Next Semantic Scale Prediction via Hierarchical Diffusion Language Models
Authors: Cai Zhou
Chenyu Wang
Dinghuai Zhang
Shangyuan Tong
Yifei Wang
Stephen Bates
Tommi Jaakkola
Venue: NeurIPS 2025 poster ()
Keywords: Discrete Diffusion Model
Theory for Discrete Diffusion
Diffusion Language Modeling
Primary area: deep_learning
TL;DR: We propose Hierarchical Diffusion Language Models, which is a discrete diffusion with a general time-varying next semantic scale prediction process for language modeling.

Abstract: In this paper we introduce Hierarchical Diffusion Language Models (HDLM) -- a novel family of discrete diffusion models for language modeling. HDLM builds on a hierarchical vocabulary where low-level tokens with detailed semantics are surjectively mapped to high-level tokens with coarse-grained meanings. In the forward process, each token is independently perturbed to its higher-level ancestor with more abstract semantics according to the scheduler, while in the reverse process the model progressively predicts the next, more detailed semantics. Taken together, HDLM provides a general time-varying next semantic scale prediction process for language modeling. We derive closed-form expressions for the diffusion Evidence Lower Bound (ELBO), and show that HDLM can be implemented in a flexible manner while including the existing MDLM as a special case. We also propose practical training techniques based on the insights. Extensive text generation experiments validate the effectiveness of HDLM, which demonstrates consistently lower validation and generative perplexity than baselines.

Next Semantic Scale Prediction via Hierarchical Diffusion Language Models
Cai Zhou! Chenyu Wang Dinghuai Zhang 2.3,
Shangyuan Tong'
Yifei Wang'
Stephen Bates 't
Tommi Jaakkola 1T
Massachusetts Institute of Technology "Microsoft Research MMila Quebec Al Institute {caiz428, wangchy) edu dinzhangdmicrosoof..o com
Abstract
this paper we introduce Hierarchical Diffusion Language Models (HDLM) - novel family of discrete diffusion models for language modeling. HDLM builds on a hierarchical vocabulary where low-level tokens with detailed semanties are surjectively mapped to high-level tokens with coarse- grained meanings. In the forward process, each token independently perturbed to its higher-level ancestor with more abstract semantics according to the scheduler, while in the reverse process the model progressively predicts the next, more detailed semantics. Taken together, HDLM provides general time-varying next semantic scale prediction process for language modeling. We derive closed-form expressions for the diffusion Evidence Lower Bound (ELBO), and show that HDLM can be implemented in a flexible manner while including the existing MDLM as special case. We also propose practical training techniques based the insights. Extensive text generation experiments validate the effectiveness of HDLM, which demonstrates consistently lower validation and generative perplexity than baselines.
Introduction
Autoregressive language models 19 36) current state-of-the-art methods for language gener- ation. However, the next-token prediction scheme fundamentally constrains their ability to revise
previously generated tokens. Alternative paradigms such diffusion models [31 13 32 have hence aroused researchers' interest due to their capability for progressive denoising and refinement. Discrete diffusion models, inherently compatible with the discrete nature of language, have recently gained popularity (20 30 26)). There are mainly two types of discrete diffusion model with different forward processes: uniform and masked (see (20 for more detailed descriptions). Nevertheless, both types have several fundamental limitations. For masked discrete diffusion, (i) all masked tokens have the same mask embeddings, lacking rich semanties; (ii) it cannot self-correct the tokens already generated, reintroducing the same non-revisability limitation of autoregressive models. For uniform discrete diffusion, uniformly perturbs each token with a random token in the forward process. Some consequent drawbacks include (i) the same token serves as noise with high probability in the noisy stage but becomes semantically meaningful when decoded, resulting inconsistency and confusion in the semantics; (ii) empirically, uniform discrete diffusion does not perform well, partially due the higher difficulty in denoising from more complex distribution.
Generalized Interpolating Discrete Diffusion (GIDD) 34 partially addresses these limitations by establishing general framework for broader class of processes, especially the interpolation between uniform and masked diffusion. GIDD combines hoth masking and uniform noise via a defined mixing rate. However, is notable that the noisy tokens including the masked tokens and uniformly
"Equal contribution. Equal senior supervision.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

[Muci]
[Nood
[iask]
[Muid]
[Naak]
[lask]
Nett
Wowter 4I
[Macc]
[lock]
(Nad
DMarki
[Chutwer
DMorkl
Sar
IClasee
[Clude 1
Diuak]
Ichadee 1]
[lakk)
Seale
(Cuntera)
IChasee
Semartic
Naski
(iaserr)
for
retuste
Blaski
Pcluewe
Senaatio
Ghewn
Kiunteri
for
giaaterr
Modcligg
Lanpage
Semvace
[Daster 4)
Next
Serrani
Idaan
Prakitinn
for
Lazgunge
Modking
Languans
1-1 Tegut
Forward proces
Scale
Predetias
Si
Medking
Backward procss
Figure I: Hierarchical Diffusion Language Model (three hierarchies are shown). (Left) When training HDLM, word tokens transit higher hierarchies independently with more abstract semantics, In the reverse process, the model learns to predict the lower level tokens. (Right) The vocabularies of three hierarchies (word, cluster, mask) and the illustration of next semantic scale prediction.
perturbed tokens- - still lack rich semantics due to their discrete nature. In addition, the self-correction ability of GIDD comes only from the uniform noise, which actually harms the performance compared to masked diffusion.
To maximize the advantages of diffusion models, namely arbitrary order generation and progressive self-refimement, we introduce Hicrarchical Diffusion Language Model (HDLM), a general and flexible framework for discrete diffusion language modeling via time-arrying next-scale prediction. Analogously to the next-scale prediction in visual autoregressive models (VAR) [33], which generate image patch tokens from coarse to fine scales (lower to higher resolutions), we introduce semantic hierarchies into language tokens (Figure D In between clean word tokens and the mask token, introduce intermediate hierarchies consisting of additional vocabularies. Each token in lower hierarchy has more detailed semantics than those in a higher hierarchy, and there exists a surjective mapping from each lower hierarchy (fine grain with larger vocabulary size) to its adjacent higher hierarchy (coarse grain with smaller vocabulary size). In the forward diffusion process, each token is progressively turned into its mapping in the next higher hierarchy scale-by-scale timesteps according to the designed scheduler. In the reverse process, the model learns to progressively predict more detailed tokens (i.e., the next scale of semantics) until the original words are recovered. Remarkably, in both forward and reverse processes, different tokens in a sentence may be in different hierarchies at a diffusion timestep, thus we denote our hierarchical discrete diffusion as time-varying next-scale prediction for language modeling, where the scale is in terms of the semantics of each token instead of spatial resolutions. Intuitively, this paradigm is consistent with the fact that some tokens are easier to predict than others, thus less denoising timesteps are needed to determine their detailed meanings in the next fine-gria fine-grained hierarchies. The flexible and adaptive denoising process inherently takes advantage of the arbitrary order generation capability of diffusion models, improving the potential surpass autoregressive language modeling.
varying diffusion
Hicrarchical discrete diffusion has several advantages over masked and uniform discrete diffusion. First, both mask and uniform noises lack rich semantics and will confuse the model, thus we introduce intermediate hicrarchies. In comparison, the intermediate hierarchies can be viewed as partially masked tokens with high-level semantics, which are inherently more expressive than the single mask token or random tokens. Next, the uncertainty contained in the vague semantics of the coarse hierarchies allows for more flexibility in decoding, which improves the decoding accuracy and provides the possibility of self-refinement; i.e., errors in previous decisions can be mitigated to some extent, while masked discrete diffusion cannot correct decoded tokens at all. Finally, compared to the inconsistency between the semantics of clean tokens and the uniform random noises in uniform discrete diffusion, the semantics in hicrarchical discrete diffusion are consistent and coherent, making it easier to accurately denoise in progressive manner.
Time-varyine next-scale prediction is novel and principled scheme for language modeling. Despite the simplicity of its intuition, we establish rigorous theories for our hierarchical discrete diffusion Markov chain (CTMC) framework. Our hierarchical discrete diffusion
based on the continuous is a special CTMC that restricts the transitions only to occur between adjacent hierarchies. We derive closed-form diffusion Evidence Lower Bound (ELBO) for hierarchical discrete diffusion training, which has clear interpretations. On the practical side, we show that the design space of hierarchical discrete diffusion flexible, compatible with both GIDD and MDLM types of training and sampling both theoretically and heuristically. Code is available at https://gitubbboomm/ooo///DDDDD

---

2 Preliminary
Discrete diffusion models generalize the continuous diffusion framework discrete state spaces 2. Specifically, given an initial discrete data point X E Z drawn from a data distribution qo (X), these models gradually degrade this data point through forward Markov chain Zo. ,,, Zr. where each state Z, evolves according to a transition kernel [r ( z 2.), starting from Z1 X and eventually arriving at a prior distribution pr (Zr), which is typically easy to sample from. We consider continuous-time Markov Chain in this paper, and select 7 1. 'he core task is then train a model to approximate the reverse transitions of this Markov chain, effectively enabling it to reverse the degradation and recover original data points from samples drawn from the prior distribution.
D3PM 3 introduces a Markov forward process q that acts independently on each token qtjs (#e|=>) = (at 2.), where Cat denotes categorical distribution, and we use 2, to denote one-hot encod- ing vectors of 2s, namely the of {x E {0, 1) MI > x; 1]. With the noise schedule chosen in [30), the marginal of 2, is g(=a|<)) [x) = Cat (z;; 0x,x + 3mm) where B = - ar and m denotes the one-hot vector of given mask token.
Generalized Interpolating Discrete Diffusion (GIDD) 34 establishes a general framework for forward processes with flexible mixing schedules. In particular, the forward process is Markov chain with marginal transitions that can be written as linear interpolation of categorical distributions between distribution n, and data distribution
ge (<x222)) = (2z (2;; 0xx + (xTT))
where x is the one-hot encoding of the data x, 3, = - a where 0 < a S 1, and T. can be any probability distribution that changes smoothly over time. Masked diffusion is special case of GIDD where TH m. By defining the mixing rate and mixing distribution as follows, GIDD the combination of masking and uniform noise:
a4
1-t C.
B
- St
m+
C,
where c (1- 1 Nec where N is the vocabulary size, and B is a constant chosen so that the desired uniform token ratio reached. The marginal forward distribution is therefore
1 C. ((1 ()x + Im c.)
However, it is notable that the noisy tokens including the masked tokens and uniformly perturbed tokens still lack rich semantics due to their discrete nature. In addition, the self-correction ability of GIDD comes only from the uniform noise, which actually harms the performance compared to masked diffusion.
3 Hierarchical Diffusion Language Model
We now introduce Hierarchical Diffusion Language Model, family of discrete diffusion models with a next semantic scale prediction scheme. We build our derivations on hierarchical discrete diffusion models, providing flexible models that compatible with prior work. See Appendix A for proofs.
3.1 Forward Process
We adopt the continuous-iim Markov Chain (CTMC) as the mathematical framework for discrete diffusion models. Instead of transporting token directly from clean data x to masks m (or vice versa) as MDL.M does 330 26), we introduce an intermediate hierarchy with expanded vocabulary to the Markov chain: x - m, The higher level tokens have disjoint support and low-level tokens with detailed semantics are surjectively mapped to high-level tokens with coarse saasse-rraieed meaning. Here = c(x) Tx is a predefined function of x (e.g., a clustering of x), where r R[[] is the matrix with [ j) = 1 if and only if c(x,) c, and 0 otherwise. The vocabulary and the mapping functions of the hierarchies can be either hard-coded or learnable (e.g., via DeepSets [39]). In our implementation, we pre-define the mappings by clustering the word token embeddings of a pre-trained model given a number of clusters; details of our "semantic lustering algorithm" are available in Appendix
3

---

By introducing this hierarchical decoding procedure, the intermediate states can be viewed as partially decoded, partially masked tokens, which enables richer semantics of the masks with moderate uncertainty. The marginal distribution of the forward process can therefore be written
ge(ze(z) = Cat(zz a,x + (x)) + Se,m m)
where Be + mm = dt. This be viewed natural extension of GIDD where
Bt.m m)
(x(x) = Cat(Tei
c(x) +
(5)
Br. 3t.m
Br, + Be,m
A significant difference in our hierarchical forward process is that the mixing distribution æ. is function of clean data x, resulting fundamentally distinct CTMC properties (analyzed as follows) as well as the new training ELBO (Section 3.3)
CTMC with block conditional transitions. We use the notation z, E R to denote the hot representation of 2t, which is the concatenation of word-, cluster- and mask-level vectors. Denote Pt|s as the cumulative transition probabilities from state 2g to state ž at times 8 < f, namely (1,(2z]z,) = Cat(z,; Pal. Z,), where the 2,-th column Ptis represents the probability when starting from state 2g. Denote the time-inhomoeeneuus generator matrix (also called the forward transition rate matrix) as Q, satisfying dt Kolmogorov forward and backward equations for CTMC state:
dæ, Q, 24, which contains instantaneous transition rates. The
iP ŌL
ÖPt= P Pale =1
P
Pijar
(6)
as
Proposition (Time-inhomoggneouu generator and cumulative conditional transition matrix of HDLM), The time-inhomogenneous generator matrix of HDLM is
[
PT
Qi =
a
LE
ICc
a a
(7)
3r,
where E 11x|C| The cumulative conditional transition matrix of HDLM is:
Ot
!vI
0
Ox
(f. as Bud
a' BE, du
BI_ Icl
Pt|s
r
0
(8)
30,0
(1
at
+
Bro
du
LE
(1
3r,
ax
Buc
B
Proposition suggests that hierarchical discrete diffusion process corresponds CTMC with block forward transition rate matrix Qt and has the mask as an absorbing state.
Generalized processes with stochastic perturbations and multiple hierarchies. At inference time, diffusion models denoise based on predictions of previous steps. The accumulation of error may result in OOD contexts compared those seen during training. To mitigate this potential gap between training and sampling, we introduce an additional cluster token corruption mechanism into the forward process. Denoting the probability that & transits to the correct cluster c(x)
& < # c(x) = (1 B),t. we assume that a word token will be
i... (c = c(x)) = corrupted into inaccurate cluster token d' * c(x) with probability 3 where N, is the number of clusters. When & 1, it reduces the standard HDLM introduced above. We hence have the following noise distribution,
B. = c(()
Σ c(c(m)
1
(1
E),,.c c+ + Br. m
N.
This perturbation mechanism helps to train the model to predict correct word tokens even from incorrect cluster tokens within inaccurate contexts with corrupted clusters, allowing the model to

---

robustly "self-correct" in a principled manner. This intuition is strongly supported by the experimental result that the generative perplexity of HDLM with 1 is 60% smaller than the baselines, demonstrating the strong generalization and self-correction capability of HDLMM
Our framework also generalizes to arbitrary number of hierarchical levels, not limited one intermediate hierarchy. Denote the word tokens as the 0-th hierarchy and the masks as the n-th, then in the forward process with
intermediate hierarchies, the tokens follow the Markov chain x - + ... C + m. Correspondingly, the model progressively decodes tokens from higher-level hierarchies with more abstract semantics to lower-level ones in the inference time. Multiple hierarchy levels allow for sophisticated control over the generation process, yet may require more careful design of the sampling algorithm. this work, we provide theoretical foundations for HDLM with arbitrary hierarchies, while leaving experimental implementations of more intermediate levels as future work. More details of the stochastic perturbation mechanism and HDLM with arbitrary hierarchy levels, including their conditional transition matrices, closed-form ELBOS and further discussions, are available Appendix A.2
3.2 Reverse Process
For noisy input 2n timestep t. the model prediction of distribution over the word-level vocabulary is xo(2,,) t) R which is shortened as X when not causing confusion. We adopt the commonly used reverse process in previous work 3 34 as follows,
La (=)=)(x)) Xe)
(=(22 = (=)2())))s)
(10)
(x)
Here qc(-(x)) := Eia, x (-(r)) where x is the predicted probability of token x;. Further-
more, we can prove that the backward rate matrix Qi of our hierarchical forward process with block
conditional transition matrix shares the same relation with the forward rate GIDD (34)
(E6, 2x) -d
EdrlE'
22)
(e(='(x0) ge (=(=x(x))
Qe zt)
(2 Xx)
(11)
qe qe( Xx)
For convenience, we omit and use Q to denote Qf while not causing confusion.
3.3 Training
Closed-form CT-ELBO of HDLM. We now derive closed-fone expressions for diffusion Evidence Lower Bound (ELBO) of HDLM based on the forward and backward rate matrix of its hierarchical process. For arbitrary noise distribution n, and schedules 0r,, Br, we have ELBO as follows (34).
Lemma 2 (Proposition H.4 in 34
For any CTMC diffusion process with marginals qe
forward rate Qi 21), and backward rate Qi(2,, 22), the continuous ELBO is
Qi(Es, 21) (<(2./z)
log 221,,20) + Q.(z,, 20) - Q2(2(, +C
logp(x) 2 E1,2,
Σ 2
g0(2)(2) Qe(e,, 21 (2)lx)
(12)
Qe(Es, =x) (22122)
log 2e (0))
= E
Σ
+
ge g((2(()) (2 (2)
(e(2)(xo)
where l. U(0, and C (og
(1))))))))) (p)))
One could casily verify that 0 for HDLM and MDLM, hence will be omitted afterwards. Now for our standard hierarchical process (with one intermediate hierarchy and without stochastic perturbations), plugging in the forward rate matrix and working through some algebra yields the following main result for HDLM:
5

---

Theorem 3 (Closed-forn CT-ELBO for HDLM with hierarchical CTMC diffusion process and block conditional transition).
logp(a)) 2 R
den
0 Br,
log
(o
+ 8
3l.m 3rm
log Σ
pa(x'))
po(a')
x'Ixx=Ix
-'=1xx=Ixx
Og 2
Pl.n
Pl.m
En,ze
+
8 820 +
82m,m]
(13)
a4
Be.m
=0
-44 x
Xo 3 Tx)
+ dec. Bl.m (Tx) log((
E den
log
(14)
Bi,
x [
Br.m
X9 O (TTxx)
= - Wi, CE(,,
sm Wmm CE(Txx,
(15)
X r
where po (x:) product, and
xol1] is the predicted probability of token Is, @ is the Hadamard (element-wise)
-a¹ Wt.m
Bl.m
Wt,c =
,i,e"
Be.m
(16)
Interestingly, Equation [3 can be interpreted combination of two cross-entropy (CE) losses corresponding to different hicrarchies. The first term applies to each cluster token, requiring the model to classify within the where word tokens are mapped the cluster. This can be viewed as if restricting the predictions of word-level tokens within the known cluster and dropping the probabilities outside the cluster; thus X9 is element-wise multiplied by r Tx and normalized by x, x, T'x, implying that the model only needs to predict word tokens within the cluster. The second term applies each mask token, which is equivalent cluster-level classification in which predicting the word tokens within the correct cluster suffices.
Analysis of training ELBO. be proved that the expectations of the weights for both the token-level and cluster-level cross-entropy loss are invariant to the schedule.
We continue with more in in-depth analysis of the HDLM ELBO. It can
Proposition 4 (Invariance of both token-level and cluster-level loss weights).
E Er,,, W.,,, E-(- =1
(17)
E2 10 mm. Er Bc. =
(18)
Proposition IS an intriguing neat property that generalization of the invariant results in MDLM 15 26 This implies that while different schedules may result in various weights We,c and Wt.m hence enabling the flexibility emphasizing different stages of the diffusion process, the expectation of each loss weight w.r.t. timesteps and samples is always fixed to 1, which is reasonable. Despite the same expectations of weights, HDLM can actually achieve lower cross-entropy loss (implying higher single-step denoising quality) compared with MDLM,, thanks to the cluster tokens to be predicted or in the context providing more information than mask tokens.
Proposition 3 suggests a standard training loss for HDLM based on the strict ELBO:
X9 C Txx
C(x,Xo,t) t) = (e We, CE(x,
+ & Wm CE(('x, Ix,]
(19)
X9 TTTTx
Consequently, we further show that MDLM is special case of our HDLM where there is only one cluster token. Despite being intuitive by construction, we can prove that HDLM ELBO reduces to MDLM's, s, which further validates that our hierarchical discrete diffusion is a generic framework compatible with previous work but provides more design space.
Remark 1. MDLM is a special case of HDLM where there is only one cluster whose embedding is the same as the mask.

---

y=1 1
y=2
y=
Act
R.
Rcc
A.m
Pcm
Pc.
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 2: Marginal probabilities of the example forward processes with at = (1 - 1)7 and 7 = 1, 3.
y= 1
y 2
V= 3
10
10
10
Wt.c
8
8
6
6
w-
We
4
4
3
2
0.0
0.4
0.8
0.0
1.0
0.6
1.0
Figure 3: Cross entropy weights of the example forward processes with a, (1 (1-1) t and y 1,2, 3.
3.4
Example Processes
We now give some examples of forward processes. While the schedules are arbitrary, we consider the following transition process. When a token becomes a cluster token at time т, will become mask token at time lm,r ~ U(T 1). Thus the rate of cluster token and mask token follow,
Be,c - 1- (1 Br,m
1 11-7
(20)
With 04 = (1 t)?, we have the following general expression (visualized in Figure 2 for y = 2))
0r = 7 {- 7 - t1 (1- 3e.
(76 (+ (1 11
{ 7
{ 1n(- 1), =1
(21)
t (1-+))n(111))) =1
Therefore, the weights Wr,e, Wm.m can be computed as (visualized in Figure 3
(7- 1) (1 1)2-2
7(1 - (1 (1-
,7 +
(1 1)
yt + ty
Wi.c
Wimm
(22)
In(1 - t)
- - In(I1n11 1t 7
t t (1 - t) In(1 tt 7/=
=1
3.5 Improved Techniques for HDLM Training and Sampling
Based our theoretical insights, we continue to analyze the HDLM training and sampling processes, proposing more practical techniques to improve the model.
Gradient of cluster-level loss. Compared with MDLM, the cluster-level cross entropy loss in HDLLM is additional. However, the loss at the cluster level would encourage predictions within the same clusters but not exactly the target word token x, causing the gradient to hold against optimizing
7

---

the cross entropy at the token level. In particular, consider the negative gradient of the cluster-level loss w.n.t. X2( t),
Exo Wm (T(TT,TXxxxxx (xx(=x;
m r (Tx x TXg
(23)
This suggests that the gradient descent would encourage Xo predicting all the word tokens within the cluster {a' c(a²)= = c(x)) even if x' are not the correct word token æ. Intuitively, cluster prediction an "easier" task, but actually benefits the model through an easy-to-hard curriculum and multi-stage decomposition: the model is allowed to make (moderate) mistakes in the earlier mask-to-cluster stage, as long as it can be corrected in the later cluster-to-token stage. To verify this hypothesis, we introduce a hard training mode, where we replace the cluster-level cross-entropy with the token-level cross-entropy (within the whole vocabulary) during training, leading to harder task:
Li (x, Xo, 1) = Oe Xa ④ (T (TT))) + 52 Wr, CE(x, xo) TT
(24)
It is straightforward that CE(X, > CE(Tx, Ixx),, so hard training mode actually utilizes a looser bound of the elbo as the training loss; the calculation of evaluation elbo remains the same. As shown in 5 the hard training mode indeed negatively affects the performance slightly, suggesting the advantage of progressively denoising and robustness of the original HDLM formulation.
Auxiliary loss. We apply the reverse process derived from the Bayesian perspective, the model always outputs the word token prediction X (z, t) for all inputs at different noise levels, and the cluster-level prediction is derived through the Bayesian posterior Ix2 instead of using another classifier head to directly predict the logits of clusters. As an ablation, we append an auxiliary cluster-level classifier head to the model which outputs cæ (2,,,)) t). Together with the main network, the auxiliary classifier is trained with the original elbo and an auxiliary cluster cross-entropy loss CB(Tx, co zz; ()). We observe mild performance drop when using this direct cluster prediction instead of the Bayesian posterior (Table 5). demonstrating the advantage of self-consistent elbo training and posterior inference.
Flexible loss weights. We also find that clipping the large and extremely small loss weights based on heuristics help in stabilizing optimization and improving performance. Actually, such weight manipulation techniques can be commonly found across broad diffusion literature [14 15 34]
Intuitively, these weights can be thought of hyperparameters over the noise schedule, which control how much effort the model devotes to the corresponding tasks during training. A careful balance, ideally through empirical tryouts, could often lead to better results 21
Force transition in decoding. When predicting the word token from a cluster token, the naive model prediction xy has logits for all tokens even outside the cluster. To keep consistent with the forward process,
optionally restrict the model decode the cluster tokens only to the word tokens within the clusters, which we call force transition (within the cluster). We adopt force transition decoding strategy by default and find it consistently helpful; see ablations for more discussion.
4 Experiments
4.1
Experimental Setup
In our experiments, we focus on language modeling, training the proposed HDLM algorithm on the widely used OpenWebTexx (OWT) [ dataset following [34] We adopt the settings in prior
work 26 34 20 30 in terms of architecture and scale DIT (24] of small (170M) and base (425M)
sizes with GPT-2 25 tokenizer on 131B tokens of training data. We take 100000 data out of 8013769
data as the validation set. Following 34 we use a context length of 512 tokens and do not use sentence packing. We evaluate the trained models via both validation perplexity (Valid. PPL. and generative perplexity (Gen. PPI.), which are standard metries in recent literature, Other details and additional results are deferred to Appendix B
8

---

Table 1: Validation perplexity and generative perplexity on OWT. The numbers of trained tokens are also reported for fair comparison. Our small models consistently surpass other discrete diffusion counterparts, and our base model is able to match the performance of autoregressive models.
Model
Train. toks.
Valid. PPL (4)
Gen. PPL (4)
GPT2
25
unk.
23.40
Llamal 10M (retrain.)
262B
16.11
SEDD
(20
262B
224.10
MDLM-small
26
(reimpl.)
131B
$27.39
163.7
SmDl+
34
(reimpl.)
131B
$25.82
170.2
HDLM-small-64
131B
$23.36
144.2
HDLM-small-
131B
$23.25
148.0
HDLM-base-Iee
131B
$19.22
139.9
4.2
Main Results
For Valid. PPL., we report the perplexity on the OWT validation set. For Gen. PPL., we generate 256 samples with length 512 using 512 denoising steps, and leverage gpt2-large [25] as the reference model. We include both autoregressive models and the main discrete diffusion models 20 26 34 as baselines, and the results are adopted from the corresponding work (marked with or reproduced in our settings (noted with reimpl.) to keep consistency. An HDLM with clusters is denoted as HDLM-n, and we also attach their sizes. Our main results are shown in Table Remarkably, HDLMs with small size outperform other discrete diffusion variants in terms of both validation perplexity and generative perplexity. Moreover, our base HDLM achieves 19.77 perplexity, surpassing or matching the autoregressive models.
Table 2: Ablations of HDLM with different numbers of clusters n. We report the validation perplexity and generative perplexity OWT, and all models are trained for 131B tokens. HDLM with cluster numbers around 64 and 128 achieve the best performance, while n = 1 recovers MDLM performance.
Model
n
Valid. PPL (L)
Gen. PPL (L)
<25.72
163.9
2
<25.79
160.2
4
$25.05
162.0
8
<24.57
146.6
HDLM-small
16
$24.17
155.2
32
<23.83
154.6
64
$23.36
144.2
128
<23,25
148.0
256
$23.65
150.4
4.3 Ablation Studies
We conduct comprehensive ablations on HDLM w.r.t. the number of clusters n. (Table 2) the
perturbation probability & (Table 3 the forward process noise schedule controlled by (Table 4
and different training modes (Table Without specifying, use HDLM-small and set 64, E
1, = and standard training mode by default. Full results are presented in Appendix B
One could observe in Table that as the number of clusters increases, the validation PPL first increases and then decreases, suggesting an optimal and moderate range for suitable numbers of clusters: approximately the square root of the vocabulary size which divide the generation into two stages with approximately equal complexities. We also verify empirically that HDLM with one cluster recovers the performance of MDLM, which is consistent with theoretical analysis.
As illustrated Table stochastic perturbations introduce harder denoising target in training, which improves the model's ability to deal with inaccurate contexts and thus benefits the real inference

---

Table 3: Ablations of HDLM-64 with different perturbation probability ç. We report the validation perplexity and generative perplexity on OWT, and all models are trained for 131B tokens. All models are forced transition within the cluster, which significantly reduces the Gen. PPL when & < 1.0.
Model
Valid. PPL (L)
Gen. PPL (4)
1.0
$23.36
144.2
HDLM-smal1-64
0.9
<23.54
69.76
0.8
$25.93
54.15
procedure: the generative PPL reduces over 51% with ç 0.9 and over 62% with ç = 0.8. Intriguingly, force transition decoding strategy is effective even for & < 1. Intuitively, force transition would lead to better performance when the cluster prediction is accurate, yet otherwise will increase the accumulation prediction error. Our empirical results imply that the robustness to inaccurate contexts may be more important than correcting the cluster tokens in-place. For reference, the Gen. PPL is 184.0 for & 0.9 without force transition.
5 Related Work
Researchers have been looking for alternatives of autoregressive models for text despite their great success, among which diffusion models are representatives. Early work in this line often leverage continuous diffusion [8] or combine diffusion with AR models [22 I0 Building on foundational
work of discrete diffusion [3] more recent work such as SEDD 20 MDLM (26]. MD4 [30] demonstrate the effectiveness of discrete diffusion in text modeling in small-scale pretraining, which is also our focus. In particular, SEDD 20 learns the concrete score (as the ratio of marginal distributions), while MDLM [26] and MD4 30 further simplify the training objective for the masking forward process. Larger-scale work LLaDA (23] and Dream (38] demonstrate the scaling potential of diffusion language models, matching state-of-the-art AR models with the same scales in practical math reasoning [11, coding [1I, or multimodal tasks 37 Most of the aforementioned recent discrete diffusion language models are masked diffusion due to the simplicity and empirical effectiveness, yet at the cost of losing self-correction ability. While there are also discrete diffusion with uniform noises [29 27]. they typically underperform the discrete counterparts. GIDD 34 unifies the two types by establishing a general framework that combines both noises. Our paper, by contrast, introduce a novel noising procedure that is both conceptually simple and practically effective, while preserving the self-correction ability.
335 [ further investigate the integration of discrete 2 explores another new paradigm to autoregressively generate text blocks and diffuse text within each block. Our work acts a different direction, the length of sequence is fixed but the semantic of each token is generated sequentially.
diffusion with AR models. Block Diffusion
6 Conclusion
We introduce Hierarchical Diffusion Language Model (HDLM), generic and flexible family of discrete diffusion model with hierarchical semantic structures. Intuitively, HDLM follows a "next semantic scale prediction" scheme to progressively decode high-level abstract tokens to low-level detailed tokens. Leveraging the continuous time Markov chain framework, we elaborate on the forward and backward process of hierarchical discrete diffusion, derive strict diffusion training ELBO and formally demonstrate their properties. Based on theoretical insights and empirical observations, we propose design principles and practical training techniques for HDLM. Our extensive experiments demonstrate the strong language modeling capability of HDLM.
Limitations and Broader Impact
Hierarchical discrete diffusion is novel and general framework for discrete data generation. However, in our implementation we only consider one intermediate layer, while extending multiple hierarchies natural. We also leave exploring applications of hierarchical discrete diffusion to other areas besides language modeling (c.g., image generation, biomolecule design, and decision making) future work.
10

---

Acknowledgment
C. Z. and S. B. were partly supported by the ARPA-H ADAPT program. C. Z. C. W., S. and T. J. were partly supported by the Machine Learning for Pharmaceutical Discovery and Synthesis (MLPDS) consortium, the NSF Expeditions grant (award 1918839) Understanding the World Through Code, and the DSO Singapore grant on next generation techniques for protein ligand binding.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 1rXiv:2303.08774. 2023.
[2] Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503. 09503, 2025.
[3] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 14:17981-1799 2021.
[4] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. arXiv preprint arXiv: 2019.
[5] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv: 131.v0131 2013.
[6] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACI-HLTT, pages 2924 2936, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.
[7] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint
2018.
[8] Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skyylion007, hatp:///1iio00oiiihhhb / OpenWebTexttCrruus 2019.
[9] Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410.1789) 2024.
[10] Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933, 2022.
[11] Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. Diffucoder: Understanding and improving masked diffusion models for code generation. arXiv preprint arXiv:2506.200 20639, 2025.
[12] Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-Im: Semi-auutreerressvve simplex- based diffusion language model for text generation and modular control. arXiv preprint arXiv:2210. 10..222017733 17432, 2022.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[14] Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 24174-24184, 2024.
11

---

[15] Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems, 36:65484-6555 2023.
[16] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprini urXiv:1412.6900 1412.6982 2014.
[17] Shufan Li, Konstantinos Kallidromitis, Hritik Bansal, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Jason Kuen, Zhe Lin, Kai-Wei Chang, and Aditya Grover. Lavida: large diffusion language model for multimodal understanding. arXiv preprint arXiv:2505.16839 2025.
[18] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion-Im improves controllable text generation. Advances in neural information processing systems, 15:4328-4343 2022.
[19] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 1rX4v:2412.19437, 2024.
[20] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310. 16834444, 2023.
[21] Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models. arXiv preprint arXiv:2410.1108/ 2024.
[22] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In Proceedings of EMNLP, pages 2381-2391, Brussels, Belgium, 2018. Association for Computational Linguistics.
[23] Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models, 2025.
[24] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF international conference on computer vision, pages 4195-4205 2023.
[25] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI, 2019. Accessed: 2024-11-15.
[26] Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:130136 36-133116110 30184, 2024.
[27] Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang. Justin Chiu, and Volodymyr Kuleshov. The diffusion duality. arXiv preprint arXiv:2506. 10892, 2025.
[28) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.106/ 2019.
[29] Yair Schiff. Subham Sekhar Sahoo, Hao Phung, Guanghan Wang, Sam Boshar, Hugo Dalla- torre, Bernardo de Almeida, Alexander Rush, Thomas Pierrot, and Volodymyr Kuleshov. Simple guidance mechanisms for discrete diffusion models. arXiv preprint arXiv:24/2.10199. 2024.
[30] Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37:103131-1001177 2024.
[31] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. ArXiv, abs/1907.05600, 2019.
[32] Yang Song, Jascha Narain Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Er- mon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ArXiv, abs/201 1.3456,, 2020.
12

---

[33] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances neural information processing systems, 37, 2024.
[34] Dimitri von Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf, and Thomas Hofmann. Generalized interpolating discrete diffusion. arXiv preprint arXiv:2503.04482 2025.
[35] Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovec, Stefano Ermon, and Arash Vahdat. Energy-basee diffusion language models for text generation. arXiv preprint arXiv:2410.21357, 2024.
[36] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.1511 15115, 2024.
[37] Ling Yang. Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505. / 15809, 2025.
[38] Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint ar 2025.
[39] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
[40] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 4791 4800, Florence, Italy, 2019. Association for Computational Linguistics.
[41] Fengqi Zhu, Rongzhen Wang. Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, et al. Llada 1.5: Variance-reduced preference optimization for large language diffusion models. arXiv preprint 1rXx::2505..9223, 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims are reflected in narrative and experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Include in the Limitation section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Assumptions are in the main text and the complete, correct proof are in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Provide experimental and implementation details; describe the algorithm and the model clearly.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Datasets are publicly available. We provide our code publicly.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttpcccpppppppppeeeeeeeeeeeeeeee httpa///guddeee/eeeeabiiissssss /Cc////oddiiisssseeeeeee ssiosolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Include all the experimental details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results stable and our evaluations are conducted with sufficient sample sizes.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Describe the computational resources in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We follow the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discuss the broader impacts the broader impact section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We admit the risks for misuse in the limitation section.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We give credits and cite them and respect the license.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets including code, data and model are publicly available with proper instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This paper does not have human subjects crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not have human subjects or crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is only used for polishing writing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
20

---

A Proofs
A.1 Proof and Analysis of Standard HDLM ELBO
Proof of HDLM ELBO. We start with proving our main result: the ELBO standard HDLM.
Theorem 5 (Closed-form CT-ELBO for HDLM with hierarchical CTMC diffusion process and block conditional transition, Theorem in the main text).
po(x)
fl.m 8
log p(zp 2 Er, 83, Bie
log
log
Σ
po(x')
Σ
po(a")
3t,m
x'Txx=Txx
-xxxx=Ixx
dg SEL +
Blom Be,m
Er, att
S2n +
32.ml
(25)
Be.
82 -aj T log
X9 (x)
Bl.m (Tx) log(Txo)
=Ei,
+ 82
(26)
Be,
Xg 1'
Be.m
X9 2 (1' ('x)x 82 We CE (x)))
-- W CE(x,
(27)
x TTTX
where po(x;) = xoli] [i] is the predicted probability of token Ig @ is the Hadamard (element-wise product, and
-a6 Br,
pt.
Wt,c
Wa.m
(28)
Bt.m
Proof.
By Lemma 2. we plugin Qe in Equation into Equation Equation (*) for different 24 {ar c(z), m) respectively:
12 We calculate the following
Qe(2si x (c(2zla)) (r) log (e(2, Xo X1) (2
2e(2' xx We Xx
Σ Re 2lzz
Σ Qi(x' =e)
(z(l de (o) (z)
(i) only when 2, c(x) that Qe(2., 22) + 0 and (((2)lz)) + 0, thus the first term being nonzero; when z' E c, the second term is nonzero.
Be Σ
po(x')
3r. Σ EEE
Σ
po (o
-aa4 Blia) Br. log
T'Ix--IX
Br,m
-(d, 32,a)
2'E'''TT'==x'
Br,
B
Be,m=
3r.
Br,
Be,m 1
(29)
Bl.m
logg Σ T'
po(a') -1
(30)
Br.m
where we utilize the facts that - (a,, + -3,,)) = Pm,. and Ex' Ex p0(x)==1 1.
(ii) 2t = c(x), the first term is nonzero only when 24 = H, and the second term nonzero
z' E (T='= = c(x), E V or z' c()), therefore
Σ
pola')
a O4
* at at log
aspo (z)
Bee -
x' T'x''
Ix
24 + Br
ai Br,
Bi,
Σ
Po(x')
Og Br
po(x')
Bee
='Xx==Xx
(31)
-0² log
Po
81.c
(32)
3r,0
Σ
po(z!)
Br,c
Txxxx
21

---

(iii 2 2. the first term is always zero since Qi 2x) = 0 for all 2n # 21, and the second term is
nonzero only when z' 2t 2, hence
=
(33)
at
Combining all three categories, have
82 Fl.m
log
E
po(x')) -1] + 82..e
-a'
log
pa(x)
+
82
di
Be,m 2'xx'=Ixx
BI,
po(x')
3e,
Txxx=Ix
(34)
where we simplify (()) to 82,, since the only cluster token that 2, could be is c(x). Taking the expectation yields,
Bl Br.m
-a4 log
po(x)
Ee [+] -e, deg
log
pa(x'))
((()
(35)
Ba.
Σ
po(a')
='Ix==Ix
=1xxx'1xxx
de
Pr.
Bl,m Be,m
8 22
+ 8 82
8 8eam
(36)
at
Be,
The second term has an expectation of zero:
Ez E2iaa(zsla)) (8n.. de + dean Be,
Bl.
Pl.m Be,
Eg
+ 8
(37)
Es at de
Br. + Be,m Be.
Br.m
aq
3e,
Be.c
Bec
(38)
(at + 8/,c + Pt.m)dt
(39)
1 Bt.m 1
0t
+ Br,0
(40)
1
+0+1
(41)
-0
(42)
The first term can be simplified into matrix form which is equivalent to weighted sum of two cross-entropy losses:
L. log Σ po(a') + SEn -a4 log
po(x))
Er,2,
82mm
(43)
3t,m
Br.
Σ
po(ar)
2'xxx==xx
al x log
X8 3 (Ix)
I'x)
Fb.m (Tx) log(Txo)
= -,,,21
821 Br,a
- Sasmm Br.m
(44)
X, [TTx
Xy ④ (xT)x) + 8z Wem CE Txo)
E2,2,
5 CF
[xo)
(45)
x, I
where (0())) product, and
xoli] is the predicted probability of token 2i, @ is the Hadamard (element- wise)
-0²
B
Wi.c 3r,c²
Wt.m
(46)
Bi.m
We therefore conclude the CT-ELBO for HDLM:
X9 O (T))x),
logp(x) 2 -Erse b W2
Wtmm + S CE(Ix, l'xe)
(47)
x I 1'x
The conclusion that MDLM is special case of HDLM directly follows.
22

---

Corollary 6. MDLM is special case where there is only one cluster (which equals to the mask).
Proof. We need to show that when the neural network parameters are identical, our MDLM ELBO
reduces to MDLM. According to the HDLM ELBO in Theorem
log)(x) $ -E,,za W0 W Cx Xo O (T)
2z.m Wm CE lx,)
(48)
Tx X, Tx
Here all word tokens belong to the same cluster, i.e., r 1x thus it always holds that the second term is zero:
CE(Tx, [xp 1) = 0 we have the following derivation:
(49)
For the first term in Equation
13
Er,,, CF Xg O
Tx) )
}]
TE,,, (2,, xo) E. [ CFFxx xo)
(50)
Xg T
which is the same as the MDLM
26
Actually, an interesting fact is that MDLM is also equivalent HDLM with [V| (vocabulary size) clusters. Hence, any cluster number in between and [V] implies a generalized model with additional benefits compared with MDLM, and it reasonable to observe an optimal cluster size (around 64 to 128 in our experiments).
Analysis of the training ELBO for the example forward processes. We now give more analysis on the example forward processes stated in the main text note that these are only the schedules used in our experiments, while HDLM supports more flexible schedules. Consider the following transition process: when a token becomes cluster token at time it will become a mask token at time tm.r (T 1). Then in the forward process, the rate of cluster token and mask token will be
Bec S/¹ 1-7 (-da))
1
(51)
Be. 1' t [--
(52)
With or (1 - t)7, we have the following general expression:
(1 t- (1-1)),,7
0r.c
(53)
-) In(1 1), 7= =1
1
(7t (1 -)² #)
Bm
(54)
t t (1 t) In(1 - 1
Therefore, the weights wt,,, Wt.m can be computed as
-)7
(7 1) (1 t)--T
-a¹
Whe
(55)
Brc
1
-(1 1) In(1 - 1) 7
11 (1 1))-1)
7 + (1 - t) #1
Wt.
(56)
3e,m
In(1 - t)
(1 1) In(1 1)
When the schedules follow the common practice that a, is always decreasing w.r.t. t and 3rm is increasing, we have of S 0, 8m 2 O. leading to non-negative weights:
Wi, 2 Wr,m 2 0, Vt [0,
(57)
23

---

While Proposition 4 guarantees that the expectations for token- and cluster-level loss weights are 1_ may not be finite. In particular,
finite, their limits at 04 and
lim Wr.e +00, if lim of + (
(58)
100
lim Wr,e +00, if lim ae ( 0
(59)
lim Wt.m + +00, if lim Br.m # 0 2-00
(60)
6-0+
In typical schedules, these conditions tend to be satisfied for instance, the linear schedule ag = 1 - t and a -1, resulting in infinite weights. However, it is reasonable for Wo,e and wo,m to be infinite, since all the cluster and mask tokens must be fully decoded to clean word tokens when
t= O. Moreover, the dominators of the weights Be,es Bt.m exactly the occurrence probabilities of cluster/mask tokens in the forward process, thus we have
= -a
(61)
2z 2m Wr, m] fm,m
(62)
which are finite as long as the time derivatives of a₁ and ft.m are finite as in common practice. The above analysis suggests high degree of freedom in designing forward processes, as we obtain proper training ELBOs in most cases under mild conditions.
A.2 Extended Forward Processes
HDLM with stochastic perturbations. We now derive the ELBO for the case with random perturbations the cluster tokens (i.e., E 1).
Theorem (Closed-fore CT-ELBO for HDLM with stochastic perturbations).
8i,m
(5 Io (po(c(z)) + E(Ne -))
1-
log p(x) 2 Er,,, 82.m Se.m
[1 (-po(c(())]]
--6
Σ log (E(No- -11
po(za) + - po po(2x)l)
No -
1-f
=x=c(((x)
-a1 log
px(
+ 8 )(
Be,
(o(2))
E(N. TT- po(=)l
1-g
pe (x)
+ )2,0)((2)
log
(63)
Bro
E(N.-1) 1-E po (z+) + pp(()))
Here we use the notation po(zs) Exgedn)== (x C' to simplify the expression. To
interpret the ELBO, denote 77 (N.-1)E which is usually greater than 1 for reasonable values
1-E
(since should not be much smaller than 1). lin particular, when & 1, 1 oo. Then Equation (63) simplifies to:
log p(z) 2 Ee,z, 82 Fl.m g log (11 - (11()))((()))))))))))))))))))( + m =.=e(o(a)
=) +
Σ
log (11 - (z)) + +1)
Br,
de
po(x)
-af log
(o(x))
S2),(()) Br,a (1 n (o((2) +
log
+ 8 d2,,))e(e)
1
Brd (1) 1)po (zx) +
(64)
Proof. The inhomogencous generator Qt,g can be obtained by replacing in Equation by D'g. which is defined by substituting every 1 in I' with & and every 0 with
N. -. We continue to calculate
1-
Equation (*).
(i) the calculation is the same those in Theorem
04
(65)
00
24

---

( 2 = c(z), which means the word token transits to a correct cluster token. Denote pa [c(x))) = Σ -r(xx=((() Po (a)),, then the second term in Equation (*) needs to sum over all of E V and
= = = c(z))
Σ Qelz', zt) (1(="|xx)) -ai (rEpo(c(x)) + Oit at (r,cEpo(c(x))) + Se,c 1-f N. po(c(x)))
- -1- af a the
qe (2)
Br,
BI,
(66)
The first term zero when 2s any of cluster and mask tokens (since Qe,e c(x))
when
2/ * c(r), and zero for any word token that is not since qe(2z O when 2, {V)z). Thus the first term is
(0(2)(a)
Σ (2(2z:2)) 20)
log qi
(67)
(<(2)(z)
2,
df at log
Po(z)aa
5.1.0
(68)
at
Br,ch
[po(c((=) N. (1 po(c(<)))
or
de log
(x(
(69)
3r,
po(22) + n1
(1 (1-po(z))) po())))
(ii) 2 E {C)c(z)), which could only happen when & < 1. The second term in Equation (+) can be computed as:
Eul'' [6(=") (<)=
xo) -a' arEpo + a6 N. (x(2,(x0) ar (e,cEDo(z2) -t,
1-6 1 (1 (1-po(2))) al + 4...
PL.
1-5 (1 po(21))
Be,
BEc
(70)
Analogously, the first term is only nonzero when 2. = v.
(<(2)(z)
log (0(2((x0) 4: ( (2e
Σ Qe(zs: 2)
(71)
(x()(())) 2o 2e Xv) )4: 2.
a4 (1 E)
p=(x))
(1-5)
3r,
log
N.
(72)
0q No 1
Brie N.
[1-8)
{po(2s) + No (1 - (p)()))
1-8
ax
at log
(x(
(73)
Br,
(pp(zz)) + (1 pz()))
(iv) 2 m, the second term sums over all cluster tokens,
E O.(5,s 21) (e (xx((xx))
[xo)
+ Bro Eveo
+ N. N (1 po(2'))
By.m
(74)
Le (z(2((o))
Br,
Br.
3e,
where we leverage Evec Po(2 1. For the first term, we need to sum over all cluster tokens whether 2, c(x) or not,
Σ Qe(2s, 21) (2(2))))) 9o (x()(())
2e (2,2,1x0) log
(75)
qe (0(2 (2.l2z)
2
-(a + (%,0)
Br,ce log
(po(c(x)) + (1 po(c(x))) N. ) Br.m
(76)
Be,e
Br
Br.m .
Brok
I-c Br. N.-1
(po(ze)e + (1 po())))
No )
I-E
1-
Bt.m
log
Br,m
Br,m
Bi,
1-E
=x=((((x)
Nc-1
Bl.m & log (po(c(x)) + -111 -po(c(=))))
1
Σ
log (npo(zs) + +11 po(2)))) (77)
+ "7 =,=o(((x)
Be,
17
25

---

Combining all types, we have
B [+]
Be
8 m
Flom Bt.
{
log (11- po(()())) + )
Σ
log (17 1)po (2e)
17 " =x=((((
-0y log
pa(x)
-aé log po(
8e o(()
+ d (s(e))(())
Be,c
(1- 1 p)po(zz) (ze)
Be,
(n (2)
Blom Be.m
Bro
+ dec,
+ 0z
(78)
Be,c
Be,
a4
For similar reason, one could easily obtain that the expectation of the bias term zero, hence we conclude the proof.
One can verify that when ç = 1, the ELBO reduces to the original version without perturbations in Theorem 3 The ELBO actually encourages correct mosk-to- prediction (through the first term) and correct cluster-to-toke prediction (through the second term), as well as out-of-cluster exploration when the current cluster state 2; is inaccurate (through the dominator of the last term).
HDLM with arbitrary hierarchy levels. For consistency and simplicity, suppose the clean tokens are the O-th hierarchy, and the mask token is the n-th hierarchy. Denote (x)) as the mapping of clean token to the i-th hierarchy. Let o4,; be the marginal schedule for the i-th hierarchy, We assume = 1.0 for simplicity, then the ELBO is given as follows.
Theorem Closed-form CT-ELBO for HDLM with arbitrary number of hierarchies).
n
Ejer d's log
Entres 1(a)== 1(2) Po po(03) Eatr Po(x") (
logp(z) 2 Er Σ 8ect
(79)
where 8 an indicator function that denotes whether 2, the i-th hierarchy.
Proof. We start with calculating the time-inhomoogneous generator for HDLM with n hierarchies. By recursion method, we have
oh 0c.0
TIV
IT
to
ahotol11 00.1
0
0
0
deea
Qt,
Ey=d ais I
mn-
0
0
T
Q
0
0
(80)
where T2 E [RCC41 maps tokens in the i-th hierarchy the i + 1-th hierarchy.
Analogously to the proof for Theorem 3 according to Lemma 2 plugin the above Qt into Equa-
tion 12) yields the following calculation:
ii (namely 2, = x)) the first term in (*) is zero, and the second term is nonzero only when
= thus
0
(81)
04,00
(ii) < < a. (namely 2, = (x(x) the mapping in the i-th intermediate hierarchy), only when
2g (z)(z) (z) that the first term is nonzero, and only when z' cx(z) or (2' E C-11
26

---

I 1i-1z' ze} the second term is nonzero, hence
-jjei ags Otii-l
( (00))--(==)=aa-le Po(x')
C111111
log
01,i-1 azii
at,
Ex"r (aa""aa""))aaa Po (2²)
Djsrah.
Ejsihh
(82)
01,1-1
at
Ejei Oes log Σ Ex"ra(t")==a())) po(x"))
(0 po(x')
02
(83)
(iii) (namely 2 m), only when 2, (2) that the first term is nonzero, and only when E {2' E Cn-1 nnn = 21} the second term is nonzero, therefore
* -jjen O/S arna- log
0it,n11 E 10') (00)) po(x')
ar.
at
at,
acn Ex"ren po 20("")
Ejen ann
at,a-1
(84)
0t,n-1
Or
Ejen O'n log
(0')en-(()))= Po(x)))
(85)
Extren Po Po(x"")
at,
where we again leverage Combining all the terms, we have the constant bias term
Ejen dhs = annn
E
Σ d.)
Σ
di
(86)
1=0
04
$=0
J
We therefore have the remaining weighted sum of cross-entropy losses:
Ejes Shis
E Eatrer Earte(or Po(x
((x) Po Po
logp(z) 2 Er
Σ
log
(87)
Qti
1=1
The result shows that the ELBO of arbitrary hierarchies has clear interpretation: it can be decom- posed into a weighted combination of cross-entropy losses on each hierarchy, where the model learns to decode token on the i-th hierarchy into the correct detailed token in the i - 1-th hierarchy. We believe this is novel and highly nontrivial theoretical contribution which could inspire more future work.
B Experimental Details
B.1 Experimental Settings
We develop models based on the DiT architecture [24] and use the GPT2 25 tokenizer. We train
two model sizes, consistent with previous research 34 26] SMALL (containing 12 layers, 12 attention heads, and a hidden dimensions of 768, totaling 92. 1M non-embedding parameters), and
BASE (containing 24 layers, 16 attention heads, and hidden dimensions 1024, totaling 321.2N M2 non-embedding parameters).
Following the setting of [B]] all models are trained with context size of 512 tokens and batch size of 512 for 500k steps, resulting in a total of 131B training tokens. We utilize 8 NVIDIA A 1100/H100 80GB GPUs and employ mixed precision training in bf16 format. For optimization, we use the Adam optimizer [6 (6 = (0.9, 0.99). = 10 10-9) with learning rate of 5 x 10-4, The learning rate is warmed up linearly for the first 10k steps and then decayed using cosine schedule to 10% of the initial learning rate. We apply weight decay of 0.02 and gradient clipping to norm of 1.0. We clip the largest value of loss weights Wt,m, We to 2.0 or 10.0 in training for stable optimization, yet do not clip while evaluating the elbo for fair comparison.
27

---

To ensure training stability, all denominators in the loss and ELBO weights were clipped to 1 x 10 For sequences exceeding 512 tokens, we randomly selected a 512-token window. Shorter sequences were padded to 512 tokens; these padding tokens were included in the loss calculation but excluded from the ELBO..
For downstream performance evaluation, we follow the practice in GIDD 34 and use the eval-harness We incorporate a custom model that estimates per-token log-likelihoods using the ELBO. Our evaluation focuses on likelihood-based multiple-choice tasks, where the per-token likelihood is calculated across both the context and the completion, but not the padding.
However, it remarkable that validation perplexity is not an effectiveness metric, as models with larger vocabulary sizes like ours naturally distribute probability mass across higher perplexity. Moreover, the elbo is only an upper bound of the exact likelihood, and our complex forward process potentially leads to loose bounds and thus not directly comparable. However, HDLM still achieve much lower perplexity than baselines with smaller vocabulary sizes, which further verify its superiority. Generative perplexity also imperfect
tokens, leading to
measures the alignment with the reference model, and simple, repetitive sequence tend to have a small value. Despite these limitations, we still report Valid. PPL and Gen. PPL as common practice in recent literature.
B.2 Additional Results
Ablation studies. In the main text, we already provide the ablation results of cluster number n and stochastic perturbation E. The results w.r.t. the forward process schedule and training mode are further presented here.
Table 4: Ablations of HDLM-64 with different schedules parameterized by r. report the validation perplexity and generative perplexity OWT, and all models are trained for 131B tokens. Intuitively, larger y introduces harder forward processes with worse validation PPI. correspondingly, but leads to better generation quality in full inference.
Model
7
Valid. PPL (4)
Gen. PPL (L)
1.0
<23.36
144.2
HDLM-small-64
2.0
$25.01
138.3
3.0
<130.32
135,9
Reported in Table are the results for HDLMs with different forward process schedules controlled by y. Intuitively, larger y lead to a harder single-step denoising task for given leading to larger validtion PPL. However, HDLM with larger would decode less tokens in the initial stage and decode more tokens when the context becomes richer, which improves the generative perplexity (i.e., the real-inference performance is enhanced). This observation is consistent with other literature that decoding more tokens in the later stage (namely at small 1) is helpful.
Table 5: Ablations of HDLM-64 with different training strategies. We report the validation perplexity and generative perplexity OWT, and all models are trained for 131B tokens.
Model
Train Mode
Valid. PPL (L)
Gen. PPL (L)
<23.36
144.2
Auxiliary
<24.36
151.9
HDLM-small-64
Hard
<25.38
151.9
Auxiliary+Hard
<,5.21
143.2
Results in Table suggest that the harder training task would slightly hurt the performance. This validates that the next semantic scale prediction paradigm of HDLM effectively decomposes the generation process into stages with balanced difficulty, bringing performance gain.
28

---

Table 6: Performance on LMIB dataset.
Model
Valid. PPI. (1)
AR Transformer (33B tokens)
22.32
BERT-Mouth
142.89
D3PM (absorb)
37..9
Diffusion-LM.
<118.6
DiffusionBert
$63.78
SEDD (33B tokens)
$32.79
MDLM (33B tokens)
$27.04
HDLM-64 (33B tokens)
$26.95
Results on LMIB. We additionally pretrain our model on LMIB [5], smaller dataset that is popular for text generation. Following [26). we set the sequence length to 128 but without sentence packing (as in (44.. We train HDLM for 1M steps, resulting in 33B tokens. The constant learning rate set to 3 x 10-1 with 2500 warmup steps, and other hyperparameters follow the settings in OWT pretraining. As shown in Table 6 HDLM is able to achieve better performance than SEDD and MDLM baselines. Notably, the improvements on OWT is more significant, suggesting the ad advantages of HDLM could be more obvious in difficult tasks.
Table 7: Zero-shot benchmark accuracy on downstream datasets.
Size
Model
Train. toks.
ARC-e
ARC-c
BoolQ
PIQA
OBQA
WinoG.
Avg.
SMALL
GPT2
unk.
43.81
19.03
48.72
62.89
16.40
51.62
40.41
Llama (retrain.)
262B
40.53
25.51
46.21
62.73
28.40
50.57
42.35
MDLM
262B
30.98
23.63
50.52
54.13
28.00
49.41
39.44
GIDD (our impl.)
131B
26.73
24.83
51.10
51.85
27.20
38.53
HDLM-4 (ours)
131B
27.15
26.62
50.43
52.50
30.00
50.99
39.62
HDLM-32 (ours)
131B
26.73
25.17
49.97
51.03
24.00
51.78
38.11
HDLM-32-5 0.9 (ours)
131B
28.07
25.94
50.61
53.21
27.00
51.07
39.32
BASE
GIDD (our eval impl.)
131B
28.32
23.12
49.57
52.83
23.80
48.30
37.66
HDLM-256 (ours)
131B
28.91
25.09
50.18
55.33
26.20
51.85
39.59
Downstream evaluation. We evaluate the general performance of our language model for language understanding based on the commonly adopted Im-eval evaluation suite. We follow the benchmark suite choice of (34] which cover a diverse range of tasks regarding general language understanding and question answering capability and can convincingly showcase the performance comparison among different models. These include ARC [ (both elementary and challenge subsets), BoolQ [6]
HellaSwag 40 PIQA 4, OpenBookQA
222, and WinoGrande (28
We demonstrate the results of this evaluation suite in Table 7 For small and base model sizes, we report the performance of HDLM-4 and HDLM-256 respectively. Our small model achieves the best on two datasets and highly competitive results on others, even in comparison with autoregressive models. The average accuracy also outperforms other diffusion models even trained for twice as long, verifying the strong generalization power of HDLM. Our large model is better than the GIDD counterpart in every dataset. Interestingly, increasing model size does not necessarily improve the performance as observed by (34] which we partially attribute to overfitting the pretraining dataset. We also observe that with the perturbed forward process, the model with & < learns to distinguish inaccurate clusters in the denoising steps, which improves its self-correction ability. Compared to its counterpart whose g 1, HDDLM-32 with = 0.9 is better in five out of six tasks, and is also superior in four tasks to HDLM-4, verifying the potential advantage of self-correction.
29

---

C Details of the Semantic Clustering Algorithm
C.1 Clustering algorithm of word embeddings
As mentioned in the main text, the clusters of the word tokens can be learned jointly with the discrete diffusion model or predefined. In our implementation, we predefine the mapping given the number of clusters with an enhanced K-means clustering algorithm, which we refer as Semantic Clustering. The algorithm determine the clusters according the token embeddings of pretrained model, for instance, a pretrained GIDD model [34] which is more compatible with our discrete diffusion. Semantic Clustering consists of K-means++ initialization, a dynamic cluster size control mechanism (to keep the sizes of the clusters relatively balanced up use-defined tolerance), as well as efficient cluster assignment and centroids updates. We adopt the converged assignment to determine our word-to-cluster mapping (i.e., r',, and in addition take the centroids of clusters as the initializations of cluster token embeddings in HDLM. Detailed algorithms are described in Algorithm | and Algorithm 2
C.2 Examples of Word Clusters
We experiment with #clusters 16, 64, 128, 256), with pretrained GIDD model of both base and small sizes. The clustering algorithm generally converges in minute. The overall semantic coherence lies between 0.3 to 0.4 for most settings, suggesting a reasonable correlation within the clusters (given the large vocabulary size and the high-dimensional nature of the word embeddings). The cluster sizes and example words of the clustering for #clusters = 32 and GIDD-small is shown validate the effectiveness of the algorithm.
in Table and Table 9 for reference, from which
C.3 Discussions
As one could imagine, the quality of the clustering method has considerable effect on the model performance. Hence, more sophisticated algorithm to predefine the hierarchical mapping is worth exploring, which is left as future work. The mapping function can even be extended to stochastic or implicit. Learning the hierarchies jointly with discrete diffusion training is also a feasible solution, where we could represent sets with deep models such as DeepSets [39]. Hierarchical clustering also provides an additional perspective for introducing multiple hierarchies into HDLM. We can show that our theoretical framework still applies to this case, and HDLM can potentially further benefit from more carefully designed or well-trained hierarchies without introducing computation overheads in practical training.
30

---

Algorithm 1: Semantic Clustering with Size Constraints (Part 1: Setup and initialization)
Input: embeddings, cluster_size, mnor_lerre lolerance, min_size_ralio, mar maaz_iize_atic use_ use_cosine
Output: Buster_ids, centroids
vocab_ size, embed_ _dii_ddimm__ t Dimensions of embeddings:
use_cosine True then
cormmalized_embedddnge + embeddingg/|lembedddigg // Normalize vectors
else cermmalized_mmbdddiiig <- embeddings;
end
rrg_cluster_siee t vocaabsize/clssee avg
min_clusler_size &- ang aug_dusler_sizz x miin_sizerralio,
mar_cluster_size < avg_cluster_size mar_size_ralio;
// Initialize centroids using K-means++ centroids +-
Function wtt.://ww..c.iieee/////////eeeed hntiaaiiiccceennnenueeueuuuallii hewer//eeentoeeeeeeeeeeeeeeeeenr
_ize,,sze use _ee_osiee);
cluster_size uos_nne) centroids + zeros(cluster__it) embed_diie __idd:
first_ide idr RandomInteger(() vocab size - 1);
centroids 00] + embeddingss[frrs idz];
if cuse_cosine True then
centroids 00 +- centroids |/cetroosddds (O]]::
end
for k = to cluster_sis size do
if use._ uis__cosinee True then similarities +- embeddings centroidal: k)
max_similariten +- max max(similaritiis, axis 1);
distances +- 1 max maz_similarities:
else
distances Coopueeee..iissaaccceeemmmnnnood
k));
end
probabilities +- distances distances?:
nexl_ida idx &- hamplle@roomiistrbbbttiooom hatiorrbbibbbttllloooobobbllll)
centroids [k] <- (osleeddddingggss____d__i_ee
usee_oo_o fese_ccoceeeee = True then
centroids <- c (entrdddd end end
return centroids;
for iteration = 1 to mar iters do
// Assignment phase
if use use_cosinee = True then af finitiees <-
CamppreessinsSilllaatttttoooreee -ererddimeeeddns.....e centroids);
cluster_ide ids argmax(a finities, axis =]; // Initial assignment
else
distances hetpreeeoe.eellaaaaacsooeemeeeed
centroids);
cluster ids t- argmin (imminndisttncss, axis = // Initial assignment
end
cluster_size _izes <- CountPointaahnllusttrrrcccuus _ids, "luster_size)
// Continue in Part 2... end
31

---

Algorithm 2: Semantic Clustering with Size Constraints (Part Update and Size Constraints)
// Continued from Part 1...
// Handle oversized clusters
oversized clusters puerstzaddclusterss t {k "lustersizess// max maa__mloutee cluster_size,
foreach cluster_ids E pnrerizzed_cuustene do
excess_cound + chuster httlsseerssiicsscusslee.et _idr) mar www_ccuuuessaar
yoinls_i*_clusler < | cccusler_ids_f cluster dussterddde-
foreach point E biciiils_in_ccusiii
do
Calculate affinity/distance to current cluster;
Find second best cluster and its affinity/disaance:
Calculate reassignment_ cost for moving this point;
end
points_ _ttossso_____oooo more Select excess count points with lowest reasiignesst
foreach poinl E ppinlss_o__ firinlsll_movee do
Reassign poind to its second-best cluster; end end
// Handle undersized clusters cluster cluster_sizes sizes t- MacountClusterss(cuseee peasster_dd_,,__ cluster_ cluster_size))
undersized. clusters < {k cluster sizes
cluster_sizes{Kt < min min_cluster cissserssttr_se)
foreach cluster jid E uundersized__luslerss do
needed_counn män_clusler_size
fatrstr_ssies_duuae
candidale_points <- {i cluster_ids(i]
clusterjdd and has high affinity cluster_iee -idr):
points_te to to_move + Select needed count best candidates from candidate_points; foreach point E points_to to mo_e do cluster_idsppoind + cluster_idı;
end end
// Update centroids
_ee__cenroidd ceros(cluster. _size, bombledddim):
for k = to cluster_sic size do cluster_man mask {i cduster_ids k): if elusser_mass > then
if uue_ uis__cosine- True then
mean_rector t- aaaaaaoooaiicddeeeeddiiii
axis = 0);
new new_centroids(k] + anean_uector//nn ||
else
new cew__enrroids(K < (aaa(oornaiizedeemeeediiiggggnd -aasstre__msk,__- axis = 0);
end
end
end
// Check convergence
use use_cosine = True then centroid_chane change 1 mean embedddin centroids(i.j) new (eceu__entroids(i_f]] و==
else
centroid_change + ||new_centroie centroids end
centroids + new weanneeeeeetree sew_centroids;
centroid change < tolerance then
break; end
// Converged
Compute final statistics and coherence scores; return cluster bics, centroids;
32

---

Table 8: Example words of clusters. Shown is the #cluster=32 example.
Cluster
Size
Example Words
862
aign, York, Washington, San, Calif, California, Tex, London, Texas, college, ville, football, England, icago, eah, Florida, Chicago. NFL. Jose, ijuana, Los, Louis, Angeles, Carolina, Virginia, Francisco, Boston, Lake, Toronto, Penn
2
2098 ility, ution, ism, ness, formation, ience, effect, result, iness, ency, interest, ality, design, view, cost, story, claim, ization, example, urity, idence, reason, ulation, ability, question, love, problem, success, erence, value, experience, mind, future
217
2016, 2015, 2014, 2017, 2013, 2012, 2011, 2018, 2010, 2008, 2009, century, 2007, 2015, decades, 2000, 2006, 2014, 2005, 2016, 2004, 2017, decade, 2003, 2001, 2002, 1990, 2013, 2012, 1980, 1999, 1998, 2018, 1970, 1996, 1997, 1995
4
776
country, Euro, Europe, North, South, countries, German, Israel, China, Japan, Syri, Russia, Canada, Russian, European, UK British, English, Muslim, India, Iraq. Indian, Chinese, Iran, istan, French. Syria, Germany, Arab, Australia
1746 ments, system, art, things, information, ways, list, data, plan, process, program, project, money, product, ories, itions, event, games, access, amount, series, ources, hours, space, period, ients, item, ilities, issues, minutes, changes, results
6
1886
God, character, film, King, Dragon, Star, Super, craft, battle, uild, Earth, Game, Dark, movie, ternal, Lord, enemy, Rock, spell, Edition, Cat, Sky, novel, magic, Dead, Master, god, enemies, Games, Hell, Death, Battle, quest, Magic, Wars
1381 erson, ason, Trump, oney, son, ohn, augh, ague, erman, inton, Obama, Clinton, White, Hill, ston, wood, ford, berg, Rich, Smith, Brown, sey, zen, bert, Johnson, kin, kins, Fox, Jones, Bush, igan, rick, stein, Lee, Sanders, sen, mond, Williams
8 948 Ed, John, Russ, Mark, David, Paul, Sam, Jack, James, Donald, Rob, Michael, Angel, Ben, Bill, William, Tom, Thom, Alex, George, Phil, Matt, Scott, Lou, Mike, Tim, Hillary, Carol, Chris, Robert, Frank, Mary, Jim, Max, Jeff, Ann
1790 ited, made, used, found, read, met, told, ished, got, called, pped, able, ived, available, took, done, seen, thought, given, won, known, sent, based, asked, started, taken, added, reported, itted, wanted, saw, began, overed, needed, lost
10 1033 also, only, again, even, now, then, very, But, ually, still, though, really, fact, ically, too, never, ately, ably, So, always, ently, already, least, later, enough, sure, actually, ever, often, course, yet, likely, today, ially, particular. anything, though
11 1064
war, attack, sex, death, milit, cult, demon, terror, arrest, attacks, Islam, society, violence, prison, ban, crime, peace, igration, criminal, intelligence, assault, politics, marriage, crisis, abuse, murder, killing, bomb, gender, icide, illegal
12
2175 Contact, Content, Truth, Memory, Thread, Eye, Info. Quality, Too, Active, Single, Following, Friends, Ord, Rules, Drop, yrics, Until, Between, Pred, Rule, Weight, Comb, Learn, Inside, Basic, Site, Sex, Normal, Upon, Due, Made
13
1764
ear, car, leg, bus, camp, cap, book, air, dam, land, area, net, room, object, house, field, pub, lab, bar, ground, pack, foot, ograph, page, gun, port, track, heart, ball, building, inside, card, base, hands, eyes, road, goal, phone, table, box, path, pen
14
1422
4. 6, 7, 8, 9, 00, 01, 201, 19, 6, 7, 8, 000, 200, 10, 12, 12. 50, 19, 11, 20, 18, 201. 30, 15, 15, 11, 14, 16, 16, 14, 13, 25, 18, 13, 30, 17, 199, 17, 25, 24, 80
15
2314
H. E, K. St, Th, Y. Ch, irst. Un, Re, Tr. Wh. Sh, Ar, Com, New, Al, Se, Le, CI, Americ, De, An, B1, 7,, Ad, Fr, Sp, Pl, Be, Ph, Ind, Sc, Or, Con, Comm, Mar, Dr. FI, Col, Sy, Te, Am, Br, Pr, Pres, Co, Mr. Su, Tw, Ob, Im. Min, Man, Res
16
1628
add, see, because, make, where, against, help, follow, find, govern, ize, during, develop, without, allow, using, ask, invest, build, including, keep, kill, why, tell, give, expect, perform, within, making, become, having, elect, sever, whether
33

---

Table 9: Example words of clusters (Cont.). Shown is the #cluster=32 example.
Cluster
Size
Example Words
17
1535
go, get, play, act, bet, look, don, think, start, count, take, going, run, lead, pass, turn, something, put, might, feel, must, come, seem, stand, let, rest, pay, didn, return, move, pop, talk, until, win, doesn, came, happen, sit, appear, grow
18
1043
K. M. N. Q, U. C, U. Q. US, X, AT, IS, AS, IT, IC, AC, ST, AM, OS, BC, AD. US, EC, AP, AM. PM, IS, OP, TV. BI, AP, IG, AN, IA, ST, EM, FL, PS, OD, AA, RA, IM, II, IP, SA, CC, CT, PC, SP, BA, BS, OC, ID, NA, PA, GB, IR
19 3630
fier, ople, nder, ause, iew, ublic, ange, stem, vern, ollow, ween, alth, illion, ption, nce, ittle, chool, erest, lease, ording, chn, reen, arge, ww, ample, uro, iven, raph, ission, ember, orth, ajor, outh, ither, arent, ceess, nect, iversity, ideo, gin, ource
20 1993 clud, includ, differe, requ, somet, velop, exper, offic, expl, comple, belie, incre, contin, riend, commun, adv, ruct, beh, ide, polit, ploy, uthor, chang. charact, techn, rese, ocr, econom, orig, polic, creat, ident, invol, rele, ortun, viol, sugg
21
1205 rub, cann, ast, bull, wal, suc, phen, lit, af, vac, upper, cro, sav, lob, root, nav, mand, buff, mo, du, flex, medium, gro, stim, pow, ped, pra, ble, bes, tor, pip, sho, CO, hyd, sup, vent, rum, spr, poly, cock, ju, vo, tort, li, pref, mel, blind, sin
22
1714 app. irect, import, ware, htt, type, function, version, init, file, text, code, method, log, webs, link, htt, struct, tool, search, Twitter, Google, email, command, Face- book, atform, O. application, print, coin, computer, software, feature, Windows
23
2353 Bas, Rad, Incre, Build, Team, Keep, Cap, etition, Real, Disc, Product, Que, King, Tra, Pan, Index, Sk, June, Bro, Looking, English. Vis, Intern, Field, Tor, Microsoft, Rober, Prov, Ohama, Supp. Control, Sun, Home, Hub, Apple, Notes
24
1217 ational, ative, ular, public, ural, health, ional, human, itional, social, local, political, individual, anal, ederal, custom, national, aterial, white. military, ental, energy, istic, personal, atural, online, legal, ological, ancial, private, sych, civil
25
1236
ATES, PAR, APTER, DAM, URES, VERY, AMES, GOD, WORLD, HOW, LOC, ITS, REL, VIS, AIN, ATIONAL, CLE, FINE, BEL, IMP, SOU, PUT, ZZ, INC, IVERS, OUR, CLE, LOT, DOWN, TAIN, GER, ITIES, LED, NET, STE
26
1427 report, law, state, government, Rep, ology, conom, school, United, business, season, police, tax, official, city, press, War, State, fund, States, market, campaign, study, community, media, University, Republic, House, service, party
27
2042
very, round, day, self,, the, view, ertain, stand, ready, for. not, pro, most, play, empt, urther, wh, year, put, now, minist, load, ext, like, cle, medi, su, time, key, face, rect, set, body, action, ruction, ground, order, down, hold, come, where
28
1751 much, those, such, end, ting, long, ef, ys, its, ax, own, set, ifc, ble, ward, show, ms, omet, say, Is, ful, und, ren, cess, ince, u, olog, up, ump, last, ures, ars, ues, cy. io, hes, air, ier, read, ank, atch, ever, point, ork, ool, alk, ement, ract, same
29
1717 new, good, great, different, fun, big, direct, better, small, vious, current, ailable, large, important, clear, certain, possible, proper, complete, young, strong, quick, true, bad, similar, entire, specific, previous, recent, significant, difficult, easy
30
1381 meric, water, iol, pot, food, wind, pie, drug, blood, oil, gas, brain, fat, gold, birth, drink, skin, cook, disease, chem, plant, species, animals, ice, cells, cohol, cancer, heat, drugs, fuel, diet, alcohol, coal, marijuama, ffee, dry, fish, egg, temperature
1308 man, people, ists, person, team, child, men, women, friend, ians, family, company, children, author, intern, others, members, president, players, former, President, mom, someone, player, cop, woman, students, exec, friends, everyone, parent
32
1611
I, S. G), * +. =, >, ? @, Lb 1h ..... www, http, HAHA
34

---

D Unconditional Generation
We finally provide some examples of unconditional generation. We choose our HDLM-128-bald model, and use the reverse process described in the main text.
Sampling Steps 512
Contents
It seems that HTC (TM) is handing a part responsibilityy to build Smart Device Projects Qualcomm, concluding the giant former handset part- nership, and the 24 Samsung Gear in Japan ordering ESPA+.Check out: Samsung TV with P35 and Sony Glass? Rooster. Samsung aforemen- tioned Desire 1520 was launched last month, and Samsung is now using the IGG Desire moniker, the only company to come in its first series of ARM chips powering the 1520 HDD screen. However, a well estimate of the upcoming Gear tells the Beyond Android forums that Samsung users will get access to its Mobile Gear program with each monthly purchase, perhaps an acknowledgement
its ad-init nature of SME lock, and that Samsung is already shooting a track for further marketing after the device has introduced. only started engineers here, next steps are in Desire 1520 and not in the HTC board. Sale pointed its backers, working on the project, much more up the supply chain, this with the plans for Galaxy launch in 2016 are taken seriously in pre-2015 where Samsung refers higher end hardware pack of chips to see Wireless Design tests on touchscreen tablets. and their market still beyond the successor, all Galaxy Note 3. Many were surprised that Samsung will not pursue Mobile Fusion during the coming initiative and Strategy period, perhaps because of the flexibility of bringing in such slab of HGDDR, where development is under pressure from certain suppliers. Native research laboratories within the companies would vary some be al possible which manufacturer and OEM has the best push to stay in research lane.
1024
It opened so slowly, revealing that I am an angry, serious person. will be typing to end "need you to send me a wallally quick enough? Hopefully it can bring people tears and hurt" psuedo for the real impacts of a heavy hand on my feelings. Unfortunately. will often be sent truly open letter long time surprisingly few people ever answer it and take seriously. Nor besides people tell me am doing anything wrong. Do you ever sit assured where I would think later? (mathing) Yes. I am talking to you for the first time. believe it too fast. Mingya Dear blood. Why did you have to read this written hard time? In desimimized way, you got beaten 3-0 by drinking alcohol against a Zerg strager in our fan club where you now yours. It and the irony is bit unusual, times, I would not check Reddit and accept was on the other side. In fact, I used to get that kind of Ape feel pulling to let me know. But not now. We are outside of Korea where players do not even know that violence punishment breaking during the game saying Thanks, you never expose your friends or people to parasitic infections of said hate. tried somewhere there is that sort of vi-Re where agents fight like "in takes the bullet" and hits back groups but now this Battle is that same kind of place. MTSong: Thanks my close friends as my translator. I will see you soon. Zona Princess -uneredential -talk -beautifully Braid: When Baran's fan got forthright about your backstabbing and he tried to insult you in intimidation, what made you realize last minute?
many
35

---

Sampling Steps 1024
Contents
Almost literally mission based on defying celestial navigation rules making observer looking at objects from bright wavelengths by century convention. Think of this image as Gaussian filter, a blur between Galileo and Zeeb, wondering what the objects are going through from the point down. Its stray segments, orbits, orbital motions, unique orbital lengths, all normal events that seem impossible for astronomers to predict. This is just the way the Discovery team wanted the modern astronomy to look, where even the telescope looks up at the sky as it pummels it instead of measuring star. These objectologist smell like liar on the chaos and like cheat astronomer. So we know the test that followed unionized framework Southwest Research Division focused on sending up the groundwork to get a reality instrument to the ground that could in decade more better upgrade the world astronomical tools. But they did so largely by accident. The Milky Way provided the Planetary Systemector naked and sent tonnes messages through the solar system, all along way) to the moon. This culture has extended creatures experienced from dusty cloud lifeworm, Cream Dwarf B (or or B or O) to the solar system forgotten white dwarf, Star Power Letter Flake Feeder. Consequently, the helloubating, a deafening cheer, this message was focused on categorizing the minor planets on earth. Somewhere near the gastrarian referred the message "Malmy" and running that is like if Lady Orr opens whats away message she has written asking, Everybody belong, go spaceship, I'm working along the planetary science studying various planets on this planet around me, the resulting announcement was useless for most of the carrier. The remaining things just were moved to stop others from moving around before they heard the signals. Everybody's ' RB, the broadcast is not observatory to reach for aliens, we're thinking more about it and the message's ' message seemed supposed to be released to verify a alien's 's structure via the 1996 SETI conference. Source: Astrophysicinos Gramento de Mancera, et al. PSA Space Center. G flipped from the corrected message box shows GAnd literally, the probe was supposed to give us information. The entire experimentation mechanism that processed the message was immediately sent back to the NASA Data Center. It would be grand to know the "Daily message box" was off hoax that this scoring point from culture.
1024
One sunny Sunday we ventured to Montreal from over Britannia: when we finally arrived in town the talk about the way of the other day on the rink was the focus. The couch was mixing pie, savanna and chocolate pizza. I was eating bargain thin desserts in my Aimost-Francee de Don diet on Sunday morning. Now instead of getting homage at the rink, I was with a milkshake hint the form of caramel cookie baked and deep-fries frip-fried corn. There were four friends at work at a restaurant. Teddy, my wife and I left sans bread but a table great full house full of work and we leaned in at scrambling. I provide replacements for these-u.de -- but-very-kreld wwekkkelllloeeses through the crust when baked the crack. Given the size of my family, my parents stepped in to buy the cookies, break something about baking. Not only that, but we spilt spaghetri-dramblee aore through the flame crust and the best bill cooled and sheathed. The candy is sensitive, the bread is thin and it was wrong, it worked. Toated it with my hand. Shut the doors and chew with my fingers. Iated tail. To the couch while the kitchen pocked and, sure enough, what thought was a pot was the grilled cheese. Only to be wrong. The cheese had become too dense with grease.
despite Fireade bacon and cheese reheat too far
36


Reviews:
[{"review_id": "kcyzcGmPDL", "reviewer": "Reviewer_7z47", "summary": "This paper introduces Hierarchical Diffusion Language Models (HDLM), a discrete diffusion model that operate on hierarchical semantic scales. Unlike standard discrete diffusion models that transition between clean tokens and masks, HDLM introduces intermediate semantic hierarchies where tokens progressively transition from fine-grained to coarse-grained representations. The authors derive closed-form ELBO expressions, demonstrate that existing models like MDLM are special cases of their framework, and show that HDLM achieves competitive performance on language modeling tasks while enabling self-correction through its hierarchical structure.", "questions": "No", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- Conceptually novel approach: The idea of hierarchical semantic diffusion with \"next semantic scale prediction\" is intuitive and well-motivated, addressing key limitations of existing discrete diffusion models (lack of semantic richness in masked tokens, inability to self-correct).\n- Strong theoretical foundation: The paper provides rigorous mathematical treatment with closed-form ELBO derivations and proves elegant properties like the invariance of token-level loss weights.\n- Unifying framework: Showing that MDLM is a special case of HDLM demonstrates the generality of the approach and provides theoretical validation.\n- Self-correction capability: The hierarchical structure naturally enables the model to refine predictions, as demonstrated by improved self-corrected perplexity.\n\nWeaknesses:\n\n- Insufficient insights into model behavior: The paper presents only quantitative metrics (perplexity, downstream task accuracy) without providing any understanding of how HDLM actually functions as a language model. Metrics alone are insufficient to understand a generative model's behavior.\n- No generation examples or analysis: The paper lacks any examples of generated text, making it impossible to assess generation quality, coherence, or how the hierarchical structure influences output. Readers cannot judge whether the model produces sensible text or how it differs qualitatively from baselines.\n- Missing generation process visualization: How does the hierarchical denoising process unfold? Examples showing the progression from mask→cluster→word tokens would illuminate the model's unique generation dynamics and help readers understand the practical implications of the hierarchical structure.\n- Lack of parameter sensitivity studies: How do key hyperparameters (number of clusters, clustering methods, noise schedules, force transition) affect generation behavior? Without these ablations, it's unclear which design choices matter and how to configure the model effectively.\n- No analysis of learned representations: What semantic patterns emerge in the hierarchical clusters? How does the model utilize the intermediate representations during generation? Such analysis would provide crucial insights into whether the theoretical benefits translate to meaningful semantic organization.", "was_revised": true, "final_justification": "The authors covered some of my concerns in their rebuttal. I believe that this paper is good, yet I cannot give it a rating of 5, mostly because of other unaddressed points in the weaknesses. I believe this paper would benefit from a more in-depth exploration of the underlying representations. I don't think that the fact that other people haven't done that is a good argument.", "rebuttal": "We address the reviewer's concerns as follows.\n\n> Insufficient insights into model behavior\n\nIn our original submissions, we already included sampled texts as qualitative results in the appendix. In our new results, we also report generative perplexity of generated samples, showing that samples generated by our HDLM are substantially better than other baselines. Regarding theoretical insights, we believe our grounded elbo derivation and other discussions in the paper already provided enough insights of the next semantic scale scheme. We can answer in details if you could specify what type of insights you are expecting.\n\n> No generation examples or analysis\n\nWe already included a lot of generation examples in the supplementary. Here are more samples.\n\n- Scientists have noticed that patients include some vascular dysfunction, and report a rapid onset of symptoms and years after giving the procedures to treat this kind of cancer anyway. Cancer more often happens to take effect for tumors, however, lowering median rates for brain tumors as much as the patient requests for curable cancer, resulting in some improvement. Patients started to have blurred episodes as an old friend cancer, and the same seems to be improving everywhere. Patient requests do tend to be history with treatment and have included patients with brain that had mostly been frozen, attached as bone, to have a bone that protrudes in the skin and lumbar defects that wouldn't tear, but they can fracture and cut with no difficulty. Jullk-hootinen has had success in the skin cancers in the breasts and substates such as kidney aisles. During the 1990s, the greatest success was in advanced patients and sub-patients in the development of new angi cells. We were to dedicate those of cancerous cancer to cancerous cells, and render our angi cells the same color and colors as our skin and other blood line tissue such as heart, brains and lungs. Except for the Ktellften, basically what I was using as a target in the 1990s. Studios Research, a program by the Strategic Fund, has been toward the study of degenerative cells called angioglial cells that have a coating of freeids neurons. It takes a lot of investments to make a product that's versatile, reliable and working at a reasonable cost. The company has made a monitor, or magnum, to analyze angi cells through a test tube. Just all your angi cells would live in the right tissues. Why allow them to live if we treat an incurable disease, like a new chemo called telov-thole, or not? I am actively investigating angi cells to help treat this defect, so if you are a multi-pathologist who does a set of alkenomalies, some people would have tumors if they could be treated with angi cell therapy. Also, there is a now-saving catheence drug. What would you do with a cancer so advanced? Doctors are brainstorming that now, is a recent device made by a biopharmaceutical company to help with a cancer, like the telov-thole implant, because it's so sensitive and strong.\"\n\n- As new and healthier China, the country enjoys trends in sustainable technologies and lifestyles. After seemingly a century when 80 percent would mean an annual case of new and compared postings, the term green it had become the metric system of all sustainability - to define and elevate what and where humans will be. A wave of what we knew could be \"We would not expect Xi to rank India's ray of green light as easily as China should achieve under his watch is performing uber statistic\". Greenpeace Arevers' Matthew Segius. With collaboration from the America Media and Development Bureau. The earth was silently driven through the river of Mount Beijing - one of four milestones reached.However, the reduction of green as Fuzhou seemed outside of his control. In 2009, China was suddenly on a fire. This afire situation can be reflective of the rapid rate of growth. China's annual GDP ratio more than doubled from less than 7 percent of the population in 2007 or 2008 to more than 14 percent now. In Africa, that same growth of the poor guy benefited with India accounting for the top 3 percent of population as much as the U.S., Africa etc. with population spread percent of that population's income. At a revolution below GDP per person today, everyone in India makes \\$4,000 while in China the poorest 50 per cent make that amount. But where human activity seem assured is the tiny concentrations of the photographed, and Xi's comments, while outwardly, were meant to express their probable hopes and aspirations. Green responses It is bizarre and distressing that this paradigm of not \"have me or haven't\" exists for too long. So much of the global green questions are unfiltered and over 92 percent have already been removed. A return to the close-door meeting where a critical shift for the external, is back unnoticed cautionary message, it comes to numbing heads of all kinds of serious provocations is learning shreds of polluted lakes for right into energy-poor regions, stolen from the Chanolan Monastery, The Great Monsoon Divide, a smuggler in the Singapore nightmare made tapping in Russian meters in Norway for electricity leaks while North Korea seeks for its electricity needs. The fact the questions have been removed from subject relations of grassroots success is key. \"While in the future we would like them would be coy if then democratic about that, if successful\".\n\n> Missing generation process visualization\n\nUnfortunately, we cannot upload figures to visualize the process. Intuitively, we are capable of both GIDD-type (Bayesian based method) and MDLM-type (analogously to masked diffusion) reverse process, where the mask tokens are gradually transit to clusters and then word tokens according to the time schedule.\n\n> Lack of parameter sensitivity studies\n\nIn Table 1 below, we provide extensive ablation study on the number of clusters.\n\nTable 1: Performance on OWT dataset.\n\n| Model (small)  | train tok | valid ppl ($\\downarrow$) | gen ppl ($\\downarrow$) |\n| -------------- | --------- | ------------------------ | ---------------------- |\n| GPT2           | unk.      | 23.40                    | -                      |\n| SEDD           | 262B      | $\\leq$24.10              | -                      |\n| MDLM (reimpl.) | 131B      | $\\leq$27.48              | 170.7                  |\n| GIDD (reimpl.) | 131B      | $\\leq$25.71              | 170.2                  |\n| HDLM-1         | 131B      | $\\leq$25.72              | 163.9                  |\n| HDLM-2         | 131B      | $\\leq$25.79              | 160.2                  |\n| HDLM-4         | 131B      | $\\leq$25.05              | 162.0                  |\n| HDLM-8         | 131B      | $\\leq$24.57              | 146.6                  |\n| HDLM-16        | 131B      | $\\leq$24.17              | 155.2                  |\n| HDLM-32        | 131B      | $\\leq$23.83              | 154.6                  |\n| HDLM-64        | 131B      | $\\leq$23.36              | 144.2                  |\n| HDLM-128       | 131B      | $\\leq$23.25              | 148.0                  |\n| HDLM-256       | 131B      | $\\leq$23.65              | 150.4                  |\n| HDLM-128-Large | 131B      | $\\leq$19.22              | 139.9                  |\n\nDue to the time restriction, we are still conducting experiments on other ablations. We will update the results when they are ready.\n\n> No analysis of learned representations\n\nIt is difficult to analyze learned representations and connect them with hierarchical structures. No papers of this type on discrete diffusion or even LLMs have done such analysis. We believe this is beyond the original scope of the paper."}, {"review_id": "oOGMwQhqwf", "reviewer": "Reviewer_UHEs", "summary": "This paper introduces Hierarchical Diffusion Language Models (HDLM), a novel class of discrete diffusion models for language generation. The core idea is to replace the standard denoising process, which typically transitions from a mask token directly to a data token, with a multi-stage, hierarchical process. In the proposed forward process, word tokens are first perturbed into \"coarse-grained\" semantic clusters before finally becoming a mask token. The reverse process learns to progressively refine these abstract cluster tokens back into specific, \"fine-grained\" word tokens, a scheme the authors term \"next semantic scale prediction\". The authors provide a theoretical foundation for this approach, derive an ELBO for training, and show that a previous model, MDLM, is a special case of their framework. They evaluate HDLM on language modeling and downstream tasks, proposing a new *self-corrected perplexity* metric to demonstrate the model's capabilities.", "questions": "8.  Could you add the self-corrected perplexity result for an HDLM with 1 cluster (= MDLM) to Table 1? \n\n9.  Given that the ELBO provides only a bound on the true likelihood -- whose tightness can vary between models -- have you considered evaluating the models using a more direct measure of generative quality, such as generative perplexity, which is a standard evaluation in the context of diffusion language models?\n\n10. The paper only considers one intermediate hierarchical level. You mention extending the theory to multiple hierarchies as a limitation. Could you briefly elaborate on the potential benefits and challenges of such an extension?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper tackles the timely and important problem of improving discrete diffusion models for language. The core idea of introducing semantic hierarchies is novel, intuitive, and offers a principled way to improve the denoising process compared to standard techniques. I find the *next semantic scale prediction* framing a good conceptual contribution.\n\n2. The work is theoretically-grounded \n\n3. The paper demonstrates that its framework is a generalization of previous work, with MDLM being a special case where only one cluster token exists.\n\n**Weaknesses**\n\n4. The paper's primary claim of effectiveness rests on a new metric, i.e., *self-corrected perplexity*. While the motivation to capture the model's refinement ability is understandable, introducing a new metric and using it alone to show your own model's superiority is not fully convincing without external validation. This is especially problematic when the model performs worse on standard metrics and achieves only marginal gains on downstream tasks. Furthermore, perplexity values derived from the diffusion ELBO are an upper bound on the true perplexity, meaning they should technically be reported with a $\\leq$ sign. Reporting these as exact values in Table 1 is imprecise.\n\n5. The writing, particularly in the results section (5.2), is not always clear. The justification for the self-corrected perplexity metric is dense and its fairness is just asserted. Key details are also missing; for instance, the notation HDLM-X is used in tables, but 'X' (representing the number of clusters) is not explicitly defined in the captions. The abstract is also vague, using generic sentences like demonstrate the effectiveness without concrete, quantitative details.\n\n6. The paper fails to provide any justification for choosing the number of clusters for different experiments. This is a critical hyperparameter of the proposed method, and its impact on performance, complexity, and other trade-offs is not discussed.\n\n7. The paper frequently uses terms like \"detailed semantics\", \"coarse-grained meanings\", and \"richer semantics\". While the operational meaning (grouping tokens via clustering) is clear, the connection to linguistic semantics is loose and based on intuition rather than a more formal, well-defined concept.\n\nOverall, I think the idea is interesting, yet tested on a single setting and -- currently -- it brings only marginal improvements over simpler baselines.\n\n*Minor*\n\nThere are several typos (e.g., L192: \"as state\" -> \"as stated\", L262: “yet which does not” -> “yet this does not”?, Table 4: xi -> \\xi).", "was_revised": true, "final_justification": "The authors added new experiments and performance metrics showing improved results. I have thus increased my score and now lean towards acceptance of this work.", "rebuttal": "We thank the reviewer for the insightful review. We have worked hard to improve our work and answer your questions as below. Hope we have addressed your concerns, and we are happy to discuss any follow-up questions.\n\n> Weakness 4: self-corrected perplexity and other metrics.\n\n* We agree with the reviewer that the original evaluation is somehow limited. While the rationality of self-corrected ppl has already been clarified in the paper, we would like to add that we are not using self-corrected ppl alone as our main metric. We first apologize for accidentally reporting the performance of all our HDLM models trained for 65B tokens (instead of 131B) in Table 1 of the original submission. This insufficient training leads to an unfair comparison with other baselines. Along with moderate hyper-parameter search, including leveraging an improved schedule ($\\gamma=1$ in the appendix) and loss weight clipping, we conduct new experiments and train all the models for 131B tokens, yielding **substantially better results on standard metrics including validation perplexity and generative perplexity** compared with MDLM and GIDD baselines retrained in the same setting. Specifically, our best HDLM obtains even lower validation perplexity than gpt-2. The corrected performance of validation perplexity and generative perplexity is updated in the table below.\n\nTable 1: Performance on OWT dataset.\n\n| Model (small)  | train tok | valid ppl ($\\downarrow$) | gen ppl ($\\downarrow$) |\n| -------------- | --------- | ------------------------ | ---------------------- |\n| GPT2           | unk.      | 23.40                    | -                      |\n| SEDD           | 262B      | $\\leq$24.10              | -                      |\n| MDLM (reimpl.) | 131B      | $\\leq$27.48              | 170.7                  |\n| GIDD (reimpl.) | 131B      | $\\leq$25.71              | 170.2                  |\n| HDLM-1         | 131B      | $\\leq$25.72              | 163.9                  |\n| HDLM-2         | 131B      | $\\leq$25.79              | 160.2                  |\n| HDLM-4         | 131B      | $\\leq$25.05              | 162.0                  |\n| HDLM-8         | 131B      | $\\leq$24.57              | 146.6                  |\n| HDLM-16        | 131B      | $\\leq$24.17              | 155.2                  |\n| HDLM-32        | 131B      | $\\leq$23.83              | 154.6                  |\n| HDLM-64        | 131B      | $\\leq$23.36              | $\\textbf{144.2}$                  |\n| HDLM-128       | 131B      | $\\leq$$\\textbf{23.25}$              | 148.0                  |\n| HDLM-256       | 131B      | $\\leq$23.65              | 150.4                  |\n| HDLM-128-Large | 131B      | $\\leq$ $\\underline{19.22}$              | $\\underline{139.9}$                  |\n\nTable 2: Performance on LM1B dataset.\n\n| Model                       | PPL ($\\downarrow$) |\n| --------------------------- | ------------------ |\n| AR Transformer (33B tokens) | 22.32              |\n| BERT-Mouth                  | $\\leq$142.89       |\n| D3PM (absorb)               | $\\leq$76.90        |\n| Diffusion-LM                | $\\leq$118.62       |\n| DiffusionBert               | $\\leq$63.78        |\n| SEDD (33B tokens)           | $\\leq$32.79        |\n| MDLM (33B tokens)           | $\\leq$27.04        |\n| HDLM-64 (33B tokens)        | $\\leq$$\\textbf{26.95}$        |\n\n\n * We appreciate the reviewer's reminder of the $\\leq$ sign and will include it in the revised version. We fully understand that diffusion ELBOs are upper bounds, and we have emphasized that the bound for HDLM tends to be looser (as explained in the paper). We omitted the $\\leq$ as it should be a common sense and will update Table 1 with the $\\leq$ in the revised version.\n\\end{itemize}\n\n> Weakness 5: writing\n\n* We will improve the writing to make it clearer. We will make the justification of self-corrected perplexity more concise, explicitly define HDLM-X where $X$ is the number of clusters, and use concrete, quantitative details in the abstract. We will also fix the typos.\n\n> Weakness 6: number of clusters\n\n* We have conducted extensive ablations on the number of clusters in our new experiments, see the table above. Regarding validation perplexity, we observe substantial improvement when increasing the number of clusters, and optimal performance is achieved when the number of clusters is around 32 to 128. The result is consistent with our intuition: a larger number of clusters would lead to more expressive intermediate states and more sophisticated clusters, making the \"mask to cluster'' step harder, yet \"cluster to token'' step much easier. Striking a balance between these two steps would improve overall sampling performance. Regarding complexity, larger number of clusters would indeed lead to a larger transition matrix definition (namely $\\Gamma\\in\\mathbb R^{|C|\\times|V|}$). However, the actual computation overhead is negligible compared to the DiT forward pass.\n\n> Weakness 7: connection to linguistics\n\n* While the strict connection to linguistics is beyond the scope of the paper, it is indeed an interesting question. We present some examples of clustered tokens in the supplementary (Table 5 in Appendix E.2), where one could easily observe patterns (e.g., same part of speech, similar meanings, and shared root or etymology). However, we would like to emphasize that machine learning models may learn in a different way from humans, thus they may benefit from abstract representations (e.g., semantic clusters in this paper). Even the strict linguistic analysis of LLMs is difficult, and more research on the connection between LLMs/dLMs and linguistics is worth exploring in the future.\n\n> Question 8: Could you add the self-corrected perplexity result for an HDLM with 1 cluster (= MDLM) to Table 1?\n\n* Self-corrected perplexity for HDLM with 1 cluster is 25.68, which is almost the same as the original validation perplexity, verifying that MDLM is not able to self-correct.\n\n> Question 9: Given that the ELBO provides only a bound on the true likelihood -- whose tightness can vary between models -- have you considered evaluating the models using a more direct measure of generative quality, such as generative perplexity, which is a standard evaluation in the context of diffusion language models?\n\n* Thanks for the reminder. It is true that ELBO is a bound of the likelihood, and as we emphasized in the paper, the bound for HDLMs may be looser due to the extended vocabulary size, yet they still outperform baselines. \n* Regarding other measures of generative quality, we report the generative perplexity in Table 1 above. For all models, we generate 256 sequences, each with a length of 512 tokens. The number of denoising steps is set to 512, and the reference model is gpt2-large. Our HDLM significant outperform MDLM and GIDD baselines (the best HDLM-64 reduces gen. ppl. by over $15\\%$).\n\n> Question 10: The paper only considers one intermediate hierarchical level. You mention extending the theory to multiple hierarchies as a limitation. Could you briefly elaborate on the potential benefits and challenges of such an extension?\n\n* This is a great question. Introducing more hierarchies naturally aligns with the``progressively denoising'' natural of diffusion models, which is also one of our main motivation. Multiple hierarchies allow for more sophisticated control and potentially stronger self-correction abilities, since intermediate hierarchies with different semantic levels are capable of providing useful information that mask tokens cannot provide, while maintaining the ability to deal with uncertainty and potential needs to self-correct. For practical challenges, defining such hierarchies may not be trivial, and classical methods such as hierarchical clustering cannot necessarily lead to optimal definitions of the transitions. As mentioned in the paper, dynamically learning the hierarchies may be a promising direction, which we leave as future work. Furthermore, since common vocabulary sizes of the models are typically 50,000 (such as gpt-2 tokenizer) to 250,000 (such as Qwen-2 tokenizer), one intermediate hierarchy may already be sufficient. More hierarchies would lead to smaller-size clusters and restricted transition matrix, limiting the expressiveness while introducing higher computation costs.\n* While we leave the experimental implementation of more hierarchies as future work, we actually derive the elbo for arbitrary number of hierarchies, which we will state and prove formally in the revised version. We briefly state the main conclusion here.\n*Suppose the clean tokens are the $0$-th hierarchy, and the mask token is the $n$-th hierarchy. Denote $c_i(x)$ as the mapping of a clean token $x$ to the $i$-th hierarchy. Let $\\alpha_{t,i}$ be the marginal schedule for the $i$-th hierarchy. Then the ELBO is given by:*\n    $$\n    \\log p(x)\\geq \\mathbb E_{t, z_t}\\Big[\\sum_{i=0}^n -\\delta_{z_t, i}\\frac{\\sum_{j\\leq i}\\alpha_{t,j}'}{\\alpha_{t,i}}\\log\\Big(\\frac{\\sum_{x':c_{i-1}(x')=c_{i-1}(x)}p_\\theta(x')}{\\sum_{x'':c_{i}(x'')=c_{i}(x)}p_\\theta(x'')}\\Big)\\Big]\n    $$\n    where $\\delta_{z_t, i}$ is an indicator function that denotes whether $z_t$ is in the $i$-th hierarchy. The result shows that the ELBO of arbitrary hierarchies can be decomposed into a weighted combination of cross-entropy losses on each hierarchy. We believe this is a **novel and highly nontrivial** theoretical contribution which could inspire more future work."}, {"review_id": "Z3ZMzGhgrQ", "reviewer": "Reviewer_M7sG", "summary": "This paper introduces an innovative class of discrete diffusion models called Hierarchical Diffusion Language Models (HDLM). In essence, HDLM employs a hierarchical vocabulary structure where lower-level tokens convey more detailed semantics, while higher-level tokens offer a more coarse-grained semantic perspective. The authors present a robust theoretical framework alongside practical experiments to convincingly demonstrate the efficacy of HDLM.", "questions": "- When applying HDLM to image generation using discrete tokens generated by VQ-VAE, what modifications are necessary? In this context, would HDLM be a suitable choice?\n- Why do you use different numbers of clusters for different model sizes in Table 2? Should the number of clusters be fixed before training, and how is this number determined?\n- Why did you choose the DiT architecture over the more commonly used LLaMA or Qwen in LLMs? Could you compare HDLM's performance with the LLaMA architecture?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- This paper is well-written, and the concept of HDLM is both novel and intriguing.\n- The theoretical results and practical design principles are robustly established.\n- With its self-correction step, HDLM outperforms all other discrete diffusion models.\n\n**Weaknesses**\n- The related work section lacks organization.\n- In Table 1, the comparison of HDLM's sampling speed with different cluster numbers against other discrete diffusion and AR models is unclear. Additionally, does the self-correction step introduce extra computational costs?\n- Why does the small HDLM outperform the base HDLM in Table 2?", "was_revised": true, "final_justification": "The authors answered all my questions. I still vote for acceptance.", "rebuttal": "We sincerely appreciate the reviewer's positive feedback. We have answered your questions as below.\n\n> The related work section lacks organization.\n\nThanks for the reminder. We will update the related work section in the revised version.\n\n> In Table 1, the comparison of HDLM's sampling speed with different cluster numbers against other discrete diffusion and AR models is unclear. Additionally, does the self-correction step introduce extra computational costs?\n\nWe would like to clarify that the main computation FLOPs occur in the forward pass of DiT, which is the same for HDLM and other baselines (e.g., MDLM) with same architectures. The only extra computation of HDLM is computing the transition matrix between states, which is negligible compared with DiT forward pass. Similarly, the computation is also almost the same for all models with self-correction step, yet it is still efficient since we only need one forward pass (rather than recursively sampling). In summary, HDLM shares almost the same sampling efficiency compared with MDLM in real-inference settings.\n\n> Why does the small HDLM outperform the base HDLM in Table 2?\n\nActually this phenomenon is also observed in GIDD paper. We attribute this to the gap between the downstream task and the pretraining corpus. Since the base model fits into the pretraining task more than the small model, they may struggle when there is a distribution shift in downstream tasks (namely overfitting). However, we indeed observe significantly better sample quality of base model in unconditional generation tasks, which can be measured by generative perplexity. In detail, for both models with 128 clusters and trained for 131B tokens, we generate 256 sequences with length 512 using 512 denoising steps. When using gpt2-large as the reference model, the generative perplexity is 139.9 for the base model, which is obviously lower than that for the small model (148.0).\n\n> When applying HDLM to image generation using discrete tokens generated by VQ-VAE, what modifications are necessary? In this context, would HDLM be a suitable choice?\n\nThis is a great question. Typically, the total number of image tokens - codebook size (e.g., 2048) times hidden dimension (e.g., 1024), is significantly larger than language vocabulary size (typically around 50,000 for gpt2 and around 250,000 for Qwen-2). Also, the visual semantics of image tokens may highly depends on the contexts, whereas language tokens may have relatively static meanings on their owns. These factors make it harder to define proper hierarchies over image tokens. It is definitely reasonable to apply HDLM, and dynamic clustering or modifications similar to existing methods like VAR (which employs next-scale prediction) could make it more suitable. Since the image generation task is beyond the scope of the paper, we leave it as the future work.\n\n> Why do you use different numbers of clusters for different model sizes in Table 2? Should the number of clusters be fixed before training, and how is this number determined?\n\nThe number of clusters is fixed as a hyper-parameter. We have conducted extensive ablations on the number of clusters in our new experiments, see the table above. Regarding validation perplexity, we observe substantial improvement when increasing the number of clusters, and optimal performance is achieved when the number of clusters is around 32 to 128. The result is consistent with our intuition: a larger number of clusters would lead to more expressive intermediate states and more sophisticated clusters, making the \"mask to cluster'' step harder, yet \"cluster to token'' step much easier. Striking a balance between these two steps would improve overall sampling performance.\n\n| Model (small)  | train tok | valid ppl ($\\downarrow$) | gen ppl ($\\downarrow$) |\n| -------------- | --------- | ------------------------ | ---------------------- |\n| GPT2           | unk.      | 23.40                    | -                      |\n| SEDD           | 262B      | $\\leq$24.10              | -                      |\n| MDLM (reimpl.) | 131B      | $\\leq$27.48              | 170.7                  |\n| GIDD (reimpl.) | 131B      | $\\leq$25.71              | 170.2                  |\n| HDLM-1         | 131B      | $\\leq$25.72              | 163.9                  |\n| HDLM-2         | 131B      | $\\leq$25.79              | 160.2                  |\n| HDLM-4         | 131B      | $\\leq$25.05              | 162.0                  |\n| HDLM-8         | 131B      | $\\leq$24.57              | 146.6                  |\n| HDLM-16        | 131B      | $\\leq$24.17              | 155.2                  |\n| HDLM-32        | 131B      | $\\leq$23.83              | 154.6                  |\n| HDLM-64        | 131B      | $\\leq$23.36              | 144.2                  |\n| HDLM-128       | 131B      | $\\leq$23.25              | 148.0                  |\n| HDLM-256       | 131B      | $\\leq$23.65              | 150.4                  |\n| HDLM-128-Large | 131B      | $\\leq$19.22              | 139.9                  |\n\n> Why did you choose the DiT architecture over the more commonly used LLaMA or Qwen in LLMs? Could you compare HDLM's performance with the LLaMA architecture?\n\nOne of the fundamental difference between diffusion language model and autoregressive model is that the former features bidirectional attention and is capable of any order generation. These characteristics cannot be achieved by LLaMA or Qwen since they have causal attention and can only do left-to-right sequential generation. Actually, DiT is widely used in continuous and discrete diffusion, as in SEDD, MDLM, GIDD, etc, and we follow their settings. The performance of LLaMA-like model is included in the tables, and it is widely accepted that existing diffusion language models still underperform state-of-the-art AR LLMs, yet the gap is being closed."}, {"review_id": "2U7UmeaHbc", "reviewer": "Reviewer_mLoq", "summary": "This paper introduces Hierarchical Diffusion Language Models (HDLM), a discrete diffusion framework for language modeling.It structures tokens into semantic hierarchies where detailed tokens map to coarse-grained ancestors. HDLM's forward process independently transitions tokens to higher abstraction levels via scheduled diffusion. The reverse process progressively predicts finer semantics through time-varying next-scale prediction.This enable tokens to occupy different semantic levels at each diffusion step. Theoretically, HDLM derives a closed-form ELBO showing it generalizes prior discrete diffusion models, introduces stochastic perturbations for self-correction, and enables flexible training schemes. Experiments demonstrate HDLM outperforms existing diffusion models in language modeling perplexity and downstream language understanding tasks while requiring less training data, with larger semantic hierarchies yielding stronger self-correction capabilities.", "questions": "- Can other models not compute the self-corrected perplexity for unified comparison?  \n- Did the authors conduct ablation studies on the tricks for HDLM mentioned in the aforementioned drawbacks? Presenting these would more convincingly validate the effectiveness of these designs.  \n- It is recommended to elaborate on the specific clustering algorithm and operations in the paper, as I believe this is a crucial step for HDLM, directly impacting the model's final performance.  \n- Adding noise to word vectors in the embedding space causes the perturbed vectors to follow a certain probability distribution around their original positions. Does this correspond to the transformation from precise semantics to coarse-grained semantics in the proposed method?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\nProposes a hierarchical diffusion framework (HDLM) that introduces semantic hierarchies between word-level tokens and mask tokens. This replaces uniform/mask noise with coarse-to-fine semantic transitions, addressing key limitations in prior discrete diffusion models. Theoretical innovations include deriving closed-form ELBO expressions for hierarchical diffusion and proving HDLM generalizes existing models HDLM’s hierarchical design enables error mitigation during decoding, validated by significant gains in self-corrected perplexity. Techniques like stochastic perturbations and force transition improve robustness and training stability. HDLM achieves state-of-the-art self-corrected perplexity among diffusion models and consistent improvements over the baseline method GIDD across all datasets. The schematic diagram is clear and easy to understand.\n\nWeaknesses:\nAlthough Figure 1 is clear and easy to understand, the methodology section of the paper is rather obscure and suffers from poor readability. \nThe experimental section is insufficient, with only limited comparisons of perplexity and downstream task performance against a few methods. The experimental setup lacks clarity, particularly regarding implementation details such as the clustering process. Additionally, the authors did not investigate the model's performance across more hierarchical levels.\n\nFurthermore, while the paper proposes several tricks for HDLM, including Stochastic Perturbation \\& Self-Correction, Hard Training Mode, Auxiliary Loss, and Force Transition, the experimental section fails to provide ablation studies for these components. This omission makes it difficult to assess their necessity and individual contributions to the model's performance.", "was_revised": true, "final_justification": "raise score to 4", "rebuttal": "Thank you for your thoughtful review. We have improved our papers accordingly, and hope that we could address your concern.\n\n> Although Figure 1 is clear and easy to understand, the methodology section of the paper is rather obscure and suffers from poor readability.\n\n* We will improve the writing by providing more intuitive interpretations over the extensive theory, and explain the method in detail according to Figure 1.\n\n> The experimental section is insufficient, with only limited comparisons of perplexity and downstream task performance against a few methods.\n\n* We conduct additional experiments by training HDLMs wiith different number of clusters and compare both validation perplexity and generative perplexity metrics with more baselines. For generative perplexity, we generate 256 sequences with 512 length using 512 denoising steps and leverage gpt2-large as the reference model. The new results validate that our HDLM substantially outperform baseline models MDLM and GIDD in terms of both validation perplexity and generative perplexity. In addition, we train HDLM on the LM1B datasets and shows that our method outperforms previous baselines such as MDLM and SEDD.\n    \nTable 1: Performance on OWT dataset.\n\n| Model (small)  | train tok | valid ppl ($\\downarrow$) | gen ppl ($\\downarrow$) |\n| -------------- | --------- | ------------------------ | ---------------------- |\n| GPT2           | unk.      | 23.40                    | -                      |\n| SEDD           | 262B      | $\\leq$24.10              | -                      |\n| MDLM (reimpl.) | 131B      | $\\leq$27.48              | 170.7                  |\n| GIDD (reimpl.) | 131B      | $\\leq$25.71              | 170.2                  |\n| HDLM-1         | 131B      | $\\leq$25.72              | 163.9                  |\n| HDLM-2         | 131B      | $\\leq$25.79              | 160.2                  |\n| HDLM-4         | 131B      | $\\leq$25.05              | 162.0                  |\n| HDLM-8         | 131B      | $\\leq$24.57              | 146.6                  |\n| HDLM-16        | 131B      | $\\leq$24.17              | 155.2                  |\n| HDLM-32        | 131B      | $\\leq$23.83              | 154.6                  |\n| HDLM-64        | 131B      | $\\leq$23.36              | $\\textbf{144.2}$                  |\n| HDLM-128       | 131B      | $\\leq$$\\textbf{23.25}$              | 148.0                  |\n| HDLM-256       | 131B      | $\\leq$23.65              | 150.4                  |\n| HDLM-128-Large | 131B      | $\\leq$ $\\underline{19.22}$              | $\\underline{139.9}$                  |\n\nTable 2: Performance on LM1B dataset.\n\n| Model                       | PPL ($\\downarrow$) |\n| --------------------------- | ------------------ |\n| AR Transformer (33B tokens) | 22.32              |\n| BERT-Mouth                  | $\\leq$142.89       |\n| D3PM (absorb)               | $\\leq$76.90        |\n| Diffusion-LM                | $\\leq$118.62       |\n| DiffusionBert               | $\\leq$63.78        |\n| SEDD (33B tokens)           | $\\leq$32.79        |\n| MDLM (33B tokens)           | $\\leq$27.04        |\n| HDLM-64 (33B tokens)        | $\\leq$$\\textbf{26.95}$        |\n\n* It is worth mentioning that our experimental settings are standard ones that other papers (such as MDLM and GIDD) also deploy, testing only on these metrics.\n\n> The experimental setup lacks clarity, particularly regarding implementation details such as the clustering process.  \n\n> It is recommended to elaborate on the specific clustering algorithm and operations in the paper, as I believe this is a crucial step for HDLM, directly impacting the model's final performance.\n\n* Most of our experiments follow the settings of previous work. We explained the experimental settings and implementation details in the Experiment section as well as in **Appendix D**. Regarding the clustering process, we included extensive details in **Appendix E**. Please let us know if you have more questions. Moreover, in our new experiments, we also conducted ablations on the number of clusters, showing that a cluster number of 32, 64, or 128 leads to optimal performance. \n\n> Additionally, the authors did not investigate the model's performance across more hierarchical levels.\n\n* If here \"more hierarchical levels\" refer to more cluster numbers, our new experiments provide ablations on the cluster numbers with up to 256 clusters. If it means more than one intermediate hierarchies, we already mention this in our limitation section. While we leave the experimental implementation as future work, we derive the novel ELBO for arbitrary hierarchical levels as our new theoretical contribution, which we will state and prove formally in the revised version. We briefly state the main conclusion here.\n\n*Suppose the clean tokens are the $0$-th hierarchy, and the mask token is the $n$-th hierarchy. Denote $c_i(x)$ as the mapping of a clean token $x$ to the $i$-th hierarchy. Let $\\alpha_{t,i}$ be the marginal schedule for the $i$-th hierarchy. Then the ELBO is given by:*\n    $$\n    \\log p(x)\\geq \\mathbb E_{t, z_t}\\Big[\\sum_{i=0}^n -\\delta_{z_t, i}\\frac{\\sum_{j\\leq i}\\alpha_{t,j}'}{\\alpha_{t,i}}\\log\\Big(\\frac{\\sum_{x':c_{i-1}(x')=c_{i-1}(x)}p_\\theta(x')}{\\sum_{x'':c_{i}(x'')=c_{i}(x)}p_\\theta(x'')}\\Big)\\Big]\n    $$\n    where $\\delta_{z_t, i}$ is an indicator function that denotes whether $z_t$ is in the $i$-th hierarchy. The result shows that the ELBO of arbitrary hierarchies can be decomposed into a weighted combination of cross-entropy losses on each hierarchy. We believe this is a **novel and highly nontrivial** theoretical contribution which could inspire more future work. Also as we explained to Reviewer UHEs, one intermediate hierarchy may already be sufficient for the language modeling task we are interested in.\n\n> Furthermore, while the paper proposes several tricks for HDLM, including Stochastic Perturbation \\& Self-Correction, Hard Training Mode, Auxiliary Loss, and Force Transition, the experimental section fails to provide ablation studies for these components. This omission makes it difficult to assess their necessity and individual contributions to the model's performance.\n\n> Did the authors conduct ablation studies on the tricks for HDLM mentioned in the aforementioned drawbacks? Presenting these would more convincingly validate the effectiveness of these designs.\n\n* Thanks for the suggestions. We agree with the reviewer that more ablation study is needed. While we have already provided the ablations on the cluster number, other ablation experiments are still being conducted, since it takes about five days to train a small size discrete diffusion model on OWT dataset with 8 A100 GPUs. We appreciate your understanding and will report the full results in the following discussion period as soon as the experiments are ready.\n\n> Can other models not compute the self-corrected perplexity for unified comparison?\n\n* We are sorry that we might not understand your question accurately. As a rough answer, the models marked with \"x\" (including all AR models and masked diffusion models like SEDD, MDLM) are not able to self correct. In comparison, the self-correction ability of HDLM improves its sampling quality.\n\n> Adding noise to word vectors in the embedding space causes the perturbed vectors to follow a certain probability distribution around their original positions. Does this correspond to the transformation from precise semantics to coarse-grained semantics in the proposed method?\n\n* According to previous papers on diffusion language models, the continuous embedding space is hard to control, interpret and generate compared to discrete counterparts. Adding noise to word vectors may result in representations that are close to other tokens, which does not necessarily result in coarse-grained semantics. From a generative perspective, the Gaussian noise in the embedding space actually simulates the (continuous form of) uniform noise diffusion rather than masked diffusion. In contrast, our discrete formulation of intermediate states has clear meanings and definitions for semantic clusters, which is not the case for continuous embedding. Moreover, our coarse-to-fine hierarchies are introduced in an additional vertical dimension (abstract semantics), while adding noise to the embeddings is still operating on the original vocabulary dimension."}]

Author rebuttal:

Meta-review: The paper introduces Hierarchical Diffusion Language Models (HDLM), a novel class of discrete diffusion models for language. The core innovation is to replace the standard, one-step denoising process (from a mask token directly to a word token) with a multi-stage, hierarchical approach. The authors provide a strong theoretical foundation for this "next semantic scale prediction" framework, deriving a closed-form ELBO and showing that existing models like MDLM are a special case. This hierarchical structure also uniquely enables a self-correction mechanism during generation.

The strengths of the paper include the novel and intuitive idea of hierarchical semantic diffusion and strong theoretical derivations. Reviewers concerned about the weak experimental results, the lack of ablation studies on key hyperparameters such as the number of semantic clusters and no qualitative analysis. These concerns were addressed by the authors during rebuttal and reviewers reached to a consensus of accepting the paper.

I recommend this paper for acceptance on the condition that the authors incorporate the extensive new experiments, results, and theory described in the rebuttal into the final version.
Decision: Accept (poster)