Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling
Authors: Tianyu Xie
Shuchen Xue
Zijin Feng
Tianyang Hu
Jiacheng Sun
Zhenguo Li
Cheng Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Masked diffusion models
Variational autoencoders
Latent variable models
Primary area: generative models

Abstract: Discrete diffusion models have recently shown great promise for modeling complex discrete data, with masked diffusion models (MDMs) offering a compelling trade-off between quality and generation speed. MDMs denoise by progressively unmasking multiple dimensions from an all-masked input, but their performance can degrade when using few denoising steps due to limited modeling of inter-dimensional dependencies. In this paper, we propose Variational Autoencoding Discrete Diffusion (VADD), a novel framework that enhances discrete diffusion with latent variable modeling to implicitly capture correlations among dimensions. By introducing an auxiliary recognition model, VADD enables stable training via variational lower bounds maximization and amortized inference over the training set. Our approach retains the efficiency of traditional MDMs while significantly improving sample quality, especially when the number of denoising steps is small. Empirical results on 2D toy data, pixel-level image generation, and text generation demonstrate that VADD consistently outperforms MDM baselines.

Published as a conference paper at ICLR 2026
VARIATIONAL AUTOENCODING DISCRETE DIFFUSION WITH ENHANCED DIMENSIONAL CORRELATIONS MOD-
ELING
Tianyu Xiel.*, "ee,,,,huun Shhuhhnn Xue Zijin Feng", Tianyang Hu Jiacheng Sun",
Zhenguo Li Cheng Zhang
1 1.2
1 School of Mathematical Sciences, Peking University
2 Center for Statistical Science, Peking University
3 Academy of Mathematics and Systems Science, Chinese Academy of Sciences
4 Huawei Foundation Model Dept
5 The Chinese University of Hong Kong, Shenzhen
ABSTRACT
Discrete diffusion models have recently shown great promise for modeling com- plex discrete data, with masked diffusion models (MDMs) offering a compelling off between quality and generation speed. MDMs denoise by progressively unmasking multiple dimensions from an all masked input, but their performance can degrade when using few denoising steps due to limited modeling of inter- dimensional dependencies. In this paper, we propose Variational Autoencoding Discrete Diffusion (VADD), novel framework that enhances discrete diffusion with latent variable modeling to implicitly capture correlations among dimensions. By introducing auxiliary recognition model, VADD enables stable training via variational lower bounds maximization and amortized inference over the training set. Our approach retains the efficiency of traditional MDMs while significantly improving sample quality, especially when the number of denoising steps is small. Empirical results on 2D toy data, pixel-level image generation, and text generation demonstrate that VADD consistently outperforms MDM baselines in sample quality with few denoising steps.
INTRODUCTION
Diffusion models (Ho et al. 2020 Song et al. 2020) have shown their remarkable successes in
generative modeling of continuous objects, e.g., image (Rombach al. 2021 Ramesh et al. 2022
audio (Kong et al. 2021 Liu et al. 2023). and video (Ho et al. 2022; Blattmann et al. 2023
Recently, diffusion models have also been extended to the discrete state space (Austin et al. 202 I
Campbell et al., 2022 Sun et al. 2023 Lou et al. 2024), achieving competitive
even superior performance to autoregressive models in tasks such as Sudoku solving and code generation. A notable example is the masked diffusion model (MDM) (Sahoo et al., 2224 Shi et al. 2024: Ou et al.) 2025). which works by progressively masking the dimensions of the data point towards an all-masked state in the forward process, and gradually unmasking (i.e., predicting the distributions of) multiple dimensions simultanen limultaneeusly in the backward process. This parallel prediction capability of MDMs offers great potentials of sampling acceleration upon autoregressive models, which typically predict one dimension at a time (Xu et al., 2024). However, the practical value of MDMs lies not just in their parallel generation capability, critically their ability to produce high-quality samples with minimal denoising steps key requirement for real-world deployment where inference speed is paramount.
Despite the inference efficiency of MDMs, their denoising distribution at each backward step is typically modeled as a product of independent categorical distributions across dimensions which may fail to capture complex inter-dimensional dependencies commonly present in real-world data. This
Email: t i saanuxieeppke 8pku. edu.ceddcccc
1 "ooresponning author. Email: chengzhang@math. wwt.tthhhppkkkke..e.eeeee edu.cc

---

Published as a conference paper at ICLR 2026
issue becomes particularly pronounced when using small number of denoising steps, where many dimensions must be unmasked simultaneously. amplifying the impact of independence assumptions. While recent efforts (Xu et al. 2024: Liu et al., 2024) have attempted mitigate this limitation, they require inner-loop sampling steps under guidance from pre-trained autoregressive models or correlation models, introducing additional computation costs.
In this work, we propose Variational Autoencoding Discrete Diffusion (VADD), a novel framework that enhances discrete diffusion models by incorporating latent variable structure into the denoising distribution. This structure enables the model to implicitly capture inter-dimensional correlations, thereby improving its approximation capacity. To address the intractability of the resulting marginal distributions, we adopt the variational autoencoding framework (Kingma & Welling 2014), jointly optimizing the denoising model and an auxiliary recognition model via a variational lower bound. We further design specialized transformer-based architecture for VADD that retains the fast inference ef- ficiency of traditional MDMs. Through experiments on 2D toy datasets, pixel-level image generation, and text generation, show that VADD consistently outperforms MDM casslines-particullae in terms of sample quality with few sampling steps- hrigh-highhggtting the benefits of modeling dimensional dependencies.
2 BACKGROUND
Masked diffusion models Let Ø5g be categorical sample with N dimensions and æn E {1,.... V be the i-th dimension of zy. We further augment the state space with a special mask state [M] = V+1. Let 8a denote one-hot vector of length V + with the value 1 at position The forward process of masked diffusion models (MDMs) defined as g< (zx.) II ((x]]a=)) and q( (x)) Cal(ar;: 2a
a, 0 (x0) for < I, where the mask schedule ar strictly decreasing function in t E [0, 1] with Ogg x and a₁ x O. Particularly, we have q( z))) = Cat(a); a, + (1 (1-a,) (m))
a, 0zg + R.=06
that will diffuse to all- mal-mdseed state [M] N at t 1.
Given vo, the posterior distribution of x takes the form q(0s) 2,, xo) = -I g( [x], x/) where
= Cat(al; dg;)s
x + [M],
q |2], x/)
-
(1)
Cat 1 ((g) +
Du- T-oee
at [M].
The symbol Cat refers to the categorical distribution. Equation backward transitions as
inspires parametrizing the
Pe (a, (an)) q(2,, [ 2o (10 1))
II g( [mt, anj pig (x),
(2)
where the denoising distribution no (m), E Nx(+1) with the constraint EV=1 14 (ax, !) =
and μ4 [x]] (at,, () is expected to match the posterior of the clean data q( ax.). The 14((x. () is
explicit and often trained by maximizing the continuous-time evidence lower bound (ELBO) (Shi
ct al. 2024 Sahoo et al. 2024)
C( 0) = S Eylam of log (xo xt)dtt < log (xoo
(3)
for all æo in the training set, equivalent to the training loss of any-order autoregressive models, as discussed in Ou et al. (2025).
Note that the backward transition distribution po (= (z:) in equation 2 is factorizable over N dimensions which helps reducing the modeling complexity of state space. However, this conditional independence structure fails capture the inter-dimensional correlations and would inevitably introduce cumulative approximation errors. As the number of dimensions unmasked simultaneously during a backward transition is proportional to (a, - at), this drawback can be more severe under a large step size.
Variational autoencoders Consider the generative modeling task on a dataset {y)...., .y.... where y: is N-dimensional continuous or discrete variable. The variational autoencoder (VAE)
2

---

Published as a conference paper at ICLR 2026
(Kingma & Welling, 2014) assumes generative model pe 2) ( ()))=)))(z) where R is
latent variable with a prior distribution p(z). and a recognition model n (2 y as an approximation for the intractable posterior (o(x)y) The generative model and recognition model can be jointly learned by maximizing the following evidence lower bound (ELBO)
Po x) ( = logro(y) - DEL
Lyy: 0, Φ) = (=ly)) log
(=ly)) S p (4)
for all data points f....... ymJ J. Here, po Su po z)dz is the marginal likelihood of y and DKL is the Kullback-Leeillrr (KI.) divergence.
A meean-ieeld structure is often assumed for the conditional distribution pe (0(y))), i.e., the dimensions of y are independently distributed conditioned on 2. As a comprehensible example, Kingma & Welling 20144) assumes the following distribution over a continuous image sample y
pe p (z) N (iag
where mg. σg R + RV are two learnable neural networks. As latent variable model, the marginal distribution Po( is still capable of modeling the complex dimensional correlations by integrating out the latent variable 2.
METHODOLOOGY
In this section, we propose Variational Autoencoding Discrete Diffusion (VADD), which extends the dimension-inndeeedden denoising distribution in MDMs by introducing a latent variable structure. This design enables VADD capture complex dependencies across dimensions. Both the denoising distribution and an auxiliary recognition model are jointly optimized under the variational autoencod- ing framework. The basic idea of VADD can also be transferred discrete diffusion models with other noise schedules (c.g., uniform distribution as the noise), which is an interesting future direction.
3.1 DENOISING MODEL IN VADD
Instead of parametrizing the backward transition pe (x. xe) as an explicit distribution, we define it as latent variable model:
/ pe (o xe, 2 p)((d)
pe (x, (x,
(6)
where p(z) is the prior distribution of the latent variable z E IRd and pa (2r [2,, z). an explicit probabilistic model, is the conditional distribution given the previous state o; and the latent variable z. Although the conditional distribution pa (x, [e, 2) may not capture the dimensional correlations of v., the marginal transition distribution Po (ar. (m)) capable of doing this by integrating out z. Throughout this paper, the prior distribution p(z) is the standard Gaussian distribution A(O,. Ta), and other multimodal distributions, e.g., Gaussian mixtures, are meaningful to explore the future.
Inspired by the To preerdiiition parameterization of MDM backward transitions in equation 2 we parametrize the conditional distribution (ox x1, II Po x/), 2) and
pa(x:/2), {{
Cat (x); x:): 1-az (xl + a-=as po 2,t,, I-ac
x 7 [M]
(7)
Cat x
x [M]
1-ac
where the n z, E I) satisfying E Me (Ix, (xx, 2,t) 1 and μà [M] (xx, z,t)
a deep model outputting the categorical probabilities of p papplar.. 2).
Intuitively, there are multiple possible ways to recover the clean data æo from the partially masked sample 2t, reflecting the multimodality of g( The latent variable z can be interpreted as a controller for high-level semantics, guiding the denoising model toward specific mode of the clean data. Figure T provides a comprehensive example on how VADD works on 2D toy examples. We see that MDLM cannot model this correlation in one step, suggested by the collapsed samples in the middle column. In contrast, VADD correctly captures this correlation and generates good samples from these multimodal distributions in one step.
3

---

Published as a conference paper at ICLR 2026
ground truth
MDLM (7
VADD (T=
ground truth
MDLM (7 1)
VADD (7
chockerboard
Figure I: One-step generation results of VADD and MDLM (Sahoo et al. 2024) on 2D examples.
Combining equation (6) and mated through
the conditional distribution of æo given æt. g (2) is approxi-
Pe (e xx)) = / a
N II 1=1
1.w0 14
(a 2, =)t))=;=(m)) Hang =i (M) p(z)dz.
(8)
However, directly maximizing the ELBO C(ru; 0) in equation 3 is no longer feasible, since ( 24) in VADD requires marginalizing out z which is intractable (equation (8). In what follows, we introduce an alternative tractable surrogate for the ELBO L( 0) in equation
3.2 THE VARIATIONAL AUTOENCODING MECHANISM
Inspired by the idea VAEs (Kingma Welling, 2014), we consider approximating the posterior distribution of the latent variable z with an auxiliary recognition model ra (æo, ax)) R pa p0(2|an 20 æz).
Now for the L( 0) in equation (3). by treating the po p (2,) itself as marginal likelihood conditioned on æt. the following equation gives lower bound of L(xo; 0):
-ay log
(2o
E(wo; 0, x)) = Eg( (x) at..
dt S L( 0).
(9)
(= (20g at,)
The L( O Φ)) in equation 9 is referred to the Double Evidence Lower Bound (DELBO), it a lower bound of ELBO (see more details in Appendix [A.]]. The equality in equation (9) holds and only if r (z) æo. æx) pe( [oo xx). i.c., the recognition model perfectly fits the posterior. In our implementation, the recognition model T 20, xx) is a diagonal Gaussian distribution, i.c..
r zx, = / (m xx), diag { (a" (z =.,)))
(zo.
(10)
where my and σ are two deep models approximating the mean and standard deviation, respectively, We find that this simple implementation works fairly well in numerical studies.
Similarly to the classical VAEs, the denoising model and recognition model can be jointly optimized by maximizing the DELBO C(arg; 0, <b) for ny in the training set. Since the rg mgg ar:) is Gaussian distribution, the reparameterization trick could be utilized to compute the gradient w.r.t. Φ. However, as the posterior distribution po(=|zo xx) is rather complex in the high-dimensional cases, naively maximizing L(x;; O, Φ) can encounter with the posterior collapse issue empirically. To tackle this problem, we borrow the idea of KL annealing from Bowman et al. (2015): Yang et al. (2017) and consider the following variant of DELBO
Ta (a(z|wo.co æ.)) p(z)
[moo 0,
Fegiee mn) (ro(= az
logro(zo/mm, =) Alog
dr, (11)
where A 1] is the KI. annealing weight. We summarize the training procedure of VADD in Algorithm
As in other MDMs, sampling from VADD also starts from an all-masked state and recovers the clean data with the backward transition (x) More specifically, given a time sequence [t: satisfying 0 = < <... < lT 1, sampling from po (a realized by recursively sampling
Algorithm 2: Sampling from VADD
Input: A sequence of time steps
0 to f1 < ... tr 1: A transition model po (or,, z). Output: The generated sample æp- Initialize Ity [M] N,
2r, ~ p(x) and Ity-1 p0 [ 2,,),
for T..... do
starting from Ety = [M] N. We summarize the sam- pling procedure of VADD in Algorithm
Sample z p(z);
Sample x;-1 ~ po (2),_, |26, 2);
4

---

Published as a conference paper at ICLR 2026
Algorithm 1: Training in VADD
Input: Number of iterations H: Optimizer Opt: KI. annealing weight As; Batch size B. Initialize po (0 [t; =) and (=(2210) zx);
for h 1,..., H do
Sample f (b) 1 1b=1 from the training data and (+ Mb=1 from Uniform(0, 1);
Sample fa: (36) 1b=1 based on ( 1b=1 and the forward masking process q( xo)):
(b) 1.
Sample fa (1) 1b=1 from the recognition model r xx00)::
Compute the KI. annealing weight An:
Compute the Monte Carlo estimate of DELBO Ex, Mc vc(0.0) : Eb= Ex, :.
0.
Update the parameters 0, Φ +- Opt(O, Φ, - Vo, MC (oc000)))
3.3 DESIGN OF THE DENOISING MODEL AND RECOGNITION MODEL FOR TEXTS
An important application of VADD is text generative modeling. However, the standard transformer architecture (Vaswani et al.) 2017) is not directly applicable for parametrizing the denoising model and recognition model for text generation. In this section, we address this limitation by specifically modifying the transformer architecture for both models and analyzing their inference complexities. Denote by the number of transformer blocks, by de the dimension of token embeddings, by de the dimension for latent variable embeddings, and by d the latent dimension.
Denoising model Recall that the the backward transition go (z. (z:) is implicitly defined by the denoising model [ 2,t) in equation 7 whose workflow is illustrated in Figure 2 a). To incorporate z, we introduce the AdaLN layer, an affine transform depending on z, that performs adaptive layer normalization (AdaLN) (Xu et al., (019).. Specifically, takes the embedding of z as input and outputs the shift and scale that will be applied to all token embeddings in the sequence. Each of the L transformer blocks consists of self-attention layer and feed-forward layer, both preceded by an AdaLN layer.
inner-block MLP first
Recognition model The recognition model rss (z) To. Ft.) takes two sequences, 2n and Tt, as input, but naively applying transformer to them will double the computational cost. Fortunately, the observation that æ, is a partially masked version 2gg inspires a simpler model design. We introduce binary vector M. E {0. 1) representing whether position is masked (= 1) not (= 0), and thus the pair (æyg x.)) can be bijectively mapped to (æo, M,) which is used as model input instead. This defines the recognition model r (2/ M,) ( æo, æ2). illustrated in Figure b). Similarly to the denoising model, an AdaLN layer is added before both the self-attention layer and feed-fore fooward layer in each transformer block, with the difference that this AdaLN layer is applied exclusively to masked tokens. This operation can be implemented by first defining a learnable mask representation vector R& (shared across blocks), then using two inner-block MLPs on Rs for outputting the shift and scale of the AdaLN layers in each block, and finally blocking their application to the unmasked tokens by multiplying M..
Remark. Although a more straightforward way for building T æy. x1.) is just ignoring the dependency o, and only taking zu input, find this will cause severe posterior collapse issue empirically, even if the KL annealing weight in the loss function [I]) is carefully tuned by us.
Complexity analysis For VADD, since both and R& are one-dimensional tensors, the complexity of the AdaLN layers is OLNdd + Ldgde), where is the sequence length. For the denoising model, the complexity of MLP is O(d..). and that of the standard transformer model is O(NVd, L(N'd,, + NNQ)). Therefore, the total complexity of the denoising model is ONNVd, + L(N2d, + Nd.) + LNd, Ld + dd.). Since the term (Ld,d, + dd.) is dominated, the overall complexity is thus to be O(VNd, + L(N2de + Nd.) ) + LNds). For the recognition model, the complexity of the AdaLN layer and transformer model remain the same in the denoising model. while the complexity of the MLP is (dd,)). Thus, the total complexity of the recognition model is O(VNd, + L(N2d, Nd)) + d. + Ldede + dd.) = O(VNd, L(N2d. Nd!) + L.Nd.).
5

---

It should be pointed out that the training cost of VADD is around 1.5x that of other MDMs with only a denoising model in our implementation (see Table 4 since two models of the same size are jointly trained in VADD using the KL-annealed DELBO loss (II). As future work for reducing the training cost, we can discard the recognition model and only optimize the generative model using Published as a conference paper at ICLR 2026
Ho(2, 2) E e Nx(V-1)
ma(2, M) o0(2, IR
Feed-Forward
Food-Forward
dg
de
2nd AdaLN de
2nd AdaL @
Leamable Mask
d
7X
Representation Vector Rs
Self-Attention
Self-Attention
1st AdaLN
de
Ast @
Ad
MLP
Embedding
Fimhedding
M
(a) Denoising model
(b) Recognition model
Figure 2: The network architecture of the denoising model and recognition model in VADD for text modeling. The feature dimensions of the tensors are marked in red font. @ M means that the module only applied the positions satisfying M'
By factorizing N out, the per-token complexity for both the denoising and recognition model is OV + LNd, + Ld?), where we omit the last term as it is one order smaller than the other terms.
alternative loss functions, e.g., score divergence. Despite the limitation of training cost, the sampling cost of VADD is the same as that of other MDMs, since the sampling procedure does not involve the recognition model.
RELATED WORKS
Several works have explored utilizing latent variable models improve text modeling (Bowman et al. 2015: Gu et al., 20188] Kaiser et al. 2018). Recently, Kong et al. (2025) used latent variable structure for the next token prediction autoregressive models and optimized with variational Bayes (Jordan et al. 1999). instead of the VAE amortizing over all training samples. [Hayakawa al. (2024) considered distilling the pretrained MDMs with model mixtures as the backward transition by optimizing the consistency loss. Di[M]O (Zhu et al.) 2025b) distills multi-step masked diffusion model into a one-step generator, by optiming the the model outputs of a student model for all possible intermediate states under the help of auxiliary model. Soft-Di[MJO (Zhu al. 2025a) integrates soft embeddings into Di[M]O distillation, which makes one-step generators end-to-end trainable and enables straightforward application of GAN-basel refinement, differentiable reward fine-tuning, and TTEO. Learnable Sampler Distillation (Fu et al. 2025) also employs a distillation approach in which a student sampler with few steps learns to align its intermediate score trajectory with that of high-quality teacher sampler with numerous steps. Our approach is clearly distinct from these distillation based methods as we train the latent-variable denoising model from scratch based on training data set, and do not rely on powerful pre-trained teacher model as the initialization weight and learning target.
The posterior collapse issue is common problem during training VAEs, i.e., the recognition model ignores the dependency on data and becomes very close to the prior distribution. From an optimization perspective, the recognition model can be less regularized by the prior through downweighting the
KL divergence term, termed as the KL annealing strategy (Bowman et al. 2015 Yang et al. 2017). As the posterior collapse issue can be partially attributed to the over-complex decoder, Gulrajani et al. (207)) proposes a hybrid architecture that combines a VAE with a weaker decoder, while Kim al. 2018) proposes strengthen the encoder. Dieng et al. 2019) adds skip connections from the

---

Published as a conference paper at ICLR 2026
Table Empirical JS divergence (J) between generated samples and ground truth samples, and NLL (L) evaluated on ground truth data. JS-T means sampling with 7 steps. The empirical JS divergences are evaluated based on 100K samples.
checkerboard
swissroll
circles
Model
JS-1
JS-5
NLL
JS-1
JS-5
NLL
JS-1
JS-5
NLL
MDLM
1.395
0.211
8.503
2.619
0.287
7.111
2.273
0.263
7.462
VADD
0.062
0.048
8.058
0.086
0.025
6.132
0.161
0.042
6.716
input directly to the decoder, reducing the burden on the latent variable. Kingma et al. (2016) uses
normalizing flows to model a highly flexible and complex prior, allowing the posterior to capture complex data dependencies instead of collapsing to the simple prior.
EXPERIMENTS
In this section, we demonstrate the effectiveness of VADD on three tasks: two-dimensional toy examples, pixel-level image generation, and text generation. For all these tasks, we reimplement MDM using the ELBO loss function defined in equation (3) along with the same training strategy as VADD. This reimplementation serves as our main baseline, which we refer to as MDLM, method concurrently proposed by three recent works (Sahoo et al. 2024 Shi al.) 2024 Ou al. 2025). Unless otherwise specified, we use the linear noise schedule 0; = f and the linear time steps t, = i/T for discretizing the backward process. For the ELBO evaluation in VADD, we use the 1000-sample lower bound variant of DELBO (see equation (20)
estimate for ELBO, which a common choice in the VAE literature (Burda et al.) 2016 Vahdat Kautz 2020 For all the negative- hiwtiie-iiiklhhhod-bbbbeeeee results are upper bounds. The experimental details can be found Appendix [ The official code is released at fhtpps : wtt.t/ttttbu.uu wtt.:/mmm//....t www./ttuiiieeccccccoooceooootooe eetisieee//DDD VADD We emphasize that VADD expected to achieve improved sample-qualit--based metrics, such as FID score and generative perplexity, with fewer sampling steps. However, we do not expect substantial improvements in likelihood-basee metrics, e.g., bits per dimension, perplexity, etc.
metrics, the autoregressive results are exact likelihoods, while the diffusion
5.1 TWO-DIMENSIONAL TOY EXAMPLES
We first apply VADD to two-dimensional toy examples to provide a concise understanding of how our method works. We consider three multimodal distributions: checkerboard, swissroll, and circles. The training set consists of 100K samples generated by the scikit- mrikittlearyy package (Pedregosa al. 2011). We rescale the data [0, 1) and discretize them with bin width of 0.01, resulting in V 100 for each dimension. Both MDLM and VADD are trained for 500 epochs with batch size of 256 and an initial learning rate of 0.0003 (annealing according to a cosine schedule). The KL annealing weight À in VADD linearly increases from 0 to in the first 100 epochs. The latent dimension is set to =
Table D reports the
(JS) divergence and negative marginal loglikelihood (NLL) of different methods, where we see that the VADD outperforms MDLM by a large margin on these quantitative metrics. Figure and Figure [Juue show the histplots of the samples generated by different methods and sampling steps, where we see that VADD is capable of generating samples that are much closer to the ground truth.
5.2 PIXEL-LEVEL IMAGE GENERATION
We then apply VADD to the lexel-level image generation task the binarized MNIST (padded to 32 x 32, V= 2) and CIFAR-10 10 (32 x 32, V 256) datasets. For both datasets, the denoising model and the recognition model adopt the UNet architecture in the PyTorrch implementation of variational
diffusion models (Kingma et al. 2021). The denoising model incorporates the latent variable by
fuusi oonddee11

---

Published as a conference paper at ICLR 2026
MDLM
8
(T 5)
VADD
3
5 6
6
3
5
4
8
S
9
7
9
(T 5)
MDLM (T = 20)
0
8
2
/
6
9
3
2
9
8
5
R
8
R
VADD
6
5
5
8
(
6
2
7
3
4
9
(T= 20)
Figure 3:
steps on the binarized MNIST dataset. Non-cherry-picked samples generated by different discrete diffusion models and sampling
Table 2: Test BPD (4) on binarized MNIST and
Table 3: FID score (L) with different sampling steps 7' on the CIFAR-10 dataset. The FID score is computed with 50K images using the clean-fid package. 'Reproduced by us.
CIFAR-10. 'Reproduced by us.
Model
#Parameters
BPD
Binarized MNIST (32 x 32)
MDLM
2.2M
0.075
T
MDLM
VADD (ours)
VADD (ours)
2.3M
0.063
10
334.3
170.3
CIFAR-IO (32 x 32)
20
261.3
108.7
D3PM LA-0.01
37M
4.40
30
203.4
84.8
MD4
28M
2.75
40
166.1
72.1
MDLM
32M
2.80
50
140.3
64.6
VADD (ours)
32M
2.74
100
76.5
50.5
adding its embedding to the pixel embeddings in each up block and down block. The recognition model employs the siamese mechanism (Koch et al.] 2013), which applies the same UNet to Do and æ, and finally aggregates the outputs. The denoising model in the MDLM baseline also adopts the same UNet architecture. Both MDLM and VADD are trained for 0.2M (binarized MNIST) or IM (CIFAR-10) iterations with a batch size of 256 and constant learning rate of 2e-4. The KL annealing weight À in VADD linearly increases from 0 to in the first 100K iterations.
Table 2 reports the bits per dimension (BPD) of different models. For the CIFAR-I0 10 results in Table 2. both MDLM and VADD are trained for 2M iterations to keep consistent with that of MD4 (Shi et al., 2024). We see that the latent variable structure provides more generative modeling capacity, as evidenced by the lower BPD of VADD. It is worth noting that the BPD reduction of VADD is much more significant on the binarized MNIST dataset, as the VAE framework generally works better on low-dimensional cases (a smaller V).
The binarized MNIST samples generated by MDLM and VADD with different sampling steps are plotted in Figure 3 We see that VADD can generate realistic digit images with even T = sampling steps, which is impossible for MDLM. For a direct metric for the sample quality, we report the Fréchet inception distance (FID) score between the generated images and the training
CIFAR-10 in T We see that VADD consistently generates more realistic images than MDLM with a small number of sampling steps T' See more image samples in Figure (MNIST) and F (CIFAR-IO) in Appendix C.2
5.3 TEXT GENERATION
Finally, we test the effectiveness of VADD for unconditional text generation, aligned with the common practice of diffusion language models. We consider two widely used datasets: One Billion Word (LMIB) and OpenOeeTeeet WebText. The denoising model and recognition model of VADD adopt the architecture in Lou al. (2024) based on diffusion transformer. with necessary modifications for incoporating the latent variable described in Section 3.3 The reimplemented MDLM baseline also adopts the architecture in Lou al. (2024). The sizes of these models all correspond to the
8

---

Published as a conference paper at ICLR 2026
GPTT2 small scale (around 125M parameters), i.e., T 12, de 784, although our VADD may have slightly more parameters introduced by the AdaLN layers (around 6%).. In our implementation, the denoising model and recognition model ignore the dependency (2025). For both MDLM and VADD, we use the AdamW optimizer with a constant learning rate of 2e-4 (after 2500 warm-up iterations) and exponential moving average decay rate of 0.9999. The KL annealing weight in VADD increases linearly from 0 to in the first 200K iterations. The results are collected after IM iterations with batch size of 512.
the time t, following Ou et al.
One Billion Word The One Billion Word (LMIB) (Chelba et al., 2013) ) is medium-sizeee and real-world dataset containing about 30M sentences. Following He et al. (2022): Lou et al. (2024). we use the standard train-test split, tokenize the data with the bert beart-baaeeee ba -aceeunnaasee sed tokenizer, and reorganize the data into sequences with a length of N = 128. The latent dimension in VADD is set to d = 128. Two additional autoregressive baselines, Transformer-XL (Dai et al. (019) with 0.8B parameters and OmniNet (Tay et al. 2021) with O.1B parameters, are considered.
The perplexities the test split of LMIB are reported in Table Our VADD achieves the best test perplexity among both the autoregressive and diffusion baselines. Note that VADD with only 1M iterations even surpasses the strongest autoregressive Transformer with 5M iterations. A possible explanation is that the sequences in the reorganized dataset contain around 75% padding tokens that won't 't be generated, making VAE especially powerful for this relatively low-dimensional problem.
Open Text The Open an spen-source repli- cate of GPT-2's WebText dataset (Radford al.] 2019). Following Lou et al. (002)). we use the last 100K docu- ments as the test split, tokenize the data with the GPT-2 tokenizer, and reorganize the data into sequences with a length N = 1024. The latent dimension in VADD is set to d 512. For the zero-shot learning task, we compute the perplexities of the trained model on the test splits of six datasets: Lambada, LMIB, WikiText, AG News, PubMed, and Arxiv, following Sahoo et al. (2024).
Table 4: Training speed (it/s) of differ- ent models with batch size of 512 on 8 H800 GPUs for the Open WebText ex- periment.
MDLM
EDLM
VADD (ours)
2.77
1.74
1.84
According to the generative perplexities with 16~ 1024 sampling steps depicted in Figure 4 VADD
consistently and significantly improves upon other MDM baselines when the number of sampling steps is small. Compared to MDLM, our VADD uses less than 50% computational cost to achieve the same sample quality. The reason lies in VADD's capability of modeling the joint distribution on multiple tokens. Table 1 reports the training speed of different models, where we see that the training speed of VADD is around 0.66 that of MDLM, which is faster than 0.5x. repoets zero-shot perplexities on six benchmark datasets. In most datasets, our VADD model achieves similar or better zero-shot perplexities than baselines, and we emphasize that MDLM uses the same architecture and training setting as VADD and is considered as the most appropriate baseline. This observation meets our expectation that VADD has more advantage the sample-quality-ban -basdd metrics than the weikeiihhoodbbaeeee metrics.
6 LIMITATIONS AND FUTURE WORKS
In the current framework, the prior distribution p(z) is assumed to be a simple standard Gaussian distribution. It is the conditional denoising model po po (ar,,,/a,, 2) that integrates the latent variable with o; and & to model the correlations among different dimensions of x,, However, this uninformed prior can suffer from the prior hole problem, i.c., there exist regions that have high probability under the prior but low probability under the model posterior. A more informed prior structure can be Po(z fe(z) t) that depends on the precedent partially masked state I; and the time variable t. More complex structures, e.g., hierarchical priors (Vahdat & Kautz] (2000). can also be employed for the prior Po (2/22,, t). Advanced training techniques (Hoffman & Johnson 2016 Aneej et al. 2020) are also applicable to alleviate the prior hole problem.

---

Published as a conference paper at ICLR 2026
OpenWeb Text
Table 5: Test perplexities (J) on LMIB. 'Reported in
Sahoo et al. (2024). 'Reproduced by us. The other results reported in their original papers. All shaded
300
MDLM
EDLM
model sizes correspond to GPT-2 small.
generative perplexity
VADD
Model
#lterations
Perplexity
100
Transformer-A XL
23.5
OmniNetr
21.5
60
Difffssion AR
Transformer'
0.5M
22.32
Transformer'
5M
20.86
40
DiffusionBert
1.9M
63.78
SEDD-Absorb
IM
32.79
10a
10t sampling time
MDLM
1M
27.04
MDLM
10M
23.00
Figure 4: Generative perplexities (J) eval-
MDLM
1M
27.70
uated by pre-trained GPT-2 large model based on 256 samples on Open WebText. All model sizes correspond GPT-2 small.
VADD (ours)
1M
20.53
Table 6: Zero-shot perplexities (L) on six benchmark datasets of models trained on the Open OpenWebTeet Text. All models are trained for 1M iterations. Results of GPT-2 are reported in Sahoo et al. 2024) "Reproduced by us. Other results are reported in their original papers.
Test dataset
Lambada
LMIB
WikiText
AG News
PubMed
Arxiv
GPT-2 (Radford al.
2019)
51.28
51.25
25.75
52.09
49.01
41.73
BEDD-Absorb (Lou et al. 2024
50.92
79.29
40.62
RADD-AO AO (Ou et al. 2025
49.43
70.71
35.25
MDLM (Shhoo et al. 2024
49.67
71.03
35.61
68.57
42.43
37.92
VADD (ours)
47.30
69.71
34.78
68.00
40.62
36.39
CONCLUSION
In this paper, we present Variational Autoencoding Discrete Diffusion (VADD), which extends the denoising distributions in MDMs with the latent variable structure, allowing for enhanced correlation modeling over different dimensions. This advantage of VADD makes it possible for correctly generating multiple dimensions simultaneously and is especially powerful with small number of sampling steps. The variational autoencoding mechanism is utilized to jointly optimize the denoising model together with auxiliary recognition model. Extensive experiments generation demonstrate that our VADD consistently outperforms the MDM baselines in terms of sample fidelity under few denoising steps. Our approach the first try of applying latent variable models and variational autoencoding mechanism to MDMs, and other expressive latent variable model designs and efficient optimization methods would be interesting future directions.
image and text
ACKNOWLEDGEMENT
This work was supported by National Natural Science Foundation of China (grant no. 12201014, grant no. 12292980 and grant no. 12292983). The research of Cheng Zhang was support in part by National Engineering Laboratory for Big Data Analysis and Applications, the Key Laboratory of Mathematics and Its Applications (LMAM) and the Key Laboratory of Mathematical Economics and Quantitative Finance (LMEQF) of Peking University. The authors appreciate Yu Zhu, Fangyu Ding, Ding Ding, and Haoxiong Liu for constructive discussion on this project. The authors grateful for the computational resources provided by the figh-peeformance Computing Platform of Peking University.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jyoti Aneja, Alexander G. Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. In Neural Information Processing Systems, 2020.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising difftusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34: 1 19981-17993, 2021.
A. Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Scung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 22563- 22575, 2023.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio, Generating sentences from a continuous space. arXiv preprint arXiv: 2015.
Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In International Conference on Learning Representations, 2016.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances Neural Information Processing Systems, 35:28266-282777 2022.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.388.
2013.
Zihang Dai, Zhilin Yang. Yiming Yang. Jaime G Carbonell, Quoc Lc. and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 2978 2978-2988, 2019.
Adji B Dieng, Yoon Kim, Alexander M Rush, and David M Blei. Avoiding latent variable collapse with generative skip models. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2397-2405. PMLR, 2019.
Feiyang Fu, Tongxian Guo, and Zhaoqiang Liu. Learnable sampler distillation for discrete diffusion models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoregressive neural machine translation. In International Conference on Learning Representattions 2018
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: A latent variable model for natural images. In International Conference Learning Representations, 2017.
Satoshi Hayakawa, Yuhta Takida, Masaaki Imaizumi, Hiromi Wakaki, and Yuki Mitsufuji. Distilla- tion of discrete diffuusin through dimensional correlations. In Workshop Machine Learning and Compression, NeurlPS 2024, 2024. URL httts:://ppereviieee net. forum? ?id= 1bx05x7kxc
Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. Diffusion- bert: Improving generative masked language models with diffusion models. arXiv preprint arXiw:2211.15029, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-668 -6511, 2020.
Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 2022.
M Hoffman and M. J Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, 2016.

---

Published as a conference paper at ICLR 2026
Michael I. Jordan, Zoubin Ghahramani, T. Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. Machine Learning. 17:183-233, 1999.
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In International Conference on Machine Learning, pp. 2390-2399. PMLR, 2018.
Yoon Kim, Sam Wiseman, Andrew Miller, David Sontag, and Alexander Rush. Semi-amortized variational autoencoders. In International Conference on Machine Learning, pp. PMLR, 2018.
D. P. Kingma and M. Welling, Auto-encoding variational bayes. In ICLR, 2014.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21707 2021.
Durk Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In /CML deep learning workshop, volume 2, pp. 1-30. Lille, 2015.
Deqian Kong. Minglu Zhao, Dehong Xu, Bo Pang, Shu Wang, Edouardo Honig. Zhangzhang Si, Chuan Li, Jianwen Xie, Sirui Xie, et al. Scalable language models with posterior inference of latent thought vectors. arXiv preprint arXivv2502.01556 2025.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference Learning Representations, 2021.
Anji Liu, Oliver Broadrick, Mathias Niepert, and Guy Van den Broeck. Discrete copula diffusion. arXiv preprint 24Xiv:24 2024.
Haohe Liu, Zehua Chen, Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark Plumbley. Audiol.DM: Text-oo-uuio generation with latent diffusion models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 21450-21474, 2023.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-firsi International Conference on Machine Learning, 2024. URL http://wwwwweeeeuteeememam
Intps:://pennevview net. Forum www.mmidd-ccccccPP-PC.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun. Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In The Thirteenth International Conference on Learning Representations, 2025.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier
Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, al. Seikit-leamn Machine learning in python. the Journal of machine Learning research, 12:2825-2830. 2011.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. ArXiv, abs/2204.06125 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Marroquin, Alexander M Rush, Yair Schiff. Justin T Chiu, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In The ehirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
12

---

Published as a conference paper at ICLR 2026
Jiaxin Shi, Kehang Han, Zhe Wang, Arnadd Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in Neural Information Processing Systems, 37: 103131-103167, 2024.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13445 2020.
Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous time discrete diffusion models. In The Eleventh International Conference on Learning Representations, 2023.
Yi Tay, Mostafa Dehghani, Vamsi Aribandi, Jai Gupta, Philip M Pham, Zhen Qin, Dara Bahri, Da-Cheng Juan, and Donald Metzler. Omninet: Omnidirectional representations from transformers. In International Conference on Machine Learning, pp. 10193-10202. PMLR, 2021.
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Neural Information Processing Systems (NeurIPS), 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. Advances in neural information processing systems, 32, 2019.
Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovec, Stefano Ermon, and Arash Vahdat. Energy-based diffusion language models for text generation. arXiv preprint arXiv:2410.21335 2024.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg- Kinkpairkpa Improved variational autoencoders for text modeling using dilated convolutions. In International conference machine learning, pp. 3881-3890. PMLR, 2017.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. In The Thirteenth International Conference on Learning Representations, 2025. URL
bttps://openzeview net. wtt.:/ooummmmmm 2tt
Yuanzhi Zhu, Wang, Stéphane Lathuilière, and Vicky Kalogeiton. Soft-di(mJo: Improving one-step discrete image generation with soft embeddings. ArXiv, abs/2509.22925 2025a.
Yuanzhi Zhu, Xi Wang, Stéphane Lathuilière, and Vicky Kalogeiton. Di[Mo: Distilling masked diffusion models into one-step generator. arXiv preprint arXiv:2503. 2025b.
13

---

Published as a conference paper at ICLR 2026
LLM Usage In the preparation of this manuscript, LLM was used polish grammar, style, and readability of the text.
A DERIVATIONS OF THE DELBO
A.1 THE LOWER BOUND OF ELBO
Let (= zo, æt)) be recognition model with support on R4, which can simply realized by assuming a Gaussian family for o zo. zz). We have
Pe (20 (=() (=/ xt)dz / po (oolo, z)p(z))de po axe)
(12)
Fib (2 at:)
Rd
Using Jensen's inequality and noting that log() concave function, we have
po (c,, 2)n(z) (z|200 )dz
log po (x,)
log
(13)
r (z) xt,)
g
po(xoo 22, 22 p)) r (z/200 2,
log
(14)
ro 20 xt
Rd
po (Io De. 2)p(z))
EE
(2/10, log
(15)
ribl 20 xc,)
The incquality (1) in holds and only if r xx) = po (z Fo xz). By noting that we have
>
T-a;
po po x 2 z)p(z)
Ü(x;; 0. 0. =
Englar (mo) (2)o
log
df
(16)
1- a2
ro (z xo. xx,)
-a6
By(ar|wo) 1- Ote
(17)
L( 0).
(18)
The inequality 17 holds if and only if for any t E [0 and æt g(x1|xo), it holds that ro (2) Fg. x,)
Pa æo æt).
Finally, we conclude that the following chain of lower bounds:
E(zo; O, Φ) S L(xo: 0) S Po po (2o)-
(19)
A.2 THE MULTI-SAMPLE VARIANT OF DELBO FOR ESTIMATING ELBO
For previous MDMs, c.g.. Sahoo et al. )024); Shi et al. 2024); Ou al. 2025). the ELBO L( 0) is reported as surrogate of the intractable likelihood log Pe (o and then used to compute the perplexities, bits per dimension, etc. However, for VADD, computing the ELBO is intractable as the integration on z is intractable. We consider the following multi-sample variant of the DELBO, termed K K-sample DELBO,
-a¹
1
K
pa x2,, z (p(*k) r am0 x.e)
(xo ④)
Beg Karalala Ct..
log
dl.
K
(20)
The K- K-sample DELBO satisfies
Σx (xo: 0, Φ) s Zx+1 (Zo; 0.4) < L(xo: 0)-
(21)
Moreover, by the strong law of large numbers, it holds that
lim Ex (zo; 0, Φ)) = L(x;: 0).
(22)
K
In all of our experiments, we use 21000 (zo: 0, Φ)) as an approximation for the L( 0) For the
Monte Carlo sampling on (zx, t), we sample 100 independent pairs (æ,, t) ~ q(xx)(())) (o)) U 00, to estimate the 22000(20; 0 ()).
14

---

Published as a conference paper at ICLR 2026
B DETAILS OF EXPERIMENTAL SETUP
B.I TWO-DIMENSIONAL TOY EXAMPLES
We consider three multimodal distributions: checkerboard, swissroll, and circles. The training set consists of 100K samples generated by the scikit-learn package (Pedregosa et al. 2011). For checkerboard, nrows=2 and ncols=2; for the swissroll, noise=0.2; for the circles, noise=0.02 and fac We rescale the data to [0, 1) and discretize them with bin width of 0.01, resulting in V = 100 for each dimension. The latent dimension set to d 2.
The denoising model μt (at, 2, takes integer- valued sample æt, and vecotr 2, and a scalar as inputs. All the activation functions in MLPs are ELU unless other specified. ( z, consists of the following two steps:
Embedding. The embedding t, emb(t), is the output of positional sinusoidal embedding Vaswani et al.) (017)) and a following MLP with channel widths [1024, 512, 512]. The embedding of 2, emb(2), is the output of an MLP with channel widths [d, 512]. The
embedding of æ. (a). is the output of an embedding module with embedding dimension 512.
Readout. We then sum up emb(( emb((:) and apply MLP with channel width [512, 512, 512, 512, $2,2V V].
The recognition model ro (2|200 xy) takes two integer- vectors Do Dy and a scalar value as input and output the vector-valued mean and standard deviation of We consider a Siamese scheme to incorporate the two inputs Zp, Tt.
Embedding. The embedding emb(t), is the output of positional sinusoidal embedding Vaswwan et al. 2017) and following MLP with channel widths [1024, 512, 512). The embeddings of By and æt, emb( (zo) and emb(x;), are the output of the same embedding module with embedding dimension 512.
Readout. We then apply MLP with channel width [512, 512, 512, 512, 512, 512] on emb(())+em(an) and emb(()++ebill)) (xx), and obtain the outputs out and out, Finally, we compute the average of outo and out,, and apply an MLP with channel width [512, 512, 2d].
We use the Adam optimizer with momentum parameters (B1, (2) (0.9, 0.999) and weight decay The initial learning rate is 3e-4 and decreases according to a cosine annealing schedule. The KL annealing weight increases linearly from 0 to 1 in the first 100 epochs. The batch size is 256. The total number of training epochs is 500. The experiments are run on a single A100 40G GPU.
B.2 PIXEL-LEVEL IMAGE GENERATION
The binarized MNIST data binarized MNIST (32 x 32, V = 2) are obtained by binarizing the grayscale MNIST data with threshold of 0.5 and padding 2 zeros each side of the image. CIFAR- 10 (32 x 32, V 256) can be directly obtained in its original form without any transformation.
The denoising model and the recognition model adopt the UNet architecture in the PyTorch imple- mentation (https:: //tithbb..ttt.../ comm addat.dtt/// /aattt/aaaiiieee iwarrittio ona -di fus conmmode s) of vari- ational diffusion models (Kingma et al. 2021).
Binarized MNIST. For both the recognition model and the denoising model, the numbers of up blocks and down blocks are 8. The embedding dimension 64. The pixel embedding dimension is 32. The dropout rate is 0.1. The number of normalization groups is 16. The number of attention heads is 1. The latent dimension is 64.
CIFAR-10. For both the recognition model and the denoising model, the numbers of up blocks and down blocks 32. The embedding dimension 128. The pixel embedding dimension is 32. The dropout rate is 0. The number of normalization groups is 32. The number of attention heads 32. The latent dimension is 128.
We make the following modifications for the denoising model and the recognition model.
15

---

Published as a conference paper at ICLR 2026
Denoising model. A special token [M] is also added to the table of the pixel embedding module. We transform the latent variable a into an embedding emb(z) with MLPs and add it to the embeddings of all pixels in each up block and down block.
Recognition model. Borrowing the idea of the siamese mechanism, a standard UNet takes (Ig. t) and (z,, t) as inputs, and outputs two tensors outo and out,, We compute the average (outo + out,)/2 and downsample it to 1 1 x 2d tensor with down convolution blocks. This x 2d tensor will give the mean and standard deviation of r (2)220, ( zt).
We use the A optimizer with (2) (0.9, 0.99), weight decay of 0.01, and a constant learning rate of 2c-4. The batch size is 256. The gradient is clipped to a maximum norm of 1. The KL annealing weight linearly increases from 0 to in the first 100K iterations. The total number of iterations is 200K for binarized MNIST and 1 1M for CIFAR-10. The exponential moving average decay rate is 0.9999, and the update frequency is 1. The experiments of binarized MNIST are on a single A100 40G GPU. The experiments of CIFAR-10 are run on 8 A100 40G GPUs.
B.3 TEXT GENERATION
For the One Billion Word dataset, we firstly detokenize the texts following Lou et (2024) tokenize the texts with the bert-base-ur cased tokenizer, following He et al. 20222; (2024). We pad and truncate the sequences to length of 128.
We then Lou et al.
For the Open WebText dataset, we firstly detokenize the text following Lou et al. (024)) We then tokenize the texts with the GPT-2 tokenizer. We concatenate and wrap them to a sequence length of 1024, including BCS and a EOS token as the first and last token of the sequence. We use the last 100K documents as the validation split.
The network architecture and inference complexity of the denoising model and the recognition model have been elucidated in Section 3.3 The model size corresponds to the GPT-2 small scale, i.e., 12 transformer blocks, an embedding dimension of 768, and 12 heads in each attention layer.
The optimizer is AdamW with (2) = (0.9, 0.999). a weight decay of 0.0. The learning rate is a constant 3c-4, with warmup phase of 2500 steps. The batch size is 512. The gradient is clipped to maximum norm of The KL annealing weight linearly increases from 0 to I in the first 100K iterations. The total number of iterations is MM. The exponential moving average decay rate is 0.9999, and the update frequency is 1. The experiments on One Billion Word are run on 1000 40G GPUs, and the experiments on OpenWebText Text are run on 8 H800 GPUs.
C ADDITIONAL EXPERI RESULTS
C.I TWO-DIMENSIONAL TOY EXAMPLES
Figure 5 shows the histplots of the samples generated by MDLM and VADD. We see that VADD generates samples that are more close to the ground truth with both 1 and 5 sampling steps. We also visualize the samples under 1, 5, 10, 20 sampling steps of VADD in Figure We see that VADD produces very similar samples in this 2D toy experiment.
C.2 PIXEL-LEVEL IMAGE GENERATION
Figure provides the binarized samples generated by VADD. The training and test negative DELBO curves for VADD and MDLM is plotted in We see that for both models, the training loss decreases steadily, and VADD will outperform MDLM in the end. We provide CIFAR-10 samples generated by VADD with different sampling steps Figure 9
We also consider a continuous DDPM (Ho et al., 2020) baseline, which is implemented by the diffusers. DDPMP pp 1 ine module with the google /ddpmccifaatt coipmn 0-32 pre-trained checkpoint. As shown in Table 7 VADD consistently outperforms the continuous diffusion baseline DDPM across all sampling steps on CIFAR-10. At 10 steps, VADD achieves an FID of 170.3 com- pared to 298.7 for DDPM, demonstrating 43% improvement. These results demonstrate that discrete diffusion can achieve competitive or superior sample quality compared to DDPM, particularly in the few-step regime. However, it should be acknowledged that the pixel-level discrete modeling of images
16

---

Published as a conference paper at ICLR 2026
ground truth
(7=1)
VADD (7
checkerboard
swissroll
circles
Figure 5: Histplots of the ground truth and the samples generated from different models and sampling steps on the 2D toy example.
ground truth
VADD (T= 1
VADD (7=
VADD (T= 10)
VADD (T=
checkertin checkerboard
SW wwissroll
circles
Figure 6: Histplots the ground truth and the samples generated from VADD with various sampling steps the 2D toy example.
quite hard, and the FID of MDM-based methods could lag behind other continuous modeling methods by a large margin.
C.3 TEXT GENERATION
Table shows the test perplexities on the WebTet dataset. To inspect how the number of particles will affect the estimate of ELBO, also perform ablation study the number of
17

---

Published as a conference paper at ICLR 2026
VADD (T 1)
3 6
3
5
5
7
VADD
3
5 6
0
6
3
5
4
J
S
7
9
7
9
(T 5)
VADD
3
3
5
0
6
?
3
9
t
2
5
1
5
9
(T = 10)
VADD (T= 20)
6 5 5
8
/
8
2 7
3
4 9
Figure 7: Samples generated by VADD with different sampling steps on the binarized MNIST.
Table 7: FID score (L) comparison between VADD and DDPM with different sampling steps T on the CIFAR-IO dataset. VADD consistently achieves lower FID scores across all sampling steps. FID is computed with 50K images using the clean- fid package.
T
10
20
30
40
50
100
VADD (ours)
170.3
108.7
84.8
72.1
64.6
50.5
DDPM
298.7
239.1
187.2
153.6
131.0
77.2
Binarized MNIST
CIFAR-10
MDLM (train)
4x
MDLM (train)
8x10-
MDLM (test)
MDLM (test)
VADD (train)
VADD (train)
Negative LELBO
7x10
VADD (test)
Negative LELBO
VADD (test)
2x
40-0
50000
100000
150000
200000
0.0
0.2
0.6
1.0
Training steps
Training steps
1e6
Figure 8: Training and test negative DELBOs on the pixel-level image generation task.
particles K (see the K-sample DELBO 20)).
Note that this estimate is identical DELBO when K 1. We see that the ELBO estimate gradually improves as K increases.
Zheng et al. 2025) points that the generative perplexity might be sensitive to the precision of categorical sampling. To this end, we compute the generative perplexity using fp64 in Table 9 where see that VADD consistently outperforms MDLM.
The generated samples from VADD trained on OpenWebText with 128, 256, 512, and 1024 sampling steps shown in Figure 13
respectively. The categorical sampling under fp32 precision.
D BROADER IMPACT
This paper presents work whose goal is advance the field of machine learning. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
18

---

Published as a conference paper at ICLR 2026
(a) 10 sampling steps
(h) 20 sampling steps
(c) 30 sampling steps
(d) 40 sampling steps
(e) sampling steps
(f) sampling steps
Figure 9: CIFAR-10 samples generated from VADD with various sampling steps.
19

---

Published as a conference paper at ICLR 2026
<lendoftext>> the symptoms.
New thoughts are possible, and experiences that will help keep a person of heightened focus, understanding and still committed to making them better and faced with a distraction. One of these persons may have even a good model of understanding for him, but there is always something else that allows the person to express his interests.
Sometimes you're roe're over it that way, then it goes away and over it think, one day it can make everything go away and tap open again, even if you feel the bridge almost comfortably, not going to a situation that suddenly forces you to move, but more difficult things take place that you couldn't 't both before, how you wanted to have family person with you when it happens, it's for so much help, as a privilege, and me both, even clandestinely, while being given authentic adult guidance, can be frustrating and disengaged, and when life turns upside down, you are on your own we're explaining the nuances of emotions. they take place without someone's 's presence.
The worst of all, though, is going to therapy the next morning, from my friend Volok Wuska. He dealt with his abusive abusive spouse for several months, assuming the gaze of his therapist but not directly expressing his own grieving feelings. He understands how you experience in healing, and effective language can be replaced with one that treated him a great deal of anxiety and anxiety. His spouse were afraid of their words, and for a day when they found themselves in grief, and being focused for a day afterward, simply for something to say. This case with a family issue and with family business at the same time and such, the only reason they didn't 't get through it was therapy helped calm down that source of tension. See, the biggest benefit this therapy reveals is that the listener and the wounded fully continue to heal.
The "them economics" of friendship is gift. At best, we know have better understanding, and there is some greater freedom for each other because, even if we talk we' 'ree still around, we can forever enjoy all life work logether with loved one in reciprocating meaningful but marginal if functional way. You can drink like your family relatives or colleagues who don't 't respect your dignity and again, "What do you screw me on the back for doing this for?"
The adult knows his feelings are over in uncomfortable circumstances. but it keeps him alive and likely to approach things that way, and allows people that learning means good and achieving good things, without doing or trying to limit our experiences or attain them. Mindful people try to remain hardy because we have our ideas, evidence and move on flow. But information, emotions and linguistics have time to be loose everywhere. If we want communicate more successfully, we have to try to avoid these situations, we have to follow you, we have to find a way of projecting, where knowledge, creativity and understanding is enjoyed from that outside mind, and what people usually throw away in the hope that life will keep hurting them even after arriving from people who might hurt them. We talk about way too much in our heads, or we'll I1 lose few jobs in that we actually practice over and over again, because instead of what we're feeling from the outside and getting a handle to it this urge becomes to believe something and take action. This is thinking that you can turn it off, and then trained to Act itself, when your enemy changes how he takes detains you, and I still believe in my part that getting a paycheck is worth it if you care. That said, a kackeating direction in his particular direction means that a person who of course has the lisp, and beliefs, will be acting on his opinions. If he isn't able to do that, he probably is not; we should go and wait for him to be listened to, but if you have something left out because of your rudeness and see it reproach and suspend it, don't 't coach yourself again with rudeness until your board of boosters, and take every opportunity to run other people you have to tell about it.
The bottom line? You choose you will, and people in your position will respect to do that. A person you choose leave on your side of the equation should know that you aren't 't supposed to take control, but you can interfere with simple tasks like toasting and photographing photos, so there may be less danger, if your statement disrupts expectations on that side.
Break down the Starbucks bar during dinner upon a your friends at Baniac a table and they give them one of their shots.
Then you go to the Arizona Tea and take it with them. Or, somewhere in the future, you are certainly going to walk away and judge any young person relating the perceptions of your lookalike and theories to reality because<dendoffexxis
Figure 10: Text sample generated by VADD trained on OpenWebText with 128 sampling steps.
20

---

Published as a conference paper at ICLR 2026
<lendoftextl> to friends while I was out, so gave out gifts of books and trying to discover new things and new goals everyday. I coaxed them around them, and done some things that I felt was important in my life; for one was to make both of my kids very happy. didn't 't want both of my kids having parent that I would be able of without making me self happy and catching me with my child happy and so happy that all that wanted success, private happiness, was, ultimately, about internalizing my relationship. I still have lot of my secrets, but as of today in this journey, these are what myself believe be real good secrets about the whole experience. Being a woman was telling what was true to me; storytelling about the things that made me serve a woman and make myself happy, and how I truly wanted a little more. When have left positive or negative mindset choose to not be the one I feel one day want to be, sometimes; it's not necessarily right in the every single moment. I did what I wanted, and I was more passionate about the work love, something my kids took care of and I cared about my children, and I honestly thought about my life and my future. This is just part of me that's s what it's really like to be happy about your life.
One change for me is when I truly knew about how everything matters. I started to not talk about meeting guys and move on, just talk of how I liked what I was going to do. I felt more commitment was important; what wanted to talk about was mentally show that I loved being around someone, rather than making a bunch of money; I would spend money on my life and really care about it. That I also made me feel confident I was masculine and that women were not; being single made me feel accepted and a better person to be masculine, and there are many more ways to make that connection; doing some of these things with real men will make this process easier through the fact that they have the women encouraging them to both be themselves and to help them relate to the other. And of course, I think men need a strong relationship as much as possible, and women need to speak well of each other. We have to ask why we can't 't take care of ourselves and stay far behind. We have to be compassionate, because that's 's if we see how in others can go beyond our comfort and to be more positive and true our self. And that's 's just as important that we believe that our friend can serve their friend; because we. can.
Being involved with ourselves, with relationship that can work, is to love the things want, and to like the things we like to think. Having more positive things built on us and our sense of self makes it really tough for us to control over it. Because men are our coaches when it comes to be much more than just single and married, we really need more help from a woman that is still alive and well; making her presence that important away. But being who we are, we've ve all been able to achieve less of our goals, maybe we desire to look even prettier but men would do these things for us; this isn't 'n't a time when men are coming from who we are, who we are and how we behave. Being single, and opening up those opportunities is about finding potential friends, and shining light into our inner lives. is about showing off where we are in a positive way.
That's why I felt like I got all the stuff I wanted without woman. There are more layers to failed relationship, where there may have been, and even during first transition time, it's difficult to imagine just how far woman has come within society. Women also tend to have more negative emotional brought-ups. Some depression is depression, depression due to their sexuality is leading state of depression, and a symptom of that alone lead them. It's also to those who have generally high expectations and positive life expectations. Don't 't imagine how a woman would has said to you to change this or break you. When health comes where gender accepted and correct. Being in person is person you truly want its love as them.
She's the guy the woman wrote about in order to protect people her mom knows are used just to press the power of self-important ideas, to shut off their entire personality, to deny themselves resentment at once for ever.
More people are accusing getting these attitude that's out in person; the women in the original author post reported believing that someone encouraged her into being a bisexual than her. Before the ex-girlfriend went into hiding in the other post, now the other post is saying that one of these threats is her being openly gay started being a cook box Oreo. <lendoftext>>
Figure 11: Text sample generated by VADD trained on OpenWebText with 256 sampling steps.
21

---

Published as a conference paper at ICLR 2026
<lendoftextl> for an average 11 hour period. T used to call attention to him and people that he looked up to in his area, so I would seek him out. Despite the fact I received so much attention afterward, I experienced some of my past urges. On the first day at Lee and Barron's 's came home with a little worry. I'm not perfect. I wanted it to be good be with him until 4PM every day. Everything he has done, I feel that deserve. If I could tolerate my mistakes and understood my strengths and weaknesses, as well as help him get my lunch out of reach, even then that I wondered why could not apologize, I should go further, thinking that if I did so, I would see again that I had changed nothing. Instead, I thought could get there the next morning during a day to find out what was needful, and given that I was both tired and my rest, I'd rather appreciate that he wants our help, who wants him to help us. He came back and saved me in those moments knowing that understood that I'm not being taken to advisor anymore, and hoped that could refine my advisor approach too. Once again, we were on the same page, he said the message that at its heart, no one should ever fail our business together. He brought the two together, he effectively told me he was working out and happy to listen upon my decisions in response to.
It wasn't 't long, however, that I caught myself caught up in his social decision-making, and also in his personal communication. Now, he speaks the truth, and speaks for the complexities that he is faced with today. Let's say it in words his version of the mantra "the word in a hand can change the mind." " Then, especially shortly after he walked away in office, I seem to have gotten off empathy on my own, didn't interact like this in life and couldn't 't change it, that was the point of the Presidential campaign. While I proud when policy and people replace the very different decision of forthcoming. I knew that I hadn't t. Anyone that has been in office, means you have to do the things we want to do, in order to continue on in the world, I know that I must give these instructions, act face to face with them, and never act outside the confines of office.
I had been intimidated by the base built, and I was definitely not confident. At the time, this is who I hoped to become. But, I had to let it be because everything was going so well. As it did, not only knew how easy it was, but did enjoy it some level. And yet, I was afraid that for someone that I was supposed to be to be expected. had to cast an eye to his life and learn on what he was now doing. When someone is doing well with certain outlook, they will only make improvement sometimes. People never deserve to be dependent on and have to learn from on some level.
Despite this transition to office, I tried staying away. Maybe wanted a speaking farewell. Instead, I left politics and put politics behind politics. I had a parent that seemed quite a brave choice to choose. therefore came as a surprise that while at my turn of events, came home showed lesson the most of people that know should be fired as myself needed to protect myself, it needs to be alright. My time at school in Berkeley is not easy to reflect upon, it's education itself, but I did learn something in it.
I learned how my friends here in this land trust me, they sent me out on this campus as a stranger with complete sincerity. Everyone opened their eyes, learned the simple contact's skills that garner respect of lots of others. From an identity gained the respect and respect to become part of such a large network. As I waited in line during the 3-hour wait alone, I wondered if everyone knew what happened and just wanted to find out how to accept me that no matter how positive the outcome. Things like this be sure was a painful situation live in.
I cannot attest to how difficult it is to learn, it's hard to imagine yourself when having interactions with people in life. I am fully determined that the higher marks my period of experience are through this college age always, that I am able to operate in the very different environment beneath the narrator's 's walls of a college where playing game with the people we have studied very different. That said, for me the result showed that I and the students in the group do pretty good job of keeping disdisappearing students up, calling them out and making some decisions based on them in theory. So, I can still get to learn what to say to them, rememberinggclendffeettt
Figure 12: Text sample generated by VADD trained on OpenWebText with 512 sampling steps.
22

---

Published as a conference paper at ICLR 2026
<lendoftextl> situations can you have with yourself in terms of expectations? "Well, now have to decide what we are going to do next. "What months have passed"? Sometimes our usual selfless conversations here are more deliberate. Be aware that on either side of your sleep schedule are constantly working at your social media, and getting a lot of value info about your job status tomorrow, where you'll be working, and where you might be with your friends. Don't tt think directly to yourself this way: "Given the amount of people you interact with, the so many experiences, how are you looking around to plan for your future and for your dreams, that may much easier to see when you go, and therefore making good actions, and the repercussions, and all of the advice that we'v ve always accepted, is not inevitable.
From what I know, that things are going to be done in line for longer than most people realize. Maybe that's one thing. We don't t need to plan those small steps. Sometimes we need to slow things down because then anticipate the steps we' ve already taken. If, one reason this takes a lot of practice is to bear with yourself. Because we have a lot of time to make things happen. We don't I have to wait for things out. No, we have stand by about the things we'v already done. Let's reflect what went before here past today, into the future. When your List comes up, you have to put those worries ahead of you and go and listen while sorta.
Even when you go looking for those things that'll be in there, at least behind the scenes, things you've already done, there are always some caveats. What do here will help you imagine these things and look around, and you put them into the hands and you see how big they are in operation. It's huge and they'll Il be hand weighed and they'll be right there. The sense of responsibility we take for ourselves in the frame of mind that we are being taught about, and expect, to be really at odds with that of so many others. Most of the time, you can make good decisions not to have one, the time, the budget, the really. But I believe it sure makes great choice to do it, consider it a strategy for going through the high points of your personal circle in your life in that you show things you already done there. "The days that reach is you."
When it comes time to finish the week, you read the regular stories for a newspaper, or read comments, The People & Stuff. Or read all the writing, The Fowls, board reading pinned to something Very Important. If, for instance, as I read the stories in the morning, there was a willingness to put the, just put down and a small willingness to try to just stick with them. I got a loss at blocking or whatever. was the board reading. "Hey. get down on my knees." "Always don't 't quit." I don't. t. If you't 've pulled yourself under from behind then you don't tend to see things through these modes of thought. Once you decide that's the challenge here, I myself very, very was invited by this. And so, here's visit to my teacher in college. It was the night when went, and it's the one remember most vividly. The Professor sat me face down, his hand on my shoulder, and said: "Take care of yourself." And said that, there's no bad example. "I know this the only way to get an absolute best understanding of that." To think of my life outside the classroom, when hadn't 't gone to college, it was finally done over. My wife used to be a college student, and as well as I, believed in it when I went back to college. Same places where used to have a chance interview with my spouse, told myself I'm not poor experience in college anyway, and it ended up helping. I've always enjoyed finding a voice in a tough time, but still had to go to. I had to settle down. I sorta did the whole thing.
Once I moved away and I found out that job you can do in your own house isn't ' even moving around, I just took it... It out the window well. I've often been calm, truly relaxed person. I've tried to be better, and maybe just leave that behind.
In the late months of the school year, it's very important to put your mind in a work mode before your move back to the City. Because, sometimes the road ahead is all before you when you decide how to sleep. Is this in your control.<tendoftexti
Figure 13: Text sample generated by VADD trained on Open WebnnWeTeext with 1024 sampling steps.
23

---

Published as a conference paper at ICLR 2026
Table 8: Perplexities the test split of OpenWeeTeee All the models are trained for 1M iterations with batch size of 512 and a context size of 1024. All model sizes correspond to GPT-2 small.
Model
Perplexity
AR (reprduced by Sahoo et al. (2024)
17.54
SEDD (reproduced by [ahooo et (2024)
24.10
MDLM (reported by Sahoo et al. (2224)
23.21
MDLM (reproduced by us)
23.07
VADD (K = 1)
22.56
VADD (K 10)
22.33
VADD (K 50)
22.36
VADD (K 100)
22.27
VADD (K 1000)
22.21
Table Generative perplexity the model trained on OpenWebTexs The results are averaged over 256 independent samples. The categorical sampling is done under fp64 precision.
Number of sampling steps (T)
16
32
64
128
256
512
1024
MDLM (reproduced by us)
330.39
191.96
141.23
124.12
115.12
108.87
104.58
VADD
194.08
124.23
98.82
89.24
85.93
82.11
78.27
24


Reviews:
[{"review_id": "YbBoeBIynz", "reviewer": "Reviewer_ZCGk", "summary": "This paper proposes variational autoencoding discrete diffusion to enhance the dimensional correlation modeling. Along with introducing additional recognition model, they also propose some adaptions to architecture design. The experiment shows that the model achieve better FID than MDLM in text and image modeling.", "questions": "See that weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-organized and easy to follow.\n\n2. The performance of model outperforms other discrete diffusion model in pixel and language tasks.", "weaknesses": "1. The paper does not technically sound to me. It is not clear why adding latent variable $z$ contribute to model performance since in the sampling, $z$ is simply sampled from prior $p(z)$ distribution without using recognition model. $z$ seems like a style vector like StyleGAN or DDGAN which just help to increase the stochasticity of the architecture.\n\n2. Why not using the train recognition model for other iteration $i < T$ like hierarchical latent VAE technique. It seems a wasteful for training   recognition model.\n\n3. The performance on pixel space seems marginal. \n\n4. The proposed architecture is simple for denoising model, and does not show much novelty and lacks ablation study on different way to integrate $z$ into the model architecture in both denoising and recognition model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T6ETtWWVqe", "reviewer": "Reviewer_UPfa", "summary": "The authors propose the Variational Autoencoding Discrete Diffusion (VADD) framework which augments masked diffusion models (MDM) with latent variable modeling (using an auxiliary recognition model) for implicitly capturing the dependencies between different tokens when unmasking during the denoising process. This results in a more expressive unmasking framework in the low timestep regime. Empirical results show that VADD outperforms the MDM framework on different benchmarks in the low timestep regime.", "questions": "See weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I specifically like the problem formulation in the paper. The idea of using latent variables to model the dependencies between different dimensions is quite straightforward yet elegant.\n\n2. The paper is well written in the sense that the method is easy to follow and the empirical results establish the advantages of the method over the baseline MDLM model.", "weaknesses": "**Looseness of the Double ELBO (DELBO) bound**: Can the authors comment on how loose is the bound in Eq. 9 compared to the original log likelihood i.e. log p(x_0) and add more intuition in the main text. This could be quite important for applications where metrics like bits per dim (bpd) are quite important.\n\n**Missing Related Work**: The related work section in the paper is extremely limited. Can the authors discuss more about related work in discrete diffusion models, VAEs and posterior collapse. There is a large body of work in discrete diffusion which might be complimentary to the framework introduced by the authors. However an extensive discussion is currently missing from the paper.\n\n**Re. Empirical Results**:\n\n1. Can the authors add continuous diffusion based baselines for image generation for CIFAR-10? Based on my understanding the image quality in Table 3 is rather bad for CIFAR-10 and continuous time methods do much better on this benchmark. In light of this maybe these results can be moved to the Appendix altogether and the remaining space could be utilized for adding more intuitions on the theoretical results in the paper. Moreover, I would request the authors to include qualitative results for CIFAR-10 in the Appendix for a better sense on the quality of the samples.\n\n2. Do the authors have insights about the scaling properties of VADD with model size in terms of perplexities for the text benchmarks?\n\n**Discussion of limitations**: There is limited discussion on the limitations of the methods. In my understanding enforcing the recognition model to lie close to the prior would most likely suffer from prior hole problems since the underlying recognition model can be quite complex. In general, the proposed framework would inherit the same problems that classical VAE based approaches suffer from. Therefore, I think its quite important to discuss these limitations in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7XyxA3maio", "reviewer": "Reviewer_qhUA", "summary": "This paper proposed Variational Autoencoding Discrete Diffusion (VADD), which is a framework that augments masked discrete diffusion models (MDMs) with a latent variable to better capture the correlations between different dimensions. Specifically, the transition probability $p_{\\theta}(x_s | x_t)$ in the backward step is parametrized as the marginal of the product distribution between prior distribution $p(z)$ of the latent variable and the conditional denoising distribution $p_{\\theta}(x_s | x_t, z)$ given the latent variable $z$. As the marginal is generally intractable, the authors introduced a “double ELBO” (DELBO) framework to jointly train an auxiliary recognition model $r_{\\phi}(z | x_s, x_t)$ that approximates the distribution $p_{\\theta}(z | x_0, x_t)$ and the conditional denoiser $p_{\\theta}(x_0 | x_t, z)$. Experiments on 2D multimodal toy distributions, pixel-level image generation on MNIST and CIFAR, as well as text generation via MDMs are provided to justify the effectiveness of proposed method, which implies that adding an amortized latent to discrete diffusion does improve the quality of samples generated by MDMs.", "questions": "For the backward transition probability parametrized by the latent variable defined in equation (6), would it possible to make the prior distribution of the latent variable $z$ dependent on the time variable $t$? The reviewer thinks that it might be more reasonable to do so, as the correlation between different dimensions also varies as time changes. (Otherwise, would it be possible for the authors to comment on why it makes sense to use the same prior distribution for all time $t$ here?)\n\nMoreover, it seems that the authors only tested the effect of VADD for MDMs. Would it be possible for the authors to comment on/briefly discuss whether the same idea of using latent variable models can also be applied to the case of uniform discrete diffusion models or not?\n\nOverall, the reviewer thinks that the paper might be considered for top ML venues like ICLR, but the authors should probably address all questions above, add papers listed below as extra references and discuss them appropriately.\n\nReferences:\n\n[1] Kong, Deqian, Minglu Zhao, Dehong Xu, Bo Pang, Shu Wang, Edouardo Honig, Zhangzhang Si et al. \"Scalable language models with posterior inference of latent thought vectors.\" arXiv e-prints (2025): arXiv-2502.\n\n[2] Hayakawa, Satoshi, Yuhta Takida, Masaaki Imaizumi, Hiromi Wakaki, and Yuki Mitsufuji. \"Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusion.\" arXiv preprint arXiv:2510.04525 (2025).\n\n[3] Li, Tianhong, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. \"Mage: Masked generative encoder to unify representation learning and image synthesis.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2142-2152. 2023.\n\n[4] Fu, Feiyang, Tongxian Guo, and Zhaoqiang Liu. \"Learnable Sampler Distillation for Discrete Diffusion Models.\" arXiv preprint arXiv:2509.19962 (2025).\n\n[5] Zhu, Yuanzhi, Xi Wang, Stéphane Lathuilière, and Vicky Kalogeiton. \"Di [M] o: Distilling masked diffusion models into one-step generator.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18606-18618. 2025.\n\n[6] Zhu, Yuanzhi, Xi Wang, Stéphane Lathuilière, and Vicky Kalogeiton. \"Soft-di [m] o: Improving one-step discrete image generation with soft embeddings.\" arXiv preprint arXiv:2509.22925 (2025).\n\n[7] Hayakawa, Satoshi, Yuhta Takida, Masaaki Imaizumi, Hiromi Wakaki, and Yuki Mitsufuji. \"Distillation of discrete diffusion through dimensional correlations.\" arXiv preprint arXiv:2410.08709 (2024).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The reviewer finds both the presentation and the motivation of the paper to be crystal clear. To the best of the reviewer's knowledge, this is probably the first work that tries to address the problem of capturing the correlation between different dimensions via the latent variable structure in MDMs, which is motivated from existing work [1] that uses the latent variable structure for next token prediction in autoregressive models. Details about the choice of hyperparameters in the experiments are also included.", "weaknesses": "(1) For the image generation task, this paper only tested the proposed methodology on relatively small-scale datasets like MNIST and CIFAR10. In contrast, existing work [2,3] applying MDMs and related variants to image generation have tested on larger-scale datasets like ImageNet 256. The authors are encouraged to include an extra set of experiments on large-scale image datasets like ImageNet to better illustrate the effectiveness of the proposed methodology. (For instance, the authors may refer to the experimental setting of [3].)\n\n(2) Given that the method proposed in this work can be categorized as a training-based type of method for improving the quality of the samples generated by MDMs, the authors might need to perform a more detailed literature review on related methods. For instance, when the authors discussed distillation-type of methods as related work at the beginning of section 4, they only mentioned [7] and missed a few other concurrent work like [4,5,6]. Also, the quality of the paper can be further improved by adding extra training-based baselines (like those based on distillation) to justify the effectiveness of VADD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bs1RIrb9Ak", "reviewer": "Reviewer_Dd6V", "summary": "The paper proposes VADD, a generative framework combining discrete diffusion models  with a variational continuous latent variable. The goal is to address the limitations of purely discrete diffusion by introducing a latent variable z that captures global structure, while the discrete diffusion handles token-level generation. The joint training objective is derived as a variational ELBO decomposition into generative model and recognition model.", "questions": "Please refer to the \"Weaknesses\" section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThis paper presents a novel idea that introduces latent space modelling to discrete diffusion models, with a VAE-type learning objective formulation. \n2.\tExperiment results on image and text generation demonstrate the effectiveness of the proposed model.", "weaknesses": "1. The motivation for introducing a latent variable into the discrete diffusion model is not sufficiently justified. As discussed in Paragraph 2 of Section 1, the issue with MDMs becomes more pronounced when the number of denoising steps is small. However, if a sufficient number of denoising steps are used, would incorporating a latent variable still offer any advantage in terms of accuracy or efficiency? The paper provides no analysis or evidence to support this claim.\n\n2. For image generation, how is the performance comparison with SOTA continuous diffusion models? The comparison is conducted with only non-latent space modeling, which seems not sufficient.\n\n3. In Section 4, a couple of related works are mentioned. Although there is difference between them and the proposed VADD model,  they need to be incorporated as baselines.\n\n4. No code is released for reproducibility checking.\n\n5. For text generation task, how is the latent space defined?\n\n6. Several symbols like “Cat” are not defined in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)