Glauber Generative Model: Discrete Diffusion Models via Binary Classification
Authors: Harshit Varma
Dheeraj Mysore Nagaraj
Karthikeyan Shanmugam
Venue: ICLR 2025 Poster (2025)
Keywords: Discrete Diffusion Models
Diffusion Models
Non-Autoregressive Models
Generative Models
Markov Chains
MCMC
Primary area: generative models

Abstract: We introduce the Glauber Generative Model (GGM), a new class of discrete diffusion models, to obtain new samples from a distribution given samples from a discrete space. GGM deploys a discrete Markov chain called the heat bath dynamics (or the Glauber dynamics) to denoise a sequence of noisy tokens to a sample from a joint distribution of discrete tokens. Our novel conceptual framework provides an exact reduction of the task of learning the denoising Markov chain to solving a class of binary classification tasks. More specifically, the model learns to classify a given token in a noisy sequence as signal or noise. In contrast, prior works on discrete diffusion models either solve regression problems to learn importance ratios, or minimize loss functions given by variational approximations.  We apply GGM to language modeling and image generation, where images are discretized using image tokenizers like VQGANs. We show that it outperforms existing discrete diffusion models in language generation, and demonstrates strong performance for image generation without using dataset-specific image tokenizers. We also show that our model is capable of performing well in zero-shot control settings like text and image infilling.

Published as a conference paper at ICLR 2025
GLAUBER GENERATIVE MODEL: DISCRETE DIFFU-
SION MODELS VIA BINARY CLASSIFICATION
Harshit Varma
Dheeraj Nagaraj Google DeepMind dheera jnagara com
Inception Labs
harshit@incen ionai..ionn..... xyz
Karthikeyan Shanmugam
Google DeepMind karthikeyanvs@rooge..cm com
ABSTRACT
We introduce the Glauber Generative Model (GGM), a new class of discrete diffu- sion models, obtain new samples from distribution given samples from dis- crete space. GGM deploys a discrete Markov chain called the heat bath dynamics (or the Glauber dynamics) denoise a sequence of noisy tokens a sample from a joint distribution of discrete tokens. Our novel conceptual framework provides an exact reduction of the task of learning the denoising Markov chain to solving a class of binary classification tasks. More specifically, the model learns to clas- sify given token in a noisy sequence as signal or noise. In contrast, prior works on discrete diffusion models either solve regression problems to learn importance ratios, or minimize loss functions given by variational approximations. We apply GGM to language modeling and image generation, where images are discretized using image tokenizers like VQGANs. We show that it outperforms existing dis- crete diffusion models in language generation, and demonstrates strong perfor- mance for image generation without using datasct-specific image tokenizers. We also show that our model is capable of performing well in zero-shot control set- tings like text and image infilling.
INTRODUCTION
2020 Song al. 2020) use continuous time, continuous space diffusion processes to sample from a target distribution. These models start with pure noise and learn 'denoise' until meaningful sample is produced and are very successful in generative modeling of images (Rombach et al. 2021 Patil et al. 2022 Saharia et al., 2022 Ramesh et (021). ). These models are trained by solving a class of regression tasks called score matching (Hyvärinen & Dayan 2005). Another popular choice is the class of auto-regressive models based on the transformer architecture, which are state-of-the-aa
Diffusion Models (Sohl-Dickstein et al. 2015 Ho et al.
generative modeling languages VVawwani et al. 2017 Kenton & Toutanova 2019, Brown et al. 2020 Team et al. 2023 Esser et al.) |021|Yu Yu et al. 2022). These models work with discrete set (the set 'tokens") ), where sequences of tokens make up meaningful text. They learn to generate tokens autoregressively by solving multi-class classification problem.
Our work considers discrete diffusion models, where the model learns to denoise sequence of to- kens to produce a sample from a given distribution over token sequences. In language generation, this can mean starting from gibberish and denoising it to obtain meaningful sentence. Such models have been studied in the literature to try and combine the successes of Diffusion models and Autore- gressive models. We propose new class of discrete diffusion models called the Glauber Generative Model (GGM) where the the denoiser is a discrete time Markov chain over token sequences, called the Glauber dynamics (also known heat bath dynamics and Gibbs sampling). This is a central object of study in Statistical Physics of spin systems, Probability Theory and Bayesian inference
work done while employed at Google DeepMind.

---

Published as a conference paper at ICLR 2025
[Martinell]] 1999| Martinelli & Olivieri 1994 Levin & Peres 2017: Gelfand & Smith 1990: Ge-
man & Geman 1984; He et al. 2016) and is a discrete analogue of Langevin Dynamics. GGM is trained by solving well defined class of binary classifccasi problems- one of the simplest ma- chine learning tasks - to classify a specific token in a sequence of noisy tokens is "signal" or "noise'. In the context of language modeling, this roughly means figuring whether word is out-of-place in a sentence. We show an exact reduction from such a classifier to denoising Glauber dynamics based sampler in Section 3
1.1 RELATED WORK
Recent works have explored diffusion models for discrete data. Approaches for diffusion on discrete data can be broadly classified into two kinds: (i) discrete diffusion in the token space where discrete time Markov chain is the denoiser, (ii) continuous diffusion in an embedding space of the discrete data. We briefly discuss these approaches and their applications in this section.
Discrete diffusion via Discrete Markov Chains: Diffusion models over discrete spaces, analo- gous to continuous space diffusion, were introduced in (Sohl-Dickstein et al., 2015) for generative modeling. Argmax flows (Hoogeboom et al.) 202I) and D3PM (Austin et al. 202T) (and refine- ments such as (Zheng et al.J (023)) solidified these ideas - by considering the noising process to be a discrete time, discrete space Markov chain. They propose a variational loss function to teach a neural network to reverse this Markov chain. More specifically, D3PM considers character-level language modeling, token-level language modeling on short sequences (128) with relatively small vocabularies (8192), and low-resolution (e.g., 32 x 32) image generation tasks with small models (e.g., 36M)). On language modeling tasks these models still perform worse than autoregressive transformer- based model with comparable parameters. Scaling these models to larger vocabularies is an active area of research. MDLM (Sahoo et al. 2024) simplifies and improves upon D3PM by making different engineering choices and considering a simplified training objective for a specific noising process. Using image tokenizers like VQGANs (Esser al., 2021). VQ-DDM (Hu et al. 2021) and VQ-Diffusion (Gu et al.] 2021) are able to apply models similar to Argmax flows and D3PM to generate images with larger resolutions. Within this framework, DiffusER (Reid et al. 2022) considers the noising Markov Chain to be Levenshtein edit operations over text for language modeling and evaluates its performance for downstream tasks such as translation and textual style transfer.
While continuous diffusion models are learnt by solving the regression task of score matching, another line of work (Lou et al. 2023 Meng et al.) (022) considers its discrete analogue called ratio matching (Hyvärinen) 2007 Sun et al. 2022). These works that argue that Argmax flow and D3PM in fact learn these importance ratios of probability distributions indirectly, and propose to learn these directly by solving a class of regression tasks. While (Meng et al., 2022) considers least squares regression in this setting, (Lou et al. 2023) uses tailored loss function. Directly learning these ratios improves the learning performance as observed in SEDD (Lou et al., (003). achieving competitive performance autoregressive models for language modeling via discrete diffusion.
Discrete Diffusion via Continuous Diffusion over Embeddings: This line of work, introduced by (Li al. 2022) and further explored in (uulaaaaii & Hashimoto, 2023 Strudel et al. 2022 Ye al.)
2023 He et al. 2022; Yuan et al.) (022)) considers embedding the discrete space in continuous space and applies continuous diffusion models. Most notably, Plaid (Gulrajani & Hashimoto 2023)
(1.3B parameters) outperforms a 124M GPT2 model in likelihood on several language modeling benchmarks. However, GPP22medium with 345M parameters (3.8x less parameters than Plaid) still outperforms Plaid on all of the benchmarks used in the evaluation - indicating a need for further improvement. Approaches like SSD-LM (Han et al. (022) and TESS (Mahabadi et al., (023) apply continuous diffusion in logit space constructed over the vocabulary and evaluate on downstream tasks such as question generation and summarization.
Diffusion--ased approaches for modeling discrete data have also been utilized in several recent applications. DiMA (Meshchaninov et al. (024) uses continuous diffusion in the embedding space of a protein language model and outperforms autoregressive language models at unconditional protein sequence generation. DFMs (Campbell et al. (024) propose a class of flow- based models designed for discrete data that achieve sthee-of-the-art results for protein co-design.
Applications:
2

---

Published as a conference paper at ICLR 2025
Glauber Dynamies: (also known Gibbs Sampling) has been studied extensively in the statis-
tical physics literature to understand spin systems (Martinelli & Olivieri 1994 Martinelli 1999).
The study of its mixing properties for various spin systems is important sub-field of probability
theory (Levin & Peres 2017 Benjamini et al. 2005 Diaconis & Ram 2000, Mossel & Sly 2010)
Gheissari & Sinclair 2022;
Guo & Jerrum (017). Gibbs sampling has found numerous applica-
tions in Statistics and Bayesian Machine Learning for sampling from discrete posterior distributions
(Gelfand & Smith
1990.
Geman & Geman
1984
Zhang et
2001
Zhu et al. 1998
1.2 OUR CONTRIBUTIONS
We introduce a novel theoretical framework for training discrete diffusion models, called GGM, based an exact reduction to solving class of x() binary classification problems where X' is the set of tokens and T is the number denoising steps. Prior works based on D3PM incur a complexity of ?TT in general learn the denoising Markov chain directly (see Remark I. Moreover, the denoising process of GGM is a time dependent Markov chain which flips one token time, in contrast with prior works which flip multiple tokens simultancously. Our empirical evaluation shows that GGM obtains a strong performance in the case of language generation, where outperforms existing discrete diffusion models (Table We show that time-indepen inn-inndeeendnt Glauber dynamics implemented with masked language models like BERT cannot attain such performance even with 32 x more steps (Figure 2ap. We also demonstrate GGM's ability to generate high quality images on 256 x 256 CelebA-HQ and FFHQ datasets, where its performance rivals several popular diffusion models and GANs, without using a dataset-specific image tokenizer. We then demonstrate perform zero-shot image and text infilling with variety of different masks.
that model For the task of image generation, all the state-of-the-a methods use dataset-specific tokenizers and additional optimizations. We believe such techniques can further boost model's performance and bridge the gap between our models and the state-of-the-art. Thus, we believe our framework is conceptually elegant, scalable, empirically competitive, and has the potential to be widely used for generative modeling.
PROBLEM SETUP
2.1 NOTATION
By we denote discrete set and call its elements 'tokens'. Given L E N, we let X denote X valued sequences of length L. Given any X and i E {0...., L r), E 2 denotes the sequence of length L obtained form x when the i-th position removed. I; E X denotes the i-th position of x. Given finite set A. we let Unif(A) denote the uniform probability distribution over the elements A. For a probability distribution Q over a finite space V, with A C Q(AA) denotes its conditional distribution over A.
2.2 GLAUBER DYNAMICS
Let X be finite set (such as all possible tokens in a language model or in VQGAN). Glauber dynamics is Markov chain to sample a joint distribution over the space X by flipping one token time. Given probability distribution P over X and Xo E 2 Glauber dynamics obtains the trajectory Xo.... ,o...., Xr by sampling X,+1 given X, follows:
1. Sample I, ~ Unif({0, 1, 1,..., n 1}) independent of X.,
2. XI+1,i = Xz,i i h.
3. X,+1,1 P(X,
X,, if I
Glauber dynamics has P as its unique stationary distribution under mild conditions over P. Even with exact knowledge of [XX,|X_, = Xt.- the iterates Glauber dynamics can be very slow to converge to the distribution P. often requiring time exponential in L. In this work consider Glauber dynamics with two differences:
1. I, is chosen from fixed permutation in round robin fashion (as in (He et al. 2016
2. The dynamics is time dependent. That is, P = X,,+)) is replaced by X Xt.-i) for i 1......, T for some carefully designed P..
3

---

Published as a conference paper at ICLR 2025
b, uuc.b,yuu bierr 4==
D,,, J: b ,0,2)
b,,,, b
[g 1, b,
Z 6
(d....
P(X)
P(KesslXest)
No
Xr
P(X P(X (eaz-nna)
Figure Example of Glauber dynamics in a discrete token space, where the tokens are characters.
In this framework, we obtain an exact sample from at time ÖL We show case this difference by implementing time-independent Glauber dynamics via bi-directional transformer based masked language models. We describe this procedure based on (Wang & Cho (019) in Appendix IF] Our experiments (see Table [ BERT-large w/ Gibbs sampling) show that with a large number of steps, the quality of generation with this method much worse than GGM (see Figure 2a).
GLAUBER GENERATIVE MODEL (GGM)
Diffusion models in the continuous domain contain 3 crucial steps: (i) forward process: where sample from the target distribution is gradually noised to obtain sample from the standard Gaussian distribution over many iterations, (ii) reverse process: which time reverses the forward process i.e. it denoises Gaussian random vector into sample from the target distribution, and (iii) model training: where the reverse process learned with data from the forward process. We follow the same recipe to describe GGM below. Consider a finite set X and L E N. Suppose P is the target over x (i.c., sequences of length from X).
3.1 FORWARD PROCESS
Fix element & X. At time 1 (O....,T 1), let II, denote distribution over X' U (Ø) We think of II, as the noise distribution. Let io, E {O...., L 11 be fixed. We pick io...., -1 in a round robin way with respect to some permutation in order ensure that each coordinate appears in io...., ło...., ir at-least T/ L times. Suppose Xo is sample from P At time the coordinate i of X, is noised using II, in order to obtain X Draw token Z₃ II independent of X,. Set:
1}
X1,s if + in
X
Xt.5 jf it and Z. &
7, if in and 7, X
That is, X X2 with probability (T) and X is sampled from (- with probability 1 - (0). It is easy to obtain Xt, Xx+1 directly from Xo as described in Appendix H the distribution of X, by P,-
We denote
Lemma 1. Suppose that I&(+X²' = I[-+x)) is the same for every t. Suppose l1) < - e for some > As - oo, the distribution of XT converges to IT(-IX)OL total variation distance. Specifically, we have:
(VLaa((T), S L(1 - L(1
The proof of this lemma is given Appendix Under the choice II, (-Ix') Unif and (o) 2. In this case, Lemma shows that the forward process converges to Uni i.e. each token chosen i.i.d uniformly at random from X.
In the rest of the paper, Glauber dynamics goes reverse in time: X Xx. as is the convention in the diffusion model literature.
4

---

Published as a conference paper at ICLR 2025
3.2 REVERSE PROCESS VIA GLAUBER DYNAMICS
Consider Pt as given in the forward process. Suppose we have X,-1 ~ Pa+1. Then, the reverse process seeks to obtain sample X₁ Pt. We call the following time dependent Glauber dynamics update to sample X2 as the reverse Glauber dynamics:
1. Xis- = Xeti. for all j * is
2. Sample Xe,r, from the probability distribution P(Xe. = -|X++1,-i -eemūr)
Lemma 2. Suppose XT ~ Py. Suppose Xo is obtained by applying the reverse Glauber Dynamics times as defined above. Then, Xo P".
Proof. Let No...., Xy denote the forward process described in Section 3.1] We prove via induction that for any E 10,..., 1]. 8e+1 Pe+1 implies X, ~ Pt. Indeed, for any E X consider:
P(X, = x) IP + 1, 1, 2 P( 2ú, 1,,1,-i, = =x-i,)
= P(X = x - P Iú, (X,1, x-1,) = P(X, = x)) (t(x)
(2)
The first step follows from the definition of the reverse Glauber Dynamics. The second step follows from the induction hypothesis. Thus we conclude the result.
Remark 1. The forward process introduced in D3PM (Austin et al. 2021 noises all tokens at once and considers general class of noising processes. We consider a specific class of noising processes, and apply one token time. As we show below, this allows us to implement the reverse process by predicting single tokens instead of predicting for all tokens like in (Austin et al. Gu el al. (027). Moreover, this approach results an exact reduction to binary classification. Prior approaches to Discrete Diffusion Models had to learn the transition probability of token value a X to token value b E X at position i, scaling quadratically with |X sized transition matrix resulting in suboptimal performance (Hu al. 2021). Our training approach described below learns the probability that token E X appearing at position i in X was present in position of X, not, which scales linearly in |2()
3.3 MODEL TRAINING
We now describe how to train neural network with data from the forward process (Section 3.1 in order to implement the reverse Glauber dynamics (Section 3.2). The following lemma is the key to learning algorithm. We refer Appendix B for its proof.
Lemma 3. Suppose E x Suppose that (X.):=o....TT denotes the forward process. Then, for any a X:
PPX = a1 BalXi111111. =x -+,)
P(Z, ma) F(Z==)
(3)
F(2 X441 Xett. =aj
Therefore, we can estimate P(X,,,, = al Xet1,-ir = x-i)) using an estimate for PPZz Ii, [x+1 x). The latter is equivalent learning P(Z, = a| I-i X a) for every a E X.
consider the distribution D.a Law((X++1,-i, 1 12,=a|Xet1,4. = a). Given a sample X, ~ ps, we can obtain Xo, X,, 7,, X according the forward process, allowing us to sample from Dr... Consider the following binary classification problem (denoted by B2(P* 11, a)) corresponding to the data distribution Dr,.:
Reduction to Binary Classification: Given E A' and time
Predict 1 1z.ma given X
That is, we try to predict if X = a because of pure noise (i.c., Z, = a) or if it is because X = (signal). This can be solved by minimizing the cross entropy loss over model class with input Xi-1,-i- Such model learns to predict PP,, = a|X = I Z-i.; X = a).
Training Neural Network: Consider mask token & X, and parameterize neural network fo : (XU (w))² x (O....,T ..,,T-- + [0, 1] to solve the sequence of binary classification problems (.(P*, (P",,l,a) for & (o...., 1).a X. The neural network input is (x', ()) where x'
5

---

Published as a conference paper at ICLR 2025
(XU 2141, X for all j # it and xi w. The output logit corresponding E X' solves B, II, a). That is, fo outputs y fo(r', t) E [0. 1] where Da models ( = X x r'i..- X, = a). Note that using a special token like w allows to compute Ya for all E X in parallel in a single forward pass using the standard transformer architecture. The loss function and
the training algorithm is given in Algorithm
Algorithm 1: Training Glauber Generative Model (GGM)
Input: Dataset D, timesteps T, optimizer opt, model positions (&.) {T.)
Result: Trained parameters
Initialize 0, initialize the optimizer state opt iniiiaaizezze)
for each iteration do
Sample Xo D, t Unif( T- 2]);
Get X,, Z,, Xx+1 forward.process (X,, (is) (TI))==) // more
details about the forward process are given in Appendix
H
Set 8/+1,5 &- X1+1, for all + in Xi+1s. + w
// is the mask token
Compute loss C(0; ((+1) -1z77 log (fo(X ()).)))))))))) - 1z. log (1- teen))) < update VoL(0: Z, X(41)); // any optimizer like Adami, etc.
end return 8:
Implementing the Reverse Process: We now use the trained neural network to approximate the reverse Glauber dynamics. By Lemma [ whenever T is large Pr & II(-|x)4. We sam- ple XT [T)))2), the pure noise distribution. Note that it is sufficient estimate P(X a.x Given X,-1, let X/41 be its masked version in the position ic. Let ú = fo(8/+1: f). By Lemma 3 we can estimate P(Xt. =a X Xe,-i.) in step 2 of [a(aa (d) 1).
the reverse process with: P(X = -(
Algorithm 2: Inference from a Glauber Generative Model (GGM)
Input: Timesteps T, trained parameters 0, model /, noise distributions (T,)
Result: Sample N, from the target distribution IT"
Initialize X ~ (+
for 1 +- T
do
Set X X + X1+1, for all + iz, 8/41, + wi Get ú = fol 10(;;110))
// is the mask token
Compute P((X,,,, =a a|Xx+11-ix)=
(T(()) No -1) for all
ILfa) ( (1
Set Xe,i + 8 for all +i; Sample &ea, P(.1Xe+1,-ix))
// can do top-p sampling, etc. here
end
return Xo:
3.4 CONVERGENCE OF GGM
We provide a bound the total variation distance (TV) between the output distribution P of Al- gorithm 2 and the target distribution P in Theorem [ We provide the proof in Appendix
Theorem 1. Suppose that II, for every and II()) = p for some E (0. 1). Suppose
that our model learns PiZz, = a|Xe+1,-i = x- X a) perfectly from Algorithm If
T > L L/8) then the output distribution Po of Algorithm satisifes:
[ng(1/1-p)
TV(Po, P²)

---

Published as a conference paper at ICLR 2025
3.5 CONDITIONAL INFERENCE
While Algorithm 2 gives unconditional generations from a target distribution Pt, we now describe simple modification obtain conditional generation without any additional training. Suppose the tokens at places J = (f1....;JK) are conditioned to be (cI,..., CK). Then define Xr such that XT, ~ (I for & J and XT.. = cs. for all k = I..... K. We run Algorithm 2 as before, but do not update the tokens in the positions given by J. Using this, our models can generate tokens under a variety of different zero-shot control settings, like prefix/suffix completion. arbitrary infill- ing, and arbitrary lexical constraints. For example, provide examples of conditional generations for language in Appendix [..]] where our models infill the middle 512 tokens given the first and last 256 tokens. Figure 4bb shows examples of conditional generations from model on CelebA-HQ where our model is able to infill a variety of different pixel-space masks. We provide more details regarding obtaining the prompt tokens and positions from the pixel masks in Appendix D
3.6 ARCHITECTURE
Following (ouu et al.
2023) closely, we design fe to be a transformer model, based on the DiT
model family (Peebles & Xie 2022). We find that encoding time using the adaLN Zero approach
used in (Peebles& & Xie 20222 improves the convergence time of our models significantly. Like (Lou jt al. 2023) and (Gulrajani& Hashimoto, 2023). we too use rotary embeddings (Su et al. 2021 to encode position. When the 2t token to the model is masked (using w), we pass the final layer representation for the zh token to the classifier that returns ÿ E [0. 1| after applying the sigmoid activation on the logits. We provide hyperparameters and more architectural details in Appendix !I
RESULTS
4.1
LANGUAGE GENERATION
We train our models on the WeeTeWeTextt dataset (Gokaslan & Cohen 2019) and evaluate on lan-
guage generation. We compare with recent diffusion- based models for language (Plaid (Gulrajani
& Hashimoto 2023), SEDD-medium (Lou et al. 2023), and MDLM (Sahoo et al.] (024)) and
GPT2 models of different sizes in Table | Following (Lou al. 2023 Han et al.) 2022 Dieleman
ctt al. 2022). we evaluate unconditional generations of length 1024 tokens using generative perplex-
ity (Gen. PPL) measured by different, larger, autoregressive model like GPT-Neo-2.78 (Black eet al.) (021). Our model, with 387M parameters, outperforms SEDD-medium (424M parameters)
and MDLM, the previous state-of-the-an discrete diffusion models. Our model performs competi- tively with Plaid (1.3B parameters) with 3.4x less parameters and after being trained on smaller dataset (OpenWebTexx instead of OpenWeTText WebText2 (Gao et al., 2020)) However, all diffusion-based approaches lie significantly behind GPT2 models of comparable sizes. Following (Wang & Cho 2019), we include a MLM-based baseline which implements time independent Glauber dynamics as described in Section 2.2 to compare to time dependent Glauber dynamics of GGM. We begin with a noisy sequence of tokens, and take multiple token-by-token passes at the token sequence. At each position, we mask the corresponding token and pass the resulting sequence model (334M parameters) (Devlin et al. 2019) to re-sample token value in this position. We gen- erate and evaluate sequences of length 5 122 for BERT (maximum length that BERT allows). This approach is able to generate short phrases but is unable to generate text that remains coherent over 512 tokens and gets poor perplexity. We provide the exact sampling algorithm in Appendix[[] When sampling from GPT2 models, we force the models to generate text of length 1024 by setting the probability of the end-of-text token to For Plaid, SEDD-medium, and MDLM, we use the de- fault sampling algorithms as used by the authors in (Gulrajani & Hashimoto, (023), (Lou et al.) 2023), and (Sahoo et al., 2024) respectively. We include example generations from our model in Hyperparameters and other training details are provided in Appendix [[]
cased BERT-large
Appendix L
Remark 2. During sampling, we use top-p sampling, whereas SEDD and MDLM do not. However, recent work (Zheng et al. 2024 Section 5) shows that when using the Gumbel-max trick for sampling from a categorical distribution (as used by SEDD and MDLM, but not by our method), using FP32 precision instead of FP64 behaves equivalently to lowering the temperature during sampling. result of truncation. When this is fixed using FP64 precision, the default samplers for SEDD and MDLM M result in a generative perplexity of about 100 (more than 3x the reported values).
7

---

Published as a conference paper at ICLR 2025
Table Generative Perplexities evaluated over 1024 unconditional generations. We do not evaluate a larger model (e.g., GPT2-xI) with smaller model (e.g., GPT2-small).
GPT2-large (774M)
GPT2-xl
GPT-nco (2.7B)
Evaluation Model
(I.6B)
Evaluated Model
Sampling Algorithm
Total Params
Gen. PPL (1)
Gen. PPL (
Gen. PPL ())
Antoregressive Models
GP12-medium RRdford 2019
1op-p, 0.8 L 1024, 1024
315M
13.0
GPT2-large (Radford et
2019
top-9; 1024 1024,7
77-M
6.5
GPT2 xl Radford et al. 2019
1. 1024,7
wpp.r. p
1024
1.6B
6.8
Masked Language Models
Gibbs sampling Wang Cho 2019
BEKT-lagg
lop-p, p I. 612, T 2048
33-1M
487.0
187.5
188.7
Gibbs sampling Wang Cho 2019
lagg
top-ps p
L = 512,
65536
334M
25.7
23.4
26.4
Diffusion Models
Plaid Gulrajani & Hashimoto
2023
L 1021,
as per
Gulrajani Hashimoto
10260
2023
1.38
19.7
19.7
17.9
default as per et 2025
SEDD-medium (Lou et al.
2023
421M
27.3
23.0
25.2
1. 1024,7
2048
default as per Sahoo
2024
MDLM Sahno al.
2024
170M
44.2
45.4
40.9
f. 1024, 7
top-p. 0.8
GGM (ours)
L 1024,7
387M
19.5
19.9
18.0
500
BERT Gibbs sampling
Generative Perplexity
150
16
64
128
Number of times position is visited
(a) Generative perplexity of GGM BERT-large with Gibbs sampling. Inference is run for KL timesteps, where K is the number of visits per position (plotted on the z-axis),
Examples of 256 256 unconditional gen- crations from our model on FFHQ. We show more examples in Appendix G
4.2 IMAGE GENERATION
We tokenize images and de-tokenize tokens using an off-the-shelf VQGAN-basee tokenizer used by
the MUSE model (Chang et al. 2023). The tokenizer is trained on large. general image corpus with downsampling ratio /
Thus, a 256 256 corresponds to 32 32 1024 tokens. We report the FID (Parmar et al. 2022) values in Table for unconditional 256 x 256 image synthesis on the CelebA-HQ dataset (Krras al. 2017) and the FFHQ dataset (Karras et al. 2018). On CelebA-
HQ, our model outperforms several autoencoder-basee baselines by large margin and beats much larger 801M parameter GPT2- based autoregressive model that also works with VQGAN tokens [sssr al. 2021). We also outperform VQ-DDM (Hu et al. 2021), another model that applies discrete diffusion similar to (Hoogeboom et al. 2021) in the latent space of a VQGAN. On FFHQ, method outperforms the autoencoder baselines, the baseline version of StyleSwin (Zhang al.
8

---

Published as a conference paper at ICLR 2025
2021). the PixeISNAIL (Chen et al. 2017) model VQGAN tokens (Esser et al. 2021), and is competitive with other kinds of models like BigGAN (Brock et al. 2018).
Comparison with SoTA methods: We find a substantial gap in FID between our approach and SoTA methods like LDMs (Rombach et al, 2021). VO-Diffusion (Gu et al. 202T), and StyleSwin (Zhang et al. 2021). Based the findings in MAGVIT-v2 (Yu et al. 2024). that this gap can be brought down significantly, if not completely eliminated, by the use of more expressive and dataset-specific tokenizers. This can bring down the required sequence length, lead to more expressive and relevant tokens in the vocabulary, and thereby ease downstream sequence modeling. For example, the dataset-specific tokenizers used in (Esser et al., 202T) and (Gu et al. 2021)) require only 256 tokens for CelebA-HQ and FFHQ while the general- purposs tokenizer used by our model leads to 1024 tokens. StyleSwin (Zhang et al. 2021) uses several techniques to boost generation quality. More specifically, the baseline StyleSwin model gets an FID of 15.0 on FFHQ (in comparison, our method achieves 12.5). However, with techniques such as style injection, double attention, wavelet discriminator, sinusoidal positional encoding at each generation scale, and balanced consistency regularization, StyleSwin is able to achieve an FID of 2.8 (Table 5 of (Zhngg et al. (02T)) We show unconditionally generated examples from model trained CelebA-HO
believe
in Figure 4a and from our model trained on FFHQ in Figure 2b and Appendix G In Appendix
show the nearest neighbors for our generations from the CelebA-HQ training data. In Figure Ba
we visualize the forward noising process applied on an real image. In Figure 3bb we visualize the reverse denoising process leading to generation.
Table 2: FID) scores computed using 50K 256 x 256 unconditionally generated images on CelebA- HQ and FFHQ. Baseline results are replicated from (Esser et al. 2021 Jiang et al., [021; Vahdat
al. 2021 Hu et al. 2021: Jiang et al. 2021 Gu et al., 2021 denotes that the model uses
dataset-specific tokenizer latent space. We provide more details in Appendix I
CelehA-HQ
FFHQ
Model Family
Model
FID ())
Model
FID (
GLOW (Kingma & Dhariwal 2018
69.0
VDVAE (r = 0.8) (Child 2020
29.8
Flow & Autoencoder
Style ALAE (Pidhorskyi et al.
2020
19.2
VDVAE (r = 1.0) (Child 2020
33.5
DC- Parmar al.
2020
15.8
VIDVAK 0.9) (Child
28.5
TransGAN Diang al.
2021
10.3
BigGAN Brook al. 2018
12.4
GAN
PGGAN Karras al.
2017
8.0
StyleSwin (buseline) (Zhung
2021
15.0
StyleSwin [hhng al.
2021
3.3
StyleSwin (Zhang et 2021
2.8
Continuous
LSGM Vahdat et
2021
7.2
VE Jolicoeur-Martieeu al.
2021
15.7
Diffusion
L4 Rombuch al.
2021
5.1
-LDM-4 (Rombach al
2021
5.0
*VQGAN PixeISNAIL CChnn al.
2017
21.9
Autoregressive
VQGAN GPT2 [Esee al.
2021
10.2
VVQGAN GPT2 (Esser
2021
9.6
VQQDDDM ReFiT) (Hu
2021
VVQ-Diffusion (Gu al.
2021
6.3
Discrete Diffusion
VQ-DDM (w/ ReFiT) (uu et al.
2021
13.2
MUSE tokens Chang
2023
MUSE tokens Chang ct 2023 GGM (ours)
+ GGM (ours)
9.8
12.5
5 DISCUSSION AND LIMITATIONS
We propose a novel conceptual framework for discrete diffusion and evaluate its performance on generative modeling tasks for language and images. We show that our proposed approach outper- forms existing discrete diffusion models for language generation and demonstrates strong perfor- mance for image generation without using dataset-specific image tokenizers. Our approach the- oretically principled and provides promising new direction for multi-modal generative modeling. Future works can explore various applications and extensions like protein modeling (Meshchaninov et al. 2024 Campbell et al. 2024) and generalist agents (Reed et al. 2022). In its current state, we find that discrete diffusion models lag behind state of the art methods like autoregressive LLMs for language, and GANs and continuous diffusion models for image generation. We note that state of the art image generation algorithms are a product of years of cutting edge research, and utilize nu- merous additional customized techniques (such data-specific tokenizers, style injection, etc.) that

---

Published as a conference paper at ICLR 2025
(a) Noising process different timesteps visualized using the MUSE tokenizer. From left to right, the timesteps 0, 256, 512, 768, 1024, 2048, 3072, 4095.
(b) Denoising process at different times visualized using the MUSE tokenizer. From left to right, the timesteps are f = 4095, 3072, 2048, 1024, 768, 512, 256, O.
st
(a) Examples 256 256 unconditional generations from our model on CelebA-HQ.
(b) Comparison of 256 256 conditional genera- tions (middle) from our model on CelebA-HO given masked inputs (top) with the ground-truth images (bottom).
can improve performance dramatically (for example, Table 5 of (Zhang et al. 2021 We believe that with further research and refinement, discrete diffusion models can obtain even better results.
REFERENCES
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg, Structured denoising diffusion models discrete state-spaces, ArXiv, abs/2107 abs/2107.03006, 2021. URL https:
//api. wew.iissmaniicccolla.. wtt.ooaaaa..rr/rooouoIaa:::
Paul Barham, Aakanksha Chowdhery, Jeffrey Dean, Sanjay Ghemawat, Steven Hand, Daniel Hurt,
Michael Isard, Hyeontaek Lim, Ruoming Pang, Sudip Roy, Brennan Saeta, Parker Schuh, Ryan Sepassi, Laurent El Shafey, Chandramohan A. Thekkath, and Yonghui Wu. Pathways: Asyn- chronous distributed dataflow for ml. ArXiv, abs/2203. 12533, 2022. URL ltttp:://aat
semanticscholas semaartiicchholarr...oooooopoopp 10:47611786
Itai Benjamini, Noam Berger, Christopher Hoffman, and Elchanan Mossel. Mixing times of the biased card shuffling and the asymmetric exclusion process. Transactions of the American Math- ematical Society, 357(8) 2005.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autore- gressive Language Modeling with Bessh-Tnsorflow, March 2021. URLhttpss: //o.i. org/ If you use this software, please cite it using these metadata.
1.0 52012281//2 /wwodo.oooooooo 5297715
James Bradbury, Roy Frostig, Peter Hawkins. Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miine and Qiao
10

---

Published as a conference paper at ICLR 2025
Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: // github. www./ooolee/////ooooooooooo
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. ArXiv, abs/1809. 1106,, 2018. URL http://///app..maaiiii....o.
org/CozpusID: :.DD::228555 52869459
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamers. Advances in neural information processing systems, 33:1877-1901, 2020. 2020.
Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co- design. ArXiv, (bs/2402.00997,, 2024. URL https: api. semanticscholar. orr/
CorpusID:26752315
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, José Lezama, Lu Jiang, Ming Yang, Kevin P. Murphy, William Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. ArXiv, abs/2301.00 1004000000744
2023. URL URLhttss://aape sttps///a.p...mmmmmcccccccooo.o. bot. http://tt.....oooo.oooooooooooaa
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and P. Abbeel. Pixelsnail: An improved autoregressive generative model. ArXiv, abs/1712.09763, 2017. URL https::/
semanticschin lar. org/ Corpus : 6199526
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. ArXiv, abs/2011.10650, 2020. URL
Corpus1D:22712733!
Jacob Devlin, Ming-Wet Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In North American Chapter of the Associ- ation for Computational Linguistics, 2019. URL /ttpss:///ppi.cee spcp:/pcc...eeetttcccccoooo. orqq/
Corpus 52967399
Persi Diaconis and Arun Ram. Analysis of systematic scan metropolis algorithms using iwahori- hecke algebra techniques. Michigan Mathematical Journal, 48(1):157-190, 2000.
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, A. Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler. Continuous diffusion for categorical data. ArXiv, abs/2211.15089, 2022. URL htte:::/ap..semaatccccot. orq/
Corpus ID:
Patrick Esser, Robin Rombach, and Björ Ommer. Taming transformers for high-resolution image synthesis, 2021.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.
Alan Gelfand and Adrian FM Smith. Sampling-based approaches to calculating marginal densi- ties. Journal of the American statistical association, 85(410):398 409, 1990.
Stuart Geman and Donald Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Transactions on pattern analysis and machine intelligence, PAMI-6 (6):721-741, 1984.
Reza Gheissari and Alistair Sinclair. Low-temperature ising dynamics with random initializations. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing, pp. 1445- 1458, 2022.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://ttyLion0000.giio. 10/ OpenWebTeextCorpus 2019.

---

Published as a conference paper at ICLR 2025
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10686-10696,, 2021. URL ht.tps: //ap:.::......////
semant ccscoo ar
ID: 24474144455
Ishaan Guirajani and Tatsunori Hashimoto. Likelihood-based diffussion language models. ArXiv,
0bs/2305.18619) 2023.
URL
258967177
Heng Guo and Mark Jerumm. Random cluster dynamics for the ising model is rapidly mixing. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. SIAM, 2017.
Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-Im: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. In Annual Meeting of the Association for Computational Linguistics, 2022. URLAttps::/ /appi.ppp...////p
orc/CorppusD::25527770
Bryan D He, Christopher M De Sa, Joannis Mitliagkas, and Christopher Ré. Scan order in gibbs sampling: Models in which it matters and bounds on how much. Advances in neural information processing systems, 29, 2016.
Zhengfu He, Tianxiang Sun, Kuan Wang, Xuanjing Huang, and Xipeng Qiu. Diffusionbert: Im- proving generative masked language models with diffusion models. In Annual Meeting the As- sociation for Computational Linguistics, 2022. URL https: /t::://ap /api. semantisccholllrr
oxc/CorpussD:22400411
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2023, URL
com comm//goll//lax
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-68511 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 14:12444-12465, 2021.
Minghui Hu, Yujie Wang. Tat-Jen Cham, Jianfei Yang, and P.N. Pug.antnn Global context with discrete diffusion in vector quantised modelling for image generation. 2022 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 11492-11501, 2021. URL
https://appp www.ssemanisss.... www.aa.rrrrrooggeeeeeeeee (D:::/ooruuSS::24400000
Aapo Hyvärinen. Some extensions of score matching. Computational statistics & data analysis, 51 (5):499-25512, 2007.
Aapo Hyvärinen and Peter Dayan. Estimation of hon-normalized statistical models by score match- ing. Journal of Machine Learning Research, 6(4), 2005.
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image gener-
ation. ArXiv, abs/2401.0961o 2023. URL https:://apt www.stscschoolal..oooooo
Corpus ID::1DD3515559
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. In Neural Information Processing Systems, 2021. URL https://aph www.nt
ar. waa..org/CCe wwwooor///oroe CtggCorpussaaaaeeeeeee :D:23422159
Marco Jiralerspong, Joey Bose, Ian Gemp, Chongli Qin, Yoram Bachrach, and Gauthier Gidel. Fea- ture likelihood score: Evaluating the generalization of generative models using samples. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2025
Alexia Jolicoeur- Marttnnna, Ke Li, Remi Piche-Taillefer Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models. ArXiv, abs/2105. 14080, 2021. URL ltttps://ap) wwwant wtt.:/aarr..roooouooosoaaa 23:3225322
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. ArXiv, abs/1710.10196, 2017. URL https://aat
semant: 1 eschol ar. org/ Corpus TD: 3568073
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 4396 4405, 2018. URL https:/ www.ntticccoolar.ccooooo
CorpusID:544B224
Jacob Devlin Ming-Wa Weig Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACI-HET, pp 4186, 2019.
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convo- lutions. ArXiv, abs/1807.03033 2018. URL /tttss///api.sematticcccc. CorpusID:496577
Tuomas Kynkaanniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. ArXiv, abs/2203.000222 2022. URL https:
//api. /ttppp/..ematttissooooo...oorrrp
cotp:aarr..rr/ooooo::::::::: :
David A Levin and Yuval Peres. Markov chains and mixing times, volume 107. American Mathe-
matical Soc., 2017.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion-
Im improves controllable text generation. ArXiv, Abs/2205511221, 2022. URL https://app. semanticsee semanticscholar.co wwtp:orpgoooooooooooeeooo : 249192356
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer-
ence on Learning Representations, 2017. URL
CorpusID:5333 ID:55922770
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by es- timating the ratios of the data distribution. ArXīv, abs/2310.16834, 2023. URL https:
Peapi.semntiioom wwt.:laa..rrr.opoooaoa ID 264411388
Rabeeh Karimi Mahabadi, Jaesung Tae, Hamish Ivison, James Henderson, Iz Beltagy, Matthew E. Peters, and Arman Cohan.
Tess: Text-to-text self-conditioned simplex diffusion. ArXiv, URL https::/app: Sttt::::/ap..semaaniisssses rremantisschin arrooollrrooo wttpaaa...rroooo///oo//ao//
abs/2305.0833 08337, 2023. 258686601
Fabio Martinelli.
Lectures glauber dynamics for discrete spin models. Lectures probability theory and statistics (Saint-Flour, 1997), 1717:93-191, 1999.
Fabio Martinelli and Enzo Olivieri. Approach to equilibrium of glauber dynamics in the one phase region: I. the attractive case. Communications in Mathematical Physics, 161(4):447-46,, 1994.
Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: General- ized score matching for discrete data. Advances in Neural Information Processing Systems, 35: 34532-34545, 2022.
Viacheslav Meshchaninov, Pavel V Strashnov, Andrey Shevtsov, Fedor Nikolaev, Nikita V. Ivanisenko, Olga L. Kardymon, and Dmitry Vetrov. Diffusion on language model embed- dings for protein sequence generation. ArXiv, abs/2403.03726, 2024. URL httph:://p///p/
semant cscho la CD:p:r/g/ooouu:t::::: aotaa/orr///ooruuuu1122222355550
Elchanan Mossel and Allan Sly. Gibbs rapidly samples colorings of g (n, d/n). Probability theory and related fields, 148(1):37-6, 2010.
13

---

Published as a conference paper at ICLR 2025
Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive genera- tive autoencoder. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 823- 2020. URL http:///aa...aammoooooooooo..ooo
227118920)
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. 2022 IEEE/CVF Conference Computer Vision and Putlern Recogni-
tion (CVPR), 11400-11410, CorpusID:24987932)
2022. httpss//app...mmaatccccc......s
Suraj
Cuenca, Nathan Stable diffusion with diffusers.
Patil, Pedro
Lambert, and
Patrick
von
Platen. 2022.
Hugging wwt.l/oog//aabbeiiffisisssss
Face
Blog,
William S. Peebles and Saining Xie. Scalable diffusion models with transformers. 2023 IEEE/CVF
International Conference Computer Vision (ICCV), pp. 4172 4182, 2022. URL https:
//api. www.sseccnnnnnolaaee.e.aamaca htt.:/chh...a.ccccoooooooaaaaaaa
Stanislav Pidhorskyi, Donald A. Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders.
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14092- 14101, 2020. httttt:s://pp
cremmantisccooesse wrtp:oaa....r/oooosss10222203350
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. URL https://api.
semanticscholar.comm :ttp:/aa.a.rr///ooouoooo10000000
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. International conference on machine learning, pp. 8821-8831. Pmlr, 2021.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo. Alexander Novikov, Gabriel Sarth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Ec- cles, Jake Bruce, Ali Razavi, Ashley D. Edwards, Nicolas Manfred Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Trans. Mach. Learn. Res., 2022, 2022.
URL pitpps://api
remanticschon
2248221188
Machel Reid, Vincent J. Hellendoorn, and Graham Neubig. Diffuser: Discrete diffusion via edit-based reconstruction. ArXiv, abs/2210. 16886, 2022. URL https://api.
Robin Rombach, A. Blautmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. 2022 IEEN/VE Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 10674-10685, 2021. URL httth:///app..
semanticscholi org/ :tt.c/hoooa...oo/ooooro.aaaa:::a :t: 245335280
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-36494, 2022.
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusson lan- guage models. ArXiv, 2024. URL htttp:::/ppp..eemniiccchhhe
www.ntmiissss..e. remanticscholaar.
277380319
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using noncquilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265. PMLR, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 134466, 2020.
14

---

Published as a conference paper at ICLR 2025
Robin Strudel, Corentin Tallec, Florent Altch'e, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will
Grathwohl, Nikolay Savinov, Sander Dieleman, L. Sifre, and Rémi Leblond. Self-conditioned embedding diffusion for text generation. ArXiv, 0bs/2211.002366 2022. URL https://api. ssemant cscholar. wwtp:oooorr//oorrrr.r.....
ID: :ttp://Cooouuu1::3373333 253397773
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced trans-
former with rotary position embedding. ArXiv, abs/2104.09864 2021. URL https://api.
semanticscholar: wttp:oorrps1:::33333131.
Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. In The Eleventh International Conference on Learning Representa- tions, 2022.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 11882886, 2015. URL https://api.
semant Gemanticscholar.c htt.aaarcc.c/oooppaa.02555555550
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint 2:X2::2312.11805, 2023.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in neural information processing systems, 34:1 12877122133,, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Alex Wang and Kyunghyun Cho. Bert has mouth, and it must speak: Bert markov random field language model. ArXiv, abs/1902.04094,, 2019. URL https::/ http stt.ppiiss.mmmiicccccceeeeeeeeta
org/CorpusID: 60441316
Jaashnng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. Dinoiser: Diffused conditional sequence learning by manipulating noises. ArXiv, abs/2302.10025,, 2023, URL
https: www.nt
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint 200. 2(3):5, 2022.
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, Boqing Gong, Mina--ssuan Yang, Irfan Essa, David Ross, and Lu Jiang. Language model beats diffusion tokenizer is key visual generation. In The Twelfth International Conference on Learning Representationss 2024.
net hwtp:/eeeet/gggaaaaaaa.aaaa -2grANCCE 4g
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. Seqdiffuseq: Text diffusion with encoder-decoder transformers. ArXiv, abs/2212. 10325, 2022. URI. https://
api. www.setetatnnnco..e.....
Bo Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo. Styleswin: Transformer-basee gan for high-resolution image generation. 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1 11294-11304, 2021. URL
semant
wwwp/ooogggI:aaioo.oooo 1D:23334477
Yongyue Zhang, Michael Brady, and Stephen Smith. Segmentation of brain mr images through hidden markov random field model and the pnnectation-maximizaate algorithm. IEEE transac- fions on medical imaging, 20(1 30555555-7 2001.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint 24X9v22409,02908, 2024.
15

---

Published as a conference paper at ICLR 2025
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffu- sion model for text generation. ArXiv, abs/2302.05737, 2023. URL https::/aaas
semanticscholar.orp wwwpehhooorrrrooooooruuooc 2.t.
Song Chun Zhu, Yingnian Wu, and David Mumford. Filters, random fields and maximum entropy (frame): Towards a unified theory for texture modeling. International Journal of Computer Vision, 27:107-126, 1998.
16

---

Published as a conference paper at ICLR 2025
A PROOF OF LEMMA I
Proof. We consider Proposition 4.7 in Levin & Peres (2017). which establishes coupling charac- terization for the total variation distance. This implies that for any random variable Y ~ [I(+ which is jointly distributed with .o....: Xr, we must have:
((Law((x), <P(X*
(5)
We construct such a joint distribution/ coupling below. Let Xo..... XT obtained using the for- ward process. Consider the event Fr,; == (31 0 S 1 S -1 Z. F & and i, = and the event Ey [ Er, This means that each coordinate of Xu has been noised at-least once in time T. Define (() = st t 0 <t t $ T -1 1, Z, # & and is = i}. We define the random variable as
follows:
Y = (Z+() under the event ET; ~ 11(122) independently otherwise
(6)
easy to show that Y ~ I)) By definition of Y, { + Xr} = EG. Therefore, we conclude from Equation equation that:
(V(L)(())), P(E)) & L(1 -e)(eee) T300 0
B PROOF OF LEMMA 3
Proof. Note the following relationships between events
1. [X++1 = x} n (Z = .) = [Zz = Φ) n (X, =x) x)
2. For + o and E X we have:
-- if =
(X+1 = n [2+ = 2) nZ
1
otherwise
3. For any E XLL
(X++1 = x) (
= nf x;,))
From the relationships above, we have:
E Xx+1 z] P(zz,= 1|X x)
P(Z: = 2, X1-1
P(X =
if # Φ and = 4 Ii.
= = Φ
P(X+41=0) =x!
P(Z, =x, B(X111=8)
if = Fis
P(X+1 PZ (P = x) + P(Z, Ii, PX = x--))
Combining these relationships, we have:
P(Z. Fi, |X++1 =x)
1 F(Z,=0) F(Z, ((7,
P(Xt.i. = Fí, Ne,-is = =x-in)
(8)
PlZ, = xi..)
[(2z Nú, X = xx) -1)
P(X,,, = =x;, xx,1 Xx, = -.,)
P(Z: = Φ)
17

---

Published as a conference paper at ICLR 2025
C PROOF OF THEOREM 1
Proof. Suppose we initialize Xy Pr in Algorithm 2 instead of ()) By Lemma and Lemma 3 we conclude that if the model learns (2, X 2- X a) pertectly, then the output of Algorithm 2 satisfies Xo ~ P That is, it yields a sample from the exact target distribution. However, we do not initialize with Xx ~ Pr, but with Xy ~ (-12)) & and this yields an output Xo ~ Po P
Notice that Algorithm is Markov Process whose output distribution p if the input distribution Pr and the output distribution is Po if the input distribution if II(-)x)*. Therefore we can write down the data processing inequality in this case as:
TV(Pa, PT) TV(P,,
L(1 - -TT//
In the second step, we have applied Lemmal I Setting the RHS above S. conclude the result.
18

---

Published as a conference paper at ICLR 2025
D ZERO-SHOT CONDITIONAL GENERATION WITH GGM
For image data, we can mask/corrupt the inputs either the token space or the original pixel space.
D.I MASKING THE TOKEN SPACE
Given tokenized image Xo ~ Pa, we generate masked images by setting Xo.a:b [T(- and run the conditional inference as specified in Algorithm Here, a denotes the slice of tokens from index to index b. In Figure 5 we show examples with 256 and b 768 (that is, Xo. and X6,768: form our prompt tokens X and the prompt positions J {0, .O....: 255, 768. .... 1023)).
D.2 MASKING IN THE PIXEL SPACE
Instead of masking the token space, can also mask the original pixel space, pre-tokenization. In this case, we simulate the previous setting using the available pixel mask. Given an image Y E [0, 1| and mask M E {0, 1] we generate two versions the masked image Yi with masked pixels set to 0 and Y2 with the masked pixels set to 1. We then tokenize these two versions and compare the tokens to get the mask the token space. That is, if X1, X2 E X2 are obtained after tokenizing Y1, Y2, we set J [j E [4] X1, X2,5} and run Algorithm A2 with this J and prompt tokens {X; X1,1 X2.hee- We show results this approach for a variety of pixel
masks in Figure and
and Figure 6
Algorithm A2: Conditional Inference from a Glauber Generative Model (GGM)
Input: Timesteps T, trained parameters 0, model f, noise distribution IT, prompt positions J, prompt tokens [Xi)ee
Result: Sample Xo from the target distribution II* with Xo.5 X3 for all j e J
Initialize X7 [I)(-)2)+ fix XT.J X3 for all j E J;
for t f- T 1 0 do
if E J then Set X + X
else
Set X(+15 f X441. for all + iz, 8/41, t wi
Get Ú fo(((+111)
Compute P(X,,, = -|X,,1,,-i) = 1((8+111e) (1 -1): 11:(())
Set X.. X for all j # Sample Xe, P(Xe, al a|Xe+1.-u)::
10
11
end
12 end
13 return Xo:
19

---

Published as a conference paper at ICLR 2025
LE
Figure 5: Comparison of 256 x 256 conditional generations (middle) from our model on CelebA-HQ given masked inputs in the token space (top) with the ground-truth images (bottom).
W
Figure 6: Comparison of 256 x 256 conditional generations (middle) from model CelebA-HQ given masked inputs in the original pixel space (top) with the ground-truth images (bottom).
E NEAREST NEIGHBORS FOR CELEBA-HQ GENERATIONS
Since the FID is computed over the training dataset, models that overfit the training data can achieve very good FID scores (Jiralerspong et al. 2024). To qualitatively demonstrate this is not the case with models, retrieve the nearest neighbors (using Inception embeddings) for generated images in Figure] 7
F GIBBS SAMPLING WITH A PRETRAINED MLM MODEL
For a given sequence length and given vocabulary X. initialize Xo,, ~ Unif ii.d.
2. Suppose we are given a pretrained masked language model (e.g, BERT), and an ir E X We denote the Glauber dynamics update probability for i-th token given x=; as P.(((==.)) We estimate this probability distribution as the logits of the pretrained model with input x, masked at the position i (denote the mask token by w).
3. For time = 0,..., T 1, obtain index i, by iterating over all positions in {....., L in round-robin fashion. Sample X X given X, 2
Xx. if X1+1.
1 (((-)ee--
(10)
20

---

Published as a conference paper at ICLR 2025
Figure 7: Nearest neighbors for 256 256 unconditional generations from GGM CelebA-HQ
Row 1, 3: unconditional generations from our model. Row 2: nearest neighbors from the training data for images. Row 4; nearest neighbors from the training data for row 3 images.
21

---

Published as a conference paper at ICLR 2025
G
MORE UNCONDITIONAL GENERATIONS FROM FFHQ
Figure 8: More 256 x 256 unconditional generations from model trained the FFHQ dataset.
22

---

Published as a conference paper at ICLR 2025
H GENERATING X FROM Xo
We assume the distribution over the tokens in the vocabulary (-)2)) does not vary with time (i.e., only (1) is allowed vary with time). In this setting, is possible to directly obtain X, from Xo as follows. For t E {o...., T 1}, we first obtain X₂ directly from Xo and then noise it for an additional step get X1+1 obtain training data. Suppose Xg E X and let i, {0. .......,... denote the position that was chosen to be flipped at time s. When this is deterministic, we know the multiset L, {i.) completely. Let m²()) denote the multiplicity of position j in I, (i.e., the position was updated (i( times in time 1). Let Tx( {s E (0,..... 1} i, = j] be the set of all timesteps when this position was visited. Note that [x(si) (t) Set X1, = Xo.: for all j & I, (these positions have not been visited yet). For j E 4, we have had sequence of (i) independent Bernoulli trials BI..... Bm (f), where at every success we would have flipped. Since these flips independent,
1}
only need to consider if there has been least success in these m²(j) trials. That is, for Y = B1 + + Bm,
(j)> we set Xt.j ~ [I(. with probability P(Y 2 1) = 1 P(Y = 0) = 1 II. (1 - Φ)) = 1 II.... (1 [Ø)) and set Xt, = Xo.; otherwise. For round-robin scheduling the positions used in all our experiments, = mod L-1, and (J) [t/L]+ + 1, This process yields X,. We perform additional
noising step to get X(+1. We briefly describe this in Algorithm 14
Algorithm A3: Forward process: generating X directly from Xo
Input: Input Xg ~ P*, timestep t, positions {is)-= distributions II, for < 0 to L do
Compute 7x (() = {s E (1....,,)) 1) = j}: [W TT,()):
Compute Py
Sample Z₁ ~ II(-1=²);
For all set Xt,j = Z, with probability Pj and Xo,, otherwise; end
Set X(+1,) +- Xe, for all j # in Sample Ze 1lii
L/ = Φ then
Set Xet1,i, Xt,i
11 else
12
Set Xe+1,4, Zz
13 end
14 return X(+1, Xr, Zzi
23

---

Published as a conference paper at ICLR 2025
I HYPERPARAMETERS AND MISCELLANEOUS TRAINING DETAILS
The below settings are common across all experiments. Our model is a 24 layer transformer model based on (Peebles & Xie 2022 Lou et 2023 with 16 attention heads with a hidden size of 1024. Our token embedding table has one additional entry for the special mask token w. We use the AdamW optimizer (Loshchilov weight decay and with no dropout and use EMA with 0.9999 over all training steps during inference. The number of timesteps T and the sequence length L fixed to 4096 and 1024 respectively. II, II for all 1, and (Ø) = 0.5. We do not explore noise schedules and other values for (T))) in this work, though exploring these can potentially lead to improved performance and faster convergence. All our models and related code is written in JAX (Bradbury et al.) 2018) and Flax (Heck et al., [023]. We use FP32 precision. We train our models on TPUvSe accelerators having 16 x 16 topology with data Sata-parallelism enabled via Pathways (Barham et al. 2022) and dataloading via PyGrain and and We provide more experiment specific details below.
Hutter 2017) (with 31 = 0.9, B2 = 0.999, c 10-8) with no
TFDS
1.1 LANGUAGE
We approximate (z) for = + Φ using a unigram language model trained on a large subset of the training data (that is. [I(=) (1 - II(Ø)) "y where n, is the frequency of token and N is the total number of tokens). We use a batch size 64 and sample 32 timesteps per example in every iteration leading to effective batch size of 2048. Our model has been trained on OpenWebTexx (Gokaslan Cohen 2019) for 2M steps. A single step here takes around 0.36s. We tokenize and de-tokenize the data using the T5 tokenizer that has vocabulary of size 32100 and train on packed sequences of length 1024. We keep the initial learning rate to 0 and warm it up linearly for 8000 steps to a peak learning rate of 10-4 and then decay it to 10-6 using cosine decay schedule over 2M steps.
1.2 IMAGE
We set II(2) = (1 - [ as the uniform distribution over the vocabulary and use batch size of 2048. Our model on CelebA-HQ (Karras et al. (017) has been trained for 1.45M steps and our model on FFHQ (Karras et al. 2018) have been trained for 3M steps. A single step here takes around 0.36s. We keep the initial learning rate to 0 and warm it up linearly for 8000 steps to a peak learning rate of 10 and then decay it to 10-6 using cosine decay schedule up to 3M steps for FFHQ and up to 1M steps for CelebA-HQ and then keep it constant at 10 During inference, for both CelebA-HQ and FFHQ, use a temperature of 1.05 for the first half of the denoising steps. We use the 'high-res' VQGAN- based tokenizer proposed in (Chang et al., 2023) without any retraining modifications. We do not finetune this further on CelebA-HQ or FFHQ. This tokenizer has a codebook of size 8192 and a downsampling factor of 8 which implies a 256 256 image is mapped to a 32 x 32 feature map (i.e., 1024 tokens).
use top-p sampling with p 0.9. Additionally, for CelebA-HQ
1.3 BASELINES
O We use the Huggingface Flax implementation for all the BERT and GPT* architectures used in this paper. We use the authors' official implementations for Plaid, SEDD, and MDLM" For in- ference with Plaid, SEDD, and MDLM, we generate using the authors' inference scripts, with a small modification of setting the probability of the end-of-text token to be to get sequences of the specified length of 1024.
httpps://ii thub. wtt./ooogg/////aaai..../oo (Apache 2.0 License)
CelebA-HQ (CCBY-NC 4 License). FFHQ (CC BY-NC-SA 4.0 License)
An open-source replica of the unreleased WebText dataset that was used to train GPT2:
Skylion007//peenweeexxl (CCO 1.0 Universal License)
-google-t//5--33 3b (Apache 2.0 License)
GGPT2-mmlll: avenai-communityyae
GPT2-medium: hop.aiicoommuiii/gg.-mmmieiiiiio medium GPT2-large:
openniicommmuniyyggt largee GPT2-xl:
(all MIT License), BERT-large cased: pppenaicoommunttygppe.. GPT-Neo-2.7B. (euuhese
2.0 License)
"llidd: fiuul22/plaidd (Unknown license), SEDD: htw.:///Ssoooeettoooooo-ieeeeeee
(MIT Li-
cense), MDLM: kuleshov-group/mdmm (Apache 2.0 License)
24

---

Published as a conference paper at ICLR 2025
J EVALUATION METRICS
J.1 GENERATIVE PERPLEXITY
For sequence (Z1,... ...... LL of length L and an autoregressive evaluation model with parame- ters 0. trained on some dataset D. we consider the following related metrics which are also used in (Han et al. 2022 Lou et al. 2023 Dieleman et al. 2022). Note that these metrics are dependent on the kind of tokenization, the evaluation model used, and the data the evaluation model was trained on. Thus, in Table [ we report generative perplexities evaluated by multiple models of different sizes and having different training datasets. The lower these metrics the better. We use the same tokenizer for tokenizing sequences from all the evaluated models.
1
NLLa, (I) =
Σ
log Po. (X1 = xil x21)
(11)
1=2
PPL., (x) = exp (NLLo, (x))
(12)
Across batch of examples (x(11)
(B1),,
NLL8, 1 B
NLLa, (x(9)
(13)
B i=1
B Σ
PPL.. = B
PPLg, (x(1)
(14)
Note that PPLa, A exp(NLLo.). We report the generative perplexities as computed using Equa- tion 14 For the baselines SED (ouu al. 2023 and Plaid (Gulrajani Hashimoto 2023), use the authors public code and released model weights with the default sampling/inferenee hyperpa- rameters provided in their training scripts. For all evaluated models, we set the probability of the end-of-text token to be 0 to get sequences of fixed lengths (= 1024 for our experiments).
J.2
FID
Fréchet Inception Distance (FID) is a metric used to measure the generation quality of images. works by fitting Gaussian distributions to the embeddings generated by Inception and then computes the Fréchet distance between the two Gaussian distributions. However, it has faced recent criticism (Jayasumana et al. 2023 Kynkaanniemi al. 2022) since the network used get the embeddings, Inception V3 (Szegedy et al. (2015). is trained only on ImageNet for the task of classification, making its reliability for encoding other kinds of images (like high-quality faces, etc.) questionable. It has also been about 8 years since Inception V3 was released. For example, using heatmap visualizations, 'Kynkaanniemi et al. (002)) shows how FID focuses more on small microphone present in an image where the intended object of interest is human face. FID also depends lot on the image interpolation filters used while resizing the generated images to 299 x 299 before feeding them to the Inception V model (Parmar et al., 2022). Our FID implementation is in JAX and following Clean-FID (Parmar et al. 2022), we always resize with the bicubic filter having antialiasing enabled. However, since replicate the results for all other basclines from prior work, we not fully sure whether or not they use the Clean-FID settings.
K SCALING
When given more data and computing power, preliminary results show that our model improves. On scaling model to 800M parameters and training more data (the RedPajama dataset), achieve a generative perplexity of 16.7 when evaluated using GPT2-xl.
Based http://////....ooooooaaaaaaaaaaa
25

---

Published as a conference paper at ICLR 2025
L EXAMPLES OF LANGUAGE GENERATIONS
L.1 CONDITIONAL
Below, we show conditionally generated text from our model. We consider the first 256 tokens and the last 256 tokens as the 'prompt' to our model. Our model infills the middle 512 tokens. As can be seen from the below examples, our model remains on topic, and generates text that is relevant to both the prefix and suffix while adhering to the same overall style.
Example Generation
-22.9. 23.0-24.9. 25.0-29.9. 30.0-34.9, junk; 35.0 kg/m2), physical activity (junk; 3.0, 3.0-8.9, 9.0-17.9, 18.0-26.9, 2unk;27.0 metabolic equivalents-hwwkk, smoking status (never, former [1-4 cigarettes per former [5-14 cigarettes per former [ [15-24 cigarettes per day), former 125-34 cigarettes per day former [35-44 cigarettes per day]. former [45 or more cigarettes per dayJ. former [unknown cigarettes per dayJ. current [1 [1-4 cigarettes per dayJ. current [5-14 cigarettes per day), cur- rent [15-26 cigarettes per dayl. current [25-34 cigarettes per dayJ. current [35-44 cigarettes per day], current [45 or more cigarentes per dayl, current [unknown cigarettes per dayl), overall dietary pattern (Alternate Healthy Eating Index score, in quintiles), total energy intake (quintiles)) sugar-swe angar-wweetened beverages consumption (quintiles), and alcohol consumption (0, 0-5, L). There is statistical data to calculate the median daily drinking consumption between 0.5 to 1.6 kg (1-4 kş) (5, 4) as the average c.0.5 v. 2-15 mg).
density, n (g) =3 mg/k///g/kg of the total alcohol consumption (10 [10mg. c.0.5 v. dataset calculate the median deviation is shown in table. To achieve the baseline deviation, as measured using the baseline index, the median deviation between 0.4 and 1.02% (1%) was estimated to be as high as 78.2% (21%). In Figure 3, r == 0.05 I0L + % of alcohol-related consumption, the comparison indicates that total concentration shows no effect the baseline peak temperature, where the average rate was 55.4% to the milk parity ratio versus peak frequency. Moreover, if r/3 4L is approximately 0.5 % of temperature density, the current average is 0.05 [1.02-1 c.1-0.02-2
c.2.0 v. 2-3 mgl, as indicated. In addition to alcohol consumption (e.g, where the current average was 0.5 96 to the relative meat parity ratio of 1-13 grams), the calorie activity was significantly reduced at 4.7 kg (3%) compared the aeromal meat intake at 2-9.5 kg (2%) at 2-2.5 kg/p at 2 p. 3-2 kg (8 min), and to the alcohol intake rate above mg (b) was measured on the baseline frequency (4 min). At 2 mg (0.5 mg-3 3 kg), alcohol consumption for the daily average, or 0.5 mg mg mg-20 mg. for both alcohol consumption and food consumption was reduced whereas total energy consumed by excessive concentration exhibited no factor in increasing frequency, causing sig- nificant decreases the variable volume (Table 6 622). In all studies, the median nonindividual levels were associated with the observed effects of low rates described above. We measured the accuracy of median rates measured in the average population at a time using the buseline index ( (10 h) to determine the size of the population of the daily population, and, as described above, much less than two times the median- median-valued rate was consumed using the quantitative analyses of the regression factors. In an index between the overall population and never smokers, Cox models with inverse probability weighting were applied in the never smokers assessing the association between coffee consumption and risk of mortality and the results did not change substantially (Table VIII in the online-only Data Supplement). Discus- sion In this analysis of 3 large ongoing cohort studies, we observed nonlinear association between coffee consumption and risk of mortality in the overall population. with moderate coffee consumption being associated with lower mortality risk and high coffee consumption not being associated with mor- tality risk. Given that this asxociation became linear and inverse after restricting it to never smokers, it is likely that the nonlinear as association observed in the total population was attributed to the residual confounding by smoking. This was farther strengthened by the observation that the positive associa- tion between coffee consumption and death attributed lung cancer and respiratory diseases the overall population. for both of which smoking is an important risk factor, disappeared when restricting to never smokers. The inverse association between coffee consumption and risk of mortality did not change substantially when using weighted Cox model among never smokers, excluding the possibility that the different associations in overall population and never smokers were because of the different compositions of total mortality. For
the level of lower water lemperature
26

---

Published as a conference paper at ICLR 2025
Example Generation 2
the same number of gold medals, the silver medals are then judged from the most to the least and then the bronze medals. Medal count ranking edit] The gold first ranking system described above is used by most of the world media, as well the IOC. While the gold first ranking system has been used occasionally by some American media outlets, newspapers in the United States primarily publish medal tables ordered by the total number of medals 1ono]19][20122]] [[22][231]]2222 has its origins in the early days of the Olympics. when the IOC did not publish or recognise medal tubles.[1] Before the 2002 Winter Olympics the difference in ranking system received scunt notice, since in recent Olympic history the country that led in total medals also led in the gold count. [citation needed] However, during the 2002 Winter Olympics Germany won the highest number of medals (36), but earned one gold medal fewer than Norway- the latter winning 13.[25] A similar situation occurred at the 2008 and
This difference in rankings
Summer Olympics, with China and U.S. topping the gold and total medal tallies respectively, then again at the 2010 Winter World Olympics. [10] U.S. may not he banned for Olympic Games The chief critic of the 2012 Olympic final ranking, which is set be one of the higgest match-taking causes, is said to have to humiliated for the final loss of the final final. "I don't have a chance to win an Olympic Cup, but I' m totally upset," he told NBC News News. "Iuust in not going to have an opportunity to win. No one doesn't ' like that either." "We have adopted the idea of who has won a Kenyan Olympic Cup. based that, but I think the idea is everybody has never won medal after that tournament, victory after the last Olympics," he said. "People do want to have a better team their standing with the best team. How do they care about competition? "I think it's exciting. think they' 're going to see that by being super close to the best team that's 's the best I would not be crippled for the first time." The U.S. medal ranking chief has told AP that he played key role in the heavyweights chapter of the ICC: "You know, I m winning and now, I'm going be well ax the first team not just win the competition, but with an emphasis on it. And you know, China will have to get away from winning, which is one thing the Chinese are doing at the heart of China." China has ranked one of the three major categories. All three top four were taken by gold medals. [(]]] [ The last two top seven winning medals had top eight winning points that were best ranked in the final six points. The highest ranking rank is champions Jharma Prasa (six of whom won the final tournament, who won in silver). Winner Straels (89 medals) 2015: 14/28 (47 points): 2012: 228: 1:2 13 (28 medals) New Zealand:
(35 caps) 2009: 10:33 11 (54 medals) 2004; 4:14:2 2013: 23:7 1:1 1:4 2015: 9:18:7 15 The 'kindom' number:[3][2] The AAF won't allow top five finishers to be listed in the report in "table of honor". - [10] The 1OC did require top six finishers be listed in the report in "table of honor". 1928: 6:5:4:3:2:1 separate lotals are listed depending on whether the militury patrol was included or not, as its status was downgraded belatedly to demonstration sport. [9] 1932: same as 1924, and described as the usual scheme in newspapers. [10] The 1908 and 1924 systems share the points for tied placings: for example, in a two-way tie for second, each gets half the sum of the points for second and third [aaace//][7]S In 2004, 3:2:1 system was used by the Australian Geography Teachers Association. This weighting values gold medal as much weight as a silver and bronze medal combined. response to the 2008 controversy over medal rank, Jeff Z. Klein in a New York Times blog post proposed a 4:2:1 system as a compromise between the total-medals and golds-first methods.[36
27

---

Published as a conference paper at ICLR 2025
Example Generation 3
easy. There's 's a reason that many of the sources in articles like this are usually anonymous: people Sear both legal and professional repercussions for speaking out. In the course of contacting over 100 different people while researching Star Citizen's 's development, I was told by multiple sources that they were worried about legal repercussions if they spoke to the press. Speaking out publicly about a previous employer carries professional peril, too; prospective future employers may see you as a risky hive. Nonetheless, over the course of the year we found that many of the people who had worked on Star Citizen were willing to talk about their experiences, which painted picture of development process riven by technical challenges, unrealistic expectations and internal strife. The other side the story, of course, is that told by Cloud Imperium Games' current staff: its director, Chris Roberts, its project leads, and the developers who have survived the upsets that drove others away. At the stage where CIG allowed us access to Roberts and other members of the Star Citizen team at its Manchester studio, we already had pretty clear picture of the problems that have dogged the project thus far. In the end, these things have shifted to the core, that you can go through so you really excited about it. There's 's a small group of young people in the small developer community that have been part of the project and have been very comfortable with them. It's amazing, we just talk about this, and the players doing it, but something that has come through the last couple years, and at the start, there is relatively different player in the room help him all the time. But somewhere there, in addition to the fature, a different place. So when you get there and think of where the time going, which is the middle end of the project, you can be sure you have too long to go, for the people, this is really exciting, but if you do know where you can go, it's definitely where the focus is you have to figure out that the competition is still there and most importantly, what they say in the days to come, there's 's great guy in the locker room. "The first thing is get involved, and it's very well programmed here and here. It's just not just the way it's going to work. It's going to be successful, but think that really just makes sense." Another difference, in one sense, is that this gay is there to do what he means to the players. "That isn't i going to benefit the players in the days to come, that's 's all that is built up, he is not there, he does everything," says Stevens. "The thing is that working together and doing things here absolutely different for everybody, but if you want to do what you have on your team at some point, and it, you just want to keep going around yourself because you are engaging in your ongoing mission, and make an investment in your life, and have the opportunity to be integrated into the game, more resources, and more experience, and experience all the time, which is everything that can do great job." The future isn't It' in the right direction. It will take time because it's time have time to evolve so that we have a of expertise and so we get to learn the right ones, that gameplay much better than complicated work yet. So after all that, though. things are gone. The Unreal Engine is one of the oldest and most-used engines in the industry, and in 2011 Unreal Engine 4 was nearing release. Crytek, on the other hand, had released CryEngine 3 in 2009, and the studio didn't 't plan releasing new version for few more years. CryEngine has powered some beautiful first-person shooters from Crytek, with richly detailed, large open environments for players to explore, but hasn't 't often been adapted to other genres. Roberts got hold of both a version of CryEngine and an early build of the Unreal Engine 4. "I was judging both and playing with both of them and ultimately decided on CryEngine because Unreal 4 back then was very early. It had all sorts of power and flexibility, and it's used lot but that point, they were still refactoring even fundamental systems. It still had time to mature and the CryEngine was just a bit more mature." Choosing a ready-made engine instead of building one from scratch meant that he would theoretically be able to build his prototype the game much sooner. Roberts' choice would have consequences
28

---

Published as a conference paper at ICLR 2025
Example Generation 4
have found no employment. In recent years new university graduates have also found their employment prospects limited by jobs offshoring. To understand the real problem, all you have to do is to look at the official information on real median family incomes. There has been no growth for decades. The population is growing, but median incomes, and the lack of jobs has resulted in declining labor force participation rate. If incomes were growing, higher payroll tax could be paid out of income gains. The way to attack the entitlements problem is to bring the jobs home. This could be done by laxing corporations according to the location, domestic or foreign, at which they add value their product. If they produce in the US, a low rate would apply. If they produce abroad, high rate would be applied that negated the savings from producing offshore. But this would not suit the interests of the owners and managers of capital. Their solutions are to raise the retirement age, to further falsify the Consumer Price Index by using the Chained CPI to understate Social Security cost of living udjustments, and to privatize Social Security and Medicare. The first falsification of the CPI was the work of higher income taxes, coupled with the smallness of capital, net net ratio of roughly 30 percent, which is meant to keep living a better part of century. But don't agree with what we do today. I'm not putting a lot of folks out of there, because it's something don't know, I'd like to write off. I may feel ever more receptive or to have degree of competence, and in year I don't It want to know. I'm because what I'm doing to me is actual politics. So think, mean lot, and think there's 's nothing that's my concern. In a day I'm answering question, and am begging say something." Yes, that's 's not true really. In the end, it is absolutely unclear. is a clear way to address the issue. would like to remember that the things we had in mind were what they were, and that we never realised all the things that had been instructive because of us. It seems like we had little higher mentality, which the reason, but, of course, that is the right answer. "I think some other place. for the reason, we have the possibility of doing less, doing less is a place where there is different thing. But we need to change so much of the time we worked, and that's 's a part of how we define society... lot of the conditions have been compared to things we have seen in the past. No debt to pay for job, savings pay for work, is about the same something that can still only be fully understood for our society on the other hand." "But if we 'v ever seen an absolute impeachment of individuality, we have a very real problem, and maybe as much towards them as possible. think our society is receptive. They need to be created by other ones, to satisfy a lot of people. And have to say, there is no difference in underlying reality, that's sss exactly what is our own society, because we do everything can do in real life, and need it right now." snapshot of what Obama is doing in speech today. has to dexcribe this, from all Americans of that time: "Our type of economy is far removed from where I would like to see it. but you have to be careful about moving from one type of society to an- other"https:/i http:://web..raett www.aae.cccrrhheo.. wwtp:oeb/20001000000 (100000600006611 l/http://www.ttfffa www.:wff.org/lss...o. cataaappppppppoo Copp=NC&puuid==a The ideological attack entitlements is again underway. Woll Street is funding it and egging it on. Wall Street looking forward to fees that will eat up all gains and these days of negative interest rates eat up capital as well. Privatization is one of the neoliberal hallmarks of Globalization.
the US prisons have been privatized and prisoners turned into almost free labor for private businesses. Privatized prisons and almost free labor have created a demand for more incarceration, which is the reason that the "free, democratic" " US has the highest incarceration rate the entire world, both in absolute numbers and as a percentage of the population. The claim always that it is cheaper to privatize than for the government to do it, but the evidence contradicts this claim. Consider the military,
29

---

Published as a conference paper at ICLR 2025
L.2 UNCONDITIONAL
Example unconditional generations from GGM with top-p, p 0.9.
Example Generation
are twins of reality, the most productive world we' 'e ever seen today. The emotional vibe of a person comes from experiences. Without having some sort of connection with, you don't I need to make real sense about what a person does every minute. But was deeply offended by the idea that never called him during the moment. knew I found a way to give him one of the most beautiful moments of my life (Help me. I am okay). In my life, I was passionate about his emotions so well he told the difficult truth. What he even conveys to me, is that he speaks thoughtful, mundane stories and he' 'ss really sad in way other than having uncomfortoble question. During my life, feel aware of the person hurt. feel like something I just don't get used and can't 't be watching me for a while. I loved a man saying the right thing always. I wanted to talk about what he called something clever and "not something special." It was a way that music caused me to feel comfortable with me (especially as a performer, where I develop physically and feel exclusively according my life) and sleep in my inner consciousness when was gone. And I think happiness is amazing. Then there human energy; music where things fly at a time. A culture like this, for the most part, is constructural and emotional. That sounds like playing piano and putting yourself into a second spot when you can speak. It's harder and it feels like a listen. I can't 7 play this incredible humor with deep feelings about the joke. He should have vacation at a production house, with friend, every person, someone without affection, and every man hanging out of the room. He all in the mood. It's really exciting moment, the breath will have to wait only a while. There's good place around us doing this in our personal fashion, in the least - sexual sex and sexual sex. We re very hard to get alone in real life at all: our relationship is with the "other." But every day, this whole day, does he feel anxious? We that walk and don't 't fully understand it. We feel the pain, but nobody looks down there. It'll take moment; and think it's time to get do that. I'm wondering if I' doing good or bad anymore, to get the right voice. He' '1l let me realize that feeling so much in his life will help me. But the high 90s guy is passionate about feelings. I see women in male-of-ne ofa--oeeee non-male group and I've seen a guy eating diet in the other, empathy can be matched by things we've up, it's so romantic, it's good (I m just not a keemer). You don't know that. As you experience it, you don't I realize that there is no self- elaborrtattt or debate, no deprecolating questions. You really need to go over here. speak for while and you'l 've gotten much of that shift in the conversation, but since you'll ''e grown a child from his humanistic energy, the value of his humanistic dreams goes. Because he doesn't necessarily thrive and the reality much less discipline, that becomes what he believes is the most important part of our society. So we are forced to recognize that our social environment the only reason our life doesn't 't exist. And what is happiness and personal vision? There's 's no time to pick up and have a fascination with the life around us. It isn't I good way to recognize because we inherited our social life, but in the longer term, have private conversations with others and each other. Our social activity not axsociated with the very complexity of our conversations. The system so. Just as a moral one, needs to help people "give me come there and find myself to like my life," and that life is something that makes it better space for us. People have built their lives to their own imagination. We' 're constantly talking about change, to believe that it isn't 't necessarily wrong, but somewhere over thousand years old. At the highest level, we had such strong psychological relationship between that process and conception. So when a person gets stuck in something new,
a few days. have the emotion I've been trying have
when he gets in. If you re talking about anything, then he's talent, humor and
before. For the first time, sometimes people
setting it
30

---

Published as a conference paper at ICLR 2025
Example Generation 2
your game, everything. You have everything to play, to talk about and to keep going. You have a kind of situation with all of the people. It's pretty easy to get around ahead of every team. They are yet unknown, and are beyond that, depending on why and the way it makes them. It's feeling like this is going to take one day or two. you don't I go without just just picking up and focusing on all the everyday opportunities that you do but also being on the best journey when you ve ready to offer yourself anything if you can even if you are so much excited. still enjoy a lot of lifestyles and learn what to do every time. I still have top (teum) staff und some excellent coaches. I still have good scoring system, but that's 's always frustrating. I am young kid, and learn how to be my best fit. always find it bad that I retire from my next job and am lucky for my job. have a diverse and strong presence of talent that makes me incredibly confortable with either team or team. There's another question of ongoing success. People work there is one great thing to find. So as I feel confortable with it, talk about it next time. feel comfortable with what you think, I want to stay with anyone else, but I am anxious to do my job. I want to get around and sit in for few days if it what I'd like do. would like stay with someone elve for a lot of time. will never ever spend some time looking for ways to screw things down, really. just like myself to get an opinion of myself. If the media deals with me. never think I will survive for several years, or if ever expect so. personally wish to just have tremendous experience and schedule each year, and hopefally l've come that point. Whether people want to play football, if want to address it, a very serious way. Obviously it's pretty hard to do, but it will definitely get my attention. Just wait! I'm happy. only have just a few questions about other questions in the conversation. If you 've heard of that question however, then you' 'e going to be told that it would not have to happen. I am able to get stay there, almost always, many more years. Because anyone who I've seen would pretty much talk about me because people weren't It really anxious. I would have hand myself down every week and be in there for good couple of weeks. would have to disagree with me as much as can, and I would have to think of people to represent me between them. And that would give me much more flexibility, in mind to really see myself playing. I'm all over there (over in time) again after one week with the team. I'm out with my friends in the NHL. That might not happen
lose my place in NBA anyway. Would make my progress break a little bit a few days later? Because have done something for myself this time of the season and so many times early on I' I'm going to play. do have a good year so when don't i talk about the Florico de la La Rio. There are some tremendous opportunities staying in the league, want to keep myself doing right here. I will be coming to my coach. You don't I see any better. wanted myself to be team, but don't i think it has siluation with me of battered than is with my roster of players. They can get to me right now and back June as much as I can today. If things will happen in the season I think you's ve got them back. It's primarily in your UFC or in your NHL schedule. I have incredible trust and respect and have both. I have new shape. Not everyone has the new mindset, in pieces. I meet for while. I won't I win playoff title. Almost three years ago has gained consecutive consecutive wins (28th in the league) have win a game for the playoff'. have to beat him 15 games today, 22th overall in the league). That's what I talked about. I feel comfortable in the game and am worried about progressing by the end of the year. If started going by now I would have picked few, but at the end that year last year I had what appears to be great turnaround. A hockey experience makes you feel like competitor. Unless it happens, although it has be the draft selection from the year of 2014 that referred to games coming overseas, and
31

---

Published as a conference paper at ICLR 2025
Example Generation 3
we're mindful of ourselves in this path, and it is possible to do something different, and how we' 're going into it. Yes, we're not that much different from the ways of me. But the race is also part of that, Where we want to run, we 'r trying to empower people. So we don't 't need the freedom we have, can pull people out, But what we have done, we are all ready to throw things up, we can have so many other things that we can't I connect with and to see all of our experiences in another way, and as we 've gotien throughout our lives in a 'generally' society. judging the world from actually seeing yourself, starting with a single person that is like working for you: not only do you know not only yourself. but that also is wasting your time and wasting time. We need people in our minds understand what elxe we do. We'll 'r trying to be such a friend, And we can come together and talk about others, and it's like bringing together a whole diversity of people in a different way. I just can't work as a person, my father, who works. And have no idea of what she is doing anymore. She and are working. There's 's a thought, and that's 's why I'm just 12 years old. So we don't I want new things; it's like 'fiction' experiences. So have try and do it and think about everything and the story before it then steps in. Wow. I'm going to be comfortable and learn the things that we really really want. And we re trying to make time changes. Of course we need to give an idea of whether we want to be able to do new things. We need to understand what we're 're doing about how we want to live and how to use it. And we' 're trying to convince people. I I' not trying make do it personally. imagine it is in different way, because you can pull things up by telling them, let's go about and work together together. This is just practice. know you can't 't make such a job for you, but you 're not really going to try something, and just find yourself and appear or see something new. It is very interesting to learn that we' 're much harder to go with. The motivation for some nifty things turn around is we engaging in time and in the world of gaming. We know it needs more PCs and PCs, and we want to share and share Well, never liked that at all. I'd like to able to read our story or stories or articles, photos or mail, and think about it. I found myself have been extremely guilty about getting away from my little friends in the past. It's just like I am not capable of constantly noticing something. Thanks for everything,
have the games and apps that
realize can't save it anyway. do not think it's hard to waste a lot on a product if want to keep up with it. am probably intending to let people say that you can't 't help people without holding out odd things online or simply knowing what they can turn into. And unless that is a challenge, do that. Reasons are all that much more difficult though. Most people are often playing with ideas, or at least there are books, and the questions that you ask will determine if things like this could be such wonderful, exciting and exciting place. But in the least when you game, let's start not just acting like that before looking at il, but forget that. The size of this game book looks to the common sense of gaming as well, but that is definitely not something a lot many of us know about the game. The game had been absolutely terrible, and had the kind of inspiration. Most creators and writers have broad thoughts about their books when we start writing it, and the 24-day period of writing can improve pretty quickly, regarding your book, your writing, or any article. It's nice to me to admit that, although about six years ago, we had scores of comic books, other sketchy characters, and hundreds of otherwise simplistic narratives that we have never embraced. We found ourselves hitting our hands once every while. We also realized that we could give tons of talk to our communityy and feel "modicious" " because some versions of this game are just staples of really awesome ideas. We also focus on completing the dialogue, sharing pictures, and wanting to talk freely about the possibilities of any things in the book. For long time, we could write or say something about the game
32

---

Published as a conference paper at ICLR 2025
Example Generation 4
all human beings had no conscious ability to accomplish it. And don't even say that the Marxist Left has right to participate in meaning that people should instead abandon ethical ideas on ideological lines that have proved so hard to bring in. They endorse capitalism and imperialism,
they are even opposed being ridd of. Hence the liberal Left itself. as being political power, had the ability to resist the same anti- government policies that destroyed (and still dispossessed) the right to embrace the "models of Western society". But what we don't 't worry about our part is because modern anarchist theory demonstrated deep tolerance and defended it the way it did. Sociological theory has a role in identifying as its origin; surprisingly. the theory is that it's something that most people now regard as polarization. The quote has a broader view: "The constant veritization of the thought of every human being and of what the rest of society dictates that we lived in the rest of our lives is part of our desire to find out the moral and moral reasons we need to be challenged, to create those we can lead. I was trying reverse the illiteralism outside Grarkburg, where racist violence is being described as false affirmation black people about the lives of the poor and poor, living jobs. That is not manifest in the fact that I believe not only on its part in this destruction, but also the violence that has to do with "the democracy." So when causes of racism are provoked here, you can't i impangulate them of being "their to slavery." As with any of those issues, in doing so, it's best to presume that white people will choose their own race, and if they all hope they'll 'Il be prepared sacrifice their lives wherever they need more. Although no one cannot treat this question as a universally valid question if was chosen or not, who hopes that won't 't ever be civilized? Because we don't 't do or do so, we can, so far. They have the power to defend and support all of these groups. They claim that all life in their society is away from the fact that they don't 't want onything. They claim that many institutions seek ways preserve their democracy. and even though they define themselves as the people of the world, they probubly do not want to become "the only man in society" among others. So, whether we interpret individuals as the real or true "fathers" of modern European society or that of being "munched" is to compare them to being "adopted" by those who serve by themselves, and by themselves, who are, unresolved or influenced by have
capitalism as whole rather than society's 's analogy or fear. But, this means everything in Europe since the 1970s give us the rosy and denial of our worldwide struggle to preserve democratic interests, and erode the extent through which we maintain a large scale of control, But for the moment, we don't 't forget real life; it is why the problem is truly true in your country. Our citizens are willing to protect our values, and blame against your government because they didn't I really protect us. But the advocates of the power capitalism wanting empower the people living in your country not because they have the capacity to abundon by simultaneously refunting the non-existing ideology axsociated with common populalism and populalism. These causes, like that of a source of power in your country, make anonymities even worse. The rich are the big political figures, and they are not focused on real facts they have answer. They do not struggle with this. immagine how gay woman living Europe disagrees about her belief that she wants to vote for justice. She always defends the people have to support the democratic agenda. It is because most politicians realize that despite being recalculiously angered by this tragedy, the movement is such that the terms of injustice must be relative to the freedom humanity needed. am glad to spend little more time in my home in Spain, to protect my husband from France and thus my husband from Italy. I asked my daughter (a mother of six) stay home during the holiday season because of the expectation of losing our family "who lived" elsewhere by claiming that we must have extended lives in favor of a friend, and greater courage in pursuit of human freedom. This is such grand accident. No, I do not have condenmation of any revolutionary movement, but if were attacking the British and the Dutch people, would want the man who was here in Europe to tell me that he had fought against himself to get justice, and I could defend his own people. These are German values, and
33


Reviews:
[{"review_id": "wSHybSXYWF", "reviewer": "Reviewer_8Yq4", "summary": "The paper proposes a new type of discrete diffusion model. The denoising process is defined as flip one token at a time, and the model predicts whether the token at the given location is noise. The proposed GGM is tested on both text and image, and shows strong generative perplexities compared with discrete diffusion model baselines.", "questions": "Please see Weakness clarification questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is innovative and yields promising results.\n- The paper is clearly written, with a thorough discussion of related work.\n- The experiments are comprehensive and well-documented.", "weaknesses": "Clarification questions:\n- In Algorithm 2, line 5, the step is executed \"for all a.\" Does this require $|\\mathcal{X}|$ forward passes per model? If so, could this impact efficiency? Is there a way to parallelize this?\n- Why can the model take a mask token as input? According to Eq. (1), $X_{t+1}$ inherits $X_t$ when $Z_t = \\phi$. When exactly does $Z_t$ take the value $\\phi$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "haphCeiv1t", "reviewer": "Reviewer_tQ8L", "summary": "This paper introduces a new approach to discrete diffusion models called the Glauber Generative Model. The key innovation is using the Glauber dynamics from statistical physics as a discrete Markov chain to denoise sequences of tokens. The authors reduce the denoising process to a series of binary classification tasks, making it simpler than previous approaches. They evaluate GGM on language and image generation tasks, showing it outperforms existing discrete diffusion models for language generation. It also demonstrates competitive performance on image generation without using dataset-specific tokenizers.", "questions": "1. Could the author provide theoretical analysis or bounds on the convergence rate of GGM compared to standard Glauber dynamics?\n2. Could the author give ablation studies about the number of denoising steps, different round-robin scheduling strategies for token positions and vocabulary size?\n3. How do the dataset-specific tokenizers and general-purpose tokenizers affect the result of image generation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Good Physical Intuition: The use of Glauber dynamics in the paper is well-motivated and mathematically principled. Glauber dynamics serves as a discrete analog to Langevin dynamics, which underlies many continuous diffusion models, making it a natural and theoretically elegant choice for discrete diffusion.\n \n2. Theoretical Innovation: The paper presents a novel theoretical framework that elegantly reduces the complex problem of discrete diffusion to simple binary classification tasks. This provides a more efficient approach compared to previous methods that required learning full transition matrices.\n\n3. Plenty of Empirical Results: GGM outperforms existing discrete diffusion models on language generation tasks while using fewer parameters. For example, it achieves better generative perplexity than previous methods with fewer parameters and comparable performance. Besides, it also achieves strong performance for image generation without using dataset-specific image tokenizers.", "weaknesses": "1. Performance Gap: Despite improvements over other discrete diffusion models, the performance of GGM is still worse than state-of-the-art autoregressive models for language generation and continuous diffusion approaches.\n\n2. Computational Requirements: The model requires a relatively large number of denoising steps compared to other approaches. While each step may be simpler, the total computational cost during generation could be large.\n\n3. Limited Ablation Studies: The paper doesn't explore how different architectural choices and hyperparameters affect performance. For example, there's limited analysis of how the number of denoising steps or the choice of noise distribution impacts results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y6dPuYqIhH", "reviewer": "Reviewer_zzst", "summary": "The paper introduces the Glauber Generative Model (GGM), a new approach to generating discrete data like text and images using a process inspired by physics called Glauber dynamics. GGM works by changing one element at a time and simplifies the learning process to basic yes/no decisions about whether each element is meaningful or random noise. When tested on language generation, GGM performs better than similar existing models while using fewer computational resources, though it does not yet match the quality of top language models. For image generation, GGM produces good results without needing specialized preprocessing. While there is still room for improvement, the authors argue that GGM's straightforward approach makes it a promising direction for future research in generation systems.", "questions": "How would the model perform if given more data and computing power? \n\nHow does the model perform on NLP tasks? \n\nHow stable is the training process? \n\nHow well does the model handle long-range dependencies in text? \n\nWhy don't you use the more common name \"Gibbs sampler\" or \"time inhomogeneous Gibbs sampler\"?\n\nIs the classification-based learning method related to noise contrastive estimation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work introduces a theoretically elegant solution by reducing a complex generative task to simple binary classification problems. This makes the approach more straightforward to implement and understand compared to previous methods.\n\nThe model achieves strong empirical results, outperforming existing discrete diffusion models in language generation while using fewer parameters. It also shows competitive performance in image generation without requiring specialized tokenizers.\n\nThe method is versatile, working effectively across different domains (text and images).\n\nThe approach is computationally efficient, with lower complexity than previous methods, making it more practical to implement and scale.", "weaknesses": "Despite its improvements over other discrete diffusion models, GGM still performs worse than state-of-the-art methods like GPT models for text generation and continuous diffusion models for image generation.\n\nThe experimentation focuses mainly on comparing with other discrete diffusion models rather than providing comprehensive comparisons against the broader range of current generation methods. \n\nThe paper does not provide detailed analysis of computational costs and training times compared to other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XnzDId0z1s", "reviewer": "Reviewer_cyno", "summary": "This paper proposes the Glauber Generative Model (GGM), a new discrete diffusion model framework. Unlike existing methods that often rely on regression or variational approximations, GGM frames the core denoising step as a binary classification problem: for each token in a noisy sequence, the model classifies it as either \"signal\" (belonging to the original data) or \"noise\" (introduced during the diffusion process). This simplification allows for efficient training using standard binary classification losses and avoids the complexities of estimating transition probabilities in high-dimensional discrete spaces. GGM utilizes the Glauber dynamics, a Markov chain that flips one token at a time based on conditional probabilities, to iteratively refine noisy input towards a clean sample. The authors apply their model to both text and image generation, demonstrating competitive performance on language modeling benchmarks and promising results on image synthesis using a generic VQGAN tokenizer. Notably, GGM outperforms other discrete diffusion models on language modeling while using fewer parameters. In image generation, it achieves decent FID scores compared to GANs and continuous diffusion models, though a gap to state-of-the-art remains. The authors also showcase the model's ability to perform zero-shot text and image infilling, highlighting the framework's potential for controlled generation.", "questions": "Will the authors open-source the code for training and evaluating the proposed models?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The reframing of discrete denoising as a binary classification task is a novel contribution. This  simplifies the learning problem compared to prior discrete diffusion models. The use of Glauber dynamics, while not new in itself, is creatively applied here within a diffusion framework and offers a theoretically grounded approach to discrete data generation.\n- The paper demonstrates strong empirical results on language modeling, exceeding the performance of other discrete diffusion models and approaching that of much larger continuous models. The image generation results, while not state-of-the-art, are compelling given the use of an off-the-shelf VQGAN tokenizer and suggest further potential with dataset-specific tokenizers.", "weaknesses": "- The model training only differs from absorbing discrete diffusion(MD4, MDLM) by using NCE-style loss vs cross-entropy based loss, and since the regular absorbing diffusion models assume factorized denoising model distribution, this seems to me that the training objective of proposed model do not differ from previous works by much. So can you use a regular absorbing diffusion model for the GGM style sampling?\n- Similarly, the defined forward process can also be seen as a special case of D3PM, and it might have simpler/different forms if considering the continuous time limit, which is absent in current work.\n- The use of Glauber dynamics, which updates one token at a time, necessitates long sampling sequences (T=4096). Exploring alternative sampling strategies or modifications to the dynamics that enable faster convergence could significantly improve efficiency. Analyzing the impact of varying sequence lengths *T* on both sample quality and computational cost would be valuable.\n- Although GGM achieves respectable FID scores, a significant gap remains compared to state-of-the-art GANs and continuous diffusion models on image generation. The authors acknowledge the limitations of the generic VQGAN tokenizer and suggest exploring dataset-specific tokenizers. However, this reliance on VQGANs, even with improvements, might inherently restrict the model's ability to capture fine-grained details and diversity compared to methods operating directly in pixel space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)