Discrete Predictor-Corrector Diffusion Models for Image Synthesis
Authors: Jose Lezama
Tim Salimans
Lu Jiang
Huiwen Chang
Jonathan Ho
Irfan Essa
Venue: ICLR 2023 poster (2023)
Keywords: discrete diffusion models
image synthesis

Abstract: We introduce Discrete Predictor-Corrector diffusion models (DPC), extending predictor-corrector samplers in Gaussian diffusion models to the discrete case. Predictor-corrector samplers are a class of samplers for diffusion models, which improve on ancestral samplers by correcting the sampling distribution of intermediate diffusion states using MCMC methods. In DPC, the Langevin corrector, which does not have a direct counterpart in discrete space, is replaced with a discrete MCMC transition defined by a learned corrector kernel. The corrector kernel is trained to make the correction steps achieve asymptotic convergence, in distribution, to the correct marginal of the intermediate diffusion states. Equipped with DPC, we revisit recent transformer-based  non-autoregressive generative models through the lens of discrete diffusion, and find that DPC can alleviate the compounding decoding error due to the parallel sampling of visual tokens. Our experiments show that DPC improves upon existing discrete latent space models for class-conditional image generation on ImageNet, and outperforms continuous diffusion models and GANs, according to standard metrics and user preference studies.

Published as a conference paper at ICLR 2023
DISCRETE PREDICTOR-CORRECTON DIFFUSION MODELS FOR IMAGE S SYNTHESIS
José Lezama's ; Tim Salimans', Lu Jiang' Huiwen Chang' Jonathan Ho', Irfan Essa
1 Google Research "Georgia Institute of Technology
ABSTRACT
We introduce Discrete Predictor-Corrector diffusion models (DPC), extending predictor-corrector samplers in Gaussian diffusion models to the discrete case. Predictor-correce samplers are a class of samplers for diffusion models, which improve on ancestral samplers by correcting the sampling distribution of intermedi- ate diffusion states using MCMC methods. In DPC, the Langevin corrector, which does not have a direct counterpart discrete space, is replaced with a discrete MCMC transition defined by learned corrector kernel. The corrector kernel is trained to make the correction steps achieve asymptotic convergence, in distribution, to the correct marginal of the intermediate diffusion states. Equipped with DPC, we revisit recent wransformer-based non-autoregressive generative models through the lens of discrete diffusion, and find that DPC alleviate the compounding decoding error due to the parallel sampling of visual tokens. Our experiments show that DPC improves upon existing discrete latent space models for class-conditional image generation on ImageNet, and outperforms continuous diffusion models and GANs, according standard metries and user preference studies.
INTRODUCTION
Generative Adversarial Networks (GANs) are the leading model class for a wide variety of content creation tasks (Goodfellow et al., 2014; Brock et al., 2018; Karras et al., 2020). Recently, however, likelihood-basse models, such as diffusion models (Dhariwal & Nichol, 2021; Ho al., 2020; 2022) and generative transformers (Ramesh et al., 2021; Esser et al., 2021b; Chang et al., 2022), have started rivaling GANs offering an alternative training paradigm with superior training stability and improved generation diversity. In particular, ADM (Dhariwal & Nichol, 2021) and CDM (Ho et al., 2022) presented diffusion models attaining better perceptual quality on the class-conditional ImageNet benchmark compared to BigGAN (Brock et al., 2018).
Sampling speed, however, is still a bottleneck hindering the practical application of diffusion models. These models can be orders of magnitude slower than GANs, due the need to take up hundreds of steps to synthesize a single image during inference. Recently, discrete diffusion has been receiving attention as a promising direction for achieving an improved trade-off between generation quality and efficiency. Like the continuous (Gaussian) diffusion process (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho al., 2020: Gu et al., 2022), these models incrementally corrupt training data until a known base distribution is reached, and this corruption process is reversed when sampling from the learned model. Unlike continuous diffusion models, the corruption applied in a latent, possibly low-dimensional, discrete space. The image generation quality of discrete diffusion models is still inferior to that of continous diffusion models. For example, the state-of-the-ar discrete diffusion model (i..., VO-Diffuusion (Gu et al., 2022)) still notably underperforms CDM (Ho et al., 2022) and BigGAN (Brock et al., 2018) on ImageNet without the guidance from external classifiers.
Contemporarily, non-uuloregressivv transformers (Chang et al., 2022; Gu et al., 2022; Zhang et al., 2021; Lezama et al., 2022) have demonstrated promising performances in both perceptual image quality and efficiency on the ImageNet benchmark. In particular, non-autoregressive transformer model named MaskGIT (Chang et al., 2022) achieves comparable generation quality to the leading diffusion model ADM ImageNet, while enjoying two onders-of-magnitus faster inference speed.
"Coreeponding author jose1 1eaale @ com

---

Published as a conference paper at ICLR 2023
It brings down the generation time of ADM from ~500 steps to only 18 steps, making it possible to generate an image within 0.1 second on a TPU device. Non-autoregrresivee transformers are trained to predict masked visual tokens, inspired by masked language models such as BERT (Devlin et al., 2019). The decoding process, similar to non-autoregressive sampling techniques from machine translation (Ghazvininejad al., 2019; Kong et al., 2020), predicts all missing tokens in parallel, starting from fully masked sequence, and subsequently following an iterative refinement schedule. By viewing non-autoregressive transformers through the lens of discrete diffusion, our analysis yields new insights into a critical issue with these models, namely the compounding decoding error, which causes a mismatch between the inference and training distributions of the intermediate latents produced during the parallel sampling process.
To tackle the compounding decoding error, we propose Discrete models that introduce iterative refinement of the intermediate diffusion states. DPC learns corrector kernel that, coupled with the reverse diffusion predictor, forms a discrete Markov Chain Monte Carlo (MCMC) predietor-corrector algorithm that has the correct marginal distribution of the intermediate latents as its limiting distribution. The proposed DPC model is a new type of discrete diffusion model that is distinct from both -on-autorerressive transformers and conventional discrete diffusion models. Compared to non-autoregressive transformers (Chang et al., 2022; Lezama et al., 2022; Gu et al., 2022), DPC introduces new techniques perform multi-step correction of intermediate states in the sampling process. Furthermore, it provides a theoretical underpinning for this model class, and shows that, in the ideal case, it can completely correct the train/test mismatch between the distributions of intermediate states. DPC advances conventional discrete diffusion by introducing a new discrete MCMC correction kernel, which can be considered discrete analogue to the Langevin corrector that was of (Song et al., 2021) for the continuous case.
(DPC) diffusion
We empirically validate that DPC is able to achieve a good quality- ceality-vs-effccinney trade-off on two tasks: class-conde coasiiooniii image generation on the ImageNet dataset and unconditional generation on the Places2 (Zhou et al., 2017) dataset. The results show that DPC performs favorably against both non-autoregressive transformers (Chang et al., 2022; Lezama et al., 2022; Esser et al., 2021b) and discrete diffusion baselines (Gu et al., 2022) while maintaining a fast inference speed. In particular, without the help of external classifiers, DPC outperforms the state-of-
continuous diffusion models (i.e., ADM (Dhariwal & Nichol, 2021) and CDM (Ho et al., 2022)) in FID (Heusel et al., 2017), thereby establishing new state-of-the-are task on ImageNet. When leveraging an external pre-trained classifier and upsampling, DPC produces state-of-the-art class-conditional generation, yielding better Inception Score (IS) and FID compared to state-of-the-art GANS, (e.g. StyleGAN-XI. (Sauer et al., 2022)) and continuous diffusion models (e.g. ADM (Dhariwal & Nichol, 2021)). Furthermore, we present user preference studies that confirm the perceptual quality provided by DPC.
the high-resolution (512x512) image synthesis
2 BACKGROUND AND PROBLEM STATEMENT
2.1 DISCRETE DIFFUSION MODELS
Let æo E {L...., KJW be a vector of discrete data, such as an image or a set of image tokens obtained by Vector- Quanti-eua (VQ) encoding with dictionary of K elements. Using discrete diffusion process g( (æt), we can sample a sequence of latent variables I1, Z...... Iy such that the final latent Ør has a simple known and fixed distribution p(æT). Starting from a sample from p(æT). we can then reverse this process using learned reverse diffusion model po (z.-1 (xx), eventually producing a sample Zo-
One instance of such a discrete diffusion process q. is the absorbing state diffusion process from (Austin et al., 2021), in which we set x; = xo @ m, where my is vector of binary masks that starts out as all ones, me 1, and ends up all zeros, my 0. In between we gradually evolve m, by randomly setting more and more of its elements to zero according to m, ~ g(m, m4-1).
The reverse diffusion model pe p0 my) can be constructed by first sampling new mask i//m) and then sampling a new value for Φ;-1 corresponding to those values of m,- that newly unmasked, iee sampling wt_1 ~ po (xx.) for those i for which m = 1, and mi 0.
2

---

Published as a conference paper at ICLR 2023
2.2
AUTOREGRESSIVE MODELS
When sampling from a discrete diffusion model with a sufficiently large number of steps I, only a single element of æ is masked/unmasked in each transition, at each step of the reverse diffusion process. In this case, Pg is an order-agnostic autoregressive model, as introduced by Uria al. (2014) and discussed in more detail by Hoogeboom al. (2022). As shown in Uria et al. (2014), these models can be trained efficiently by maximizing the likelihood
only need sample single xi-1
Σ É mn
log po = = Dt, m,),
where the sum is over all possible orderings of the data that are consistent with my; this has lower variance than just maximizing the likelihood for the single ordering that was actually sampled by ((), but is otherwise equivalent. Alternatively, can be interpreted as denoising the corrupted data Tt towards the original clean data æp, trained by maximizing the log- oiglliiel log Py (20o/z m&) for the whole vector vo at once, under a one-step model pa defined by
(20 (22,, m.) = II pa (= (#gg (2,, m,),,
(2)
where we use the Po notation distinguish this factorized one-step denoising model from the full multi-step generative model (=())) = Eæ II (= z2). In the following we use po for all distributions affected by using the one-step model in the generative process.
2.3 NON-AUTOOEGGRESSIV TRANSFORMERS FOR IMAGE SYNTHESIS
Transformer-base based models generate images in two stages (Esser et al., 2021b; Ramesh et al., 2021). First, the image is quantized into a grid of discrete tokens by a VQ-autoencoder (van den Oord et al., 2017). In the second stage, an autoregressive transformer decoder (Vaswani et al., 2017; Chen et al., 2020a) is learned on the flattened token sequence to generate image tokens sequentially using autoregressive decoding. In the end, the generated codes are mapped to pixel space using the VQ-decoder learned in the first stage (Esser et al., 2021b).
Recently, non-autoregressive transformers (Chang et al., 2022; Gu et al., 2022: Zhang et al., 2021) are proposed to improve the second stage, adapted from non-autoregressive machine trans- lation (Ghazvininejad et al., 2019; Kong et al., 2020). An example of such model is the Masked Generative Image Transformer (MaskGIT) (Chang et al., 2022), which follows the masked modeling in BERT (Devlin et al., 2019) (equation 2). During decoding, MaskGIT starts with all the image tokens masked out. In each inference step, it uses parallel decoding, i.e. predicting all tokens simul- tancously while only keeping the ones with the highest prediction scores. The remaining tokens masked out and will be re-predicted in the next iteration. The mask ratio is made decreasing, according to a cosine function, until all tokens are generated within a few iterations of refinement.
2.4 COMPOUNDING DECODING ERROR IN NON-AUTOREGRESSIV TRANSFORMERS
To sample from autoregressive model, one would typically sample one element at time, which is costly when the total number of elements N in x is large. By ignoring the dependencies between the zi_ one can accelerate this process by sampling several x;_1 in parallel each transition. This idea has been explored in discrete diffusion models by (Gu et al., 2022: Hoogeboom et al., 2022), and exploited by non-autoregressive generative transformers (Chang et al., 2022). As suggested in prior work (Austin et al., 2021; Gu et al., 2022: Savinov et al., 2021; Lezama et al., 2022), there exists a close relation between -toreeroreeeeeivvve transformers and discrete diffusion models. For example, training of state-of-the-an MaskGIT (Chang et al., 2022), can be modeled by the discrete diffusion process with the absorbing state (MMASK] (Austin al., 2021). MaskGIT's parallel sampling may be understood as a reverse diffusion process using the one-step model in (2).
the masked-modelineg
However, parallel sampling introduces errors as the factorized Ñe does not exactly match the original generative model pe for the joint distribution of the elements of æ. As a result, the marginal inference distribution po (zx) will deviate from the training distribution q(x,) as sampling progresses. We refer to this compounding decoding error, small differences between q(x,-1/x,) and po( (z,) can accumulate into large differences between q(x;) and (a,) after many sampling steps (ef. teacher forcing error).
3

---

Published as a conference paper at ICLR 2023
2.5
CORORCTORR SAMPLERS
Predictor-Corrector samplers were proposed by Song et al. (2021) for use in Gaussian diffusion models: before applying each time transition in the generative model pa (3 [())) these samplers correct the distribution of ze by applying one or more steps of Langevin MCMC at constant timestep t, thereby bringing the distribution of xe closer to g(x)). The corrector has the form (O, T),
Ty. f- 3e + 6180(17, (F, 1) + v 22,
where 80(2 1) V log ge (a(e)) the learned score function and e, is step size.
DISCRETE PREDICTOR-CORRERR
SAMPLERS
In this section, we address the critical issue of compounding decoding error due to parallel sampling in von-autoregressive transformers. We propose to mitigate the gap between g(xr,) and (x;;) by applying learned discrete corrector MCMC step to iteratively improve intermediate diffusion states. This allows our models to exploit the computational advantages of parallel sampling in non-autoregressive transformers, while maintaining high synthesis quality.
To reduce the compounding decoding error in po(#:), we adjust the sampled æ; to more closely resemble samples from g(x;), before applying the time-transition from e; to 24-1- In the discrete diffusion case, however, there is no direct counterpart to the score function that gives the rate of change required to improve the likelihood of sample (Campbell et al., 2022). Instead, propose to utilize an MCMC corrector step of the following form:
to po ( (2, m²)
(3)
x ~ Pa @...).
(4)
where P& is a learned corrector distribution!, trained so that when applying this procedure multiple times, the distribution of ar, converges to g(3:),, making this valid corrector kernel (Section 3.1).
Since 2; = Bg @ my the corrector distribution can equivalently be stated in terms of the mask m,, which resampled at every corrector step, m, ~ @0, binary variables, of which a known number are equal to one at given time step I, and the others are zero. Thus, the corrector distribution Po (m( @ t) categorical distribution over all possible binary masks with k non-zero elements. We model this distribution using the Plackett- model (Plackett, 1975), which gives
l). The mask collection of
explI (#o. t)]
Ps (m Lo.
II- Cu c,
Ee expl( (æo, ()] + Eimm expll' (1, (do, t)]
where (a t) is the W-dimensional output of neural network corrector model, with elements (#0, representing the relative log-odds (logits) of m being positive for the given an. and where over all possible permutations C₁ of the indices of the k non-zero elements of the mask m,- In numpy notation: Cą np permute nonzerrom.)). We provide more background on this model in Appendix A.
In practice, we can sample from the mask model by forming perturbed logits by adding i.i.d. Gumbel noise, 7' = 1's (ão, t) + G", where G ~ Gumbel(0, 1), and setting the mask to 1 for the top-k elements of i, with is determined by t. This is known as the Gumbel-top-k trick (Kool et al., 2019).
3.1 DISCRETE PREDICTOR- COORRECTOR TRAINING
The learned corrector function P4 is trained to achieve detailed balance between the MCMC corrector steps (3) and (4), and the training marginal q(xx). Let J Jax ax.)) be the transition kernel of the Markoy chain given by (3) and (4). Then, this Markov chain has ((x;) stationary distribution if
(=( = No (2 a )
Z (+b)
We use the subscript 0 for models of the reverse process and é for the corrector model, which goes in the forward direction. The random variables described by the models are determined by what is inside the brackets.
4

---

Published as a conference paper at ICLR 2023
with ((bo) Ezz, p0( (1))) Assuming (6), the detailed balance condition of (a with
respect to q(x+)) follows from:
g(x;) (e)))(aa(())))) - g( Σ
Poo (1rl)) (o (ate)
Pol (co |e/ (g(())))
= =g(2))
Σ
Po (oo (2:))
7 (20)
&o
po ( |x ( ( 20t) = g(2%)) an)),
= g (ar))
(9)
((bo)
where the last step is derived from applying the same operations in the other direction.
The goal of training the corrector kernel is then to obtain (o( x (doo æ (a;)) (equation 6), which can be done by minimizing the KL divergence
K (āo (x) ( (((e:))lpp(()))
Z ( -E ) ( las (q(=)) log po + C, (10)
where C' are terms that do not include øb Importantly, the denoiser po is pre-trained and its parameters are frozen during the training of the corrector pa- Thus, the corrector distribution can be trained by first obtaining masked sample from the training distribution q(æx), and then using the denoising model ño (áo ax)) (with frozen weights 0) to obtain a corresponding æo. after which Pa (xe (ăn) can be trained by maximum likelihood. Since a, completely determined by &g and the mask m, this is equivalent to training Pe to maximize the likelihood of the sampled mask m..
Under our chosen model in (5), evaluating the likelihood for all mask elements jointly would require summing over all possible masks, which is computationally expensive. Another simple training each mask element using Pø (m | 1) (13 (dt: where is the sigmoid function. In our experiments find this simplified loss to be very effective. We discuss this choice in greater depth in Appendix A.
objective binary cross entropy applied independently
Following MaskGIT, we use the cosine schedule for determining the masking ratio during both training and inference. Let I denote the time horizon. For S sampling iterations, we set the masking ratio to (t) = 1 cos(t/S π n/2), for {1...., T/S). During training of both the generator Pe and corrector Pe. we first sample time t, i.e. t ~ L0, before inputting it to compute (t). Likewise, during inference, we compute 7(t) using the current decoding step. In practice, this scheduling gives more weight to the transitions with lower masking rates, in similar fashion the cosine weighing scheme of other diffusion models (e.g., (Hoogeboom et al., 2021)))
3.2 EFFICIENT IMPLEMENTATION USING SHORTCUT TIME TRANSITIONS
Our predictor-corrector sampler consists of taking one predictor step:
Øt-1 ~ (o xz)) (step 1),
followed by one multiple corrector steps:
t. (0 (step 2) followed by k:-1 po(01-1 (step 3).
Under the exact diffusion distribution g, we have that
Σ g( ( = ((<x((( (2)))
Since our model (x (x:) is trained to approximate (1))) as well as possible, we then also have
Σ po
(1 ( lxx),
to a close approximation. Hence, step 1 and step of our predictor sampler can be combined into a single step 2. (0) Using this result. the shorteut time transition becomes
3o ( (x.) followed by #t-1 ~ Pa (0x ((a)-
This avoids sampling step, and offers more efficient way of implementing DPC. In the special case of a single corrector step, this implementation of DPC also closely resembles the non-autoregressive transformer Token-Critic (Lezama et al., 2022).
5

---

Published as a conference paper at ICLR 2023
BLY
Figure 1: Random samples from ImageNet 512x512 512 class-conditional generation for selected classes: 'tusker', 'koala', 'model T and 'steam locomotive'. Left: StyleGAN-XI. (NFE = 1). Center: ADM
Classifier Guidance Upsampling (NFE 250 x 3). Right: DPC-light(5) (NFE 66).
EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Datasets, Baselines, and Metrics We evaluate the proposed DPC model for the tasks of class- conditional and unconditional image generation. For class-conditional image generation, we use standard ImageNet dataset resolutions 256 x x56 and 512x512. For unconditional generation, use the Places2 dataset (Zhou et al., 2017) of 1.8M images at resolution 512x512.
We compare to state-of-te oftt--ofthhe-rt models class-conditional image generation, which include (1) non-autoregressive generative transformers: MaskGIT (Chang et al., 2022) and Token-Crite (Lezama al., 2022): (2) continuous diffusion models: ADM (Dhariwal & Nichol, 2021) and CDM (Ho et al., 2022); (3) discrete diffusion model: VQ-Diffusion (Gu et al., 2022): (4) GAN models: BigGAN (Brock et al., 2018) and StyleGAN-XL (Sauer et al., 2022). We FID (Heusel et al., 2017). Inception Score (IS) (Salimans et al., 2016), and Precision vs. Recall (Kynkäänniemi et al., 2019) to evaluate perceptual quality. We also report the number of neural function evaluations (NFE) required by each method. ImageNet generation is evaluated against the training set, and Places2 against the validation set, following Chang al. (2022). Numbers for the baseline models are quoted from their respective papers except for MaskGIT for which we used a provided model. In addition, we conduct user preference studies further evaluate the perceptual quality and diversity of the models' samples.
We evaluate two versions of DPC, varying the number of correction steps c(t) each intermediate state t, where we take c(t) to include the corrector step in the shortcut time transition (section 3.2):
Refinement Schedule

---

Published as a conference paper at ICLR 2023
DPC-full(C) c(t) C
DPC-light(C): c(t) = min(sc(t, m), -,, T2)),
with (t) max(1, 00, (---)) r)) a slanted step function, making c(t) trapezoid with the top vertices at t = Ti and = T2. In DPC-light, c(t) concentrates more correction steps around t = 71 72- To further reduce the cost, presents results varying C, and shows that more DPC correction steps effectively lead to improved sampling performance. Ablation studies for 71, 72 and /i are included in Appendix B.
stop the reverse process at T 5. Table
Upsampling in Discrete Latent Space Following the success of cascaded upsampling approaches in continuous diffusion models (Dhariwal & Nichol, 2021; Ho et al., 2022), we experiment with using cascaded super-resolution stage in the discrete latent space, and observe significant improvements compared to single resolution modeling (Table 2). Specifically, we train an upsampling discrete denoising model model the sequence of high-resolution visual tokens, conditioned on the low- resolution sequence. The upsampling model is trained with the objective of (2), using a cosine masking schedule. We refer Appendix C for further details on the upsampling stage.
Implementation Details The input images are discretized into a set of 10-bit integers using a VQGAN encoder (Esser et al., 2021b), e.g. a 512x512 image is represented as grid of 32x32 integer indices over a codebook of 1024 elements. For class-conditional generation, we prepend class token to the sequence of visual tokens. The denoiser Po and corrector P6 are transformer models in which, following MaskGIT (Chang et al., 2022) and Token-Critic (Lezama et al., 2022), the denoiser has 24 layers and 16 heads. Importantly, we used the same denoiser po for MaskGIT, Token Criiii and the proposed DPC, which was trained using maximum likelihood of the one-step model in (2). Thus, the comparison between these methods is restricted to the sampling scheme. The corrector is a 20-layer, h2-head transformer. The denoiser was trained for 600 epochs and the corrector for 300 epochs on 32 TPUs v4 with batch size 256. The VQGAN encoder was trained at 256x256 resolution on the same datasets. The upsampling stage is 16layer transformer and was trained for 300 epochs to denoise the encodings of 512x512 images conditioned on the encoding of ground truth 256x256 images. For DPC, unless otherwise noted, we use T = 18 diffusion steps, C' = 5 and Ti 4, 72 We control the sampling temperature of ps with the scale parameter 3 of the Gumbel noise, which is set to f = 0.6 for DPC-full and B = 0.5 for DPC-light.
4.2 MAIN RESULTS
Quantitative Evaluation Table I presents the quantitative comparison for the class-conditional generation on ImageNet. To compare the base modeling capacity of each method, no external pre- trained classifiers upsampling stages used during training sampling. As shown, the proposed DPC yields the best IS and FID scores on both 256 256 and 512x512 resolutions. DPC-light(5)) achieves good balance between the fidelity and the number of function evaluations (correction + time-transitions). For example, DPC-light(5) outperforms the previous best discrete diffusion VQ-Diffusion model by large margin in FID while using fewer diffusion steps. improves the
ImageNet 256
ImageNet 512x
Model
Type
NFE
FID +
IS f
Prec t Rec f
FID 4
IS f Prec f Rec f
BiGGAN
GAN
6.95
202.6
0.86
0.24
8.43
177.9
0.85
0.25
ADM
cont.
250
10.94
101.0
0.69
0.63
23.24
58.06
0.73
0.60
CDM
cont.
250
4.88
158.7
MaskGIT
transf.
6.56
203.6
0.79
0.48
8.48
167.1
0.78
0.46
MaskGIT (6-66)
transf.
5.09
198.7
0.79
0.52
7.10
166.6
0.77
0.50
Token-Critic
transf.
36
1.69
174.5
0.76
0.53
6.80
182.1
0.73
0.50
VQ-Diffusion
discrete
100
11.9
(PC-fulI(S)
discrete
180
4.45
244.8
0.78
0.52
6.06
218.9
0.80
0.47
DPC-light(S)
discrete
66
4.8
249.0
0.80
0.50
6.09
228.1
0.81
0.46
Table I: Bare model comparison on class-conditional generation on ImageNet (models do not use external classifiers or upsampling during sampling or training). We list the model type where "cont." is short for continuous diffusion, "discrete" for discrete diffusion, and "transf." for transformer.

---

Published as a conference paper at ICLR 2023
Model
NFE
FID
IS f Prec 1 RRec f
Model
quality
diversity
SylcGAN-XL
3.58
219.8 0.73
0.43
StyleGAN-XL
77.6%
54.42%4.44
#0.50
ADM+G+U
250x3
3.85
221.7
0.84
0.53
ADM+G+U
67.6%
10.4 30.223..2% 1:050
MaskGIT+U
18+6
5.10
221.8 0,80
0.51
MaskGIT
68.8%
10.16 54.0% 1:050
MaskGIT+R+U
18x5
1.26
326.9
0.81 0.50
Token-Critic 68.0% 10.46 48.4% 1:0.50
DPC-ligh((5)+U
6646
3.62
249.4 0.77
0.56
DRC- 66
3.54
350.2 2 0.78 0.55
Table 3: Proportion of times our DPC- light(5) model is preferred over other state-of-the-ar class-conditional genera- tion models in the user studies conducted on the ImageNet (512x512) benchmark.
Table 2: Performance of methods that leverage an up- sampling stage (U) and/or pre-trained classifier for guidance (G) or rejection (R), on ImageNet 512x512.
quality of non -oooereereeeesivvee transformers MaskGIT and Token- Critic with a reasonable increase in NFE (36) which is still less than that of the continuous diffusion models (250). We show DPC's performance for the unconditional generation on the Places2 dataset in Table 4.
In Table 2, we examine the comparison when the condiiiondi generation is aided by pre- re-trrined classifiers used gradient guidance (G) or rejection samplers (R), or when leveraging an upsampling stage (U) for cascaded generation. Using external classifiers improves the generation quality but makes it difficult to compare models given the distinct types of classifier architectures being used such as DeiT (Touvron et al., 2021) in SyleGAN-XL, UNet with CLIP attention (Radford et al., 2021) in ADM, or ResNet (He et al., 2016) in MaskGIT. Nevertheless, we present results for this setup by using a rejection sampling scheme based on a ResNet-50 classifier pretrained on ImageNet, with a specified acceptance rate of 25%. We apply the rejection sampling in samples generated at 256 resolution and then apply 6 upsampling decoding iterations to obtain the final 512x512 samples. For reference, ADM+G+U (Dhariwal & Nichol, 2021) uses x28x 128 generation followed by upsampling to 512x DPC achieves the best FID and Inception Score under this setup among the models we consider, and is highly competitive when using only generation and upsampling.
User Preference Study To further understand the perceptual differences between the compared models, we conduct two user preference studies on Amazon Mechanical Turk to verify the visual quality and diversity of our model's s samples. We compare our DPC-light(5), without classifier rejection or upsampling, to StyleGAN-XIL ADM with guidance and upsampling, and MaskGIT and Token-Critic without classifier rejection. The compared images are generated using the public models of StyleGAN-XL, ADM, and MaskGIT, obtained from their official websites.
For the quality test, we present two randomly sampled images of the same class side-by-side (one from DPC-light and one from the compared model), and ask the graders to select which one looks more realistic. For the diversity test, we show graders two sets of six randomly sampled images cach for four different classes, and ask them to select which set is more diverse looking. The quality and diversity tests are carried out on each of the 1,000 ImageNet classes, with 14 different graders, for a total of 14,000 comparisons. The comparison to each method is organized into 250 task groups.
As shown in Table 3, DPC is preferred for quality more times than all the compared methods. is noteworthy that the compared models in Table 3 represent the best published class-conditional models on the ImageNet benchmark, including GAN, diffusion, and transformer models. The differences in the quality test are statistically significant at the v-vauue level of 0.05. It also appears that DPC shows preferred diversity compared to StyleGAN-XL and ADM+G+U.
Qualitative Evaluation In Figure 1, we show random samples from our ImageNet 512x512 DPC- light(5) model, and from the publicly available implementations of StyleGAN-XI and ADM with classifier guidance and upsampling. These are the same models used in the user studies. We refer to Appendix D and the supplementary material for more comprehensive qualitative comparisons.
5 RELATED WORK
Diffusion Models The most prominent type of diffusion model that operates entirely in continuous space is the Gaussian diffusion model "Sohl-Dicksteee et al., 2015; Song & Ermon, 2019; Ho et al., 2020; Song al., 2021; Kingma al., 2021; Tzen & Raginsky, 2019; Kadkhodaie & Simoncelli,
8

---

Published as a conference paper at ICLR 2023
DPC-full (3-0.6)
DPC-light (3=0.5)
Model
NFE
FID 4 IS Pre
Rec
NFE
FID.
1St
NFE
FID4
1st
MaskGIT
18
31.3
8.6 0.65 0.12
36
23.87
101.0
26
15.69
128.9
MaskGIT
66
23.3
9.9 0.62 0.17
2
72
10.59
162.9
30
10.77
161.3
IPC-ful(()
36
20.3
9.6 0.62 0.47
3
108
7.25
194.5
38
8.13
187.6
DPC-full(S)
180
16.3
12.1 0.59 0.50
4
144
6.23
211.3
50
7.13
202.2
DPC-light(S)
66
16.2 11.5 0.55 0.50
5
180
6.06
218.9
66
6.09
228.1
6
216
6.20
222.1
86
5.99
233.0
Table 4: Performance on unconditional generation in the Places2 validation dataset at 512x512. We include MaskGIT's per- formance for reference.
7
252
6.53
221.1
108
5.86
245.1
Table 5: ImageNet 512 x522 performance when varying C' (corrections per timestep).
2021), which is formulated as a forward and a learned reverse process that are both parameterized conditional Gaussian distributions. These models have attained high quality generation results in image and audio tasks (Dhariwal & Nichol, 2021; Ho et al., 2022; Saharia et al., 2021 Nichol al., 2021; Ramesh et al., 2022; Chen al., 2020b; Kong et al., 2021b; Lugmayr et al., 2022).
Discrete Diffusion Recently, D3PM (Austin et al., 2021) and OA-ARDM (Hoogeboom et al., 2022) applied discrete diffusion (Sohl-Dickstein et al., 2015; Hoogeboom et al., 2021) to image modeling, focusing density estimation and compression of raw image pixels. Campbell et al. (2022) introduce a framework model discrete diffusion continuous time, and demonstrate that a combination of the denoising model and forward noising process valid corrector. In contrast to Campbell et al. (2022), our learned corrector can directly target the most unlikely tokens, achieving efficient sampling with one order of magnitude fewer steps. While these methods limit their analysis to oow-rssouution images (e.g., x2x32), VQ-Diffusion (Gu et al., 2022) is the state-of-the of-thhe-rt discrete diffusion model for ImageNet image synthesis, also employing to predict missing VQ-GAN tokens iteratively. Different from our model, VQ-Diffusion may still suffer from the compounding decoding error discussed in this paper. Our results show that our model outperforms VQ-Diffusion in both quality and efficiency on the ImageNet benchmark, while further scaling discrete diffusion models from the x56 the 512x512 resolution.
rnsplaee-replace diffusion strategy
Non-autoregressive transformers for image synthesis While early works (van den Oord et al., 2016; Salimans et al., 2017; Parmar et al., 2018; Chen et al., 2020a) modeled images directly in pixel space, various recent works (Ramesh et al., 2021; Esser et al., 2021b;a; Ding et al., 2022) used autoregressive transformers over the discrete latent space provided by VQVAE or VQGAN. Closest to work is class of non-autoregressive transformers (Chang al., 2022; Zhang et al., 2021; Lezama et al., 2022; Kong et al., 2021a) which have recently demonstrated improved quality and efficiency over conventional autoregressive transformer models like Esser et al. (2021b). There two major differences to these non-autoregressive transformers, First, we propose new techniques to perform multi-step correction steps which, shown in our experiments, are essential for achieving a desirable quality-vs- -effeiiiinnc trade-off. Second, we show that, in the ideal case, the proposed method can correct the train/test mismatch in the distributions of intermediate states. Finally, we further demonstrate the applicability of cascaded upsampling in the discrete latent space.
6 CONCLUSION
Parallel sampling from generative transformers dramatically improves efficiency compared to full autoregressive sampling. However, from the perspective of discrete diffusion models, it exacerbates deviations from the ideal sampling distribution that accumulate in the marginal distributions of intermediate diffusion states. To mitigate this compounding decoding error, we proposed DPC, new discrete diffusion model based on a learned MCMC corrector kernel that refines the samples of these intermediate states. Empirically, we demonstrated that the repeated application of the learned corrector improves the samples of non-autoregressive transformer, measured on the standard ImageNet class-conditional generation task. User preference studies showed that DPC is competitive with state-of-the-art generative vision transformers, GANs, and continuous diffusion models.

---

Published as a conference paper at ICLR 2023
ACKNOWLED
We would like to thank Ming-Hsuan Yang and Douglas Eck for helpful comments during early stages of this work. We also thank the anonymous reviewers for their insightful comments and constructive feedback that helped to improve this paper.
REFERENCES
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981-17999 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv:1607.06450, 2016.
arXiv preprint
Ralph Allan Bradley and Milton E Terry. Rank analysis incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-3455 1952.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint / 1096, 2018.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. arXiv preprint vrXiv:2205.14987. 2022.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. MaskGIT: Masked generative image transformer. pp. 11315-11325 2022.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In /CML, 2020a.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. WaveG- rad: Estimating gradients for waveform generation. arXiv preprint 3XXi:2000900711: 2020b.
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep
bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. CoRR, abs/2105.05233, 2021.
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. arXiv preprint arXiv:2204. 142/7, 2022.
Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer. Imagebart: Bidirectional context with multinomial diffusion for autoregressive image synthesis. Advances in Neural Information Processing Systems, 34, 2021a.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021b.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In EMNLP-IJCNIPP, 2019..
Ian J. Goodfellow, Jean Pouget-Abadic Mchdi Mirza, Bing Xu. David Warde-Farley. Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NewrIPS, 2014.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. pp. 10696-10706, 2022.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
10

---

Published as a conference paper at ICLR 2023
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Muchine Learning Research, 23(47):1-33, 2022.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. CoRR, abs/2102.05379, 2021.
Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In International Conference on Learning Representations, 2022.
Tzu-Kuo Huang, Ruby C Weng, Chih-Jen Lin, and Greg Ridgeway. Generalized Bradley-Teryy models and multi-class probability estimates. Journal of Machine Learning Research, 7(1), 2006.
David R Hunter. MM algorithms for generalized Bradley-Terry models. The annals of statistics, 32 (D:338-4066 2004.
Zahra Kadkhodaie and Eero Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in a denoiser. Advances in Neural Information Processing Systems, 34, 2021.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In CVPR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1412 2014.
Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. CoRR, abs/2107.00630, 2021.
Xiang Kong, Zhisong Zhang, and Eduard Hovy. Incorporating a local translation mechanism into non-autoregressive translation. arXiv preprint arXiv:2011.06113 2020.
Xiang Kong, Lu Jiang, Huiwen Chang, Han Zhang, Yuan Hao, Haifeng Gong, and Irfan Essa Blt: Bidirectional layout transformer for controllable layout generation. arXiv preprint arXiv:2112.05112. 2021a.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao and Bryan Catanzaro. DiffWave: A versatile diffusion model for audio synthesis. In 9th International Conference on Learning Representations, ICLR, 2021b.
Wouter Kool, Herke Van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. In International Conference on Machine Learning, pp. 3509--3508. PMLR, 2019.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila, Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
José Lezama, Huiwen Chang, Lu Jiang, and Irfan Essa. Improved masked image generation with Token-Crinne Crinnn In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXIII, pp. 70-86. Springer, 2022.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 11461-11471, 2022.
11

---

Published as a conference paper at ICLR 2023
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. International Conference on Machine Learning. PMLR, 2018.
Robin L Plackett. The analysis of permutations. Journal of the Royal Statistical Society: Series C (Applied Statistics), 24(2):193-202, 1975.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. arXiv, 2022.
Chitwan Saharia, William Chan, Huiwen Chang, Chris A Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. arXiv preprint arXiv:2/1 1.00526, 2021a
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2 104.07636, 2021b.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. PixeICNN++: Improving the PixeICNN with discretized logistic mixture likelihood and other modifications. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022.
Nikolay Savinov, Junyoung Chung. Mikolaj Binkowski, Erich Elsen, and Aaron van den Oord. Step-unrolled denoising autoencoders for text generation. arXiv preprint arXiv:21 2021.
Jascha Sohh-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribu- tion. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé
Jégou. Training data-efficient image transformers distillation through attention. International Conference on Machine Leurning. PMLR, 2021.
Belinda Tzen and Maxim Raginsky. Neural stochastic differential equations: Deep latent gaussian
models in the diffusion limit. arXiv preprint arXiv: 1005.0999000000000 09883, 2019.
12

---

Published as a conference paper at ICLR 2023
Benigno Uria, lain Murray, and Hugo Larochelle. A deep and tractable density estimator. In Proceedings of the 31th International Conference on Machine Learning. ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pp- 467-475. .org 2014.
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML, 2016.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Isabelle Guyon, Ulrike von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett (eds.), NewrIPS, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017
Zhu Zhang. Jianxin Ma, Chang Zhou, Rui Men, Zhikang Li, Ming Ding, Jie Tang, Jingren Zhou, and Hongxia Yang. M6-UFC: Unifying multi-modal controls for conditional image synthesis. arXiv preprint arXiv:2 1421111,.,, 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligenee 40(6)) 2017.
13

---

Published as a conference paper at ICLR 2023
A LEARNING THE MASK MODEL
Let (1') i=I..N be set of N logits, given by the output of a neural network taking Ty, t as input, i.e. 1' = 1. ( 1). We perturb these logits by adding i.i.d. Gumbel noise, i.e. i + G with G ~ (umbel(O, 1), and we rank-order these perturbed logits to get ranking c. Then the Plackett- Luce model (Plackett, 1975) states that the probability of obtaining an ordering is given by
N
exp(l"))
p(c/@o)
II
(11)
N Σラ exp(lo')
If we only care about the top-k ranking C* = {c' . then the probability of that partial ranking can be obtained as
exp(1%)
p(ck (00)
(12)
* ش-= exp(1(t) R
with EN ()=((+1) exp((")).
We define our mask m, be vector with one boolean element mi for each element P. If 7 is among the ky largest elements we set m 1, and otherwise we set my = 0. Using numpy notation, np nonzero For sampling mask m, we don't care about
we now have np. np. sort(cr)) the exact order Cs, but only about the elements in this top-k ordering. The probability of sampling mask m, can thus be found by summing over all the partial rankings e2 that are consistent with this mask:
exp(1%)) Ej= ())
p(m=
E I
(13)
R
O (m,) ] i=1
exp Σ
Σ E
cx(m.) i=1 Lyz +
iim!=1
with R
Σ (20,
(14)
lmm==
Evaluating the mask likelihood in (13) requires summing over all partial orderings C₁ consistent with the mask m,, which is computationally expensive. Next we discuss alternative formulations that are more computationally efficient.
Under the Plackett-Luce model, the logits 1 encode the relative log-odds of i ending higher than j in the sampled ranking c. That is p(2 {)) = like this, the Plackett-Luce model is also known the Bradley-Terryy model (Bradley & Terry, 1952: Hunter, 2004; Huang et al., 2006).) Thus, one alternative is to consider the relative scores of masked and unmasked elements, which yields the following objective:
P). (When applied to pairs of elements
Cpairwise
E (Ie, me) g(.., m.) ((og) m
Σ {mmf=1
Σ
log (or (1)( -1,(
(15)
jami =0
Another, simpler loss function which we found to work well in practice, is binary cross entropy independently applied to each mask element using [a = 1|@0, t) ( (0o. t)), where a the sigmoid function. The reasoning here is that, if the cutoff between ending in the top-k and ending outside of is represented by we may reasonably approximate the probability of ending in the top-k as p(i ck) zs all 1). Furthermore, since the Plackett-Luce distribution is invariant shifts in the logits I', the absolute level of can be chosen at will. We therefore choose 0, resulting in (mm; 1/ f) a (T ()), which fit against the marginal distribution of observed mask
14

---

Published as a conference paper at ICLR 2023
Model
FID
IS Prec Ree
factorized 6.41
209.5 0.78 0.45
pairwise
10.89
231.6 0.77 0.38
Table 6: Performance comparison between corrector models trained with the factorized loss (16) and the pairwise loss (15), for the same number training steps (200 epochs), on ImageNet 512x512. In both cases we use the DPC-fulI(S) sampling schedule.
elements q(m)) using maximum likelihood:
factorized
E (se, me) n g( m.) &o po (2ig (.))
Σ mi-log (o (fs( -(-m)))))) (1 o o (Co(00, t)))
(16)
Table 6 shows quantitative comparison of the corrector models trained with Lpiiruisee and factorized. We train both models for the same number of training steps (200 epochs), and sweep sampling hyperparameter (3) for better FID, We found the factorized loss to be more efficient and we use this loss in all the experiments reported in the main manuscript.
B ABLATION STUDIES
Figure we show resulting FID and Inception Score when varying the parameter the sampling temperature 3. Our observation is that the resulting sampling quality, as measured by FID and Inception Score, is highly sensitive to the value of 3, and that the optimal | value also depends on the number of correction steps C' and the refinement schedule c(t).
In Figure 3, we present an ablation study on the shape the trapezoidal function c(t) for the DPC- light model. We vary the starting point of the top side of the trapezoid, Ti and its width 72 T1. We observe slightly better metrics by concentrating the trapezoid around the first steps of the decoding process.
DPC-FulIIS)
(PC-fuli3)
222
325
2312
-20
18:
175
(RC incoptinn
13
DPC-lghh(S)
DPC-lightE3)
42
202
30-
Inccupttion
12
13-
Figure 2: Ablation study for sampling temperature parameter 3. We show the effect of B on FID and Inception Score for models DPC-full(5), fPC-full(3), DPC-light(5) and DPC-light(3). We notice high sensitivity to B and strong correlation between the effect on FID and Inception Score.
C UPSAMPLING IN DISCRETE SPACE
We perform upsampling in the discrete latent space by learning to generate the sequence of visual tokens corresponding to high-resolution image x given the sequence of visual tokens from low-resolution downsampled version æ/ For the experiments the paper we use the 512x512 and x56x256 resolutions, and use the same VQ-encoder for both resolutions, yielding sequences of
15

---

Published as a conference paper at ICLR 2023
m1
b-
241
02
04-
124
Incapption
6.
50
245
Incoption Score
239
222
200
Figure 3: Ablation study for refinement schedule parameter 71. We show the effect of varying the location of the top left vertix of the trapezoid тi and its width 7i on FID and Inception Score, for DPC-light(5). We observe that concentrating more refinement steps towards the begining of the decoding process is beneficial for FID, and around the = 4 for Inception Score. In all experiments the number of diffusion steps is T 18.
Masking Rate FID
IS f Prec Rec f
0%
5.19
174.5
0.78 0.50
5%
5.22
173.5
0.78 0.50
10%
5.28
172.0
0.78 0.50
Table 7: Effect of random perturbation applied to the low-resolution sequence when training the upsampling stage. The perturbation consists in masking portion of the tokens and replacing them with the predictions of the low-resolution denoising model. We use the one-step denoising model (without corrector) for this experiment.
length 1024 (32x32) and 256 (16x 16), respectively. The upsampling model pu is trained with the one-step denoising objective:
Capsampling
log Il Pa (ad (i)lar 2t mi).
(17)
(mo . )
where the mask rate in m, follows cosine scheduling function for the denoiser model ño. We use transformer to model the high resolution sequence, with cross-attention to the low-resolution sequence every layer and independent positional encodings for each resolution.
During inference, we first generate an image (sequence of visual tokens 2 in 256x256 resolution. We then perform 6 decoding steps with the upsampling model P., following the cosine scheduling function. For simplicity, when using DPC with upsampling, we only apply corrector steps in the low resolution, and use the prediction confidence in Pu to keep/reject tokens (similarly to MaskGIT (Chang et al., 2022)) in each upsampling decoding iteration.
Following observations in cascaded continuous diffusion, where the low resolution images are augmented with random perturbations to reduce exposure bias (Ho et al., 2022), we experimented with randomly perturbing the low-resolution sequence. To randomly perturb the low-resolution sequence we randomly mask subset of tokens and ran the partially masked sequence through the low-resolution denoiser pa. Table 7 shows the results obtained for different strengths of perturbation (masking rate). These results suggest that in the discrete case, low-rese row-resouution augmentation does not improve the upsampling stage. Thus, for simplicity we did not use any augmentation for the experiments in Section 4.
16

---

Published as a conference paper at ICLR 2023
Figure 4: Random samples from ImageNet 512x512 class-conditional generation for selected classes: 'lorikeet', 'wallaby', 'flat-coated retriever', 'groenendael', 'golfcart' and 'racing car', Left: StyleGAN-XL (NFE = 1). Center: ADM Classifier Guidance Upsampling (NFE 250 3).
Right: DPC-light(5) (NFE 66).
D FURTHER QUALITATIVE RESULTS
ImageNet We present further 512x512 ImageNet class-conditional samples in Figure 4, comparing the proposed DPC-light(S) to StyleGAN-XL (Sauer et al., 2022) and ADM with Guidance and Upsampling (Dhariwal & Nichol, 2021). In Figure 5 further show random samples of random classes for each of the three methods. We provide further qualitative results in the Supplementary Material.
Places2 In Figure 6 show samples from the Places2 dataset (Zhou al., 2017), by the proposed DPC-light(S) (FID 16.2) and MaskGIT (FID 26.3), as well as original images from the dataset for reference.
17

---

Published as a conference paper at ICLR 2023
iarm
Figure 5: Random samples from random classes from ImageNet 512 class-conditional genera- tion for selected classes. Left: StyleGAN-XL (NFE = 1). Center: ADM Classifier Guidance +
Upsampling (NFE 250 x 3). Right: DPC-light(S) (NFE 66).
Figure 6: Unconditional image generation in the Places2 dataset (Zhou et al., 2017) at 512 x 512 resolution. Random samples from left: Ground Truth, center: MaskGIT (FID 23.3, NFE 66), right: DPC-light(5) (FID 16.2, NFE = -66).
E FURTHER IMPLEMENTATION DETAILS
All the transformers used this work have embedding dimension 768 and hidden dimension 3,072, learnable positional embedding (Devlin et al., 2019), LayerNorm (Ba et al., 2016), and truncated normal initialization (stddev= 0.02). The following training hyperparameters were used for both transformers: dropout rate 0.1, Adam optimizer (Kingma & Ba. 2014) with 81 0.9 and 32 = 0.96. We used RandomResizeenddCop for data augmentation and the denoiser is trained with label smoothing 0.1.
18

---

Published as a conference paper at ICLR 2023
F USER PREFERENCE STUDY DETAILS
diferrt mathect.
sade burtonu cimorts atanso cateegg
Hight mesernaiiti
Thank yours muce
Figure Screenshot for the quality user preference study.
In Figure 7 we present a screenshot for the quality user study. Users were presented with two generated images of the same class, one sampled from our method and from the compared method, and were shown the following text prompt:
Here are two images generated from the same input category using different methods. Please use radio buttons above choose the more realistic looking one. It would be great you could check both sets images carefully, as they may contain subtie artifacts that are not immediately obvious. Thank you so much!
In Figure we show a screenshot for the diversity user preference study. Users were presented with two grids of random samples, for method and one for the compared method. The grid contained images for each of classes. The users were shown the following text prompt:
each tab, we have two sets images generated from the same input categories using different methods.
Please use the radio buttons above to choose the image set with more diverse content, lighting and background.
Note that you must select one in each tab before you submit.
It would great you could check both sets images carefully, as they may contain subtle differences that are not immediately obvious. Thank you so much!
Figure& 8: Screenshot for the diversity user preference study.
19


Reviews:
[{"review_id": "zEYgruq1Ey7", "reviewer": "Reviewer_bius", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the Discrete Predictor-Corrector diffusion model (DPC). This model can synthesis better images, which is discussed both qualitatively and quantitatively. This model is evaluated on class-conditional image generation on the ImageNet dataset and unconditional generation on the Places2 dataset.", "summary_of_the_review": "The paper is well-written and easy to follow. The experimental results are stable. While the main claim is that the DPC model improves the generated images' quality compared with previous discrete diffusion. The novelty would raise if this paper could show that the DPC model extends the ability or accelerate the computation of the diffusion model. ", "strength_and_weaknesses": "Strength:\n(1) The paper is well-written and easy to follow.\n(2) The discrete predictor-corrector algorithm makes sense.\n(3) The experimental results are stable, showing this model is effective. The results are evaluated on some widely used metrics, and the user study is reasonable.\n\nWeaknesses:\n(1) The diffusion model is computationally expensive, so it would be better to analyze the computation cost or time consumption for training and inference.\n(2) The paper focuses on ImageNet and Places2 datasets. These datasets have less space to improve compared with complex datasets. So it would be better to include more datasets (e.g., LSUN, CelebAHQ). It is also worth trying to contain many objects in one image, although it is difficult to deal with.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "xGGXV7ZDd6", "reviewer": "Reviewer_mMR5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new sampling method that improves the quality of generated images from discrete diffusion models.  In the new method, denoising steps are interleaved with ‘corrector’ steps that mitigate the problem of compounding decoder error.", "summary_of_the_review": "The paper represents a useful advance in improving the cost/quality tradeoff when sampling images using discrete diffusion models. It introduces a technique that could be more broadly useful.", "strength_and_weaknesses": "The results are impressive. The sampling method is elegant and some reasonable argument is made about why it works.\n\nI would like to see more discussion of why this new learned corrector works well. The only explanation really offered by the authors in 3.1 is that it approximately satisfies detailed balance for $q(x_t)$, but many different correctors would approximately satisfy detailed balance for this distribution, and some will have better properties (mixing time, accuracy of the stationary distribution) than others. Campbell et al. 2022 http://arxiv.org/abs/2205.14987 introduce a simple corrector for discrete-state-space diffusion models, which requires no extra training. Compared to the present learned corrector, Campbell’s corrector seems like a closer analogue to the Langevin corrector described by Song et al.. Campbell et al. also justify their corrector in terms of detailed balance. It does not have the flexibility to selectively re-mask those pixels that do not ‘match’ other pixels (which I expect the learned corrector to do), and intuitively I expect the simple corrector to have lower-entropy transitions and longer mixing time than the learned corrector. It would be nice to see an empirical comparison.\n\nA theoretical question - does small KL divergence in (10) imply small deviation from the ideal stationary distribution in (6)? It’s possible for two transition  matrices to be arbitrarily close to each other as measured using e.g. Lp norm, while having very different stationary distributions.\n\nLugmayr et al. 2022  https://arxiv.org/abs/2201.09865 also do (continuous state-space, conditional) image generation by interleaving forward and reverse diffusion steps.  How does it relate to the present work?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "jJLfUJzdYeX", "reviewer": "Reviewer_spPv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work extended the Predictor-Corrector sampling algorithm originally developed in the continuous diffusion models to the discrete case, which is termed discrete Predictor-Corrector (DPC). Since, in the discrete diffusion models, there is no direct counterpart to the score function, this work proposed to learn a new MCMC corrector kernel, whose limiting distribution is the true marginal distribution of the intermediate data. Besides, this work provides new insights into interpreting the parallel decoding in the non-autoregressive transformer methods as a reverse diffusion process using one-step sampling, and reveals the compounding decoding error wherein. This work focuses on applying DPC to the discrete diffusion models in the latent space of VQ-GAN. On the high-resolution image synthesis tasks, such as ImageNet (256x256 and 512x512) and Places2 (512x512), this work demonstrates that DPC performs comparable to the latest non-autoregressive methods (e.g., MaskGIT, Token-Critic) and outperforms the state-of-the-art continuous diffusion models (e.g., ADM, CDM) and discrete diffusion models (e.g., VQ-Diffusion). \n", "summary_of_the_review": "Overall, I think the idea is interesting and can be useful for other discrete diffusion models, and the insights into understanding the improving the non-autoregressive transformer methods from the discrete diffusion model perspective could also inspire the better development of non-autoregressive transformers. The experiments are well-executed and relatively strong (although I have the concern about the larger computational cost due to the extra corrector steps). Thus, my initial recommendation is an accept.\n", "strength_and_weaknesses": "Strengths:\n- The proposed method DPC is well-motivated (to reduce the compounding decoding error introduced by the parallel decoding) and novel (by learning a new MCMC corrector kernel that serves the similar purpose of the Langevin corrector as in the continuous case). \n- The insights into understanding the improving the non-autoregressive transformer methods (e.g., MaskGIT, Token-Critic) from the discrete diffusion model perspective are very interesting. It shows that the proposed method generalizes MaskGIT and Token-Critic with a better theoretical justification. \n- It shows the applicability of cascaded upsampling in the discrete latent space, and observes the large improvements over single resolution modeling.\n- Experiments on the challenging image datasets (ImageNet and Places2) demonstrate the effectiveness of the proposed method by comparing various state-of-the-art generative models (BigGAN, StyleGAN-XL, ADM, CDM, VQ-Diffusion, MaskGIT, and Token-Critic).\n- The writing is overall clear and the paper is easy to follow.\n\nWeaknesses:\n- The corrector steps in DPC introduce more computational cost, particularly in inference. Compared to MaskGIT and Token-Critic (two similar methods as special cases of DPC without corrector steps), as shown in Table 1 and Table 3, the improvement of DPC in image quality (while it may sacrifice the diversity) comes with a larger NFE. For example, Token-Critic has 4.69 FID with 36 NFEs vs. DPC-full (5) has 4.45 FID with 180 NFEs. \n- It is confusing to me how exactly the corrector steps are repeated in the shortcut time transition. I understand that the original version of DPC contains one predictor step (step 1) and one or multiple corrector steps (step 2 and step 3). Since we combine step 1 and step 2 in the shortcut time, how do we perform the predictor step and corrector steps separately? \n- I think it would be better to add the background of the Predictor-Corrector sampling in the continuous case to make this work more accessible to a broader audience.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster