Diffusion Models Beat GANs on Image Synthesis
Authors: Prafulla Dhariwal
Alexander Quinn Nichol
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: generative models
diffusion models
score-based models
denoising diffusion probabilistic models
image generation
neural networks
attention

Abstract: We show that diffusion models can achieve image sample quality superior to the current state-of-the-art generative models. We achieve this on unconditional image synthesis by finding a better architecture through a series of ablations. For conditional image synthesis, we further improve sample quality with classifier guidance: a simple, compute-efficient method for trading off diversity for fidelity using gradients from a classifier. We achieve an FID of 2.97 on ImageNet 128$\times$128, 4.59 on ImageNet 256$\times$256, and 7.72 on ImageNet 512$\times$512, and we match BigGAN-deep even with as few as 25 forward passes per sample, all while maintaining better coverage of the distribution. Finally, we find that classifier guidance combines well with upsampling diffusion models, further improving FID to 3.94 on ImageNet 256$\times$256 and 3.85 on ImageNet 512$\times$512.

Diffusion Models Beat GANs on Image Synthesis
Prafulla Dhariwal"
Alex Nichol" OpenAl al alex@openai.com com
OpenAl prafulla@openai.com com
Abstract
We show that diffusion models achieve image sample quality superior to the current state-of ttate-of-the-art generative models. We achieve this on unconditional image synthesis by finding better architecture through series of ablations. For condi- tional image synthesis. we further improve sample quality with classifier guidance: simple, compute-cfficee -fficceett method for trading off diversity for fidelity using gradi- ents from a classifier. We achieve an FID of 2.97 on ImageNet 128 128, 4.59 on ImageNet 256x256 and 7.72 ImageNet 512x512, and match BigGAN-deep even with as few as 25 forward passes per sample, all while maintaining hetter coverage the distribution. Finally, find that classifier guidance combines well with upsampling diffusion models, further improving FID to 3.94 on ImageNet 256 256 and 3.85 ImageNet 512x512.
Introduction
Figure Selected samples from our best ImageNet 512x512 model (FID 3.85)
Over the past few years, generative models have gained the ability generate human-like natural language [ high-quality synthetic images [8] 34 57 and highly diverse human speech and music
These models can be used in variety of ways, such generating images from text prompts or learning useful feature representations [88 10 While these models are already capable
56
"Equal contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

of producing realistic images and sound, there is still much room for improvement beyond the current state-of-the-are and better generative models could have wide-ranging impacts on graphic design. games, music production, and countless other fields.
GANs 2 currently hold the oate--of-the-are on most image generation tasks
+4
as measured
by sample quality metrics such as FID [29]. Inception Score [6]] and Precision
38 However, some
of these metries do not fully capture diversity, and has been shown that GANs capture less diversity than state-of-the-art train, collapsing without carefully selected hyperparameters and regularizers
sreeihoodd-assee models [57 4o 4
Furthermore, GANs are often difficult to
4 77. While GANs N
their drawbacks make them difficult to scale and apply to new domains. As result, much work has been done to achieve GAN-like sample quality with likelihood-biae based models [22] [57 31 48 [12. While these models capture more diversity and are typically easier to scale and train than GANs, they still fall short in terms of visual fidelity. Furthermore, except for VAEs, sampling from these models is slower than GANs in terms of wall-clock time.
hold the saate-of-hhe-se
Diffusion models are a class of likelihood-basee models which have recently been shown to produce high-quality images [63 66 31 40
while offering desirable properties such as distribution coverage, stationary training objective, and easy scalability. These models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound [3I. This class of models already holds the state-of- th-tte- [7 on CIFAR-10 37, but still lags behind GANs on difficult generation datasets like LSUN and ImageNet. We hypothesize that this gap exists for at least two reasons: first, that the model architectures used by recent GAN literature have been heavily explored and refined; second, that GANs are able to trade off diversity for fidelity, producing high quality samples but not covering the whole distribution. We aim to bring these benefits to diffusion models, first by improving model architecture and then by devising a scheme for trading off diversity for fidelity.
The rest of the paper is organized as follows. In Section 2. we give a brief background of diffusion models based on Ho et al. 3 and the improvements from Nichol and Dhariwal 4 and Song
et al. 164,, and we describe our evaluation setup. In Section 3 we introduce simple architecture
improvements that give a substantial boost to FID. In Section 4 we describe a method for using
we show that models with our improved architecture achieve state-of-the-an on unconditional image synthesis
gradients from classifier to guide a diffusion model during sampling. Finally, in Section 5
tasks, and with classifier guidance achieve state-of-the-arr on conditional image synthesis.
Background
In this section, we provide a brief overview of diffusion models. For a more detailed mathematical description, we refer the reader to Appendix C On high level, diffusion models sample from distribution by reversing a gradual noising process. In particular, sampling starts with noise xr and produces gradually less-noisy samples 27-1. 2T-2,- ... until reaching final sample To- In particular, a diffusion model learns to produce a slightly more "denoised" xt-1 from Zt- Ho et al. 311 parameterize this model using a function cạ(x,, t) which predicts the noise component of noisy sample 24- To train this function, each sample in a minibatch is produced by randomly drawing a data sample Zg. a timestep t,, and noise c, which together give rise to noised sample 2/, (Equation 3 Appendix C). The training objective is then [[cg(Zr, t) - c i.e. a simple mean-squared error loss between the true noise and the predicted noise (Equation 12
Appendix C).
Ho al. 331 show that, under reasonable assumptions, we can then model the denoising distribution
Pa( (z) of æ given as a diagonal Gaussian N (x;-11 juo (are, l), Eo ()), where the mean
(Zr, [Z t) can be calculated as a function of cø t) (Equation 13 Appendix Cp. Ho et al. [ observe that the simple mean-squared error objective, Liimple works better in practice than the actual variational lower bound Lvb that can be derived from interpreting the denoising diffusion model as VAE. They also note that training with this objective and using their corresponding sampling procedure is equivalent to the denoising score matching model from Song and Ermon (65) who use Langevin dynamics to sample from denoising model trained with multiple noise levels to produce high quality image samples. We often use "diffusion models" as shorthand to refer to both classes of models.
Following the breakthrough work of [ong and Ermon papers have proposed improvements to diffusion models. Nichol and Dhariwal 49
65 and Ho et al.
several recent
find that fixing
2

---

the variance Eg(r,, t) to a constant as done in Ho et al. [1] is sub-optimal for sampling with fewer diffusion steps, and propose to parameterize [ t) as a neural network whose output is Appendix C are the variances in Ho et al. 31 corresponding to upper and lower bounds for the reverse process variances. Additionally, Nichol and Dhariwal 49 propose a hybrid objective for training both (z f) and Eg(x,, tt) using the weighted sum simple ALvb. Learning the reverse process variances with their hybrid objective allows sampling with fewer steps without much drop in sample quality. We adopt this objective and parameterization, and use it throughout our experiments.
interpolated as (x t) = exp( y log 3r + (1 - v) log Br) Here, 3. and B. (Equation 5
propose DDIM, which formulates an alternative non- Markovian noising process that has the same forward marginals as DDPM, but allows producing different reverse samplers by changing the variance of the reverse noise. By setting this noise to 0, they provide a way to turn any model (I t) into a deterministic mapping from latents to images, and find that this provides an alternative way to sample with fewer steps. We adopt this sampling approach when using fewer than 50 sampling steps, since Nichol and Dhariwal [4 found it to be beneficial in this regime.
Song et al.
Sample Quality Metries:
For comparing sample quality across models, we perform quantitative evaluations using the following metrics. While these metrics are often used in practice and correspond well with human judgement, they are not a perfect proxy, and finding better metries for sample quality evaluation is still an open problem.
We use FID 29
our default metric for overall sample quality comparisons it captures both fidelity and diversity and has been the de facto standard metric for state-of-the-art generative models as proxies for separately measuring fidelity and diversity, respectively. We include sFID 48 as a metric that better captures spatial relationships than
[3] 64 R 3J. We use Precision and Recall KX
FID, and also include Inception Score (IS) 6 as another proxy for fidelity. When comparing against other methods, we re-compute these metries using public samples or models whenever possible. This for two reasons: first, some papers 33
compare against arbitrary subsets of the training set which are not readily available; and second, subtle implementation differences can affect the resulting FID values [5 For consistent comparisons, we use the full training set as the reference batch [29
8 and evaluate metrics for all models using the same codebase.
3
Architecture Improvements
Ho et al. aclottdd the UNet architecture 158 for diffusion models, which Aellicoeur-Martinee et al.
32 found to substantially improve sample quality over the previous architectures [65] 39 used for denoising score matching. The UNet model uses a stack of residual layers and downsampling convolutions, followed by a stack of residual layers with upsampling convolutions, with skip con- nections connecting the layers with the same spatial size. In addition, they use a global attention layer at the 16x 16 resolution with a single head, and add a projection of the timestep embedding into each residual block. [Sog et al. [67] found that further changes to the UNet architecture improved performance on the CIFAR-I0 10
37 and CelebA-64 440 datasets. We show the same result on ImageNet 128 128, finding that architecture can indeed give substantial boost to sample quality on a much larger and more diverse datasets at higher resolution.
We explore the following architectural changes: increasing depth versus width, holding model size relatively constant; increasing the number of attention heads; using attention at 32x32, 16x 16, and 8x8 resolutions rather than only 16x 16; using the BigGAN g residual block for upsampling and downsampling the activations, following [7]; and finally; rescaling residual connections with 72' following
67
33 34
We train models with the above architecture changes on ImageNet 128 x 128 and compare them on FID, evaluated at two different points of training, in Table D Aside from rescaling residual connections, all of the other modifications improve performance and have a positive compounding effect. On wall-clock (Figure 5 Appendix [ we find that increased depth hurts training time most, so we opt not to use this change in further experiments. We also study other attention configurations that better match the Transformer architecture [72]. We try two configurations: constant attention heads, or constant channels per head. Table shows our results, indicating that more heads or fewer channels per head improves FID. On wall-clock (Figure Appendix A. we see that 64 channels is best so we opt to use 64 channels per head as our default. We note that this choice also better matches modern transformer architectures, and is on par with our other configurations in terms of final FID.
3

---

Table Ablation of various architecture changes, evaluated 700K and 1200K iterations
Attention resolutions
BigGAN up'downsample
Rescale reshlock
FID 700K
FID
Channels
Depth
Heads
1200K
160
2
1
16
X
x
15.33
13.21
128
4
-0.21
-0.48
4
-0,54
-0.82
32,16,8
-0.72
-0.66
-1.20
-1.21
0.16
0.25
160
2
4
32,16,8
x
-3.14
-3.00
Table 2: Ablation of attention heads. More heads lower channels per heads both improve FID. The base model was a smaller version of the best model from Table !
Number of heads
Channels per head
FID
14.08
-0.50
-0.97
-1.17
32
-1.36
64
-1.03
128
-1.08
We also experiment with layer 49 that refer to adaptive group normalization (AdaGN), which incorporates the timestep and class embedding into each residual block after group normalization operation (75) similar to adaptive instance norm [3 and FiLM (54] We define this layer as AdaGN(h, y) yss GroupNorm(()) + yb. where l is the intermediate activations of the residual block following the first convolution, and y [u.: yb] is obtained from linear projection of the timestep and class embedding. We had already seen AdaGN improve our earliest diffusion models, and so had included by default in all runs. We explicitly ablate this choice (Table 6 Appendix A. and find that FID becomes worse by 2.02 when we remove the adaptive group normalization layer.
In the rest of the paper, with residual blocks per resolution, multiple heads with 64 channels per head, attention 32, 16 and 8 resolutions, BigGAN residual blocks for up and downsampling, and adaptive group normalization for injecting timestep and class embeddings into residual blocks.
this final improved model architecture
default: variable width
4 Classifier Guidance
In addition to employing well designed architectures, GANs for conditional image synthesis 45
make heavy use of class labels. This often takes the form of class-conditional normalization statisties
20 [4] as well as discriminators with heads explicitly designed to behave like classifiers p(y xr) 46
As further evidence that class information is crucial the success of these models, Lucic et find that it is helpful to generate synthetic labels when working in a label-limited regime. Given this observation for GANs, makes sense to explore different ways to condition diffusion models on class labels. We already incorporate class information into adaptive group normalization layers (Section 3 Here, we explore a different approach: exploiting a classifier p(y|z) to improve a diffusion generator. Sohl-Dickstein et diffusion model can be conditioned using the gradients of a classifier. particular, we can train classifier P& ( (,) f) on noisy images 2t, and then use gradients V,, Po (3) to guide the diffusion sampling process towards an arbitrary class label y.
[ and Song al. [67] show one way achieve this, wherein pre-trained
For class conditional diffusion sampling, we reproduce the derivation from Sohl- et al. 6% Appendix D.2 For DDIM, we perform a score-based derivation in Appendix D.3 inspired by Song et al.
67 The resulting sampling algorithms we use for guidance are Algorithms and respectively. Both algorithms incorporate class information by adding the gradients of classifier to each sampling step with appropriate step size. In these algorithms, choose the notation

---

Algorithm| 1 Classifier guided diffusion sampling, given diffusion model (No(Zz)) Eo(xe) classi-
fier Po(y)an (t)); and gradient scale 8.
Input: class label y, gradient scale 8
Jr <- sample from NO I)
for all from 7 to 1 do
p4, - pa (zz), [x()))
It-1 f sample from N( + sE Va, logps((los), 2)
end for return
Algorithm 2 Classifier guided DDIM sampling, given diffusion model c0(x:), classifier Po(u(z))) and gradient scale
Input: class label y. gradient scale 8
<- sample from (O,
for all from T to 1 do
8 (x
vI ag Vz, log Po (v)2z)
2t-1 var-1
+ /1 Ō
Vas
end for return 20
Samples from an unconditional diffusion model with classifier guidance to condition on the class "Pembroke Welsh corgi". Using classifier scale 1.0 (left; FID: 33.0) does not produce convincing samples in this class, whereas classifier scale 10.0 (right; FID: 12.0) produces much more class-consistent images.
Figure 2:
(olaa d) (y) and (0(2,, L) eo (are) for brevity, noting that they refer separate functions for each timestep / and at training time the models must be conditioned on the input 1.
To apply classifier guidance to a large scale generative task, train classification models ImageNet. Our classifier architecture is simply the downsampling trunk of the UNet model with an attention pool [55] at the 8x8 layer to produce the final output. We train these classifiers the same noising distribution as the corresponding diffusion model, and also add random crops to reduce overfitting.
In initial experiments with unconditional ImageNet models, we found it necessary to scale the classifier gradients by a constant factor larger than 1. When using scale of 1, we observed that the classifier assigned reasonable probabilities (around 50%) to the desired classes for the final samples, but these samples did not match the intended classes upon visual inspection. Scaling up the classifier gradients remedied this problem, and the class probabilities from the classifier increased to nearly 100%. Figure 2 shows example of this effect. To understand the effect of scaling classifier gradients, note that V. log logp(y) = V, log }p(y|=)", where result, the conditioning process is still theoretically grounded in a re-normalized classifier distribution proportional to ()) When s > 1, this distribution becomes sharper than p(y(æz), since larger values amplified by the exponent. In other words, using larger gradient scale focuses more on the modes of the classifier, which is potentially desirable for producing higher quality (but less diverse) samples.
an arbitrary constant. As
In the above derivations, we assumed that the underlying diffusion model was unconditional, modeling p(z). It is also possible to train conditional diffusion models, p(zly), and use classifier guidance in
5

---

Table Effect of classifier guidance on sample quality. Both conditional and unconditional models were trained for 2M iterations on ImageNet x56x256 with batch size 256.
Conditional
Guidance
Scale
FID
sFID
IS
Precision
Recall
X
26.21
6,35
39.70
0.61
0,63
XXY
1.0
33.03
6.99
32.92
0.56
0.65
10.0
12.00
10.40
95,41
0.76
0.44
X
10.94
6.02
100.98
0.69
0.63
/
V
1.0
4.59
5,25
186.70
0.82
0.52
10.0
9.11
10.93
283.92
0.88
0.32
procsion
16
0.9
14-
10
200
0.5
150
0.4
100
gradient scale
gradient
Figure 3: Change in sample quality as we vary scale of the classifier gradients for class-conditional ImageNet 128x x 128 model.
the exact same way. Table 3 shows that the sample quality of both unconditional and conditional models can be greatly improved by classifier guidance. We see that, with high enough scale, the guided unconditional model can get quite close the FID of an unguided conditional model, although training directly with the class labels still helps. Guiding a conditional model further improves FID,
Table 3 also shows that classifier guidance improves precision at the cost of recall, thus introducing trade-off in sample fidelity versus diversity. We explicitly evaluate how this trade-off varies with the gradient scale in Figure 3 We see that scaling the gradients beyond 1.0 smoothly trades off recall (a measure of diversity) for higher precision and IS (measures of fidelity). Since FID and sFID depend on both diversity and fidelity, their best values are obtained an intermediate point. We also compare our guidance with the truncation trick from BigGAN (Figure C Appendix A.2). We find that classifier guidance is strictly better than BigGAN- deep when trading off FID for Inception Score. Less clear cut is the precision/recall trade-off, which shows that classifier guidance is only a better choice up until a certain precision threshold, after which point it cannot achieve better precision.
5 Results
To evaluate our improved model architecture on unconditional image generation, we train separate diffusion models on three LSUN (77 classes: bedroom, horse, and cat. To evaluate classifier guidance, train conditional diffusion models on the ImageNet [59] dataset 128x 128, 256 256, and 512x512 resolution.
Table summarizes our results. ADM refers to our ablated diffusion model, and ADM-G additionally uses classifier guidance. Our diffusion models can obtain the best FID on each task, and the best sFID all but one task. With the improved architecture, we already obtain state-of-the-art image generation on LSUN and ImageNet 64 x4. For higher resolution ImageNet, we observe that classifier guidance allows models substantially outperform the best GANs. These models obtain perceptual quality similar to GANs, while maintaining a higher coverage the distribution as measured by recall, and can even do so using only 25 sampling steps. We also evaluate the computational requirements for training our models (Table 10 Appendix B and find that we can obtain competitive sample quality while using the or less compute than the corresponding BigGAN-deep or StyleGAN2 model.
Figure compares random samples from the best BigGAN-deep model to our guided diffusion model. While the samples are of similar perceptual quality, the diffusion model contains more modes than the GAN, such as zoomed ostrich heads, single flamingos, different orientations of cheeseburgers, and

---

Table 4: Sample quality comparison with state-of afrt generative models for each task. LSUN diffusion models are sampled using 1000 steps (see Appendix D ImageNet diffusion models are sampled using 250 steps, except when we use the DDIM sampler with 25 steps. *No BigGAN-deep model was available at this resolution, so we trained our own. Values are taken from previous paper, due to lack of public models samples. Resullts use two-resolution stacks. Results use compute-imtensive classifier rejection sampling.
Model
FID sFID
Prec
Rec
Model
FID
sFID
Prec
Rec
LSUN Bedrooms 256x256
ImageNet 128x
DCTransformer
6.40
6.66
0.44
0.56
BigGAN-deep B
6.02
7.18
0.86
0.35
DDPM AI
4.89
9.07
0.60
0.45
LOGAN! 74
3.36
IDDPM 49
4.24
8.21
0.62
0.46
ADM
5,91
5.09
0.70
0,65
StyleGAN
3
2.35
6.62
0.59
0.48
ADM-G (25 steps)
5.98
7.04
0.78
0.51
ADM (dropout)
1.90
5.59
0.66
0.51
ADM-G
2.97
5.09
0.78
0.59
ImageNet 256
LSUN Horses 256
DCTransformer
36.51
8.24
0.36
0.67
StyleGAN2 3
3.84
6.46
0.63
0.48
VQ-VAE-21 5
31.11
17.38
0.36
0.57
ADM
2.95
5.94
0.69
0.55
VQ-VAF-2 (RS)
57
10
ADM (dropout)
2.57
6.81
0.71
0.55
VQ-GAN: 21
15.97
19.05
0.63
0.58
VQ-GAN (RS)18
21
5.06
7.34
0.79
0.48
IDDPM 40
12.26
5,42
0.70
0.62
LSUN Cats 256x256
SR3 6U
11.30
DDPM 3I
17.1
12.4
0.53
0.48
BigGAN-deep
8
6.95
7.36
0.87
0.28
StyleGAN2
7.25
6.33
0.58
0.43
ADM
10.94
6.02
0.69
0.63
ADM (dropout)
5.57
6.69
0.63
0.52
ADM-G steps)
5.44
5.32
0.81
0.49
ADM-G
4.59
5.25
0.82
0.52
ImageNet 64
ImageNet 512x512
digepN-deepp
4.06
3.96
0.79
0.48
BigGAN-deep [
8.43
8.13
0.88
0.29
IDDPM 4J
2.92
3.79
0.74
0.62
ADM
23.24
10.19
0.73
0.60
ADM
2.61
3.77
0.73
0.63
ADM-G (25 steps)
8.41
9.67
0.83
0.47
ADM (dropout) 2.07
4.29
0.74
0.63
ADM-G
7.72
6.57
0.87
0.42
tinca fish with no human holding We also check our generated samples for nearest neighbors in the DnceptionV33 feature space in Appendix and we show additional samples in Appendices M
We also compare guidance using two-stage upsampling stack. Nichol and Dhariwal
4 and
Saharia et al. 60 train two-stage diffusion models by combining a low-resolution diffusion model with a corresponding upsampling diffusion model. In this approach, the upsampling model is trained to upsample images from the training set, and conditions on low-resolution images that are concatenated channel-wist to the model input using simple interpolation (e.g. bilinear). During sampling. the low-resolution model produces sample, and then the upsampling model is conditioned on this sample. This greatly improves FID on ImageNet 256x 256, but does not reach the same performance as state-of-the-an models like BigGAN-deep
40
60
as seen in Table
In Table we show that guidance and upsampling improve sample quality along different axes. We use the upsampling stack from Nichol and Dhariwal
49 combined with our architecture improve- ments, which we refer as ADM-U. While upsampling improves precision while keeping a high recall, guidance provides knob to trade off diversity for much higher precision. We achieve the best FIDs by using guidance lower resolution before upsampling to higher resolution, indicating that these approaches complement one another.
6 Related Work
Score based generative models were introduced by Song and Ermon 66 as a way modeling a data distribution using its gradients, and then sampling using Langevin dynamics 73 Ho et al. [3] found connection between this method and diffusion models [33) and achieved excellent sample quality by leveraging this connection. After this breakthrough work, many works followed up with
7

---

Figure 4:
Samples from BigGAN-deep with truncation 1.0 (FID 6.95, left) vs samples from our diffusion model with guidance (FID 4.59, middle) and samples from the training set (right).
Table 5: Comparing our single, upsampling and classifier guided models. The upsamplers are 64-2256 and 128-512. When combining guidance with upsampling, we only guide the lower resolution model. All models are sampled using 250 sampling steps.
Model
FID
sFID
IS
Prec
Rec
Model
FID
sFID
Prec
Rec
ImageNet 256 256
ImageNet 512x
ADM
10.94
6.02
100.98
0.69
0.63
ADM
23.24
10.19
58.06
0.73
0.60
ADM, ADM-U
7.49
5.13
127.49
0.72
0.63
ADM. ADM-U
9.96
5.62
121.78
0.75
0.64
ADM-G
4.59
5.25
186.70
0.82
0.52
ADM-G
7.72
6.57
172.71
0.87
0.42
ADM-G, ADM-U
3.94
6.14
215.84
0.83
0.53
ADM-G, ADM-U
3.85
5.86
221.72
0.84 0.53
more promising results: Kong et al. 36 and Chen et al. ||| demonstrated that diffusion models
work well for audio: Joltteuu-Martinauu al. 3> found that GAN-like setup could improve samples from these models; Song et al. 67 explored ways to leverage techniques from stochastic
differential equations improve sample quality obtained by score-based models; [Song et proposed methods to improve sampling speed; Nichol and Dhariwal 660 demonstrated promising results on the difficult ImageNet generation task
and Nichol and Dhariwal
40
40
and Saharia al.
using upsampling diffusion models. Also related to diffusion models, and following the work of
Sohl-Dickstein et al.
63 Goyal al. 277 described a technique for learning a model with learned
iterative generation steps, and found that it could achieve good image samples when trained with
likelihood objective.
One missing element from previous work on diffusion models is a way to trade off diversity for fidelity. Other generative techniques provide natural levers for this trade-off. [rock al. X introduced the truncation trick for GANs, wherein the latent vector is sampled from truncated normal distribution. They found that increasing truncation naturally led to a decrease in diversity but an increase in fidelity. More recently, Razavi et al. 5 proposed to use classifier rejection sampling to filter out bad samples from an autoregressive likelihood- based model, and found that this technique improved FID. DeVries et al. [6 found that filtering out low-density regions of the training set improves GAN training performance. Most likelihood-baat based models also allow for now-eemper sampling (,, which provides a natural way to emphasize modes of the data distribution (see Appendix[]]

---

Other tikelihood-basee models have been shown to produce 71 and VQ- VQ-VAE-2 [57] are autoregressive models trained on top of quamtized latent codes, greatly reducing the computational resources required to train these models on large images. These models produce diverse and high quality images, but still fall short of GANs without expensive rejection sampling and special metrics to compensate for blurriness. DCTransformer [X] is a related method
image samples. VQ-VAE
which relies on a more intelligent compression scheme. VAEs are another promising class of likelihood-based models, and recent methods such as NVAE [6 and VDVAE 12 have successfully been applied to difficult image generation domains, Energy-based models are another class of 13
tikeihhodd-basee models with rich history [I] challenging, and Xie et al. images from these models. Du and Mordatch T9
Sampling from the EBM distribution is 76 demonstrate that Langevin dynamics can be used to sample coherent further improve upon this approach, obtaining
30
high quality images. More recently, Gaan et al. [24 incorporate diffusion steps into an energy-based model, and find that doing improves image samples from these models.
Other works have controlled generative models with a pre-trained classifier. For example, an emerging body of work 2 53 2 aims to optimize GAN latent spaces for text prompts using pre-trained CLIP [55] models. More similar to our work, [Song et al. [7 uses a classifier to generate class-conditional CIFAR-10 images with diffusion model. In some cases, classifiers can act as stand-alone generative models. For example, Santurkar et
al. 62 demonstrate that robust image classifier can be used as a stand-alone generative model, and Grathwohl et al. [2]] train a model which is jointly classifier and an energy-based model.
Limitations and Future Work
While we believe diffusion models are extremely promising direction for generative modeling, they are still slower than GANs at sampling time due to the use of multiple denoising steps (and therefore forward passes). Since our diffusion models are also larger than the competing GAN generators, each forward pass takes anywhere from 5-20 times longer too. A promising direction to reduce this latency gap is Luhman and Luhman 43), who explore way to distill the DDIM sampling process into a single step model. The samples from the single step model are not yet competitive with GANs, but are much better than previous single-step likelihood- -aaeeood--aaed models. Future work in this direction might be able to completely close the sampling speed gap between diffusion models and GANs without sacrificing image quality.
Unlike GANs, Flows, and VAEs, diffusion models do not learn an explicit latent representation. While DDIM provides way to encode images into implicit latent space, it is unclear how semantically meaningful this latent representation is compared to those of other model classes. This could make difficult to use diffusion models for representation learning or image editing applications.
The effectiveness of classifier guidance demonstrates that we can obtain powerful generative models from the gradients of classification function. This could be used to condition an image generator with a text caption using a noisy version of CLIP (55, similar to recent methods that guide GANs using text prompts 23
5 2 Our proposed classifier guidance technique is currently limited to labeled datasets. the future, our method could be extended to unlabeled data by clustering samples to produce synthetic labels 42 or by training discriminative models to use for guidance. This also suggests that large unlabeled datasets could be leveraged the future to pre-train powerful diffusion models that can later be improved by using classifier with desirable properties.
8 Societal Impact
Our proposed technique makes generative models more accessible in terms of compute costs, es- pecially because new classifiers can be trained and used on top of existing high-qualiny diffusion models. While we believe this is generally a benefit of these models, it could also have negative societal implications. For example, cheaper generative models could enable bad actors to generate fake news, propaganda images, or doctored photos. Additionally, the wide-spread deployment of these models could displace jobs in art, graphic design, animation, and photography. One could imagine, however, that democratizing generative models could also have positive impacts in the long run, creating new types of jobs such as generative photo editing. Intentionally deceitful generated

---

images are a more direct concern, and detecting and mitigating propaganda and fake news based on generative models an ongoing area of research 3 [
Conclusion
We have shown that diffusion models, class of likelihood-based models with stationary training objective, can obtain better sample quality than state-of-the-art GANs. Our improved architecture is sufficient to achieve this unconditional image generation tasks, and our classifier guidance technique allows us to do so on class-conditional tasks. In the latter case, we find that the scale of the classifier gradients can be adjusted to trade off diversity for fidelity. These guided diffusion models can reduce the sampling time gap between GANs and diffusion models, although diffusion models still require multiple forward passes during sampling. Finally, by combining guidance with upsampling, we can further improve sample quality on high-resolution conditional image synthesis.
References
[1] David Ackley, Geoffrey Hinton, and Terrence Sejnowski. learning algorithm for boltzmann machines. Cognitive science, 911):147-16, 1985.
[2] Adverb. The big sleep. 13510380530334064663 2021.
[3] Shruti Agarwal, Hany Farid, Yuming Gu, Mingming He, Koki Nagano, and Hao Li. Protecting world leaders against deep fakes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
[4] Hadeer Ahmed, Issa Traore, and Sherif Saad. Detection of online fake news using n-gram analysis and machine learning techniques. pages 127-138, 10 2017. ISBN 978-3-319-69154-1 doi:
[5] Hadeer Ahmed, Issa Traore, and Sherif Saad. Detecting opinion spams and fake news using text classification, Security and Privacy, e() 2018. doi: https://doi.rrg/100000////3/ wiley. com/ doi abs/ /0..//0.. 11022/00..0000pp......
URL
https://oalinelibrrree
[6] Shane Barratt and Rishi Sharma. A note on the inception score. arXiv: axiiv 2018.
[7] Andrew Brock, Theodore Lim, J. M. Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. 0009. 2016.
[8] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv: 1809://009.709000 2018.
[9] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv:2005. /4165 2020.
[10] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703. PMLR, 2020.
[11] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arr arXin:2009.00713 2020.
[12] Rewon Child, Very deep vaes generalize autoregressive models and can outperform them on images. axXiv22011.10630 2021.
[13] Peter Dayan, Geoffrey E Hinton, Radford Neal, and Richard Zemel. The helmholtz machine. Neural computation, 7(5):889-904 1995.
10

---

[14] Harm de Vries, Florian Strub, Jérémie Mary. Hugo Larochelle, Olivier Pietquin, and Aaron Courville. Modulating early visual processing by language. (rXiw.7707.00683 2017.
[15] DeepMind. Biggan-deep 128x on tensorflow hub. https://ffhubb.de//ddepmm biggan-deeps 128/1 2018.
[16] Terrance DeVriss, Michal Drozdzal, and Graham W. Taylor. Instance selection for gans. arXiv:2007.15255 2020.
[17] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: generative model for music. arXh::2005.00347 2020.
[18] Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning.
a:Xiv:1907.02544 2019.
[19] Yilun Du and Igor Mordatch. Implicit generation and generalization in energy- based models. arXiv: 096899903088999
2019.
[20] Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. (rXivv:610.07629 2017.
[21] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. (rXiv:2012.098811 2020.
[22] Jeffrey De Fauw, Sander Dieleman, and Karen Simonyan. Hierarchical autoregressive image models with auxiliary decoders. arXiv::1903.000 1903.04933 2019.
[23] Federico A. Galatolo, Mario G. C. A. Cimino, and Gigliola Vaglini. Generating images from caption and vice versa via clip-guided generative latent space search. (rXiv:2102.01645 2021.
[24] Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P. Kingma. Learning energy- based models by diffusion recovery likelihood. krXiv:20/2.08125) 2020.
[25] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. 2014.
[26] Google. Cloud tpus. https://clout http://loood..gglee..ooo wwwg.o.ooooooooooo com/tpu/ 2018.
[27] Anirudh Goyal, Nan Rosemary Ke, Surya Ganguli, and Yoshua Bengio. Variational walkback:
Learning transition operator as stochastic recurrent net. arXiv:1711.00228) 2017.
[28] Will Grathwohl, Kuan-Chich Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv:1972.03263 2019.
[29] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems 30 (NIPS 20/7), 2017.
[30] Geoffrey Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
[31] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. 200iv:2006. /239 2020.
[32] Alexia Jolicoeur-Martnnauu, Rémi Piché-Taillen Taillee,, Rémi Tachet des Combes, and Ioan- nis Mitliagkas. Adversarial score matching and improved sampling for image generation. 200902209 2020.
[33] Tero Karras, Samuli Laine, and Timo Aila. A style- based generator architecture for generative adversarial networks. arXivvarXiv: /8/2.04948 2019.
[34] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. arXiv: 2019.

---

[35] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arxiv: 1412.6980 2014.
[36] Zhifeng Kong. Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. (rX1vv2009.0976] 2020.
[37] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-10 (Canadian Institute for Advanced Research), 2009. http:/tt:///www. www.ww.wwww...to.owooooo.t.o.... edu/ www/ekrii/eaeaa "kriz/cffar html
[38] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. (7Xiv:9904.0699) 2019.
[39] Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid. Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. arXi:: 2016.
[40] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learnin face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[41] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization arXiv: 2017.
[42] Mario Lucic, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, and Sylvain Gelly. High-fidelity image generation with fewer labels. laXXi:1103.00221 2019.
[43] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. ( 20Ii::2000.02388 02388 2021.
[44] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. (rr\vv7710.037400 2017.
[45] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv:1411.174 2014.
[46] Takeru Miyato and Masanori Koyama. cgans with projection discriminator, arx: 1802.000 2018.
[47] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXi::1802.05599 2018
[48] Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W. Battaglia. Generating images with sparse representations. UrXiv:2103.03841 2021.
[49] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. 4:X1v2210..09672 2021.
[50] NVIDIA. Stylegan2. httpps///ggthhbb..mm///aa///////
2019.
[51] Gaurav Parmar, Richard Zhang, and Jun- Yan Zhu, On buggy resizing libraries and surprising subtleties in fid calculation. 1rXiv:2104.1122 2021.
[52] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, performeforeeece deep learning library. 17xiv:1972.01703 2019.
[53] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. arXiv:2103.17249 2021.
[54] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. arX1v:1009.07871 2017.
[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. 2000000003 2021.
12

---

[56] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Nya Sutskever. Zero-shot text-to-image generation. (rXIv:2102./2092) 2021.
[57] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. vrXiv:1906.00444/ 2019.
[58] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. (rXiv:/505.04597 2015.
[59] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. arXiv:1409.003. 1409.0555 2014.
[60] Chitwan Saharia, Jonathan Ho. William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXxvvarXX1::2104007666 2021.
[61] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. arrivv 1o06:00060099 2016.
[62] Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Image synthesis with a single (robust) classifier. 4rXiv:1906.0945 2019.
[63] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. 1003 2015.
[64] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. 200.022000.02502 2020.
[65] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. arXiv:2006. 09017 2020.
[66] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution.
2020.
[67] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic diffcrential equations. arXiv: 2020.
[68] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. (512vv15/2.000507 2015.
[69] Arash Vahdat and Jan Kautz. a0Xiv:2007.03888 2020.
Nvae:
A deep hierarchical variational autoencoder.
[70] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. 16X9::6609903499 2016.
[71] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning, arXiv:1711.00937 2017.
[72] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv:1706.03762 2017.
[73] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-/I), pages 681-688. Citeseer, 2011.
[74] Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy Lillicrap. Logan: Latent optimisation for generative adversarial networks. arXiv: 2019.
[75] Yuxin Wu and Kaiming He. Group normalization. 0rXiv:1803.08444 2018.
(76] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Ying Nian Wu. theory of generative convnet. arXiv:1602.03260 2016.
13

---

[77] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. ar0iv:/506.0033 1506.033655 2015.
[78] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. arx1v:1612.03242 2016.
[79] Ligeng Zhu. httpss///ttta..oooo//aaaay////or 2018.
14

---

Checklist
Did you include the license to the code and datasets? [No] We were unable find the official license for either ImageNet or LSUN. We do include a license for our released source code.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section g
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Appendix
(b) Did you include complete proofs of all theoretical results? [Yes] We provide detailed derivations in Appendix D
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] include the source code and instructions for running it.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix K
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] Each experiment requires several hundred GPU days to run, so it is quite costly to compute error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix B
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Our codebase
references the codebase it is forked from 20 31
We also cite PyTTrrch [52] which
we use throughout our experiments.
(b) Did you mention the license of the assets? [Yes] A license file is included with the code release.
(c) Did you include any assets either in the supplemental material URL? [Yes]
The code is included in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "OSt7t08VtYB", "reviewer": "Reviewer_yLx5", "summary": "### Summary\nThe paper aims to improve diffusion models for image generation. The paper proposes new network architecture and classifier guidance. The paper shows experimental results for unconditional and conditional image generation on several tasks with better image quality.\n\n### Strengths\n- The proposed new architecture and classifier guidance improves diffusion models, with ablation results well presented.\n- The paper shows better image quality for synthesized samples on LSUN (bedroom, horse, and cat) and ImageNet (64x64, 128x128, 256x256, and 512x512) than StyleGAN and BigGAN.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper, though with better synthesized images than existing works, has limited technical novelty. A major part of improvement comes from the manual design of network architecture.\n- The diffusion model still runs slower than GAN at sampling time. I wonder if the authors could provide a detailed latency comparison between GAN and diffusion models.\n- Have authors used any data augmentation during training? If not, would the data augmentation methods, such as StyleGAN2 with adaptive discriminator augmentation (ADA), help the training of diffusion models? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "LhL80pl7AM", "reviewer": "Reviewer_NtJ2", "summary": "This paper introduces a variety of architecture changes based on Ho et al. proposed UNet architecture for diffusion models, and shows that diffusion models can achieve better image generation quality than the state-of-the-art generative models, including StyelGAN2. Besides, the classifier guidance, which needs to train a time-variant classifier, is proposed to further improve the sample quality in the conditional image synthesis setting. In particular, the classifier guidance can trade off diversity for fidelity by tuning the classifier gradient strength. Experiments show its better image generation quality than state-of-the-art conditional generative models, such as Big-GANs.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ This paper is well-written, the proposed improvements are easy to follow and well-motivated, and the experiment settings are clearly stated.\n+ A combination of simple architecture improvements achieves state-of-the-art image generation performance when evaluating more complex datasets, such as LSUN, ImageNet 64/128/256/512. This result shows the power of diffusion models in image generation while avoiding the drawbacks of GAN-based models, which I think could add value to the generative modeling community.\n+ A new conditional generation method for diffusion models, called classifier guidance, achieves better image quality by trading off the image diversity, and it can be applied to a pre-trained unconditional diffusion model. \n+ The comparison with other baselines has been well evaluated on extensive datasets together with good ablation studies and good analysis of experimental results. \n\n\nWeaknesses:\n- When comparing with StyleGAN2, only LSUN datasets with resolution 256x256 are considered. As FFHQ is a more commonly used dataset in StyleGAN2, I wonder how the proposed ADM compares to StyleGAN2 on FFHQ of resolution 1024x1024? \n- Since the proposed method is mainly based on diffusion sampling [59] and DDIM sampling [60], I wonder how the architecture improvements apply to the SDE version of score-based models [63] and the corresponding conditional SDEs? \n- The introduction mentions that diffusion models have better “scalability” than GANs. What does “scalability” mean?\n\n\n================\n\nPost-rebuttal update:\n\nI thank the authors for clarifying my concerns. I have also read the authors' rebuttals and other reviews. This is a good paper where the architecture improvements and good empirical results will escalate the wider use of diffusion models. Thus, I recommend an acceptance. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors  have adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "EJcdXc-m8jI", "reviewer": "Reviewer_jJSL", "summary": "This paper builds directly on very recent work in diffusion modeling and investigates architectural design choices. The authors adapt several best practices from other lines of work, retooling them and enhancing them to fit the needs of diffusion models where appropriate, such as by adding timestep-conditional affine transforms to the normalization layers similar to class-conditional FiLM-like layers. They also study and modify the classifier guidance algorithms to improve class-conditional generation, potentially from otherwise unconditional models. These design choices are thoroughly ablated, and results are presented on several benchmark datasets, and compared against up-to-date baselines.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a solid paper, with well-reasoned and explicated design choices supported by thorough ablations and strong results. Aside from the headline result, nothing in here is especially “surprising” per se, but the core of the work--examining aspects of the model and training process in detail, identifying elements in need of improvement and identifying techniques to improve them, and demonstrating how far these improvements can go--is undeniably valuable, particularly in the context of the burgeoning interest in this class of generative models. The paper is well presented and easy to follow, and to the best of this reviewer’s knowledge all of the immediately relevant work is appropriately covered.\n\nMy sole complaint with this paper is a nit: the choice of title. Where the body of the paper is a solid piece of work, I think that it was a poor decision to choose a title that (true as it may be) sounds like a boast about one’s place on the high scores. What happens when this paper’s results are beaten--should that paper say “Model X beats Diffusion Models on Image Synthesis?” The title of a paper should be descriptive of the contribution, and highlighting the high score rather than the solid work that underlies this papers feels very much like the authors are playing to social media virality, or to some petty dispute between camps of people who hold irrationally strong opinions about one method or another for whatever reason. To me, this is made worse by the fact that it is *utterly unnecessary*: the results are very strong, the samples compelling, the work solid. A name like, e.g. “Architectural Improvements for Highly Performant Diffusion Models” would still get the message across (high performance) without falling prey to this. Additionally, “...Beats GANS on Image Synthesis” strikes me as grammatically subpar (I would think “at image synthesis” or “on the task of image synthesis” would make more sense). I would encourage the authors to, in the future, bear in mind that when one writes a paper which has the trappings of seminality, one has the chance to set an example for the rest of the field, and that great papers receive a mountainous degree of extra scrutiny on even the slightest of details.\n\nApologies for writing over 250 words on such a particular and utterly trivial nit (I don’t even feel especially strongly about it, but I think the authors can do better). This nit does not play into my evaluation of the paper itself for the purposes of this review: I rank it at an 8--obvious accept, possible contender for a highlight.\n\n\nNotes:\n\n-The manuscript references the U-net architecture in the context of diffusion models, but is missing the original U-net reference (Ronneberger et al). While this reference is certainly not hurting for citations, I think it’s still proper to include it, especially as it shouldn’t take up much space in the manuscript.\n\n-The AdaGN layer is a sensible idea that I would not be surprised to see become standard practice for all diffusion models. The authors may wish to highlight this contribution (arguably incremental though it may be) in the introduction or abstract rather than grouping it in with the other architectural improvement.\n\nMinor:\nTypos such as “sqaured.” Please spellcheck your paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "A31OeOxm0wm", "reviewer": "Reviewer_i9rh", "summary": "The authors show through extensive experiments that diffusion models can outperform GANs by a substantial margin in image synthesis. The authors make the following contributions:\ni) they propose architectural improvements to the standard UNet architecture, used in diffusion models.\nii) they devise classifier guidance as a way to trade diversity for fidelity and achieve improved scores\niii)  they show that combining classifier guidance with upsampling further improves sample quality.\niv) the guided models require fewer sampling steps.\n", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an excellent paper with very important experimental contributions. I believe that this paper will be of similar importance for diffusion models as StyleGAN or BigGAN papers have been for GANs.\n\nPros:\n\n- The paper (and the Appendix) is extremely well-written. Very clear and very easy to follow. The figures are very well chosen and very helpful for the paper understanding.\n- I think that the idea of using a classifier to trade diversity for fidelity is very novel. \n- The authors show the superiority of their approach over GANs through a very extensive experimental evaluation. The improvements are quite remarkable.\n- It is impressive that using the same compute the authors manage to achieve better results than StyleGAN. The discussion in the Appendix is quite interesting.\n- The proposed architecture might be the canonical thing to try from now on since it seems to offer more than 3 points FID improvement.\n- I like that the gradients from the classifier are re-weighted using the covariance matrix. This comes from an interesting theoretical analysis in the Appendix.\n\nCons:\n- I think the sentence: \"these guided diffusion models can reduce the sampling time gap between GANs and diffusion models\" (L 298-299) is not well justified in the paper. Where does the speedup come from? How much exactly is this speedup? I did not see any explicit discussion in the main paper. If the statement is true, this is a major selling point of the guidance method and I would like to see a more in-depth discussion.\n- Generating images is cute, but I would definitely like to see some applications on inverse problems (i.e. inpainting, denoising, etc). This is a very important research area and I want to see if the classifier guidance could actually be harmful to solving inverse problems.\n- Unlike GANs, trading coverage for fidelity requires training a new network (the classifier) for the proposed method. This is I think an important disadvantage of the proposed technique. I saw in the Appendix that the authors tried to achieve a similar thing without re-training using variations of temperature scaling. These attempts failed but I still wonder whether some other attempt could offer control of the trade-off between fidelity and diversity without training separate networks.\n- The main innovation, classifier guidance, is only useful when you have labels in the first place. This is acknowledged in the paper, which I think is good, however, when I read the paper's title I was kind of disappointed to see that the improvements in the unconditional generation only come from better architecture.\n- Paragraph starting in L174: it is unclear to me how the conditional models are conditioned. Is it an architectural choice (like adding class embeddings to the input?) I would like a clarification on that.\n\n\n================\n\nPost-rebuttal update:\n\nI thank the authors for their response. I read everything post-rebuttal and I maintain my original evaluation. In my opinion, this is a very good paper and I recommend clear acceptance.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed the limitations and the potential negative societal impact of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously agree that the paper should be accepted. The paper presents state-of-the-art FID scores on the task of class-conditional image synthesis on ImageNet, outperforming BigGAN. The paper presents the idea of classifier guidance useful for score-based and diffusion models and ablates various aspects of diffusion models. I agree with two of the reviewers that the title of the paper is not well chosen. Unfortunately the authors did not comment on their choice of title, but I encourage them to change the title. Some discussion of possible interaction between FID and classifier guidance can be helpful. I recommend accepting the paper as a spotlight.
Decision: Accept (Spotlight)