Title: Diffusion Models Beat GANs on Image Synthesis
Authors: Prafulla Dhariwal
Alexander Quinn Nichol
Venue: NeurIPS 2021
Keywords: generative models
diffusion models
score-based models
denoising diffusion probabilistic models
image generation
neural networks
attention

Abstract: We show that diffusion models can achieve image sample quality superior to the current state-of-the-art generative models. We achieve this on unconditional image synthesis by finding a better architecture through a series of ablations. For conditional image synthesis, we further improve sample quality with classifier guidance: a simple, compute-efficient method for trading off diversity for fidelity using gradients from a classifier. We achieve an FID of 2.97 on ImageNet 128$\times$128, 4.59 on ImageNet 256$\times$256, and 7.72 on ImageNet 512$\times$512, and we match BigGAN-deep even with as few as 25 forward passes per sample, all while maintaining better coverage of the distribution. Finally, we find that classifier guidance combines well with upsampling diffusion models, further improving FID to 3.94 on ImageNet 256$\times$256 and 3.85 on ImageNet 512$\times$512.

TL;DR: We achieve state-of-the-art image generation on ImageNet and several LSUN classes with diffusion models.

Review discussion:
- The paper, though with better synthesized images than existing works, has limited technical novelty. A major part of improvement comes from the manual design of network architecture.
- The diffusion model still runs slower than GAN at sampling time. I wonder if the authors could provide a detailed latency comparison between GAN and diffusion models.
- Have authors used any data augmentation during training? If not, would the data augmentation methods, such as StyleGAN2 with adaptive discriminator augmentation (ADA), help the training of diffusion models?
No
We are grateful for your thoughtful feedback!
- Regarding novelty, it is true that most of the architectural improvements we ablate existed before this paper was written, and we only show that diffusion models can benefit from these.
However, we actually do something quite novel with classifier guidance, which is a major source of improvement on the class-conditional tasks. We are not aware of any previous paper on diffusion or score matching that has applied classifier guidance to models which are already class-conditional, nor have any papers explored rescaling the classifier gradients. These ideas are essential to our results, and it was not obvious before writing this paper that using classifier guidance in this way had any reason to boost quality or allow us to trade-off diversity for fidelity.
- Our models are an order-of-magnitude larger than the competing GANs, and are 8-20x slower per forward pass as a result.
Additionally, our models require at least 25 forward passes per sample for good results. This amounts to roughly a 300x slowdown per sample compared to GANs. We’ve updated the limitations section to include this.
- Similar to other literature on diffusion, we don’t use any data augmentation for diffusion training other than random flips on LSUN. It is likely that adaptive augmentation would help, although probably less so at convergence--especially for larger datasets where we don’t witness any overfitting.
Strengths:
+ This paper is well-written, the proposed improvements are easy to follow and well-motivated, and the experiment settings are clearly stated.
+ A combination of simple architecture improvements achieves state-of-the-art image generation performance when evaluating more complex datasets, such as LSUN, ImageNet 64/128/256/512. This result shows the power of diffusion models in image generation while avoiding the drawbacks of GAN-based models, which I think could add value to the generative modeling community.
+ A new conditional generation method for diffusion models, called classifier guidance, achieves better image quality by trading off the image diversity, and it can be applied to a pre-trained unconditional diffusion model.
+ The comparison with other baselines has been well evaluated on extensive datasets together with good ablation studies and good analysis of experimental results.
Weaknesses:
- When comparing with StyleGAN2, only LSUN datasets with resolution 256x256 are considered. As FFHQ is a more commonly used dataset in StyleGAN2, I wonder how the proposed ADM compares to StyleGAN2 on FFHQ of resolution 1024x1024?
- Since the proposed method is mainly based on diffusion sampling [59] and DDIM sampling [60], I wonder how the architecture improvements apply to the SDE version of score-based models [63] and the corresponding conditional SDEs?
- The introduction mentions that diffusion models have better “scalability” than GANs. What does “scalability” mean?
Post-rebuttal update:
I thank the authors for clarifying my concerns. I have also read the authors' rebuttals and other reviews. This is a good paper where the architecture improvements and good empirical results will escalate the wider use of diffusion models.
Thus, I recommend an acceptance.
No
Thank you for taking the time to read our paper and provide thoughtful feedback!
- Agreed that higher-resolution datasets are worth exploring. For this paper, 512x512 was the highest resolution we tested, and even there an upsampling stack gave better results than direct diffusion. Given this, the best 1024x1024 results will likely also use upsampling. We expect future work to touch on high-resolution synthesis more to understand why direct diffusion isn’t as good as upsampling for high resolutions.
- In the past, architecture improvements (like the UNet architecture) that helped for diffusion have also helped for score matching / SDEs, so it seems likely this trend will continue. Our guidance algorithm for DDIM is derived from the score-based interpretation of diffusion models, so it’s likely that the same trick would work for score-based models / SDEs as well.
- When we refer to scalability, we mean scaling model sizes. As shown in [46], for diffusion models FID vs model size follows a smooth and predictable trend.
When training GANs, due to the adversarial training objective, it is sometimes difficult to increase model size without hurting stability or sample quality. One example can be found in Appendix H of BigGAN, where the authors double the depth of their model (with no other changes) and find that this hurts performance.
This is a solid paper, with well-reasoned and explicated design choices supported by thorough ablations and strong results. Aside from the headline result, nothing in here is especially “surprising” per se, but the core of the work--examining aspects of the model and training process in detail, identifying elements in need of improvement and identifying techniques to improve them, and demonstrating how far these improvements can go--is undeniably valuable, particularly in the context of the burgeoning interest in this class of generative models. The paper is well presented and easy to follow, and to the best of this reviewer’s knowledge all of the immediately relevant work is appropriately covered.
My sole complaint with this paper is a nit: the choice of title. Where the body of the paper is a solid piece of work, I think that it was a poor decision to choose a title that (true as it may be) sounds like a boast about one’s place on the high scores. What happens when this paper’s results are beaten--should that paper say “Model X beats Diffusion Models on Image Synthesis? ” The title of a paper should be descriptive of the contribution, and highlighting the high score rather than the solid work that underlies this papers feels very much like the authors are playing to social media virality, or to some petty dispute between camps of people who hold irrationally strong opinions about one method or another for whatever reason.
To me, this is made worse by the fact that it is *utterly unnecessary*: the results are very strong, the samples compelling, the work solid. A name like, e. g. “Architectural Improvements for Highly Performant Diffusion Models” would still get the message across (high performance) without falling prey to this.
Additionally, “...Beats GANS on Image Synthesis” strikes me as grammatically subpar (I would think “at image synthesis” or “on the task of image synthesis” would make more sense). I would encourage the authors to, in the future, bear in mind that when one writes a paper which has the trappings of seminality, one has the chance to set an example for the rest of the field, and that great papers receive a mountainous degree of extra scrutiny on even the slightest of details.
Apologies for writing over 250 words on such a particular and utterly trivial nit (I don’t even feel especially strongly about it, but I think the authors can do better). This nit does not play into my evaluation of the paper itself for the purposes of this review: I rank it at an 8--obvious accept, possible contender for a highlight.
Notes:
-The manuscript references the U-net architecture in the context of diffusion models, but is missing the original U-net reference (Ronneberger et al).
While this reference is certainly not hurting for citations, I think it’s still proper to include it, especially as it shouldn’t take up much space in the manuscript.
-The AdaGN layer is a sensible idea that I would not be surprised to see become standard practice for all diffusion models. The authors may wish to highlight this contribution (arguably incremental though it may be) in the introduction or abstract rather than grouping it in with the other architectural improvement.
Minor:
Typos such as “sqaured.” Please spellcheck your paper.
No
We appreciate your thorough and thoughtful review! Your feedback is insightful and has helped us improve our paper.
- We have added a reference to the original UNet paper.
- Agreed that Adaptive GroupNorm is an important piece here, but we only ablated it in the context of diffusion, and it seemed reasonable to group it with other architectural improvements under one theme.
- We have fixed the “sqaured” typo and a few others
This is an excellent paper with very important experimental contributions. I believe that this paper will be of similar importance for diffusion models as StyleGAN or BigGAN papers have been for GANs.
Pros:
- The paper (and the Appendix) is extremely well-written. Very clear and very easy to follow. The figures are very well chosen and very helpful for the paper understanding.
- I think that the idea of using a classifier to trade diversity for fidelity is very novel.
- The authors show the superiority of their approach over GANs through a very extensive experimental evaluation. The improvements are quite remarkable.
- It is impressive that using the same compute the authors manage to achieve better results than StyleGAN. The discussion in the Appendix is quite interesting.
- The proposed architecture might be the canonical thing to try from now on since it seems to offer more than 3 points FID improvement.
- I like that the gradients from the classifier are re-weighted using the covariance matrix. This comes from an interesting theoretical analysis in the Appendix.
Cons:
- I think the sentence: "these guided diffusion models can reduce the sampling time gap between GANs and diffusion models" (L 298-299) is not well justified in the paper. Where does the speedup come from? How much exactly is this speedup? I did not see any explicit discussion in the main paper. If the statement is true, this is a major selling point of the guidance method and I would like to see a more in-depth discussion.
- Generating images is cute, but I would definitely like to see some applications on inverse problems (i.e. inpainting, denoising, etc). This is a very important research area and I want to see if the classifier guidance could actually be harmful to solving inverse problems.
- Unlike GANs, trading coverage for fidelity requires training a new network (the classifier) for the proposed method. This is I think an important disadvantage of the proposed technique. I saw in the Appendix that the authors tried to achieve a similar thing without re-training using variations of temperature scaling. These attempts failed but I still wonder whether some other attempt could offer control of the trade-off between fidelity and diversity without training separate networks.
- The main innovation, classifier guidance, is only useful when you have labels in the first place. This is acknowledged in the paper, which I think is good, however, when I read the paper's title I was kind of disappointed to see that the improvements in the unconditional generation only come from better architecture.
- Paragraph starting in L174: it is unclear to me how the conditional models are conditioned. Is it an architectural choice (like adding class embeddings to the input?) I would like a clarification on that.
Post-rebuttal update:
I thank the authors for their response. I read everything post-rebuttal and I maintain my original evaluation. In my opinion, this is a very good paper and I recommend clear acceptance.
No
Thanks for taking the time to review our work. We are grateful for your considerate review and thoughtful questions!
- Regarding the speedup: guidance offers a speedup by reducing the required number of sampling steps to get good samples (since sampling time scales linearly with number of steps). With guidance, our models get better samples with 25 DDIM steps than they do with hundreds of steps without guidance (see Table 4).
- Agreed that inverse problems are interesting! We don’t explore these in this paper, but hopefully guidance could be used to help with these problems in the future.
- For explicit conditioning, we feed the class label embedding into the adaptive groupnorm in the same path as the timestep embedding. L126 mentions this briefly, but we simply adopted this technique from [46].

Meta-review:
role: PC
data: comment: The reviewers unanimously agree that the paper should be accepted. The paper presents state-of-the-art FID scores on the task of class-conditional image synthesis on ImageNet, outperforming BigGAN. The paper presents the idea of classifier guidance useful for score-based and diffusion models and ablates various aspects of diffusion models. I agree with two of the reviewers that the title of the paper is not well chosen. Unfortunately the authors did not comment on their choice of title, but I encourage them to change the title. Some discussion of possible interaction between FID and classifier guidance can be helpful. I recommend accepting the paper as a spotlight.

Decision: Accept (Spotlight)