Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The authors show through extensive experiments that diffusion models can outperform GANs by a substantial margin in image synthesis. The authors make the following contributions:
i) they propose architectural improvements to the standard UNet architecture, used in diffusion models.
ii) they devise classifier guidance as a way to trade diversity for fidelity and achieve improved scores
iii)  they show that combining classifier guidance with upsampling further improves sample quality.
iv) the guided models require fewer sampling steps.

main_review: This is an excellent paper with very important experimental contributions. I believe that this paper will be of similar importance for diffusion models as StyleGAN or BigGAN papers have been for GANs.

Pros:

- The paper (and the Appendix) is extremely well-written. Very clear and very easy to follow. The figures are very well chosen and very helpful for the paper understanding.
- I think that the idea of using a classifier to trade diversity for fidelity is very novel. 
- The authors show the superiority of their approach over GANs through a very extensive experimental evaluation. The improvements are quite remarkable.
- It is impressive that using the same compute the authors manage to achieve better results than StyleGAN. The discussion in the Appendix is quite interesting.
- The proposed architecture might be the canonical thing to try from now on since it seems to offer more than 3 points FID improvement.
- I like that the gradients from the classifier are re-weighted using the covariance matrix. This comes from an interesting theoretical analysis in the Appendix.

Cons:
- I think the sentence: "these guided diffusion models can reduce the sampling time gap between GANs and diffusion models" (L 298-299) is not well justified in the paper. Where does the speedup come from? How much exactly is this speedup? I did not see any explicit discussion in the main paper. If the statement is true, this is a major selling point of the guidance method and I would like to see a more in-depth discussion.
- Generating images is cute, but I would definitely like to see some applications on inverse problems (i.e. inpainting, denoising, etc). This is a very important research area and I want to see if the classifier guidance could actually be harmful to solving inverse problems.
- Unlike GANs, trading coverage for fidelity requires training a new network (the classifier) for the proposed method. This is I think an important disadvantage of the proposed technique. I saw in the Appendix that the authors tried to achieve a similar thing without re-training using variations of temperature scaling. These attempts failed but I still wonder whether some other attempt could offer control of the trade-off between fidelity and diversity without training separate networks.
- The main innovation, classifier guidance, is only useful when you have labels in the first place. This is acknowledged in the paper, which I think is good, however, when I read the paper's title I was kind of disappointed to see that the improvements in the unconditional generation only come from better architecture.
- Paragraph starting in L174: it is unclear to me how the conditional models are conditioned. Is it an architectural choice (like adding class embeddings to the input?) I would like a clarification on that.


================

Post-rebuttal update:

I thank the authors for their response. I read everything post-rebuttal and I maintain my original evaluation. In my opinion, this is a very good paper and I recommend clear acceptance.

limitations_and_societal_impact: The authors addressed the limitations and the potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper builds directly on very recent work in diffusion modeling and investigates architectural design choices. The authors adapt several best practices from other lines of work, retooling them and enhancing them to fit the needs of diffusion models where appropriate, such as by adding timestep-conditional affine transforms to the normalization layers similar to class-conditional FiLM-like layers. They also study and modify the classifier guidance algorithms to improve class-conditional generation, potentially from otherwise unconditional models. These design choices are thoroughly ablated, and results are presented on several benchmark datasets, and compared against up-to-date baselines.
main_review: This is a solid paper, with well-reasoned and explicated design choices supported by thorough ablations and strong results. Aside from the headline result, nothing in here is especially “surprising” per se, but the core of the work--examining aspects of the model and training process in detail, identifying elements in need of improvement and identifying techniques to improve them, and demonstrating how far these improvements can go--is undeniably valuable, particularly in the context of the burgeoning interest in this class of generative models. The paper is well presented and easy to follow, and to the best of this reviewer’s knowledge all of the immediately relevant work is appropriately covered.

My sole complaint with this paper is a nit: the choice of title. Where the body of the paper is a solid piece of work, I think that it was a poor decision to choose a title that (true as it may be) sounds like a boast about one’s place on the high scores. What happens when this paper’s results are beaten--should that paper say “Model X beats Diffusion Models on Image Synthesis?” The title of a paper should be descriptive of the contribution, and highlighting the high score rather than the solid work that underlies this papers feels very much like the authors are playing to social media virality, or to some petty dispute between camps of people who hold irrationally strong opinions about one method or another for whatever reason. To me, this is made worse by the fact that it is *utterly unnecessary*: the results are very strong, the samples compelling, the work solid. A name like, e.g. “Architectural Improvements for Highly Performant Diffusion Models” would still get the message across (high performance) without falling prey to this. Additionally, “...Beats GANS on Image Synthesis” strikes me as grammatically subpar (I would think “at image synthesis” or “on the task of image synthesis” would make more sense). I would encourage the authors to, in the future, bear in mind that when one writes a paper which has the trappings of seminality, one has the chance to set an example for the rest of the field, and that great papers receive a mountainous degree of extra scrutiny on even the slightest of details.

Apologies for writing over 250 words on such a particular and utterly trivial nit (I don’t even feel especially strongly about it, but I think the authors can do better). This nit does not play into my evaluation of the paper itself for the purposes of this review: I rank it at an 8--obvious accept, possible contender for a highlight.


Notes:

-The manuscript references the U-net architecture in the context of diffusion models, but is missing the original U-net reference (Ronneberger et al). While this reference is certainly not hurting for citations, I think it’s still proper to include it, especially as it shouldn’t take up much space in the manuscript.

-The AdaGN layer is a sensible idea that I would not be surprised to see become standard practice for all diffusion models. The authors may wish to highlight this contribution (arguably incremental though it may be) in the introduction or abstract rather than grouping it in with the other architectural improvement.

Minor:
Typos such as “sqaured.” Please spellcheck your paper.

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a variety of architecture changes based on Ho et al. proposed UNet architecture for diffusion models, and shows that diffusion models can achieve better image generation quality than the state-of-the-art generative models, including StyelGAN2. Besides, the classifier guidance, which needs to train a time-variant classifier, is proposed to further improve the sample quality in the conditional image synthesis setting. In particular, the classifier guidance can trade off diversity for fidelity by tuning the classifier gradient strength. Experiments show its better image generation quality than state-of-the-art conditional generative models, such as Big-GANs.

main_review: Strengths:
+ This paper is well-written, the proposed improvements are easy to follow and well-motivated, and the experiment settings are clearly stated.
+ A combination of simple architecture improvements achieves state-of-the-art image generation performance when evaluating more complex datasets, such as LSUN, ImageNet 64/128/256/512. This result shows the power of diffusion models in image generation while avoiding the drawbacks of GAN-based models, which I think could add value to the generative modeling community.
+ A new conditional generation method for diffusion models, called classifier guidance, achieves better image quality by trading off the image diversity, and it can be applied to a pre-trained unconditional diffusion model. 
+ The comparison with other baselines has been well evaluated on extensive datasets together with good ablation studies and good analysis of experimental results. 


Weaknesses:
- When comparing with StyleGAN2, only LSUN datasets with resolution 256x256 are considered. As FFHQ is a more commonly used dataset in StyleGAN2, I wonder how the proposed ADM compares to StyleGAN2 on FFHQ of resolution 1024x1024? 
- Since the proposed method is mainly based on diffusion sampling [59] and DDIM sampling [60], I wonder how the architecture improvements apply to the SDE version of score-based models [63] and the corresponding conditional SDEs? 
- The introduction mentions that diffusion models have better “scalability” than GANs. What does “scalability” mean?


================

Post-rebuttal update:

I thank the authors for clarifying my concerns. I have also read the authors' rebuttals and other reviews. This is a good paper where the architecture improvements and good empirical results will escalate the wider use of diffusion models. Thus, I recommend an acceptance. 
limitations_and_societal_impact: The authors  have adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: ### Summary
The paper aims to improve diffusion models for image generation. The paper proposes new network architecture and classifier guidance. The paper shows experimental results for unconditional and conditional image generation on several tasks with better image quality.

### Strengths
- The proposed new architecture and classifier guidance improves diffusion models, with ablation results well presented.
- The paper shows better image quality for synthesized samples on LSUN (bedroom, horse, and cat) and ImageNet (64x64, 128x128, 256x256, and 512x512) than StyleGAN and BigGAN.

main_review: - The paper, though with better synthesized images than existing works, has limited technical novelty. A major part of improvement comes from the manual design of network architecture.
- The diffusion model still runs slower than GAN at sampling time. I wonder if the authors could provide a detailed latency comparison between GAN and diffusion models.
- Have authors used any data augmentation during training? If not, would the data augmentation methods, such as StyleGAN2 with adaptive discriminator augmentation (ADA), help the training of diffusion models? 
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The reviewers unanimously agree that the paper should be accepted. The paper presents state-of-the-art FID scores on the task of class-conditional image synthesis on ImageNet, outperforming BigGAN. The paper presents the idea of classifier guidance useful for score-based and diffusion models and ablates various aspects of diffusion models. I agree with two of the reviewers that the title of the paper is not well chosen. Unfortunately the authors did not comment on their choice of title, but I encourage them to change the title. Some discussion of possible interaction between FID and classifier guidance can be helpful. I recommend accepting the paper as a spotlight.
Decision: Accept (Spotlight)