Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the algorithmic design space of diffusion models. In doing so, the authors make the following contributions:
(a) identify and characterize the degrees of freedom w.r.t. sampling, network parameterization, training
(b) propose to disentangle sampling from training and consequently, design schemes for both deterministic and stochastic sampling based on higher-order Runge Kutta solvers that significantly reduce the number of function evaluations
(c) improve training by better preconditioning of inputs, outputs, training losses and augmentation schemes
Empirically, these improvements translate to sota sample quality on standard image generation benchmarks
strengths_and_weaknesses: Strengths:

- This paper presents a timely and important contribution to an area of growing significance: deep generative modeling using diffusion models. Arguably, these models, while clearly having a lot of potential given their recent successes, are notorious to train requiring several training tricks.  
- Related to the above, the paper also stands out in contrasting existing models and algorithms --- in a way, the community can look at this paper both as an excellent survey of some major works in diffusion models while providing a generalized characterization that enables the development of newer models.
- Finally, the resulting improvements on CIFAR-10 and Imagenet are impressive -- the paper truly achieves the best of both worlds as the number of function evaluations reduce very significantly while also improving the sample quality metrics in both scenarios of deterministic and stochastic sampling. I also liked learning about some negative results and intuitions on their failures --- such as the use of stochastic samplers leading to oversaturation --  I have observed this to be a common problem in the results of some past works in diffusion models, but couldn't come across a satisfying explanation.

Weaknesses:

- While such a paper is not easy to write as it covers a bunch of distinct contributions, I found it a little hard to review constantly shuttling between the main text and the appendix. Some of the very important and interesting discussions such as those concerning the step sizes in Section 3 were delegated to the appendix, making readability slightly hard.
- At certain places, it is hard to distinguish what is prior work vs what is new --- this isn't a question re: novelty of this paper, but more on the exposition lacking discussion and details. For example, Jolicoeur-Martineau et al. (duly cited) also explore the use of higher order solvers for sampling-- what exactly is different between the current work and the one before? Is it the scheme for setting timesteps to minimize truncation error in Eq. 5 or something else?
- I think some of the claims can be placed better in context or need more empirical justification. For example, in L103-104, the authors highlight there are no dependencies between the components but does this hold broadly for all parameters in Table 1? One extreme interpretation of this could be eg, we can mix and match preconditioning schemes, i.e., skip scaling from model A (say VP), output scaling from model B (say DDIM), etc. and still train a diffusion model reasonably well. Given the brittleness of these models, I am not sure if such arbitrary mix-and-match schemes will translate to a functioning model with reasonable performance.

questions: - See last 2 points in the weaknesses para above and please let me know your thoughts
- I wasn't quite clear on why the authors choose to discuss the generalized ODE in Eq. 4 when eventually they propose to set s(t)=1 which reduces to Eq (1)
- Is there any specific reason to exclude Imagenet-64 for evaluating the effect of preconditioning in Table 2? I am particularly curious because I was hoping to contrast it with the numbers in Fig. 4c for stochastic sampling. Since we might intuitively expect that stochastic sampling could relatively be more beneficial for diverse datasets, I wonder if that intuition carries over empirically.
- Finally, while the paper does characterize the design space reasonably well, I am curious if the authors found that their final diffusion model is robust to different choices (not just their prescribed choice) of some of the design parameters in Table 1.
limitations: The authors discuss negative results in the context of stochastic sampling. Perhaps a broader discussion of negative results and context around their empirical proposals could help --- concretely, one example of such a discussion point is: which design choices are data-dependent and on what properties (eg, size, dimensionality, modality)? Such information could vastly aid practitioners looking to train and deploy these models on their custom data and modalities.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes a reformulation of continuous-time diffusion/score-based models that is more modular and easier to analyze. With this new formulation, authors carefully analyze different design choices in those models, including discretization for sampling, parameterization for the score model, stochasticity in the sampling process, noise distributions in training, and scaling schedules of inputs. Experiments demonstrate significant improvements in various settings, creating new/near SOTA results on CIFAR-10 and ImageNet-64.
strengths_and_weaknesses: This is an excellent work with many strengths:

1. Unlike previous works, authors formulate diffusion/score-based models around the probability flow ODE. This novel perspective leads to a new analysis on the role of stochasticity in sampling, showing that previous SDE-based formulations are only a special case of combining  the probability flow ODE and the Langevin diffusion SDE with different relative weights. Stochasticity only exists in the Langevin component, which effectively corrects for potential errors in solving the probability flow ODE. This interpretation motivates authors' new stochastic samplers that outperform existing ones on image datasets. I'd like to point it out that a similar reasoning was also used to form the predictor-corrector samplers in ref. [41], where the Langevin component is the "corrector", and the "predictor" can be either a probability flow ODE or a reverse-time SDE. 

2. The analysis on deterministic sampling is very convincing, backed by both insightful illustrations (Figure 3) and strong empirical improvements. I especially like the analysis on how to choose the noise schedule $\sigma(t)$ to make the probability flow ODE easy to solve. It is also a valuable contribution to show that 2nd order ODE solver (Heun's method) significantly accelerates sampling.

3. The parameterization of $D_\theta$ in equation (7) is a nice contribution, capturing the intuition that we should predict the denoised image for bigger noise levels, and predict the noise itself for smaller noise levels. The analysis on choosing $\lambda(\sigma)$, $p_{\text{train}}(\sigma)$ and others provides an effective set of hyperparameters for improved empirical performance.

There are no major weaknesses. Below are some thoughts if I have to nitpick:

1. I don't fully agree that the proposed reformulation is more modular than previous ones. It seems to me that all design choices in this work can be translated to the variational formulation of diffusion models, or the continuous-time formulation of score-based generative models. The authors' new formulation seems to facilitate the analysis of stochastic samplers the most. Other contributions can be made with existing formulations with appropriate modifications.

2. Derivations of hyperparameters are not fully principled. They are more or less based on intuitions (such as fixing the magnitudes of input and output signals), or discovered from experiments. This raises a question on whether the same design choice can perform as well for other data domains.
questions: The stochastic sampler in Algorithm 2 is different from solving equation (6) with numerical SDE solvers. Why not perform stochastic sampling with existing numerical SDE solvers? How do they fare against the sampler in Algorithm 2?
limitations: Authors discussed limitations implicitly as avenues for future work, such as the precise interaction between stochasticity and the training objective. A comprehensive discussion on negative societal impacts was also included.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors explore a wide variety of design choices made when designing diffusion models, both those which are typically given attention explicitly in prior work (e.g. the noise schedule) and ones which have been more implicitly agreed upon (e.g. the choice of of ODE solver). They highlight a set of potential sources of generalization to improve sampling - e.g., the lack of need for the sampling process to correspond to model or training details. They generalize a set of the stochastic diffusion model SDE with a single equation (Eq. 6) corresponding to moving forward and backward in time, and propose an algorithm to reduce stochastic sampling errors. In essence, prior works have explored relatively constrained subspaces of the decisions which define a particular diffusion model - this paper attempts to unify these decisions through a shared framework, allowing for new non-obvious generalizations, and then to quantitatively understand their optima. 
strengths_and_weaknesses: This paper is remarkably thorough, considering an impressively broad range of parameters and demonstrating repeatedly why such a generalized framework is useful by applying it and making nontrivial algorithmic/formal contributions. They also serve to provide an important baseline, reducing the gap between these models and those leveraging recent advances such as cascaded diffusion models, without sacrificing (as much) theoretical interpretability. The practical consequences of the suggestions made in this paper are significant, allowing for substantially more efficient sampling, and addressing what has historically been a significant limitation of diffusion models. The paper itself is excellent and beyond its practical and theoretical contributions, represents a powerful and remarkably accessible survey of diffusion model literature (excluding a few nuances like approaches consisting of multiple diffusion models).

One noteworthy detail of this paper is the repeated highlighting on decisions which, while they could be and often are made with some theoretical motivation, are in fact best left to be empirically determined (such as the relative rates of noise decay and injection over time). While it is useful to have these empirical results, and empirical results often yield useful theoretical generalizations, it is less practical to need to perform a “case-by-case … grid search” for each new design decision. In addition, FID and other Inception-based metrics have well-known limitations e.g. [1, 2, 3]. While FID is widely used to evaluate generative models, when analyzing many hyperparameters and exploring some relatively small differences in image quality, it is difficult to know whether some empirical decisions are actually improving the generated images or overfitting to FID. It would be helpful to understand/interpret the tradeoffs made by these decisions in terms of other metrics as well, such as precision and recall [4].

[1] An Improved Evaluation Framework for Generative Adversarial Networks, Liu et al 2018
[2] Pros and cons of GAN evaluation measures, Borji 2019
[3] A Note on the Inception Score, Barratt and Sharma 2018
[4] Improved Precision and Recall Metric for Assessing Generative Models, Kynkäänniemi et al 2019.
questions: What kinds of diversity/fidelity tradeoffs are implied by the various suggestions of this paper?
limitations: The limitations are reasonably well addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes to consider 1) network parameterization (preconditioning) 2) training and 3) sampling of diffusion models in separation. This allows for putting many existing works in a common framework and for a simplified investigation of the three aspects in isolation. Considering, for example, the sampling process in isolation, the authors show how to improve several pre-trained networks from the literature. Furthermore, making "optimal" choices for each of the three aspects leads to new state-of-the-art models on several image benchmark datasets. Additionally, the paper proposes to train diffusion models on augmented data, importantly conditioning the network on the augmentation process. Lastly, the paper shows that stochastic sampling outperforms deterministic sampling for "suboptimal models", whereas deterministic sampling is preferred for "well-trained models".
strengths_and_weaknesses: Strengths:
* The paper is extremely well-written and all claims are well-supported by experiments.
* The paper has a great impact on the current field of diffusion models: the independently proposed training recipes, network parameterizations (preconditioning), and sampling schemes can potentially be used in many future works.

Weaknesses:
* The paper lacks related work in certain places; to list only a few:
    * Model parameterization (preconditioning):
        * [PD] introduces v-diffusion parameterization
        * [LSGM, CLD] introduce mixed score parameterizations
    * Sampling:
        * [PNDM, DEIS] apply linear multistep methods to the Diffusion Model ODE (DEIS was just recently proposed, so missing this citation is not a big deal)
        * [LFS] learns efficient samplers
        * [PD, KD] accelerating sampling via distillation
* Otherwise, the paper has very little weaknesses; see Questions below. 

[PD] - Progressive Distillation for Fast Sampling of Diffusion Models

[LSGM] - Score-based Generative Modeling in Latent Space

[CLD] - Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

[PNDM] - Pseudo Numerical Methods for Diffusion Models on Manifolds

[DEIS] - Fast Sampling of Diffusion Models with Exponential Integrator

[LFS] - Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality

[KD] - Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed
  
questions: Main questions: 
* Figure 5(b): The figure is of very high value as it shows that stochasticity benefits/hurts suboptimal/well-trained models. I am, however, curious if the figure would look similar for more challenging datasets such as ImageNet. My suspicion is that for ImageNet a well-trained model may still be considerably different than the true data model and therefore I suspect that stochasticity would even improve a well-trained model. Could the authors comment on this?

* Network parameterization (preconditioning): The proposed network parameterization seems very similar to the v-diffusion parameterization introduced in PD. The denoiser in v-diffusion is $\alpha_t x_t - \sigma_t F_\theta(x_t, t)$, where $F_\theta$ is the neural network. Could the authors comment on connections/advantages of v-diffusion in the proposed network parameterization as well as loss function?

Extra (research) questions (for which I don't necessarily expect an answer):
* Do the authors believe that variance-learning diffusion models (as, for example, introduced in iDDPM) could also be unified with their approach?

limitations: N/A
review_title: null

Meta-review: metareview: Ratings: 8/9/8/7.
Confidence: 4/4/4/5.
Discussion among reviewers: No.

Summary: This is an excellent paper analyzing the design space of diffusion models. The paper clarifies the design space by disentangling the effects of (1) parameterization, (2) sampling, and (3) training separately. The researchers uniformily agree that the paper is well written and that the empirical results are impressive. Given the enormous interest in diffusion models in the research community, and the likely high impact of advancements in this subfield, this paper is well timed, and will probably be very well received by the NeurIPS community. 

Decision: I highly recommend to accept this paper.
confidence: Certain
Decision: Accept