Elucidating the Design Space of Diffusion-Based Generative Models
Authors: Tero Karras
Miika Aittala
Timo Aila
Samuli Laine
Venue: NeurIPS 2022 Accept (2022)
Keywords: generative modeling
denoising diffusion
image generation
score matching
differential equations

Abstract: We argue that the theory and practice of diffusion-based generative models are currently unnecessarily convoluted and seek to remedy the situation by presenting a design space that clearly separates the concrete design choices. This lets us identify several changes to both the sampling and training processes, as well as preconditioning of the score networks. Together, our improvements yield new state-of-the-art FID of 1.79 for CIFAR-10 in a class-conditional setting and 1.97 in an unconditional setting, with much faster sampling (35 network evaluations per image) than prior designs. To further demonstrate their modular nature, we show that our design changes dramatically improve both the efficiency and quality obtainable with pre-trained score networks from previous work, including improving the FID of a previously trained ImageNet-64 model from 2.07 to near-SOTA 1.55, and after re-training with our proposed improvements to a new SOTA of 1.36.

Elucidating the Design Space of Diffusion-Based Generative Models
Tero Karras
Miika Aittala NVIDIA maittala@nvidia.c com
NVIDIA ckarras@nvidia.cmm com
Timo Aila
Samuli Laine NVIDIA slaine@nvidia.com com
NVIDIA taila@nvidia.com
Abstract
We argue that the theory and practice of diffusion- generative models are currently unnecessarily convoluted and seek remedy the situation by presenting design space that clearly separates the concrete design choices. This us identify several changes to both the sampling and training processes, as well as preconditioning of the score networks. Together, our improvements yield new state-of-the-arr FID of 1.79 for CIFAR-10 in class-conditional setting and 1.97 in an unconditional setting, with much faster sampling (35 network evaluations per image) than prior designs. To further demonstrate their modular nature, show that our design changes dramatically improve both the efficiency and quality ob- tainable with pre- pre-trained the FID of a previously trained ImageNet-64 model from 2.07 to SOT-SOTA 1,55, and after re-training with our proposed improvements new SOTA of 1.36.
networks from previous work, including improving
Introduction
Diffusion frffusiio--based generative models [45] have emerged as powerful new framework for neural image
synthesis, in both unconditional [16] 36
48 and conditional [17
35 36
38
39 41
42
48
settings,
even surpassing the quality of GANs 13
in certain situations 9 They are also rapidly finding use
in other domains such audio 27 37 and video [19] generation, image segmentation 4 54 and
language translation
34 As such, there is great interest in applying these models and improving them further in terms of image/distribution quality, training cost, and generation speed.
The literature on these models dense on theory, and derivations of sampling schedule, training dynamics, noise level parameterization, etc., tend to be based directly as possible on theoretical frameworks, which ensures that the models are on a solid theoretical footing. However, this approach has danger of obscuring the available design space coupled package where no individual component can be modified without breaking the entire system.
proposed model may appear tightly
As our first contribution, we take look at the theory behind these models from practical standpoint, focusing more on the "tangible" objects and algorithms that appear in the training and sampling phases, and less on the statistical processes from which they might be derived. The goal is to obtain better insights into how these components are linked together and what degrees of freedom are available in the design of the overall system. We focus the broad class of models where neural network used to model score [22] of noise level dependent marginal distribution of the training data corrupted by Gaussian noise. Thus, our work is in the context of denoising score matching (51).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

0=00
(a) Noisy images drawn from p(z; 0)
(b) Ideal denoiser outputs D(
Figure I: Denoising score matching on CIFAR-10. (a) Images from the training set corrupted with varying levels of additive Gaussian noise. High levels of noise lead oversaturated colors; normalize the images for cleaner visualization. (b) Optimal denoising result from minimizing Eg. analytically (see Appendix B.3). With increasing noise level, the result approaches dataset mean.
Our second set of contributions concerns the sampling processes used to synthesize images using diffusion models. We identify the best-performing time discretization for sampling, apply higher- order Runge-Kutta Kutta method for the sampling process, evaluate different sampler schedules, and analyze the usefulness of stochasticity in the sampling process. The result of these improvements is significant drop in the number of sampling steps required during synthesis, and the improved sampler can be used drop-in replacement with several widely used diffusions models [36 48).
The third set of contributions focuses on the training of the score-modeling neural network. While continue to rely the commonly used network architectures (DDPM [16]. NCSN (47). provide the first principled analysis of the preconditioning of the networks' inputs, outputs, and loss functions in diffusion model setting and derive best practices for improving the training dynamics. We also suggest an improved distribution of noise levels during training, and note that non-leaking typically used with GANs is beneficial for diffusion models well.
augmentation 25
Taken together, our contributions enable significant improvements result quality, e-g., leading to record FIDs of 1.79 for CIFAR-I0 (28 and 1.36 for ImageNet [ in 64 resolution. With all key ingredients of the design space explicitly tabulated, we believe that our approach will allow easier innovation on the individual components, and thus enable more extensive and targeted exploration of the design space of diffusion models. Our implementation and pre-traue pre-trained models are available at https:
wwt.:/ooo///i/ii.. http//Vlaabb/edee
2 Expressing diffusion models in common framework
Let us denote the data distribution by (z),, with standard deviation dama and consider the family of mollified distributions p(æ:; o) obtained by adding i.i.d. Gaussian noise of standard deviation to the data. For Omax 2> data: p(æ;; max)) is practically indistinguishable from pure Gaussian noise. The idea of diffusion models is randomly sample noise image 2n ~N o max I), and sequentially denoise it into images æ; with noise levels level æ; p(æ;; o.). The endpoint N of this process is thus distributed according the data.
max > > ... ON = 0 so that at each noise
Song et al. 48 present a stochastic differential equation (SDE) that maintains the desired distribution p sample z evolves over time. This allows the above process to be implemented using stochastic solver that both removes and adds noise at each iteration. They also give a corresponding "probability flow" ordinary differential equation (ODE) where the only source of randomness is the initial noise image an- Contrary to the usual order of treatment, we begin by examining the ODE, as it offers a fruitful setting for analyzing sampling trajectories and their discretizations, The insights carry stochastic sampling, which we reintroduce as a generalization in Section 4
ODE formulation. A probability flow ODE 48 continuously increases or reduces noise level of the image when moving forward backward in time, respectively. To specify the ODE, we must first choose a schedule (t) that defines the desired noise level at time t. For example, setting a(t) x vE mathematically natural, as it corresponds to constant-speet heat diffusion [12). However, we will show in Section 3 that the choice of schedule has major practical implications and should not be made on the basis of theoretical convenience.
The defining characteristic of the probability flow ODE that evolving a sample xa ~ p( o (ta))
from time fg (either forward or backward in time) yields sample 26 p( (t,)). Following previous work [88. this requirement is satisfied (see Appendix B. and B.2) by
dæ -t()) (t) Ve log (t)) dt,
(I)

---

Table 1: Specific design choices employed by different model families. N is the number of ODE solver iterations that we wish to execute during sampling. The corresponding sequence of time steps is (lo. ",,.,,,,.. .N), where IN = If the model was originally trained for specific choices of N and { the originals are denoted by M and {u;}. respectively. The denoiser is defined as Do(x; a) = Cakipl jar + Coul (ar Fo (Gin (a) Jan; Cmoise (o)); Fo represents the raw neural network layers.
VE (88
iDDPM [36]
DDIM (46
Ours ("EDMP)
Sampling (Section 3)
ODE solver
Euler
Euler
Euler
2nd order Heun
Time steps
x-
mus (o lo mux))
where
(
N-T (amin - -mes 2))"
Schedule
(t)
vi
Scaling
(t)
1/ 1/Vottie CfRiet
Network and preconditioning (Section 5)
Architecture of Fo
DDPM+
NCSN+
DDPM
(any)
Skip scaling Capp(a)
aua (o² + dam))
Output scaling (o)(a)
Denn/ V +
Input scaling (a)
1/vo++1
1/vo'+1 +1
1/ 0z + -ana
Noise cond. Canse
(M - (0)
1n(5a)
M 1 --rr min, (1g a
In(o)
Training (Section
Noise distribution
(a)- ~ 1(
In(o)~
= Uy j ~ 1e(0 1}
In(a) ~ ~N(P ((/Pmoon/ Pan)
In(omax))
Loss weighting A(o)
1/0²
1/o²
1/0² (note: ")
(o 1 )))) /(0 odnal?
Parameters
A = 19.9, Aai
= 0.1
Gais 0.02
a = sin (i wil
oan = 0.002, daas = 80 0.5,p =
10 9,6 10
100
C 0.001, Cs = 0.008
M 1000
M 1000, 8
Punn 1.2, Put 1.2
DDDMM also employs second loss La
In our tests, Jo yielded better FID than used by iDDPM
where the dot denotes a time derivative. V log p(æ; σ) is the score function 22 vector field that points towards higher density of data at given noise level. Intuitively, an infinitesimal forward step of this ODE nudges the sample away from the data, at a rate that depends on the change in noise level. Equivalently, a backward step nudges the sample towards the data distribution.
Denoising score matching. The score function has the remarkable property that it does not depend on the generally intractable normalization constant of the underlying density function p(æ; a)) [22]. and thus can be much easier to evaluate. Specifically, if D(x; o)) is a denoiser function that minimizes the expected Le denoising error for samples drawn from Pdata separately for every i...
Eyrppnn En~A(o.. Dy + n:; o) - y1l.. then Ve log o) = (D(z;a) = (2
where y is training image and is noise. In this light, the score function isolates the noise component from the signal in æ. and Eq. ammplifies (or diminishes) it over time. Figure Iillustrate the behavior of ideal D in practice. The key observation in diffusion models is that (x;; 0) can be implemented as a neural network Do σ)) trained according to Eq. 2 Note that Do may include additional pre- and post- post-processing steps, such as scaling to an appropriate dynamic range; we will return to such preconditioning in Section
Time-dependent signal scaling. Some methods (see Appendix C.1) introduce an additional scale schedule st) and consider x s(f) to be scaled version of the original, non-scaled variable a. This changes the time-dependent probability density, and consequently also the ODE solution trajectories. The resulting ODE is generalization of Eq.
dx =
s(t) s(1)
s() ((t) (t) Vx logp
(5))
dt.
(4)
Note that we explicitly undo the scaling of x: when evaluating the score function to keep the definition of p(æ;; a) independent of s()).
Solution by discretization.
The ODE be solved is obtained by substituting Eq. into Eq. 4 to
define the point-wise gradient, and the solution can be found by numerical integration, i.c., taking
3

---

RID
110
FDD
200
20
Original sampler
reimplemcttation Heus our (1))
50
20
100
50
Black box RSC4
Our (t) & al))
10
NEE-S
32
256
(a) Uncond. CIFAR-IO, VP ODE (b) Uncond. CIFAR-I0 10, VE ODE (c) Class-cond. ImageNet-64, DDIM
Figure 2: Comparison of deterministic sampling methods using three pre-trained models, For each curve, the dot indicates the lowest NEE whose FID is within 3% of the lowest observed FID.
finite steps over discrete time intervals. This requires choosing both the integration scheme (e.g.. Euler or a variant Runge-Kutta), as well as the discrete sampling times {1o, (1,...,,, ......,,,h .. prior works rely Euler's s method, but show in Section 3 that 2nd order solver offers better computational tradeoff. For brevity, we do not provide a separate pseudocode for Euler's s method applied to our ODE here, but it can be extracted from Algorithm I by omitting lines 6-8.
Many
Putting it together. Table [ presents formulas for reproducing deterministic variants of three earlier methods in our framework. These methods were chosen because they are widely used and achieve state-of-the-ar performance, but also because they were derived from different theoretical foundations. Some of formulas appear quite different from the original papers as indirection and recursion have been removed; see Appendix C for details. The main purpose of this reframing to bring into light all the independent components that often appear tangled together in previous work. In our framework, there are no implicit dependencies between the components - any choices (within reason) for the individual formulas will, in principle, lead to functioning model. In other words, changing one component does not necessitate changes elsewhere in order to, e.g., maintain the property that the model converges to the data in the limit. In practice, some choices and combinations will of course work better than others.
3 Improvements to deterministic sampling
Improving the output quality and/or decreasing the computational cost of sampling are common topics in diffusion model research (e.g., [10 24
30
31
32
36
43 50 52 53 (6)). ). Our hypothesis
largely independent of the other components, such as network architecture and training details. In other words, the training procedure of D3 should not dictate a (t), s(t), and {tz}. nor vice versa; from the viewpoint of the sampler, D& is simply black box 152 (33 We test this by evaluating different samplers on three pre-trained models, each representing different theoretical framework and model family. We first measure baseline results for these models using their original sampler implementations, and then bring these samplers into our unified framework using the formulas in Table [ followed by our improvements. This allows to evaluate different practical choices and propose general improvements to the sampling process that are applicable all models.
is that the choices related to the sampling process
We evaluate the "DDPM++ cont. (VP)" and "NCSN+ cont. (VE)" models by Song et al. [48] trained unconditional CIFAR-10 28 at 32x corresponding to the variance preserving (VP) and variance exploding (VE) formulations 448. originally inspired by DDPM [16] and SMLD 47 We also evaluate the "ADM (dropout)"" model by Dhariwal and Nichol [9] trained on class-conditional Im- ageNet [8 at 64 x 64, corresponding to the improved DDPM (iDDPM) formulation [36]. This model was trained using discrete of M 1000 noise levels. Further details are given in Appendix C.
We evaluate the result quality in terms of Fréchet inception distance (FID) [55 computed between 50,000 generated images and all available real images. Figure 2 shows FID function of neural function evaluations (NFE), i.c., how many times Dg is evaluated to produce a single image. Given that the sampling process is dominated entirely by the cost of Dg, improvements in NFE translate directly to sampling speed. The original deterministic samplers are shown in blue, and the reimple- mentations of these methods in unified framework (orange) yield similar but consistently better results. The differences are explained by certain oversights in the original implementations as well as more careful treatment of discrete noise levels in the case of DDIM; see Appendix C. Note that our reimplementations are fully specified by Algorithm I and Table I even though the original codebases structured very differently from each other.

---

Algorithm 1 Deterministic sampling using Heun's 2nd order method with arbitrary (() and s(1). o(),, s(1), 4E(o.....N)) 2: sample 2n (0, o (26))) (16) 1)
1: procedure HEBUNSAMPLEK((D
P Generate initial sample at to Solve Eq. Lver N time steps
3: for E (0. .0.....,.. 13 do
4: d - (t4) ((t:)
x: - a(t))((t)) o(ti)
s(t.) Do (((t)) so(())
ai
b Evaluate dæ/dt at t;
(t.,) s(t.))
5:
Bi+1 f- 2; + (fi+1 ti)di
Take Fuler step from i. to fs+1
6:
if (l:4())) * 0 then
b Apply 2nd order correction unless goes to zero ((tr4))o(t+t))
7: d' +- ((u+1) 8(t++)
a(t++) a(t++1)
0:41
a(t+11)
Do
2++1 a(t++1)
o(t..n) Eval. dæ/dl at 1++1
8: 1:41 æ. + (t141 (.) (3 d..)
Explicit trapezoidal rule at t441 > Return noise-free sample at ty
9:
refurn ZV
Discretization and higher-order integrators. Solving an ODE numerically is necessarily an approximation of following the true solution trajectory. At each step, the solver introduces truncation error that accumulates over the course of N steps. The local error generally scales superlinearly with respect step size, and thus increasing N improves the accuracy of the solution.
The commonly used Euler's 's method is a first order ODE solver with (hh) local error with respect to step size h. Higher-order Runge- Kunge-Kutta methods 44 scale more favorably but require multiple evaluations of De per step. Linear multistep methods have also been recently proposed for sampling diffusion models [30 [56.. Through extensive tests, we have found Heun's 's 2nd order method 2 previoully explored in the context of diffusion models by Maliceeu--Martinaa et al. [24] to provide an excellent tradeoff between truncation error and NFE. As illustrated in Algorithm T it introduces an additional correction step for 24 to account for change in dx/dt /tt between t, and f This correction leads (hh) local error the cost of one additional evaluation of Do per step. Note that stepping to a = O would result in a division by zero, so we revert to Euler's method in this case. We discuss the general family of 2nd order solvers in Appendix D.2.
(a.k.a. improved Fuler, trapezoidal
The time steps {t;} determine how the step sizes and thus truncation errors are distributed between different noise levels. We provide detailed analysis in Appendix D.1, concluding that the step size should decrease monotonically with decreasing a and it does not need to vary on a per-sample basis. We adopt a parameterized scheme where the time steps are defined according sequence of noise levels } i.e., 4 = (ax). We set dieN = (Ai + B) and select the constants A and B so that ao max and -1 miin which gives
σicN
(amax" + N'_ (a
o max 2))² and ON O.
Here controls how much the steps near mmin are shortened at the expense of longer steps near mmax- Our analysis in Appendix D shows that setting p nearly equalizes the truncation error each step, but that in range of 5 to 10 performs much better for sampling images. This suggests that errors near mii have a large impact. We set p for the remainder of this paper.
Results for Heun's 's method and Eq. Blare shown as the green curves in Figure 2 improvement in all cases: Heun's 's method reaches the same FID Euler's method with considerably lower NFE.
We observe consistent
Trajectory curvature and noise schedule. The shape of the ODE solution trajectories is defined by functions a (t) and s(t). The choice of these functions offers a way to reduce the truncation errors discussed above, as their magnitude can be expected scale proportional to the curvature of day dal/ /l. We argue that the best choice for these functions is (t) = l and s()) = 1, which is also the choice made in DDIM 46). With this choice, the ODE of Eq. simplifies to dæ /t//t = x - D(x; t)) and and é become interchangeable.
An immediate consequence that at any E and t, a single Euler step to image Dø(æ; t). The tangent of the solution trajectory therefore always points towards the denoiser output. This can be expected to change only slowly with the noise level, which corresponds to largely linear solution trajectories. The ID ODE sketch of Figure 3 supports this intuition; the solution trajectories approach linear at both large and small noise levels, and have substantial curvature in only small region in between. The same effect can be seen with real data in Figure Tp. where the
yields the denoised
5

---

20
0
0-
-1
-20
-20
-2
-40
(a) Variance preserving ODE 48
(b) Variance exploding ODE
48
(c) DDIM [46 / /Our ODE
Figure 3: A sketch of ODE curvature ID where Pdsta is two Dirac peaks at æ =1. Horizontal t
axis is chosen to show [0, 25] in each plot, with insets showing E [0, near the data. Example local gradients are shown with black arrows. (a) Variance preserving ODE of Song et al. [48] has solution trajectories that flatten out horizontal lines at large a. Local gradients start pointing towards data only at small a. (b) Variance exploding variant has extreme curvature near data and the solution trajectories increases the solution trajectories approach straight lines that point towards the mean of data. As the trajectories become linear and point towards the data manifold.
curved everywhere. (c) With the schedule used by DDIM [46] and us, as
change between different denoiser targets occurs in relatively narrow range. With the advocated schedule, this corresponds to high ODE curvature being limited to this same range.
The effect of setting (t)) t and (t) 1 is shown as the red curves in Figure 2 As DDIM already employs these same choices, the red curve identical the green one for ImageNet-64. However, VP and VE benefit considerably from switching away from their original schedules.
Discussion. The choices that we made this section to improve deterministic sampling are summarized in the Sampling part of Table T Together, they reduce the NFE needed to reach high- quality results by a large factor: 7.3x for VP, 300x for VE, and 3.2x x for DDIM, corresponding to the highlighted NFE values in Figure 2 In practice,
can generate 26.3 high-quality CIFAR- 10 images per second on a single NVIDIA V100. The consistency of improvements corroborates our hypothesis that the sampling process is orthogonal further validation, we show results for the adaptive RK45 method [11 using our schedule as the dashed black curves in Figure the cost of this sophisticated ODE solver outweighs its benefits.
how each model originally trained. As
4 Stochastic sampling
Deterministic sampling offers many benefits, e.g., the ability to turn real images into their corre- sponding latent representations by inverting the ODE. However, it tends to lead to worse output quality [46] 48 than stochastic sampling that injects fresh noise into the image in each step. Given that ODEs and SDEs recover the same distributions in theory, what exactly is the role stochasticity?
Background. The SDEs of Song et al. 48 can be generalized 20
sum of the probability flow ODE of Eq. [and time-varying Langevin diffusion SDE 14 (see Appendix B.5):
55
dæ (1)a logp(æ;; di + ((1)a(1)) log x; (1) di 23(() (1) dui, (6)
probability flow ODE (Eq.
deterministic noise decay
injection
Langevin diffusion SDD
where is the standard Wiener process. dæ+ and dæ_ are now separate SDEs for moving forward and backward in time, related by the time reversal formula of Anderson [. The Langevin term can further be seen injection term, whose net noise level contributions cancel out. As such, (t) effectively expresses the relative rate at which existing noise is replaced with noise. The SDEs of Song et al. [48] recovered with the choice B(t) = (t)) whereby the score vanishes from the forward SDE.
combination of deterministic score-based denoising term and a stochastic noise
This perspective reveals why stochasticity is helpful in practice: The implicit Langevin diffusion drives the sample towards the desired marginal distribution at given time, actively correcting for any errors made in earlier sampling steps. On the other hand, approximating the Langevin term

---

Algorithm 2 Our stochastic sampler with (()) = 1. and s(1) 1.
1: procedure
o)) (o. NY, (o, Swose)
2: sample 2o ~ (0, & l)
min
N v2-1) if t. (Suinnun
3: for (0,.... ......... .N 1) do
P 7
4: sample <; ~ (O, mosse
otherwise
5:
i 4 + 7it
D. Select temporarily increased noise level é b Add new noise to move from i; to i Evaluate dæ dæ/dt /dt io
6:
é; f 2; + -V,
7:
d: (#: (Do ti) /
8:
R++1 + c. (t++11 f.
Take Euler step from f; to t4+1
9:
if 4+1 * then
10:
d' +- (i++1 (1 (1:1)))
b Apply 2nd order correction
3+41 < R. + (l141 - (i) + d?)
12:
return
with discrete SDE solver steps introduces error in itself. Previous results 3 non-zero (t) is helpful, but as far as we can tell, the implicit choice for (3()) in Song et al. [8] enjoys no special properties. Hence, the optimal amount of stochasticity should be determined empirically.
24 46 48
suggest that
Our stochastic sampler. We propose stochastic sampler that combines our 2nd order deterministic ODE integrator with explicit Langevin-like "churn" of adding and removing noise. A pseudocode is given in Algorithm 2 At each step i, given the sample æ; at noise level (= (ff))) we perform two sub-steps. First, we add noise to the sample according to a factor 7/ 2 0 to reach higher noise level t, + Yetz Second, from the increased -oise sample &,, we solve the ODE backward from & to f4+1 with a single step. This yields a sample 2;+11 with noise level 1441, and the iteration continues. We stress that this is not a general-pune purpose SDE solver, but a sampling procedure tailored for the specific problem. Its correctness stems from the alternation of two sub-steps that each maintain the correct distribution (up to truncation error in the ODE step). The rredictor-ooreecte sampler of Song et al. 48 has conceptually similar structure to ours.
To analyze the main difference between our method and Euler-Maruyama, we first note a subtle discrepancy in the latter when discretizing Eq. 6 One can interpret Euler-Maruyama as first adding noise and then performing an ODE step. not from the intermediate state after noise injection, but assuming that æ and a remained at the initial state at the beginning of the iteration step. In our method, the parameters used to evaluate Dg on line 7 of Algorithm 2 correspond the state after noise injection, whereas an Maruyaaa -like method would use æ,; t, instead of &;; fy. In the limit of A, approaching zero there may be no difference between these choices, but the distinction appears to become significant when pursuing low NFE with large steps.
Practical considerations. Increasing the amount of stochasticity is effective in correcting errors made by earlier sampling steps, but has its own drawbacks. We have observed (see Appendix E. that excessive Langevin-like addition and removal of noise results in gradual loss detail in the generated images with all datasets and denoiser networks. There is also drift toward oversaturated colors at very low and high noise levels. We suspect that practical denoisers induce a slightly non- conservative vector field in Eq. 3 violating the premises of Langevin diffusion and causing these detrimental effects. Notably, our experiments with analytical denoisers (such as the one in Figure lT)) have not shown such degradation.
If the degradation is caused by flaws in Do ()) they can only be remedied using heuristic means during sampling. We address the drift toward oversaturated colors by only enabling stochasticity within a specific range of noise levels f; E [Sumin: Sumax] For these noise levels, we define "/i = Setu//, where Schum controls the overall amount of stochasticity. We further clamp 7/8 to never introduce more new noise than what is already present in the image. Finally, we have found that the loss of detail can be partially counteracted by setting Smoise slightly above to inflate the standard deviation for the newly added noise. This suggests that a major component of the hypothesized non-conservativity of Da(æ; (æ;; o)) is tendency remove slightly too much noise- most likely due to regression toward the mean that can be expected to happen with any Leg-triined denoiser 29
Evaluation. Figure 4 shows that our stochastic sampler outperforms previous samplers 24 36
by significant margin, especially at low step counts. Marronncr- et al. 24 use a standard
7

---

F1
F1
FTD
32
4.5
3.0
4.0
28
2.6
3.5
2.27
22
24
20
Detemmnsstic
Optinel ottings
Sun
3.0
2.5
200
1.8
Orgiiall sampler
1.6
Mariness
2.0
NFF-16
57
64
125
250
512
1021
2018
16
52
04
125
250
512
1024
2018
30
0.6
125
200
512
1031
2016
(a) Uncond. CIFAR-IO, VP
(b) Uncond. CIFAR-10, VE
(c) Class-cond. ImageNet-64
Figure 4: Evaluation of our stochastic sampler (Algorithm 2 The purple curve corresponds to optimal choices for { Sumin Simxx Snoise J: orange, blue, and green correspond to disabling effects of Sunin.tmax and/or Snoise- The red curves show reference results for our deterministic sampler (Algorithm[]) D. equivalent to setting Sehum = O. The dashed black curves correspond to the original stochastic samplers from previous work: Euler-Maruyand [48] for VP, predictor-correet [48] for VE, and iDDMM (3) for ImageNet-64. The dots indicate lowest observed FID.
higher-order adaptive SDE solver 440 and its performance is good baseline for such solvers in general. Our sampler has been tailored to the use case by, e.g., performing noise injection and ODE step sequentially, and is not adaptive. It is open question if adaptive solvers can be a net win over a well-tumed fixed schedule in sampling diffusion models.
Through sampler improvements alone, are able to bring the ImageNet-64 model that originally achieved FID 2.07 9 1.55 that is very close to the state-of-the-art, previously, FID 1.48 has been reported for cascaded diffusion [17]. 1.55 for Hlassifier-free guidance (18), and 1 1.52 for StyleGAN- XL [44]. While our results showcase the potential gains achievable through sampler improvements, they also highlight the main shortcoming of stochasticity: For best results, must make several heuristic choices either implicit or explicit that depend on the specific model. Indeed, we had to find the optimal values of (Setura Sunin: S Swoise (Appendix E.2). This raises a general concern that using stochastic sampling as the primary means of evaluating model improvements may inadvertently end up influencing the design choices related to model architecture and training.
case-by-case basis using grid search
5 Preconditioning and training
There are various known good practices for training neural networks in a supervised fashion. For example, it is advisable to keep input and output signal magnitudes fixed to, e.g., unit variance, and to avoid large variation gradient magnitudes on per-sample basis [T] Training a neural network to model D directly would be far from ideal for example, as the input of clean signal y and noise n ~ (00, o² r²T), its magnitude varies immensely depending on noise level a. For this reason, the common practice is to not represent Do as neural network directly, but instead train a different network Fo from which Do derived.
n is combination
Previous methods [36] 46 48 address the input scaling via a o-dependent normalization factor and
attempt to precondition the output by training Fs predict n scaled to unit variance, from which the signal is then reconstructed via (z = x a (-). This has the drawback that at large a, the network needs to fine-tune output at the correct scale; note that any errors made by the network are amplified by factor of o.. In this situation, would seem much easier to predict the expected output D( or) directly. In the
output carefully to cancel out the existing noise n exactly and give the
same spirit as previous parameterizations that adaptively mix signal and noise (e.g.. [10 43 50). we propose to precondition the neural network with o-dependent skip connection that allows it to estimate either y n, something in between. We thus write Da in the following form:
Do(#; a) Ca)ip()) x Cour(a) Fe (Gn(() æ;; Cnoise
where Fg is the neural network to be trained, Csup(()) modulates the skip connection, Cm((r) and Cour (a) scale the input and output magnitudes, and Croise (a) maps noise level a into a conditioning in- put for Fg. Taking a weighted expectation of Eq. 2 the noise levels gives the overall training loss E.z.y.m [>(a) [D + n; o)) y/13]. where a ~ Purinn y ~ Platas n ~ (O T²I). The probabil-
ity of sampling given noise level is given by Porin(o) (0) and the corresponding weight is given by

---

Table Evaluation of our training improvements. The starting point (config A) is VP & VE using our deterministic sampler. At the end (configs E,F), VP & VE only differ in the architecture of Fo.
CIFAR-IE 10
at 32x32
FFHQ 26
64x64
AFHQv2 ? 64x64
Conditional
Unconditional
Unconditional
Unconditional
Training configuration
VP
VE
VP
VE
VP
VE
VP
VE
Bascline [48 ("pre-tuined)
2.48
3.11
3.01
3.77'
3.39
25.95
2.58
18.52
B + Adjust hyperparameters
2.18
2.48
2.51
2.94
3.13
22.53
2.43
23.12
+ Redistribute capacity
2.08
2.52
2.31
2.83
2.78
41.62
2.54
15.04
+ Our preconditioning
2.09
2.64
2.29
2.94
3.39
2.79
3.81
E + Our loss function
1.88
1.86
2.05
1.99
2.60
2.81
2.29
2.28
Non-leaky augmentation
1.79
1.79
1.97
1.98
2.39
2.53
1.96
2.16
NFE
35
35
35
79
79
79
79
X(o).
We can equivalently express this loss with respect to the raw network output F6 in Eq.
Es y.
A(o) Com(()) (a)² Fo (cinlo) (y - n2); Cnoise (o))
Cskip (y m
(8)
effective weight
output
effective training target
This form reveals the effective training target of Fg. allowing us determine suitable choices for the preconditioning functions from first principles. As detailed in Appendix B.6, we derive our choices shown in Table 1 by requiring network inputs and training targets to have unit variance (Cms Cout). and amplifying errors in F6 as little as possible (Crsip). The formula for Cnoise is chosen empirically.
Table 2 shows FID for series of training setups, evaluated using our deterministic sampler from Section 3 We start with the baseline training setup of Song et al. [48], which differs considerably between the VP and VE cases; we provide separate results for each (config A). To obtain a more meaningful point of comparison, we re-adjust the basic hyperparameters (config B) and improve the expressive power of the model (config C) by removing the lowest-resolution layers and doubling the capacity of the highest-resolution layers instead; see Appendix F.3 for further details. We then replace the original choices of {Cm. Cout, Cnoise, Cskip } with our preconditioning (config D). which keeps the results largely unchanged except for VE that improves considerably at 64 64 resolution. Instead of improving FID per se, the main benefit of our preconditioning that it makes the training more robust, enabling to turn our focus redesigning the loss function without adverse effects.
Loss weighting and sampling. Eq. Shows that training Fe as preconditioned in Eq. 7 incurs an effective per-sample loss weight of X(r)Cou((a)² To balance the effective loss weights, we set X(a) = 1/mou (g)2, which also equalizes the initial training loss over the entire a range as shown in Figure 5 (green curve). Finally, we need select Prain (o), i.e., how to choose noise levels during training. Inspecting the per-o loss after training (blue and orange curves) reveals that a significant reduction is possible only at intermediate noise levels; at very low levels, is both difficult and irrelevant to discern the vanishingly small noise component, whereas at high levels the training targets are always dissimilar from the correct answer that approaches dataset average. Therefore, we target the training efforts to the relevant range using a simple log-normal distribution for Purin as detailed in Table I and illustrated Figure 5) (red curve).
Table 2 shows that our proposed Ptrain and À (config F) lead to a dramatic improvement FID) in all cases when used in conjunction with our preconditioning (config D). In concurrent work, Choi et al. [6] propose similar scheme to prioritize noise levels that are most relevant w.r.t. forming the perceptually recognizable content of the image. However, they only consider the choice of À in isolation, which results in a smaller overall improvement.
Augmentation regularization. To prevent potential overfitting that often plagues diffusion models with smaller datasets, we borrow augmentation pipeline from the GAN literature [25]. The pipeline consists of various geometric transformations (see Appendix F.2) that we apply to a training image prior adding noise. To prevent the augmentations from leaking to the generated images, we provide the augmentation parameters as a conditioning input to Fø: during inference we set the them to zero guarantee that only non-augmented images generated. Table 2 shows that data augmentation provides a consistent improvement (config F) that yields new state-of-the-ar FIDs of 1.79 and 1.97 for conditional and unconditional C10AR-10. beating the previous records of 1.85 44 and 2.10 (5)]

---

1ne
FID
RDD
CIFAR-15 PFIQ 64
adiiinal VE, original
Original
modal
Distibuttonno of
VEL our model
2.4
1.0
3.5
2.2
0.8
2.0
0.0
1.5
2,5
1.57
0.2
0.0
2.0
1:2
Sa
Sawa-1
(a) Loss & noise distribution
(b) Stochasticity on CIFAR-10
(c) Stochasticity on ImageNet-64
Figure 5: (a) Observed initial (green) and final loss per noise level, representative of the the x2x32 (blue) and 64 64 (orange) models considered in this paper. The shaded regions represent the standard deviation over 10k random samples. Our proposed training sample density is shown by the dashed red curve. (b) Effect of Sehum
unconditional CIFAR-10 with 256 steps (NFE 511). For the original training setup of Song et al. [48], stochastic sampling is highly beneficial (blue, green), while deterministic sampling (Schum = 0) leads to relatively poor FID. For training setup, the situation is reversed (orange, red); stochastic sampling is not only unnecessary but harmful. (c) Effect of Sehum on class-conditional ImageNet-64 with 256 steps (NFE 511). In this more challenging scenario, stochastic sampling turns out to be useful again. Our training setup improves the results for both deterministic and stochastic sampling.
Stochastic sampling revisited. Interestingly, the relevance of stochastic sampling appears to diminish as the model itself improves, as shown in Figure When using our training setup in CIFAR-10 (Figure 5D), the best results were obtained with deterministic sampling, and any amount of stochastic sampling detrimental.
ImageNet-64. As a final experiment, we trained a class-conditional ImageNet-64 model from scratch using our proposed training improvements. This model achieved a new state-of-the-ar FID of 1.36 compared to the previous record of 1.48 [17. We used the ADM architecture (9] with no changes, and trained it using our config E with minimal tuning; see Appendix F.3 for details. We did not find overfitting to be a concern, and thus chose not employ augmentation regularization. As shown in Figure Se. the optimal amount of stochastic sampling was much lower than with the pre-trained model, but unlike with CIFAR-10, stochastic sampling was clearly better than deterministic sampling. This suggests that more diverse datasets continue to benefit from stochastic sampling.
Conclusions
Our approach of putting diffusion models to a common framework exposes a modular design. This allows a targeted investigation of individual components, potentially helping to better cover the viable design space. In our tests this let us simply replace the samplers in various carlier models, drastically improving the results. For example, in ImageNet-64 our sampler turned average model (FID 2.07) to a challenger (1.55) for the previous SOTA model (1.48) [17. and with training improvements achieved SOTA FID of 1.36. We also obtained new state-of-the -fhee-----e--ttttt results on CIFAR-11 10 while using only 35 model evaluations, deterministic sampling, and a small network. The current high-resolution diffusion models rely either on separate super-resolution steps [7
35 39, subspace projection (3), we believe that our contributions are orthogonal these extensions. That said, many of our parameter values may need to be re-adjusted for higher resolution datasets. Furthermore, we feel that the precise interaction between stochastic sampling and the training objective remains an interesting question for future work.
very large networks 9 [8). or hybrid approaches
(38 41 50
Societal impact. Our advances in sample quality can potentially amplify negative societal effects when used in a large-scale system like DALL-E 2, including types of disinformation or emphasizing sterotypes and harmful biases electricity; our project consumed 250MWh on an in-house cluster of NVIDIA V100s.
B3 The training and sampling of diffusion models needs a lot of
We thank Jaakko Lehtinen, Ming-Yu Yu Liu, Tuomas Kynkäänniemi, Axel Sauer, Arash Vahdat, and Janne Hellsten for discussions and comments, and Tero Kuosmanen, Samuel Klenberg, and Janne Hellsten for maintaining our compute infrastructure.
Acknowledgments.
10

---

References
[1] B. D. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications,
1233):313-326, 1982.
[2] U. M. Ascher and L. R. Petzold. Computer Methods for Ordinary Differential Equations and Differential- Algebraic Equationn. Society for Industrial and Applied Mathematics, 1998.
[3] F. Bao, C. Li, Zhu,
Zhang. Analytic-DPM DPM: an analytic estimate of the optimal reverse variance
diffusion probabiliitic models. In Proc. ICLR, 2022.
[4] D. Baranchuk, A. Voynov, I. Rubachev, V. Khrulkov, and A. Babenko. Label-efficient semantic segmenta- tion with diffusion models. Proc. ICLR, 2022.
[5] C. M. Bishop. Neural networks for pattern recognition. Oxford University Press, USA, 1995.
[6] Choi, J. Lee, C. Shin, S. Kim, H. Kim, and Yoon. Perception prioritized training of diffusion models.
In Proc. CVPR, 2022.
[7] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha. StarGAN v2: Diverse image synthesis for multiple domains. In Proc.
CVPR, 2020.
[8] J. Deng, W. Dong, R. Socher. L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proc. CVPR, 2009.
[91 P. Dhariwal and A. Q. Nichol. Diffusion models heat GANs on image synthesis. In Proc. NeurIPS, 2021. [10] T. Dockhom, A. Vahdat, and K. Kreis. Score-based generative modeling with critically-damped Langevin
diffusion. In Proc. ICLR, 2022.
[11] J. R. Dormand and P. J. Prince. A family of embedded Kunge-Kutta formulac. Journal of computational and applied mathematics, 6(1):11-26, 1980.
[12] J. B. J. Fourier. G. Darboux, et al. Théorie analytique de la chaleur, volume 504. Didot Paris, 1822. [13] Goodfellow, J. Pouget-Abadie, M. Mirza, Xu. D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. In Proc. NIPS, 2014.
[14] U. Grenander and M. L. Miller. Representations of knowledge in complex systems. Journal of the Royal Statistical Society: Series B (Methodological), 56(4):549-55) 1114499591111111 1994.
[15] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by two time-scale update rule converge to local Nash equilibrium. In Proc. NIPS, 2017.
[16] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion prohabilistic models. In Proc. NeurlPS, 2020.
[17] J. Ho, C. Saharia, W. Chan, D. Fleet, M. Norouzi, and T. Salimans. Cascaded diffuusion models for high fidelity image generation. Journal of Machine Learning Research, 23, 2022.
[18] J. Ho and T. Salimans. Classifier-free free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[19] J. Ho, T. Salimans, A. A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet. Vidco diffusion models. In Proc. ICLR Workshop Deep Generative Models for Highly Structured Data, 2022.
[20] C.-W. Huang. J. H. Lim, and A. C. Courville, A variational perspective on ceftusion-based generative models and score matching. In Proc. NeurlPS, 2021.
[21] L. Huang, J. Qin, Y. Zhou, F. Zhu, L. Liu, and L. Shao. Normalization techniques in training DNNs: Methodology, analysis and application. CoRR, hts/2009.128336 2020.
[22] A. Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(24):695-70) 2005.
[23] B. Jing, G. Corso, R. Berlinghieri, and T. Jaakkola. Subspace diffusion generative models. In Proc. ECCV, 2022.
[24] A. Setiiceuu-Maattinen K. Li, R. Piché-Taillefer, T. Kachman, and I. Mitliagkas. Gotta go fast when generating data with score-based models. CoRR, abs/2105. 14080, 2021.
[25] T. Karras, M. Aittala, Hellsten, Laine, J. Lehtinen, and Aila. Training generative adversarial networks with limited data. In Proc. NewrIPS, 2020.
[26] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks.
In Proc. CVPR, 2018
[27] Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro. DiffWave: A versatile diffusion model for audio synthesis. In Proc. ICLR, 2021.
[28] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[29] Lehtinen, J. Munkberg, J. Hasselgren, S. I Laine, T. Karras, M. Aittala, and T. Aila. Noise2Noise: Learning image restoration without clean data. In Proc. ICMI, 2018.
[30] I.. Liu, Y. Ren, 7. Lin, and 7. Zhao. Pseudo numerical methods for diffusion models on manifolds. In Proc. ICLR, 2022,
[31] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. Solver A fast ODE solver for diffusion prohahilistic model sampling around 10 steps. In Proc. NeurlPS, 2022.
[32] E. Luhman and T. Luhman. Knowledge distillation in iterative generative models for improved sampling speed. CoRR, abs/2101.0233 2021.
11

---

[33] P. Mishkin. L. Ahmad, M. Brundage, G. Krueger. and G. Sastry. DALL-E 2 preview - risks and limitations. OpenAI, 2022.
[34] E. Nachmani and S. Dovrat, Zero-shot translation using diffusion models. CoRR, abs/2111 2021.
[35] A. Nichol, Dhariwal, Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. GLIDE:
Towards photorealistic image generation and editing with text-guided diffusion models. In Proc. ICML,
2022.
[36] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion prohabilistic models. In Proc. ICMI, volume
139. pages 8162-8171, 2021.
[37] V. Popov, I. Vovk, V. Gogoryan, T. Sadekova, and M. Kudinov. Grad-TTS: A diffusion probabilistic model for text-to-speech. In Proc. ICML, volume 139, pages 8599-8608, 2021.
[38] K. Preechakul, N. Chatthee, S. Wizadwongsa, and S. Suwajanakom. Diffusson autoencoders: Toward a
meaningful and decodable representation. In Proc. CVPR, 2022.
[39] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hicrarchical text-conditional image generation with CLJP latents. Technical report, OpenAl, 2022.
[40] A. Roberts. Modify the improved Fuler scheme integrate stochastic differential equations. CoRR, abs/1210.0933, 2012.
[41] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. Proc. CVPR, 2022.
[42] C. Saharia, W. Chan, H. Chang, C. A. Lee, J. Ho, T. Salimans, J. Fleet, and M. Norouzi. Palette:
Image-to-image diffusion models. In Proc. SIGGRAPH, 2022.
[43] T. Salimans and J. Ho. Progressive distillation for fast sampling of diffusion models. In Proc. ICLR, 2022.
[44] A. Sauer, K. Schwarz, and A. Geiger. StyleGAN-XI XL: Scaling StyleGAN to large diverse datasets. In Proc.
SIGGRAPH, 2022.
[45] J. Sohl- Dichs-Dic,s,,in, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proc. ICML, pages 2256 206--2265,2200 2015.
[46] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In Proc. ICLR, 2021.
[47] Y. Song and S. Ermon. Generative modeling by estimating gradicnts of the data distribution. In Proc. NewrIPS, 2019.
[48] Y. Song. J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In Proc. ICLR, 2021.
[49] E. Süli and D. F. Mayers. An Introduction to Numerical Analysis. Cambridge University Press, 2003.
[50] A. Vahdat, K. Kreis, and J. Kautz. Score-based generative modeling in latent space. In Proc. NeurlPS.
2021.
[51] P. Vincent. A connection between score matching and denoising autoencoders. Newral Computation. 23(7))1661-1677, 2011.
[52] D. Watson, W. Chan, J. Ho, and M. Norouzi. Learning fast samplers for diffuusio models by differentiating
through sample quality. In Proc. ICLR, 2022.
[53] D. Watson, Ho. M. Norouzi, and W. Chan. Learning to efficiently sample from diffusion probabilistic
models. CoRR, abs/2106. 03802.. 2021.
[54] J. Wolleb, Sandkühler, Bieder, Valmaggia, and P. C. Cattin. Diffusion models for implicit image segmentation ensembles. In Medical Imaging with Deep Learning, 2022.
[55] Q. Zhang and Y. Chen. Diffusion normalizing flow. In Proc. NeurIPS, 2021.
[56] Q. Zhang and Y. Chen. Fast sampling diffusion models with exponential integrator. CoRR, Abs/2204.13900 2022.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Section 6 The main limitations
of the analysis relate to the set of tested datasets and their limited resolution.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [No] We follow common application-speeific assumptions about the probability distributions, functions and other components, but do not exhaustively specify them, or consider pathological
corner cases.
(b) Did you include complete proofs all theoretical results? [No] Our equations and algorithms build on previously known results, and highlight their practical aspects through mostly readily verifiable algebraic manipulations (Appendix B). We do not explicitly present all details of the derivations, and assume that the previous results are sufficiently rigorously proven the respective literature.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? (Yes] Our implemen- tation and pre-trained models are available at /https://giteeee wwt.thtttbb.../.m// www.coo////ooaee Satee/VVaabb/eeee edm
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] Appendix F.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] Shaded regions in Figures and
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Section
4. If you using existing assets (e.g., code, data, models) curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Appendix F.5.
(c) Did you include any new assets either in the supplemental material or URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "kmfBuNSIYOm", "reviewer": "Reviewer_er1y", "summary": "This paper studies the algorithmic design space of diffusion models. In doing so, the authors make the following contributions:\n(a) identify and characterize the degrees of freedom w.r.t. sampling, network parameterization, training\n(b) propose to disentangle sampling from training and consequently, design schemes for both deterministic and stochastic sampling based on higher-order Runge Kutta solvers that significantly reduce the number of function evaluations\n(c) improve training by better preconditioning of inputs, outputs, training losses and augmentation schemes\nEmpirically, these improvements translate to sota sample quality on standard image generation benchmarks", "questions": "- See last 2 points in the weaknesses para above and please let me know your thoughts\n- I wasn't quite clear on why the authors choose to discuss the generalized ODE in Eq. 4 when eventually they propose to set s(t)=1 which reduces to Eq (1)\n- Is there any specific reason to exclude Imagenet-64 for evaluating the effect of preconditioning in Table 2? I am particularly curious because I was hoping to contrast it with the numbers in Fig. 4c for stochastic sampling. Since we might intuitively expect that stochastic sampling could relatively be more beneficial for diverse datasets, I wonder if that intuition carries over empirically.\n- Finally, while the paper does characterize the design space reasonably well, I am curious if the authors found that their final diffusion model is robust to different choices (not just their prescribed choice) of some of the design parameters in Table 1.", "limitations": "The authors discuss negative results in the context of stochastic sampling. Perhaps a broader discussion of negative results and context around their empirical proposals could help --- concretely, one example of such a discussion point is: which design choices are data-dependent and on what properties (eg, size, dimensionality, modality)? Such information could vastly aid practitioners looking to train and deploy these models on their custom data and modalities.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This paper presents a timely and important contribution to an area of growing significance: deep generative modeling using diffusion models. Arguably, these models, while clearly having a lot of potential given their recent successes, are notorious to train requiring several training tricks.  \n- Related to the above, the paper also stands out in contrasting existing models and algorithms --- in a way, the community can look at this paper both as an excellent survey of some major works in diffusion models while providing a generalized characterization that enables the development of newer models.\n- Finally, the resulting improvements on CIFAR-10 and Imagenet are impressive -- the paper truly achieves the best of both worlds as the number of function evaluations reduce very significantly while also improving the sample quality metrics in both scenarios of deterministic and stochastic sampling. I also liked learning about some negative results and intuitions on their failures --- such as the use of stochastic samplers leading to oversaturation --  I have observed this to be a common problem in the results of some past works in diffusion models, but couldn't come across a satisfying explanation.\n\nWeaknesses:\n\n- While such a paper is not easy to write as it covers a bunch of distinct contributions, I found it a little hard to review constantly shuttling between the main text and the appendix. Some of the very important and interesting discussions such as those concerning the step sizes in Section 3 were delegated to the appendix, making readability slightly hard.\n- At certain places, it is hard to distinguish what is prior work vs what is new --- this isn't a question re: novelty of this paper, but more on the exposition lacking discussion and details. For example, Jolicoeur-Martineau et al. (duly cited) also explore the use of higher order solvers for sampling-- what exactly is different between the current work and the one before? Is it the scheme for setting timesteps to minimize truncation error in Eq. 5 or something else?\n- I think some of the claims can be placed better in context or need more empirical justification. For example, in L103-104, the authors highlight there are no dependencies between the components but does this hold broadly for all parameters in Table 1? One extreme interpretation of this could be eg, we can mix and match preconditioning schemes, i.e., skip scaling from model A (say VP), output scaling from model B (say DDIM), etc. and still train a diffusion model reasonably well. Given the brittleness of these models, I am not sure if such arbitrary mix-and-match schemes will translate to a functioning model with reasonable performance.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XyqsKy2paNd", "reviewer": "Reviewer_2v7C", "summary": "The paper proposes to consider 1) network parameterization (preconditioning) 2) training and 3) sampling of diffusion models in separation. This allows for putting many existing works in a common framework and for a simplified investigation of the three aspects in isolation. Considering, for example, the sampling process in isolation, the authors show how to improve several pre-trained networks from the literature. Furthermore, making \"optimal\" choices for each of the three aspects leads to new state-of-the-art models on several image benchmark datasets. Additionally, the paper proposes to train diffusion models on augmented data, importantly conditioning the network on the augmentation process. Lastly, the paper shows that stochastic sampling outperforms deterministic sampling for \"suboptimal models\", whereas deterministic sampling is preferred for \"well-trained models\".", "questions": "Main questions: \n* Figure 5(b): The figure is of very high value as it shows that stochasticity benefits/hurts suboptimal/well-trained models. I am, however, curious if the figure would look similar for more challenging datasets such as ImageNet. My suspicion is that for ImageNet a well-trained model may still be considerably different than the true data model and therefore I suspect that stochasticity would even improve a well-trained model. Could the authors comment on this?\n\n* Network parameterization (preconditioning): The proposed network parameterization seems very similar to the v-diffusion parameterization introduced in PD. The denoiser in v-diffusion is $\\alpha_t x_t - \\sigma_t F_\\theta(x_t, t)$, where $F_\\theta$ is the neural network. Could the authors comment on connections/advantages of v-diffusion in the proposed network parameterization as well as loss function?\n\nExtra (research) questions (for which I don't necessarily expect an answer):\n* Do the authors believe that variance-learning diffusion models (as, for example, introduced in iDDPM) could also be unified with their approach?\n", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is extremely well-written and all claims are well-supported by experiments.\n* The paper has a great impact on the current field of diffusion models: the independently proposed training recipes, network parameterizations (preconditioning), and sampling schemes can potentially be used in many future works.\n\nWeaknesses:\n* The paper lacks related work in certain places; to list only a few:\n    * Model parameterization (preconditioning):\n        * [PD] introduces v-diffusion parameterization\n        * [LSGM, CLD] introduce mixed score parameterizations\n    * Sampling:\n        * [PNDM, DEIS] apply linear multistep methods to the Diffusion Model ODE (DEIS was just recently proposed, so missing this citation is not a big deal)\n        * [LFS] learns efficient samplers\n        * [PD, KD] accelerating sampling via distillation\n* Otherwise, the paper has very little weaknesses; see Questions below. \n\n[PD] - Progressive Distillation for Fast Sampling of Diffusion Models\n\n[LSGM] - Score-based Generative Modeling in Latent Space\n\n[CLD] - Score-Based Generative Modeling with Critically-Damped Langevin Diffusion\n\n[PNDM] - Pseudo Numerical Methods for Diffusion Models on Manifolds\n\n[DEIS] - Fast Sampling of Diffusion Models with Exponential Integrator\n\n[LFS] - Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality\n\n[KD] - Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed\n  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TwVg5ExbRhs", "reviewer": "Reviewer_XT6R", "summary": "In this paper, the authors explore a wide variety of design choices made when designing diffusion models, both those which are typically given attention explicitly in prior work (e.g. the noise schedule) and ones which have been more implicitly agreed upon (e.g. the choice of of ODE solver). They highlight a set of potential sources of generalization to improve sampling - e.g., the lack of need for the sampling process to correspond to model or training details. They generalize a set of the stochastic diffusion model SDE with a single equation (Eq. 6) corresponding to moving forward and backward in time, and propose an algorithm to reduce stochastic sampling errors. In essence, prior works have explored relatively constrained subspaces of the decisions which define a particular diffusion model - this paper attempts to unify these decisions through a shared framework, allowing for new non-obvious generalizations, and then to quantitatively understand their optima. ", "questions": "What kinds of diversity/fidelity tradeoffs are implied by the various suggestions of this paper?", "limitations": "The limitations are reasonably well addressed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is remarkably thorough, considering an impressively broad range of parameters and demonstrating repeatedly why such a generalized framework is useful by applying it and making nontrivial algorithmic/formal contributions. They also serve to provide an important baseline, reducing the gap between these models and those leveraging recent advances such as cascaded diffusion models, without sacrificing (as much) theoretical interpretability. The practical consequences of the suggestions made in this paper are significant, allowing for substantially more efficient sampling, and addressing what has historically been a significant limitation of diffusion models. The paper itself is excellent and beyond its practical and theoretical contributions, represents a powerful and remarkably accessible survey of diffusion model literature (excluding a few nuances like approaches consisting of multiple diffusion models).\n\nOne noteworthy detail of this paper is the repeated highlighting on decisions which, while they could be and often are made with some theoretical motivation, are in fact best left to be empirically determined (such as the relative rates of noise decay and injection over time). While it is useful to have these empirical results, and empirical results often yield useful theoretical generalizations, it is less practical to need to perform a “case-by-case … grid search” for each new design decision. In addition, FID and other Inception-based metrics have well-known limitations e.g. [1, 2, 3]. While FID is widely used to evaluate generative models, when analyzing many hyperparameters and exploring some relatively small differences in image quality, it is difficult to know whether some empirical decisions are actually improving the generated images or overfitting to FID. It would be helpful to understand/interpret the tradeoffs made by these decisions in terms of other metrics as well, such as precision and recall [4].\n\n[1] An Improved Evaluation Framework for Generative Adversarial Networks, Liu et al 2018\n[2] Pros and cons of GAN evaluation measures, Borji 2019\n[3] A Note on the Inception Score, Barratt and Sharma 2018\n[4] Improved Precision and Recall Metric for Assessing Generative Models, Kynkäänniemi et al 2019.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EHQP3cnRbLU", "reviewer": "Reviewer_YPzX", "summary": "This work proposes a reformulation of continuous-time diffusion/score-based models that is more modular and easier to analyze. With this new formulation, authors carefully analyze different design choices in those models, including discretization for sampling, parameterization for the score model, stochasticity in the sampling process, noise distributions in training, and scaling schedules of inputs. Experiments demonstrate significant improvements in various settings, creating new/near SOTA results on CIFAR-10 and ImageNet-64.", "questions": "The stochastic sampler in Algorithm 2 is different from solving equation (6) with numerical SDE solvers. Why not perform stochastic sampling with existing numerical SDE solvers? How do they fare against the sampler in Algorithm 2?", "limitations": "Authors discussed limitations implicitly as avenues for future work, such as the precise interaction between stochasticity and the training objective. A comprehensive discussion on negative societal impacts was also included.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is an excellent work with many strengths:\n\n1. Unlike previous works, authors formulate diffusion/score-based models around the probability flow ODE. This novel perspective leads to a new analysis on the role of stochasticity in sampling, showing that previous SDE-based formulations are only a special case of combining  the probability flow ODE and the Langevin diffusion SDE with different relative weights. Stochasticity only exists in the Langevin component, which effectively corrects for potential errors in solving the probability flow ODE. This interpretation motivates authors' new stochastic samplers that outperform existing ones on image datasets. I'd like to point it out that a similar reasoning was also used to form the predictor-corrector samplers in ref. [41], where the Langevin component is the \"corrector\", and the \"predictor\" can be either a probability flow ODE or a reverse-time SDE. \n\n2. The analysis on deterministic sampling is very convincing, backed by both insightful illustrations (Figure 3) and strong empirical improvements. I especially like the analysis on how to choose the noise schedule $\\sigma(t)$ to make the probability flow ODE easy to solve. It is also a valuable contribution to show that 2nd order ODE solver (Heun's method) significantly accelerates sampling.\n\n3. The parameterization of $D_\\theta$ in equation (7) is a nice contribution, capturing the intuition that we should predict the denoised image for bigger noise levels, and predict the noise itself for smaller noise levels. The analysis on choosing $\\lambda(\\sigma)$, $p_{\\text{train}}(\\sigma)$ and others provides an effective set of hyperparameters for improved empirical performance.\n\nThere are no major weaknesses. Below are some thoughts if I have to nitpick:\n\n1. I don't fully agree that the proposed reformulation is more modular than previous ones. It seems to me that all design choices in this work can be translated to the variational formulation of diffusion models, or the continuous-time formulation of score-based generative models. The authors' new formulation seems to facilitate the analysis of stochastic samplers the most. Other contributions can be made with existing formulations with appropriate modifications.\n\n2. Derivations of hyperparameters are not fully principled. They are more or less based on intuitions (such as fixing the magnitudes of input and output signals), or discovered from experiments. This raises a question on whether the same design choice can perform as well for other data domains.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept