Gotta Go Fast When Generating Data with Score-Based Models
Authors: Alexia Jolicoeur-Martineau
Ke Li
Rémi Piché-Taillefer
Tal Kachman
Ioannis Mitliagkas
Venue: ICLR 2022 Submitted (2022)
Keywords: score-based
generative model
denoising diffusion
SDE
diffusion process

Abstract: Score-based (denoising diffusion) generative models have recently gained a lot of success in generating realistic and diverse data. These approaches define a forward diffusion process for transforming data to noise and generate data by reversing it (thereby going from noise to data). Unfortunately, current score-based models generate data very slowly due to the sheer number of score network evaluations required by numerical SDE solvers. 
   
In this work, we aim to accelerate this process by devising a more efficient SDE solver. Existing approaches rely on the Euler-Maruyama (EM) solver, which uses a fixed step size. We found that naively replacing it with other SDE solvers fares poorly - they either result in low-quality samples or become slower than EM. To get around this issue, we carefully devise an SDE solver with adaptive step sizes tailored to score-based generative models piece by piece. Our solver requires only two score function evaluations, rarely rejects samples, and leads to high-quality samples. Our approach generates data 2 to 10 times faster than EM while achieving better or equal sample quality. For high-resolution images, our method leads to significantly higher quality samples than all other methods tested. Our SDE solver has the benefit of requiring no step size tuning.

Under review as a conference paper at ICLR 2022
AN EFFICIENT SDE SOLVER FOR SCORE-BASED DIFFU- SION MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Score-based (denoising diffusion) generative models have recently gained lot of success in generating realistic and diverse data. These approaches define forward diffusion process for transforming data to noise and generate data by reversing it (thereby going from noise to data). Unfortunately, current score-based models generate data very slowly due to the sheer number of score network evaluations required by numerical SDE solvers.
In this work, we aim to accelerate this process by devising a more efficient SDE solver. Existing approaches rely on the Euler-Maruyama (EM) solver, which uses a fixed step size. We found that naively replacing it with other SDE solvers fares poorly they either result in low-quality samples or become slower than EM. To get around this issue, we carefully devise an SDE solver with adaptive step sizes tailored to score-based generative models piece by piece. Our solver requires only two score function evaluations per step, rarely rejects samples, and leads to high-quality samples. Our approach generates data 2 to 10 times faster than EM while achieving better or equal sample quality. For high-resolution images, our method leads to significantly higher quality samples than all other methods tested. Our SDE solver has the benefit of requiring no step size tuning.
INTRODUCTION
Score-based generative models (Song and Ermon 2019 2020 Ho al. 2020
cct al. 2020; Song et al.] 2020a| Pichō-Taillefe (2021) have been very successful at generating data from various modalities, such images (Ho al. 2020 Song et al. 2020a), audio (Chen et al. 2020
Kong et al.) 2020 Mittal et al., 2021 Kameoka et al. (2020), and graphs (Niu et al. 2020). They have
further been used effectively for super-resolution (Saharia al. 2021: Kadkhodaie and Simoncelli
2020), inpainting (Kadkhodaie and Simoncelli] 2020), source separation (Jayaram and Thickstun
2020). and image-to-image translation (Sasaki et al. 202T). In most of these applications, score- based models achieved superior performances in terms of quality and diversity than the historically dominant Generative Adversarial Networks (GANs) (Goodfellow et al., 2014)).
Score-based models can be understood in two main classes: those based on a Variance Exploding (VE) diffusion process (Song and Ermon 2019) and those based on a Variance Preserving (VP) one (Ho et al. 2020). Both diffusion processes progressively transform real data into Gaussian noise; NO, o - 1) for VE where ma is very large, and NO I) for VP.
The diffusion process (VE, VP, etc.) is then reversed in order to generate real data from Gaussian noise. Reversing the process requires the score function, which is estimated with neural network (known score network). Although very powerful, score-based models generate data through an undesirably long iterative process; meanwhile, other state-of-the-an methods such as GANs generate data from single forward pass of neural network. Increasing the speed of the generative process is thus an active area of research.
Chen et al. (2020) and Roma- al. (2021) proposed faster step size schedules for VP diffusions that still yield relatively good quality/diversity metrics. Although fast, these schedules are arbitrary, require careful tuning, and the optimal schedules will vary from model to another.

---

Under review as a conference paper at ICLR 2022
VE-deep (32x32 CIFAR-10)
VP-deep (32x32 CIFAR-10)
10
Euler Maruyama
Euler Muler-Meruyama
Our SDE Salver
DDIM
FID (50K samples)
9
FID (50K samples)
15-
5
10
4-
200
400
600
800
1000
200
400
500
800
1000
Number of function evaluations
Number of function evaluations
VE-deep (256x256 LSUN-Church)
VE-deep (256x256 FFHQ)
400
*
Euler Maruyaaa
Euler-Maruyama
Our SDE Salver
250
Our SDE Solver
FDD (5K samples)
350
FID (5K samples)
300
200
250
150
200
150
100
100
250
500
750
1000
1250
1500 1750
2000
250
500
750
1000
1250
1500 1750
2000
Number of function evaluations
Number of function evaluations
Figure 1: Comparison between novel SDE solver at various values of error tolerance and Euler- Maruyama for an equal computational budget. We measure speed through the Number of Function Evaluations (NFE) and the quality of the generated images through the Fréchet Inception Distance (FID; lower is better). See Table 1 for more details.
Block et al. (2020) proposed generating data progressively from low to high-resolution images and show that the scheme improves speed. Similarly, Nichol and Dhariwal (2021) proposed generating low-resolution images and then upscale them since generating low-resolution images is quicker. They further suggested to accelerate VP-based models by lcarning cennension-speciiie noise rather than assuming equal noise everywhere. Note that these methods do not affect the data generation algorithm and would thus be complementary
Nichool
methods.
Song et al. (220a) and Song et al. (2020b) proposed removing the noise from the data generation algorithm and solve Ordinary Differential Equation (ODE) rather than Stochastic Differential Equation (SDE); they report being able to converge much faster when there is no noise. Although it improves the generation speed, [Song et al. (2020a) report obtaining lower-quality images when using the ODE formulation for the VE process (Song et al. 2020a).. We will later show that our SDE solver generally leads to better results than ODE solvers at similar speeds.
Thus, existing methods for acceleration often require considerable step size/schedule tuning (this also true for the baseline approach) and do not always work for both VE and VP processes. To improve speed and remove the need for step size/schedule tuning, we propose to solve the reverse diffusion process using SDE solvers with adaptive step sizes.
It turns out that off-the-shelf SDE solvers are ill-suited for generative modeling and exhibit either (1) divergence, (2) slower data generation than the baseline, or (3) significantly worse quality than the baseline (see Appendix A.. This can be attributed to distinct features of the SDEs that arise in score-based generative models that set them apart from the SDEs traditionally considered in the numerical SDE solver literature, namely: (1) the codomain of the unknown function is extremely high-dimensional, especially the case of image generation; (2) evaluating the score function is computationally expensive, requiring forward pass of a large mini-batch through large neural network; (3) the required precision of the solution is smaller than usual because we are satisfied as long the error is not perceptible (e.g., one RGB increment
image).
2

---

Under review as a conference paper at ICLR 2022
We devise our own SDE solver with these features in mind, resulting in an algorithm that can get around the problems encountered by off-the-shele solvers. To address high dimensionality, we use the & norm rather than the &o norm measure the enror across different dimensions prevent single pixel from slowing down the solver. To address the cost of score function evaluations while (1) take the minimum number of score function evaluations needed
still obtaining high precision, for adaptive step sizes (two evaluations), and (2) use extrapolation to get high precision at no extra cost. To take advantage of the reduced requirement for precision, we set the absolute tolerance for the error according to the range of RGB values.
Our main contribution new SDE solver tailored to score-based generative models with the following benefits:
Our solver is much faster than the baseline methods, i.c. reverse-diffusion method with Langevin dynamics and Euler-Maruyama (EM);
It yields higher quality/diversity samples than EM when using the same computing budget;
It does not require any step size or schedule tuning;
It can be used to quickly solve any type of diffusion process (e.g., VE, VP)
BACKGROUND
2.1 SCORE-BASED MODELING WITH SDES
Let x(0) E ndd be a sample from the data distribution Pdata The sample is gradually corrupted over time through Forward Diffusion Process (FDP), common type of Stochastie Differential Equation (SDE):
dx = f(x, t)dt g(t)dw.
(1) TR. is the diffusion coefficient and w(t) is the
where f(x, t) RR IR Rd is the drift, g(()
Wiener process indexed by f E [0. 1]. Data points and their probability distribution evolve along
the trajectories {x( and {pr {x) respectively, with pa = Pdata- The functions and g are chosen such that x1) be approximately Gaussian and independent from x(0). Inference is achieved by reversing this diffusion, drawing x(1) from its Gaussian distribution and solving the Reverse Diffusion Process (RDP) equal to:
dx [f(x g(t)2 V log (x)) dt g())dw,
where Vx log Pe (x) is referred to as the score of the distribution at time (Hyvärinen 2005) and w(t) is the Wiener process in which time flows backward (Anderson) 1982).
One can observe from Equation 2 that the RDP requires knowledge of the score (or Pt), which we do not have access to. Fortunately, it can be estimated by neural network (referred to the score network) by optimizing the following objective:
A(t) (x()))) Vx(e) log px
(()) (x(()) (xx()))))))))))))))))))))))))))) (()) (0)~ppan
(3)
where A(t): : IR + IR is a weighting function generally chosen to be inversely proportional to:
E
Vx(1) lo (xx)((())))())))))))
One can demonstrate that the minimizer of that objective 0* will be such that 8g- (x, 1) = Vx log Pe (x) (Vincent] 2011), allowing us to approximate the reverse diffusion process. As can be the objective requires the ability to generate samples from the FDP at arbitrary times Thankfully, as long the drift is affine (i.e., f(x, = Ax + B), the transition kernel p(x(t)(x(0)) (0)) will always be normally distributed (Särkkä and Solin 2019). which means that we can solve the forward diffusion of the Gaussian transition kernel is trivial compute, making
evaluating
in single step. Furthermore, the the loss an inexpensive training objective.
There primary choices for the FDP in the literature, which discuss below.
3

---

Under review as a conference paper at ICLR 2022
2.2 VARIANCE EXPLODING (VE) PROCESS
The Variance Exploding (VB) process consists in the following FDP:
d (t) dw.
dx
dt
Its associated transition kernel is:
x(t) ((x(()).00 - = -Z xN(x(0),
In practice, we let (() min
where min 0.01 and mmaz max; Σ ||x
x(0 the maximum Euclidean distance between two samples from the dataset fx(*) N (Song and Ermon 2020). Using the maximum Euclidean distance ensures that x x(1) does not depend on x(0); thus, x(1) is approximately distributed as (O o² (1)I).
2.3 VARIANCE PRESERVING (VP) PROCESS
The Variance Preserving (VP) process consists in the following FDP:
dx = - + /B(tjdw.
Its associated transition kernel is:
x(1) ((() N(x(0)) ( B(s)ds e So ((s)do) I).
In practice, let B()) Bmin + 1. (Bmax 3min), where 8min 0.1 and Bmaz 20. Thus, x(1)
is approximately distributed as N(0,I) I) and does not depend on x(0)
2.4 SOLVING THE REVERSE DIFFUSION PROCESS (RDP)
There are many ways to solve the RDP; the most basic one being Marur-aa (K Kloeden and Platen 1992). the SDE analog Euler's method for solving ODEs. Song et al. (2020a) also proposed Reverse- Diffuusio,,, which consists in ancestral sampling (Ho et al. 2020) with the same discretization used in the FDP. With the Reverse-Diffusion, (Song al.) 2020a) obtained poor results unless applying an additional Langevin dynamics step after each Reversion- approach Predictor-Correcte (PC) sampling, with the predictor corresponding to Devvere--ifffuiin and the corrector to Langevin dynamics. Although using a corrector step leads to better samples, PC sampling only heuristically motivated and the theoretical underpinnings are not yet understood. Nevertheless, (Song et al.) 2020a) report their best results (in terms of lowest Fréchet Inception Distance (Heusel al.] (201]) using the Reverse- Difffusion with Langevin dynamics for VE models. For VP models, they obtain their best results using Fuler-Maruyama.
step. They named this
3 EFFICIENT METHOD FOR SOLVING REVERSE DIFFUSION PROCESSES
3.1 SETTING UP THE ALGORRIHHM
We start with general algorithm for solving an SDE (similar to most ODE/SDE solvers). We choose the various options/hyper- parameers based mixture of theory and experiments; ablation study of the different hyper-parameter parameters can also be found in Appendix B
3.1.1 INTEGRATION METHOD
Solving the RDP to generate data can take an undesirably long time. One would assume that solving SDEs with high-order methods would improve speed over Euler-Maruyamae just like high-order ODE solvers improve speed over Euler's 's method when solving ODEs. However, this is not always the case: while higher-order solvers may achieve lower discretization errors, they require more function
4

---

Under review as a conference paper at ICLR 2022
evaluations, and the improved precision might not be worth the increased computation cost (Lehn iet al. 2002 Lamba 2003).
Our preliminary attempts at using SDE solvers with the DifferentialEquations. Julia package (Rackauckas and Nie 2017a) confirmed that higher-order methods were significantly slower (6 to times slower; see Appendix[Ap Lamba's 's algorithm (Lamba 2003), low-order adaptive method, yielded the fastest results, thus motivating us to restrict our search to the space of low-order methods. Still, the resulting images were of lower quality.
Using a fixed step-size while solving an ODE/SDE requires some tuning and one should be able to advance faster (from 1 to 1 0) in regions of low noise. To gain more speed, one can dynamically adjust the step size over time; this is common approach used in most fast ODE/SDE solvers. Such technique generally use two integration methods: lower-order (x') method and a higher-order (x") method. The local error E = x' - x" is used to determine how stable the lower-order method the current step size; the closer zero, the more appropriate the step size is. From this information, we can dynamically adjust the step size and decide whether or not to accept the proposed sample of the lower-order method. Alternatively, one can select x" as the proposal, which we will refer to as extrapolating.
Rather than using the Improved Fuler ODE solver (Süli and Mayers 2003) as in Lamba (2003) or a high-order stochastic Runge- Kunge-Kutta method (Rößler) 2010) in Rackauckas and Nie (201 (b) (which did not work well in our preliminary attempts with the Julia package) we instead rely on the more recent Improved Euler SDE solver (Roberts 2012) our higher order method. This method is very similar to the classical Improved Euler ODE solver, but it is made to work with SDEs instead of ODEs. Importantly, this method only requires two score function evaluations and re-uses the same score function evaluation used for EM, meaning that it is only twice as expensive as EM. Similarly to Lamba's 's algorithm, this method, albeit quick, leads to images of poor quality. However, by using extrapolation (taking x" instead of x' as our proposal), we were able to match and improve over the baseline approach (EM). Thus. using the stochastic Improved Euler was the key taking bigger steps without sacrificing precision. Note that Lamba's 's algorithm cannot use extrapolation due to its use of a non- non-stochastic ODE solver (Improved Euler).
An algorithm has strong- condgg-odderpoo p when the local error from t to t + h is 0h**+1 )). Suler-Maruyama has strong-order 0.5 while Improved Euler has strong-order (Roberts) 2012). The highest strong- order found in the DifferentialEquations/ Julia package (Rackauckas and Nie 2017a) are order 1.5 1.5. Thus, our method obtains a balance between methods that are I) low precision, but fast and 2) high precision, but slow.
3.1.2 TOLERANCE
In ODE/SDE solvers, the local error is divided by tolerance term. Traditionally, the mixed tolerance 5(x'): IRd + Rd is calculated as the maximum between the absolute and relative tolerance:
8(x)) maxtse eret xx().
where the absolute value |- applied element-wise.
The DifferentillEquationsj jl Julia package instead calculates the mixed tolerance through the maxi- mum of the current and previous sample:
8 8(x', x xprev)= = max Grel max([x'), | l)).
(5)
Having no trivial prior for which approach to use, we tried both and found the latter approach (Equation 5) to converge much faster for VE models (see Appendix B).
Given our focus on image generation, we can set tata a priori. During training and at the end of the data generation, images are represented as floating- point tensors with range [ymin ymaz]- When evaluated, they must be transformed into 8-bit color images; this means that images are scaled to the range [0, 255) and converted to the nearest integer (to represent one of the 256 values per color channel). Given the 8-bit color encoding, an absolute tolerance eabs Mmas Vussn corresponds to 256 tolerating local errors of at most one color (e.g., æ'y with Red=5 and z" with Red=6 is accepted, but I'y with Red=5 and 2i with Red=7 is not) whannel-wise. One-color differences are not perceptible and should not influence the metrics used for evaluating the generated images. For VP models, which
5

---

Under review as a conference paper at ICLR 2022
have range [-1, this corresponds cabs 0.0078 while for VE models, which have range [0, 1], this corresponds to Cabs = 0.0039.
To control speed/quality, vary Erel, where large values lead to more speed but less precision, while small values lead to the converse.
3.1.3 NORM OF THE SCALED ERROR
The scaled error (the error scaled by the mixed tolerance) is calculated as
x'
x"
Eg
(xx, x xrev
Many algorithms use q 00 (Lamba max(x), ..... Xx) over all & elements of Although this can work with low-dimensions SDEs, this is highly problematic for high-dimensional SDEs such as those in image-space. The reason is that a single channel of single pixel (out of 65536 pixels for 256 x 256 color image) with large local error will cause the step size to reduced for all pixels and possibly lead to a step size rejection. Indeed, our experiments confirmed that using q == slows down generation considerably (see Appendix
2003
Rackauckas and Nie 2017b), where ||x||>>
B To that effect, we instead use scaled &g norm:
x' x" 8(x', Xpren)
|x|
2
71
3.1.4 HYPERPARAMETERS OF THE DYNAMIC STEP SIZE ALGORITIIM
Upon calculating the scaled error, accept the proposal x" if Eg < and increment the time by Whether or not it is accepted, we update the next step size h in the usual way:
h min(hmma 0h F.y
(6)
where hmax is the maximum step size, A is the safety parameter which determines how strongly adapt the step size (0 being very safe; being fast, but high rejections rate), and r is an exponent- scaling term.
Although ODE theory tells us that we should let r p+1 with p being the order of the lower- order integration method, there is no such theory for SDEs (Rackauckas and Nie 2017b). Thus, as Rackauckas and Nie (2017b) suggest, resorted to empirically testing values and found that any r E [0.5, works well on both VE and VP processes, but that r E [0.8, 0.9) is slightly faster (see Appendix B We arbitrarily chose r 0.9 the default setting.
Finally, we defaulted to setting 0 = 0.9 for the safety parameter as is common in the literature, and choose hmax to be equal to the largest step size possible, namely the remaining time
3.1.5 HANDLING THE MINI-BATCH
Using the same step size for every sample of a mini-batch means that every images would be slowed down by the other images. Since every image's RDP is independent from another, apply different step size to each data sample; some images may converge faster than others, but we wait for all images have converged.
3.2 ALGORITHM
In Section 3.1 We defined every aspect of the algorithm needed to numerically solve Equation 2 for image generation. The algorithm thus consists in using adaptive step sizes through Equation 6 with the hyperparameters defined in the previous subsection (g = oo, 9 = 0.9, r = 0.9, cabs with Maller-Maryyama the low-order method and Improved Euler the high-order method. The resulting algorithm is described in Algorithm[T This algorithm straightforward to parallelize across the batch dimension.
Minas Ven
256

---

Under review as a conference paper at ICLR 2022
Note that this algorithm is only for solving an RDP; more general version for solving an arbitrary forward-ting time diffusion process can be found in Appendix [ Additionally, we present a demonstration in Appendix [ showing that the extrapolation step conserves the stability and convergence of the EM
step.
Algorithm Dynamic step size extrapolation for solving Reverse Diffusion Processes
Require: s0: Cress Cabes Penat 0.01, r 0.9, 0 0.9
> for images: cabs
Uman Vonin
256
t 2-11.
h <- Initialize x
hinit
Xprev fx X
while > 0 do
Draw z 1)
x' f- x x- (x((x,)) () + hg(())20 1) Vhgg(1)2
Ealer-Maruyama
- x- hf(x',t -t, + hi(t - so) - h) + Vhg(t- h)z
x" (x' + +X)
Improved Euler (SDE version)
t max Cabbo Crel max([x'), 1xprol)))))))
b Element-wise operations
F2 -- 1 11(x' - x") /0112
if Eg < n
then
Accept b. Extrapolation
X 4 x"
t
Xprev f- x'
4 min OhE2')
> Dynamic step size update
return X
4 EXPERIMENTS
To test Algorithm on RDPs, we apply various pre-trained models from [Song et al.) (2020a). To start, we generate low-resolution images (32x32) by testing the VP, VE, VP-deep, and VE- deep models on CIFAR-10 (Krizhevsky et al. 2009). Then, we generate higher-resolutions images (256x256) by testing the VE models on LSUN Church (Yu al. 2015), and Flickr-Faces- (FFHQ) (Karras et al. (219)). See implementation details in Appendix D We used four or less V100 GPUs to the experiments.
To measure the performance of the image generation, we calculate the Fréchet Inception Distance (FID) (Heusel al., 2017) and the Inception Score (IS) (Salimans al., (0166), where low FID and high IS correspond to higher quality/diversity. We compare our method to the three base solvers used in [Sng al. (2020a): Reverse-Diffusion with Langevin dynamics, Euler-Maruyama (EM), and Probability Flow, where the latter solves an ODE instead of an SDE using Runge-Kutta 45 (Dormand and Prince 1980). We also compare against the fast solver by (Song et al. 2020b) called denoising diffusion implicit models (DDIM), which is only defined for VP models. We define the buseline approach as the solver used by [ong al. (2020a) which leads to the lowest FID (EM for VP models and Reverse- Diffussio with Langevin for VE models). For our algorithm, the only free hyperparameter the relative tolerance which we set to Erel E (0.01, 0.02, 0.05, 0.1, 0.5).
The FID and the Number of score Function Evaluations (NFE) are described in Table I for low- resolution images and Table 2 for high-resolution images. The Inception Score (IS) described for CIFAR-10 in Appendix B
4.1 PERFORMANCE
Compared to EM, we observe that our method is immediately advantageous in terms of qual- ity/diversity for high-resolution images, along with to 3x speedups (Eret 0.02). While this advantage becomes less obvious in terms of the FID on CIFAR-10, our method still offers > 5x computational speedups apparent disadvantage (Eret (0.02, 0.05)).
7

---

Under review as a conference paper at ICLR 2022
Table Number of score Function Evaluations (NFE) Fréchet Inception Distance (FID) on CIFAR- 10 (32x32) from 50K samples
Method
VP
VP-deep
VE
VE-deep
& Langevin
1999 / 3.41
1999 3.28
1999 / 2.40
1999 2.21
Euler-Maruyama
1000 / 2.55
1000 2.49
1000 2.98
1000 3.14
DDIM
1000 / 2.86
1000 / 2.69
Ours (Crel 0.01) Bolerr-Mauyymma (same NFE)
329 / 2.70
330 2.56
738 / 2.91
736 3.06
329 10.28
330/ 10.00
738 / 2.99
736 3.17
DDIM (same NFE)
329 4481
3.7/
Ours (Eret 0.02)
274 2.74
2.4
490 / 2.87
488 2.99
Buler-Maruyama (same NFE)
274 14.18
274/ 13.67 2.74
490 / 3.05
488 / 3.21
DDIM (same NFE)
274 5.75
Ours (cret 0.05)
179 2.59
180 2.44
271 3.23
270 3.40
Fuler-Maruyama (same NFE)
179 25.49
180 / 25.05
271 3.48
270 3.76
DDIM (same NFE)
/
180 9.25
Ours (Cret = 0.10)
147 2.95
151 2.73
170 / 8.85
170 10.15 /.56
Fuler-Maruyama (same NFE)
147 /47/33..8
151 / 31.93
170 5.12
DDIM (same NFE)
/4.551555
151 / 11.38
Ours (Crel 0.50)
49 72.22
48 / 82.42
52 266.75
50 / 307.32
Euler-Maruyama (same NFE)
49.92.9
48 95.77
52 169.32
50 271.27
DDIM (same NFE)
49/ 37.24
48 / 38.71
Probability Flow (ODE)
145 2.86
183 / 7.64
181 5.53
Table Number of score Function Evaluations (NFE) Fréchet Inception Distance (FID) on LSUN- Church (256x256) and FFHQ (256x256) from 5K samples
Method
VE (Church)
VE (FFHQ)
Reverse-Diffusine & Langevin
3999 / 19.14
3999 / 16.42
Euler-Maruyama
2000 / 42.11
2000 / 18.57
Ours (Cret 0.01)
1104 / 25.67
1020 /15.68
Euler-Maruyana (same NFE)
1104 / 68.24
1020 / 20.45
Ours (Erel 0.02)
1030 / 26.46
643 / 15.67
Maller-Mauyyama (same NFE)
1030 / 73.47
643 44.42
Ours (tret 0.05)
648 / 28.47
336 18.07
Buler-Maruyama (same NFE)
648 145.96
336 114.23
Ours (cret = 0.10)
201 44.92
198 24.02
Fuler- Fuler- Maruyamm (same NFF)
201 / 417.77
198 284.61
Probability Flow (ODE)
434 234.214.47
369 135.50
8

---

Under review as a conference paper at ICLR 2022
Deerss--iffuiion with Langevin achieves the lowest FID for VE models CIFAR-10, 10, though at the cost of a 4x computational overhead over our method. Furthermore, their advantage vanishes for VP models and when generating nighrresolution images.
We further compare our SDE solver to EM given the same computational budget and observe that method is always immensely preferable in high-resolutions and for VP models. For VE models on CIFAR-10, we observe that our algorithm leads to better FID as long as the NFE is sufficiently large (270). Note that since our algorithm takes two score function evaluations per step, EM has the advantage of doing twice as many steps given the same NFE, which appears to be a factor more important than the order the method low budget in low-resolution VE. Nevertheless, comparing for equal number of iterative step, the results still point to our method being always preferable. For high-resolution images, we see that EM cannot converge on moderate to small NFEs due to the high-dimensionality, making of our SDE solver the way to go.
Generally, we observe that the VE process cannot be solved as fast as the VP process; this due to the enormous Gaussian noise in the VE process causing larger local errors. This reflects the issue mentioned in Section 3.1.1 regarding high-order SDE solvers not always being beneficial in terms of speed for SDEs with heavy Gaussian noise. In practice, for VE, the algorithm uses a small step size the beginning to ensure high accuracy and eventually increases the step size as the noise becomes less considerable.
4.2 SOLVING ODE INSTEAD OF AN SDE
We that our SDE solver generally does better than Probability Flow, especially in high-resolution, where we obtain greatly lower FIDs with similar budget. Our algorithm attains the same NFE as Probability Flow when Eyel = 0.10 for low-resolution images and when 0.05 Crel 0.10 for high-resolution images, For the same budget, Probability Flow has higher FID than our approach on all but low-resolution VE models. However, in that case, our algorithm achieves much lower FID when Grel I 0.05, albeit slower. In high-resolution, Probability Flow leads to very poor FIDs, suggesting no convergence.
4.3 DDIM
Contrary to Song et al. (2020) FID of DDIM worsens significantly when the NFE decreases. This could be due to differences between Song al. (2020a) continuous-time score-matching and the DDIM training procedure and architecture. Nevertheless, the FIID increase engendered by a reduced budget is much less dramatic than for EM. As of note, DDIM succeeds in maintaining lower FIDD than our solver at extremely small NFEs (< 50), albeit with extremely poor FID.
5 LIMITATIONS
Although we tested our approach wide range of settings, we nevertheless only tested on continuous-tin time image generation models. We did so because solving the SDE requires continuous- time and the only such pre-trained models time of publishing are the one by [Song al. (2020a).
Although our approach removes step size and schedule tuning, we still need to choose a value of the relative tolerance, which indirectly affects the number steps taken; one could theoretically tune this hyper-parameter to optimize certain metric, going against the point of removing tuning. Still, letting Gre 0.01 for precise results and Gret = 0.05 for fast results are reasonable choices, as all evidence points the FID being stable w.r.t. crel.
6 CONCLUSION
We built an SDE solver that allows for generating images of comparable (or better) quality to Euler-Maruyan at much faster speed. Our approach makes image generation with score-basee based models more accessible by shrinking the required computational budgets by a factor of 2 to 5x, and presenting sensible way of compromising quality for additional speed.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances Neural Information Processing Systems, pages 11918-11930, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. arXiv preprint 1xX1v:2006.09011, 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilisticc models. arXiv preprint arxiv:2006. 1239, 2020.
Alexia Dolicoeur-Martinean Rémi Piché- Taillffai Rémi Tachet des Combes, and loannis Mitliagkas. Adversarial score matching and improved sampling for image generation. arXiv preprint arXiv:2009.05475, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 133456, 2020a.
Rémi Piché-Taillefer Hamiltonian monte carlo and consistent sampling for score-based generative modeling. Master's thesis, Université de Montréal, Québec, 2021.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprinl urXiv:2009.00713. 2020.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint Siti:2009.0976// 2020.
Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprint 17Xiv:2103.1609 2021.
Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, and Nobukatsu Hojo. Voicegrad: Non-paun con-parallel any-to-many voice conversion with annealed langevin dynamics. arXiv preprint arXiv:2010.02977, 2020.
Chenhao NIu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permu- tation invariant graph generation via score-based generative modeling. International Conference on Artificial Intelligence and Statistics, pages 4474-4484. PMLR, 2020.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636, 2021.
Zahra Kadkhodaie and Eero P Simoncelli. Solving linear inverse problems using the prior implicit in denoiser. arXiv preprint arXiv:2007.136400 2020.
Vivek Jayaram and John Thickstun. Source separation with deep generative priors. In International Conference on Machine Learning, pages 4735-4735. PMLR, 2020.
Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104.053:3 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 2672-2680. Curran Associates, Inc., 2014. WRLLhttp://paar adversarial -nettallleee...... .ddf
Robin San-Roman, Eliya Nachmani, and Lior Wolf. Noise estimation for generative diffusion models.
arXiv preprint 11XX::2104.02000 2021.
Adam Block, Youssef Mrouch, Alexander Rakhlin, and Jerret Ross. Fast mixing of multi-scale langevin dynamics underthe manifold hypothesis. arXiv preprint arXiv:2006. //66,, 2020.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint arXiv: /02.09672. 2021.
10

---

Under review as a conference paper at ICLR 2022
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020b.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(Apr):695-709, 2005.
Brian DO Anderson.
Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):31-3326 1982.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- tion, 2367):1661-1674 2011.
Simo Särkkä and Arno Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
Peter E Kloeden and Eckhard Platen. Stochastic differential equations. In Numerical Solution of Stochastic Differential Equations, pages 103-160. Springer, 1992.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626-6633 2017.
Jürgen Lehn, Andreas RöBler, and Oliver Schein. Adaptive schemes for the numerical solution of sdes a comparison. Journal of computational and applied mathematics, 138(2):297-308, 2002.
Lamba. An adaptive timestepping algorithm for stochastic differential equations. Journal of computational and applied mathematies, 161(2)4177430, 2003.
Christopher Rackauckas and Qing Nie. Differentialequations.]]
performant and feature-rich
ecosystem for solving differential equations in julia. The Journal of Open Research Software, 5(1), 2017a. doi: 10.5334/jors.551 URL / dimensions ai/ /dtaiis/ Exported from https://app.dimeesiionssaa
publication/pu 1085583166
2019/05/05.
Endre Süli and David Mayers. introduction to numerical analysis. Cambridge university press, 2003.
Andreas RöBler. Runge-kutta methods for the strong approximation of solutions of stochastic differential equations. SIAM Journal on Numerical Analysis, 48(3):922-952, 2010.
Christopher Rackauckas and Qing Nie. Adaptive methods for stochastic differential equations via natural embeddings and rejection sampling with memory. Discrete and continuous dynamical systems. Series B, 22(7):2731, 2017b.
AJ Roberts. Modify the improved euler scheme to integrate stochastic differential equations. arXiv preprint arXiv: 1210.09// 2012.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 150. 150..03366..3333 03365, 2015.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pages 2234-2242. 2016.
John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematies, 6(1):19-26, 1980.
11

---

Under review as a conference paper at ICLR 2022
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106(496):1022-1614 2011.
Crispin Gardiner. Stochastic methods, volume 4, Springer Berlin, 2009.
Ludwig Amold. Stochastic differential equations. New York, 1974.
Sergej S Artemiev and Tatjana A Averina. Numerical analysis of systems of ordinary and stochastic differential equations. Walter de Gruyter, 2011.
Yoshihiro Saito and Taketomo Mitsui, Stability analysis of numerical schemes for stochastic differen- tial equations. SIAM Journal Numerical Analysis, 23(6):2254-2200 1996.
12

---

Under review as a conference paper at ICLR 2022
APPENDICES
A DIFFERENIALEBUUATIN...
Table 3: Short experiments with various SDE solvers from Differeeiiallquationns// on the VP model with small mini-batch.
Method
Strong-Order
Adaptive
Speed
Euler-Marus Buler-Maruyama (EM)
0.5
No
Baseline speed
SOSRA (RöBler
2010
1.5
Yes
5.92 times slower
SRA3 RöBler 2010
Yes
6.93 times slower
Lamba EM (detault) LLmma 2003
0.5
Yes
Did not converge
Lamba EM (atol=1e-3)
(Lamba
2003
0.5
Yes
2 times faster
Lamba EM (atol=1e-3, rtto=lo-o)) (La3m (Lamba 2003
0.5
Yes
1.27 times faster
Euler-Heun
0.5
No
1.86 times slower
Lamba Euler-Heun (Lamba 2003
0.5
Yes
1.75 times faster
SOSRI (RöBler] 2010)
1.5
Yes
8.57 times slower
RKMil (at various tolerances) (Kloeden and Platen 1992)
1.0
Yes
Did not converge
ImplicitRK.Mi (Kloeden and Platen 1992)
1.0
Yes
Did not converge
ISSEM
0.5
Yes
Did not converge
Here, we report the preliminary experiments ran with the Differential|quationss]] Julia package (Rackauckas and Nie| 2017a) before devising our own SDE solver. As can be seen, most methods either did not converge (with warnings of "iinttability detected") or converged, but were much slower than Suler-Maruyama. The only promising method was Lamba's method (Lamba 2003). Note that an algorithm has strong-order p when the local error from t to + h is O (h)+t)).
B EFFECTS OF MODIFYING ALGORITHM
Table Effect of different settings on the [Inception score (IS) / Fréchet Inception Distance (FID) Number of score Function Evaluations (NFE)) from 10k samples (with mini-batches of 1k samples) with the VP- CIFARIO model.
Change(s) in Algorithm
IS
FID
NFE
No change [g 2, 0.9, 8(x'x 1 X
9.38
4.70
3972
Small modifications
8(x')
9.26
4.69
4166
No Extrapolation (thus, using Euler-Man (alle-Maruyyma)
9.58
11.73
3978
a = xc .5 r = .8
9.48
4.90
14462
9.41
4.69
4104
9.36
4.68
3938
9.41
4.69
4048
Variations o Lamba 2003) Algorithm
0.5, Lamba integration
7.80
52.98
1468
0.5, Lamba integration, Extrapolation
7.32
64.65
1438
0.5, Lamba integration, = 0C
9.28
21.09
2360
0.5, Lamba integration,
0 - 0.8
9.21
18.82
2346
As can be seen, most chosen settings lead to better results. However, seems have little impact the FID. Still, using r e [0.8, 0.9] lead to a little less score function evaluations and sometimes lead to lower FID.
13

---

Under review as a conference paper at ICLR 2022
Table Effect of different settings on the [Inception score (IS) / Fréchet Inception Distance (FID) Number of score Function Evaluations (NFE)] from 10k samples (with mini-batches of 1k samples) with the VE CIFARIO model.
Change(s) in Algorithm
IS
FID
NFE
No change q = 2, 0.9, 8(x', 8 Xprew)
9.39
4.89
8856
Small modifications
8(x')
9.39
4.99
17514
No Extrapolation (thus, using Euler-Manue Maruyamma)a
9.58
6.57
8802
q 0.5
9.41
5.03
39500
9.47
4.87
9594
r 0.8
9.45
4.84
8952
=
9.43
4.93
8784
Variations of Lamba 2003) Algorithm
0.5, Lamba integration
9.08
18.28
2492
r 0.5, Lamba integration, Extrapolation
3.70
169.78
2252
0.5, Lamba integration, q 00
9.42
6.80
5886
r = 0.5, Lamba integration, co. = 0.8
9.35
6.20
2970
We notice that using = oc and 6(x') lead to higher NFE as we expect. However, they also generally lead to higher FID, thus lower quality/diversityy which is not expected! We hypothesized that this might be due to the large number of steps taken when using q = oo and ((x'). To test this, we trained the VE and VP models with Euler-Maruyama with 10k steps instead of Ik steps and indeed obtained higher FIDs. This means that taking too many steps leads to worse performance in score-based models.
Worse quality from taking more steps should typically not happen as more steps should mean a more precise trajectory. We hypothesize this be caused by the difference between using the actual score function instead of using the pre-trained score-network; given the errors in the score network, it may be that taking could also be due to the metric, as no existing quality/diversity metrics for generative models is truly perfect; but we believe that this hypothesis is less plausible than the increasing errors from using the score-network.
many steps leads to some deviations from the right solution. Alternatively, this
14

---

Under review as a conference paper at ICLR 2022
C DYNAMIC STEP SIZE ALGORITHM FOR SOLVING ANY TYPE OF SDE (RATHER THAN JUST REVERSE DIFFUSION PROCESSES)
Assume, have Diffusion Process of the form:
dx f(x, t)dt + g f)dw.
The algorithm to solve it is represented in Algorithm 2 The differences are:
it is in forward-tin
the range of time must be given
The diffusion can depend which leads slightly more complicated formulation that depends on some random number 8 = +1 (Roberts 2012).
we retain the full trajectory instead of only the ending
we retain the noise after rejection ensure that there no bias in the rejections
Algorithm 2 Dynamic step size extrapolation for solving arbitrary (forward-time) Diffusion Processes
Require: 80, lbegin lend, Crel, Cabs, hinit = 0.01, r = 0.9, 0 = 0.9
t tbegin
h f hinst
Initialize x(1)
X Xpree + X
Draw OO
while < tend do if Stratonovich SDE or g(x, t) = g(x) then
8 8+0
else
Itō diffusion
Draw & ~ Uniform((-1,1)
x' < x(t) + hx(((()), t) + vhg(x(t), () - 8)
Euler- Maruyaaa
X + x(1) + 1-15f(x h) V l1 8)
x" + 1(('' +X)
Improved Euler (SDE version) >> Element-wise operations
5 - max((Cabr> Crel max([x'), (xxxxx)))x(xr
Eg 1 11(x' -x") /8112
F2 < then
Accept
1 {- h
x(t) + x"
b Extrapolation
Xprev + x'
Draw z I), 4- min(t, OhEz')
Dynamic step size update
return x
15

---

Under review as a conference paper at ICLR 2022
D IMPLEMENTATION DETAILS
We started from the original code by Song et al. 2020a) but changed a few settings concerning the SDE solving. This creates the VP and VP-deep models, we obtained 2.55 and 2.49 instead of the original 2.55 and 2.41 for the baseline method (EM). For the VE and VE-deep models, obtained 2.40 and 2.21 instead of the original 2.38 and 2.20 for the baseline method (Reverse -iiffuusonn with Langevin).
very minor difference between their reported results and ours. For
As done in Song et (2020a), we used the optimal hignal-t-noise ratio of 0.01 for the Langevin conrector.
When solving the SDE, time followed the sequence to 1, 4--1 1= where 1000 for CIFAR-10, N = 2000 for LSUN, = lc - 3 for VP models, and c = le - 5 for VE models.
Meanwhile, the actual step size h used in the code for Guler-Maruyame (EM) was equal n. Thus, there was a negligible difference between the step size used the algorithm (h } and the actual step size implied by (h = )) Note that this has little to no impact.
The bigger issue is at the last predictor step was going from = cto N < 0. Thus, f was made negative. Furthermore the sample was denoised t while assuming i = There are two ways to fix this issue: 1) take only step from 1 to & = 0 and do not denoise (since you cannot denoise with the incorrect or with 0), or 2) stop f and then denoise. Since denoising is very helpful, we took approach 2; however, both approaches are sensible.
Finally, denoising was not implemented correctly before. Denoising was implemented as one predictor step Rleverse-Diffusion or EM) without adding noise. This corresponds to:
- h f t) g(t) log log)((x)]
At the last iteration, this incorrect denoising would be:
d(o)(t)) 1 N
x x
Vx Vxlogp((x)
dt
Umin
2 log
Omax
Vx log px
Omin
R3 X
for VE and
Biin
V log
N
for VP.
Mcanwhile, the correct way to denoise based on Tweedie formula (Efron 2011 is:
x - x + Var(x(1) Vx logp,(x))
where (ar( is the variance of the transition kernel: Var[x min 0.01 for VE and (x) = 1. This means that the correct Tweedie formula corresponds to
x イ x log
x x
for VE and
- X + Vx log
for VP.
As can be seen, denoising has very small impact VE so the difference between the correct and incorrect denoising is minor. Meanwhile, for VP the incorrect denoising lead to tiny change, while the correct denoising lead to large change. In practice, we observe that changing the denoising method to the correct one does not significantly affect the FID with VE, but lowers down the FID significantly with VP.
16

---

Under review as a conference paper at ICLR 2022
E
INCEPTION SCORE ON CIFAR-10
Table 6: Inception Score CIFAR-10 (32x32) from 50K samples
Method
VP
VP-deep
VE
VE-deep
Secvers--Diffusiinn & Langevin
9.94
9.85
9.86
9.83
Euler-Maruyama
9.71
9.73
9,49
9.31
Ours (Grel = 0.01)
9,46
9.54
9.50
9.48
Ours (Grel 0.02)
9.51
9.48
9.57
9.50
Ours (Gret = 0.05)
9.50
9.61
9.64
9.63
Ours (Erel = 0.10)
9.69
9.64
9.87
9.75
Probability Flow (ODE)
9.37
9.33
9.17
9.32
STABILITY AND BIAS OF THE NUMERICAL SCHEME
The following constructions rely on the underlying assumption of the stochastic dynamics being driven by a wiener process. More so, also assume that the Brownian motion is time symmetrical. Both assumptions are consistent and widely used in the literature; for example, see (Gardiner 2009) (Arnold 1974).
The method described in Algorithm gives us a significant speedup in terms of computing time and actions. Albeit the speed up comes from piece-wise step in the algorithm combining the traditional Euler Maruyama (EM) with form of adaptive step size predictor-correctt the stability and the convergence of the EM scheme conserved by introducing the extra adaptive stepsize of our new scheme. As first step, we define the stability and bias in Stochastic Differential Equation (SDE) numerical solution.
Here we show that both
We denote R(A) as the real value of a complex-valued X.
The linear SDE defined in the following way:
dx; = Ax,dl + adw;
with its numerical counterpart
Yn+1 = (ht) yn Znz
where the 2, are random variables that do not depend on Yo- In or À and the EM scheme is
Yn+1 = (1 - hA) + Zn.
numerical scheme is asymptotically unbiased with step size h > 0 if, for a given linear SDE driven by two-sided Wiener process, the distribution of the numerical solution y,, converges as
n + oo to the normal distribution with zero mean and variance 2|X]
a² AAttemeev and Averina, 2011 1).
This stems from the fact that a solution of a linear SDE 8 is a Gaussian process whenever the initial condition is Gaussian (or deterministic): thus, there are only two moments that control the bias in the algorithm:
d²
lim
E [
0.
lim E [
|X]
numerical scheme with step size h is numerically stable in mean the numerical solution y!!)) applied to linear SDE satisfies
lim EE [yx]]
0,
and is stable mean square (Saito and Mitsui 1996) if have that
ly, )))) o²
lim lim
000
29R(A)
17

---

Under review as a conference paper at ICLR 2022
In what follows, will trace the criteria for bias through algorithm and show that it remains unbiased. By construction, the first EM step remains unbiased, while for the RDP, we write down the time reverse Wiener process as
y n++1= = (1 + Ah) yn + Zn
the reverse time steps h i.e., L nh, 2nh,
y Nn+1
(1+ X - -1) nY
= =1+ h)) [[1 X 8))
= (1 h) E [5
= A1+ 11)) [Yul
Thus,
1 + A h)l < <1,
then
lim
E
(h) 0.
100
In Algorithm 1, we are performing consecutive steps forward and backwards in time so = 2h such that
+ A4 < 1.
Thus, the scheme is both numerically stable and unbiased with respect to the mean.
Next, we focus the numerical solution in mean square:
E
1 n+1
A h)) E |5 P] + h
1 A 13) 1 A( h)) [ + 0²h + 0
1 A 11) 11)2(x+1)))
1 + A( h)) 2(
EE [lvol] +
0
2MR + 1212 (t h)
Under the same assumption of consecutive steps, we have that
11 + Ah 2(n+1)
[]] = 11
Ah
2(x+1)
E lyoll
0²
28((A) + |\|² h
lim E [|nm+1] P]
00000
2R(() |^| h
lim lim E (11=+=11)) 121) h An-+00
o²
2A((A)
Assuming the imaginary part of X is null, we have that
lim lim E
[|
(T) 2 2|
0²
Thus, the numerical scheme is stable and unbiased in the mean square.
Following the two steps for computation of x' and š; the step size decreases and does not change size: thus. all the above statements hold, and the entire algorithm is stable and unbiased with respect both the mean and square mean.
18

---

Under review as a conference paper at ICLR 2022
G
SAMPLES
(a) Dynamic-ster step Extrapolation (c 0.01)
(b) Dynamic-ster step Extrapolation 0.02)
(c) Dynamic-stop Extrapolation (c = 0.05)
(d) Dynamic-step Extrapolation (e = 0.10)
Figure 2: VP CIFAR10
19

---

Under review as a conference paper at ICLR 2022
(a) Dynamic-step step Extrapolation (e 0.01)
(b) Dynamic-stor step Extrapolation (e 0.02)
(c) Dynamic-step Extrapolation (e 0.05)
(d) Dynamic-step Extrapolation 0.10)
Figure 3: VP-deep CIFAR10
20

---

Under review as a conference paper at ICLR 2022
(a) Dynamie-step Extrapolation (e 0.01)
(b) Dynamic-step Extrapolation (e 0.02)
(c) Dynamic-step Extrapolation (e 0.05)
(d) Dynamic-step Extrapolation 0.10)
Figure 4: VE CIFAR10
21

---

Under review as a conference paper at ICLR 2022
(a) Dynamic-step Extrapolation (e 0.01)
(b) Dynamic-step Extrapolation (e 0.02)
(c) Dynamic-step Extrapolation (e 0.05)
(d) Dynamic-step Extrapolation 0.10)
Figure 5: VE-deep CIFARIO
22

---

Under review as a conference paper at ICLR 2022
(a) Dynamie-step step Extrapolation (e 0.01)
(b) Dynamic-step Extrapolation (e 0.02)
(c) Dynamic-step Extrapolation (e 0.05)
(d) Dynamic-step Extrapolation 0.10)
Figure 6: LSUN-SSUNULNN (256x256)
23

---

Under review as a conference paper at ICLR 2022
(a) Dynamic-step Extrapolation (e 0.01)
(b) Dynamic-step Extrapolation (e 0.02)
(c) Dynamic-step Extrapolation (e 0.05)
(d) Dynamic-step Extrapolation 0.10)
Figure 7: VE FFHQ (256x256)
24


Reviews:
[{"review_id": "RD-snkZRn-Q", "reviewer": "Reviewer_ybV4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The major contribution of the paper seems to be an application of existing numerical solvers (Roberts, 2012) to sampling from SDE models. The technical novelty could be limited as the solver already exists despite the tricks used to improve the empirical results. \n2. There is no theoretical analysis of the method.\n3. It seems that the major contribution is on the empirical side. However, the empirical results are not very impressive especially when NFE is small. For instance, in Table 1, DDIM outperforms the proposed method when NFE=49. As the goal of the paper is to improve the sampling speed of score-based models, having a good result when NFE is small enough is important. An FID of 72.29 using 49 NFE might not be impressive since NFE=49 is already a large budget. Wondering when the proposed method can outperform DDIM when NFE=100.\n4. There are some formatting issues (e.g., too much space) on the first page. In the tables, some best values are in bold while some are not. It would be good to be consistent.\n5. Algorithm 1 should be better explained in section 3.2. The required parameters and their selections should also be explained more clearly.\n6. The paper writing can be improved. For instance, in section 3.1.1, \"the stochastic Improved Euler’s method\" should be explained more clearly.  \"Dynamic step size algorithm\" is an important contribution of the paper, the setting should be better explained. If possible, can move some details from the appendix to the main paper.\n7. It would also be interesting to perform experiments on even higher-dimensional images if model checkpoints are available.", "summary_of_the_paper": "Score-based/diffusion-based generative models can achieve high sample quality. However, their sampling speed is slow due to the large number of evaluations required by numerical SDE solvers. This works aims to accelerate the sampling process by using a more efficient SDE solver. The proposed approach generates data 2 to 10 times faster than the baselines while achieving reasonably well sample qualities.", "summary_of_the_review": "The major contribution of the paper seems to be an application of existing numerical solvers to sampling from SDE models. The novelty could be limited as the solver already exists (despite the tricks proposed to improve empirical results). Given that the theoretical contribution is limited, the major contribution would be on the empirical side. However, the empirical results are not impressive enough: when using a small number of sampling steps, the proposed method has much worse results compared to DDIM. As the goal of the paper is to efficiently sample from SDE models, having strong performance when the sampling steps are small is important.\n\nThe paper writing can be improved. There are some formatting issues.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MswWB7t__VX", "reviewer": "Reviewer_Px4w", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The research problem of this paper is significant and interesting to the community of deep generative models, especially score-based models and denoising diffusion probabilistic models. The writing of this paper is clear. \n\nStrengths:\n- A novel SDE solver with a stability and bias analysis.\n- The empirical success of the algorithm, i.e., high generation quality with less steps (function evaluations). \n- The algorithm can readily be adapted to other score-based models with SDE. \n\nI am not an expert in numerical solvers for differential equations, so I will leave evaluation of this part to other reviewers.\n\nWeaknesses (and questions):\n- One major concern is about the title. The title is too exaggerated and does not reflect the content of the paper. Instead, I would like to see a title that is precise, for example, \"An efficient SDE solver for score-based models\". \n- Fig 1: What about the original Improved Euler baseline? \n- Section 3: The several components of the algorithm seem to be parameter tuning and engineering trick. There is no theoretical or even intuitive justification for why they offer better performance (Appendix F does not answer this question). In addition, I do not see any ablation study on the contribution of each component. \n- Section 3.1.3: The last equation ($\\ell_2$ norm) in this subsection cannot be understood.\n- The algorithm: Although the algorithm does not need stepsize or scheduler tuning, there are other hyperparameters involved so some tuning is still needed.\n- Experiments: How often do you reject under different NFE? There should be experiments on the relationship between NFE and percentage of rejection. Next, there should be a figure similar to Fig 1 but the x-axis is the time spent rather than NFE. \n- Appendix A: Are those methods $\\ell_2$ or $\\ell_{\\infty}$? \n- Appendix E: What can we read from this table?", "summary_of_the_paper": "The paper presents a new SDE solver for the reverse process in score-based models. The algorithm is fast and offers high quality, and avoids some hyperparameter tuning. There is theoretical analysis on the stability and bias of the algorithm. The paper also has experiments comparing the proposed algorithm to several baseline methods. ", "summary_of_the_review": "The algorithm seems to be empirically successful. However, the paper lacks justification for why their algorithm can be better. There is also much space for improvements in the experiments (especially, on the overall generation speed measured by seconds). In addition, the title is exaggerated. \n\n---------\nUpdates after the rebuttal\n\nI appreciate that the authors agreed to use a more precise title. Some of my concerns are also addressed. Therefore, I will increase my score to 5. However, I still think the experiments lack some key components such as time spent and number of rejections, although they might strongly correlate to NFE. The empirical results are good but not impressive enough. I encourage the authors to further improve the method and make additional theoretical/empirical justifications. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JdElWxd7Oga", "reviewer": "Reviewer_4pA6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n1. This paper proposes the first adaptive step-size numerical SDE solver for score-based generative modeling. Experimental results demonstrate clear improvement over previous methods like Euler-Maruyama methods and probability flow ODEs. It also demonstrates improvement over DDIM for a moderate budget of iteration numbers.\n\n2. The method itself is simple and clear, and only requires tuning one hyperparameter. \n\n## Weaknesses\n\n1. No theoretical understanding of the proposed numerical SDE solver. It would be better to include an analyze on the convergence order of the proposed approach.\n\n2. Results of predictor-corrector for VP SDEs in Table 1 are significantly worse than results reported in the original score sde paper. What could be the reason for this? Did you use the correct signal-to-noise ratio for the Langevin corrector?\n\n3. Some minor writing issues. The first page has a footnote \"equal contribution\", while all author names should be anonymized. In section 3.1.3, $E_q$ is defined to be a scalar (the $L_q$ norm), but it is referred as a vector in the expression for computing $\\| x\\|_2 $.\n\n4. In section 3.1.5, it is mentioned that a different step size is applied to each data sample. How is this implemented in deep learning frameworks? Will it cause noticeable slowdown on GPUs?", "summary_of_the_paper": "This paper proposes a new numerical solver for stochastic differential equations and demonstrated significant improvement over existing ones such as the Euler-Maruyama solver in terms of the quality/computation trade-off for score-based generative modeling with SDEs.", "summary_of_the_review": "This paper proposes a simple and effective numerical SDE integrator for score-based generative modeling based on SDEs. Paper can be stronger with a deeper theoretical understanding.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BJuozbDdJk", "reviewer": "Reviewer_RPn6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1)\nI understand the paper's nature lies in the empirical side. However, some claims can be further elaborated through formal analyses. For example, Norm calculation in 3.1.3 can be analyzed by following algorithmic complexity, i.e. big-oh notations. I suggest that authors find such analysis opportunity throughout the paper. Moreover, it would be great if you can come up with a table to organize such complexity comparisons.\n\n2)\nI cannot follow why Eq 5 could be better in both quality and speed at the same time. Eq 5 has more constraints to setup the tolerance, so the speed-up seems to be an obvious benefit. However, the quality could be damaged while I also note that the quality is not being too much hurt from your result report. Why is that?\n\nThis question goes same to utilizing L2 norm, instead of L-$\\infty$ norm\n\n3)\nIs there any theoretic argument on the integration method? Which can be proved or contests as a proposition or a theorem?\nI could not fully comprehend the argument in the current text, and it just looks like swapping SDE solvers to find the best match.\n\n4)\nThese improvements can be further investigated through ablation studies, which seems to be must-do, in my opinion. Without the ablation study, I cannot argue which technique contributed to more or less.", "summary_of_the_paper": "This paper presents a number of treatments to speed up and improve the generation quality in the process of the reverse diffusion process, diffusion-based generative models. This paper is mostly empirical by nature, and authors suggested five techniques in improving the SDE solver performance.", "summary_of_the_review": "Good paper with interesting and essential ideas in improving the usability of diffusion models. However, the current experimental result and the method justification are weak.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes numerical method for solving SDEs that empirically are faster than previous approaches. Two reviewers felt the paper was above threshold, while two felt it was below threshold for acceptance. While the paper is borderline in this sense, all four reviewers noted that the paper lacked a theoretical justification and rested on empirical evidence for the usefulness of the approach. Several reviewers also pointed out that these empirical results are on the weak side. While the paper may add a potentially useful learning trick to the optimization literature, these two significant concerns put it on the side of a borderline reject.
Decision: Reject