Pseudo Numerical Methods for Diffusion Models on Manifolds
Authors: Luping Liu
Yi Ren
Zhijie Lin
Zhou Zhao
Venue: ICLR 2022 Poster (2022)
Keywords: diffusion model
generative model
numerical method
manifold

Abstract: Denoising Diffusion Probabilistic Models (DDPMs) can generate high-quality samples such as image and audio samples. However, DDPMs require hundreds to thousands of iterations to produce a sample. Several prior works have successfully accelerated DDPMs through adjusting the variance schedule (e.g., Improved Denoising Diffusion Probabilistic Models) or the denoising equation (e.g., Denoising Diffusion Implicit Models (DDIMs)). However, these acceleration methods cannot maintain the quality of samples and even introduce new noise at high speedup rate, which limit their practicability. To accelerate the inference process while keeping the sample quality, we provide a new perspective that DDPMs should be treated as solving differential equations on manifolds. Under such a perspective, we propose pseudo numerical methods for diffusion models (PNDMs). Specifically, we figure out how to solve differential equations on manifolds and show that DDIMs are simple cases of pseudo numerical methods. We change several classical numerical methods to corresponding pseudo numerical methods and find that pseudo linear multi-step method is the best method in most situations. According to our experiments, by directly using pre-trained models on Cifar10, CelebA and LSUN, PNDMs can generate higher quality synthetic images with only 50 steps compared with 1000-step DDIMs (20x speedup), significantly outperform DDIMs with 250 steps (by around 0.4 in FID) and have good generalization on different variance schedules.

Published as a conference paper at ICLR 2022
PSEUDO NUMERICAL METHODS FOR DIFFUSION MODELS ON MANIFOLDS
Luping Liu, Yi Ren, Zhijie Lin & Zhou Zhao Zhejiang University {luping. 1iu, rayeren, linzhi jie, thaozhou)@zzj..ee edu.
ABSTRACT
Denoising Diffusion Probabilistic Models (DDPMs) can generate high-quality samples such image and audio samples. However, DDPMs require hundreds to thousands of iterations to produce final samples. Several prior works have successfully accelerated DDPMs through adjusting the variance schedule (e.g.. Improved Denoising Diffusion Probabilistic Models) or the denoising equation (e.g., Denoising Diffusion Implicit Models (DDIMs)). However, these acceler- ation methods cannot maintain the quality of samples and even introduce new noise high speedup rate, which limit their practicability. To accelerate the inference process while keeping the sample quality, we provide a fresh perspec- tive that DDPMs should be treated solving differential equations on manifolds. Under such a perspective, we propose pseudo numerical methods for diffusion models (PNDMs). Specifically, we figure out how solve differential equations manifolds and show that DDIMs are simple cases of pseudo numerical meth- ods. We change several classical numerical methods to corresponding pseudo numerical methods and find that the pscudo linear multi-step method is the best in most situations. According to our experiments, by directly using pre-trained models on Cifar10, CelebA and LSUN, PNDMs generate higher quality syn- thetic images with only 50 steps compared with 1000-step DDIMs (20x speedup), significantly outperform DDIMs with 250 steps (by around 0.4 in FID) and have good generalization on different variance schedules." !
INTRODUCTION
Denoising Diffusion Probabilistic Models (DDPMs) (Sohi-Dickstein all 2015: Ho alll 2020)
is a class of generative models which model the data distribution through an iterative denoising process reversing multi-step noising process. DDPMs have been applied successfully to a variety of applications, including image generation cHo all M120; Song et al., 20205), text generation
(Hoogeboom et all, 2021: Austin all 2021). 3D point cloud generation (Luo Hu. 2021), text-
to-speech (Kong et all 2021: Chen et all 2020) and image super-resolution (Saharia et all 2021).
Unlike Generative Adversarial Networks (GANs) (Goodfellow et all 2014), which require careful
hyperparameter tuning according to different model structures and datasets, DDPMs can use similar model structures and trained by simple denoising objective which makes the models fit the noise in the data. To generate samples, the iterative denoising process starts from white noise and progressively denoises it into the target domain according to the noise predicted by the model at every step. However, critical drawback of DDPMs is that DDPMs require hundreds to thousands of iterations to produce high-quality samples and need to pass through a network at least once at every step, which makes the generation of large number of samples extremely slow and infeasible. In contrast, GANs only need one pass through network.
There have been many recent works focusing on improving the speed of the denoising process. Some works search for better variance schedules, including Nichol Dhariwal (2021) and Watson (02T). Some works focus on changing the inference equation, including Song et all (2020a)
"Corresponding author OOu implementation is available htttss://ggtthiie com/ ng- 11u/ PNDM

---

Published as a conference paper at ICLR 2022
and Song al (202()). Denoising Diffusion Implicit Models (DDIMs) (Song et all 20203)
relying
on a non non-Markon Markovian process accelerate the denoising process by taking multiple steps every itera- tion. Probability Flows (PFs) (Song et all 20201) build a connection between the denoising process and solving ordinary differential equations and use numerical methods of differential equations to accelerate the denoising process. Additionally, we introduce more related works in Appendix AA
However, this direct connection be- tween DDPMs and numerical meth- ods (e.g., forward Euler method, lin- ear multi-step method and Runge- Kutta method (Timothy, 20TZ)) has weaknesses in both speed and effect (see Section B.I.). Some numerical methods are straightforward, like the forward Euler method, but they can only trade quality for speed. Some numerical methods can accelerate the reverse process without loss of qual- ity, like the Sungge-Kutaa method, but they need to propagate forward more times along a neural network at every step. Furthermone, we also notice that numerical methods can introduce noticeable noise at high speedup rate, which makes high-order numerical methods (c.g.. Runge-Kutta method) even less effective than DDIMs. This phenomenon also mentioned in Salımans & Ho (2022).
Figure 1: 5, 10, 20, 50 and 100-steps generated results using DDIMs, classical numerical methods and PNDMs.
To figure out the reason for the performance degradation in classical numerical methods, we conduct some analyses and find that classical numerical methods may sample data far away from the main distribution area of the data, and the inference equations of DDPMs do not satisfy a necessary condition of numerical methods the last several steps (see Section B.2)
To tackle these problems, we design new numerical methods called pseudo numerical methods for diffusion models (PNDMs) to generate samples along specific manifold in R", which is the high- density region of the data. We first compute the corresponding differential equations of diffusion models directly and self-consistently, which builds a theoretical connection between DDPMs and numerical methods. Considering that classical numerical methods cannot guarantee to generate samples on certain manifolds, we provide brand-new numerical methods called pseudo numerical methods based
theoretical analyses. We also find that DDIMs are simple cases of pseudo numerical methods, which means that we also provide a new way to understand DDIMs better. Furthermore, find that the pseudo linear multi-step method is the fastest method for diffusion models under similar generated quality.
Besides, we provide detailed theoretical analysis of our new theory and give visualization results to support our theory intuitively. According to our experiments, our methods have several advantages:
Our methods combine the benefits of DDIMs and high-order numerical methods successfully. We theoretically prove that our new methods PNDMs are second-order convergent while DDIMs are first-order convergent, which makes PNDMs 20x faster without loss of quality on Cifar10 and CelebA.
Our methods can reduce the best FID of pre-trained models with even shorter sampling time. With only 250 steps, our new denoising process can reduce the best FID by around 0.4 points Cifar10 and CelebA. We achieve new SOTA FID score of 2.71 on CelebA.
Our methods work well with different variance schedules, which means that our methods have good generalization and can be used together with those works introducing better variance sched- ules to accelerate the denoising process further.
2 BACKGROUND
In this section, we introduce some backgrounds. Firstly, we present the classical understanding of DDPMs. Then provide another understanding based on [Song all (20205), which inspires us to use numerical methods to accelerate the denoising process of diffusion models. After that, we introduce some background numerical methods used later in this paper.
2

---

Published as a conference paper at ICLR 2022
2.1 DENOISING DIFFUSION PROBABILISTIC MODELS
DDPMs model the data distribution from Gaussian distribution to image distribution through an iterative denoising process. Let zo be an image, then the diffusion process is Markov process and the reverse process has similar form to the diffusion process, which satisfies:
2++1 ~N
3, 3,1), t = 2t-1 ~ (x) 8), (2,,, t)I), -
1
(1) Here, 3e controls the speed adding noise to the data, calling them the variance schedule. N the total number of steps of the denoising process. 1s and Bg are two neural networks, and 0 are their parameters.
Ho al (2020) get some statistics estimations of ps and So. According to the properties of the conditional Gaussian distribution, we have:
((0)) NN(a0 (1 at))),
q( [t, (1, v0) (ūt 50),
Here, a₁ = 1- ā, = ITi ai The = vor-
1-3: zo + var(1-ar-1) and B, = 1-ā:-1
1-&; Bt. Then
this paper sets 3g = 3, and designs objective function to help neural networks represent μlg.
Objective Function The objective function is defined by:
L = BBg [1 - 110 (are, 1)19
1 xe(
30
)
no (2o,
Va+
v1 - 84.
(3)
B2
|le - colooo Va + a t)]?
Exxoe
012(1 - (z))
Here, (0 c) Va vI between μ4 and 6o μ40 ya,
estimate of the noise The relationship (xz VI-ai co). Because N(00, 1), we assume that the mean
Oet, (N(0 1), ())))
and variance of are 0 and 1.
2.2 STOCHASTIC DIFFERENTIAL EQUATION
According to Song et all (20206)) there is another understanding of DDPMs. The diffusion process can be treated as solving a certain stochastic differential equation d.r = (V1 (t) 1)x(t))dd +
/B(t)dw. According to Anderson (1982), the denoising process also satisfies a similar stochastic
differential equation:
dr = ((V1- - B(t) - 1)x(t) ((t)ca dt + 3(())d
(4)
This is Variance Preserving stochastic differential equations (VP-SDEs). Here, we change the do- main of from [1, .] [0, 1]. When N tends infinity. {B {z;
) become continuous functions 8(t) and (t) on [0, 1]. Song et all (20206) also show that this equation has an ordinary differential equation (ODE) version with the same marginal probability density as Equation (+))
dr = ((/I- 3(t) - 1)x(t) 1 ((t)co(x(()))) dt.
(5)
This different denoising equation with no random item and the same diffusion equation together is Probability Flows (PFs). These two denoising equations show us new possibility that we can use numerical methods to accelerate the reverse process. As far as we know, DDIMs first try to remove this random item, so PFs can also be treated as an acceleration of DDIMs, while VP-SDEs are an acceleration of DDPMs.
2.3 NUMERICAL METHOD
Many classical numerical methods can be used to solve ODEs, including the forward Euler method, Runge-Kutta method and linear multi-step method (Timothy, 2017).
3

---

Published as a conference paper at ICLR 2022
Forward Euler Method For certain differential equation satisfying de (x,t). The trivial numerical method is forward Euler method satisfying Z146 = 23 + (z).
Runge-Kutta Method Runge-Kutte Kutta method uses more information at every step, it can achieve higher accuracy R. Runge-Kutte Kutta method satisfies:
{
h1 f(re;
2 f(x: k t + 2
lig - far + k22 t+ ())
K4 f(a + ōkg, + 8)
Z1+6 = 21 + (k1 + 2k2 + 2k + ks).
Linear Multi-Step Method Linear multi-step method is another numerical method and satisfies:
T1+6 = Tt + 24
(55 - 5- + 37 9fc-2s), St [( 1).
PSEUDO NUMERICAL METHOD FOR DDPM
In this section, we first compute the corresponding differential equations of diffusion models to build direct connection between DDPMs and numerical methods. As byproduct, we can directly use pre-trained models from DDPMs. After establishing this connection, we provide detailed analyses on the weakness of classical numerical methods. To solve the problems in classical numerical meth- ods, we dive into the structure of numerical methods by dividing their equations into gradient part and transfer part and define pseudo numerical methods by introducing nonlinear transfer parts. We find that DDIMs can be regarded as simple pseudo numerical methods. Then, We explore the pros and cons of different numerical methods and choose the linear multi-step method to make numeri- cal methods faster. Finally, we summarize our findings and analyses and safely propose our novel pseudo numerical methods for diffusion models (PNDMs), which combine our proposed transfer part and the gradient part of the linear multi-step method. Furthermore, we analyze the convergence order of pseudo numerical methods to demonstrate the effectiveness of our methods theoretically.
3.1 FORMULA TRANSFORMATION
According to Song et al. (2020)), the reverse process of DDPMs and DDIMs satisfies:
Et vI (see(zz,)) 1)
Tt = / Vā-1
+ / 11 ā ozc0 + acc.
(8)
Vāz
Here, o, controls the ratio of random noise. If or equals one, Equation (8) represents the reverse process of DDPMs; if a, equals zero, this equation represents the reverse process of DDIMs. And only when a, equals zero, this equation removes the random item and becomes a discrete form of certain ODE. Theoretically, the numerical methods that be used on differential equations with random items are limited. And Song et all (20206) have done enough research in this case. Empirically, Song et all (2020)) have shown that DDIMs have better acceleration effect when the number of total steps is relatively small. Therefore, our work concentrate on the case equals zero.
To find the corresponding ODE of Equation (B), we replace discrete with continuous version t - 8 according to (Song et all 2020a) and change this equation into a differential form, namely, subtract I, from both sides this equation:
co co t)
Ft-5 - F4 = (āt-- - ax)
(Van(Vorss Vor)
Varl
ā)-s)ā + /(1 (1-a))āau @a)
Because is a continuous variable from 0 to t, we can now compute the derivative of the generation ca(m(t).))
data 2y and get that lim Te- aá(()) s(t)
20(0)/1-a(1 Here, a(I) the continuous
6-30
20(E)
version of N like the definition of (t)). Therefore, the corresponding ODE when tends to
2To achieve higher accuracy, more information is not enough. The reason why these methods achieve higher accuracy be found in Appendix B.2
4

---

Published as a conference paper at ICLR 2022
zero of Equation is:
dr a'
x(t) ct(x(t),t)
(10)
dt
(ā(1) 20(1)/1- a()
3.2 CLASSICAL NUMERICAL. METHOD
After getting the target ODE, the easiest way to solve it is through classical numerical methods. However, We notice that classical numerical methods can introduce noticeable noise at high speedup rate, making high-order numerical methods (e.g., Runge-Kutta method) even less effee- tive than DDIMs. This phenomenon is also mentioned in Salimans & Ha (2022). To make better use of numerical methods, we analyze the differences between Equation (II)) and usual differen- tial equations and find main problems when we directly use numerical methods with diffusion models.
The first problem is that the neural network eo and Equa- tion (II)) are well-defined only limited area. Equation (2) shows that the data I; is generated along curve close to an arc. According to Figure 2. most of I; is concentrated in band with a width of around 0.1, namely the red area in Figure D. This means that the neural network eg can- not get enough examples to the noise successfully away from this area. Therefore, and Equation (II),, which con- tains Eg, are only well-defined in this limited area. How- ever, all classical numerical methods generate results along a straight line instead of an arc. The generation process may generate samples away from the well-defined then introduce new errors. In Section A.3 we will give more visualization results to support this.
and
Figure 2: the density distribution of the norm the data.
The second problem is that Equation (ITI) is unbounded at most cases. We find that for most linear variance schedules 3r Equation (II)) tends to infinity when tends to zero (see Appendix A.4), which does not satisfy the condition of numerical methods mentioned in Section A This is an apparent theoretical weakness that previous works have not explored. On the contrary, the original DDPMs and DDIMs, the prediction of the sample and the noise in the data are more and more precise as the index tends zero (see Appendix (..)). This means original diffusion models do make a significant error in the last several steps, whereas using numerical methods on Equation (II)) does. This explains why DDIMs are better than higher-order numerical methods.
3.3 PSEUDO NUMERICAL METHOD ON MANIFOLD
The first problem above shows that we should try to solve our problems on certain manifolds. Here, target manifolds are the high-density region of the data x; of DDPMs, which defined by (x(20, e) = /a + 11-va-- methods solve differential equations on manifolds that have analytic expressions. Unfortunately, it's challenging to use the above expression of manifolds. Because we do not know the target To in the reverse process and random items c are hard to handle, too.
11-āāge, āgc, e (0, 1). Emst & Geriard (1996) show several numerical
In this paper, we design a different way that we make our new equation of denoising process more fits with the equation original DDIMs to make their results share similar data distribution. Firstly, we divide the classical numerical methods into two parts: gradient and transfer parts. The gradient part determines the gradient at each step, while the transfer part generates the result the next step. For example, linear multi-step method can be divided into the gradient part f' = & (55 ft -95 f f f7 - 3s) and the transfer part Zt+5 = Ie + 8f'. , All classical numerical methods have the same linear transfer part, while gradient parts different.
We define those numerical methods which use a nonlinear transfer part as pseudo numerical meth- ods. And an expected transfer part should have the property that when the result from the gradient part is precise, then the result of the transfer part is as close to the manifold as possible and the error of this result small as possible. We find that Equation (H) satisfies such property.
5

---

Published as a conference paper at ICLR 2022
Property 3.1 If the precise noise in 24, then the result of 22-6 from Equation (9) is also precise.
And we put the proof of this property in Appendix Therefore, we use:
ol 6) Vār- Vā
(ā-- at) + V(L ((--a))ā-o))
(11)
as the transfer part and Eg as the gradient part. That if Eg is precise, the result of z also precise, which means that can determine the direction of the denoising process to generate the final results. Therefore, such choice also satisfies the definition of a gradient part. Now, we have our gradient part Eg and transfer part 6.
This combination solves the two problems mentioned above successfully. Firstly, our new transfer parts do not introduce new errors. This property also means that it keeps the results at the next step on the target manifold because generating samples away a kind of error. This shows that we solve the first problem. Secondly, know that the prediction of eg is more and more precise in the reverse process in the above subsection. And our new transfer part can generate precise results according to the precise prediction of eg. Therefore, pseudo numerical methods, while classical numerical methods can introduce obvious error at the last several steps. This shows that we solve the second problem, too. We also find that their combination 6(z,, co (T, t), t, - 1) is just the inference equation used by DDIMs, so DDIMs is simple case of pseudo numerical methods. Here, we define DDIMs DDIMs", emphasizing that is pseudo numerical method.
generation results
more and more precise using
3.4 GRADIENT PART
Because we split numerical methods into two parts, we can use the same gradient part from different classical numerical methods freely (e.g.. linear multi-step method), although we change the transfer part our inference equation. Our theoretical analyses and experiments show that the gradient part from different classical methods can work well with our new transfer part (see Section B.R. H.2). By using the same gradient part of the linear multi-step method, we have:
{
e, = ea (x((t))) t)
Algorithm 1 DDIMs
e z (55e, - 59c, + 37 -9e-38)
1: ~N(0
It+5 = t,,
2: for t = 1.... d,0
(12)
3: Iı
By using the same gradient part of Runge-Kutta Kutta method, we have:
4: end for
5: return
et to(x,tt)
Algorithm 2 PNDMs
F eI,,et, t,, 5))
1: N(0,
c
co(st,t
2: for T T 1,7 2,
It. Ct = (TRK, 1,()
x tf,
4: end for
e = cola,,t+ 2)
(13)
5: for f = T 1.... 1,0
6: Ft, PLMS t
end for
ed co(a2,t+
8: return 20
ef = (e) + 2e? 2e +t)
T 6(x,, e,, 8).
Abbreviate Equation (12) and as T1+6. C: = PLMS( (eplpertt, + 8), Iets.e =
1RK 8).
Here, we have provided three kinds of pseudo numerical methods. Although advanced numerical methods can accelerate the denoising process, some may have to compute the gradient part cg more times at every step, like the Kunge-Kutta method. Propagating forward four times along neural network makes
denoising process slower. However, we find that the linear multi-step method can reuse the result of eg four times and only compute eø once at every step. And theoretical analyses tell us that the Runge-Kutte Kutta and linear multi-ster method have the same convergence order and similar results.

---

Published as a conference paper at ICLR 2022
Therefore, the gradient part of the lin- ear multi-step method and our new transfer part as main pseudo numerical methods for diffusion models (PNDMs). In Table IIl we show the relationship between different nu- merical methods. Here, we can see PNDMs combine the benefits of higher-order classical numerical methods (in the gradient part) and DDIMs (in the transfer part).
order
first
non-first
lincar
forward Euler
linear multi-step,
Runge-Kutta...
nonlinear
DDIM
PNDM
Table The relationship between different numer- ical methods.
3.5 ALGORITIMM
We can provide our whole algorithm of the denoising process of DDIMs now. According Song atall (20221)), the algorithm of the original method satisfies Algorithm M1 And our new algorithm of DNPMs uses the pseudo linear multi-step and pseudo Runge-Kutta method, which satisfies Algo- rithm 2 Here, we cannot use linear multi-step initially because the linear multi-step method cannot start automatically, which needs at least three previous steps' information generate results. So we use the Kutge-Kutta method to compute the first three steps' results and then use the linear multi-step method to calculate the remaining.
We also use the gradient parts of two second-order numerical methods to get another pseudo nu- merical method. We introduce the details of this method in Appendix A.3 We call it S-PNDMs, because its gradient part uses information from two steps at every step. Similarly, we also call our first PNDMs F-PNDMs, which use data from four steps, when we need to distinguish them.
3.6 CONVERGENCE ORDER
Change the transfer part of numerical methods may introduce unknown error. To determine the influence of our new transfer part theoretically, we compute the local and global error between the theoretical result of Equation (IC) xt +6) and our new methods, we find that æ(t +6) FDDIM 5) O(5²) and
(14) If the target ODE satisfies Lipschitz condition and local error Eloca O then there are C' and such that the global error Cglobal satisfies Ggiobal CC& + cl + 2h + ...) I C²8k-1_ And we have that the convergence order is equal to the order of the global error. The detailed proof can be found in Appendix A.d. Therefore, we get the following property:
xx + 8) -SD- (x 6) = -(5*).
Property 3.2 S/F-PNDMss have thinl-order local error and second-order convergent.
EXPERIMENT
4.1 SETUP
We conduct unconditional image generation experiments on four datasets: Cifar10 (32 32) (nizhevsky ct all, 2009), CelebA (64 x 64) cLa 20L)), LSUN-church (256 x 256) and LSUN-bedroom (256 x 256) (YueLaa 2016). According to the analysis in Section B. we can
pre-trained models from prior works in our experiments. The pre-trained models for Cifar10, LSUN- church and LSUN-bedrood
are taken from Ho (202I) and the pre-trained model for CelebA is taken from Song et al, (2U2IT). In these models, the number of total steps N is 1000 and the variance schedule linear variance schedule. And also pre-trained model for C0 which uses cosine variance schedule from improved denoising diffusion probabilistic models (iDDPMs (Nichol
Dhanwa ()2I))
4.2
SAMPLE EFFICIENCY AND QUALITY
To analyze the acceleration effect, test Fenchel Inception Distance (FID (Heusel all 2018) on different datasets under different steps and different numerical methods, including DDIMs, S-PNDMs, F-PNDMs and classical fourth-order numerical methods (FONs) (e.g., Runge-Kutta

---

Published as a conference paper at ICLR 2022
Table 2: Image generation measured in FID Cifar and CelebA. PFs use black box ODE solvers and use the number of score func- tion evaluations as the step of PFs. DDIM is retest of DDIM. The bold results mean the best ones using the
FID step
dataset
10
20
50
100
250
1000
time
model
DDIM
13.4
6.84
4.67
4.16
4.04
Cifar10
13.8
3.89
3.69
3.71
3.72
DDIM
18.5
10.9
6.99
5.52
4.52
4.00
0.337
Cifar10
FON
13.1
7.41
5.26
4.65
4.12
3.71
0.390
(linear)
S-PNDM
11.6
7.56
5.18
4.34
3.91
3,80
0.344
F-PNDM
7.03
5.00
3.95
3.72
3.60
3.70
0.391
same pretrained model. We
DDIM
14.5
8.79
5.86
4,92
4.30
3.69
0.505
use the 50-step, 512 batch an RTX- 3090 to test the computa- tional cost and the column time is the average compu- tational cost per step in sec- onds. And we put the results of standard deviation in Ap-
Cifar10
(cosine)
S-PNDM
8.64
5.77
4.46
3,94
3.71
3.38
0,517
size experiment
F-PNDM
7.05
4.61
3.68
3.53
3.49
3.26
0.595
CelebA
DDIM
17.3
13.7
9.17
6.53
3.51
DDIM
16.9
13.4
8.95
6,36
4.44
3.41
1.237
CelebA
FON
16.0
11.6
8.13
6.70
5.14
4.17
1.431
(linear)
S-PNDM
12.2
9.45
5.69
4.03
3.19
2.99
1.258
F-PNDM
7.71
5.51
3.34
2.81
2.71
2.86
1.433
pendix A.
method and linear multi-step method). On Cifar10 and CelebA, we first provide the results of previ- ous works DDIMs. Then, we use the same pre-trained models to test numerical methods mentioned in this paper and put the results in Cifar 10 CelebA (linear). We also use models from iDDPMs to test nonlinear variance schedules and put the results Cifar10 (cosine). Song et all (202005 do not provide detailed FID results of probability flows (PFs) under different steps, we retest the results using its pretrained models by ourselves.
Efficiency Our two baselines are DDIM and PF. DDIM is simple case of pseudo numerical meth- ods, and PF is a case of classical numerical methods. However, PF uses a much bigger model than DDIM and uses some tricks improve the sample quality. To ensure the experiment's fairness, use fourth-order numerical methods on Equation (II) and the model from DDIM. In Table Q we find that the performance of FON is limited when the number of steps is small. By contrast, our new methods, including S-PNDM and F-PNDM, can improve the generated results regardless of whether the number of steps used is large or small. According Cifar10 / CelebA (linear), F-PNDM can achieve lower FID than 1000 steps DDIM using only 50 steps, making diffusion models 20x faster without losing quality.
DOIM
We draw line chart of computation cost with FID accord- ing to the results of Cifarl0 (linear) above in Figure Be- cause F-PNDM uses the pseudo Runge-Kutte method to generate the first three steps, is slower than other methods at the first several steps. Therefore, S-PNDM can achieve the best FID initially, then F-PNDM becomes the best and the acceleration is significant.
S-NNDM
F FNNND
Quality When the number of steps is relatively big, the results of FON become more and more similar that of pseudo numerical methods. This is because all the meth- ods are solving Equation (II), and their convergent results should be the same. However, pseudo numerical methods still work better using large number of steps empirically. F-PNDM can improve the best FID around 0.4 using pre- trained models and achieves a new SOTA FID score of 2.71 on CelebA, which shows that our work can not only accelerate diffusion models but also improve the sample quality topline. We also notice that the FID results of F-PNDM converge after more than 250 steps. The FID results will fluctuate around value then. This phenomenon is more pronounced when we test our methods on LSUN (see Table EL 8)
187
10P
157
time
Figure 3: The FID results under dif- ferent computation costs and different numerical methods unit of time is the computational cost of 1-step DDIM, which is 0.337s.
C0 The
According to Cifar10 (cosine), the cosine variance schedule can lower FID using relatively large number of steps. More analyses about variance schedule can be found in Appendix A. What's s more, test our methods other datasets and provide image results in Appendix A. We can draw similar conclusions on our methods' acceleration and sampling quality, regardless of the datasets and the size of the images.
FID results in Appendix A and
8

---

Published as a conference paper at ICLR 2022
3.5
3.4
006
00:4
6B 20:4000
Fauile
Foa-23
PRDM-
PM04411 t0
PA00-22
0034-0003
1MG
Mep
step
C6:55
BOMILE
ED
FONCO
FON22
Figure 4: The upper part shows the change of norm with the number of steps using different methods and different steps. The lower part shows the generation curves two points using different methods and different steps. DDIM-n means n-step DDIM method. Experiments in this subsection all use the Cifarl0 dataset and we use the 1000-step DDIM's s result as our target result.
4.3 SAMPLE ON MANIFOLDS
Here, we design visualization experiments to show the effort of our new methods and support our analyses. Because it is hard to visualize high-dimensional data, we use the change of global char- acteristic norm and local characteristic pixel to show the change of the data under different steps. For pixel, we randomly choose two positions p .p². Then for a series of images TT, -r- derived from the reverse process,
denote yy the value of Fyz position p². Then draw polyline (yi- y4 : in R.2 For norm, we first count the distribution of the norm of the training datasets under different steps and use this make heat map as the background. After that, we draw the norm of our generated results using different methods and steps above this heat map.
In Figure EL we can see that the FON may run far away from the high-density area of the data, which explains why FON may introduce noticeable noise. However, PNDM can avoid this problem and appropriately fit the target result. More visualization results supporting our analysis can be found in Appendix A.I. What's more, we design toy example to test our new methods without the influence of neural networks and get similar conclusions as to the real cases above. We put the detailed results in Appendix A.S.
5 DISCUSSION
In this paper, we provided DNPMs, new numerical method suitable for solving the corresponding ODES of DDPMs. DNPMs can generate high-quality images using fewer steps without loss of qual- ity successfully. Based the idea of this work, further improvement can be explored in future works: 1) find a better variance schedule for DNPMs: although we tested DNPMs on linear variance schedule and cosine variance schedule in this work, there might be another variance schedule more suitable for our proposed numerical methods. 2) Find higher-order convergent pseudo numerical methods: we analyzed the convergence order of S/F- DNPMs, which are both second-order conver- gent. However, F-DNPMs achieve better FID than S- S-DNPMs in most cases. We think this is because the result between our transfer part and target ODE has higher-order error, which limits the con- vergence order of F-DNPMs. This error from the change of the transfer part is theoretical but does not influence the quality of images according to the property of Equation (LI). However, making the transfer part higher-order convergent and finding the influence of such change is still exciting and needs more research. 3) Extend PNDMs to more general applications: when we proved the convergence order of DNPMs. found that other kinds of transfer parts keep the convergence order unchanged too, which means that pseudo numerical methods can be used on more general applications, like certain neural ODEs (Chen all 2019 Dupont al] 2019).

---

Published as a conference paper at ICLR 2022
REFERENCES
Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982. ISSN 03044149. doi: 10.10160030044441222000051111111
Jacob Austin, Daniel D. Johnson, Jonathan Ho. Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete spaees, 2021.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. arXiv preprint, 2020.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Dif-
ferential Equations. orXiv:1806.077. 07366 [cs, stat], December 2019.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models. CoRR, htss2108.02238814 2021.
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented Neural ODEs. arXiv: [cs, Stat), October 2019.
Hairer Ernst and Wanner Gerhard. Solving Ordinary Differential Equations II: Stiff and Differential- Algebraic Problems. Springer Series in Computational Mathematics 14. Springer- Verlag Berlin Heidelherg, 2 edition, 1996.
lan Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. 1rXxiv1706.000 [es, stat), January 2018.
Jonathan Ho, Ajay Jain, and Picter Abbecl. Denoising diffusion probabilistic models. arXiv, 256, 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax Flows and Multinomial Diffusion: Towards Non-Autoreeresssive Language Models. arXiv preprint, 2021.
Myconghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff- TTS: A Denoising Diffusson Model for Text-to-Speech. arXiv:2104.0144 [cs, eess), April 2021.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and I1-Chul Moon. Score Matching Model for Unbounded Data Score. arXiv:2106.05527 [cs, stat], August 2021.
Zhifeng Kong and Wei Ping. On Fast Sampling of Diffusion Probabilistic Models. arXiv:2106.00132 [es], June 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A Versatile Diffusion Model for Audio Synthesis. / [es, eess, stat), March 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Max W. Y. Lam, Jun Wang, Rongjie Huang, Dan Su, and Dong Yu. Bilateral Denoising Diffusion Models. arXiv:2108.115: 11514 [cs, eess], August 2021.
Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, Peng Liu, and Zhou Zhao. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. arXiv preprint arXiv:2 10..020550244 2, 2021.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) December 2015.
10

---

Published as a conference paper at ICLR 2022
Shitong Luo and Wei Hu. Diffusion Probabilistic Models for 3D Point Cloud Generation. arXiv preprint, 2021.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. CoRR, abs/2102.09672, 2021.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.0766 2021.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep Unsuper- vised Learning using Nonequilibrium Thermodynamics. pp. 10, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. arXiv preprint, pp. 1-19, 2020a.
Yang Song and Stefano Ermon. Improved Techniques for Training Score-Based Generative Models. arXiv:2006. 09010 [es, stat], October 2020a.
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribu- tion. arXiv: 1907.05600 [es, stat), October 2020b.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. pp. 1-32, 2020b.
Sauer Timothy. Numerical Analysis. Pearson, 3 edition, 2017. ISBN 201 97801 44694444 013469645X.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based Generative Modeling in Latent Space. arXiv:2106.05531 [es, stat], June 2021.
Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning Efficiently Sample from Diffusion Probabilistic Models. CoRR, abs/2106.0338 2021.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN:
Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop. arXiv:1506.03365 [cs], June 2016.
A APPENDIX
A.1 RELATED WORK
DDPMs have been well developed in the last few years. Some works concentrate improving the quality and the speed of DDPMs and making DDPMs more practical. [Song et al. (2020a) introduce new inference equations accelerate DDPMs. Nichol & Dhanwal (2021), Watson all (2124) and
Kong & Ping (202T) choose to find better variance schedules to improve the images quality. Vahdai et atal (2021) combine the advantages of DDPMs and Variational Autoencoders and get better results. Furthermore, Kim all (2021) try to solve an existing bottleneck that the inference equations of DDPMs are unbounded in some situations.
Song et all (20205) find the similarity between DDPMs and noise conditional score networks (NC- SNs (Song Ermon, 20206), which is that they both use process similar to Langevin dynamics to produce samples. Therefore, some works (Song & Ermon, 202(0; Kim et all 202.)) that can improve the results of NCSNs also can be used DDPMs. Additional, Song et al. (20200) com- bine DDPMs and NCSNs under the framework of neural differential equations (Chen al 2019
Dupont et all, 211 9). Therefore, numerical methods widely used in neural differential equations can also be applied accelerate DDPMs. Our work successfully combines the advantages of 'Song all (20208) and Song et all (20206). We use transfer part from DDIMs and use different gradient parts from different numerical methods. Although we and Song et all (20220) solve certain differential
11

---

Published as a conference paper at ICLR 2022
equations derived from DDPMs,
different target different equations and different numerical methods, which get better results.
The application of DDPMs is not limited to unconditional image generation. Some works apply DDPMs to various types of data successfully, including text-to-speech (Chen et all 2020; Lam et all (021),, singing voice (Liu et all 202T), 3D Point Cloud (Luo & Hu, 202II, text generation Austin el all 2021). Additionally, DDPMs can also be used to generate conditional samples, too
eong et al, 2021: Chol (02T))
A.2 CONVERGENT ORDER OF METHOD
We use the forward Euler method and linear multi-step method to show what is the order of a method. Assume that 2G is precise and compute the error at T248- For forward Euler method, have:
Ct,6 r(t + 6) 21,5
8² ( - (x(() + (x(ettt))
= x(t) + )f(x((t),t) +
2
(15)
8 (c) -
8²
2 M
Here, we assume that l" continuous, so f" is bounded close area. x(1 + 8) is the precise result and F5,5 is the numerical result from / to / + 8.
For linear multi-step method, we have:
e,, (1 + 8) It.55
8² (x(t). + +...+ 8² x(t) + IT f()(t),t)+ 2
f (z(1,,))
4!
(16)
a(())
+
Σ bed
Σ (-(s
(- - 1)0)*
8th (a(0))))
(k))
2=2
k==
=0(85) Here, {6,} satisfy Ed= b, = and the following equations for j E ... ,33:
1
(-1) (-2) + (-3)bb1 =
(17)
j+1
We call the error at Ft,s local error, and the error at Tt+MM Mr (M is big enough but finite) global error. Assume the local error of our method has order k+1 and the target ODE satisfies Lipschitz condition, then:
x0t + M5) F4,As6 S|a(t + M8) F14(M-188. + [c+ x1,
Se++(M + l (M 1)6) (1s-61
Se(+(M-1)666 + 1 C2+ +...
508++1 (1 1 4 ... + e 11)kh
(18)
iLB -1
- S Cok+1 +1 eLb
-c
L8
=0(8)).
Therefore, the global error will be one order lower than the local error. From Equation (15), we can the forward Euler method has local emror ( and global error ( call it the first-order numerical method. And the linear multi-step method has local error ((85) and global error ((*)),
and we call it the fourth-order numerical method.
In addition, assuming that numerical method has kth-order global error, we can compute the con- vergence speed of this numerical method:
x Z+4 ztt + T)
25
(28)*
lim 80 x 25 z(t
2
(19)
T
5k
12

---

Published as a conference paper at ICLR 2022
Here, 24 2 is the result at t+T and move every step. This shows that the fourth-order method can converge to the exact solution faster than the first-order method when 8 - 0, which means that we a bigger iteration interval ò to achieve similar global enorr and bigger iteration interval means that we can iterate fewer times to get results with high quality.
A.3 PSEUDO SECOND-ORDER METHOD
We introduce two second-order numerical methods. First is improved Euler method satisfying:
h f),,,t
k2 f(z + 8k1, + 8) I++6 = Iz + 5 (Ai + k2) 2
(20)
Second is another linear multi-step method called second-order linear multi-step method satisfying:
T++5 2i (3f: ft-6)
(21)
And the corresponding pseudo improved Euler methods satisfying:
e (x(.
r olIe; t,,t,
e eØ ( 8)
(22)
ef = { + Et)
T&t5 6(z,, t,t++
Pseudo second-order linear multi-step method satisfying:
(x)e
é = { ct-
(23)
It+5 = HlIe, 1.1
Similar to what we do to get F-PNDMs, We com- bine them to get S-PNDMs. Abbreviate Equation
Algorithm 3 S-PNDMS
1: 2r N).
12 and (25) as
2: = T do
Fe+ó; Gt = PIF((a,, (et) 8),
3: 2t, 6: P1E
4: end for
Z145, = t
5: for
= - ... 1, do
6: Jz; 6t PLMS (I++1, (ephps,, 1+
7: end for
8: return 370
A.4 THE EXISTENCE OF A DERIVATIVE
Because is usually obtained by multiplying linear variance schedule B, So have
āt at at +b6+c
(24)
and og = 1, 80 c = 0. Now we have
8 Ft
lim
6-+0
04-5 ai
Zt
c0(x(tt)
= -lim
6-+0
8
Vau (v 04-8 Vo.)
Vāi(V(1
2) v(1 az ar)āā-6)
(25)
0t-8 Ci
Zt
c0 E0(I),t)
= (oar+
It.
Ed (0(
= -lim
6-+0
8
2nr,
2 Cgaa
2 2 a,āg
c
(e(x,)))
co(I).t)
=(2at + b)ā
(2at + +b(
20,
2-1 ā,00
vI αi
13

---

Published as a conference paper at ICLR 2022
(=u is well-defined, b must equal to zero, or (2at + b) (e(().)) vT-a; will tend to infinity. This is a strong condition that most variance schedules do not satisfy. In practice, DDPMs can choose the variance schedule very freely. This means that treating DDPMs as ODEs directly is not proper and has theoretical weakness.
To make lims-+0 St -t-6
A.5 RELATIONSHIP BETWEEN i, 6O AND Ft.
Relationship between & and Eg. In Figure 5 whether in the co domain or the sample/inage domain when the step-index tends to zero. Therefore, say that the noise becomes more and more precise when step, namely t, tends to zero.
that the denoising process tends to converge,
opsilor
08ts
10"
10a
200
400
600
800
1000
step
Figure 5: The norm the difference between two adjacent terms under different steps
Relationship between 6 and To prove Property 3.1. assume that x; V v aie, NN the neural network and cg = NN(a,, t). Because we assume that the gradient part is precise, then have Eg e. Then for all t' < t, we have:
E1 /1 ōcco
xe = Vor
+ V Cyyt8
Vā
aaao vI
-
vI age
(26)
- (rr-arre(zz
Vor + v1 āvc.
Here, we can find that Zy Vārzo + VI- Epe is also precise, so Property [.] is true.
A.6 ORDER ANALYSIS PSEUDO METHOD
For the convenience of theoretical analysis, we generalize the problem. Let ((x(t), 6, = f(x(t),t,8) + g(t,t)( ())(()t(()))) t) and have the property f(z(t),t,0) gt, 0) Then
have:
<(1) z(0) Σ
E(v(t (y(t+ 8) y(t))
6-40
=x(0)= Σ (S(x(1),,,)) +((1,))((((1)
(27)
= =x(0)= + /.
af
0g (x()),t,))+
06 (t, (t,)))(((())))
06
Now, Equation (10) becomes a special case of this more general version and, in this special case, have:
( -
(Vale+))
f(x(t),t,0)
x(())
Va
(28)
v(1 /(t=a(t)aa(+a
0(t,0) VI- alt 0
va(t)
14

---

Published as a conference paper at ICLR 2022
Now, compute the local error of S-PNDMs. We first compute the theoretical and numerical results of different numerical methods. We have:
xtt + 6)
0S
og 06 (t. ()((((().))) +
=x(t)= +
(a(t), t,) +
08
82 0f
00 (()))(((()))))) 0660)
(z)t),t,0)
06
05
og (t, (e(x(())) + +0(8))+
(29)
=x())
(o(t),t,0)
08
06
82
f (r(1),t0))
041
0f
(x(1), + (.)))((((())))) +
(x (1), (,
2
000l
060r
06
8
0gg
8g
(4,0)(((())))
(1,0)
2
000t
08
and
(t. 5)
=x(1) + (100000000000000((00000000000000
0f =x(1) + (o(t),t,0))
808
03
08 (0)0000))))))
06
2 08
(6 00 08
(1,0) (10) +
82 8
(t,0) ((((0))))+((((+++000000)0))0
2 08
=x(1) + 0f (z(1, 4,0
8 0s
08 (,(0)440)+0()))
06
8 0g
(4,0 + 8 0
(((x)()))++c((d)) +0), 8), 6))
05
2 062
=x(t) + 05 (,(t),,,0) 1,0) +
82 O
(30)
2 052 (2((),,,)++0())))
06
(8 0g
82 8g
(4,0) +
(1,0)
2 (c(x(t).t+<<(((()))) (c(x((),)))
08
2 08
0f
(tt)),,0) + 8² j2f
=x(t) +
082 (z(t),t.0++0())))
08
00 (t.0)
0g
) (c(((t)) - 82 8gg
2 082 (t. ()(())))).
08
=x(1) +
0s
(x())
0g
(1, ((()))(()))))) + +08)))+
08
26
82 0
0g (z(1),
0 (t, (())))
age (1, (()))(((()))))
08
05
Then we compute the difference between the theoretical and numerical results. We have:
xt
+ 5)
Is..NDMM (x + 66)
82
f2
Sf
080r (x(t), 0f 08
8P
(=(0) +
(x(t), 0) +
2
080t
08
(31)
82
.gg
8g
f (x) 1,
0g
(t, 1)+0)
(1,0) ((()))))+0(())
2
080t
05
880%
06
15

---

Published as a conference paper at ICLR 2022
In this special case, compute the derivatives of some items needed in Equation (BI). We have:
0g
a
(11 a(1)) + 5)
38
(t,,)
05
05
V a(t
Va(t)
(32)
-a'(t+
VI a(t)a'(t (t))
2 - a(((+6))
2 (t +0)
0f
Va(l+
(o(1,
x(1))
08
08
Va
(33)
d'(++ z(t)
2/a(t)a(t + 8)
f
6)
r(1) + -d'(t 8)2
4Va(t)a(t 5)3 2())6==
(34)
052
2/a(t)a +
8f
(z(t), = at)
x(t) a'(t)²
=(t) ((t)
(35)
060t
2a(t)
2a(t)2
0f (x))),,,0) d'(t)
(36)
080r
20((1)
0g9
-a((1
V1 V1-a a())a"(++))
(1, 6) 6-0 2/1 a((+)))
082
2/a(t)a(t++ + 5)
-d((1 8)2
V1- a1-a(t)a'(( (t + +0)
1s=0
4V1 - a(((+
4Va(1)a(( + 873
(37)
d1(()
VI V1- a(t)a"
2/1 (()() a(())
2Va(t)a(t)
-((1)2
VI a1)a(t)'((1)
4/1 a(t))
4Va(t)a(t)3
8g
0
ad((t
VT-a((ja(()
(t, 6=0
0s0t
01
2/1 ((t))
2/a(t)a(t)
(1)
a(t)a"(t)
(38)
2V1-a((1))
2 2Va(()a(1)
(t))
a(t)al(1)
d'(t)²
V1-a(t)
2a(1)2
4 4 a())
Now we can compute the final result of Equation (1I). We split it into three parts and the values of the first two terms. We have:
0²
f2
8z
(x) Of (z(t),
052 (x(t),t,0)+ +
a80t
080x
08
(t)
20(1) x(t) - d'(1)
2x())=(()) x(t)
aa)t) r(f) a((t)²
2a(1) da(1)² ta())==(1)) a(1)
+ 20(() 20(()) x(t)
at(t)
(39)
2a(1)
-0 and
92g
8g
84 (z(t), t,
Ög
(t.
0) +
(t)
060t
082
060x
a6
a())
a' al)))
a(tt)
('(t)
v 11- -TTaaaaa'( (1)
4a(1)²
4a(t) /1 att)
2a(t)
2/11 a((t)
20(())
(40)
a' a'(1)
a'(t)
-a'(t)
4a(t)² / a(t)
2a(1)
2/11 - -a()aatt))
-0
16

---

Published as a conference paper at ICLR 2022
Finally, get the final result of Equation (III):
z(t + 6) TS-PNDM (z 6) = O(8))
(41)
And the computation the convergence order of F-PNDMs is similar, and we ignore it here. There- fore, Property (32 is true.
A.7 VARIANCE SCHEDULE
According to Cifarl0 10 (cosine) in Table 2, PNDMs can be used on both linear variance schedule and cosine variance schedule. However, also notice cosine variance schedule make FID lower when we use relatively big generation steps, but the effort is limited when the number of steps is small. F-PNDM uses information from four consecutive steps, so the smoothness of the schedule is more important for F-PNDM than DDIM. According to this experiment, our work can be used with works that pay attention to variance schedules to improve the acceleration effect further. However, variance schedule that fits pseudo numerical methods better remains to be found in further work.
A.8 TOY EXAMPLE
Here, we design a toy example to test our new methods without the influence neural networks. We randomly generate the initial input equations (0(x)= = (sin æ [0], cos x [1] to replace the neural networks in real cases. Let ø in Equation (I is unchanged and a; a(l) 1 L, then we get:
(m1, m2), my (O, 1) and use simple analytic
(o(z),
Var-
(ār-6 - (it)
21
Vaslv 8)- + v(1 ar)āz_8) E
Vā
(42)
1 (1 (f-
V1- 21
VI- V(t 61 t)+ + /t(1 - (t 51) c
Here, we use three different numerical methods to generate zo.
For DDIM, we have:
Tt-6 x4 + (a(=),
(43)
For FON, according to Equation (IC), we have:
e = a't))
Et
(0(x))
2ā(t)
20(() / a(t)
co(x))
(44)
2(1 t)
2(1 (1-t)vv/
8
Tt- 8 = +
(55e)- - 59e'4s + 375/426 9e(436)
24
For F-PNDM, we have:
e' (55ca (z,) - 59cg(1146)) + 37 (c0(21436))
24
(45)
Zt-6 = 2, c'
Then we draw the corresponding generation curves in Figure R We find that the result is similar to the real cases. The main difference here is that FON can correct its results while the real case cannot. The reason is that the gradient is well-defined everywhere, while real cases, the gradient meaningful the high-density region of the data z, of DDPMs.
A.9 MORE FID RESULTS
Here, We provide our more detailed FID results on Cifar10, CelebA, LSUN-church and LSUN- bedroom in Table & g R
17

---

Published as a conference paper at ICLR 2022
3DHM-1200
DDHM-100
B3/M-000
23
DON--
FOM-S
PHDM-5
DON 1000
18
D3IN-1
BD1A30
Figure 6:
The generation curve of our toy example.
steps
5
10
20
25
40
50
100
125
200
250
500
1000
DDIM
13.4
6.84
4.67
4.16
4.04
DDIM
44.5
18.5
10.9
9.61
7.65
6.99
5.52
5.19
4.69
4.52
4.17
4.00
FON
98.0
13.1
7.41
6.41
5.50
5.26
4.65
4.54
4.23
4.12
3.84
3.71
S-PNDM
22.8
11.6
7.56
6.79
5.57
5.18
4.34
4.18
3.97
3.91
3.81
3,80
F-PNDM
13.9
7.03
5.00
4.76
4.10
3.95
3.72
3.64
3.60
3.60
3.64
3.70
D
28.7
14.5
8.79
7.83
6.41
5,86
4.92
4.75
4.42
4.30
3.98
3.69
S-PNDM
18.3
8.64
5.77
5.45
4.76
4.46
3.94
3.85
3.69
3.71
3.60
3.38
F-NDDM
18.2
7.05
4.61
4.32
3.85
3.68
3.53
3.46
3.47
3.49
3.44
3.26
Table 3: Cifar10 image generation measured in FID. The upper part uses linear variance schedule and the bottom half uses cosine variance schedule. The first line shows the FID provided by Song etal (2(22R))
steps
10
20
25
40
50
100
125
200
250
500
1000
DDIM
17.3
13.7
9.17
6.53
3.51
DDIM
24.4
16.9
13.4
12.3
9.99
8.95
6.36
5.74
4.78
4.44
3.75
3.41
FON
60.2
16.0
11.6
10.6
8.89
8.13
6.70
6.28
5.45
5.14
4.49
4.17
S-PNDM
15.2
12.2
9.45
8.42
6.50
5.69
4.03
3.72
3.30
3.19
3.01
2.99
F-PNDM
11.3
7.71
5.51
4.75
3.67
3.34
2.81
2.75
2.71
2.71
2.77
2.86
Table 4: CelebA image generation measured in FID. All of them use linear variance schedule.
steps
5
10
20
25
40
50
100
125
200
250
DDIM
19.5
12.5
10.8
10.6
*DIM
48.8
18.8
11.7
11.0
10.1
10.0
9.84
9,83
9.85
9.88
S-PNDM
20.5
11.8
9.20
9.13
9.31
9.49
9.82
9.88
10.0
10.0
F-PNDM
14.8
8.69
9.13
9.33
9.69
9.89
10.1
9.99
10.1
10.1
Table 5: LSUN- image generation measured in FID. All of them use linear variance schedule.
18

---

Published as a conference paper at ICLR 2022
steps
10
20
25
40
50
100
125
200
250
DDIM
17.0
8.89
6.75
6.62
DDIM
51.3
16.4
8.47
7.41
6.27
6.05
5.97
6.03
6.23
6.32
S-PNDM
18.1
10.2
6.50
6.02
5.74
5.81
6.29
6.44
6.69
6.75
F-PNDM
12.6
6.99
5.68
5.74
6.17
6.44
6.91
6.96
7.03
6.92
Table LSUN-bedroom image generation measured in FID. All of them use linear variance sched- ule.
A. MORE IMAGE RESULTS
Here, we show more generated images on Cifar10, CelebA, LSUN-church and LSUN-bedroom in
Figure 01 B S I
I.. T2.
10
20
50
100
250
500
5
10
20
50
100
250
500
DDIM
FON
PNDM
DDIM
FON
PNDM
Figure 7: 5, 10, 20, 50, 100, 250, 500-steps generated results using DDIMs, classical numerical
methods and PNDMs on C0
10
20
50
100
250
500
DDIM
FON
3
PNDM
DDIM
FON
PNDM
Figure 8: 5, 10, 20, 50, 100, 250, 500-steps generated results using DDIMs, classical numerical
methods and PNDMs on CelebA.
19

---

Published as a conference paper at ICLR 2022
Figure 9: Generated images of PNDMs on Cifar10.
Figure 10: Generated images of PNDMS on CclebA.
20

---

Published as a conference paper at ICLR 2022
20
100
FON
PNDM
10
20
50
100
FON
PNDM
10
20
50
100
DCIM
FON
PNDM
Figure 11: 5, 10, 20, 50, 100-steps generated results using DDIMs, classical numerical methods and PNDMs on LhUN-church.
21

---

Published as a conference paper at ICLR 2022
100
FON
H
H
PKDM
10
20
50
100
DDIM
FON
PNDM
20
100
DDIM
Figure 12: 5, 10. 20, 50, 100-steps generated results using DDIMs, classical numerical methods and PNDMs on LSUN-bedroom.
22

---

Published as a conference paper at ICLR 2022
A.11 MORE VISUALIZATION RESULTS
Here, we put more visualization results similar to Figure E in Figure I
30:4-5
D0H-10
B3
7O45
FON-10
T3N-22
-M0963
PROM LS
18
PND 20
DOIN-000
04
4.0
412
475
410
4/2
D2
POR
PON
34
38
DOIN-10
DD4920
20865
FOR-13
FON-20
28
PNDM-25
PADAA-40
PLDN-5D
3084-1022
D2IN-100
001551000
628
600
120
330
123
156
120
1.0
130
Figure 13: Visualization results under 5, 10, 20, 25, 40 and 50 steps.
A. 12
FID RESULT WITH STANDARD DEVIATION
Here, report the mean and standard deviation of FID results, tested over four sampling runs.
FID step
dataset
10
20
50
100
250
1000
model
DDIM
18,50=1006
10.86:.08
6.95+.04
5.49+t.06
4.52+.02
4.02+E.04
Cifar 10 (linear)
FON
13.00
7.33+.06
5.24
4.64+.04
4.12+.03
3.73+.03
S-PNDM F-PNDM
11.58.1.10
7.53-L.07
5.15
4.34.1.03
3.93
3.83.1.03
6.12:1.07
5.04.1.07
4.011.02
3.75.L.04
3.67
3.78.1.04
Table Image generation measured in FID Cifar10. DDIM* means kind of pseudo numerical method and also a retest of DDIM.
23


Reviews:
[{"review_id": "wbh8Qpg1358", "reviewer": "Reviewer_HRpL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I thank the authors for this submission. I believe there is value in this work both from a theoretical and practical perspective. In general, I am willing to accept the paper. I have two main suggestions:\n\n1. Please double-check the writing and notations. Some sentences are hard to understand and equations contain errors, e.g. Eq. 2, hat{alpha}_t, wrong index.\n\n2. As DDPMs are relatively new, I was expecting a bit more elaborate introduction.\n\n3. I was honestly quite disappointed with the presentation at times. Some equations are stated without any intuition. If you are short of space, I would rather move some of the experimental parts to a supplementary.\n\n4. Sec. 4.3 and in general, I would also work with toy problems and easily visualizable data. This will provide additional insight for the reader.", "summary_of_the_paper": "This paper introduces a framework to treat Denoising Diffusion Probabilistic Models (DDPMs) as solving differential equations on manifolds. This goal is faster sampling without significant loss of quality.", "summary_of_the_review": "I am not very familiar with DDPMs but as far as I can tell, the numerical technique introduced makes sense and leads to improved results. I, therefore, recommend acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jDHwJB3qaw", "reviewer": "Reviewer_jhKu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The beginning of the article (Section 1 and Section 2) is well written and presents the motivation behind the introduction of these pseudo numerical methods in a very didactic way. However, Section 3 needs a lot of polishing. These are some comments and suggestions: \n\n1. The contribution of this work is based on the assumption that $\\sigma_t = 0$. The conditions required to satisfy this assumption deserve to be commented.\n\n\n2. The mention of the manifolds is very implicit and requires to be clarified: what is the definition of the manifold defined at the beginning of Section 3.2?\n\n3. The paragraph at the beginning of Section 3.3, supposed to give the intuition behind the introduction of the transfer part defined in (11), is not clear and should be reformulated. For example, what do you mean by ‘We find that Equation (9) has the property that if ϵ is the precise noise in $x_{t}$, then the result of $x_{t−\\delta}$ is also precise, no matter how big $\\delta$ is’?\n\n4. In Algorithm 2, PLMS and PRK were not defined\n\n\n5. It would be preferable to put the theoretical results of Section 3.6 in a clearly stated theorem. This would highlight the theoretical contributions of this work. \n\n\n\n\n\n\n", "summary_of_the_paper": "This article studies pseudo numerical methods that improve and accelerate upon exiting numerical methods for Denoising Diffusion Probabilistic Models (DDPMs). The crux of this work is the observation that existing work on this topic does not take into consideration the structure of the high-density region of the data. ", "summary_of_the_review": "The novelty of the methodology presented in this paper qualifies this work to be accepted to the conference conditionally to improve the clarity of Section 3. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "R924us_t71b", "reviewer": "Reviewer_urhS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduces a class of pseudo numerical methods for DDPM, this is based on a previous work that already establishes the relationship between DDPMs and a certain class of differential equations on manifolds. The pseudo numerical methods separate the gradient and transfer step of classical numerical algorithms, and for each part choose the best of both worlds. With this they are able to provide faster converengence and efficient update steps because gradients do not need to be recomputed for every step. While I would not guarantee that methods do not exist already in classical optimization, I have not seen them in any related application. \n\nThe paper is overall well written, and the contribution and main ideas are explained clearly. Starting at Section 3.3 (to 4) the derivations become slightly confusing, and it takes a lot of referencing back to find all the variable names again. A legend for the variables and a bit more redundant explanation would probably help here. The same goes for the derivations in the appendix which I could not follow completely. \n\nExperiments are done on four datasets with different resolutions using pretrained models for the manifold description. The results show that the introduce pseudo numerical methods converge much faster and provide better results in less iterations than the previous DDIM method and classical numerical methods. The qualitative examples in the appendix look good, but some of them are too smooth to compete with general sota generators (I am not sure if the smoothness here is related to the pretrained models that were used?). \n\nMinor comments:\n- The authors claim that their implementation is in the supplementary. I was not able to find any supplementary (except the appendix) but I am honestly not sure if this is a failure of me using OpenReview... The implementation should definitely be published in the end though. \n- I think it is bad practice to move the related work section in the appendix. \n- The large figures in the appendix are very hard to understand because the subfigures are not separately titled. \n- There are some grammatical errors throughout the text which should be proofread again. ", "summary_of_the_paper": "The paper proposes a new efficient method for denoising diffusion probabilistic models (DDPM) (generative models that optimize for the closest solution on a manifold) based on the observation that this can be seen a solving a set of differential equations on a manifold. This allows efficient pseudo numerical methods to be applied here which have many advantageous properties over classical optimization methods, including less optimization steps and guaranteed manifold solutions due to separating the gradient part from the transfer part in the optimization. Results are shown on four datasets comparing to two reasonable but not sota baselines. ", "summary_of_the_review": "The idea of separating the gradient and transfer part is (to the best of my knowledge, I am not an expert though) novel, and I can see many applications besides the ones proposed in this paper. The shown results might not be exactly state-of-the-art in terms of generating images, but show clear advantages over traditional numerical methods. Therefore I recommend accept. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "4NDo13d7bhU", "reviewer": "Reviewer_c9bY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Clarity.\nOverall, I believe that the clarity of the submission should be enhanced.\n\nFirst, the introduction can be improved to better stress the motivation of the submission. If I understand correctly, this work builds on Probability Flows (Song et al., 2020), which leverage the existence of a deterministic process whose trajectories have the same densities as the original diffusion process. This deterministic process satisfies an ODE that depends on the score original drift and diffusion terms but also on the score function. Consequently classical numerical ODE solvers (e.g. RK) can be leveraged to sample data from the probabilistic model. Authors then state that results obtained via this approach are subpar and suggest that this is due to the solvers tendency to sample data far from the data manifold.\n\n1/ Why is this true? Would be necessary to give some intuition and to refer to a theoretical analysis\n\n2/ What is the precise problem? Is the issue that the model oversample the data distribution's tail (i.e. fails to fit the distribution properly) or that the numerical methods fail (in what sense?) in these areas as the score is undefined (or hard to estimate)? SMLD (Song and Ermon, 2020)'s motivation to inject noise is built on the latter.\n\nThe authors then provide \"pseudo numerical methods for diffusion models (PNDMs)\" which produces trajectories that \"iterate data on the high-density region of the data\", hence tackling the aforementioned issue.\n\nAlso, perhaps this is question of taste, but I believe that the Background Section (and most of the paper)'s clarity could be greatly improved by taking the perspective of Song et al. (2020b), that is, a continuous diffusion process (forward / perturbating data) and the associated reverse diffusion process (generating data).\n\nAdditionally, Section 3.1 is challenging to follow. Would perhaps be better to put less equations but spend more time explaining why and how they matter.\n\n## Strengths\nFirst, the proposed method is conceptually simple and showcase previously proposed methods (DDIMs) as a special case where the update is a (first-order) Euler step.\n\nThen, the submission shows strong empirical results on common datasets like CelebA, with faster convergences or significantly better FID (for the same number of steps) yielding SOTA. Authors report a ~x20 speed-up wrt DDIM, but this in number of steps and fourth order methods trade-off convergence speed for computational cost. Figure 3 suggests that the speed-up is around x15 for CIFAR which is still significant (although it would ideal to report the runtime directly in Table 2).\n\nFinally, Figure 4 is quite nice as it empirically illustrate the proposed method ability to sample trajectories that like closer to the data manifold, which was the original motivation.\n\n## Weaknesses\nI think that the main weakness is the writing.\nAlthough I believe that Section 4.3 would deserve a deeper empirical analysis as it is directly investigating the core motivation of this submission.\n\n## Relation to prior work\n- Perhaps worth citing Sohl-Dickstein et al., 2015 in Section 1?\n- Citations for Runge-Kutta and the Linear Multi-Step methods appear to be missing.\n- It is not entirely clear to me what method is meant by Probability Flows (Song et al., 2020b), is it with Variance Exploding SDE (SMLD) or Variance Exploding SDE (SMLD) (cf Table 1 from that paper).\n\n\n## Additional feedback.\n- \"However, classical numerical methods (Sauer, 2017) have problems when they are applied to DDPMs.\" -> What class of numerical methods? The Euler and RK methods are ODE solvers, although extensions exist for SDEs. It is unclear how they can be applied to DDPMs. Or is it implicitly implied that they are used for the the corresponding deterministic process (probability flow)?\n- \"to iterate our data on the high-density region\" -> would suggest to reformulate\n- Eq 3: $\\epsilon_\\theta$ is not defined. Would advise to do so so for the paper to be self-contained, especially as $\\epsilon_\\theta$ is used through the entire paper.\n- Table 2: Error bars / confidence intervals are missing. Bold is not defined. As methods have different computational cost per step would be very useful to additionally show this metric.\n- Figure 3: Time has no unit.\n- Figure 4: Axis names are missing.\n", "summary_of_the_paper": "Highlighting the high computational complexity for sampling from Denoising Diffusion Probabilistic Models (DDPMs) (e.g. wrt GANs), authors build on the connection between diffusion processes and ODEs to propose efficient (pseudo-)numerical methods so as to sample data from the data manifold. The main idea is to combine the discrete update proposed in DDIMs, with a fourth-order gradient estimation given by the Runge-Kutta or linear multi-step methods. The motivation being that such gradient estimator should yield trajectories that stay closer to the data manifold. They empirically assess their method(s) on CIFAR10 and CelebA in terms of sample quality (measured by FID), and show that they get a ~x20 speed-up wrt DDIMs or a significant improvement in FID with the same number of steps.\n", "summary_of_the_review": "I personally find this submission interesting and significant, yet believe that clarity needs to be improved to enable readers take the most of the paper's insights.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new DDPM model based on solving differential equations on a manifold.  The resulting numerics appear to be favorable, with faster performance than past models.

Most of the reviews thought the main result was of interest and were impressed with the performance.  Reviewer c9bY points out some challenging issues and analytical questions that remain unanswered in the text; they also have some simpler textual revisions that seem less important.

In general, this paper has the misfortune of receiving reviews whose confidence appears to be low.  While partially this is a byproduct of the noisy machine learning review system, the difficulty of the text itself is substantial and made the paper less than approachable; the authors are encouraged to continue to revise their text based on feedback from as many readers as possible.  That said, the authors were quite responsive to reviewer comments during the rebuttal phase, which significantly improved the text.

Overall this is a borderline case, and the AC also had some difficulty following details of this technically dense paper.  Given the positive *technical* assessments of the work and at least one reviewer defending the paper's clarity, the AC is willing to give this paper the benefit of the doubt.
Decision: Accept (Poster)