Denoising Diffusion Implicit Models
Authors: Jiaming Song
Chenlin Meng
Stefano Ermon
Venue: ICLR 2021 Poster (2021)
Keywords: generative models
variational autoencoders
denoising score matching
variational inference

Abstract: Denoising diffusion probabilistic models (DDPMs) have achieved high quality image generation without adversarial training, yet they require simulating a Markov chain for many steps in order to produce a sample. To accelerate sampling, we present denoising diffusion implicit models (DDIMs), a more efficient class of iterative implicit probabilistic models with the same training procedure as DDPMs. In DDPMs, the generative process is defined as the reverse of a particular Markovian diffusion process. We generalize DDPMs via a class of non-Markovian diffusion processes that lead to the same training objective. These non-Markovian processes can correspond to generative processes that are deterministic, giving rise to implicit models that produce high quality samples much faster. We empirically demonstrate that DDIMs can produce high quality samples $10 \times$ to $50 \times$ faster in terms of wall-clock time compared to DDPMs, allow us to trade off computation for sample quality, perform semantically meaningful image interpolation directly in the latent space, and reconstruct observations with very low error. 

Published as a conference paper at ICLR 2021
DENOISING DIFFUSION IMPLICIT MODELS
Jiaming Song, Chenlin Meng & Stefano Ermon Stanford University
(tsong, chen in, ermon) @cs. stanford.
ABSTRACT
Denoising diffusion probabilistic models (DDPMs) have achieved high qual- ity image generation without adversarial training, yet they require simulating a Markov chain for many steps in order produce sample. To accelerate sam- pling, we present denoising diffusion implicit models (DDIMs), a more efficient class of iterative implicit probabilistic models with the same training procedure DDPMs. In DDPMs, the generative process is defined as the reverse of a particular Markovian diffusion process. We generalize DDPMs via class of non-Matt non-Markovian diffusion processes that lead to the same training objective. These non-Markovian processes can correspond to generative processes that are deterministic, giving rise to implicit models that produce high quality samples much faster. We empirically demonstrate that DDIMs can produce high quality samples 10x to 50x faster terms of wall-clock time compared to DDPMs, allow us to trade off computation for sample quality, perform semantically meaningful image interpolation directly in the latent space, and reconstruct observations with very low error.
INTRODUCTION
Deep generative models have demonstrated the ability to produce high quality samples in many domains (Karras et al., 2020; van den Oord et al., 2016a). In terms of image generation, genera- tive adversarial networks (GANs, Goodfellow et al. (2014)) currently exhibits higher sample quality methods such as variational autoencoders (Kingma Welling, 2013), autore-
than likelihood gressive models (van den Oord et al., 201 and normalizing flows (Rezende & Mohamed, 2015; Dinh et al., 2016). However, GANs require very specific choices in optimization and architectures in order to stabilize training (Arjovsky et al., 2017; Gulrajani al., 2017; Karras et al., 2018; Brock et al., 2018), and could fail to cover modes of the data distribution (Zhao et al., 2018).
Recent works on iterative generative models (Bengio et al., 2014), such as denoising diffusion prob- abilistic models (DDPM, Ho et al. (2020)) and noise conditional score networks (NCSN, Song & Ermon (2019)) have demonstrated the ability to produce samples comparable to that of GANs, with- out having to perform adversarial training. To achieve this, many denoising autoencoding models trained to denoise samples corrupted by various levels of Gaussian noise. Samples are then produced by a Markov chain which, starting from white noise, progressively denoises it into an im- age. This generative Markov Chain process either based on Langevin dynamics (Song & Ermon, 2019) or obtained by reversing a forward diffusion process that progressively turns an image into noise (Sohl-Dickstein al., 2015).
A critical drawback of these models is that they require many iterations to produce a high quality sample. For DDPMs, this is because that the generative process (from noise data) approximates the reverse of the forward diffusion process (from data to noise), which could have thousands of steps; iterating over all the steps required to produce a single sample, which is much slower compared to GANs, which only needs one pass through network. For example, it takes around 20 hours to sample 50k images of size 32 x 32 from a DDPM, but less than a minute to do so from GAN Nvidia 2080 Ti GPU. This becomes more problematic for larger images sampling 50k images of size 256 x 256 could take nearly 1000 hours on the same GPU.
To close this efficiency gap between DDPMs and GANs, we present denoising diffusion implicit models (DDIMs). DDIMs are implicit probabilistic models (Mohamed & Lakshminarayanan, 2016) and are closely related to DDPMs, in the sense that they are trained with the same objective function.

---

Published as a conference paper at ICLR 2021
Py
P6
133
22
31
CD
33
x
Pt
1a zo)
glanglms
g (
Figure 1: Graphical models for diffusion (left) and Mon-Markov (right) inference models.
In Section 3, we generalize the forward diffusion process used by DDPMs, which is Markovian, non-Markovian ones, for which we are still able design suitable reverse generative Markov chains. We show that the resulting variational training objectives have shared surrogate objective, which exactly the objective used to train DDPM. Therefore, we can freely choose from a large family of generative models using the same neural network simply by choosing different, non- Markovian diffusion process (Section 4.1) and the corresponding reverse generative Markov Chain. In particular, are able to use non-Markovian diffusson processes which lead to "short" generative Markov chains (Section 4.2) that can be simulated in a small number of steps. This can massively increase sample efficiency only at minor cost in sample quality.
In Section 5, we demonstrate several empirical benefits of DDIMs over DDPMs. First, DDIMs have superior sample generation quality compared to DDPMs, when accelerate sampling by 10x to 100x using our proposed method. Second, DDIM samples have the following "consistency" prop- erty, which does not hold for DDPMs: if start with the same initial latent variable and generate several samples with Markov chains of various lengths, these samples would have similar high-level features. Third, because of "consistency" DDIMs, we can perform semantically meaningful image interpolation by manipulating the initial latent variable in DDIMs, unlike DDPMs which interpolates near the image space due to the stochastic generative process.
BACKGROUND
Given samples from a data distribution g(xa). we are interested in learning model distribution (o())) that approximates g( and is easy to sample from. Denoising diffusion probabilistic mod- els (DDPMs, Sohl-Dickstein al. (2015); Ho al. (2020)) are latent variable models of the form
(=() f po (x dx where Po( == Po(aT)
/
T
II I=l
Po 1
(1)
where æ1.... TT are latent variables in the same sample space as zu (denoted as X)., The parame- ters 0 are learned to fit the data distribution (xx)) by maximizing a variational lower bound:
max (x [log Pa (20)) < max Beg [log pe (Forr) loog g aan)
where q (x) is some inference distribution over the latent variables. Unlike typical latent vari- able models (such as the variational autoencoder (Rezende et al., 2014)), DDPMs are learned with fixed (rather than trainable) inference procedure q( (.I.:/)))) and latent variables are relatively high dimensional. For example, Ho et al. (2020) considered the following Markov chain with Gaussian transitions parameterized by decreasing sequence a1:7 E 10"
( (2)) II g 1), where (( =N
O4
11-1; (1
Oi at-1
04-1
where the covariance matrix is ensured to have positive terms on its diagonal. This called the forward process due to the autoregressive nature of the sampling procedure (from 2n DT). We call the latent variable model pa (Z which is Markov chain that samples from Zr to Do the generative process, since approximates the intractable reverse process q(x;-1/æ;). Intuitively, the forward process progressively adds noise to the observation Co. whereas the generative process progressively denoises a noisy observation (Figure 1, left).
A special property of the forward process is that
q( [x xo) := q( (odialoooddrr (o)dd1.(-)) Nixx varzo, (1 02) I));
2

---

Published as a conference paper at ICLR 2021
so we can express xy linear combination of zo and noise variable
z V Va,zo + /1 11-a4p a,6, where e~ (O(
(4)
When set or sufficiently close to 0, g(ær] mo) converges standard Gaussian for all 2o, it is natural to set po (TT) ;= (0, f). If all the conditionals are modeled as Gaussians with trainable mean functions and fixed variances, the objective in Eq. (2) can be simplified to
Ioy(Eo) Σ (( [lle8' [Ilo)((arre (Varao + v1 oxft))
<ell2]
27
(5)
where = ((0 )2 is set set of T functions, each 6D trainable parameters 8(0), and that depends on 01:r. In Ho et al. (2020), the objective with y is optimized instead to maximize generation performance of the trained model; this is also the same objective used in noise conditional score networks (Song & Ermon, 2019) based on score matching (Hyvärinen, 2005; Vincent, 2011). From trained model, By is sampled by first sampling Ør from the prior pe (o(r)) and then sampling Pt-1 from the generative processes iteratively.
(1) X + X (indexed by () is function with (.,..... 7T is vector of positive coefficients in the objective
The length I of the forward process is an important hyperparameter in DDPMs. From a variational perspective, large T' allows the reverse process to be close to a Gaussian (Sohl-Dickstein et al., 2015), so that the generative process modeled with Gaussian conditional distributions becomes a good approximation; this motivates the choice of large T values, such as T 1000 in Ho et al. (2020). However, as all iterations have be performed sequentially, instead of parallel, to ob- tain sample Zp. sampling from DDPMs is much slower than sampling from other deep generative models, which makes them impractical for tasks where compute is limited and latency is critical.
VARIATIONAL INFERENCE FOR NON-MARKOVIAN FORWARD PROCESSES
Because the generative model approximates the reverse of the inference process, we need to rethink the inference process in order to reduce the number of iterations required by the generative model. Our key observation is that the DDPM objective in the form of L,, only depends on the marginals" g(x,|/o)) but not directly the joint / Since there are many inference distributions (joints) with the same marginals, we explore alternative inference processes that are non-Markovian, which leads to new generative processes (Figure 1, right). These non-Markovian inference process lead to the same surrogate objective function as DDPM, as we will show below. In Appendix A, we show that the Mon-Markovian perspective also applies beyond the Gaussian case.
3.1 NON-MARKOVIAN FORWARD PROCESSES
Let us consider family Q of inference distributions, indexed by real vector a E RT 20
lo (I mo) = qo (xr (xT(ar)) an) II qa (0)-
(6)
and for all where yo (xT(20)) =N(VaTan aao (1-ar)
go (x Dr, vo) N (Var-iz0 + v 11 a:-1 of
0e v 10,300
,07
VI
The mean function is chosen to order to ensure that (x N( ar (1 ag I) for all t (see Lemma of Appendix B), so that it defines a joint inference distribution that matches the "marginals" as desired. The forward process can be derived from Bayes" rule:
go (x, (2-11, = Jo (Zt [x,, Zo Zo) (x) (20)
(8)
qo (x 1200
Please refer to Appendix C.2 for details.
i We slightly abuse this term (as well as joints) when only conditioned on mo.
We overload the term "forward process" for cases where the inference model not diffusion.
3

---

Published as a conference paper at ICLR 2021
which is also Gaussian (although do not use this fact for the remainder of this paper). Unlike the diffusion process in Eq. (3), the forward process here is no longer Markovian, since each 2y could depend on both - and Vo. The magnitude of a controls the how stochastic the forward process is; when a - O, we reach an extreme case where as long as we observe 2g and D, for some t, then 2 become known and fixed.
3.2 GENERATIVE PROCESS AND UNIFTED VARIATIONAL. INFERENCE ORJECTIVE
Next, we define a trainable generative process pe (Zo:T) ) where each pb (xt-1 leverages knowl- edge of do (a: (2,, zn)- Intuitively, given noisy observation we first make a prediction' of the corresponding zp. and then use to obtain a sample æt-1 through the reverse conditional distribution go (Zt-11 1 De, To), which we have defined.
For some Ty ~ ((za) and C (0, I). 2e can be obtained using Eq. (4). The model co (axe) then attempts to predict from It, without knowledge of 2g. By rewriting Eq. (4), one can then predict the denoised observation, which is a prediction of 20 given Zz::
p (xx) == (xt - V1 - az Eg (x:))/
(9)
We can then define the generative process with fixed prior po (xT)) - NN( I) and
( (a1),o9]) ( (2.- Fe, 1(() (zx)) otherwise,
if =
Po (z:-1
(10)
where Go (2x--1 Pt, 16 ( ) is defined as Eq. (7) with æo replaced by 16 (x:). We add some Gaussian noise (with covariance of for the case of supported everywhere.
ensure that the generative process
We optimize via the following variational inference objective (which is a functional over cB): (xI:)(z)) (
Ja (Ca) == Experra (marr) (frar) [log Zo
- log Pa (xo.T))
(11)
-Baorra.. (w0rr)
lo (o (20) Σ log gr (x. 3r, 2o)
(t) Σ log Po (x,-1 1 (x,) log po (xT)
where we factorize 4o (aI (20) according to Eq. (6) and po (mo:T) according to Eq. (1).
From the definition of Jz, it would appear that a different model has be trained for every choice of since it corresponds However, Jg is equivalent to Ly for certain weights y, as we show below.
different variational objective (and different generative process).
Theorem 1. For all a > O, there exists 7 E R.o and C E RR such that J. = L, C.
The variational objective Ly is special in the sense that if parameters 9 of the models e0 are not shared across different f, then the optimal solution for eo will not depend on the weights y (as global optimum achieved by separately maximizing each term the sum). This property of L., has two implications. On the one hand, this justified the use of L1 as surrogate objective function for the variational lower bound in DDPMs; on the other hand, since Ja is equivalent to some Lyy from Theorem 1, the optimal solution of Jo is also the same that of L1- Therefore, if parameters not shared across t in the model cø, then the L1 objective used by Ho et al. (2020) can be used as a surrogate objective for the variational objective J, well.
4 SAMPLING FROM GENERALIZED GENERATIVE PROCESSES
With In the objective, are not only learning generative process for the Markovian inference process considered in Sohl-Dicke bocl-Dickstein et al. (2015) and Ho et al. (2020), but also generative processes for many Mar-Markvvian forward processes parametrized by that we have described. Therefore, we can essentially use pretrained DDPM models as the solutions to the new objectives, and focus on finding a generative process that is better at producing samples subject to needs by changing
"I carming distribution the predictions also possible, hut empirically found little benefits of it.
4

---

Published as a conference paper at ICLR 2021
CK1
3
ED
gg mo)
g( 201 30)
Figure 2: Graphical model for accelerated generation, where [1,3].
4.1 DENOISING DIFFUSION IMPLICIT MODELS
From (TI.)) in Eq. (10), one can generate sample -;-1 from a sample æ, via:
Ct
vI 0rc (t) (x+)
+ 1 o V1- of (x.)
2t-1 Va-1
(12)
"direction pointing to random noise
predicted ano"
where er (0, I) is standard Gaussian noise independent of æt, and define oxg Different choices of a values results in different generative processes, all while using the same model cp, so re-training the model is unnecessary. When = v(1 -z- 1)((1 - ad as/a,-1 for all f, the forward process becomes Markovian, and the generative process becomes DDPM.
We note another special case when o = 0 for all 12; the forward process becomes deterministic in the generative process, the coefficient before the random
given 2t-1 and Vg. except for noise becomes zero. The resulting model becomes an implicit probabilistic model (Mohamed & Lakshminarayanan, 2016), where samples are generated from latent variables with a fixed procedure (from to zo). We name this the denoising diffusion implicit model (DDIM, pronounced /d:m/), because is an implicit probabilistic model trained with the DDPM objective (despite the forward process longer being diffusion).
4.2 ACCELERATED GENERATION PROCESSES
In the previous sections, the generative process is considered as the approximation to the reverse process; since of the forward process has T steps, the generative process is also forced to sample T steps. However, as the denoising objective L1 does not depend on the specific forward procedure as long as 4 (2 (axg) is fixed, we may also consider forward processes with lengths smaller than T, which accelerates the corresponding generative processes without having to train a different model.
Let consider the forward process as defined not on all the latent variables T1.T, but on subset . Trs where т is increasing sub-sequence of [1...., 77 of length S. In particular, we define the sequential forward process over :-.....: Bry such that g( (mo) = (Va,,, Zo, -1-o0 )) matches the "marginals" (see Figure 2 for an illustration). The generative process now samples latent variables according to reversed(7). which we term (sampling) trajec- tory. When the length the sampling trajectory is much smaller than we may achieve significant increases in computational efficiency due to the iterative nature of the sampling process.
Using a similar argument as Section 3, we can justify using the model trained with the I, ob- jective,
changes needed in training. We show that only slight changes to the updates in Eq. (12) are needed to obtain the new, faster generative processes, which applies to DDPM, DDIM, well all generative processes considered in Eq. (10). We include these details in Appendix C.1.
In principle, this means that we can train a model with an arbitrary number of forward steps but only sample from some of them in the generative process. Therefore, the trained model could consider many more steps than what is considered (Ho et al., 2020) or even continuous time variable (Chen al., 2020). We leave empirical investigations of this aspect future work.
"Athhough this is covered in Theorem 1, can always approximate by making o, very small.
5

---

Published as a conference paper at ICLR 2021
4.3 RELEVANCE TO NEURAL ODES
Moreover, we can rewrite the DDIM iterate according to Eq. (12), and its similarity to Euler inte- gration for solving ODEs becomes more apparent:
(V1-CL1
1
Tz
I V
Tt +
01-1
ai Cx
c
(8)
(xx)
(13)
a1-1
at
ac-1
We can reparameterize (vI- 11-0//e /l//v// va) with À and (x/val with H()) then sampling 2n with Equa- tion (13) can be treated integration over the following ODE:
H()) S CAHAANN 1 H(M), H(M) 10,
(14)
M
for some very large M (which corresponds to the case of a 0). This suggests that with enough T (discretization steps), the we can also reverse the generation process (going from t = 0 to 7), which encodes Zn to Ty and simulates the reverse of the ODE in Eq. (14). This suggests that unlike DDPM, we can use DDIM to obtain encodings of the observations (as the form æT), which might be useful for other downstream applications that requires latent representations of a model.
EXPERIMENTS
In this section, we show that DDIMs outperform DDPMs in terms of image generation when fewer iterations are considered, giving speed ups of 10x to 100x over the original DDPM generation process. Morcover, unlike DDPMs, once the initial latent variables zт fixed, DDIMs retain high- level image features regardless of the generation trajectory, so they are able to perform interpolation directly from the latent space. DDIMs can also be used to encode samples that reconstruct them from the latent code, which DDPMs cannot do due to the stochastic sampling process.
the same trained model with T 1000 and the objective being L, from Eq. (5) with 7 1; as we argued in Section 3, no changes are needed with regards to the training procedure. The only changes that make is how we produce samples from the model; we achieve this by controlling (which controls how fast the samples are obtained) and (which interpolates between the deterministic DDIM and the stochastic DDPM).
For each dataset,
We consider different sub-sequences 7 of [l,..., 2' and different variance hyperparameters a in- dexed by elements of r. To simplify comparisons, we consider a with the form:
Tr, (n) m ar D)/ (1)111_,,)))))
(15)
where 7) E R20 is hyperparameter that we can directly control. This includes original DDPM generative process when = and DDIM when 17 O. We also consider DDPM where the random noise has a larger standard deviation than o (1), which we denote as Ô: Ôr, = VI- (T, This is used by the implementation in Ho al. (2020) only to obtain the CIFAR10 samples, but not samples of the other datasets. We include more details in Appendix D.
5.1 SAMPLE QUALITY AND EFFICIENCY
In Table 1, we report the quality of the generated samples with models trained on CIFARIO and CelebA, as measured by Frechet Inception Distance (FID (Heusel et al., 2017)), where we vary the number of timesteps used to generate a sample (dim(r)) and the stochasticity of the process (n). As expected, the sample quality becomes higher as we increase dim(r), presenting a trade- off between sample quality and computational costs. We observe that DDIM (1) = 0) achieves the best sample quality when dim(r) is small, and DDPM (1) 1 and δ) typically has worse sample quality compared to its less stochastic counterparts with the same dim(r), except for the case for dim(r) 1000 and ő reported by Ho et al. (2020) where DDIM is marginally worse. However, the sample quality of becomes much worse for smaller dim(r), which suggests that it ill-suited for shorter trajectories. DDIM, on the other hand, achieves high sample quality much more consistently.
In Figure 3. we show CIFARIO and CelebA samples with the same number of sampling steps and varying For the DDPM, the sample quality deteriorates rapidly when the sampling trajectory has

---

Published as a conference paper at ICLR 2021
Table 1: CIFARIO and CelebA image generation measured in FID. 7 1.0 and are cases of DDPM (although Ho et al. (2020) only considered T 1000 steps, and S < T can be seen as simulating DDPMs trained with steps), and 17 = 0.0 indicates DDIM.
CIFARIO (32
32)
CelebA (64 x 64)
10
20
50
100
1000
10
20
50
100
1000
0.0
13.36
6.84
4.67
4.16
4.04
17.33
13.73
9.17
6.53
3.51
0.2
14.04
7.11
4.77
4.25
4.09
17.66
14.11
9.51
6.79
3.64
0.5
16,66
8.35
5,25
4.46
4.29
19.86
16.06
11.01
8.09
4.28
1.0
41.07
18.36
8.01
5.78
4.73
33.12
26.03
18.48
13.93
5.98
6
367.43
133.37
32.72
9.99
3.17
299.71
183.83
71.71
45,20
3.26
dim
dim(r)
dim(T)
dim(r)
0.0
0.0
0.0
0.0
- 0.5
0.5
0.5
0.5
1.0
100
Figure CIFARIO and CelebA samples with dim(r) = 10 and dim(r) = 100.
10 steps. For the case of o, the generated images seem to have more noisy perturbations under short trajectories; this explains why the FID) scores are much worse than other methods, as FID is very sensitive to such perturbations (as discussed in Dolicocur-Martinee
et al. (2020)).
In Figure 4, we show that the amount of time needed to produce sample scales linearly with the length of the sample trajectory. This suggests that DDIM is useful for producing samples more efficiently, samples
be generated in much fewer steps. Notably, DDIM is able to produce samples with quality comparable to 1000 step models within 20 to 100 steps, which is a 10x to 50x speed up compared to the original DDPM. Even though DDPM could also achieve reasonable sample quality with x00x steps, DDIM requires much fewer steps to achieve this; on CelebA, the FID score of the 100 step DDPM is similar to that of the 20 step DDIM.
5.2 SAMPLE CONSISTENCY IN DDIMS
For DDIM, the generative process is deterministic, and æo would depend only on the initial state æT. In Figure 5, we observe the generated images under different generative trajectories (i.e. different 7) while starting with the same initial æT. Interestingly, for the generated images with the same initial 2T, most high-level features are similar, regardless of the generative trajectory. In many cases, samples generated with only 20 steps are already very similar ones generated with 1000 steps in terms of high-level features, with only minor differences in details. Therefore, would appear that ar alone would be informative latent encoding of the image; and minor details that affects sample quality are encoded in the parameters, as longer sample trajectories gives better quality samples but do not significantly affect the high-level features. We show more samples in Appendix D.4.
CIFAR10
Bedroom
20
1000
Hours
Hours
100
0.5
10
30
100
300
1000
10
30
100
300
1000
steps
Figure 4: Hours to sample 50k images with one Nvidia 2080 Ti GPU and samples different steps.
7

---

Published as a conference paper at ICLR 2021
sample timesteps
20
50
1000
sample timesteps sample timesteps
100
100
Figure 5: Samples from DDIM with the same random æT and different number of steps.
5.3
INTERPOLATION IN DETERMINISTIC GENERATIVE PROCESSES
erstock
erstoccl
Figure 6: Interpolation of samples from DDIM with dim(r) 50.
Since the high level features of the DDIM sample is encoded by ær, would exhibit the semantic interpolation effect similar to that observed in other implicit proba- bilistic models, such GANs (Goodfellow et al., 2014). This is different from the interpolation procedure Ho et al. (2020), since DDPM the same ær would lead to highly diverse Zo due to the stochastic generative process In Figure 6, we show that simple interpolations in Ø7 lead to semantically meaningful interpolations between two samples. We include more details and samples in Appendix D.5. This allows DDIM control the generated images on high level directly through the latent variables, which DDPMs cannot.
interested to whether
5.4 RECONSTRUCTION FROM LATENT SPACE
As DDIM is the Fuler integration for particular ODE, it would be interesting see whether it can encode from 2y to ØT (reverse of Eq. (14)) and reconstruct my from the resulting or (forward
Athough it might possible if interpolates all T noises, like what is done in Song & Ermon (2020).
8

---

Published as a conference paper at ICLR 2021
Table 2: Reconstruction error with DDIM on CIFAR-I0 test set, rounded to 10-4
s
10
20
50
100
200
500
1000
Error
0.014
0.0065
0.0023
0.0009
0.0004
0.0001
0.0001
of Eq. (14))². We consider encoding and decoding on the CIFAR-10 test set with the CIFAR-10 model with S steps for both encoding and decoding; we report the per-dimension mean squared error (scaled to [0, 1l) Table 2. Our results show that DDIMs have lower reconstruction error for larger S values and have properties similar to Neural ODEs and normalizing flows. The same cannot be said for DDPMs due to their stochastic nature.
6 RELATED WORK
Our work is based on large family of existing methods on learning generative models as transi- tion operators of Markov chains (Sohl-Dickstein et al., 2015; Bengio et al., 2014; Salimans et al,, 2014; Song et al., 2017; Goyal et al., 2017: Levy et al., 2017). Among them, denoising diffusion probabilistic models (DDPMs, Ho et al. (2020)) and noise conditional score networks (NCSN, Song & Ermon (2019; 2020)) have recently achieved high sample quality comparable to GANs (Brock et al., 2018; Karras et al., 2018). DDPMs optimize a variational lower bound to the log- log-likelihood, whereas NCSNs optimize the score matching objective (Hyvärinen, 2005) over a nonparametric Parzen density estimator of the data (Vincent, 2011; Raphan & Simoncelli, 2011).
Despite their different motivations, DDPMs and NCSNs are closely related. Both denoising autoencoder objective for many noise levels, and both use procedure similar to Langevin dynamics to produce samples (Neal et al., 2011). Since Langevin dynamics is discretization of gradient flow (Jordan et al., 1998), both DDPM and NCSN require many steps to achieve good sample quality. This aligns with the observation that DDPM and existing NCSN methods have trouble generating high-quality samples in a few iterations.
DDIM, on the other hand, is an implicit generative model (Mohamed Lakshminarayanan, 2016) where samples are uniquely determined from the latent variables. Hence, DDIM has certain prop- erties that resemble GANs (Goodfellow al., 2014) and invertible flows (Dinh et al., 2016), such as the ability to produce semantically meaningful interpolations. We derive DDIM from purely variational perspective, where the restrictions of Langevin dynamics are not relevant; this could par- tially explain why iterations. The sampling procedure of DDIM is also reminiscent of neural networks with continuous depth (Chen et al., 2018; Grathwohl et al., 2018), since the samples it produces from the same latent variable have similar high-level visual features, regardless of the specific sample trajectory.
able observe superior sample quality compared to DDPM under fewer
7 DISCUSSION
We have presented DDIMs an implicit generative model trained with denoising auto-encoding / score matching objectives from a purely variational perspective. DDIM is able to generate high- quality samples much more efficiently than existing DDPMs and NCSNs, with the ability to perform meaningful interpolations from the latent space. The Mar-Mvikovian forward process presented here seems to suggest continuous forward processes other than Gaussian (which cannot be done in the original diffusion framework, since Gaussian is the only stable distribution with finite variance). We also demonstrated discrete case with multinomial forward process Appendix A, and it would be interesting to investigate similar alternatives for other combinatorial structures.
Moreover, since the sampling procedure of DDIMs is similar to that of an neural ODE, it would be interesting to see if methods that decrease the discretization error in ODEs, including multi- (Butcher & Goodwin, 2008), could be helpful for further improving sample quality in fewer steps (Queiruga et al., 2020). It is also relevant to investigate whether DDIMs exhibit other properties of existing implicit models (Bau ct al., 2019).
step methods such as
Since øy and Tg have the same dimensions, their compression qualities are not our immediate concem.

---

Published as a conference paper at ICLR 2021
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. arXiv preprint
arXiv: 1eXiv:1701.077 07875, January 2017.
David Bau, Jun-Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and An- tonio Torralba. Seeing what a gan cannot generate. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4502-4511, 2019.
Yoshua Bengio, Eric Laufer, Guillaume Alain, and Jason Yosinski. Deep generative stochastic networks trainable by backprop. In International Conference on Machine Learning, 226-234, January 2014.
Christopher M Bishop. Puttern recognition and machine learning. springer, 2006.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11099 September 2018.
John Charles Butcher and Nicolette Goodwin. Numerical methods for ordinary disferential equa- lions, volume 2. Wiley Online Library, 2008.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. WaveG- rad: Estimating gradients for waveform generation. arXiv preprint urXiv:2009.00715 September 2020.
Ricky T Q Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differ- ential equations. arXiv preprint arXiv:1806.073e8. 07366, June 2018.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. arXiv preprint 160iv 088003 May 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Anirudh Goyal, Nan Rosemary Ke, Surya Ganguli, and Yoshua Bengio. Variational walkback: Learning a transition operator stochastic recurrent net. In Advances in Neural Information Processing Systems, pp. 4392-4402, 2017.
Will Grathwohl, Ricky T Q Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. FFJORD: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367. October 2018.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Advances in Neural Information Processing Systems, pp- 5769-5779, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two Time-Scale update rule converge to a local nash equilibrium. arXiv preprint arXiv: 100.000000000000,, June 2017.
Jonathan Ho. Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. / 1239, June 2020.
Aapo Hyvärinen. Estimation of normmazal statistical models by score matching. Journal of Machine Learning Researc h. 6:695-709, 2005.
Alexia Jolicoeur-Martineau,, Rémi Piché-Taillefer, Rémi Tachet des Combes, and Ioannis Mitliagkas. Adversarial score matching and improved sampling for image generation. September 2020.
Richard Jordan, David Kinderlehrer, and Felix Otto. The variational formulation of the fokker- planck equation. SIAM journal on mathematical analysis, 2911):1-17 1998.
10

---

Published as a conference paper at ICLR 2021
Tero Karras, Samuli Laine, and Timo Aila. A Based generator architecture for generative adversarial networks. arXiv preprint arXiv:1812.04999. December 2018.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality stylegan. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110-8119, 2020.
Diederik Kingma and Max Welling. Auto-Encoding variational bayes. arXiv preprint arXiv: 14v10, December 2013.
Daniel Levy, Matthew Hoffman, and Jascha Sohl-Dickstein. Generalizing hamiltonian monte carlo with neural networks. arXiv preprint arXiv:1711.09260 2017.
Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv preprint arXiv:1610.03483, October 2016.
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
Alejandro F Queiruga, N Benjamin Erichson, Dane Taylor, and Michael W Mahoney. Continuous- in-depth neural networks. arXiv preprint arXiv:2008.02388, 2020.
Martin Raphan and Eero P Simoncelli. Least squares estimation without priors supervision.
Neural computation, 23(2):374-420, February 2011. ISSN 0899-7667, 1530-888X.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.05770 May 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and
approximate inference deep generative models. arXiv preprint arXiv:1401.4083 2014.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Tim Salimans, Diederik P Kingma, and Max Welling. Markov chain monte carlo and variational inference: Bridging the gap. arXiv preprint arXiv:1410.6466 October 2014.
Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pp. 245-254, 1985.
Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. arXiv preprint arXiv:1503.03588 March 2015.
Jiaming Song, Shengjia Zhao, and Stefano Ermon. A A-nice-me: Adversarial training for meme. arXiv preprint arXiv:1706.0r
June 2017.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint arXiv: 1007:v9997.000000 July 2019.
Yang Song and Stefano Ermon. Improved techniques for training Score-Based generative models. arXiv preprint 1rXiv:2006.009/1, June 2020.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv preprint (ttt::609.04999 September 2016a.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint 161779960, January 2016b.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- fation, 23(7):1661-1174 2011.
11

---

Published as a conference paper at ICLR 2021
Sergey Zagoruyko and Nikos Komodakis. arXivv1605.07116, May 2016.
Wide residual networks.
arXiv preprint
Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah Goodman, and Stefano Ermon. Bias and generalization in deep generative models: An empirical study. In Advances in Neural Information Processing Systems, pp. 10792-10801, 2018.
12

---

Published as a conference paper at ICLR 2021
A NON-MARKOVIAN FORWARD PROCESSES FOR A DISCRETE CASE
In this section, we describe a non-Markoviag forward processes for discrete data and corresponding variational objectives. Since the focus of this paper is to accelerate reverse models corresponding to the Gaussian diffusion, we leave empirical evaluations as future work.
For categorical observation Dg that is one-hot vector with K possible values, define the forward process as follows. First, we have g( (20) as the following categorical distribution:
g( Ca + -1 lx)
(16)
where 1x E I* is vector with all entries being 1/ K, and at decreasing from on = 1 for t = to QT fort T. Then we define q( (2,, 2o) as the following mixture distribution:
(a+)
with probability de with probability (ak-1- -
(011-1/20 11 vo)
Cat(xv)
(17)
Cat(1x)
with probability (1 - -04-1) - az)o
equivalently:
q x0) = Cat (o,30 + (ax-1 0200)) + ((1 - -4--))))))))))))) at)o,)lk),
(18)
which is consistent with how we have defined q( (00)
Similarly, define reverse process po (2r--1 (2:z)
Pe (2 = Cal (a.2r + (a - 0,04)58 ( (xx)) + ((1 01 (1-)))))))))) (19)
where f/ (ax) maps æ, K-dimensional vector. As (1-0;-1) az) - O. the sampling process will become less stochastic, in the sense that it will either choose W; or the predicted to with high probability. The KL divergence
DKL DKL(9(z-110z, xp) (
(20)
is well-defined, and is simply the KL divergence between two categoricals. Therefore, the resulting variational objective function should be easy to optimize as well. Moreover, as KI. divergence convex, we have this upper bound (which is tight when the right hand side goes to zero):
D æo) (1)
(a.-1 0, ) DEL
((t) (xx))).
The right hand side simply a multi-class classification loss (up to constants), so can arrive at similar arguments regarding how changes in do not affect the objective (up to re-weighting).
B PROOFS
Lemma 1, For do (21.))0)) defined in Eq. (6) and qo (2: 11 2o) defined in Eq. (7), we have:
go (x, (.)) ((aar v v
(21)
Proof. Assume for any < T. go (x, we) = (Vaata v1 - a, holds, if:
4o ( 1w0) = NW(oor-ia, /1-az-1
(22)
then we can prove the statement with an induction argument for t from T to 1, since the base case (t = T') already holds.
First, we have that
qo (x1-1/200) / go (x) (xt dd
and
ao (x1 N(Varzo [a
(23)
ao (0 [r,, vo) N (VVor-100 v 0t-1 - of. Dt vaizo
(24)
vI az:
13

---

Published as a conference paper at ICLR 2021
From Bishop (2006) (2.115), have that q( (x)) is Gaussian and
(21-1001))) = v0 + V 06-1 - oi
Varao v (0
(25)
v1 at
= Vaa-1200
(26)
and
Covlgo [20)] oi Ct-1 of (1-0 -1 04-1)
(27)
0t
Therefore, 4r (1 1200) = NNvaan vI 04-1 I), which allows us to apply the induction argument.
Theorem 1. For all a > O, there exists 7y E Ro and E R, such that Jg La + C.
Proof. From the definition of Jg:
T 2o (xTT(220) >
T
Jo(e)) = Exg.r (w0or))
log do (xt-1 [x,, vn)
Σ
log Po (1)
(x. 12x))
(28)
T
=Exer=((tor)
Σ D 11 (
xx))
log Po (2olzz1)
where we use denote "equal up to value that does not depend on eœ (but may depend on qo)"
For & > 1:
En (K ( xx)) (x.-
DKL (9o (x 2 (2) 1/ (x:))))
|1:00 - (1) (zc:))l ll
-g (00.0))((20.0)
(29)
201
(xx; c//vor
-(x;
C
(xx))) 04 2
= -1
(30)
20
||e - (0 (xxe)11 27
-Bo / TTmie
2dofau
(31)
where d the dimension of æn- For = 0:
2
100
f0
(a)
1)
log pb (xo|x=))
E Emg m1))
(32)
20
||e c/ (x) 112
= -a
2d
(33)
Therefore, when ?tt = 1/(200 for all t. E (1,.... :.....TT T). we have
T Σ
JE()))
[1c66 (x.) - =ellg] = Ly(E))
(34)
2do7a,
1=1
for all eg. From the definition of "=" we have that Ja In C.
14

---

Published as a conference paper at ICLR 2021
C ADDITIONAL DERIVATIONS
C.I ACCELERATED SAMPLING PROCESSES
In the accelerated case, we can consider the inference process to be factored as:
y0,r (20: = go.r (2rg (2)) II da.r (ZT, (r,, 20) II ya.r (2r (20)) 187
(35)
where т is a sub-sequence of [L...... T] of length with Ts T, and let 7 := (L.....,]) be its complement. Intuitively, the graphical model of (ær,) and æn form chain, whereas the graphical model heF and zo forms a star graph. We define:
ax,r (x. = NN (1 - ())
WEFU
(36)
Ir. v0r,
do.r (1T,,) [ Fo)
1 Do + V O1 - o
Vi E [S]
v1 an.
where the coefficients chosen such that:
go.r (zr, (zo) MN 20. (1 Cr, ar,)
Vi E [S]
(37)
i.e, the "marginals" match.
The corresponding "generative process" is defined as:
po == (o(r))
II
Pg (Ti) (I lxt) )x
II
Pg 10 (xc (mog ax:)
(38)
use to produce samples
in variational objective
where only part of the models are actually being used produce samples. The conditionals are:
Pg (Tc)
(K (x,,) = Jo.r (2., Ir,, ff ((re) (IT..)) if [S},
(39)
(1) ps
( xx) = N (xx) (am)).ff I) otherwise,
(40)
where we leverage la (Z TT. 2D0 as part of the inference process (similar what we have done in Section 3). The resulting variational objective becomes (define TT.-., 2 for conciseness):
(EE) Ea (moor)) [log Lo (x (zo) log pe (xFo))
(41)
E (2omT)
DKI( (20) (arol axt)
(1)
(42)
lET
Σ Drt.(um,,aaa |r To)le
(T)) (xT,, , ,l2r;)))
+
where each KL divergence is between two Gaussians with variance independent of 0. A similar argument to the proof used in Theorem can show that the variational objective J can also be converted to an objective of the form Ly.
C.2 DERIVATION OF DENOISING OBJECTIVES FOR DDPMS
We note that in Ho al. (2020), diffusion hyperparameter 3, is first introduced, and then relevant variables 0ę := 3g and āg I Og are defined. In this paper, we have used the notation a, to represent the variable a in Ho et al. (2020) for three reasons. First, it makes it more clear that we only need to choose one set of hyperparameters, reducing possible cross-references of the derived variables. Second, it allows us to introduce the generalization as well as the acceleration case easier, because the inference process is no longer motivated by a diffusion. Third, there exists an isomorphism between a1.r and 1,... T, which is not the case for 3r.
IIn this section teal color notations used Ho al. (2020).
15

---

Published as a conference paper at ICLR 2021
In this section, use Bi and dt to be more consistent with the derivation in Ho et al. (2020), where
Ol
(43)
01-1
0t
B
(44)
Ox--1
can be uniquely determined from or (i.e. oi))
First, from the diffusion forward process:
v
2o + arll ac-1) as-1
q<<;-112; 22, vo) -
8t.
3.1
az
at
1 az
Ho et al. (2020) considered specific type of P8 (x1-1 lx1):
(1)
Po (z, 22) = N (Ho( 0,
(45)
which leads to the following variational objective:
T
E (marr)
q( (z0) +
log q ( 20, 20) -
Σ tm/
log (() (x, xx,)
(46)
DKL DKL(9(21-11014 zo ao))
log Po (1)
(xo x11)
Le-1
One can write:
T--1 Fg
[10 1) (0 20)))
(47)
207
Ho al. (2020) chose the parametrization
Br 6 (xe: VI aq
Ma pa
(48)
which can be simplified to:
B? (0
T Eano,
||e <olvarito + v1 o4e,
1)11
(49)
D EXPERIMENTAL DETAILS
D.1 DATASETS AND ARCHITECTURES
We consider 4 image datasets with various resolutions: CIFARIO (32 32, unconditional). CelebA (64 x 64), LSUN Bedroom (256 x 256) and L.SUN Church (256 x 256). For datasets, we set the hyperparameters according the heuristic in (Ho al., 2020) to make the results directly comparable. We use the same model for each dataset, and only compare the performance different generative processes. For CIFAR10, Bedroom and Church, obtain the pretrained checkpoints from the original DDPM implementation; for CelebA, we trained our own model using the denoising objective L1.
Our architecture for (t) (xx) follows that Ho et al. (2020), which a U-Net (Ronneberger et al., 2015) based on Wide ResNet (Zagoruyko & Komodakis, 2016). We use the pretrained models from Ho al. (2020) for CIFAR1O, Bedroom and Church, and train our own model for the CelebA 64 64 model (since a pretrained model is not provided). Our CelebA model has five feature map resolutions from 64 x 64 to 4 x 4, and we use the original CelebA dataset (not CelebA-HQ) using the pre-processing technique from the StyleGAN (Karras et al., 2018) repository.
16

---

Published as a conference paper at ICLR 2021
Table 3: LSUN Bedroom and Church image generation results, measured in FID. For 1000 steps DDPM, the FIDs are 6.36 for Bedroom and 7.89 for Church.
Bedroom (256 x 256)
Church (256 x 256)
dim(T)
10
20
50
100
10
20
50
100
DDIM (1) 0.0)
16.95
8.89
6.75
6.62
19.45
12.47
10.84
10.58
DDPM (1) 1.0)
42.78
22.77
10.81
6.81
51.56
23.37
11.16
8.27
D.2 REVERSE PROCESS SUB-SEQUENCE SELECTION
We consider two types of selection procedure for t given the desired dim(() T:
Linear: we select the timesteps such that т, |ci| for some
Quadratic: we select the timesteps such that 7; = [ax²] for some c.
The constant value c is selected such that T-1 is close to T. We used quadratic for CIFAR1O and linear for the remaining datasets. These choices achieve slightly better FID than their alternatives in the respective datasets.
D.3 CLOSED FORM EQUATIONS FOR EACH SAMPLING STEP
From the general sampling equation in Eq. (12), we have the following update equation:
xr. VI ar, Co (T.) (xT.)
(n) = van-1
+ V1 ar.. - (n) (T.
(T.) + o, (n)
Var.
where
ar
Qr.
(n) = "V
1 ar,
ar
For the case of & (DDPM with larger variance), the update equation becomes:
vI VI- Cr (Ti) (xr,,
Fr.
Vir
+ 11 ar, or, (1)2.ez)
(T)) (T:) (T)) + δz
which uses a different coefficient for compared with the update for = 1, but uses the same coefficient for the non-stochastic parts. This update is more stochastic than the update for 7) = 1, which explains why it achieves worse performance when dim(7) is small.
D.4 SAMPLES AND CONSISTENCY
We show more samples in Figure 7 (CIFARIO), Figure 8 (CelebA), Figure 10 (Church) and consis- tency results of DDIM in Figure (Celeb)).
D.5 INTERPOLATION
To generate interpolations on a line, we randomly sample two initial ør values from the standard Gaussian, interpolate them with spherical linear interpolation (Shoemake, 1985), and then the DDIM to obtain Zn samples.
sin - (1))) ET
sin(ao) (1)
T
sin(
sin
2
(50)
(0) |1a [|
(ar
where 0 arccos
These values used to produce DDIM samples.
To generate interpolations on grid, we sample four latent variables and separate them in to two pairs; then slerp with the pairs under the same a, and use slerp the interpolated samples across the pairs (under an independently chosen interpolation coefficient). We show more grid interpolation results in Figure (CelebA), Figure 12 (Bedroom), and Figure 13 (Church).
17

---

Published as a conference paper at ICLR 2021
Figure 7: CIFARIO samples from 1000 step DDPM, 1000 step DDIM and 100 step DDIM.
Figure 8: CelebA samples from 1000 step DDPM, 1000 step DDIM and 100 step DDIM.
sample timesteps
50
100
1000
Figure 9: CelebA samples from DDIM with the same random Φr and different number of steps.
18

---

Published as a conference paper at ICLR 2021
Figure 10: Church samples from 100 step DDPM and 100 step DDIM.
Figure 11: More interpolations from the CelebA DDIM with dim(7) 50.
19

---

Published as a conference paper at ICLR 2021
Figure 12: More interpolations from the Bedroom DDIM with dim(r) 50.
Figure 13: More interpolations from the Church DDIM with dim(r) 50.
20


Reviews:
[{"review_id": "H_F-YtuXKxI", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary : This paper develops a variant (DDIM) of an existing method (DDPM) with the goal of accelerating it greatly while still maintaining performance. The authors are working in the context of a denoising process that runs in the reverse direction to a sequence of steps that each add a small amount of Gaussian noise to the original data. The proposal is to introduce an auxiliary function that breaks the Markov assumption by leaking some information in a controlled way about the training points x0, and then use this auxiliary function as scaffolding to train the actual Markov chain of denoising functions.\n\nThis paper builds on other works in the recent literature and proposes something useful and novel. They propose something relatively down-to-earth and then they methodically analyze the consequences and derive all the mathematical formulas that follow. I feel that the dose of mathematical content is just appropriate for what they set out to do. Less would be too vague, more would be excessive.\n\nThe direction that they are proposing is relevant, contrary to certain other papers that involve a lot of correct equations but don't take us anywhere interesting. In my mind, this is a very good clean paper. \n\nI appreciated the authors taking the time to mention the connection with ODEs, and I thought they would mention Neural ODEs in section 7 as a nod to the recent surge of interest in ODEs in the context of Deep Learning sparked by that paper.\n\nThe only bad things that I could possibly say about this paper would involve comparing it to certain other wildly creative papers, and to say that it's not as innovative or throught-provoking as those papers. And that's not a fair thing to say.\n\nI would like to ask the authors if, with their framework, there is a need to train a completely different epsilon_t for every t=1..T ? I understand these models to be U-Nets, as mentioned in the appendix. I presume that the thing that makes this reasonable is the fact that S < T, and only S different models need to be trained?", "extra_scores": {}, "extra_text": {}}, {"review_id": "32Iem2FenH_", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a change to the recently popular diffusion models, motivated by increasing the speed of sampling. This is accomplished by changing the “forward” process which adds noise to the data. In the original diffusion models, this forward process is a Markov process whose marginals and conditionals can be computed efficiently in closed form. This paper proposes to replace this Markov forward process with a non-markovian process that is designed to have the same marginals. The generative model, in this case, changes such that to predict the next step in the process, the model must first predict the “clean” sample at the end of the chain which is then used to give an estimate for the next step in the chain. \n\nIntriguingly, the objective for training this new generative model is identical to training a standard diffusion model. Thus, the models differ only at sampling time. Under the new interpretation, we can sample from a family of models after training. This family can be tuned to increase the speed of sampling, at the cost of some sample quality. Inside this family, there also exists an implicit generative model which can be sampled from in a more deterministic fashion than the other members -- hence the name of the paper. \n\nThe authors present a number of image generation experiments and study the impact of the sampling parameters on sample quality. They find that in settings where the efficiency of sampling is greater than the original diffusion model, the proposed approach achieves higher sample quality -- although never as high as the original diffusion model. \n\nBecause the model can be sampled in a deterministic fashion, the authors note that fixing the latent variables during sampling results in consistent samples. This allows a much more controlled generation compared to the original diffusion model. \n\n\nStrong areas:\n\nI quite enjoyed this paper. These diffusion models have gained considerable attention and this work addresses what is likely their largest issue -- the slow speed of sampling. Most interesting to me is these new ideas can be applied to the original diffusion models without retraining. \n\nI find that this work provides more insight into how these diffusion models work and which choices in the original presentation are important for their performance and which (like the inference process) can be changed. These are important insights and should be impactful for further research on this class of models. \n\n\nWeaknesses:\n\nWhile this is a strong paper, there are a few issues in my opinion. I found some of the experimental details difficult to understand. Were all the results presented generated from the same DDIM? Or were the models trained separately for each sampling parameter set? This should be made more clear. \n\nIn section 3.2 you claim some equivalence between the original DDPM objective and the variational bound in your model. This looks like re-weighting the various terms in the objective. In the DDPM work, they discuss the impact of this re-weighting. I would have liked to see a similar discussion here. They found it improved sample quality but decreased likelihood. I would have loved a comparison of the various training procedures but I did not find one. Along the same line, you present no likelihood results. You should be able to generate a lower-bound on likelihood using your model (except for the eta=0 model I believe). I am curious how the various sampling schemes impact likelihood. It is not necessary for the proposed approach to be appealing but would definitely complete the picture. \n\n\nNit-picky issues:\n\nI think the abstract could be a bit more descriptive. You should add some information saying what differs between your approach and standard diffusion models. In reading the abstract, it was not clear to me that using a non-markovian forward process would make sampling faster. This became clear as I read the paper. You could add a little more information to make that clear in the abstract.\n\n\nMy recommendation:\n\nI think this paper was clearly written and proposed a strong contribution to the field of generative modeling. I think the insights presented here give us more understanding of diffusion models and while also improving their sampling speed. I will recommend accepting this paper. \n\nSome questions:\n\nDo you think these same insights could be applied to discrete diffusion models proposed in the original work on non-equilibrium thermodynamics? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-OEVebC7R1n", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### DESCRIPTION \nThis paper consider tweaks to denoising diffusion models, exploring non-Markovian inference models, as well as shorter and possibly deterministic generative trajectories. \n\n#### DISCUSSION\n\nI'm having difficulty placing this work in the correct context, and disentangling exactly what the contributions are. From my understanding of denoising diffusion models (as presented in Sohl-Dickstein et al (2015) and Ho et al (2020)), the following hold:\n\n(i) At test time, the variance of each step of the generative process can be set to a different value than at training time, effectively evaluating a different model than the one that was trained. This is exploited by Ho et al (2020) to remove the weights in the variational lower bound at training time (they effectively set sigma such that the weights are 1 at training time).   \n\n(ii) At test time, it is possible to select a subset of the timesteps used at training time, and generate a corresponding schedule for those selected timesteps, possibly leading to more efficient sample generation at the expense of sample quality.\n\n(iii) Any inference process which satisfies the Markov property (as in Sohl-Dickstein et al or Ho et al) admits the factorization given in eq. (6). \n\nNow, I would have thought the first port of call would be to compare the proposed method in this paper to (i) and (ii). Moreover, while I *think* the choice of q(xt-1 | xt, x0) in eq. (7) indeed means that q(xt | xt-1, x0) =\\= q(xt | xt-1) (i.e. the process which starts at data and ends at noise is indeed non-Markovian), the lack of the comparisons with (i) and (ii) makes it unclear whether this step actually needs to be introduced, and the fact that the parameter sigma appears in both the mean and variance of q(xt | xt-1, x0) ties the mean and variance in a somewhat opaque way. \n\nI'm also a little confused by the choice of language in the paper. Traditionally, a diffusion process describes a continuous-time stochastic process which satisfies the Markov property. 'Denoising diffusion models' stretch the term diffusion to describe a finite discretization of this diffusion process, but explicitly retain the Markov property, and have a clear interpretation as a generative models which iteratively denoises a white noise sample. If you further make the inference process non-Markovian, so that you have a non-continuous, non-Markovian process, does it make sense to persist in using the word diffusion to describe the model? \n\nThis may be a nitpick, but I'm also not sure about the sense in which this model can be described as 'implicit'. The term implicit is typically used to describe generative models which specify the data-generating procedure mechanistically as a series of prescribed operations (possibly stochastic) applied to some given inputs, so that the resulting output distribution is defined 'implicitly' by these generative steps, and *crucially* marginalizing over any associated latent variables is intractable. The 'implicit' model described in this paper is formulated as the deterministic limit of a standard latent variable generative model fit using amortized stochastic variational inference. Recent work has examined normalizing flows as the deterministic limit of a Markovian generative model (e.g. \"Stochastic Normalizing Flows\", Wu et al (2020), \"SurVAE Flows\", Nielsen et al 2020) and indeed, the sentence \"The resulting model becomes an implicit probabilistic model, where samples are generated from latent variables with a fixed procedure.\" from this paper could equally describe a standard normalizing flow, which is certainly not an implicit model. \n\nAs far as the experiments are concerned:\n\n- It seems the takeaway message from section 5.1 is that the proposed model better adapts to evaluation with shorter trajectories when eta is small, and that the original DDPM doesn't fare well with shorter trajectories. The sample quality gets worse the shorter the trajectory (which is expected), but the best results are still given by evaluating the full DDPM model with 1000 steps.   \n\n- In section 5.2, I suppose it's expected that samples generated using different length trajectories from the same starting point are similar given that the generative process likely corresponds to some ODE discretization (as you've mentioned in the conclusion), but it's good to have it tested.   \n\n- In section 5.3, I don't really understand the purpose of interpolations for demonstrating the capabilities of generative models. What exactly do they demonstrate? \n\n#### EXTRA NOTES\n\nWhy are 'joint' and 'marginals' repeatedly placed in quotation marks? \n\n\"The magnitude of σ controls the how stochastic the forward process is.\", \"...the stochasticity of the process\", \"...compared to its less stochastic counterparts\" What does it mean for something to be more or less 'stochastic' than something else? How can 'stochasticity' be used as a quantitative measure?\n\n\"Fenchel Inception Distance\" -> \"Frechet Inception Distance\"\n\n#### CONCLUSION \nOverall, I think fundamental idea of improving the long sampling trajectories of DDPMs is certainly interesting. However, I'm finding it difficult to understand the extent to which the proposed non-Markovian inference process is really necessary i.e. I'd like to see baseline comparison with points (i) and (ii) above. Right now, I'm not able to disentangle which components of the proposed method are actually necessary.  ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work provides additional insights into a class of generative models that is rapidly gaining traction, and extends it by potentially providing a faster sampling mechanism, as well as a way to meaningfully interpolate between samples (an ability which adversarial models, currently the most popular class of generative models, also have). The revised manuscript includes an extension to discrete data, which could potentially amplify the impact of this work. The authors have also run additional experiments in response to the reviewers' comments.

Reviewer 1 raised several concerns about the choice of language (i.e. referring to the proposed model as a diffusion model, and the precise meaning of 'implicit' in the context of generative models). This is a fair point, as the authors introduce changes that affect the Markovian nature of the "diffusion" process, and a diffusion process is supposed to be Markovian by definition.

However, I think there is something to be said for the authors' argument of using the word 'diffusion' to clearly link this work to the prior work on which it is based. Given that technically speaking, the original DDPM work already 'abuses' the term to refer to a discrete-time process, it is difficult to argue compellingly that 'diffusion' should not feature in the name of the proposed model. Referring to 'non-Markovian diffusion processes' however seems more problematic, as this is a direct contradiction. If the authors wish to use this phrase, adding a few sentences to the introduction that justify this use would be helpful, and personally I feel this would be sufficient to address the issue (I noted that Section 4.1 already acknowledges that the forward process is no longer a diffusion). Plenty of work in our field abuses notation and this is justified simply with the phrase "with (slight) abuse of notation..."; I don't think this would be any different.

Reviewer 1 is technically correct that 'stochastic' is an absolute adjective, i.e. something can only be stochastic or deterministic, there is nothing in between, and there are no degrees/levels of stochasticity or determinism. In practice however, it is quite often used in a comparative sense, and I believe I have in fact been guilty of this myself! I do not feel that it causes any ambiguity in this case. Indeed, the phrase 'degree of stochasticity' seems to be in relatively common use in literature. While there may be more correct terms to use, I subscribe to the descriptivist view on language, and I do not think the comparative use of 'stochastic' is a major issue here. The alternatives I can think of seem potentially more cumbersome (e.g. I wager that 'more/less entropic' would be more poorly understood than 'more/less stochastic'). Still, I recommend that the authors consider potential alternatives in the future, to avoid any confusion.

Overall, I think the reviewers' major concerns have been addressed in the revised manuscript. Given that all reviewers consider the idea worthwhile, I will join them in recommending acceptance.
Decision: Accept (Poster)