Score-Based Generative Modeling through Stochastic Differential Equations
Authors: Yang Song
Jascha Sohl-Dickstein
Diederik P Kingma
Abhishek Kumar
Stefano Ermon
Ben Poole
Venue: ICLR 2021 Oral (2021)
Keywords: generative models
score-based generative models
stochastic differential equations
score matching
diffusion

Abstract: Creating noise from data is easy; creating data from noise is generative modeling. We present a stochastic differential equation (SDE) that smoothly transforms a complex data distribution to a known prior distribution by slowly injecting noise, and a corresponding reverse-time SDE that transforms the prior distribution back into the data distribution by slowly removing the noise. 
Crucially, the reverse-time SDE depends only on the time-dependent gradient field (a.k.a., score) of the perturbed data distribution. By leveraging advances in score-based generative modeling, we can accurately estimate these scores with neural networks, and use numerical SDE solvers to generate samples. We show that this framework encapsulates previous approaches in score-based generative modeling and diffusion probabilistic modeling, allowing for new sampling procedures and new modeling capabilities. In particular, we introduce a predictor-corrector framework to correct errors in the evolution of the discretized reverse-time SDE. We also derive an equivalent neural ODE that samples from the same distribution as the SDE, but additionally enables exact likelihood computation, and improved sampling efficiency. In addition, we provide a new way to solve inverse problems with score-based models, as demonstrated with experiments on class-conditional generation, image inpainting, and colorization. Combined with multiple architectural improvements, we achieve record-breaking performance for unconditional image generation on CIFAR-10 with an Inception score of 9.89 and FID of 2.20, a competitive likelihood of 2.99 bits/dim, and demonstrate high fidelity generation of $1024\times 1024$ images for the first time from a score-based generative model.

Published as a conference paper at ICLR 2021
SCORE-BASED GENERATIVE MODELING THROUGH STOCHASTIC DIFFERENTIAL EQUATIONS
Yang Song*
Jascha Sohl-Dickstein
Diederik P. Kingma
Stanford University
Google Brain
Google Brain durk Gurkegoogle.c com
stanford.edu wtw.onnsssssssaaooo.e.... yangsonge
jaschasd@yoogle.ccoo com
Abhishek Kumar
Stefano Ermon
Ben Poole
Google Brain
Stanford University
Google Brain
abhishkegoogle com
ermonêca stanford. edu
poolebegoogle.com com
ABSTRACT
Creating noise from data is easy; creating data from noise is generative modeling. We present a stochastic differential equation (SDE) that smoothly transforms com- plex data distribution to known prior distribution by slowly injecting noise, and a corresponding reverse-time SDE that transforms the prior distribution back into the data distribution by slowly removing the noise. Crucially, the reverse-time SDE depends only the time-dependent gradient field (a.k.a., score) of the perturbed data distribution. By leveraging advances in score-based generative modeling, we accurately estimate these scores with neural networks, and numerical SDE solvers to generate samples. We show that this framework encapsulates previous approaches in score-based generative modeling and diffusion probabilistic mod- eling, allowing for new sampling procedures and new modeling capabilities. In particular, we introduce a predictor-corrector framework correct errors in the evolution of the discretized reverse-time SDE. We also derive an equivalent neural ODE that samples from the same distribution as the SDE, but additionally enables exact likelihood computation, and improved sampling efficiency. In addition, we provide new way to solve inverse problems with score-based models, as demon- strated with experiments on class-conditional generation, image inpainting, and colorization. Combined with multiple architectural improvements, we achieve record- record-breaking breaking performance for unconditional image generation CIFAR-10 10 with an Inception score 9.89 and FID of 2.20, a competitive likelihood of 2.99 bits/dim, and demonstrate high fidelity generation of 1024 x 1024 images for the first time from a score-based based generative model.
INTRODUCTION
Two successful classes probabilistic generative models involve sequentially corrupting training data with slowly increasing noise, and then learning to reverse this corruption in order to form generative model of the data. Score matching with Langevin dynamics (SMLD) (Song & Ermon, 2019) estimates the score (i.e., the gradient of the log probability density with respect data) each noise scale, and then uses Langevin dynamics to sample from a sequence of decreasing noise scales during generation. Denoising diffusion probabilistic modeling (DDPM) (Sohl-Dickstein al., 2015; Ho et al., 2020) trains sequence of probabilistic models to reverse each step of the noise corruption, using knowledge of the functional form of the reverse distributions to make training tractable. For continuous state spaces, the DDPM training objective implicitly computes scores at each noise scale. We therefore refer to these two model classes together as score score-based generative models.
Score-based generative models, and related techniques (Bordes et al., 2017; Goyal et al., 2017: Du & Mordatch, 2019), have proven effective at generation of images (Song & Ermon, 2019; 2020; Ho et al., 2020), audio (Chen et al., 2020; Kong et al., 2020), graphs (Niu et al., 2020), and shapes (Cai
Work partially done during an internship Google Brain.

---

Published as a conference paper at ICLR 2021
Forward SDE (data + noise)
Figure 1: Solving a reverse- time SDE yields score-based generative model. Transform- ing data to simple noise dis- tribution can be accomplished with continuous- This SDE can be reversed if we know the score of the distribu- tion at each intermediate time step, Vx log Pa (x).
x(O)
dx = f(x, t)dt + g(t)dw
x(T)
SDE.
score function
x(0)
[(xx, t)
V po + g())d
x(T)
Reverse SDE (noise data)
al., 2020). To enable new sampling methods and further extend the capabilities of score-based generative models, we propose a unified framework that generalizes previous approaches through the lens of stochastic differential equations (SDEs).
Specifically, instead of perturbing data with a finite number of noise distributions, we consider continuum of distributions that evolve over time according to diffusion process. This process progressively diffuses data point into random noise, and is given by prescribed SDE that does not depend on the data and has no trainable parameters. By reversing this process, we can smoothly mold random noise into data for sample generation. Crucially, this reverse process satisfies a reverse-time SDE (A (Anderson, 1982), which can be derived from the forward SDE given the score of the marginal probability densities as a function of time. We can therefore approximate the reverse-time SDE by training a time-dependent neural network to estimate the scores, and then produce samples using numerical SDE solvers. Our key idea is summarized in Fig.
Our proposed framework has several theoretical and practical contributions:
Flexible sampling and likelihood computation: We can employ any general-purposs SDE solver to integrate the reverse-time SDE for sampling. In addition, we propose two special methods not viable for general SDEs: (i) Predictor-Correctorr (PC) samplers that combine numerical SDE solvers with soore-based MCMC approaches, such as Langevin MCMC (Parisi, 1981) and HMC (Neal et al., 2011); and (ii) deterministic samplers based on the probability flow ordinary differential equation (ODE). The former unifies and improves over existing sampling methods for score-based models. The latter allows for fast adaptive sampling via black-box ODE solvers, fleiile data manipulation via latent codes, uniquely identifiable encoding, and notably, exact likelihood computation.
Controllable generation: We can modulate the generation process by conditioning on information not available during training, because the conditional reverse-time SDE can be efficiently estimated from unconditional scores. This enables applications such as class-conditional generation, image inpainting, colorization and other inverse problems, all achievable using single unconditional score-based model without re-training.
Our framework provides a unified way explore and tune various SDEs for improving score-based generative models. The methods of SMLD and DDPM can be amalgamated into framework discretizations of two separate SDEs. Although DDPM (Ho al., 2020) was recently reported to achieve higher sample quality than SMLD (Song & Ermon, 2019; 2020), we show that with better architectures and new sampling algorithms allowed by our framework, the latter can catch up- up-it it achieves new state-of-the thee----tthhrttttt Inception score (9.89) and FID score (2.20) on CIFAR- 10, as well as high-fidelity generation of 1024 1024 images for the first time from a score-based model. In addition, we propose new SDE under our framework that achieves likelihood value of 2.99 bits/dim on uniformly dequantized CIFAR-10 images, setting a new record on this task.
Unified framework:
2 BACKGROUND
2.1 DENOISING SCORE MATCHING WITH LANGEVIN DYNAMICS (SMLD)
Let Po (x xx) == \ x, I be perturbation kernel, and (x) == Pdata (x) (x x)dx, where Patta (x) denotes the data distribution. Consider sequence of positive noise scales Tmin = σi < 02 NN mma- Typically, amin is small enough such that Pa... (x) s Paata (x), and Fmax is
2

---

Published as a conference paper at ICLR 2021
large enough such that Pa (x) (x 0, max I). Song & Ermon (2019) propose train Noise Conditional Score Network (NCSN), denoted by se (x, σ), with a weighted sum of denoising score matching (Vincent, 2011) objectives:
N
0* = arg min
Σ of (x) (x1x) (s - Vx log (x x)112
(I)
Given sufficient data and model capacity, the optimal score-based model Syw Sx,((,0) 0) matches Vx Po (x) almost everywhere for E {o.) For sampling, Song & Ermon (2019) run M steps of Langevin MCMC to get sample for each Po, (x) sequentially:
x + E/S0* (x""= o,) + - /2Ezz," m 1.2,...,, ,M,
(2)
where > O is the step size, and z!" is standard normal. The above is repeated for N, 1, ..... in turn with x ~ Nix 0, mmxx) I) and x² x, when < N. As M - oc and c - for all i, x becomes an exact sample from Po (x) = Pdata(x) under some regularity conditions.
2.2 DENOISING DIFFUSION PROBABILISTIC MODELS (DDPM)
Sohl-Dickstein et al. (2015); Ho al. (2020) consider a sequence of positive noise scales 0 < 81, ... .N < 1. For each training data point X Pxata(x), a discrete Markov chain {xo, X1. XN) constructed such that p(x; x,-1) N(xx; /I 1,, RQTI), and therefore Po, (x: xo) = NXx; /a,xo, (1 a,))),, where o, = [[;-1 (0]1-1))))))) B,). Similar to SMLD, we can denote the perturbed data distribution Po. (x) == PP xx)Pa, (x x)dx. The noise scales are pre- scribed such that x is approximately distributed according to (O, 1). A variational Markov chain in the reverse direction is parameterized with (x = WX(;;1; VT= (x; + 8:50 (x,, i)), B,I), and trained with a re-weighted variant of the evidence lower bound (ELBÓ):
0+ arg min Σ (1--aj)
(1-a;) Bx Px) (x (3) i) Vx log
(x x 12]
(3)
0
After solving Eq. (3) to get the optimal model Sø* (x, i), samples can be generated by starting from XN I), and following the estimated reverse Markov chain below
X VI- (x; + BiSgs (xs, 0)) + v Biz,, N 1
We call this method ancestral sampling, since it amounts to performing ancestral sampling from the graphical model I po xx). The objective Eq. (3) described here is Liimpte in Ho et al. (2020), written in a form to expose more similarity to Eq. (1). Like Eq. (1), Eq. (3) is also a weighted sum of denoising score matching objectives, which implies that the optimal model, Sae (x, i), matches the score of the perturbed data distribution, Vx log Po, (x). Notably, the weights of the i-th summand logPo,
in Eq. (1) and Eq. (3), namely of and (1 az)), related to corresponding perturbation kernels in the same functional form: o2 V log Po. (x x) x and (1 - a:)[c//B[[/V logPo, Po, (X x)]l
SCORE-BASED GENERATIVE MODELING WITH SDES
Perturbing data with multiple noise scales is key the success of previous methods. We propose to generalize this idea further to an infinite number of noise scales, such that perturbed data distributions evolve according to an SDE as the noise intensifies. An overview of our framework is given in Fig.
3.1 PERTURBING DATA WITH SDES
Our goal is to construct diffusion process (x(t) indexed by a continuous time variable [o, T]. such that x(0) ~ Pv. for which we have dataset of i.i.d. samples, and x(T) ~ pr. for which we have a tractable forn to generate samples efficiently. In other words, Po is the data distribution and Pr is the prior distribution. This diffusion process can be modeled as the solution to an Itô SDE:
dx f(x, t)dt g(t)dw
(5)
3

---

Published as a conference paper at ICLR 2021
Data
Forward SDE
Prior
Reverse SDE
Data
(0)
dr (jdt + a(t)du
(T)
dr = (a,,) - V.
s(t)dui
=(0)
SDC
Probebility flow ODE
x
(a(a)
(z)
(x)
Figure 2: Overview score-based generative modeling through SDEs. We can map data to a noise distribution (the prior) with SDE (Section 3.1), and reverse this SDE for generative modeling (Section 3.2). We can also reverse the associated probability flow ODE (Section 4.3), which yields a deterministic process that samples from the same distribution as the SDE, Both the reverse-time SDE and probability flow ODE can be obtained by estimating the score Vx log (x( (Section 3.3).
where w is the standard Wiener process (a.k.a,, Brownian motion), f(-,t) R - Rd is a vector- valued function called the drift coefficient of x(t), and g(-) R the diffusionn coefficient of x()). For ease of presentation we assume the diffusion coefficient is scalar (instead of d d matrix) and does not depend on x, but our theory can be generalized to hold in those cases (see Appendix A). The SDE has unique strong solution as long the coefficients are globally Lipschitz both state and time (Øksendal, 2003). We hereafter denote by pe px(x) the probability density of x(t), and use P&o (x() x(s)) to denote the transition kernel from x(s) to x()), where 0 & s s<t <
scalar function known as
Typically, pr is unstructured prior distribution that contains information of Po. such Gaussian distribution with fixed mean and variance. There are various ways of designing the SDE in Eq. (5) such that it diffuses the data distribution into fixed prior distribution. We provide several examples later in Section 3.4 that are derived from continuous generalizations of SMLD and DDPM.
3.2 GENERATING SAMPLES BY REVERSING THE SDE
By starting from samples of x(T) Pr and reversing the process, we can obtain samples x(0) ~ Pa- remarkable result from Anderson (1982) states that the reverse of diffusion process also diffusion process, running backwards in time and given by the reverse-time time SDE:
dx
[f(x t) 2 log Pe (x)Jdt g(t)dw,
where w is standard Wiener process when time flows backwards from T to 0, and dt is an infinitesimal negative timestep. Once the score of each marginal distribution, Vx log (x)), is known for all f, derive the reverse diffusion process from Eq. (6) and simulate it to sample from Po-
3.3 ESTIMATING SCORES FOR THE SDE
The score of distribution can be estimated by training a score-based model on samples with score matching (Hyvärinen, 2005; Song et al., 2019a). To estimate V. log logg)xx). we can train a lime-dependent score-based model so(x, f) via continuous generalization Eqs. (1) and (3):
0* arg min E, ((1)) Exco) So (x(),)) - Vx(0) log Por(((()()) x 11]] (7)
Here À [o.T] R.o is positive weighting function, is uniformly sampled over [0, T].
x(0) ~ (x( and x(t) ~ Por (x(() x(0)). With sufficient data and model capacity, score matching ensures that the optimal solution to Eq. (7), denoted by Sø* (x, t), equals Vx log (x() for almost all
x and t. As in SMLD and DDPM, we can typically choose 1x1/E| || x(t) log Poe (x(t) xx()) [
Note that Eq. (7) uses denoising score matching, but other score matching objectives, such sliced
4

---

Published as a conference paper at ICLR 2021
score matching (Song al., 2019a) and finteediifference score matching (Pang al., 2020) also applicable here.
We typically need to know the transition kernel (ot(x(t) x(0)) to efficiently solve Eq. (7), When f(., t) is affine, the transition kernel always a Gaussian distribution, where the mean and variance are often known in closed-forms and can be obtained with standard techniques (see Section 5.5 in Särkkä Solin (2019)). For more general SDES, we may solve Kolmogorov's forward equation (Øksendal, 2003) to obtain Poz (x(() x()).. Alternatively, we can simulate the SDE sample from Poo (x() x()) and replace denoising score matching in Eq. (7) with sliced score matching for model training, which bypasses the computation of Vx(1) log Pos (x(t) x()) (see Appendix A).
3.4 EXAMPLES: VE, VP SDES AND BEYOND
The noise perturbations used in SMLD and DDPM can be regarded discretizations of two different SDEs. Below we provide brief discussion and relegate more details to Appendix B.
When using total of N noise scales, each perturbation kernel Po, (x xo) of SMLD corresponds to the distribution of x; in the following Markov chain:
x; = x; x
Zi-1, i= 1,..,,... N,
(8)
where 2 ~ NOO,I), I), and we have introduced do = 0 to simplify the notation. In the limit of N xc, (1 becomes a function (f). z, becomes z(t), and the Markov chain {x]]=_ becomes a continuous stochastic process (x(t)} where we have used a continuous time variable E [o. 1] for indexing, rather than an integer i. The process (x(O)= is given by the following SDE
[[oo(t)] dw.
dx= -
dt
Likewise for the perturbation kernels (Pag (x xn)) of DDPM, the discrete Markov chain is
x; /1 3,x + / /B,21-1, i 1,
(10)
As N co. Eq. (10) converges the following SDE,
dx -
8(t)x dt + v B(t dw.
(11)
Therefore, the noise perturbations used in SMLD and DDPM correspond to discretizations of SDEs Eqs. (9) and (11). Interestingly, the SDE of Eq. (9) always gives a process with exploding variance when - co, whilst the SDE of Eq. (11) yields process with fixed variance of one when the initial distribution has unit variance (proof in Appendix B). Due to this difference, we hereafter refer to Eq. (9) the Variance Exploding (VE) SDE, and Eq. (11) the Variance Preserving (VP) SDE.
Inspired by the VP SDE, we propose a new type of SDEs which perform particularly well on likelihoods (see Section 4.3), given by
dx - 1 B()x di + V 3(0(1 e-25 (())ds Jdw.
(12)
When using the same B(1) and starting from the same initial distribution, the variance of the stochastic process induced by Eq. (12) is always bounded by the VP SDE at every intermediate time step (proof in Appendix B). For this reason, name Eq. (12) the sub-VP SDE.
Since VE, VP and sub- VP SDEs all have affine drift coefficients, their perturbation kernels por (x (t) x(0)) all Gaussian and can be computed in closed-forms, discussed in Section 3.3. This makes training with Eq. (7) particularly efficient.
4 SOLVING THE REVERSE SDE
After training a time-dependent score-basee core-based model 5g. we can use it to construct the reverse-time SDE and then simulate it with numerical approaches to generate samples from Po-
5

---

Published as a conference paper at ICLR 2021
Table 1: Comparing different reverse-time SDE solvers on CIFAR-10. Shaded regions are obtained with the same computation (number of score function evaluations). Mean and standard deviation are reported over five sampling runs. "P1000" or "P2000": predictor-only samplers using 1000 or 2000 steps. "C2000": corrector-only samplers using 2000 steps. "PC1000": Predictor-Correcto (PC) samplers using 1000 predictor and 1000 corrector steps.
Variance Exploding SDE (SMLD)
Variance Preserving SDE (DDPM)
FID."
Sampler
P1000
P2000
C2000
PC1000
P1000
P2000
C2000
PC1000
Predictor
ancestral sampling
4.98 1
4.88 1
3.62
3.24
3.24
3.21
reverse diffusion
4.79 + .07
4.74 .8
20.43 .07
3.60 .22
3.21 + 02
3.19 A2
19.06 + .s
3.18 :
probability flow
15.41 15
10.54 .08
3,51 414
3,59 044
3.23 + .03
3.06 ::
4.1 GENERAL-PURPOSS NUMERICAL SDE SOLVERS
Numerical solvers provide approximate trajectories from SDEs. Many general-purpose numerical methods exist for solving SDEs, such as Euler- Miller-Mruyamaa and stochastic Runge- Kutta methods (Kloe- den & Platen, 2013), which correspond to different discretizations of the stochastic dynamics. We can apply any of them to the reverse-time SDE for sample generation.
Ancestral sampling, the sampling method of DDPM (Eq. (4)), actually corresponds one special discretization of the reverse-time VP SDE (Eq. (11)) (see Appendix E). Deriving the ancestral sampling rules for new SDEs, however, can be non-trivial. To remedy this, we propose reverse diffusion samplers (details in Appendix E), which discretize the reverse-time SDE in the same way as the forward one, and thus can be readily derived given the forward discretization. As shown in Table 1, reverse diffusion samplers perform slightly better than ancestral sampling for both SMLD and DDPM models on CIFAR-10 (DDPM-type Mttpe ancestral sampling also applicable SMLD models, see Appendix F.)
4.2 PREDICTOR -ORCOTOT---RRETORRRORO SAMPLERS
Unlike generic SDEs, we have additional information that can be used to improve solutions. Since we have a score-based model Sg* (x, t) Vx (x), we can employ score-based MCMC approaches, such as Langevin MCMC (Parisi, 1981; Grenander & Miller, 1994) or HMC (Neal et al., 2011) to sample from Pt directly, and correct the solution of numerical SDE solver.
Specifically, at each time step. the numerical SDE solver first gives an estimate of the sample the next time step, playing the role of a "predictor". Then, the score-based MCMC approach corrects the marginal distribution of the estimated sample, playing the role of "corrector". The idea is analogous to Predictor-Corrector methods, a family of numerical continuation techniques for solving systems of equations (Allgower& & Georg, 2012), and similarly name our hybrid sampling algorithms Predictor-Corrector (PC) samplers. Please find pseudo-code and a complete description in Appendix G. PC samplers generalize the original sampling methods of SMLD and DDPM: the former uses an identity function as the predictor and annealed Langevin dynamics as the corrector, while the latter uses ancestral sampling as the predictor and identity as the corrector.
We test PC samplers on SMLD and DDPM models (see Algorithms 2 and 3 in Appendix G) trained with original discrete objectives given by Eqs. (1) and (3). This exhibits the compatibility of PC samplers to score-based models trained with a fixed number of noise scales. We summarize the performance of different samplers in Table 1, where probability flow is predictor to be discussed in Section 4.3. Detailed experimental settings and additional results are given in Appendix G. We observe that our reverse diffusion sampler always outperform ancestral sampling, and corrector-only methods (C2000) perform worse than other competitors (P2000, PC1000) with the same computation (In fact, we need way more corrector steps per noise scale, and thus more computation, to match the performance of other samplers.) For all predictors, adding one corrector step for each predictor step (PC1000) doubles computation but always improves sample quality (against P1000). Moreover, is typically better than doubling the number of predictor steps without adding corrector (P2000), where we have to interpolate between noise scales an ad hoc manner (detailed in Appendix G) for SMLD/DDPM models. In Fig. 9 (Appendix G), we additionally provide qualitative comparison for 9

---

Published as a conference paper at ICLR 2021
Table 2: NLLS and FIDs (ODE) on CIFAR-10. 10.
Table CIFAR-10 sample quality.
Model
NLL Test | FID 1
Model
FID!
IS1
Re (Dinh et al., 2016)
3.49
Conditional
iResNet (Behrmann al., 2019)
3.45
BigGAN (Brock et al., 2018)
14.73
9.22
Glow (Kingma& & Dhariwal, 2018)
3.35
StyleGAAN-ADD (Karmas et al., 2020a)
2.42
10.14
MintNet (Song et al., 2019b)
3.32
Residual Flow (Chen al., 2019)
3.28
46.37
Unconditional
FFIORD (Grathwohl et al., 2018)
3.40
StyleGANZ-ADA (Karras et al., 2020a)
2.92
9.83
Flow++ (Ho et al., 2019)
3.29
NCSN (Song & Ermon, 2019)
25.32
8.87 +: .12
DDPM (L) (Ho et al., 2020)
S 3,70'
13.51
NCSNv2 (Song Ermon, 2020)
10.87
8.40
DDPM (Lammee) (Ho al., 2020)
3.75"
3.17
DDPM (Ho al., 2020)
3.17
9.46 + .11
DDPM++
2.78
9.64
DDPM
3.28
3.37
DDPM++ cont. (VP)
2.55
9.58
DDPM cont. (VP)
3.21
3.69
DDPM++ cont. (sub-VP)
2.61
9.56
DDPM cont. (sub-VP) VP)
3,05
3,56
DDPM++ cont. (deep. VP)
2.41
9.68
DDPM+ cont. (VP)
3.16
3.93
DDPM++
(deep. sub-VP)
2.41
DDPM++ cont. (sub-VP)
3.02
3.16
NCSN++
2.45
9.73
DDPM++ cont. (deep, VP)
3.13
3.08
NCSN++ cont. (VF)
2.38
9.83
DDPM++ cont. (deep, sub-VP)
2.99
2.92
NCSN++ cont. (deep, VE)
2.20
9.89
models trained with the continuous objective Eq. (7) 256 256 LSUN images and the VE SDE, where PC samplers clearly surpass predictor-only samplers under comparable computation, when using a proper number of corrector steps.
4.3 PROBARILITY FLOW AND CONNECTION TO NEURAL ODES
Score-based models enable another numerical method for solving the reverse-time SDE. For all diffusion processes, there exists a corresponding deterministic process whose trajectories share the same marginal probability densities {x) as the SDE. This deterministic process satisfies an ODE (more details in Appendix D.1):
1
f(x, 1) (oxp:))
(13)
which can be determined from the SDE once scores are known. We name the ODE in Eq. (13) the probability flow ODE. When the score function is approximated by the time-dependent score-based model, which is typically neural network, this is an example of a neural ODE (Chen et al., 201 ().
Exact likelihood computation Leveraging the connection neural ODEs, we can compute the density defined by Eq. (13) via the instantaneous change of variables formula (Chen et al., 201 8). This allows us to compute the exact likelihood on any input data (details in Appendix D.2), As an example, we report negative log- aggliikeihhoos (NLLs) measured in bits/dim on the CIFAR-10 dataset in Table 2. We compute log- ligeliils on uniformly dequantized data, and only compare to models evaluated in the same way (omitting models evaluated with variational dequantization (Ho et al., 2019) or discrete data), except for DDPM (L/Isimple whose ELBO values (annotated with *) are reported discrete data. Main results: (i) For the same DDPM model in Ho et al. (2020), we obtain better bits/dim than ELBO, since our likelihoods are exact: (ii) Using the same architecture, we trained another DDPM model with the continuous objective in Eq. (7) (i.e.. DDPM cont.), which further improves the likelihood; (iii) With sub-VP SDES, we always get higher likelihoods compared to VP SDEs; (iv) With improved architecture (i.e., DDPM++ cont., details in Section 4.4) and the sub-VI VP SDE, we can set new record bits/dim of 2.99 on uniformly dequantized CIFAR- 10 even without maximum likelihood training.
Manipulating latent representations By integrating Fq. (13), we can encode any datapoint x(0) into a latent space x( (T)). Decoding can be achieved by integrating corresponding ODE for the reverse- time SDE. As is done with other invertible models such neural ODEs and normalizing flows (Dinh et al., 2016; Kingma & Dhariwal, 2018), we can manipulate this latent representation for image editing, such interpolation, and temperature scaling (see Fig. 3 and Appendix D.4).
Uniquely identifiable encoding Unlike most current invertible models, our encoding is uniquely identifiable, meaning that with sufficient training data, model capacity, and optimization accuracy, the encoding for an input is uniquely determined by the data distribution (Roeder et al., 2020). This because forward SDE, Eq. (5), has no trainable parameters, and its associated probability flow
7

---

Published as a conference paper at ICLR 2021
ODE Evaiuation Points
NFE-14
NTE-86
NIE-348
Interpolation
Evaluation timepoint
Pracision
1e:1
1e.3
1
0.0
10"
10
10"
Evaluation number
Figure 3: Probability flow ODE enables fast sampling with adaptive stepsizes as the numerical precision is varied (leff), and reduces the number of score function evaluations (NFE) without harming quality (middle). The invertible mapping from latents to images allows for interpolations (righ().
ODE, Eq. (13), provides the same trajectories given perfectly estimated scores. We provide additional empirical verification on this property in Appendix D.5.
Efficient sampling As with neural ODEs, we can sample x(0) Po by solving Eq. (13) from different final conditions x(T) ~ pr. Using a fixed discretization strategy we can generate com- petitive samples, especially when used conjuction with correctors (Table 1, "probability flow sampler", details in Appendix D.3). Using a black-box ODE solver (Dormand & Prince, 1980) not only produces high quality samples (Table 2, details in Appendix D.4), but also allows us to explicitly trade-off accuracy for efficiency. With a larger error tolerance, the number of function evaluations can be reduced by over 90% without affecting the visual quality of samples (Fig. 3).
4.4 ARCHITECTTRE IMPROVEMENTS
We explore several new architecture designs for score-based models using both VE and VP SDEs (details in Appendix H), where we train models with the same discrete objectives as SMLD/DDPM. We directly transfer the architectures for VP SDEs sub-VP SDEs due to their similarity. Our optimal architecture for the VE SDE, named NCSN++, achieves an FID of 2.45 on CIFAR-10 with PC samplers, while our optimal architecture for the VP SDE, called DDPM++, achieves 2.78.
By switching to the continuous training objective in Eq. (7), and increasing the network depth, we can further improve sample quality for all models. The resulting architectures are denoted as NCSN++ cont. and DDPM++ cont. in Table for VE and VP/sub- VP SDEs respectively. Results reported in Table 3 are for the checkpoint with the smallest FID over the course of training, where samples are generated with PC samplers. In contrast, FID scores and NLL values in Table 2 are reported for the last training checkpoint, and samples are obtained with black-box ODE solvers. As shown Table 3, VE SDEs typically provide better sample quality than VP/sub-VP SDEs, but also empirically observe that their likelihoods are worse than VP/sub-Ve VP SDE counterparts. This indicates that practitioners likely need to experiment with different SDEs for varying domains and architectures.
Our best model for sample quality, NCSN+ cont. (deep, VE), doubles the network depth and sets new records for both inception score and FID on unconditional generation for CIFAR-10. Surprisingly, we can achieve better FID than the previous best conditional generative model without requiring labeled data. With all improvements together, also obtain the first of high-fidelity samples on CelebA-HQ 1024 x 1024 from score-based models (see Appendix H.3). Our best model for likelihoods, DDPM++ cont. (deep, sub-VP), similarly doubles the network depth and achieves log-like likelihood of 2.99 bits/dim with the continuous objective in Eq. (7). To our best knowledge, this the highest likelihood on uniformly dequantized CIFAR-10.
CONTROLLABLE GENERATION
The continuous structure of our framework allows us to not only produce data samples from Po. but also from p=(x(0) y) if pely x(() is known. Given forward SDE in Eq. (5), we can sample
8

---

Published as a conference paper at ICLR 2021
3
Figure 4: Left: Class-conditional samples on 32 x 32 CIFAR-10. 10. Top four rows are automobiles and bottom four rows are horses. Right: Inpainting (top two rows) and colorization (bottom two rows) results on 256 x 256 LSUN. First column is the original image, second column is the masked/gray- scale image, remaining columns are sampled image completions or colorizations.
from (x(((t) y) by starting from pr (x(T) y) and solving a conditional reverse-time SDE:
dx (f(x,1) (V + Vx Vx Pog x)]]d/ + g({)dw.
(14)
In general, we can use Eq. (14) to solve a large family of inverse problems with score- base- generative models, once given estimate of the gradient of the forward process, Vx p: x())) In some cases, possible to train a separate model to learn the forward process po x(t) and compute its gradient. Otherwise, we may estimate the gradient with heuristics and domain knowledge. In Appendix L4, we provide a broadly applicable method for obtaining such an estimate without the need of training auxiliary models.
We consider three applications of controllable generation with this approach: class-conditions generation, image imputation and colorization. When y represents class labels, we train time-dependent classifier (y x(()) for class-conditional sampling. Since the forward SDE is tractable, we can easily create training data (x( for the time-dependent classifier by first sampling (x(0).y) y) from a dataset, and then sampling x(t) ~ (xx(x(t) x()).. Afterwards, we may employ mixture of cross-entropy losses over different time steps. like Eq. (7), to train the time-dependent classifier (y x())). We provide class- conditionn CIFAR- 10 samples in Fig. 4
(left), and relegate more details and results to Appendix
Imputation is special case of conditional sampling. Suppose we have an incomplete data point y where only some subset, Qyy) is known. Imputation amounts to sampling from p( ((((0) M(y)). which we can accomplish using an unconditional model (see Appendix I.2). Colorization is a special case of imputation, except that the known data dimensions are coupled. We can decouple these data dimensions with orthogonal linear transformation, and perform imputation in the transformed space (details in Appendix I.3). Fig. (right) shows results for inpainting and colorization achieved with unconditional dependent score-based models.
6 CONCLUSION
We presented a framework for score- based generative modeling based on SDEs, Our work enables a better understanding of existing approaches, new sampling algorithms, exact likelihood computation, uniquely identifiable encoding, latent code manipulation, and brings new conditional generation abilities to the family of score-based generative models.
While our proposed sampling approaches improve results and enable more efficient sampling, they remain slower at sampling than GANs (Goodfellow et al., 2014) on the same datasets. Identifying ways of combining the stable learning of score-based generative models with the fast sampling of implicit models like GANs remains an important research direction. Additionally, the breadth of samplers one can use when given access to score functions introduces number of parammtprr..sers.. Future work would benefit from improved methods to automatically select and tune these hyper- parameters, well as more extensive investigation the merits and limitations of various samplers.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
We would like thank Nanxin Chen, Ruiqi Gao, Jonathan Ho, Kevin Murphy, Tim Salimans and Han Zhang for their insightful discussions during the course of this project. This research was partially supported by NSF (#1651565, #1522054, #1733686), ONR (N000141912145) AFOSR (FA95501910024), and TensorFlow Research Cloud. Yang Song was partially supported by the Apple PhD Fellowship in AI/MI.
REFERENCES
Eugene I. Allgower and Kurt Georg. Numerical continuation methods: an introduction, volume 13. Springer Science Business Media, 2012.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Process. Appl., 12(3):
313-326, May 1982.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and förn-Henrik Jacobsen. Invertible residual networks. In International Conference Machine Learning, pp. 573-582, 2019.
Florian Bordes, Sina Honari, and Pascal Vincent. Learning to generate samples from noise through infusion training. arXiv preprint Niti:1703306975 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference on Learning Representations, 2018.
Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.007/3 2020.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in neural information processing systems, pp. 6571-6583, 2018.
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jöm-Henrik Jacobsen. Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems, pp. 9916-9926, 2019.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint
2016.
John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19-26, 1980.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In H. Wallach, H. Larochelle, A. Beygelzimer. F. d' d'Alché-Ban Buch Fox, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 8608-3618 Curran Associates, Inc., 2019.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106(496): 2011.
Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- fion processing systems, pp. 2672-2680, 2014.
Anirudh Goyal Alias Parth Goyal, Nan Rosemary Ke, Surya Ganguli, and Yoshua Bengio. Variational walkback: Learning a transition operator as a stochastic recurrent net. In Advances in Neural Information Processing Systemx, pp. 4392 4402, 2017.
10

---

Published as a conference paper at ICLR 2021
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. In International Confer- ence on Learning Representations, 2018.
Ulf Grenander and Michael Miller. Representations of knowledge in complex systems. Journal of the Royal Statisfical Society: Series B (Methodologicall) 56(4):549-581, 1994.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pp. 2722-2730, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
Neural Information Processing Systems, 33, 2020.
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Communications in Nkatitics-Simmlltiin and Computation, 19(2):333-450, 1990.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(Apr):695 7099,-7000000 2005.
Alexia Sesicoeur-Martinece Rémi Tiché-Taillefee Rémi Tachet des Combes, and Ioannis Mitliagkas. Adversarial score matching and improved sampling for image generation. arXiv preprint arXiv:2009.0544 05475, 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020b.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Peter E Kloeden and Eckhard Platen. Numerical solution of stochastic differential equations, vol- ume 23. Springer Science & Business Media, 2013.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Dimitra Maoutsa, Sebastian Reich, and Manfred Opper. Interacting particle solutions of fokker-planck equations through pradient-log-denssie estimation. arXiv preprint arXiv:2006600702, 2020.
Radford M Neal al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permu- tation invariant graph generation via score- based generative modeling, volume 108 of Proceedings of Machine Learning Research, pp. 4474 4484, Online, 26-28 Aug 2020. PMLR.
11

---

Published as a conference paper at ICLR 2021
Bernt Øksendal. Stochastic differential equations. In Stochastic differential equations, pp. 65- 84. Springer, 2003.
Tianyu Pang, Kun Xu, Chongxuan Li, Yang Song, Stefano Ermon, and Jun Zhu. Efficient learning of generative models via finiteddifference score matching. arXiv preprint arXiv:2007.03317, 2020.
Giorgio Parisi. Correlation functions and computer simulations. Nuclear Physics B, 180(3):378-388 1981.
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems, pp. 14837-14847, 2019.
Geoffrey Roeder, Luke Metz, and Diederik P Kingma. On linear identifiability of learned representa- tions. arXiv preprini 201i::0000000000 2020.
Simo Särkkä and Aro Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
John Skilling. The eigenvalues of mega-dimenssonal matrices. In Maximum Entropy and Bayesian Methods, pp. 455-466 Springer, 1989.
Jascha Sohl-Dickstein, Erie Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In International Conference Machine Learning, pp. 2256-2265, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances Neural Information Processing Systems, pp. 11895-11907, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in Neural Information Processing Systems, 33, 2020.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Proceedings of the Thirty- Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Isruel, July 22-25, 2019, pp. 204, 2019a.
Yang Song, Chenlin Meng, and Stefano Ermon. Mintnet: Building invertible neural networks with masked convolutions. In Advances in Neural Information Processing Systems, pp. 11002-11012, 2019b.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Kridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurlPS, 2020.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- tion, 2317)11661-1674, 2011.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03328 2015.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07144 2016.
Richard Zhang. Making convolutional networks shift-invariant again. In ICML, 2019.
12

---

Published as a conference paper at ICLR 2021
APPENDIX
We include several appendices with additional details, derivations, and results. Our framework allows general SDEs with matrix-valued diffusion coefficients that depend the state, for which we provide detailed discussion in Appendix A. We give full derivation of VE, VP and sub-VP SDBs in Appendix B, and discuss how use them from practitioner's perspective in Appendix C. We elaborate on the probability flow formulation of our framework in Appendix D, including derivation of the probability flow ODE (Appendix D.1). exact likelihood computation (Appendix D.2), probability flow sampling with fixed discretization strategy (Appendix D.3), sampling with black- box ODE solvers (Appendix D.4), and experimental verification on uniquely identifiable encoding (Appendix D.5). We give full description of the reverse diffusion sampler in Appendix E, the DDPM-type ancestral sampler for SMLD models in Appendix F, and Cerdictor-Correate samplers in Appendix G. We explain our model architectures and detailed experimental settings in Appendix H. with 1024 x 1024 CelebA-HQ samples therein. Finally, we detail on the algorithms for controllable generation in Appendix and include extended results for causs-conditionll generation (Appendix I.1), image inpainting (Appendix I.2), colorization (Appendix I.3), and a strategy for solving general inverse problems (Appendix 1.4).
A THE FRAMEWORK FOR MORE GENERAL SDES
In the main text, introduced our framework based simplified SDE Eq. (5) where the diffusion coefficient is independent of x(t). It turns out that our framework can be extended to hold for more general diffusion coefficients. We can consider SDEs in the following form:
dx = f(x, t) G(x, t)dw,
(15)
where f(:,t)= Rd Rd and G(.,t) Rd pddd We follow the Itô interpretation of SDEs throughout this paper.
According to (Anderson, 1982), the reverse-time SDE is given by (ef Eq. (6)) dx [f(x t) V |GG ()(x,))
t) Vx V log (x) G t)w,, (16)
where we define V F(x) == (V (x)) throughout the paper.
(x)) for matrix-valued function
F(x) = (x), fx
The probability flow ODE corresponding to Eq. (15) has the following form (cf., Eq. (13), see detailed derivation in Appendix D.1):
dx = f(x,t) - 2 P (G(xx -
log (o)g))))))) jar. (17)
Finally for conditional generation with the general SDE Eq. (15), we can solve the conditional reverse-time SDE below (cf Eq. (14), details in Appendix I):
T dx {f(x, t) - V [G(x, t) G(x, + () Vx log
- -G(x,t) Vx log x)) G
(18)
When the drift and diffusion coefficient of an SDE are not affine, it can difficult to compute the transition kernel Pn: (x(t) x(0)) in closed form. This hinders the training of buseed models, because Eq. (7) requires knowing V Vx(1) log Pox (x(() x()).. To overcome this difficulty, we can replace denoising score matching Eq. (7) with other efficient variants of score matching that do not require computing Vx(() log Poe Poe (x(t) x())). ). For example, when using sliced score matching (Song
al., 2019a), our training objective Eq. (7) becomes
(19) where A: [0, T R is positive weighting function, (), E[v] = 0. and Covlv] I. We can always simulate the SDE to sample from por (x(() x(()), and solve Eq. (19) to train the time-dependent score-based model ())
0* arg min E,
Ex) Ev-p Ex
J/ |/s (x(() vT sx)
13

---

Published as a conference paper at ICLR 2021
B VE, VP AND SUB- SUB-VP& SDES
Below we provide detailed derivations to show that the noise perturbations of SMLD and DDPM discretizations of the Variance Exploding (VE) and Variance Preserving (VP) SDEs respectively. We additionally introduce sub-VP SDEs, a modification to VP SDEs that often achieves better performance in both sample quality and likelihoods.
First, when using a total of N noise scales, each perturbation kernel Po; (x xo) of SMLD can be derived from the following Markov chain:
X; = = V of 1--1, i= 1, 1....,............... ,,
(20)
where 2;-1 (O, Piata and we have introduced ou 0 simplify the notation. In the limit of N - oc, the Markov chain {x.]/=1 becomes a continuous stochastic process (x())
{ becomes function (t), and z, becomes z(t). where we have used a continuous time variable
t [0. 1] for indexing, rather than an integer {1,2, .N]. Let x (+) x,, (+) d,
and z [ 2, for = .., .N. We can rewrite Eq. (20) as follows with AL 1 and
N
t e {0, N N. N-13 )
x(t At) x(t) Vo At) 0²(t) z(t) x(t) +V (
dloo(t)) At ztt),
dt
where the approximate equality holds when At < 1. In the limit of At - O, this converges to
[t)
dx
dw,
(21)
dt
which is the VE SDE.
For the perturbation kernels (Po, (x xo)) used in DDPM, the discrete Markov chain is
3xx-1 + v ViRi-1, i= N,
(22)
where 7i-1 (O, I). To obtain the limit of this Markov chain when N cc, define auxiliary set of noise scales 13, = N. and re- re-write Eq. (22) as below
B N x-1
B. V
VN == .N.
(23)
In the limit of N - 0c, 1B liz1
becomes function 3(t) indexed by t e [0. Let 3 (+) We can rewrite the Markov chain Eq. (23) as the following with Al
B.,
x(*) Xj, 2( (t) Z;
and telte {0, 1, ..,,, N
N-1):
x(t + At) = -V- 3(+ At x(t) B(t + z(t)
1
x(1
2 3(1+ x))x(() BB(L+ At) z(1)
1
~ x(1)
2 3(1) x(1 + (()) 2(1),
(24)
where the approximate equality holds when At 1. Therefore, in the limit of At Eq. (24) converges to the following VP SDE:
dx 3()x di + /B(1) dw.
(25)
So far, we have demonstrated that the noise perturbations used in SMLD and DDPM correspond to discretizations of VE and VP SDEs respectively. The VE SDE always yields a process with exploding variance when t - c0. In contrast, the VP SDE yields process with bounded variance. In addition, the process has a constant unit variance for all f E [0, oo) when p(x(0)) has a unit variance. Since the VP SDE has affine drift and diffusion coefficients, obtain an ODE that governs the evolution of variance
Eq. (5.51) in Särkkä & Solin (2019) to
d dt
3(1)(I - Evp()),
14

---

Published as a conference paper at ICLR 2021
where Evp(t)= Cov[x((t] for (x(t))==o obeying VP SDE. Solving this ODE, we obtain
(lp(() + elo -(()d Evp(() I):
(26)
from which is clear that the variance Evp(t) is always bounded given Evr(0). Moreover, (t
I if (vp(0) = I. Due to this difference, we name Eq. (9) as the Variance Exploding (VE) SDE, and Eq. (11) the Variance Preserving (VP) SDE.
Inspired by the VP SDE, we propose new SDE called the sub- VP SDE, namely
dx - dt V((((1 e 2 B(x)ds dd..
- Bt)
(27)
Following standard derivations, it is straightforward to show that E[x(()] is the same for both VP and sub-VP SDEs; the variance function of sub-VP SDEs is different, given by
vpu)- I+
-2 Pg)dd I+
St ((s)ds (E (E)) 21).
(28)
where Sahh.VP v() = Co)(x(()) for process (x()) obtained by solving Eq. (27). In addition, we observe that (i) Ssub-vp(t) x Etp(t) for all > 0 with (s)--p(()- = Evp(0) and shared B(s); and (ii) lim Ept)-v((t) lim,-=x Evp(t) = if lim, So (()dds = ap. The former is why we name Eq. (27) the sub- VP SDE its variance always upper bounded by the corresponding VP SDE. The latter justifies the use of sub-VP SDEs for score-based generative modeling, since they can perturb any data distribution to standard Gaussian under suitable conditions, just like VP SDEs.
VE, VP and sub-VP VP SDEs all have affine drift coefficients. Therefore, their perturbation kernels Poe (x(t) x()) are all Gaussian and can be computed with Eqs. (5.50) and (5.51) in Särkkä & Solin (2019):
(VE SDE)
(xx(x(() x())
((x())=x(0e=={ (() I Ie 55 (s)da)) (x(((t)x(0)e) (
(VP SDE)
(29)
a(s)ds -1-c e - (1))
(sub-VI VP SDE)
As result, all SDEs introduced here can be efficiently trained with the objective in Eq. (7).
C SDES IN THE WILD
Below we discuss concrete instantiations of VE and VP SDEs whose discretizations yield SMLD and DDPM models, and the specific sub-VP SDE used in our experiments. In SMLD, the noise scales {or I. is typically geometric sequence where Jmin is fixed to 0.01 and max is chosen according to Technique in Song & Ermon (2020). Usually, SMLD models normalize image inputs
to the range [o, 1]. Since {o a geometric sequence, we have o xt) = a. O
Omu
Anii
for i 1, ... .N. In the limit of N co, we have (t)) m corresponding VE SDE is
Emm Tuin
for (0, 1]. The
dx = Omin
T'axx
22 J
Omax dw, te (0, 17,
(30)
and the perturbation kernel can be derived via Eq. (29):
(x((x( x()) = N (x(t); x(());((0) 2 C
Umnax
2t
1).
te (0,
(31)
min
Tmin
There is one subtlety when / = 0: by definition, (0) ao 0 (following the convention in Eq. (20)), but a (0*) )++ == lim (t) = Fmin * 0. In other words, (t) for SMLD is not differentiable since (0)) a(0*), ), causing the VE SDE in Eq. (21) undefined for t 0. In practice,
bypass this issue by always solving the SDE and its associated probability flow ODE in the range e [e, for some
small constant > and we
10-5 in our VE SDR experiments.
15

---

Published as a conference paper at ICLR 2021
Variance of Perturbation Kernels
Perturbation Kernels
Varlance Perturbation Kernels
2500
SMLDD original
Scaling Factor of Means
1.0
DORM original SDE
1.0
2000
SDE
0.8
0.8
Variance
1500
0.6
Varfance
1000
0.4
0.4
500
0.2
0.2
DDRM original
VP SDE
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
(a) SMLD
(b) DDPM (mean)
(c) DDPM (variance)
Figure 5: Discrete-time perturbation kernels and our continuous generalizations match each other almost exactly. (a) compares the variance of perturbation kernels for SMLD and VE SDE; (b) compares the scaling factors of means of perturbation kernels for DDPM and VP SDE; and (c) compares the variance perturbation kernels for DDPM and VP SDE.
For DDPM models, (8 typically an arithmetic sequence where Br Bui
N N(N-1) (5max
Bmin) for 2,2. N. Therefore, 3(t) 3min + ((3max Bmin) for E [0, 1] in the limit of
N cc. This corresponds to the following instantiation of the VP SDE:
dx - (8min + {(3max xdtt + Bmin + 1(3max Pmin )dw, tE [0, 1J, 2
(32)
where x(0) ~ Pdaaa(()) In our experiments, we let Bmin = 0.1 and Bmax = 20 to match the settings in Ho et al. (2020). The perturbation kernel is given by
Poc (() x())
=N (:P(:an- Bain) t x(0,, I Ie - ( St t E [0, 1J.
(33)
For DDPM, there is no discontinuity issue with the corresponding VP SDE; yet, there are numerical instability issues for training and sampling
0, due to the vanishing variance of x(t) as f 0. Therefore, same as the VE SDE, we restrict computation to t E [e, 1] for a small E > 0. For sampling, we choose 10-3 so that the variance of x(e) in VP SDE matches the variance of x in DDPM; for training and likelihood computation, we adopt e = 10-5 which empirically gives better results.
As sanity check for our SDE generalizations SMLD and DDPM, we compare the perturbation kernels of SDEs and original discrete Markov chains in Fig. 5. The SMLD and DDPM models both use N 1000 noise scales. For SMLD, we only need to compare the variances of perturbation kernels since means are the same by definition. For DDPM, we compare the scaling factors of means and the variances. As demonstrated Fig. the discrete perturbation kernels of original SMLD and DDPM models align well with perturbation kernels derived from VE and VP SDEs.
For sub- VP SDEs, we use exactly same 3()) as VP SDEs. This leads the following perturbation kernel
Por (x)(x(t) x())
= =N (x(t); - P (amn)-BBau--- 2 x(0). - - :
-Bai)) -aac) t.ai Raau) (21), (
t [0, 1]. (34)
We also restrict numerical computation to the same interval of [c, 1] as VP SDEs.
Empirically, we observe that smaller generally yields better likelihood values for all SDEs. For sampling, it is important to use an appropriate for better Inception scores and FIDs, although samples across different e look visually the same to human eyes.
D PROBABILITY FLOW ODE
D.1 DERIVATION
The idea of probability flow ODE inspired by Maoutsa et al. (2020), and one can find the derivation of simplified case therein. Below we provide derivation for the fully general ODE in Eq. (17). We
16

---

Published as a conference paper at ICLR 2021
consider the SDE in Eq. (15), which possesses the following form:
dx f(x, l)de
where f(),t) Rd - Rd and G(,, () Rd Rddxd The marginal probability density px(((t)
evolves according Kolmogorov's forward equation (Fokker-Planck equation) (Øksendal, 2003)
@p((x) δt = Σ
à dzn
[fx(x
1
Σ
a²
Σ
Gu(x (Xxxx (pxxx)
(35)
2
We can easily rewrite Eq. (35) obtain
@pe(x)
2² dr,0r,
d
()x()))
Σ
Gu(xx (xxxx)))
ôt
ozi
2
1-1 5=1
Σ
a
[fx(x,
Σ
0
Σ k=1
Ga(xxt)G5aa(e Gx, (x) (36)
dai
==1
1=1
Note that
2
()p(x))
Σ Ga(x, (x, (xxxx à
p((x)
log (x())
day
V
(G(x,t)
(xx Vx (o)),(
based on which we can continue the rewriting of Eq. (36) to obtain
ēpp(x)
())( +
Ga(xx (Gsxxx
0t
dz.
dz
Zxz
Σ do,
[f((x, ())((x)]
-1t.:3//...../..sss:/1=////s///. () log (x())
[G(x
das
Σ do,
[J((x;
(ogp))))))))))))))))())))))))))) pa(xx)
à (x,(xx))((x))
(37)
dx.
where we define
f(x, () = f(x, - [G(x,))G((x V- - () log
Inspecting Eq. (37), we observe that equals Kolmogorov's forward equation of the following
SDE with G(x, t) = (Kolmogorov's 's forward equation in this case is also known as the Liouville equation.)
dx = f(x, t)dt + G(x, t)dw,
which is essentially an ODE:
dx = f(x, l)dt,
same as the probability flow ODE given by Eq. (17). Therefore, we have shown that the probability flow ODE Eq. (17) induces the same marginal probability density px(x) as the SDE in Eq. (15).
17

---

Published as a conference paper at ICLR 2021
D.2 LIKELIHOOD COMPUTATION
The probability flow ODE in Eq. (17) has the following form when we replace the score Vx logPr(x)
with the time-dependent score-based model So (x l):
f(x, () v [G(x, (x, () 1 O((x
dx
(38)
2
(x,1)
With the instantaneous change of variables formula (Chen et al., 2018), we can compute the log- likelihood of po (x) using
Po (x(0)) log (x(()) + V (x))
(39)
where the random variable x(t) as function of t can be obtained by solving the probability flow ODE in Eq. (38). In many cases computing V fx t) is expensive, so we follow Grathwohl et al. (2018) to estimate it with the Skiling-Hutccinng trace estimator (Skilling, 1989; Hutchinson, 1990). In particular, we have
fo(x, 1) = Ep(e)LE Vfo(x, ()e],
(40)
where Vfg denotes the Jacobian of (), and the random variable satisfies Epte)]e] 0 and Covpce/c I. The vector-Jacobian product c (T t) can be efficiently computed using reverse- mode automatic differentiation, at approximately the same cost evaluating fo (oxx, (). As a result, we can sample
p(e) and then compute efficient unbiased estimate to V fo t) using E Vfo((x, f)e. Since this estimator is unbiased, over a sufficient number of runs. Therefore, by applying the seilling-Hutchinson estimator Eq. (40) Eq. (39), we can compute the log-likelihood any accuracy.
attain an arbitrarily small error by averaging
In our experiments, we use the RK45 ODE solver (Dormand & Prince, 1980) provided by S..py.integratee solve_ivp in all cases. The bits/dim values in Table 2 are computed with atol=le-5 and rtol=le-5, same as Grathwohl et al. (2018). To give the likelihood results of our models in Table 2, we average the bits/dim obtained the test dataset over five different runs with = 10-5 (see definition of e in Appendix C).
D.3 PROBABILITY FLOW SAMPLING
Suppose we have a forward SDE
dx =
dx=f(xxtb) f(x,t)dtt + G(f)dw,
and one of its discretization
X + f((x;) G i= 0,1,,... NN
(41)
where 2, ~ NN I). We assume the discretization schedule time fixed beforehand, and thus we absorb the dependency on At into the notations of f; and G, Using Eq. (17), we can obtain the following probability flow ODE:
dx [(xx1) 2
1 G())G()) Vxlop=x) dt.
1xg
(42)
We may employ any numerical method to integrate the probability flow ODE backwards in time for sample generation. In particular, we propose a discretization in similar functional form to Eq. (41):
x; x - (x;+())+) +
2 G.+1 GT (xi+1, i 1), i= 0,1, 1.1.....N---- 1,
where the score-based model Sg* (x;, is conditioned on the iteration number This a determin- istic iteration rule. Unlike reverse diffusion samplers ancestral sampling, there is additional randomness once the initial sample xw is obtained from the prior distribution. When applied to SMLD models, we can get the following iteration rule for probability flow sampling:
x; x _ (oē+ of son (x+1), = 0. ,N-L --1 1.
(43)
Similarly, for DDPM models, we have
x, = =2 VI x + = 1 (x + +1); i= 0,1,, .. .
(44)
18

---

Published as a conference paper at ICLR 2021
D.4 SAMPLING WITH BLACK-BOX BOX ODE SOLVERS
For producing figures in Fig. 3, same settings in Ho et al. (2020). All FID scores of our models Table 2 are computed on samples from the RK45 ODE solver implemented in sc scipy- integrate............ s ve vv with atol=le-5 and rttl=loeeee--.. We use
DDPM model trained 256 256 CelebA-HQ with the
10 for VE SDEs and e 10-3 for VP SDEs (see also Appendix C).
Aside from the interpolation results in Fig. 3, demonstrate more examples of latent space manipulation in Fig. 6, including interpolation and temperature scaling. The model tested here is DDPM model trained with the same settings in Ho al. (2020).
Although solvers for the probability flow ODE allow fast sampling, their samples typically have higher (worse) FID scores than those from SDE solvers if corrector used. We have this empirical observation for both the discretization strategy in Appendix D.3, and black-box ODE solvers introduced above. Moreover, the performance of probability flow ODE samplers depends on the choice of the SDE their sample quality for VE SDEs is much worse than VP SDEs especially for high- dimensional data.
D.5 UNIQUELY IDENTIFIABLE ENCODING
As a sanity check, we train two models (denoted as "Model A" and "Model B*) with different architectures using the VE SDE on CIFAR-10. Here Model A is NCSN+ model with layers per resolution trained using the continuous objective in Eq. (7), and Model all the same except that uses 8 layers per resolution. Model definitions are in Appendix H.
We report the latent codes obtained by Model and Model B for random CIFAR-I0 image in Fig. 7. In Fig. 8, we show the dimension-wist differences and correlation coefficients between latent encodings on a total of CIFAR-10 10 images. Our results demonstrate that for the same inputs, Model and Model B provide encodings that are close in every dimension, despite having different model architectures and training runs.
E REVERSE DIFFUSION SAMPLING
Given forward SDE
dx = f(x, t)dt + G(f)dw,
and suppose the following iteration rule a discretization of it:
X;+1 x, f((x,) + G,z, i = 0, 1, -
(45)
where z; ~ NO, I). Here we assume the discretization schedule of time is fixed beforehand, and thus we can absorb into the notations of f, and G,
Based on Eq. (45), we propose to discretize the reverse-time SDE
dx = [f(x, !) G())G(I)Vv logp: + G(I)dw,
with similar functional form, which gives the following iteration rule for i e (0, 1.... ,N ,N 1}:
x; x fi+1 (x++1) + G G G (X++1, i 1) + G
(46)
where our trained score-based model Sa* (x,, i) is conditioned on iteration number i.
When applying Eq. (46) to Eqs. (10) and (20), we obtain new set of numerical solvers for the reverse-time VE and VP SDEs, resulting in sampling algorithms shown in the "predictor" part of Algorithms and 3. We name these sampling methods (that are based on the discretization strategy in Eq. (46)) reverse diffissin samplers.
As expected, the ancestral sampling of DDPM (Ho et al., 2020) (Eq. (4)) matches its reverse diffusion counterpart when 3 0 for all i (which happens when At since 8: B,At, Appendix B),
19

---

Published as a conference paper at ICLR 2021
Figure Samples from the probability flow ODE for VP SDE on 256 x 256 CelebA-HQ Top:
spherical interpolations between random samples. Bottom: temperature rescaling (reducing norm of embedding).
20

---

Published as a conference paper at ICLR 2021
Latent value
100
Model
Model
-100
20
40
100
Dimension
Figure 7: Comparing the first 100 dimensions of the latent code obtained for random CIFAR-10 image. "Model A" and "Model B" are separately trained with different architectures.
Model
600
x1(7)
vs.
Model
r=0.96
Count
400
Mode|
Model (shuffled)
200
vs. Model (shuffled)
Model A
100
200
300
Difference in encodings
Correlation Coefficient
Figure 8: Left: The dimension-win difference between encodings obtained by Model A and B. As baseline, we also report the difference between shuffled representations these two models. Right: The dimension-wind -iise correlation coefficients of encodings obtained by Model A and Model B.
because
(xi+1 + B44180 (x+1,, 1)) + v B1412
Bi+1
Bi+ + o(8411) (x5+1 + Bet (x++1. 1)) + v B1412441
2 Bi+1
(xi+1 + (x011 1)) + V 11+1
Bi+1
Xi+1 + 1: (x
34 Saa (x+1, 1) / B1412i+11
Bi+1
Xi+1
11+180
(x
31+12++1
2
3++1
Xi+1 10 1) v 35-12
2
1
2
31+1
0(33-1)
Xi+1 + Bi+ 18x (xi-1, i 1) / B1412a
2
-(2 / B x;-1+ Bi+ (xi+1, i 1) / BBt12i+1.
Therefore, the original ancestral sampler of Eq. (4) is essentially a different discretization to the same reverse-time SDE. This unifies the sampling method in Ho et al. (2020) numerical solver to the reverse-time VP SDE in our continuous framework.
F ANCESTRAL SAMPLING FOR SMLD MODELS
The ancestral sampling method for DDPM models can also be adapted to SMLD models. Consider sequence of noise scales 02 < ... < ax as in SMLD. By perturbing data point Xo with these noise scales sequentially, we obtain Markov chain X X1
XN, where
p(x; x--1) (xx;;; (o} I), = 1,2, N.
21

---

Published as a conference paper at ICLR 2021
Algorithm 1 Predictor-Corrector (PC) sampling
Require:
N: Number of discretization steps for the reverse-time SDE
M: Number of corrector steps
1: Initialize x ~ (r(x)
2: for i = N 1 to 0 do
3: X; < Predictor(x;+1]
4: for
to .M do
5:
X; - Corrector(x:)
6:
return Xg
Here assume oo 0 to simplify notations. Following Ho al. (2020), can compute
(xixxiii -ب + (1 (-)z)
of -لر of
20
q(x,-1 x,, Xo) /
xo:
of
If we parameterize the reverse transition kernel as pe (x:-1 1 x,) = 1o i), 1, then
I Bgl DRL (q(x; X;, (o)) po x:)
o
X; (1
0/-1 0
Bgg
Xo (x(();;))
+
2T
1
o. 0 (x(x
Ex
(x0,
+C,
2r
a:
where L,-1 one representative term in the ELBO objective (see Eq. (8) Ho al. (2020)), is constant that does not depend on 0, NN I), and (x0, z) + ozz. We can therefore parameterize (x(x;, i) via
Ne (x i) + (o of-1)s8 (x) 1);
of (o -
where so (x;, i) is estimate 2/0,. As in Ho al. (2020), we let Ti
Through
of
ancestral sampling on IT:h1 po po(x; - x;), we obtain the following iteration rule
x;-1 = x; + (o? of (xi, +
of 2 1,2, N, (47)
where XN 0f I). 0% denotes the optimal parameter of sø, and z; NNO,,), I). We call Eq. (47) the ancestral sampling method for SMLD models.
G
SAMPLERS
Predictor-Correcctor (PC) sampling The predictor can be any numerical solver for the reverse- time SDE with fixed discretization strategy. The corrector can be any score-based MCMC approach. In PC sampling, we alternate between the predictor and corrector, as described in Algorithm 1. For example, when using the reverse diffusion SDE solver (Appendix E) the predictor, and annealed Langevin dynamics (Song & Ermon, 2019) as the corrector, we have Algorithms 2 and for VE and VP SDEs respectively, where {G No) N are step sizes for Langevin dynamics as specified below.
The corrector algorithms We take the schedule of annealed Langevin dynamics in Song & Ermon (2019), but re-frame it with slight modifications in order to get better interpretability and empirical performance. We provide the corrector algorithms in Algorithms and respectively, where we call the "signal-to-noise" ratio. We determine the step size using the norm of the Gaussian noise || norm of the acore-based model ||s@* " ll, and the signal-to-noise ratio r. When sampling a large batch of samples together, we replace the norm ||-11, with the average norm across the batch. When the batch size is small, suggest replacing ||z||| with Vd, where d is the dimensionality of z.
22

---

Published as a conference paper at ICLR 2021
Algorithm 2 PC sampling (VE SDE)
Algorithm 3 PC sampling (VP SDE)
1: xN .V min I)
1: xx (0,1))
2: for i = N 1 to
0 do
2: for
t do
3: x Xi+1
(oit1
oi 80x (x41
3: x < (2 v1
B441lx141 + 11 (X:41.
1)
4:
10)
4: (0
5: x; - x;
8++1 - oiz
5: X; - x, + /B.-12
Predictor
6: for j = to M do
6: for j to M do
Corrector
7: z ~~ I.
7: I..
8:
x, x, 6g50+ (x,, 0i) V2,z
8: x, x, + 6 (x,, + /2e,z
9: return
9: return X0
Algorithm 4 Corrector algorithm (VE SDE).
Algorithm 5 Corrector algorithm (VP SDE).
Require: {o,)]=1, T, .N,
Require: {3.), r,, M.
I: x (N, 0 max I)
1: x 10,
2: for < to do
2: for i < No 1 do
3: for 1 . do
3: for j -
M do
4:
z ~ ~N(0,11
4: z ~ IN
5:
Sø* (
5:
g - Søn (x 1x))
6:
c 2(r |2/22/ ||g )
2r |z||2 [g])2))
7:
x - x +eg + 2e z
7: x - x +eg + 2
8:
xf_ XM
8:
xf x
return x
return xB
Denoising For both SMLD and DDPM models, the generated samples typically contain small noise that is hard to detect by humans. As noted by Dolicoeur-Marting et al. (2020), FIDs can be significantly worse without removing this noise. This unfortunate sensitivity to noise is also part of the reason why NCSN models trained with SMLD has been performing worse than DDPM models in terms of FID, because the former does not use a denoising step at the end of sampling, while the latter does. In all experiments of this paper ensure there is single denoising step the end of sampling, using Tweedie's 's formula (Efron, 2011).
3000
3000
Total functtor exelustors
Too! score fur cilon www.nnnnnnnnnccoooooo aislussoon
2000
2200
Langevin stops per noise level
Langevin per noise
Figure 9: PC sampling for LSUN bedroom and church. The vertical axis corresponds the total computation, and the horizontal axis represents the amount of computation allocated to the corrector. Samples are the best when computation is split between the predictor and corrector.
Training We use the same architecture in Ho et al. (2020) for our score-based models. For the VE SDE, we train model with the original SMLD objective in Eq. (1); similarly for the VP SDE, we use the original DDPM objective in Eq. (3). We apply a total number of 1000 noise scales for training both models. For results in Fig. train an NCSN+ model (definition in Appendix H) on
23

---

Published as a conference paper at ICLR 2021
Table 4: Comparing different samplers on CIFAR-10, where "P2000" uses the rounding interpolation between noise scales. Shaded regions are obtained with the same computation (number of score function evaluations). Mean and standard deviation are reported over five sampling runs.
Variance Exploding SDE (SMLD)
Variance Preserving SDE (DDPM)
FID. Sampler
P1000
P2000
C2000
PC1000
P1000
P2000
C2000
PC1000
Predictor
ancestral sampling
4.98
4.92
3.62
3.24
3.11
3.21
reverse diffusion
4.79 a. 07
4.72 a 07
20.43 a 07
3.60
.22
3.21 a m
3.10 : 40
19.06 1. 06
3.18
probability flow
15.41 +
12.87 0.09
3.51
.14
3,59 044
3.25 +
3.06 +:
Table 5: Optimal signal-to-noise ratios of different samplers. "P1000" "P2000": predictor-only samplers using 1000 or 2000 steps. "C2000": corrector-only samplers using 2000 steps. "PC1000": PC samplers using 1000 predictor and 1000 corrector steps.
VE SDE (SMLD)
VP SDE (DDPM)
Sampler
P1000
P2000
C2000
PC1000
P1000
P2000
C2000
PC1000
Predictor
ancestral sampling
0.17
0.01
reverse diffusion
0.22
0.16
0.27
0.01
probability flow
0.17
0.04
256 x 256 LSUN bedroom and church_outdou (Yu et al., 2015) datasets with the VE SDE and our continuous objective Eq. (7). The batch size is fixed to 128 on CIFAR-10 and on LSUN.
Ad-hoc interpolation methods for noise scales Models in this experiment are all trained with 1000 noise scales. To get results for P2000 (predictor-only sampler using 2000 steps) which requires 2000 noise scales, we need to interpolate between 1000 noise scales at test time. The specific architecture of the coiee-conditional score-based model in Ho et al. (2020) uses sinusoidal positional embeddings for conditioning on integer time steps. This allows us to interpolate between noise scales test time an ad-hoc way (while it is hard do so for other architectures like the one Song Ermon (2019)). Specifically. for SMLD models, we keep min and F fixed and double the number of time steps. For DDPM models, we halve Amin and Sams before doubling the number of time steps. Suppose (x, i) - denote the corresponding interpolated score-based model at 2.V time steps. We test two different interpolation strategies for time steps: linear interpolation where syg (x, so i/2) and rounding interpolation where s/g xx i) so (x, |i/2]). We provide results with linear interpolation in Table 1, and give results of rounding interpolation in Table 4. We observe that different interpolation methods result in performance differences but maintain the general trend of predictor-corrector methods performing on par or better than predictor-onle or corrector-only samplers.
score-based model trained on N time steps, and let fs& (x, i) ) 2
Hyper-parameters of the samplers For Predictor-Corrector and corrector-only samplers on CIFAR-10, search for the best signal-to-noise ratio (r) over grid that increments at 0.01. We report the best in Table 5. For LSUN bedroom/church. ootdooo, we fix r to 0.075. Unless otherwise noted, we use one corrector step per noise scale for all PC samplers. We two corrector steps per noise scale for corrector-onle samplers on CIFAR-10. 10. For sample generation, the batch size 1024 on CIFAR- 10 and 8 LSUN bedroom/church outdoor.
H ARCHITECTURE IMPROVEMENTS
We explored several architecture designs to improve score-based models for both VE and VP SDEs. Our endeavor gives rise to new state-of-the-ee sample quality on CIFAR-10, new state-of-tt of-the---- likelihood uniformly dequantized CIFAR-10, and enables the first figh-fidelity image samples of resolution 1024 x 1024 from score-basee based generative models. Code and checkpoints are sourcee https:/githbb..omyyaggsssgg///// sde.
24

---

Published as a conference paper at ICLR 2021
H.1 SETTINGS FOR ARCHITECTURE EXPLORATION
Unless otherwise noted, all models are trained for 1.3M iterations, and we save one checkpoint per 50k iterations. For VE SDEs, we consider two datasets: 32 x 32 CIFAR- 10 (Krizhevsky et al., 2009) and 64 x 64 CelebA (Liu et al., 2015), pre-pu pre-processed following Song & Ermon (2020). We compare different configurations based on their FID scores averaged over checkpoints after 0.5M iterations. For VP SDEs, we only consider the CIFAR-10 dataset save computation, and compare models based on the average FID scores over checkpoints obtained between 0.25M and 0.5M iterations, because FIDs turn to increase after 0.5M iterations for VP SDES.
All FIDs are computed on 50k samples with tensorf 1 low gan. For sampling, we use the PC sampler discretized 1000 time steps. We choose reverse diffusion (see Appendix E) the predictor. We use one corrector step per update of the predictor for VE SDEs with a signal-to-noise ratio of 0.16, but save the corrector step for VP SDEs since correctors there only give slightly better results but require double computation. We follow Ho et al. (2020) for optimization, including the learning rate, gradient clipping, and learning rate warm-up schedules. Unless otherwise noted, models are trained with the original discrete SMLD and DDPM objectives in Eqs. (1) and (3) and use batch size of 128. The optimal architectures found under these settings are subsequently transferred to continuous objectives and deeper models. We also directly transfer the best architecture for VP SDEs to sub- uub-VP SDEs, given the similarity of these two SDEs.
Faise
False
depm
True
True
biggan
3.5
3.0
3.0
2.5
2.5
numnrre,, harnures_bblcce
Progressive Arch, none inpur, ikpurskipp, none
4.0
output skip
inpur, skip. outpur_ skip skip
3.5
3.5
none, residual
3.0
Input skip, residual residual
2.5
OFAR-10
CelebA
dataset
dataset
Figure 10: The effects of different architecture components for score- score-based models trained with VE perturbations.
Our architecture mostly based on Ho al. (2020). We additionally introduce the following components to maximize the potential improvement of score-based models.
1. Upsampling and downsampling images with anti-aliasing based Finite Impulse Re- sponse (FIR) (Zhang, 2019). We follow the same implementation and syper-parameters in StyleGAN-2 (Karras et al., 2020b).
2. Rescaling all skip connections by 1/v2. This has been demonstrated effective in several best- in-class GAN models, including ProgressiveGAN (Karras al., 2018), StyleGAN (Karras et al., 2019) and StyleGAN-2 (Karras et al., 2020b).
3. Replacing the original residual blocks in DDPM with residual blocks from BigGAN (Brock et al., 2018).
4. Increasing the number of residual blocks per resolution from 2 4.
25

---

Published as a conference paper at ICLR 2021
5. Incorporating progressive growing architectures. We consider two progressive architectures for input: "imput skip" and "residual", and two progressive architectures for output: "output skip" and "residual". These progressive architectures are defined and implemented according to StyleGAN-2.
We also tested equalized learning rates, a trick used in very successful models like Progressive- GAN (Karras et al., 2018) and StyleGAN (Karras al., 2019). However, we found harmful an early stage of our experiments, and therefore decided not to explore more on it.
The exponential moving average (EMA) rate has a significant impact on performance. For models trained with VE perturbations, we notice that 0.999 works better than 0.9999, whereas for models trained with VP perturbations is the opposite. We therefore use an EMA rate of 0.999 and 0.9999 for VE and VP models respectively.
H.2 RESULTS ON CIFAR-10 10
All architecture components introduced above can improve the performance of score basee models trained with VE SDEs, as shown in Fig. 10. The box plots demonstrate the importance of each component when other components can vary freely. On both CIFAR-10 10 and CelebA, the additional components that we explored always improve the performance on average for VE SDEs. For progressive growing, it is not clear which combination of configurations consistently performs the best, but the results are typically better than when no progressive growing architecture is used. Our best score-based model for VE SDEs 1) uses FIR upsampiin//dowssampiigg 2) rescales skip connections, 3) employs BigGAN-type residual blocks, 4) uses 4 residual blocks per resolution instead of 2, and 5) uses "residual" for input and progressive growing architecture for output. We name this model "NCSN++", following the naming convention of previous SMLD models (Song & Ermon, 2019; 2020).
We followed a similar procedure to examine these architecture components for VP SDEs, except that we skipped experiments on CelebA due limited computing resources. The NCSN++ architecture worked decently well for VP SDEs, ranked 4th place over all 144 possible configurations. The top con- figuration, however, has slightly different structure, which uses no FIR upsampliggddowsaamliigg and no progressive growing architecture compared to NCSN++. We name this model "DDPM++", following the naming convention of Ho et al. (2020).
The basic NCSN++ model with 4 residual blocks per resolution achieves an FID of 2.45 on CIFAR-10, whereas the basic DDPM++ model achieves an FID of 2.78. Here in order to match the convention used in Karras et al. (201 8); Song & Ermon (2019) and Ho et al. (2020), we report the lowest FID) value over the course of training, rather than the average FID value over checkpoints after 0.5M iterations (used for comparing differrnnt models of VE SDEs) or between 0.25M and 0.5M iterations (used for comparing VP SDE models) in our architecture exploration.
Switching from discrete training objectives to continuous ones in Eq. (7) further improves the FID values for all SDEs. To condition the NCSN+ model on continuous time variables, we change positional embeddings, the layers in Ho et al. (2020) for conditioning random Fourier feature embeddings (Tancik et al., 2020). The scale parameter of these random Fourier feature embeddings fixed 16. We also reduce the number of training iterations to 0.95M to suppress overfitting. These changes improve the FID on CIFAR-10 from 2.45 to 2.38 for NCSN+ trained with the VE SDE, resulting in a model called "NCSN+ cont.". In addition, we can further improve the FID from 2.38 to 2.20 by doubling the number of residual blocks per resolution for NCSN++ cont.. resulting in the model denoted as "NCSN++ cont. (deep)". All quantitative results are summarized in Table 3, and we provide random samples from our best model in Fig. 11.
discrete time steps, to
Similarly, we can also condition the DDPM++ model on continuous time steps, resulting in a model "DDPM++ cont.". When trained with the VP SDE, it improves the FID of 2.78 from DDPM++ to 2.55. When trained with the sub-VP SDE, it achieves an FID of 2.61. To get better performance, we used the Euler-Maruyama solver as the predictor for continuously
models, instead of the ancestral sampling predictor or the reverse diffusion predictor. This is because the discretization strategy of the original DDPM method does not match the variance of the continuous process well when' t - 0, which significantly hurts FID scores. As shown in Table 2, the likelihood values are 3.21 and 3.05 bits/dim for VP and sub-VE VP SDEs respectively. Doubling the depth, and trainin with
26

---

Published as a conference paper at ICLR 2021
Figure 11: Unconditional CIFAR-10 samples from NCSN+ cont. (deep, VF).
27

---

Published as a conference paper at ICLR 2021
Figure 12: Samples on 1024 x 1024 CelebA-HQ from modified NCSN+ model trained with the VE SDE.
28

---

Published as a conference paper at ICLR 2021
0.95M iterations, we improve both FID and bits/dim for both VP and sub- VP SDEs, leading to model "DDPM++ cont. (deep)". Its FID score is 2,41, same for both VP and sub- sub-VP SDEs. When trained with the sub-VP SDE, it can achieve likelihood of 2.99 bits/dim. Here all likelihood values are reported for the last checkpoint during training.
H.3 HIGH RESOLUTION IMAGES
Encouraged by the success of NCSN++ on CIFAR-IO, 10, we proceed to test it on 1024 x 1024 CelebA- HQ (Karras et al.. 2018), a task that was previously only achievable by some GAN models and VQ-VAR-2 (Razavi et al., 2019). We used batch size of 8, increased the EMA rate to 0.9999, and trained model similar to NCSN+ with the continuous objective (Eq. (7)) for around 2.4M iterations (please find the detailed architecture our code release.) We use the PC sampler discretized at 2000 steps with the reverse diffusion predictor, one Langevin step per predictor update and signal-to-noise ratio of 0.15. The scale parameter for the random Fourier feature embeddings is fixed to 16. We use the "input skip" progressive architecture for the input, and "output skip" progressive architecture for the output. We provide samples in Fig. 12. Although these samples are not perfect (e.g. there are visible flaws on facial symmetry), believe these results are encouraging and can demonstrate the scalability of our approach. Future work on more effective architectures are likely to significantly advance the performance of score-based generative models on this task.
CONTROLLABLE GENERATION
Consider forward SDE with the following general form
dx = f(x, t)de + G(x, t)dw,
and suppose the initial state distribution is po (x( y). The density time is Px(x(t) y) when conditioned on y. Therefore, using Anderson (1982), the reverse-time SDE is given by
dx = (f(x, t) - V [G(x, (x 1) - G(x, G), T) log x yy) + G(x, f)dw. (48)
Since Pe (x(() y (x)) x(t)), the score Vx log (x(() y) can be computed easily by Vx log (x y) = Vx lolp..(x(())) + Vx logp(y x(t))-
(49)
This subsumes the conditional reverse-time SDE in Eq. (14) as a special case. All sampling methods we have discussed so far can be applied to the conditional reverse-time SDE for sample generation.
L11
SAMPLING
When y represents class labels, we can train a time-dependent classifier (y x(t)) for class- conditional sampling. Since the forward SDE is tractable, we easily create pair of training data (x(t). y) by first sampling (x(0). y) from a dataset and then obtaining x(f) ~ Pue (x((t) x(0).. Afterwards, may employ mixture of cross-entropy losses over different time steps, like Eq. (7), to train the time-dependent classifier (y x(t)).
To test this idea, we trained Wide ResNet (Zagoruyko & Komodakis, 2016) Wwide-ResNet-22--
on CIFAR-10 with VE perturbations. The classifier is condi- tioned on log o, using random Fourier features (Tancik et al., 2020), and the training objective is a simple sum of cross-entropy losses sampled at different scales. We provide a plot to show the accuracy of this classifier over noise scales in Fig. 13. The score-based model an unconditional NCSN++ (4 blocks/resolution) in Table 3, and we generate samples using the PC algorithm with 2000 discretization steps. The class-conditional samples are provided in Fig. 4, and an extended set of conditional samples is given in Fig. 13.
1.2 IMPUTATION
Imputation a special case of conditional sampling. Denote by 2(x) and f((x) the known and un- known dimensions of respectively, and let t) and G denote f(.,t) t) and G(- t) restricted to the unknown dimensions. For VE/VP SDEs, the drift coefficient f(-, t) is element-wise. and the diffusion coefficient G(:,)) is diagonal. When f(:,t) is element-wise, fo(: t) denotes the same
29

---

Published as a conference paper at ICLR 2021
class: airplane
class: automobile
class: bird
class: cat
class: deer
class: dog
class: frog
class: horse
class: ship
class: truck
Accuracy vs. noise scale
0.8
Accuracy
0.6
10-2
10-1
10a
Figure 13: Class-conditional image generation by solving the conditional reverse-time SDE with PC.
The curve shows the accuracy of noise-conditional classifier over different noise scales.
30

---

Published as a conference paper at ICLR 2021
clement-wist function applied only to the unknown dimensions. When G(.,t) t) is diagonal, G denotes the sub- uub-matrix restricted to unknown dimensions.
For imputation, goal is to sample from p({((x(0)) 2(x(0)) y). Define new diffusion process ztt) = (x)(,)), and note that the SDE for z(t) can be written as
dz (z, Gz f)dw.
The reverse-time SDE, conditioned on (2(x(()) = y, given by
dz = (z) v [G, (2,
(n(2,t)Gpp t) V2 V, log ((((()) - y)] + Go(z, ()dw.
Although pr(z(t) [((( = y) general intractable, it can be approximated. Let denote the event ((x(()) We have
Pr(x(1) (2((()) y) ps (z(1) A) pe(z(() [(((())), ()p((((((1)) 4)dA(x(1))
(x(((()))) = [(x(t)),
() [(x(O)))
(e(2(t) ((x(t))),
where (((((t) is a random sample from (52(x(t)) Therefore,
A), which is typically a tractable distribution.
Vx (og ((x(0)) = = Vz (z) ((((()))
Vz logg ((())(((()):((((((()]
where [z(t); ((((())] denotes a vector u(t) such that M(u(t) = ((x(()) and M(u(t)) = 2(),, and the identity holds because V log (og):(((((;;((((()))) Va log p({((((t)) Va log (z((t) ((x(())))
Vz log)))(((() ((((())) +
We provided an extended set of inpainting results in Figs. 14 and 15.
1.3 COLORIZATION
Colorization is special case of imputation, except that the known data dimensions are coupled. We can decouple these data dimensions by using an orthogonal linear transformation to map the gray-scale image to a separate channel different space, and then perform imputation complete the other channels before transforming everything back to the original image space. The orthogonal matrix we used to decouple color channels
0.577
0.816
0.577
0.408
0.707
0.577
0.408
-0.707
Because the transformations are all orthogonal matrices, the standard Wiener process w(I) will still be a standard Wiener process in the transformed space. allowing us build an SDE and use the same imputation method in Appendix I.2. We provide an extended set of colorization results in Figs. 16 and 17.
1.4 SOLVING GENERAL INVERSE PROBLEMS
Suppose have two random variables and y, and know the forward process of generating y from x, given by p(y x). The inverse problem is to obtain x from y, that is, generating samples from p(x y). In principle,
estimate the prior distribution p(x) and obtain p(x y) using Bayes' rule: p(x y) = p(x)p(y x)/p(y). In practice, however, both estimating the prior and performing Bayesian inference non-trivial.
Leveraging Eq. (48), based generative models provide one way to solve the inverse problem. Suppose have diffusion process (x())){ generated by perturbing with SDE, and
31

---

Published as a conference paper at ICLR 2021
time-dependent score-bases model (x(()) trained to approximate Vx po (x(())) Once have an estimate Vx log Pr (x((t) y), we can simulate the reverse-time SDE in Eq. (48) to sample from Po (x(0) y) p(x y). To obtain this estimate, first observe that
f pi(x(t) yy(), v)p(y(t) y)dy(t),
Vx P: yy) = Vx log
where yt) defined via (t) and the forward process p(y(t) x x())) Now assume two conditions:
p(y(t) | y) tractable. We can often derive this distribution from the interaction between the forward process and the SDE, like in the case of image imputation and colorization.
(x((()) y) R px(((t) y())). For small f, y(t) is almost the same as y so the approximation holds. For large t, y becomes further away from x(t) in the Markov chain, and thus have smaller impact on x(t). Moreover, the approximation error for large t matter less for the final sample, since is used early in the sampling process.
Given these two assumptions, have
Vx (og y)) = Vx log (x(((t) (x())(((() yydy
Vx lo (x(t) (t)))
Vx lx f)) + V P) xx())
Sga (x(t), f) + Vx log
x(t)).
(50)
where (() is sample from p(y(1) y). Now can plug Eq. (50) into Eq. (48) and solve the resulting reverse-time SDE to generate samples from p(x y).
32

---

Published as a conference paper at ICLR 2021
9
9
Figure 14: Extended inpainting results for 256 256 bedroom images.
33

---

Published as a conference paper at ICLR 2021
9
9
Figure 15: Extended inpainting results for 256 x 256 church images.
34

---

Published as a conference paper at ICLR 2021
fre No
Bore Now
More Nov
prr Now
ore Now
oore Nov
Now
ore
Figure 16: Extended colorization results for 256 256 bedroom images.
35

---

Published as a conference paper at ICLR 2021
Figure 17: Extended colorization results for 256 x 256 church images.
36


Reviews:
[{"review_id": "zLwPct3Kc3k", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY\n\nThe submission proposes a score-based generative model, which uses an SDE to map the\ndata distribution to a simple noise distribution and the corresponding reverse-time SDE to\ngenerate samples by mapping the noise to the data space. The proposed model builds\nupon and generalises two existing models (SMLD and DDPM) by transforming the data\nusing a continuous SDE dynamics as opposed to perturbing the data with a finite number\nof noise distributions utilized by these models.\n\n##################################################################\n\nREASON FOR SCORE\n\nThe paper provides a clear motivation for the proposed modifications to SMLD and DDPM,\nas well as a clear technical description of these modifications, their analysis and discussion.\nI think the proposed model and sampling algorithms offer substantial conceptual\nimprovements to the existing models and should be of interest to the community. The\npaper is well written and structured.\n\n##################################################################\n\nPROS\n\n+ Clear motivation for the work.\n+ Detailed technical description of the proposed models.\n+ Interesting discussion of similarities and differences between SMLD, DDPM, and the SDE\nbased model, as well as corresponding sampling algorithms.\n+ Extensive experiments.\n\n##################################################################\n\nCONS\n\n- I found the discussion of equivalent neural ODE and its differences to reverse SDE\n a bit short, especially given that it is used in multiple experiments.\n- The case of using general SDEs (not only those derived from SMLD and DDPM) is\nmentioned only briefly, leaving it unclear if using a general SDE would require\nrelatively simple changes, or if the proposed model is limited to SDEs derived from\nSMLD and DDPM.\n\n##################################################################\n\nQUESTIONS and COMMENTS\n\n- Is it correct that the function \\sigma(t) in Eq. (6) is assumed to be monotonic\nand bounded by \\sigma_max, while \\beta(t) in Eq. (8) is bounded by 1, but doesn't have to\nbe monotonic?\n\n- In the case of general SDE for noise perturbations in Eq. (9), are there any assumptions on\ndrift and diffusion function (such as monotonicity or boundedness)?\n\n- In the case of SDEs (6) and (8), \\nabla_x p_{0t} (x(t)) is available in closed-form. Is it always\nnecessary to have such a closed form expression in order to compute the objective (11), or\ncan it be estimated somehow without it? (I guess for a general SDE, there is typically no\nclosed-form \\nabla_x p_{0t} (x(t)) available)\n\n- Why do you think the FID values for the PC sampler with corrector are higher than without it\nfor VP SDE? (Table 1b)\n\n- In section 4.2: \"[...] PC samplers significantly outperform the corrector-only method, and can\nimprove over predictor-only approaches for most cases without extra computation.\" Why does\nfull PC sampler (with predictor and corrector) not incur extra computation in comparison to\nprediction-only approaches. Don't we need to evaluate the approximate score function\ns_\\theta(x, i) for each of the M steps in the corrector sampler?\n\n- In section 4.3: \"[...] deterministic process whose trajectories induce the same evolution\nof densities\". Does it mean that ODE (12) and reverse SDE (10) map the same noise\ndistribution p(x(T)) to the same distribution in the data space? If so, are there any\nadvantages of using a reverse SDE instead of equivalent ODE if the latter is easier to solve\nnumerically and admits an exact computation of likelihoods?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "qL0RSCfEg8t", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper generalizes a family of score-based generative models that rely on sequences of noise scalings of the data and extends them to the continuous domain, which leads to an SDE-based framework. By using score-matching, the forward SDE, which transforms data into a tractable noise distribution, can be reversed and thus used as a generative model. This is then improved by employing a two-phase algorithm with a prediction step, followed by a tunable number of correction steps. Further, reformulating the problem as a neural ODE allows for exact likelihood computations and reduces the number of required function evaluations. The framework enables unconditional, as well as conditional samples.\n\nI find the paper to be very well written and straightforward. As someone who does not have neural SDEs or Langevin Samplers as a core competence, I was able to follow all of the writeup, which is remarkable. I think the framework is nice and there is substantial novel innovation to justify accepting this paper. The experiments are convincing.\n\nA few questions and remarks:\n- You claim that you unify current methods into a common framework. While I see that you attempt to do this (i.e. putting the algorithms side-by-side, etc.), but in essence, you still handle VE and VP SDEs separately throughout. My suggestion would be to either really try to unify them into a single formulation, or alternatively, tone down the claims of unification, maybe just say that you show commonalities.\n- In Figure 2, you claim that the results are best when computation is \"split\" between the predictor and corrector. However, this is a very imprecise statement. An equal split would mean just 1 step of corrector, but I don't see clear evidence that that's best. Do you have numerical evidence that a 1-to-1 split is best, or what do you mean by \"split\"? Otherwise, you could just say that M is a tunable hyperparameter.\n- Also in Figure 2, it seems that there is a clear shift at some point where the samples go from low to high quality. Do you have any numerical indication (without looking at a test set, FID, etc.) of how a practicioner could notice that running for more steps would or wouldn't help?\n- In Table 1, what is the last row? I guess it's just employing the corrector, but maybe a label for the row would be nice.\n- Also in Table 1, it looks like the corrector helps for VE SDEs, but hurts for VP SDEs. Do you have an explanation for this? Maybe it's somewhere in the text, but if it is, I've missed it, so maybe point me to it.\n- Given that you can compute exact likelihoods, is it possible to compute the exact NLL for any real dataset, like a test dataset?", "extra_scores": {}, "extra_text": {}}, {"review_id": "7j3xdLuSZN", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary and contributions\nThis paper proposes a generalized framework for score-based generative modeling (SBGM). The proposed method subsumes previous SBGM techniques of score matching with Langevin dynamics (SMLD aka NCSN) and denoising diffusion probabilistic modeling (DDPM) and shows how they correspond to different discretizations of Stochastic Differential Equations (SDEs). The  continuous-time SDE generalizes the idea of a finite number of perturbation kernels used by previous methods to a continuum of them. The authors propose a forward SDE that transforms the data distribution into a known noise distribution and the corresponding reverse-time SDE that converts samples from this noise distribution to the data distribution. A predictor-corrector sampling framework is studied that leads to improved performance of both NCSN and DDPM frameworks. The paper also shows the equivalence of the proposed SDE to Neural ODEs which allows exact computation of the log-likelihood using the continuous change of variables formula. Quantitative experiments on the CIFAR10 dataset show that the proposed framework leads to significant improvements over previous SBGMs. Qualitative results on the CelebA-HQ dataset demonstrate the ability of the method to scale to high resolution images.\n\n#### Strengths\nThis paper makes significant technical and empirical contributions to the emerging area of score-based generative models. The generalized SDE framework subsumes recent works in this area and is also connected to Neural ODEs, enjoying exact likelihood calculation, which may be relevant to the normalizing flows and generative modeling community is general. The empirical evaluation is particularly well-done. It bridges the gap between the performance of NCSN and DDPM models leading to state-of-the-art performance. The authors also demonstrate the ability of the method to generate high quality images of human faces when trained on CelebA-HQ dataset. Preliminary experiments on class conditional generation, imputation, and image colorization demonstrate the wide applicability of the proposed method.\n\n#### Weaknesses\nThe paper does not suffer from any obvious weaknesses. The quantitive experiments could be strengthened by the addition of results on another dataset but the empirical evaluation is sufficient in its current state.\n\n#### Additional feedback\nQuestions:\n- In equation 11, how is the weighting function $\\lambda$ chosen?\n- In equation 11, apart from being able to sample from the transition kernel, it should also have a closed-form density for the evaluation of the score. Is my understanding correct?\n- In equation 21, should there be a discretization step-size corresponding to $\\Delta t$?\n- In table 1 (a), why does SMLD with corrector only perform so poorly? As far as I understand it is equivalent to NCSN. Can the authors clarify if I misunderstood something?\n\n-------\n\nPost Rebuttal: I thank the authors for clarifying on my questions and updating the manuscript. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "7-AEi3izVVK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper presents a generative model based on stochastic differential equations (SDEs), which generalizes two other score-based generative models score matching with Langevin dynamics (SMLD) and denoising diffusion probabilistic modeling (DDPM). The recipe to sample from the data distribution is based on (i) the observation that both SMLD and DDPM can be formulated as the discretization of an SDE, (ii) the finding from Anderson (1982) about the reverse of an Ito process, (iii) a score model. A novel aspect of the presented technique is the use of the score model as \"predictor\", which gives the initial sample from the MCMC sampler that serves as \"corrector\". Finally, the Ito process induced by the reverse SDE is formulated in a deterministic manner, leading to a neural-ODE based generative model.\n\nPros:\n- The authors did a good job at showing connections between the previous score-based generative models and their model. I believe on its own this is a nice contribution.\n- The method is thoroughly analyzed. I went through the derivations and didn't find any errors.\n- Experiments show that combining the predictor and corrector routines leads to better performance, a nice validation of the theoretical claims.\n- As promised, the model achieves SOTA on several tasks.\n\nCons:\n- I'm having difficulty seeing the transformation of the reverse SDE into an ODE (from eq.10 to eq.12). Is it as simple as multiplying the second term with 1/2 and discarding the Brownian motion? Also, eq.12 is a simple ODE system, which has nothing to with a process as far as I understand. Maybe more explanation or pointers in Maoutsa et al., 2020 would be nice.\n- The paper lacks the discussion on the benefits/downsides of different SDE solvers, discretization time steps, etc.\n- As such, the paper lacks a \"toy example\" experiment, for example, on a simple 2D dataset like half-moons. A visual demonstration of the SDEs and probability flow (maybe corresponding vector fields and Brownian motion over time) would be interesting.\n\nAdditional comments:\n- A typo (intead) right below eq.9\n- Best performing rows can be bold in Table 1.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that this is a well-written and interesting paper that will be of interest to the ICLR and broader ML community.
Decision: Accept (Oral)