Maximum Likelihood Training of Score-Based Diffusion Models
Authors: Yang Song
Conor Durkan
Iain Murray
Stefano Ermon
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: generative models
density estimation
score matching
score-based generative models
diffusion models
stochastic differential equations
normalizing flows
neural ODEs
likelihood
continuous normalizing flows

Abstract: Score-based diffusion models synthesize samples by reversing a stochastic process that diffuses data to noise, and are trained by minimizing a weighted combination of score matching losses. The log-likelihood of score-based diffusion models can be tractably computed through a connection to continuous normalizing flows, but log-likelihood is not directly optimized by the weighted combination of score matching losses. We show that for a specific weighting scheme, the objective upper bounds the negative log-likelihood, thus enabling approximate maximum likelihood training of score-based diffusion models. We empirically observe that maximum likelihood training consistently improves the likelihood of score-based diffusion models across multiple datasets, stochastic processes, and model architectures. Our best models achieve negative log-likelihoods of 2.83 and 3.76 bits/dim on CIFAR-10 and ImageNet $32\times 32$ without any data augmentation, on a par with state-of-the-art autoregressive models on these tasks. 

Maximum Likelihood Training of Score-Based Diffusion Models
Yang Song*
Conor Durkan" School of Informatics University of Edinburgh conor. durkan@ed. ac uk
Computer Science Department
Stanford University yangsongêcs. stanford... edu
lain Murray School Informatics University of Edinburgh i murray@ed. ac. uk
Stefano Ermon
Computer Science Department
Stanford University ermonêcs sttesssttafooroo edu
Abstract
Score-based diffusion models synthesize samples by reversing a stochastic process that diffuses data to noise, and are trained by minimizing weighted combination of score matching losses. The log-likelihood of score-based diffusion models can be tractably computed through connection continuous normalizing flows, but log- ligliiol is not directly optimized by the weighted combination of score matching losses. We show that for specific weighting scheme, the objective upper bounds the negative log-likell leelikkllihodd, thus enabling approximate maximum likelihood training of score-based diffusion models. We empirically observe that maximum likelihood training consistently improves the likelihood of score-based diffusion models across multiple datasets, stochastic processes, and model architectures. Our best models achieve negative log- li-liikiiihoods of 2.83 and 3.76 bits/dim on CIFAR-10 and ImageNet 32 without any data augmentation, on par with state-of-ther afrt autoregressive models on these tasks.
Introduction
Score-based generative models [44, 45, 48] and diffusion probabilistic models [43, 19] have recently achieved state-of-the- sample quality in a number of tasks, including image generation [48, 11], audio synthesis [5, 27, 37], and shape generation [3]. Both families of models perturb data with sequence of noise distributions, and generate samples by learning to reverse this path from noise to data. Through stochastic calculus, these approaches can be unified into single framework [48] which we refer to as score-based diffusion models in this paper.
The framework of score- score-based diffusion models [48] involves gradually diffusing the data distribution towards given noise distribution using stochastic differential equation (SDE), and learning the time reversal of this SDE for sample generation. Crucially, the reverse-time SDE has closed-form expression which depends solely on the time-dependent gradient field (a.k.a,, score) the perturbed data distribution. This gradient field can be efficiently estimated by training neural network (called a score-bas based model [44, 45]) with a weighted combination of score matching losses [23, 56, 46) as the objective. A key advantage of score-based diffusion models is that they can be transformed into continuous normalizing flows (CNFs) [6, 15], thus allowing tractable likelihood computation with numerical ODE solvers.
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Compared to vanilla CNFs, ccore-based diffusion models much more efficient train. This is because the maximum likelihood objective for training CNFs requires running an expensive ODE solver for every optimization slep, while the weighted combination of score matching losses for training score-based models does not. However, unlike maximum likelihood training, minimizing combination of score matching losses does not necessarily lead to better likelihood values. Since better likelihoods are useful for applications including compression [21, 20, 511, semi-supervised generative models, we seek a training objective for score-based diffusion models that is as efficient as score matching but also promotes higher likelihoods.
learning [10], adversarial purification [47], and comparing against likelihood-
We show that such an objective can be readily obtained through slight modification of the weighted combination of score matching losses. Our theory reveals that with specific choice of weighting, which we term the likelihood weighting. the combination of score matching losses actually upper bounds the negative log-likelihood. We further prove that this upper bound becomes tight when our score-based model corresponds to the true time-dependent gradient field of a certain reverse-time SDE. Using likelihood weighting increases the variance of our objective, which we counteract by introducing a variance reduction technique based on importance sampling. Our bound is analogous to the classic evidence lower bound used for training latent-variable models in the variational autoencoding framework [26, 391, and can be viewed as a continuous titrrouuuss-teeeeee generalization of [43].
minimize the weighted combination of score matching losses for approximate maximum likelihood training of score-based diffusion models. Compared to weight- ings in previous work [48], we consistently improve likelihood values across multiple datasets, model architectures, and SDEs, with only slight degradation of Fréchet Inception distances [17]. Moreover, our upper bound on negative log-likelihoos allows training with variational dequantization [18], with which we reach negative log-likelihood of 2.83 bits/dim on CIFAR-10 10 [28] and 3.76 bits/dim on ImageNet 32 32 [55] with no data augmentation. Our models present the first instances of normalizing flows which achieve comparable likelihood to cutting- gutting-edge autoregressive models.
With likelihood weighting,
2 Score-based diffusion models
Score-based diffusion models are deep generative models that smoothly transform data to noise using a diffusion process, and synthesize samples by learning and simulating the time-reversal of this diffusion. The overall approach is illustrated in Fig. 1.
2.1 Diffusing data to noise with an SDE
Let p(x) denote the unknown distribution of dataset consisting of D-dimensional i.i.d. samples. Score-based diffusion models [48] employ stochastic differential equation (SDE) to diffuse p(x) towards a noise distribution. The SDEs are of the form
dx f(x, t) dt + g(t) dw,
where f(: :D R.P is the drift coefficient, g()) R is the diffusion coefficient, and w E R denotes a standard Wiener process (a.k.a., Brownian motion). Intuitively, we can interpret dw as infinitesimal Gaussian noise. The solution of an SDE is diffusion process (x(1))ee(or7)) where [o. 7] is fixed time horizon. We let (x()) denote the marginal distribution of x(t), and Pox (x' x) denote the transition distribution from x(0) to x(t). Note that by definition we always have Po p when using an SDE to perturb the data distribution.
The role of the SDE is to smooth the data distribution by adding noise, gradually removing structure until little of the original signal remains. In the framework of score-based diffusion models, we choose fxx t), gt)., and T such that the diffusion process (x(t)))(( approaches some analytically tractable prior distribution x (xx) at & T, meaning pr()) x (xx.. Three families of SDEs suitable for this task are outlined in [48], namely Variance Exploding (VE) SDEs, Variance Preserving (VP) SDEs, and subVP SDEs.
2.2 Generating samples with the reverse SDE
Sample generation in score-based diffusion models relies on time-reversal of the diffusion process. For well- drift and diffusion coefficients, the forward diffusion described in Eq. (1) has an

---

Data
Noise
dx = f(x, dt + g(t) dw
x(0)
Score function V log (x)
x(T)
dx
[x f -(1)²
de + g(t) dw
Samples
Prior
Figure 1: We can SDE to diffuse data simple noise distribution. This SDE can be reversed once we know the score the marginal distribution at each intermediate time step, Vx log pa (x).
associated reverse-time diffusion process [1, 16] given by the following SDE
dx
[x () Vx log px()) dt + g())
where W is now a standard Wiener process in the reverse-time direction. Here dt represents an infinitesimal negative time step, meaning that the above SDE must be solved from
to This reverse-time SDE results in exactly the same diffusion process (x(t))te(o,7) as Eq. (1), assuming it is initialized with x(T") generative models, and its functional form reveals the key target for learning: the dependent score function Vx log pa (x). Again, see Fig. for a helpful visualization of this two-part formulation.
(x). This result allows for the construction of diffusion-basee
In order to estimate V log (x) from given dataset, fit the parameters of neural network so( x, 1), termed a score-based model, such that 80 xx 1) s Vx log p. (x) for almost all x E IR and I e [0, T]. Unlike many likelihood-based generative models,
ccore-based model does not need to satisfy the integral constraints of a density function, and is therefore much easier to parameterize.
Good score-based models should keep the following least squares loss small
1 I
Jsm( 0; x(-) =
2 2. (A(t) V lo so(x, t) dt,
(3)
where x: [0, T] R is a positive weighting function. The integrand features the well-known
score matching [23] objective (x) Vx
log log)((x) sx t)ll We therefore refer to Eq. (3) as
weighted combination of score matching losses.
With score matching techniques [56, 46), we can compute Eq. (3) up to an additive constant and
minimize it for training score-based models. For example, we can use denoising score matching [56] to transform JsM (0; X()) into the following, which is equivalent up to a constant independent of f:
1 T
JDSM 0 x()) = 2 J
E Ep(x)po: (x (A(t) Vx log Por x) so(x', t) [)] dt.
(4)
Whenever the drift coefficient Se (x, 1) is linear in x (which is true for all SDEs in [48]), the transition density Poc(x' x) is tractable Gaussian distribution. We can form a Monte Carlo estimate of
both the time integral and expectation in JDSM 0; X(-)) with sample (t, x'),, where is uniformly drawn from [o, T].
p(x) is a sample from the dataset, and x' Por x). The gradient Xx log Por (x' x) can also be computed in closed form since Por x) is Gaussian.
After training score-based model sø(x, t) with JDSM
( ( X(-)), we can plug into the reverse-time SDE in Eq. (2). Samples are then generated by solving this reverse-time SDE with numerical SDE solvers, given an initial sample from 7(x ()) at T. Since the forward SDE Eq. (1) is designed such that pr(xx x) se (x)., the reverse-time SDE will closely trace the diffusion process given by Eq. (1) in the reverse time direction, and yield an approximate data sample / (as visualized in Fig. 1).
3 Likelihood of score-based diffusion models
The forward and backward diffusion processes in score-based diffusion models induce two probabilis-
tic models for which we can define likelihood. The first probabilistic model, denoted pe is given by the approximate reverse-time SDE constructed from our score-based model so [ l). In particular, suppose {*o (e(or)))))) is stochastic process given by
SDD (x),
dx = [x t) 2 s0(8, t)] dt + g(t) dw, xo(T) ~
(5)
3

---

We define p as the marginal distribution of x0) The probabilistic model po is jointly defined by the score-based model 80(x, 1), the prior т, plus the drift and diffusion coefficients of the forward SDE in Eq. (1). We can obtain a sample xo(0) ~ De by numerically solving the reverse-time SDE in Eq. (5) with an initial noise vector (2()) π.
The other probabilistic model, denoted Pa (x), is derived from the SDE's associated probability flow ODE [32, 48]. Every SDE has a corresponding probability flow ODE whose marginal distribution at each time matches that of the SDE, so that they share the same pO (x) for all time. In particular, the ODE corresponding to the SDE in Eq. (1) is given by
dx
1 -
[t -
lxg,
(6)
dt
Unlike the SDEs in Eq. (1) and Eq. (2), this ODE describes fully deterministic dynamics for the process. Notably, it still features the same time-dependent score function Vx log Px(x). By approximating this score function with our model s0(x, t), the probability flow ODE becomes
dx t(x -2 20 (tt)sso (x..
dt
In fact, this ODE is an instance of continuous normalizing flow (CNF) [15], and how the ODE dynamics transform volumes across time in exactly the same way as these traditional flow-based models [6]. Given prior distribution т T(x), and a trajectory function Na: [0, - R satisfying the ODE in Eq. (7), we define D as the marginal distribution of xo( when (T) ~ и.
quantify
Similarly to Pe SDF the model p8 DDD is jointly defined by the score- score-based model sx t). the prior æ, formula [6],
and the forward SDE in Eq. (1). Leveraging the instantaneous
caange-of-variabee
can evaluate log Pe
ODE (x) exactly with numerical ODE solvers. Since pg ODE is a CNF, we can generate ODDE by numerically solving the ODE in Eq. (7) with an initial value xT() ~ т.
a sample Xe (0) Pg
(x)(x) is tractable, training pg ODE with maximum likelihood will require Although computing log P (x)
calling an ODE solver for every optimization step [6, 15]. which can be prohibitively expensive for large-scale scale score-based models. Unlike 2 we cannot evaluate log pg (x) exactly for an arbitrary data point x. However, we have lower bound on log pe (x) which allows both efficient evaluation and optimization, as will be shown in Section 4.2.
(x)
4 Bounding the likelihood of score-based diffusion models
Many applications benefit from models which achieve high likelihood. One example is lossless compression, where log- eigllikeld directly corresponds to the minimum expected number of bits needed to encode a message. Indeed, popular likelihood-basee models such as variational autoen- coders and normalizing flows have already found success in image compression [51, 20, 21]. Despite some known drawbacks [50], likelihood is still one of the most popular metries for evaluating and comparing generative models.
Maximizing the likelihood of based diffusion models can be accomplished by either maximizing the likelihood of SDE or poDE. Although p&DE is continuous normalizing flow (CNF) and its log- likelihood is tractable, training with maximum likelihood is expensive. As mentioned already, requires solving an ODE at every optimization step in order to evaluate the log-like lide on a batch of training data. In contrast, training with the weighted combination of score matching losses much more efficient, yet in general it does not directly promote high likelihood of either DE or PODE.
In what follows, we show that with specific choice of the weighting function A(1), the combination of score matching losses TsM (0; X(-)) actually becomes an upper bound on DKL.P || PSDE), and can therefore serve efficient proxy for maximum likelihood training. In addition, we provide related lower bound on log p
SDE (x) that can be evaluated efficiently on any individual datapoint x.
4.1 Bounding the KL divergence with likelihood weighting
It is well-known that maximizing the log- likelihoodo of a probabilistic model is equivalent to minimiz- ing the KL divergence from the data distribution to the model distribution. We show in the following theorem that for the model p this KL divergence can be upper bounded by TsM (0: A(-) when using the weighting function X(t) g(t) where g(t) is the diffusion coefficient of SDE in Eq. (1).

---

Table 1: SDEs and their corresponding weightings for score matching losses.
SDE
Formula
A(t) in [48]
likelihood weighting
VE
dx o(t) dw
(t)
(
VP
dx = 1(())x + dB
1 e S (x) ds
8(1)
subVP
dx - (())(1 -e 2% dr)) dw
(1 e So day)
((0)(1 - e d()dd)
Theorem 1.
Let p(x) be the data distribution, x(x) be a known prior distribution, and Pe be
defined as in Section 3. Suppose (x(O)) is stochastic process defined by the SDE in Eg. (1) with x(0) P. where the marginal distribution of x(t) is denoted as Pt. Under some regularity conditions detailed in Appendix A, we have
DKL(P
Po P0 < Is g())) + (KL || a).
(8)
Sketch of proof. Let l and denote the path measures of SDEs in Eq. (1) and Eq. (5) respectively. Intuitively, μ is the joint distribution of the diffusion process (x(1))(=(o,rr given in Section 2.1, and v represents the joint distribution of the process (Xe(t)) defined Section 3. Since we can marginalize l and v to obtain distributions p and SDE the data processing inequality gives (p pe SDE) & (K D). From the chain rule for the KL divergence, we also have ( || v) = DII,(PT || 4) + (xp( (M(- x(T) z) v(- xx(T)) z)]], where the KL divergence in the final term can be computed by applying the Girsanov theorem [34] to Eq. (5) and the reverse-time SDE of Eq. (1).
When the prior distribution π is fixed, Theorem guarantees that optimizing the weighted combination of score matching losses JSM( ( g is equivalent minimizing an upper bound on the KL divergence from the data distribution p to the model distribution SDE Due to well-known equivalence between minimizing KI. divergence and maximizing likelihood, we have the following corollary.
Corollary 1. Consider the same conditions and notations in Theorem 1. When a is fixed prior distribution that does not depend on O, we have
- -Eptx)logg SDE (x)] & JSM JsM(0; (;)(())) + Ci = (DsM((0;9())) + Cz,
where C1 and C2 are constants independent 0. O.
In light of the result in Corollary 1, we henceforth term X(1) = g()² the likelihood weighting. The original weighting functions in [48] are inspired from earlier work such as [44, 45] and [19], which are motivated by balancing different score matching losses the combination, and justified by empirical performance. In contrast, likelihood weighting is motivated from maximizing the likelihood of a probabilistic model induced by the diffusion process, and derived by theoretical analysis. There are three types of SDEs considered in [48]: the Variance Exploding (VE) SDE, the Variance Preserving (VP) SDE, and the sub VP SDE. In Table 1, we summarize all these SDEs and contrast their original weighting functions with likelihood weighting. For VE SDE, our likelihood weighting incidentally coincides with the original weighting used in [48], whereas for VP and subVP SDEs they differ from another.
Theorem 1 leaves two questions unanswered. First, what are the conditions for the bound to be tight (become an equality)? Second, is there any connection between DiDDE and ODE under some conditions? We provide both answers in the following theorem.
Theorem 2. Suppose p(x) and q(x) have continuous second-orde order derivatives and finite second moments. Let {x(())eefo.T) be the difsusion process defined by the SDE in Eq. (1), We use pt and qe to denote the distributions of x(t) when x(0) ~ p and x(0) q. and assume they satisfy the same assumptions in Appendix A. Under the conditions qr π and s s t) = V log qe (x) for all t e [0, T7, we have the following equivalence in distributions
Po
SDE
Po
ODE
q.
(9)
Moreover, we have
DKL(D
|
p0 ) JsM g(-))) + DRL || .)
(10)
5

---

Sketch of proof. When so(x,t) matches Vx log qe (x), they both represent the time-dependent score
of the same stochastic process so we immediately have Pe SDE 9. According to the theory of probability flow ODEs, we also have pe ODE
poe To prove Eq. (10), we note that DKL ll
Pe ) = (plg)) DKL (prlloT) Sn at DKL (p: gt) dt (prl())))) di DKL(Pt) q:) dt.
We can now complete the proof by simplifying the integrand using the Fokker-Planck equation of p
and g: followed by integration by parts.
In practice, the conditions of Theorem 2 are hard to satisfy since our score-based model s0(x, t)
will not exactly match the score function Vx log (x(x) of some reverse-time diffusion process with the initial distribution 9r = æ. In other words, our score model may not be valid time-dependent score function of stochastic process with appropriate initial distribution. Therefore, although score matching with likelihood weighting performs approximate maximum likelihood training for P6 SDE we emphasize that it is not theoretically guaranteed to make the likelihood of po better. That said, poDE will closely match y if our score-based model well-approximates the true score such that se (x, t) & Vx log p(()) for all x and e T.. Moreover, we empirically observe in our experiments (see Table 2) that training with the likelihood weighting is actually able to consistently improve the likelihood of po ODE across multiple datasets, SDEs, and model architectures.
4.2 Bounding the log-likelihood on individual datapoints
The bound in Theorem is for the entire distributions of and pa SDE but we often seek bound the log-likeli ogglikkeihood for an individual data point x. In addition, JsM X()) in the bound is not directly computable due to the unknown quantity Vx log (x)), and can only be evaluated up to additive constant through JDSM 0; X(-)) (as we already discussed in Section 2.2). Therefore, the bound in Theorem is only suitable for training purposes. To address these issues, we provide the following bounds for individual data points.
Theorem 3. Let (xc(x' x) denote the transition distribution from (x(x) to Pr(x)) for the SDE in Eq. (1). With the same notations and conditions in Theorem 1, we have
SDE log Pe (x) = Lg (xx) C2SM(())
(x)
(11)
where L is defined as
- -Bpur (x)) (og m(x')] I,
(xxx [ so(',t)) + ((()) |0((x 1012 2Vx
-112-22x- -(xxxt))
and C LDSM(X) (x) is given by
2 I. Exoc(e((>)) (o())² [/00||10xxxx ()) Vx pos xx) de
- (xe) (oog((())]]])
T Jo Epoxtxrx00 [o(0) V log x) 20x f(xxx( .)] dt.
Sketch of proof. For any continuous data distribution p, we have - Exx pODE) + H(p), where H(p) denotes the differential entropy of p. The KI. term can be bounded according Theorem 1, while the differential entropy has an identity similar Theorem 2 (sce Theorem 4 in Appendix A). Combining the bound of DKL D)DE) and the identity of H(p), we that holds for all continuous distribution Removing the
p P (x)] = (K
obtain a bound on - Exp lo (
expectation over p both sides then gives us a bound on log (x)) for an individual datapoint x. We can simplify this bound to CSM (x) and CDSM(X (x) with similar techniques to [23] and [56]. O
We provide two equivalent bounds LSM and (x) The former bears resemblance to score
matching while the second resembles denoising score matching. Both admit efficient unbiased estimators when f(,()) linear, as the time integrals and expectations can be estimated by samples of the form (4,'), where /. is uniformly sampled over [0, T) and
LSM(x) and CDSM (x)
x' ~ Poe x). Since the transition distribution Par(x' x) is a tractable Gaussian when f(. (,, () is linear, we can easily sample from it as well as evaluating Vx log Por(x' x) for computing CDSM (x).

---

Moreover, the divergences Vx s0(x, and Vx f(x, 1) in (x) and (x) have efficient unbiased estimators via the
trick [42, 22).
We can view CDSM(x (x) continuous-time generalization of the evidence lower bound (ELBO) in diffusion probabilistic models [43, 19J. Our bounds Theorem 3 are not only useful for optimizing and estimating log 12 (x), but also for training the drift and diffusion coefficients f(x, t) and g(t) jointly with the score-based model se (x, t); we leave this avenue of research for future work. In addition, we can plug the bounds in Theorem 3 into any objective that involves minimizing -logpe SDF (x) to obtain an efficient surrogate. Section 5.2 provides an example, where we perform variational dequantization to further improve the likelihood of score-based diffusion models.
Similar to the observation in Section 4.1, (x) and CDSM(x are not guaranteed to upper bound (x). However, they should become approximate upper bounds when so (x, t) is trained ODE
- log pe ODE
sufficiently close the ground truth. In fact, we empirically observe that log Pg (x) $ Lg (x)(x)
CDSM (x) holds true for x sampled from the dataset in all experiments.
4.3 Numerical stability
So far we have assumed that the SDEs are defined in the time horizon [0, T] in all theoretical analysis. In practice, however, we often face numerical instabilities when - 0. To avoid them, we choose a small non-zero starting time >> O, and train/evaluate score-based diffusion models in the time horizon [c, 7] instead of [0, 7j. Since é is small, training score-based diffusion models with likelihood weighting still approximately maximizes their model likelihood. Yet at test time, the likelihood bound as computed in Theorem 3 is slightly biased, rendering the values not directly comparable to results reported in other works. We use Jensen's 's inequality to correct for this bias in our experiments, for which we provide a detailed explanation in Appendix B.
4.4 Related work
Our result in Theorem 2 can be viewed as a generalization of De Bruijin's 's identity ([49), Eq. 2.12) from its original differential form an integral form. De Bruijn's identity relates the rate of change of the Shannon entropy under an additive Gaussian noise channel to the Fisher information, a result which can be interpreted geometrically as relating the rate of change of the volume of distribution's 's typical set to its surface area. Ref. [2] (Lemma I) builds on this result and presents an integral and relative form of de Bruijn's 's identity which relates the KL divergence the integral of the relative Fisher information for distribution of interest and a reference standard normal. More generally, various identities and inequalities involving the (relative) Shannon entropy and (relative) Fisher information have found use in proofs of the central limit theorem [24]. Ref. [31] (Theorem 1) covers similar ground to the relative form of de Bruijn's 's identity, but is perhaps the first to consider its implications for learning in probabilistic models by framing the discussion in terms of the score matching objective ([23), Eq. 2).
Improving the likelihood of score-based diffusion models
Our theoretical analysis implies that training with the likelihood weighting should improve the likelihood of score-based diffusion models. To verify this empirically, we test likelihood weighting with different model architectures, SDEs, and datasets. We observe that switching to likelihood weighting increases the variance of the training objective and propose to counteract with importance sampling. We additionally combine our bound with variational dequantization [18] which narrows the gap between the likelihood of continuous and discrete probability models. All combined, we observe consistent improvement of likelihoods for both pg and PODE across all settings. We term the model pø ODE trained in this way ScoreFlow, and show that it achieves excellent likelihoods on CIFAR-10 10 [28] and ImageNet 32 x 32 [55], on a par with cutting-edge autoregressive models.
5.1 Variance reduction via importance sampling
As mentioned in Section 2.2, we typically use Monte Carlo sampling to approximate the time integral in JosM (0; X()) during training. In particular, we first uniformly sample a time step / ~ 2[[0 T). and then use the denoising score matching loss at t estimate for the whole time integral. This
7

---

VP SDE
subVP SDE
20.0
w/o importance sampling
w/o importance sampling
Loss scale)
importance sampling
importance sampling
10.0
200
400
600
800
200
400
600
800
Training iteration (k)
Training iteration (k)
Figure 2: Learning curves with the likelihood weighting on the CIFAR-10 dataset (smoothed with
exponential moving average). Importance sampling significantly reduces the loss variance.
Monte Carlo approximation is much faster than computing the time integral accurately, but introduces additional variance to the training loss.
We empirically observe that this Monte Carlo approximation suffers from a larger variance when using our likelihood weighting instead the original weightings in [48]. Leveraging importance sampling, we propose new Monte Carlo approximation that significantly reduces the variance of learning curves under likelihood weighting, as demonstrated in Fig. 2. In fact, with importance sampling, the loss variance (after convergence) decreases from 98.48 to 0.068 on CIFAR- 10, and decreases from 0.51 to 0.043 on ImageNet.
Let X(t) = axt)² denote the weightings in [48] (reproduced in Table 1), and recall that our likelihood weighting ((t) = g Since at))² empirically leads lower variance, can use proposal ()) from g(t)² to att)² with importance sampling, where Z is normalizing constant that ensures SP(() dt 1. Specifically, for any function h(t), we estimate the time integral So ((t)) dt with
distribution p(t) == /a1))² z to change the weighting in JDSM
J ((1)2((t)dr 2 S p(t)a(())((() di (Z)(1)2h(),
(12)
where i is a sample from p(t). When training score-baa based models with likelihood weighting. h(t) corresponds to the denoising score matching loss at time t.
Ref. [33] also observes that optimizing the ELBO for diffusion probabilistic models has large variance, and proposes to reduce it with importance sampling. They build their proposal distribution based on historical loss values stored thousands of discrete time steps. Despite this similarity, our method is casier to implement without needing to maintain history, can be used for evaluation, and is particularly suited to the continuous-timg setting.
5.2
Variational dequantization
Digital images are discrete data, and must be dequantized when training continuous density models like normalizing flows [12, 13] and score-based diffusion models. One popular approach this is uniform dequantization [53, 50], where we add small uniform noise over [0, 1) to images taking values uniformly dequantized data implicitly maximizes a lower bound on the log-like ligeliilldood of a certain discrete model Po(x), Due the gap between po (x(x) and (x(x), comparing the likelihood of continuous density models to models which fit discrete data directly, such as autoregressive models [55] or variational autoencoders, naturally puts the former
in (0, 1, 255). As shown in [50], training continuous model Pe(x)
disadvantage.
To minimize the gap between px(x) and Px(x), ref. [18] proposes variational dequantization, where separate normalizing flow model q x) is trained to produce the dequantization noise by optimizing the following objective
max E xxx Bu po (x + u) - log (u x)]
(13)
Plugging in the lower bound on log po (x) from Theorem 3, we can optimize Eq. (13) to improve the likelihood of score-based diffusion models.

---

Table 2: Negative log-likeling log-likelihood (bits/dim) and sample quality (FID scores) on CIFAR-10 10 and ImageNet 32 x 32. Abbreviations: "NLL" for "negative log-likelihood": "Uni deq." for "Uniform dequantization", "Var. deq." for "Variational dequantization", "LWW" for "likelihood weighting", and "IS" for "importance sampling". .iiing".nnn Bold indicates best result in the corresponding column. Shaded rows represent models trained with both likelihood weighting and importance sampling.
CIFAR-10
ImageNet 32 x 32
Model
SDE
Uni. deg.
Var. deg.
Uni. deq.
Var. dey.
FID
FID]
NLL,
Bound]
NLL1
Bound]
NLL
Bound
NLL.
Bound]
Buseline
VP
3,16
3.28
3,04
3.14
3.98
3,90
3.96
3.84
3.91
8.34
Baseline LW
VP
3.06
3.18
2.94
3.03
5.18
3,91
3,96
3.86
3.92
17.75
Baseline 1.W
VP
2.95
3.08
2.83
2.94
6.03
3.86
3.92
3.80
3.88
11.15
Deep
VP
3.13
3.25
3.01
3.10
3.09
3.89
3.95
3.84
3.90
8.40
Deep LW
VP
3.06
3.17
2.93
3.02
7.88
3.91
3.96
3.86
3.92
17.73
Doep+ LW+ IS
VP
2.93
3.06
2.80
2.92
5.34
3.85
3.92
3.79
3.88
11.20
Baseline
subVP
2.99
3.09
2.88
3.20
3.87
3.92
3.88
8.71
Baseline I.W
suhVP
2.97
3.07
2.86
2.96
7.33
3.87
3.92
12.99
Baseline LW +
subVP
2.94
3.05
2.84
2.94
5.58
3.84
3,91
3.79
3.87
10.57
Deep
subVP
2.96
3.06
2.85
2.95
2.86
3.86
3.91
3.81
3.87
8.87
Deep + LW
subVP
2.95
3.05
2.85
2.94
6.57
3.88
3.93
3.83
3.88
16.55
Deep LW IS
subVP
2.90
3.02
2.81
2.90
5,40
3.82
3,90
3.76
3.86
10.18
5.3 Experiments
We empirically test the performance of likelihood weighting, importance sampling and variational dequantization across multiple architectures of score-based models, SDEs, and datasets. In partieular, we consider DDPM++ ("Baseline" in Table 2) and DDPM++ (deep) ("Deep" in Table 2) models with VP and subVP SDEs [48] on CIFAR- 10 [28] and ImageNet 32 32 [55] datasets. We omit experiments on the VE SDE since (i) under this SDE our likelihood weighting is the same as the original weighting in [48]; (ii) we empirically observe that the best VE SDE model achieves around 3.4 bits/dim on CIFAR- 1-FAR-1000 10 in our experiments, which is significantly worse than other SDEs. For each experiment, we report - EFoog ODE [x)] ("Negative log-likelihood"
Table 2), and the upper bound |E[DDSM( [x]] on -Blog Pe (x)] ("Bound" in Table 2). In addition, we report FID scores [17] for samples from pa pa produced by solving the corresponding ODE with the Dormand Princee RK45 [14] solver. Unless otherwise noted, we apply horizontal flipping as data augmentation for training models on CIFAR-10, so as to match the settings in [48, 19]. Detailed description of all our experiments can be found in Appendices B and C.
We summarize all results in Table 2. Our key observations are follows:
1. Although Theorem 3 only guarantees E EDCS have SDF # ODE we still find that B[CDSM (x)] ("Bound" in Table 2) >/ -Ellog Pg ("NLL" in Table 2) all our settings.
(x)) > - E[llo SDE (xP)( and in general we ODF (x)]
2. When all conditions are fixed except for the weighting in the training objective, having a lower value of the bound for yoDE always leads to lower negative lig-likelihood for pe
ODE
3. With only likelihood weighting, we can uniformly improve the likelihood of p and the bound of pd SDE on CIFAR- 10 across model architectures and SDEs, but it not sufficient to guarantee likelihood improvement ImageNet 32 32.
4. By combining importance sampling and likelihood weighting, uniformly better likelihood for pg ODE and bounds for SDE across all model architectures, SDEs, and datasets, with only slight degradation of sample quality as measured by FID [17].
able to achieve
5. Variational dequantization uniformly improves both the bound for p and the negative log- likeliiel (NLL) of po ODE in all settings, regardless of likelihood weighting.
Our experiments confirm that with importance sampling, likelihood weighting is not only effective for maximizing the lower bound for the log-likelihood of p SDE but also improving the log- lieeiikd of Po ODE In agreement with [19, 33), we observe that models achieving better likelihood tend have worse FIDs. However, emphasize that this degradation of FID is small, and samples actually

---

have no obvious difference in visual quality (see Figs. 3 and 4). To trade likelihood for FID, can use weighting functions that interpolate between likelihood weighting and the original weighting functions [48]. Our FID scores are still much better than most other likelihood- based models.
We term pg QDEE a ScoreFlow when its correspond- ing score-based model so (x, t) is trained with likelihood weighting. importance sampling, and variational dequantization combined. It can be viewed as a continuous normalizing flow, but is parameterized by score-based model and trained in a more efficient way. With variational dequantization, show ScoreFlows obtain competitive negative log- aggliiellhoods (NLLs) of 2.83 bits/dim on CIFAR-10 and 3.76 bits/dim on ImageNet 32 x 32. Here the ScoreFlow on CIFAR-I0 trained without horizontal flipping (different from the setting in Table 2). As shown Table our results are on a par with the state- of-the-an art autoregressive models on these tasks, and outperform all existing normalizing flow
Table 3: NLLS on CIFAR-IO and ImageNet 32x32.
Model
CIFAR-I0
ImageNet
FFJORD [15]
3,40
Flow++ [18]
3.08
3.86
Gated PixelCNN [35]
3.03
3.83
VFlow [4]
2.98
3.83
PixeICNN+ [40]
2.92
NVAE [54]
2.91
3.92
Image Transformer [36]
2.90
3.77
Very Deep VAE [8]
2.87
3.80
PixcISNAIL [7]
2.85
3,80
8-VAE [38]
2.83
3.77
Sparse Transformer [9]
2.80
ScoreHlow (Ours)
2.83
3.76
models. The likelihood for CIFAR- 10 can be significantly improved by incorporating advanced data augmentation, as demonstrated in [25, 41]. While we do not compare against these approaches, we believe that incorporating the same data augmentation techniques could also improve the likelihood
of ScoreFlows.
Conclusion
We propose an efficient training objective for approximate maximum likelihood training of score- based diffusion models. Our theoretical analysis shows that the weighted combination of score matching losses upper bounds the negative ligeliieldood when using a particular weighting function which we term the likelihood weighting. By minimizing this upper bound, we consistently improve the likelihood of score-based diffusion models across multiple model architectures, SDEs, and datasets. When combined with variational dequantization, we achieve competitive likelihoods on CIFAR-10 and ImageNet 32 32, matching the performance of best-in-class autoregressive models.
Our upper bound analogous the evidence lower bound commonly used for training variational autoencoders. Aside from promoting higher likelihood, the bound can be combined with other objectives that depend on the negative log-likelhhoo, and also enables joint training the forward and backward SDEs, which we leave as a future research direction. Our results suggest that score- based diffusion models are competitive alternatives continuous normalizing flows which enjoy the same tractable likelihood computation but with more efficient maximum likelihood training.
Limitations and broader impact Despite promising experimental results, we would like em- phasize that there is no theoretical guarantee that improving the SDE likelihood will improve the ODE likelihood, and this is explicitly a limitation of our work. Score-based diffusion models also suffer from slow sampling. In our experiments, the ODE solver typically need around 550 and 450 evaluations of the score-based model for generation and likelihood computation on CIFAR-10 10 and ImageNet respectively, which is considerably slower than alternative generative models like VAES and GANs. In addition, the current formulation of score-based diffusion models only supports continuous data, and cannot be naturally adapted to discrete data without resorting to dequantization. Similarly to other deep generative models, score-based diffusion models can potentially be used to generate harmful media contents such as 'deepfakes' and might reflect and amplify undesirable social bias that could exist in the training dataset.
Author Contributions
Yang Song wrote the code, ran the experiments, proposed and proved Theorems 1 and 3, and wrote most of the paper. Conor Durkan proposed and proved first version of Theorem 2, and wrote the paper. lain Murray and Stefano Ermon co-advised the project and provided helpful edits to the draft.
10

---

Acknowledgments and Disclosure of Funding
The authors would like to thank Sam Power, George Papamakarios, Adji Dieng for helpful feedback, and Duoduo for providing her photos in Fig. 1. This research supported by NSF (#1651565 #1522054, #1733686), ONR
AFOSR (FA955019100244) ARO (W91INF-21-1-- 0125), Sloan Fellowship. and Google TPU Research Cloud. This research was also supported by the EPSRC Centre for Doctoral Training in Data Science, funded by the UK Engineering and Physical Sciences Research Council (grant EP/LO16427/1), and the University of Edinburgh. Yang Song was supported by the Apple PhD Fellowship in AI/ML.
References
[1] B. D. Anderson. Reverse-Time Diffusion Equation Models. Stochastic Processes and their Applications, (2(3):3133226 1982.
[2] A. R. Barron. Entropy and the Central Limit Theorem. Annals of Probability, 14(1):336-345 1986.
[3] R. Cai, G. Yang, H. Averbuch-Elor, Z. Hao, S. Belongie, N. Snavely, and B. Hariharan. Learning Gradient Fields for Shape Generation. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[4] J. Chen, C. Lu, B. Chenli, J. Zhu, and T. Tian. Vflow: More expressive generative flows with variational data augmentation. In International Conference on Machine Learning, pages 1660-1669, PMLR, 2020.
[5] N. Chen, Y. Zhang, H. Zen, R. J. Weiss, M. Norouzi, and W. Chan. WaveGrad: Estimating Gradients for Waveform Generation. arXiv preprint wrXiv:2009.00715 2020.
[6] R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural Ordinary Differential Equations. In Advances in neural information processing systems, pages 6571 6583, 2018.
[7] X. Chen, N. Mishra, M. Rohaninejad, and P. Abbeel. Pixelsnail: An improved autoregressive generative model. In International Conference on Machine Learning, pages 864-872. PMLR, 2018.
[8] R. Child. Very deep VAEs generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, 2021.
[9] R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 011.1944...0000 2019.
[10] Z. Dai, Z. Yang. F. Yang, W. W. Cohen, and R. Salakhutdinov. Good semi-supervised learning that requires a bad GAN. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 5513-6523, 2017.
[11] P. Dhariwal and A. Nichol. Diffusion models beat GANs on image synthesis. arXiv preprint arXiv:2105.05253 2021.
[12] L. Dinh, D. Krueger, and Y. Bengio. NICE: Non-Linear Independent Components Estimation. arXiv preprint
2014.
[13] L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density estimation using real NVP. In 5th Interna- tional Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[14] R. Dormand and P. Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19-22) 1980.
[15] W. Grathwohl, R. T. Q. Chen, J. Bettencourt, I. Sutskever, and D. Duvenaud. FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models. In International Conference on Learning Representations, 2019.
[16] U. G. Haussmann and E. Pardoux. Time reversal of diffusions. The Annals of Probability, pages 1188-1205, 1986.
[17] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by two time-scale update rule converge to local Nash equilibrium. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. Vishwanathan, and R. Garnett, editors, Advances
11

---

in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6626-6637, 2017.
[18] Ho, X. Chen, A. Srinivas, Y. Duan, and P. Abbeel. Flow++: Improving flow-based generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pages 2722-2730. PMLR, 2019.
[19] Ho, A. Jain, and P. Abbeel. Denoising Diffusion Probabilistic Models. Advances in Neural Information Processing Systems, 33, 2020.
[20] J. Ho, E. Lohn, and P. Abbeel. Compression with flows via local bits-back coding. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages
2019.
[21] E. Hoogeboom, J. W. T. Peters, R. van den Berg, and M. Welling, Integer discrete flows and lossless compression. H. M. Wallach, H. Larochelle, A. Beygelzimer, F, Alché-Bucc E. B. Fox, and R. Garnett, editors, Advances in Neurul Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 12134-12144, 2019.
[22] M. F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics -Smullatio and Computation, 18(3):1059 1076, 1989.
[23] A. Hyvärinen. Estimation of Non-Normalizee Statistical Models by Score Matching. Journal of Machine Learning Research, 6(Apr):695-70% 2005.
[24] O. Johnson and A. Barron. Fisher Information incqualities and the Central Limit Theorem. Probability Theory and Related Fields, 129(3):391-4099 2004.
[25] H. Jun, R. Child, M. Chen, J. Schulman, A. Ramesh, A. Radford, and I. Sutskever, Distribution augmentation for generative modeling. In International Conference on Machine Learning. pages 5006-5019, PMLR, 2020.
[26] D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations, 2014.
[27] Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro. DiffWave: A Versatile Diffusion Model for Audio Synthesis. arXiv preprint arXiv:2009.09761, 2020.
[28] A. Krizhevsky, V. Nair, and G. Hinton. The CIFAR-IO Dataset. online: http://www. cs. toronto. edu/kriz/cifar. html, 55, 2014.
[29] C. Léonard. Some properties of path measures. In Séminaire de Probabilités XLVI, pages 207-230. Springer, 2014.
[30] X. Li, T.-K. L. Wong, R. T. Q. Chen, and Duvenaud. Scalable Gradients for Stochastic Differential Equations. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, 2020.
[31] S. Lyu. Interpretation and Generalization of Score Matching. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, pages 359-366, 2009.
[32] D. Maoutsa, S. Reich, and M. Opper. Interacting particle solutions of Fokker-Planck equations through grradiet-log-densite estimation. arXiv preprini arXiv:2006. 200000000000702. 2020.
[33] A. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.09672 2021.
[34] B. Oksendal. Stochastic differential equations: an introduction with applications. Springer Science & Business Media, 2013.
[35] A. v. d. Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, and K. Kavukcuoglu. Conditional image generation with pixelenn decoders. Proceedings of the 30th International Conference Neural Information Processing Systems, pages 4797-4805, 2016.
[36] N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, N. Shazeer, A. Ku, and D. Tran. Image transformer. In International Conference on Machine Learning, pages 4055 4064. PMLR,
2018.
12

---

[37] V. Popov, I. Vovk, V. Gogoryan, T. Sadekova, and M. Kudinov. Grad-TTS: A diffusion probabilistic model for text-to-speech. arXiv preprint arXiv:2 2021.
[38] A. Razavi, A. van den Oord, B. Poole, and O. Vinyals. Preventing posterior collapse with delta-vaes. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[39] J. Rezende, S. Mohamed, and D. Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Proceedings of the 31 International Conference Machine Learning, volume 32, 2014.
[40] T. Salimans, A. Karpathy, X. Chen, and D. P. Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. OpenReview.net, 2017.
[41] S. Sinha and A. B. Dieng. Consistency regularization for variational auto-encoders. arXiv preprint arXiv:2105. 14859, 2021.
[42] Skilling. The eigenvalues of mega-dimensionn matrices. In Maximum Entropy and Bayesian Methods, pages 455-466. Springer, 1989.
[43] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep Unsupervised Learning Using Nonequilibrium Thermodynamicss In International Conference on Machine Learning. pages 2256-2265. 2015.
[44] Y. Song and S. Ermon. Generative Modeling by Estimating Gradients of the Data Distribution.
In Advances in Neural Information Processing Systems, pages 1 11918-11930, 2019.
[45] Y. Song and S. Ermon. Improved Techniques for Training Score- Based Generative Models. Advances in Neural Information Processing Systems, 33, 2020.
[46] Y. Song. S. Garg. J. Shi, and S. Ermon. Sliced Score Matching: A Scalable Approach to Density and Score Estimation. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, page 204, 2019.
[47] Y. Song, T. Kim, S. Nowozin, S. Ermon, and N. Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. In International Conference Learning Representations, 2018.
[48] Y. Song, Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-Based Generative Modeling Through Stochastic Differential Equations. In International Conference on Learning Representations, 2021.
[49] A. Stam. Some Inequalities Satisfied by the Quantities of Information of Fisher and Shannon. Information and Control, 2(2): June 1959.
[50] L. Theis, A. van den Oord, and M. Bethge. A note on the evaluation of generative models. In
Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[51] J. Townsend, T. Bird, and D. Barber. Practical lossless compression with latent variables using bits back coding. In International Conference on Learning Representations, 2019.
[52] B. Tzen and M. Raginsky. Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit. 1905. 09883, 2019.
[53] B. Uria, I. Murray, and H. Larochelle. Rnade: the realevalued neural autoregressive density- estimator. In Proceedings of the 26th International Conference on Neural Information Process- ing Systems-Volann Volume 2, pages 2175-2183, 2013.
[54] A. Vahdat and J. Kautz. NVAE: A deep hierarchical variational autoencoder. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[55] A. Van den Oord, N. Kalchbrenner, and K. Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pages 1747-1756. PMLR, 2016.
[56] Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7):1661-1674, 2011.
13

---

[57] K. Yang, J. Yau, L. Fei-Fei, J. Deng, and O. Russakovsky. A study of face obfuscation in imagenet. arXiv preprint arXiv:2103.06191, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] The limitations of our theory are discussed inline in Section 4.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Discussed in conclusion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] The full set of assumptions are in Appendix A.
(b) Did you include complete proofs of all theoretical results? [Yes] The full proofs are all provided in Appendix A.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] Code is released at https://ittub..com/yan -ao.l/aaansooooorrreee..oo f_ow..
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] All experimental details are in Appendix C.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] Training our models is expensive and we do not have enough resource/tmee for multiple repetitions of our experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Compute and resource types are given Appendix C.
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cited the creators of CIFAR-10 10 and wown-sampled ImageNet.
(b) Did you mention the license of the assets? [No] Licenses are standard and can be found
online.
(c) Did you include any new assets either in the supplemental material as URL? [Yes] Code and checkpoint are released at https://github.com wttpl/uuw.ooooogooooaaaaeaa wownungson//oorree.o.oo.... flow.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] All datasets used in our work are publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] We mentioned privacy issues of the ImageNet dataset inline Appendix C.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "z2i6lMX-A0S", "reviewer": "Reviewer_CUCw", "summary": "The authors show that for a specific weighting of score matching losses at various noise levels, the objective upper bounds the negative log-likelihood. They further bound the log-likelihood of individual datapoints to develop a continuous-time generalization of the ELBO.\n\nUsing the proposed objective the authors optimize score-based diffusion models towards likelihood.\nThe authors further make use of 1) importance sampling of time steps for reduced variance, 2) variational dequantization for a tighter bound on the likelihood and 3) the use of the equivalent ODE during evaluation, which allows exact likelihood computation (apart from the dequantization) for an approximation to the SDE model.\nAs a result of this, state-of-the-art likelihood performance is presented on CIFAR-10 and ImageNet 32x32, beating even the best autoregressive models.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Explicitly outlining the connections to maximum likelihood estimation (MLE) for continuous-time score-based diffusion models is novel, concurrent with [1], which was an important missing piece. It is nice to see a continuous-time extension of the ELBO. One thing I found missing was a connection to the discrete-time case as in DDPM [2] where a link between the ELBO and the score-matching loss is also established. I feel a discussion on this would clarify connections and thus strengthen the paper.\n\nAs for the results, the ablation study in Table 2 is informative, showing that \n1. likelihood weighting tends to improve likelihood, empirically backing up the choice, while degrading the FID score.\n1. the performance of the equivalent ODE tends to perform better than the SDE despite this not being guaranteed.\n1. the proposed importance sampling improves performance, along with the use of variational dequantization and depth.\n\nFurthermore, the best numbers reported, as summarized in Table 3, are quite impressive, setting new records on two highly benchmarked datasets, which have been dominated by autoregressive models for years. This demonstrates clearly that score-based diffusion models have great potential, not just for image synthesis, but also for tasks relying on the likelihood (e.g. compression).\n\nAs for the clarity: I believe parts of Section 4 could be compressed and simplified to improve clarity. Please correct me if I'm wrong. \nThm. 1 seems superfluous given Thm. 2 which turns the inequality into an equality (obtained through Eq. 25-27 with $q:=p_{\\theta}^{\\text{SDE}}$). Also, for Thm. 2, it seems $q_t$ is defined as the process solving Eq. (5) and thus $q := q_0 := p_{\\theta}^{\\text{SDE}}$ trivially, so it seems $q$ could be dropped?  \nFinally, would not the requirement for $p_{\\theta}^{\\text{ODE}} = p_{\\theta}^{\\text{SDE}}$ be that $s_{\\theta}(x,t) = \\nabla_x \\log p_t(x)$ instead of $\\nabla_x \\log q_t(x)$?\n\nMinor fix on Line 127: $s_{\\theta}(x) \\rightarrow s_{\\theta}(x, t)$.\n\n[1] A Variational Perspective on Diffusion-Based Generative Models and Score Matching (Huang et al., 2021)  \n[2] Denoising Diffusion Probabilistic Models (Ho et al., 2020)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations of the work are discussed in Sec. 4 on Line 202-210 and Line 244-247.\nPotential societal impacts of the work are not discussed. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "xsBXtnMDft", "reviewer": "Reviewer_Vbrn", "summary": "This paper presents likelihood weighting to improve the log-likelihood performance of the score-based model. It is widely known that the score-based model is an excellent generative model to sample an instance from the high-dimensional data distribution. However, its encoding capability has remained question. This paper argues that the presented likelihood weighting can be universally applicable to the score-based models to improve such encoding capability without hurting the generation performance. There were series of theoretic derivations to induce the likelihood weighting, and a benchmark experiment was conducted. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1)\nThe main question is improving negative log-likelihood without damaging FID. Table 2 shows that \"DEEP+LW+IS\" achieves the best NLL with mediocre FID because the best NLL case results in 5.70 FID when the best FID is 2.86. When we observe Table 2, FID score almost doubles whenever we introduce likelihood weighting. There is no discussion about FID in Table 2 except \"with only slight degradation of sample quality as measured by FID\" in line 299.\n\nThen, these questions arise.\n- Why do you need additional importance sampling to limit the damage on FID? What is the theory behind of this observation?\n- Still, the degradation of the FID is notieable. Can you verify this sample quality in other metrics, i.e. IS?\n- Is there any trade-off coefficient that you can introduce to barter NLL with FID?\n\n\n2)\nWhy do you report only NLL in Table 3 without FID? To empirically verify your claim, don't you need to observe the two aspects at the same time? Or, is it because your score model has definately better FID over the flow, transformer, and other models?\n\n\n3)\nLikelihood weighting seems to increase the loss variance as you mentioned in line 252. It seems that importance weighting is introduced to limit this side effect in an adhoc manner. To reason thie phenomena further, there should be inequality checking of loss variances between the case of using $\\lambda$ as before and $\\lambda=g(t)^2$. \n\n4)\nVE-SDE uses the same weighting of likelihood weighting and the weighting in the prior work. Then, people may just use the VE-SDE variation of score-based models to optimize the NLL. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nThere is no discussion at all. I understand that authors may not discuss the societal impact, but they can surely include the limitations that they expect from their models.\n\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "9hazwWz2OM9", "reviewer": "Reviewer_gpyk", "summary": "This paper shows that, with a specific weighting of the score matching loss, one can upper bound the negative log-likelihood of the density induced by the reverse SDE parameterized by the score function. The authors then propose to use importance sampling to minimize this weighted score matching loss and obtain state-of-the-art likelihood estimates on CIFAR10 and ImageNet 32x32. ", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is a strong submission with some theory and impressive empirical results, which furthermore bridges the long-standing gap between autoregressive models and non-autoregressive likelihood-based models. \n\nIn general, the paper is quite well written and quite easy to follow. Below I have some nits, which I hope the authors could address. \n\n1. Thm 1 & 3: The main proof technique can be summarized by an application of data processing inequality (which upper bound marginal KL divergence with the joint, functional KL divergence) and Girsanov's theorem (for computing the functional KL). My main critique for Theorem 1 is that it does not hold for empirical distribution, as the differential entropy will be unbounded. Then the authors derive Theorem 3 to bound the likelihood of individual datum. However, I do not quite follow the step where the expectation E_p(x) is removed. Do you mean by taking p(x) to approximate a point mass centered at x? If so please make the statement rigorous. Also, I'm not sure why in the LHS of the equation after line 649, there is the conditional expectation p0t and x', is this a typo?\n\n2. Thm 2: in the remark after the theorem, it's said that the densities induced by the SDE and ODE will be close if s\\approx score of the data. There seems to be some logical leap here, since this will only hold true provided that the score of the model (\\nabla log q, not s) approximates the score of the data (\\nabla log q). It is still unclear how the two densities differ from or relate to each other, quantitatively and non-asymptotically (asymptotically it's clear when the score function s is perfect, then everything is fine), and why the ODE's likelihood is improved. \n\n3. Bias in importance sampling: g^2/sigma^2 is not integrable as it blows up too fast as t->0, which means defining p(t) to be proportional to it will not result in a valid probability density. In Appendix B, it's said that the integration starts from some t_0, which should resolve the integrability issue, but this may result in bias in the ELBO estimation for both training and testing. Does the reported ELBO have this bias? I imagine this bias is negligible if t_0 is sufficiently small, has it been tested? Also, I'd suggest somehow include this in the main text so that people who want to build on the framework of this paper can be made aware of it. \n\n4. VE SDE: Line 284, it is said that the weighting for VE SDE is the same as in [1]. But if I understand correctly, in [1], lambda is taken to be 1/E[nabla log p_0t], which will correspond to the \"conditional\" variance, not the diffusion coefficient sigma^2(t), aka likelihood weighting.  \n5. SOTA claim: It'd be better to include some other SOTA results with data augmentation in the table, such as [1,2]. It wouldn't harm the contribution of the paper as long as the SOTA claim is within the category of standard training with random horizontal flips.\n\n[1] Score-Based Generative Modeling Through Stochastic Differential Equations.\n[2] Distribution Augmentation for Generative Modeling\n[3] Consistency Regularization for Variational Auto-Encoders", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As pointed out in the main review above. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "8aN0mRUcFsl", "reviewer": "Reviewer_3AeC", "summary": "The paper studies score-based denoising-diffusion generative models and makes one major contribution. It shows how the training objective, a mixture or score-matching losses defined along different times of the diffusion, can be adapted to enable maximum likelihood training (using a lower bound objective). This is rigorously derived and proven analytically. The result is a specific weighting of the score-matching losses for different times t. The paper validates the derived methodology and empirically demonstrates very strong likelihood performance, state-of-the-art among models without data augmentations. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Overall impression:__ The main contribution of the paper, demonstrating how to train score-based diffusion models in a maximum likelihood fashion, using a lower bound objective, is a very important contribution. Score-based diffusion models recently generated much attention and demonstrated very high image quality synthesis, but their likelihoods were worse than those of some other models (autoregressive models, VAEs, etc.). This work solves this problem, demonstrates how to actually train the model directly towards good likelihood, and outperforms all previous models (when comparing to models without data augmentation). Consequently, I think this is a significant contribution and I expect future works on score-based diffusion models to often rely on the weightings derived in this paper. That being said, regarding the overall quality the paper lacks some discussions of important questions, and could discuss the context and limitations better. More specific feedback below.\n\n__Clarity:__ I think the paper is well written and easily readable.\n\n__Originality:__ The main contribution is original. The main contribution consists of several theorems that have not been appeared before, to the best of my knowledge. The proofs of the theorems seem solid and elegant. The paper also proposes importance sampling techniques and uses variational dequantization during training. However, these aspects are rather incremental.\n\n__Weaknesses, Open Questions, Suggestions:__\n1. The authors distinguish between $p^{ODE}$ and $p^{SDE}$, presenting it as if these are different generative models (e.g. in line 291 $p^{ODE}\\neq p^{SDE}$). However, the authors also write that both $p^{ODE}$ and $p^{SDE}$ share the same marginal $p_t$ for all $t$ (line 117). Doesn't this imply that they define the same generative distribution and therefore model? Could we argue that $p^{ODE}$ and $p^{SDE}$ are merely two different ways of writing the *same* generative model, with one allowing us to sample with black-box ODE solvers, and the other, for example, with ancestral sampling (and differences in practice mainly being due to numerical issues and different solvers used)? With this interpretation it is also not surprising that the likelihoods/bounds of $p^{ODE}$ and $p^{SDE}$ improve together. Or is the issue that the statement \"$p^{ODE}$ and $p^{SDE}$ share the same marginal $p_t$ for all $t$\" is only true for valid score functions modeling proper distributions, while the learnt score function $\\mathbf{s}_\\theta$ may not satisfy these conditions? To me it rather feels the crucial difference is in the way they are evaluated - in one case via actual integration with error guarantees due to ODE solver tolerances, as in Normalizing flows, in the other case using a stochastic ELBO-like estimator. I got a bit confused and feel this point could be explained more clearly.\n2. The models that achieve the strongest likelihoods tend to have worse FIDs than models trained without the likelihood weighting. There is a trade-off between NLL and FID. This is not discussed at all in the paper. How can we overcome this? However, looking at it from another perspective, the achieved FID scores, even for the models trained with the likelihood-weighting, are much better than typical FIDs obtained by previous autoregressive models, VAEs, flows, etc., which were also trained with likelihood objectives - and this is great! But why are FIDs so much better here? I think these aspects are quite interesting and important and deserve discussion.\n3. One of the main drawbacks of current score-based diffusion models is their slow synthesis speed. This aspect is not discussed at all in the paper. What is the number of function evaluations, i.e. network calls, of the ODE solver when sampling? I think this needs to be reported and discussed as it is an important metric when working with diffusion models and can vary wildly. Ideally, even wall-clock time could be measured. It's also important when comparing to other types of likelihood-based models.\n4. There is no thorough discussion of Related Works. I think contrasting the proposed *ScoreFlow* and its advantages and disadvantages to other types of likelihood-based generative modeling techniques and putting it into a broader context a bit more thoroughly would be nice.\n5. The authors say they do not compare to the VESDE, since the VESDE likelihood weighting is the same as the weighting in [1]. It seems, however, that [1] did not evaluate likelihood for their models. Also, [1] obtained very strong FID score (although when using predictor-corrector sampling) even though they trained with a weighting corresponding to the likelihood-weighting derived in this work. This stands in contrast to the worse FID scores obtained here whenever using the likelihood weighting. I feel that exactly because the weighting used in [1] already is the maximum likelihood weighting, it would be very interesting to evaluate VESDE models towards likelihood or at least to discuss this.\n6. Just a suggestion: If space is an issue for addressing some of my comments, I think the explanation of Variational Dequantization can be put in the Appendix entirely, as this is a standard technique. Also the proposed importance sampling is straight-forward and could be cut much shorter.\n\n__Conclusion:__ In conclusion, I think this is a good paper that deserves to be published, as I think its main contribution is very important and it quantitative results are strong. However, I think the paper would significantly profit from addressing the above concerns and also considering the feedback regarding \"Limitations and Societal Impact\".\n\n[1] Song et al. \"Score-Based Generative Modeling through Stochastic Differential Equations\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and Societal Impact were not addressed in the paper in an optimal manner.\n\n__Regarding limitations:__ I think obvious and well-known limitations of current score-based diffusion models, including the models and methods proposed in the paper, are: \n1. The slow synthesis speed, since a deep network needs to be called very often to obtain the final results (by the ODE solver). This stands in contrast to models such as GANs or VAEs, from which one can sample very efficiently.\n2. The used diffusion processes are defined for continuous data (or dequantized discrete data, if applicable, like for typical images), but cannot be used easily for truly discrete or categorical datatypes (e.g. language data, graphs, binary data, etc.). \n3. As discussed, there seems to be some trade-off between strong FIDs and NLLs. \n\nI think such limitations could be acknowledged and discussed better.\n\n__Regarding Societal Impact:__ The authors state that they did not discuss societal impact, since the method per se does not add any new negative impacts. I basically agree with this statement. However, this work essentially tackles generative modeling, which indeed can have negative societal impacts, such as synthesizing fake images, etc. This is well-known, but I think a pointer in this direction to acknowledge this would not hurt.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows how training continuous-time denoising diffusion models (a.k.a. score-based modes) can be converted to maximum-likelihood training with a proper weighting of the training objective. Although this connection is known to some degrees from classical literature (see below), the paper does an excellent job of showing how following this connection for training yields "SOTA" likelihood results with modern score-based generative models and proper importance sampling. Given the unanimously positive reviews and the potential impact of this work, I am very happy to recommend this paper for acceptance. 

That said, I'd like to encourage the authors to make the following changes for the camera-ready version:
- I appreciate coming forward with the updated numbers after discovering the numerical problems. Please make sure that the claims in this submission are adjusted after updating the numbers in the final camera-ready version. 
- The connection between the KL divergence and score matching objective was also discussed in classical papers such as [1]. Please add a small section discussing these earlier works.

[1] Lyu, Interpretation and Generalization of Score Matching. 
Decision: Accept (Spotlight)