A Variational Perspective on Diffusion-Based Generative Models and Score Matching
Authors: Chin-Wei Huang
Jae Hyun Lim
Aaron Courville
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Diffusion model
Neural SDE
Neural ODE
normalizing flows
score matching
hierarchical VAE
generative modelling
maximum likelihood
variational lower bound

Abstract: Discrete-time diffusion-based generative models and score matching methods have shown promising results in modeling high-dimensional image data. Recently, Song et al. (2021) show that diffusion processes that transform data into noise can be reversed via learning the score function, i.e. the gradient of the log-density of the perturbed data. They propose to plug the learned score function into an inverse formula to define a generative diffusion process. Despite the empirical success, a theoretical underpinning of this procedure is still lacking. In this work, we approach the (continuous-time) generative diffusion directly and derive a variational framework for likelihood estimation, which includes continuous-time normalizing flows as a special case, and can be seen as an infinitely deep variational autoencoder. Under this framework, we show that minimizing the score-matching loss is equivalent to maximizing a lower bound of the likelihood of the plug-in reverse SDE proposed by Song et al. (2021), bridging the theoretical gap.

A Variational Perspective on Diffusion-Based Generative Models and Score Matching
Chin-Wed Huang, Jae Hyun Lim, Aaron Courville University of Montreal & Mila
(chin-wei. haan-wei.hhang, jae. hoo.. 1im, aaron. courvilleeemmoatreedllc
Abstract
Discrete-time diffusion-basee generative models and score matching methods have shown promising results in modeling high-dimensional image data. Re- cently, Song et al. (2021) show that diffusion processes that transform data into noise can be reversed via learning the score function, iee the gradient of the log- density of the perturbed data. They propose to plug the learned score function into inverse formula to define generative diffusion process. Despite the empirical success, a theoretical underpinning of this procedure is still lacking. In this work, we approach the (continuous-time) generative diffusion directly and derive vari- ational framework for likelihood estimation, which includes continuous-time nor- malizing flows as special case, and can be seen an infinitely deep variational
autoencoder. Under this framework, we show that minimizing the sooremmtthhing loss is equivalentt to maximizing lower bound of the likelihood of the plug-in reverse SDE proposed by Song et al. (2021), bridging the theoretical gap.
Introduction
Generative modeling can be thought of as inverting an inference process. If the inference process is invertible, then one can focus on transforming the data into a tractable distribution (Dinh et al., 2016). If the inference process is deterministic yet non-in non-invertible, one could learn invert stochas- tically (Dinh et al., 2019; Nielsen et al., 2020). Most generally, both inference and generation can be stochastic. This is known the variational autoencoder (Kingma Welling, 2014; Rezende al., 2014, VAE).
Under the variational framework, one has lot of flexibility in choosing the generative and infer- ence models. Recent work on diffusion-based modeling (Sohl-Dickstein et al., 2015; Ho et al., 2020) can be thought of as removing one degree of freedom, by freezing the inference path. The infer- ence model is fixed discrete-time Markov chain, that slowly transforms the data into tractable prior, such as the standard normal distribution. The generative model is another Markov chain that is trained to revert this process iteratively. Diffusion brasusion-bbsedd models have been shown to perform remarkably well on image synthesis (Dhariwal & Nichol, 2021), rivaling the performance of state- of-the-art Generative Adversarial Networks (Brock et al., 2018).
Song al. (2021) connect liffusioo-based model and score matching (Hyvärinen & Dayan, 2005), by looking at the stochastic differential equation (SDE) associated with the inference process. They realize that the dynamic of the inference process can be inverted if one has access to the score function of the perturbed data, by solving another SDE reversed in time. They then propose to learn the score function of the inference process and substitute the approximate score into the formula of the reverse SDE to obtain a generative model. We call the resulting generative model the plug-in reverse SDE.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Figure I:
Three special cases of generative SDEs. The stars indicate the initial values, followed by some random sample paths. Left: trained with no diffusion o = 0 (í.e. neural ODE). Middle: trained with some fixed diffusion a Right: trained with fixed inference process, f and g (i.e. the plug-in reverse SDE).
Conceptually simple as this learning procedure may seem, little is known about how the score match- ing loss relates to the plug-in reverse SDE. In this paper, we propose a variational framework suitable for likelihood estimation for general generative diffusion processes, and use this framework to con- nect score matching with maximum likelihood. We do so by combining two important theorems in stochastic calculus: the Feynman-Kac formula for representing the marginal density of the gen- erative diffusion as an expectation (Section 3), and the Girsanov theorem for performing inference in function space (Section 4). We derive functional evidence lower bound that consistently ex- tends discrete-time diffusion models to have infinite depth, í.e. the number of layers goes to infinity generative and inference SDEs, ing objective equivalent to minimizing the (implicit) score matching loss (Section 6). Our theory suggests that by matching the score, one actually maximizes lower bound on the log marginal density of the plug-in reverse SDE, laying a theoretical foundation for this learning procedure. We further generalize result to family of marginal-equivaler plug-in reverse SDEs, including an equivalent ODE as a limiting case.
(Section 5). Finally, by reparameterizznng
obtain train-
Notation:
We use (Y,, 3) denote the inference process (where Y, is the data), and (X,, t) to denote the generative process (where Xo is a random variable following an unstructured prior). We and to distinguish the two directions, and always integrate the differential equations from 0 to T 0 (different from the literature, where sometimes
might see integration from T to 0). B, and B, denote the Brownian motions associated with the inference and generative SDEs, respectively. B₁ is a reparameterization of B, (see Section 4). q(y, s) and p(æ, t) denote the probability density functions of Y, and X,, respectively. We let Sy denote time-indexed parameterized function that will be used to approximate the score V log ( s). V is the gradient wrt the spatial variable (r or y, which we sometimes call position), o, i, and 0z, are partial derivatives, and H, denotes Hessian.
2 Background
Assume Yo follows the data distribution g3y, 0), and Y, satisfies the Itō SDE (Øksendal, 2003)
dy = J(,, ds + s) dB.,,
where f and y are chosen such that the density d(v, s) will converge to some tractable prior Po as s + T. Following Song et al. (2021), we assume g is position-independent "reverse" SDE, whose marginal density evolves according to q(y, s), reversed in time, for example!
possible to find
dX = (gg T V log q(X, - 5) dt + gdB
(2)
If Xo Po. then the density p(a, L) of X, is equal to g(z. T l). This means that if we have access to the score function V log g, we can solve the above SDE to obtain Xy d Yo. Song et al. (2021)
See Appendix G for a family of equivalent (reverse) SDEs indexed by some parameter X, of which equation (2) special case with À= = 0.

---

Method
Loss
F-K
F-P
LESM
B (Y, 5) V (ogg(Y
o(y.s)
plu. T -s)
LISM
1 Sy (Y, s)
R + V. (A se)
c(y. 5)
-V V.uy,. - -s)
LSsM
E l
s) ll
+ + ((A So
b(y.s) 5)
-ulyI T
LPMM
1 So 8)
V log g Y. l
n(y.5)
a(T - 5) Po(y)
((V)
Table 1: Score matching losses.
follows the
Rademacher distribution.
Table 2: Feynman-Kaaa coefficients.
propose to approximate the score via parameterized score function Sy by minimizing
Ev. F.
(1so
log (Y,,, ()
ds
where the expectation in the integral is known as the explicit score matching (ESM) loss LESM. and Ass) is a positive definite matrix² that serves as a weighting function for the overall loss. LESM not immediately useful, since do not have access the ground truth score log q. few alternative losses can be used, which are all equal to one another up to constant, including implicit score matching (Hyvärinen & Dayan, 2005, ISM), sliced score matching (Song et al., 2020, SSM), and denoising score matching (Vincent, 2011, DSM). The losses are summarized in Table 1, and are related through the following identity (see Appendix A for the derivation):
1. [I((u,,))) LISM LsSM LDSM - Er, ((((v-llo))) LESM 2
where I(g) = B log 9 []] is constant. After training, Song et al. (2021) plug into (2) to define a generative model. We refer this SDE as the plug-in reverse SDE. The plug-in reverse SDE has been demonstrated to have impressive empirical results, but theoretical underpinning of this learning framework is still lacking. For example, it is unclear how the training objective (minimizing the score matching loss) relates to the sampling procedure, e.g. whether the probability distribution induced by the plug-in reverse SDE gets closer to the data distribution in the sense of any statistical divergence or metric. We seek to answer the following question in this paper: How will minimizing the score-matching loss impact the plug-in reverse SDE? We first provide a framework to estimate the likelihood of generative SDEs, and then get back to this question in Section 6.
3 Marginal density and stochastic instantaneous change of variable
Let X, be diffusion process solving the following Itô SDE
dx (xX,t) t) dt + (X,tt)
(4)
with the initial condition X,- ~ Po- which induces a family of densities X, ~ p(-,t) We use this SDE as the generative SDE, and we are interested in log,(æ, T)) for maximum likelihood. The density p(a, 1) follows the Kolmogorov forward (or the Fokker Planck) equation:
Oip(z,t)= = Σ (e p(x,t))
Σ 1i,,,, p(x,
(5)
with the initial value p(-.0) = Po(.), where D ooo0 the diffusion matrix. We can expand the Fokker Planck and rearrange the terms to obtain
0p((x,t) -V m(a,t) + Σ 0h Ph t)
p(x,t)+
Σ
pi(z,t) + 2., Diy(E 0x,p(z,t) + t) p(r,
(6)
We use this matrix to induce a Mahalanobis norm ||=|| := = Ae. which be used in Section Forr generality, we use the notation f and to describe generative SDE, which will be set to " and g when come back to the discussion of the plug-in in reverse SDE in Section 6.
3

---

so that all coefficients of the same order are grouped together. For simplicity, we assume the diffu- sion term o is independent of z throughout the paper. Then (6) reduces to
O.p(2, () = - (V p(x, 1)) p(z, () u(x, Vp(a, + D(I) (x,
(7)
where: denotes the Frobenius inner product between matrices. Even with this simplification, solv- ing (7) is not trivial. Fortunately, can estimate this quantity using the Feynman Kac formula, which tells us that the solution certain second-orde linear partial differential equations have a probabilistic representation.
Theorem (Feynman-Kac representation, Chapter 5.7 of Karatzas & Shreve (2014)), Let > O. Let y and & be the spatial and temporal arguments to the function E C (R' x [O, solving
0,r cv Vu
Ve A: Hg =0
with the terminal condition v(y,T) T) = hly), where A = 2mm=" for some matrix-valued function mly. s). Under the assumption stated in Appendix if B', is Brownian motion and Y, solves
dY = b(Y, ds + (Y, s) dB',,
with the initial datum Y- y,, then
v(y, h(YT) exp ([ Ye = y
(10)
To estimate the density p(- T) of (7), we can apply the change of variable p(ar, 1) == v(x, 1) by letting the Feynman-Kau -ac (F-K) coefficients correspond to their Bokkeer-Plande (F-P) counterparts according to Table 2. This way, solving (8) backward equivalent to solving (7) forward, and we have the following representation of the marginal density at T:
u(Y,, T - s) ssds) ds p(x, T) = E Po(YT) exp (%- -v
Yo 76=2/
(11)
where Y, is a diffusion process solving
dY = =- -dd - s) dB
(12)
Remark (Marginalizationn. This representation can be interpreted as a mixture of continuous time flows. Assume sample path of the Brownian motion is given, and are interested in how the density evolves following the dynamic (4). In the infinitesimal setting, it can been seen as applying the invertible map + x u(x, () + (I) where AB, == B(++)) Bi the Brownian increment. Since the diffusion term is independent of the spatial variable, it can be seen as constant additive transformation, which is volume preserving, so it will not be taken into account when computing the change of density. The only contribution to the change of density will be from the nog-determinant of the Jacobian of idd+pu uttt which means we can simply apply the instantancous change of variable formula (Chen et al., 201 8). This will be the conditional density given the entire {B, 2 0}. and marginalizing out results in the expectation in (11). See Appendix C for details.
Our framework also works with the general case where a depends on x. but the formulae need to be adapted account for the spatial partial derivatives. See Appendix D for the derivation.
4 Inferring latent Brownian motion
As our goal is to estimate likelihood, we would like to compute the log density value using (11). However, this involves integrating out all possible Brownian paths, which intractable. To resolve this, we view the Brownian motion latent variable, and perform inference by assigning higher probability to sample paths that are more likely to generate the observation. One can view this as a VAE, except have infinite dimensional latent variable.
Formally, let (2,F,PP) PP) be the underlying probability space for which B' is a Brownian motion. Suppose Q is another probability measure on (S, F) equivalent to P; that is, IP and Q are similar in

---

the sense that they have the same measure zero sets. This allows us to apply the change- of measure trick and lower bound the oog-iiklihood with finite quantity using Jensen's inequality:
dF
+ log L.
logp(x, T) 2 Be log dQ
V uss Yo
(13)
Note that ay continuous wrt a third measure, say Lebesgue, then the derivative can be expressed as the ratio of the two densities. However, since we are dealing with an infinite dimensional space, we are immediately faced with the following problems:
dP is the Radon-Nikodym derivative of P wrt Q. When both measures are absolutely
1. Is there measure Q (equiv. to P) for which de can be easily computed, or at least numer- ically approximated?
ae
2. Can we find a reparameterization (similar to the Gaussian reparameterizationn) of B', under the new law Q to estimate the gradient needed for training?
We resort to the Girsanov theorem, which describes general framework for dealing with the change of measure of Gaussian random variables under additive perturbation. It allows us to consider the law of a diffusion process as Q. See Appendix E for an explanation using the more familiar notion of probability densities.
Theorem 2 (Girsanov theorem, Theorem 8.6.3 of Øksendal (2003)). Let B, be an Itô process solving
dß. alw, 8) ds + dB',
(14)
for w E Ω
S dsds) ds)
0
8 S T and Bo = O, where a(w,s) satisfies the Novikov's condition
exp
2
So a²
< co. Then B, is a Brownian motion wrt Q where
dQ
dlp (w) == exp alw. 5) .B', - 1 2 S [|a(w, ds
(15)
Equation (14) provides standarization formula of B', under Q, which means we can "invert" it to reparameterize B', This leads to the following lower bound.
Theorem (Continuous-time ELBO). Let Q be defined via the density (15). Then the RHS of (13) can be rewritten as
[
E
|la(w, [la(w,))ll2 + log Pog - V. uds
Yo = =: =0
(16)
where the expectation taken wrt the Brownian motion B., and Y, solves*
dY (-p + σa) ds + adB.
(17)
We call Y, solving (17) the inference SDE, and 8o0 the continuous- time ELBO (CT-ELBO). Remark 2 (Computation). This lower bound can be numerically estimated by using any black box SDE solver, by augmenting the dynamic of with the accumulation of ||a||2 and V n.. Computing the divergence term V yu directly can be expensive, but it be efficiently estimated using the Hutchinson trace estimator (Hutchinson, 1989) along with reverse-mode automatic differentiation, similar Grathwohl al. (2018). As the parameters of both the generative and inference models are decoupled from the random variable B., their gradients can be estimated via the reparameteri- zation trick (Kingma& & Welling, 2014; Rezende et al., 2014). Furthermore, backpropugation can be computed using an adjoint method with a constant memory cost (Li al., 2020).
Remark 3 (Drift a). (i) In general, the drift term of the approximate posterior to the latent Brownian motion can be amortized, so that will encode the information of individual datum x. (ii) The regularization ||a||2 ensures that is kept close to 0, since it represents the deviation of the measure
'Note that μ and run backward in time from T. whereas forward.
5

---

Theorem (Variational gap and optimal inference SDE). The variational gap can be written as it induces (t.e. Q) from the classical Wiener measure (which is centered Gaussian measure). (iii) When the diffusion coefficient a is O, the inference SDE reduces to the reverse dynamic of the generative ODE, and if a = O in this case, the lower bound is tight. (iv) There is generally no constraint on the form of alw, 8), so one can potentially augment it with additional dimensions to have non-Markovian inference SDE. For simplicity, we let the inference SDE be Markovian model, i.c. a = a(y,s), s). This is justified by the following theorem.
T =0= / [[la(w,)) o' Vlogp(Y - ds.
log 7) -0 =
s)]]
(18)
In particular, -0 p(x, T) and only alw, 8) can be written as alw, alY, (w), s) for almost every E [) and Ω. and aly, 8) - pog s) almost everywhere.
Remark 4 (Variational gap). Even though the inference SDE seemingly takes simple form, is sufficiently flexible in that this type of variational problem can be generally solved by taking the supremem over all progressively measurable processes a(w, s) (Boué et al., 1998). In fact, the above theorem shows that 8h log p(xr, T) if and only if a(y,s) a T V log T 8). This means non-amortized Markovian inference process is powerful enough.
Infinitely deep hierarchical VAE
Before we make the connection to score matching, we formally address the common belief that "diffusion models can be viewed the continuous limit of hierarchical VAEs" (Tzen & Raginsky, 2019), and show that the CT-ELBA consistently extends their discrete-time counterpart. We do so by inspecting the ELBO of hierarchical VAE defined as discretized generative and inference SDEs. We assume the generative model (i.e. the decoder) follows the transitional distributions
p(#+4 (x:) NN (x()),
(19)
o Ato?
(20)
where At = T/L is the step size and L is the number of layers. For the inference model (i.e. the encoder), we assume
(( = (( 04+1) (=(x)= = -x+ ((-)), At) o (i)t)a((, iAt)) 07 (iAt)
(21)
(22)
These transition kernels constitute a hierarchical variational autoencoder of T. stochastic layers, whose marginal likelihood can be lower bounded by
L-1 Σ log p( (x:) =: 81
logp(x1) 2 Ey logp(æo) +
(23)
which we refer as the discrete-time ELBO (DT-ELBO). The reconstruction error of the stochastic layer can be seen as some form of finite difference approximation to differentiation, which gives rise to V yu in the CT-ELBO in the infinitesimal limit (as At approaches 0). The regularization of ||a|| pops up when we compare the difference between iu; and f., using the Gaussian reparameterization compute the reconstruction error. We formalize this idea in the following theorem.
Theorem (Consistency). Assume pt, a, a -2 a, ||a|| and their derivatives up to the fourth order are all bounded and continuous, and that is non-singular. Then E3 - 0oD as L +
This theorem tells us that the CT-ELBO we derive for continuous-time diffusion models is not that different from the traditional ELBO, and that maximizing the CT-ELBO can be seen as training an infinitely deep hierarchical VAE. We present the proof in Appendix F. which formalizes the above intuition, using Taylor's theorem to control the polynomial approximation error, which will go to as the step size At vanishes when the number of layers L increases to infinity.

---

By [log at
-100
By [log ax (Pr,
By Be]l["]
> A
1 620
840
860
960
980
1000
Figure 2: Left: Samples from plug-in reverse SDEs with different values (rows). We use the same score function So trained on the Swiss roll dataset, and plug it into (27). For generation, we use the Euler Maruyama method with step size of At 1/1000. We visualize the samples for the i-th iterates (columns), which approximately represent the same marginal distribution when the score function is well trained. Right: Lower bound on the marginal likelihood of continuum of plug-in reverse SDEs. The lower bound is optimized when the score matching loss is minimized, which will push up the entire dark blue curve.
Score-based generative modeling
Recall that our goal is to analyze the plug-in reverse SDE and draw connection to score matching. To this end, we reparameterize the generative (4) and inference (17) SDEs as
dx (gg) So f) dt + gdB, and dY = fds ds + gd..
(24)
by letting g So. where the time variable is reversed (T t) for the generative process, and forward time (8) for inference. The ELBO (16) can be rewritten as
-0 = Ey, (oog Yo x] - L Ey,
Sa
1 gg +
(gg' Se f) Yo ds. (25)
Comparing the integrand the implicit score matching loss Table 1, we immediately see that the network So approximates V log q(y, s), the score function of the marginal density of Y,. That is, matching the score of g(y,)) amounts to maximizing the lower bound on the marginal likelihood of the plug-in reverse SDE.
Recently, Durkan & Song (2021) also attempt establish the equivalency between maximum like- lihood and score matching, by showing the following relationship between the forward KL diver- gence and a weighted sum of score matching loss (aka the Fisher divergence):
1 2 /." (.)).)) llV - V vl) 1)1 ds,
DKL(g(3,
()llr(,)))) Dkl(g(v.0)llr(x.00
(26)
where r(y, ) is the density of Y, solving the same inference SDE with the initial condition you ~ r(., 0), assuming Ty.TT) rly T). However, it is inaccurate to claim that score matching is equivalent to maximum likelihood. This is because if we simply let r(y. 0) = p(y, T), i.e. the density of the generative SDE evaluated
y. r(y,8) will not necessarily be the same either p(y, T s) or Sy 8). This means the KI. divergence is not equal to the integral
the weighted score matching loss | S V log 1 = In fact, the latter corresponds to a lower bound on the likelihood (the cross-entropy term the KL) up to some constant, as equation (25) suggests.
More generally, we can apply our analysis to a family of plug-in reverse SDEs indexed by some parameter À <
dx = (11 9 f) di + VT and dY = (f 2 logg) ds + vI BB,
(27)
'We follow the Euler-Maruyant (EM) scheme. Other discretization scheme may also work; leave that for future work.
'This refers to the vl of the paper arXiv. This version was later on replaced with a new version where they derived a similar bound ours.
7

---

where we assume is diagonal for simplicity. We defer the formal discussion to Appendix G, but the essence is that this inference SDE induces the same marginal distribution as (1), and the generative SDE is its corresponding plug-in reverse. Equation (27) includes the original plug-in reverse SDE (24) and an equivalent ODE as special cases with X 0 and = 1. Denote its corresponding CT-ELBO by 8 Specifically, (25) becomes E Then we have the following relationship.
Theorem 6 (Plug-in reverse SDE ELBO, abridged). For A L
X 4(1 - A) /
By, By 1
By; (Y,. v Vlog (Y., 3)13 ds (28)
We state the full theorem in Appendix H. where we rearrange the terms to show that the average CT-ELBO of the X-plug-in reverse SDE is also equivalent to the ISM loss, similarly to (25) but up to some multiplying and additive constants. The implication is that while minimizing the score matching loss, we implicitly maximize the likelihood of a continuum of plug-in reverse SDEs which include the ODE as limiting case (X + 1). See Figure 2 (right) for illustration. This suggests the likelihood of the equivalent ODE can be improved by minimizing the score matching loss, as the ODE's likelihood will be close to plug-in reverse SDEs with À 1, which explains the good likelihood of the equivalent ODE reported in Song et al. (2021). In practice, we can only estimate the ELBO of the case À 0 since otherwise there will be some constant we do not have access to, but their gradients can all be estimated via score matching.
6.1 Computational trade-off
Having general framework for estimating the likelihood of diffusion processes allows us com- pare a wide family of models, including continuous-ting time flows and plug-in reverse SDEs trained by score matching. We compare the two by measuring the negative ELBO throughout training to highlight their computation-estimatio trade-off. We train the models on the Swiss roll toy data. For continuous-time flow, we = 0, using the Hutchinson trace estimator following Grathwohl et al. (2018). The ELBO in this case is tight since will be penalized to be O. We use the torchdiffeo library (Chen al., 2018) for numerical integration for fairer comparison" For plug-in reverse SDE, we train the drift network a using SSM and DSM (for DSM the loss is weighted to reduce variance, which introduces some bias; see the next subsection). We use the variance- preeeerviig inference SDE from Song et al. (2021), which allows us to sample Y, using a closed form formula, for sam- pled uniformly between [o, T). The trained models are visualized Figure 1, the learning curves presented in Figure 3.
From the learning curve figures, we that likellood decreases rapidly for the continuous-time flow in terms of the number of parameter updates. But once the x-axis is normalized by runtime, the convergence speed becomes almost indistinguishable. This because for continuous numerical integration takes time, whereas for plug-in reverse SDEs, we train on random time step s; that is, within fixed amount of time the latter can make more parameter updates the cost of noisier gradients. Note that both models have constant memory cost (wrt T or L, the number of integration steps), so large batch size can be used to reduce variance for training.
flows,
6.2 Bias and variance trade-off
The integral in equation (25) can be estimated by sampling (Y,, s), and using the Hutchinson trace estimator to estimate the divergence, which corresponds to implicit score matching. However, in practice the variance of this estimator is very high when the norm of the Jacobian Vs, is large. Another popular approach is to use the denoising estimator (recall the identity from (3)).
Ey, 1
(V,, V log gYY,)V)) 12g Yo x
(29)
The inference SDE is typically chosen so that Y, can be easily sampled, e.g. following A (( of), where μ, and og are functions of Yo and s. In this case, if we reparameterize Y, Mg + age where
"Black-bor box SDE solvers such as torchsde (Li et al., 2020) might not be optimized for the deterministic case, since their stochastic adjoint method scales OL log L) in time whereas deterministic numerical solvers usually faster. This matters for our runtime comparison.
8

---

Dannsising
Dennaing
60
Siioe
Nog E.B E0
60
Siice
Nog
10"10
14"10"
10
33
2x
2x100
30K
50K
1000
2000
3000
4000
5030
6030
7000
Norations
Time (Seconds)
Figure 3: Neural ODE vs plug-in reverse SDE (denoising or slice score matching). The learning curves presented as function of iterations (left) and runtime (right) to emphasize the computa- tional distinction between the two families models.
2=110
Standard Eror
BPD
16"
10
6-100
6x110
0K
25K
40K
50K
8CK
103K
120K
140K
0X
20K
40K
6OK
B0K
100K
120K
140K
herations
leerations
10
Standard Enter
2x110
Model
3
0'"10"
deteaoed
4x100
Ilerations
Figure 4: Likelihood estimation on MNIST (first row) and CIFARIO (second row). Sş and a denote which model we parameterize. Y-axes are bits-per-dim and the standard error of BPD of the test set. The debiased curves improve upon the original biased gradient estimator (Song et al., 2021) since it maximizes a proper ELBO. Shaded area reflects the uncertainty estimated by 3 random seeds.
~~ ~N,00, I), then the score becomes V log = - Since ds + as - 0, this estimator normally has unbounded variance. Song & Ermon (2019); Song et al. (2021) propose to remedy this by multiplying the DSM loss by o²/g² (assuming is scalar for simplicity), that the target has constant magnitude on average E 21 c which would result biased gradient estimate with much smaller variance. We can debias this estimator by sampling ~ q(s) xx g²/0²² This ratio, however, is usually not normalizable in practice (as it integrates oo). As alternative, we consider the following unnormalized density āe(s) = (s=)//o² for 8 E 8e). and (e(s) (s)/o² for E [se. T'. We experiment with this debiased procedure by sampling and chosen to be the variance-preservinn preserving SDE. s, small so that the bias is negligible.
--
x qe. for
We train the model on MNIST (LeCun et al., 1998) and CIFARIO (Krizhevsky et al., 2009). We present the learning curves and the standard error of the estimate of the ELBO in Figure 4. The lower bound is estimated using the Hutchinson trace estimator with sampled uniformly from [0. 7), with the same batch size, the only thing that will affect the dispersion is the magnitude of Sg. Since smaller values of are more likely to be sampled under Je. the debiased model will see samples with less perturbation more often. On the contrary, sampling uniformly will bias the model to learn from noisier data, causing the learned score to be smoother. We also experiment with parameterizing Sş vs parameterizing a. We find the latter parameterization be helpful since the relationship Sy y La has the effect of negating the multiplier in the reweighted loss, i.e. 1 l. a + ell²). This is similar to the noise conditioning technique introduced in Song & Ermon (2020).
7 Related work
Diffusion-based generative models Our work lays a theoretical foundation for Song (2021), which recognizes that conditional denoising score matching (Song & Ermon, 2019, 2020) and

---

discrete-time diffusion- generative models (Sohl-Dickstein et al., 2015; Goyal et al., 2017; Ho al., 2020) can be viewed as learning to revert an inference process (using the plug-in reverse SDE). Different from Ho et al. (2020), which shows the ELBO of discrete time diffusion process can be likened to DSM (Section 3.2 of the paper), we show that ISM loss naturally arises from the Fokker-Planck equation of the marginal density, via the Fenman- Kac representation and the Girsanov change of measure. This line of work has been successfully applied to modeling high di- mensional natural images (Dhariwal & Nichol, 2021; Saharia et al., 2021), audio (Kong et al., 2020), 3D point cloud (Cai et al., 2020; Zhou et al., 2021), 1), and discrete data (Hoogeboom et al., 2021).
Time-reversal of diffusion processes Plenty of works have studied the reverse-time diffusion processes (2), including Anderson (1982); Föllmer (1985); Elliott & Anderson (1985); Haussmann & Pardoux (1986). These are different from our marginal-equivalers (reverse) processes (27) when Sg V log4, q, since the latter is related by the marginals only.
Score matching for energy-based models Besides the connection to difffusion models, score matching is also often used as a method for learning energy based models (EBM)- see Song & Kingma (2021) for comprehensive review useful techniques- When used as an EBM, sam- pling from the conditional score model can be achieved by running the annealed Langevin diffusion (Neal, 2001), which is connected to free-energy estimaton in physics (Jarzynski, 1997), wherein the path integral is essentially a Feynman Kac representation.
De Bruijn's 's identity To connect maximum likelihood and score matching, Durkan & Song (2021) shows that KL divergence can be represented as integral of weighted Fisher divergence, gener- alizing the case of Lyu (2009) where the inference perturbation is a simple Brownian motion. This type of formulas fall into the category of de Bruijn's 's identity (Cover, 1999) for relative entropy. A similar differential form result can be found in Wibisono et al. (2017).
Learning SDEs Tzen & Raginsky (2019); Li et al. (2020) also propose to learn a neural SDE by applying Girsanov's theorem. The key difference is that they treat the SDE entirely latent variable, with an additional emission probability, whereas we use the Feynman-Kaa formula to directly express the marginal density as an expectation, side-stepping the need smooth out the density using the emission probability (which will be Dirac point mass in our case). In their case, the inference direction is the same the generative direction, since they infer the latent SDE directly, whereas we apply Girsanov to the Feynman- Kac diffusion (opposite the generative direction). Xu al. (2021) further apply neural SDE as an infinitely deep Bayesian neural network.
8 Conclusion and Discussion
In this work, we derive a general variational framework for estimating the marginal likelihood of continuous-time diffusion models. This framework allows us to study a wide spectrum of models, including continuous- work, we show that performing score matching with a particular choice of mixture weighting is equivalent to maximizing lower bound on the marginal likelihood of a family of plug-in reverse SDEs, of which the one used in Song et al. (2021) and the equivalent ODE are special cases. Em- pirically, we validate our theory by monitoring the ELBO while performing score matching, and discuss the implication of the choice of mixture weighting and the potential of debiasing via non- uniform sampling. We emphasize that our theory does not explain the impressive sample quality of this family of models, which is still an open research problem and we leave it for future work.
normalizing flows and score-based generative models. Using our frame-
This work introduces general framework to estimate the likelihood of diffusion-based models, which allows the parameters of both the generative and inference SDEs to be learned, using nu- merical solver with constant memory cost (as per Remark 2). The training time can be reduced via the connection to score matching and the reverse-time time parameterization (24) as long as f and g take a simple form (so that Y; can be sampled without numerical integration). For example, one can generalize the Drnstein-Uhlenberr variance-presering SDE, by parameterizing the integral of ʃ using monotone network (Sill, 1998; Kay & Ungar, 2000; Daniels & Velikova, 2010; Huang et al., 2018). This has been explored in concurrent work by Kingma et al. (2021) in a different framework.
process to have non-linear (in time) ʃ and g, similar to the
10

---

Acknowledgements
We would like thank David Kanaa, Ricky Chen, Simon Verret, Rémi Piché-Taillefer Alexia JolicoeurMaartnnauu and Faruk Ahmed for giving their feedback on this manuscript. We would also like to thank the INNF+ 2021 reviewers and NeurIPS 2021 reviewers for their constructive suggestions, which help us improve the clarity of the paper. Chin-Wei is supported by the Google PhD fellowship.
We also acknowledge the Python community (Van Rossum & Drake Jr, 1995; Oliphant, 2007) for developing the tools that enabled this work, including numpy (Oliphant, 2006; Van Der Walt al., 2011; Walt et al., 201 I; Harris et al., 2020), PyTTorch (Paszke et al., 2019), Matplotlib (Hunter, 2007), seaborn (Waskom al., 2018), and SciPy (Jones et al., 2014).
References
Anderson, B. D. Reverse-time diffusion equation models. Stochastic Processes and their Applica- tions, 12(3):113322, 1982.
Boué, M., Dupuis, P., et al. A variational representation for certain functionals of brownian motion. The Annals of Probability, 26(4): 1998.
Brock, A., Donahue, J., and Simonyan, K. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 1809 11096, 2018.
Cai, R., Yang, G., Averbuch-Eloo H., Hao, Z., Belongie, S., Snavely, N., and Hariharan, B. Learning gradient fields for shape generation. In Computer Vision ECCV 2020 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part III, 2020.
Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. Neural ordinary differential equations. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pp. 6572-6583, 2018.
Cover, T. M. Elements of information theory. John Wiley Sons, 1999.
Daniels, H. and Velikova, M. Monotone and partially monotone neural networks. IEEE Transactions on Neural Networks, 21(6):906-99) 2010.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2/05.05233 2021.
Dinh, L., Sohl-Dickstein, J., and Bengio. S. Density estimation using real nvp. In International Conference on Learning Representations, 2016.
Dinh, L.,, Sohi-Dickstein, J., Larochelle, H., and Pascanu, A rad approach to deep mixture models. arXiv preprint arXiv: /ttx:v:99033077144 2019.
Durkan, C. and Song, Y. On maximum likelihood training of score-based generative models. arXiv preprint 1:Xi::21010092: 2021.
Elliott, R. J. and Anderson, B. D. Reverse time diffusions. Stochastic processes and their applica- tions, 19(2):327-333 1985.
Föllmer, An entropy approach to the time reversal of diffusion processes. In Stochastic Differen- fial Systems Filtering and Control, pp. 156-163. Springer, 1985.
Goyal, A., Ke, N. R., Ganguli, S., and Bengio, Y. Variational walkback: Learning a transition operator stochastic recurrent net. Advances in Neurul Information Processing Systems, 2017.
Grathwohl, W. Chen, R. T., Bettencourt, J., Sutskever, I., and Duvenaud. D. Ffjord: Free-form continuous dynamics for scalable reversible generative models. In International Conference on Learning Representations, 2018.
Harris, C. R., Millman, K. van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., al. Array programming with numpy. Nature, 585(5)))): 357-362, 2020.

---

Haussmann, U. G. and Pardoux, E. Time reversal of diffusions. The Anmals of Probability, PP- 1188-1205. 1986.
Ho, J., Jain, A., and Abbeel, Denoising diffusion probabilistic models. Advances in neural information processing systems, 2020.
Hoogeboom, E., Nielsen, D., Jaini, P., Forré, P., and Welling, M. Argmax flows and multinomial diffusion: Towards non-autoregressive language models. arXiv preprint arXi:22102.05379 2021.
Huang, C.-W., Krueger, D., Lacoste, A., and Courville, A. Neural autoregressive flows. In Interna-
tional Conference on Machine Learning, pp. 2078-2087 2018.
Hunter, J. D. Matplotlib: 2d graphics environment. Computing in science & engineering, 9()): 90, 2007.
Hutchinson, M. F. stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics-Simulatiion and Computation, 18(3):155--1076 1989.
Hyvärinen, A. and Dayan, P. Estimation of non-normalized statistical models by score matching.
Journal of Machine Learning Research, 6(4), 2005.
Jarzynski, C. Equilibrium free-energy differences from nonequilibrium measurements: master- equation approach. Physical Review 56(5):5018, 1997.
Jones, E., Oliphant, T., and Peterson, P. {SciPy): Open source scientific tools for (Python). 2014.
Karatzas, I. and Shreve, S. Brownian motion and stochastic calculus, volume 113. springer, 2014.
Kay. H. and Ungar, L. H. Estimating monotonic functions and their bounds. AIChE Journal, 46 (12):2426-2434, 2000.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Kingma, D. P., Salimans, T., Poole, B., and Ho. J. Variational diffusion models. arXiv preprint arXiv:2107.006300 2021.
Kong. Z., Ping, W., Huang. J., Zhao, K., and Catanzaro, B. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.0977 2020.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-bard learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Li, X., Wong, T.-K. L., Chen, R. and Duvenaud, D. Scalable gradients for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, pp. 3870-3882. PMLR, 2020.
Lyu. S. Interpretation and generalization of score matching. In Proceedings of the Twenty- Conference on Uncertainty in Artificial Intelligence, pp. 359-366, 2009.
Milshtein, G. Approximate integration of stochastic differential equations. Theory of Probability & Its Applications, 19(3):557-562, 1975.
Neal, R. M. Annealed importance sampling. Statistics and computing, 11(2):125-139, 2001.
Nielsen, D., Jaini, P., Hoogeboom, E., Winther, O., and Welling, M. Survae flows: Surjections to bridge the gap between vaes and flows. Advances in Neural Information Processing Systems, 33, 2020.
Øksendal, B. Stochastic differential equations. In Stochastic differential equations, pp. 65-84. Springer, 2003,
Oliphant, T. E. A guide NumPy, volume 1. Trelgol Publishing USA, 2006.
12

---

Oliphant, T. E. Python for scientific computing. Computing in Science & Engineering, 9(3):10-20, 2007.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, nigh-performance deep learning library. In Advances in neural information processing systems, pp. 1926-8037, 2019.
Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate infer- ence in deep generative models. In International conference on machine learning, pp. 1278-1286.
PMLR, 2014.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolutional networks for biomedical image
segmentation. International Conference on Medical image computing and computer-assistdd intervention, pp. 234-241. Springer, 2015.
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, J. and Norouzi, M. Image super-resolution via iterative refinement, arXiv preprint arXiv:2104.0766:: 2021.
Sill, J. Monotonic networks. In Advances in neural information processing systems, pp- 661-6667 1998.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, pp- 2256-2265. PMLR, 2015.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In Advances in neural information processing systems, 2019.
Song, and Ermon, Improved techniques for training score-based generative models. arXiv preprint (rXiv22006.0001 2020.
Song, Y. and Kingma, D. P. How to train your energy-base models.
arXiv preprint
arXiv:2101.03288, 2021.
Song, Y., Garg, S., Shi, J., and Ermon, S. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty Artificial Intelligence, pp. 574-584, PMLR, 2020.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021.
Tzen, B. and Raginsky, M. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv preprint TXXv:1905500883 2019.
Van Der Walt, S., Colbert, S. C., and Varoquaux, G. The numpy array: a structure for efficient numerical computation. Computing in Science & Engineering, 13(2):22, 2011.
Van Rossum, G. and Drake Jr, F. L. Python reference manual. Centrum voor Wiskunde Infor- matica Amsterdam, 1995.
Vincent, P. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-1674 2011.
Walt, S. v. d., Colbert, C., and Varoquaux, G. The numpy array: structure for efficient numerical computation. Computing in science & engineering, 13(2):22-30, 2011.
Waskom, M., Botvinnik, O., O' Kane, D., Hobson, P., Ostblom, J., Lukauskas, S., Gemperline, D. C., Augspurger, T., Halchenko, Y., Cole, J. B., Warmenhoven, J., de Ruiter, J., Pye, C., Hoyer, S., Vanderplas, J.. Villalba. S., Kunter, G., Quintero, E., Bachant, P., Martin, M., Meyer, K., Miles,
A., Ram, Y.. Brumner, T., Yarkoni, T., Williams, M. L., Evans, C., Fitzgerald, C., Brian, and
Qalich, A. mwaskom/scaborn: v0.9.0, July 2018.
Wibisono, A., Jog, V., and Loh, P.-.. Information and estimation in fokker-planck channels. In 20/7 IEEE International Symposium Information Theory (ISIT), pp. 2673-2677. IEEE, 2017.
13

---

Xu, W., Chen, R. T., Li, X., and Duvenaud, D. Infinitely deep bayesian neural networks with stochastic differential equations. arXiv preprint arXiv:2102.06559, 2021.
Zhou, L., Du, Y., and Wu, 3d shape generation and completion through point-voxel diffusion. arXiv preprint arXi::204.006670 2021.
14


Reviews:
[{"review_id": "VyQRpBJScb", "reviewer": "Reviewer_k6w1", "summary": "This paper rigorously shows how to calculate a variational lower bound on the marginal log density of a continuous time diffusion model. The bound contains implicit score matching, and it is calculated using Girsanov’s theorem from a Feynman-Kac representation of the marginal density obtained from the Fokker-Planck equation.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an excellent paper. I appreciate that the authors are rigorous for continuous time processes, and I like how the authors derive a marginal density without an emission probability.\nMy only concerns are regarding its framing in the context of prior work. My suggestion is to be more precise in describing the contributions as showing that *implicit* score matching arises from a variational lower bound on log likelihood, as opposed to existing and concurrent work showing that *denoising* score matching arises from a variational bound derived differently, starting with discrete time argument in DDPM (which, in section 7, should also be credited for showing that denoising score matching is learning to reverse an inference process).\n\nIf the goal of deriving implicit score matching is made clear throughout the paper, I think it will clarify to the reader the advantages of their method of deriving the variational lower bound as opposed to more traditional approaches, e.g. the approach from the original DPM paper, or modifying the approach from Tzen and Raginsky. And from this perspective, it is quite remarkable that the implicit score matching formula arises from Girsanov and Feynman-Kac, and I commend the authors on this finding.\n\nIt would be nice if the authors could discuss their ELBO in a more general setting, e.g. in which both forward and reverse processes have learnable drifts, or the setting of Tzen and Raginsky. How will training work in that case? Furthermore, can the authors discuss exactly when their method has advantages or disadvantages over other bounds, for example in lines 286-293? In the variational inference setting of Tzen and Raginsky, what does the authors’ proposed ELBO look like, and how does it compare to Tzen and Raginsky’s ELBO?\n\nIt appears that sections 3 and 4 use notation that refers to general SDEs (e.g. equation 4), but sections 2 and 6 use the same notation to refer to score-based generative modeling SDEs (e.g. equation 2). The change in notation is a bit confusing to the reader.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "GbB5g3PEkth", "reviewer": "Reviewer_dz2y", "summary": "This paper presents a framework for studying continuous diffusion models. Thanks to this it demonstrates the plug in generative model corresponds to a continuous version of the discrete diffusion models and is equivalent to minimising an ELBO on the likelihood of the model. \nIt does so by first transforming the change of density associated to the SDE into a second order linear differential equation and then expressing its solution as an expectation over another SDE. Solving this expectation is performed via variation inference on the Brownian paths and yields the continuous-time ELBO.\nThis CT-ELBO is then compared with its discrete time counter part. Finally it is compared to the score matching loss which demonstrates the soundness of such method for learning the generative model and continuous time diffusion models.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nTo the best of my knowledge this paper present completely new results. The approach to derive the different expression is novel as well. Overall I think the proposed CT-ELBO could lead to many interesting future works for variation of the current continuous diffusion models.\n\nQuality:\nI have to admit that my background and the time I was able to spend on this review does not let me judge accurately the quality of the methods presented in this paper. However I think the methodology proposed by the authors to express the CT-ELBO is very nice and I don’t see fundamental flaws in those steps.\nI think the authors could have expressed the role of lambda explicitly in the manuscript as it is discussed later in the experiments. I also believe authors should have discussed more thoroughly why the claims in Durkan and Song are not accurate.\n\nClarity:\nOverall I think the paper is quite clear and make a really great at presenting complex materials into a comprehensive way.  However I also have to admit section 6 was a bit harder to follow on my side (except the first paragraph) and could maybe be improved.\n\nSignificance:\nThis work is very important as 1) It provides formal results for using plug-in score matching in diffusion generative models. 2) It provides the tools for creating new SDE based generative models.\n\nAdditional remarks:\n• EQ 13: ds missing?\n• I did not get how the first part of 15 disappear into 16.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "/", "review_text": "", "extra_scores": {"ethical_concerns": "/", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "4I6QcjkqH8c", "reviewer": "Reviewer_99PV", "summary": "The authors derive a general variational formulation for likelihood estimation in diffusion based generative modeling and draws an equivalence relation between maximising the associated continuous time (CT) ELBO and minimizing specific score-matching losses.\n\nThe authors also provide a more formal interpretation to “diffusion models can be viewed as a continuous limit of hierarchical VAEs” and interesting commentary on bias/ variance trade-off for weighted losses in score-matching.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although previous attempts have been made to connect score-matching and likelihood objectives, this paper provides stronger results with more insight.\nFormal interpretation to “diffusion models can be viewed as a continuous limit of hierarchical VAEs” is novel, though incremental by itself.\nThe results appear correct. Although not the focus of this work, experiments are limited however, see below.\nWork is clear, though perhaps poorly organised. Perhaps re-stating existing theorems (Girsanov/ Feynman Kac) is unnecessary and the excess space may be used to provide more discussion on the paper’s contributions.\nThis work provides insights into score-matching losses used in practice and the link to likelihood, I believe this will be used and built upon by researchers.\n\nStrengths:\nDerive novel CT-ELBO and connect score matching to ELBO in continuous time. This is a general ELBO and allows comparison for various ODE and SDE reverse diffusion generative models. \nUnlike previous work [2], the derived connection to specific score-matching losses is exact rather than an upper bound and provides insight into how various score-matching losses used in practice relate to likelihood and family of plug-in reverse SDEs. This formulation is elegant and provides better understanding. \nMaximising ELBO in continuous time using ODE/ SDE integrators is computationally prohibitive, this paper derives an equivalency to a more scalable, albeit noisier approach using score-matching. I believe this will be of interest to practitioners as it gives a better understanding of score matching losses used in practice.\nThe authors formally confirm that limit of likelihood of discretized diffusion (i.e. infinite hierarchical VAE) matches continuous time likelihood\nInteresting discussion on bias-variance trade-off via weighting scheme in score matching losses\n\nWeaknesses:\nThe authors spend a significant proportion of the paper stating existing theorems (Girsanov, Keynman Kac representation), yet leave original Theorem 5 and discussion on plug-in reverse SDEs to the appendices. \nSection 6 discussing the implications of main theoretical contribution is overly brief. \nAlthough the bias/ variance trade-off from weighting is explored for images MNIST/ CelebA, the main contribution regarding likelihood based learning is only verified on low dimensional Swiss Roll data. \nNo experiments comparing numerical SDE integrators, just ODE. \n\nMinor:\nI do not think Durkan [2] claims score matching is equivalent to max likelihood as you state in line 195, just that score matching upper bounds likelihood and is a proxy.\nMissing references:\nReverse SDE [1] (for instance)\nNotation undefined:\nHessian H_p not defined in equation (7) line 88\n\n[1] U. G. Haussmann. E. Pardoux. \"Time Reversal of Diffusions.\" Ann. Probab. 14 (4) 1188 - 1205, October, 1986. \n[2] Durkan, Conor, and Yang Song. \"On maximum likelihood training of score-based generative models.\" \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limited societal impact, this paper is mainly theoretical. ", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "15"}, "extra_text": {}}, {"review_id": "0YWKHgWLWxS", "reviewer": "Reviewer_xxp9", "summary": "The paper develops a variational framework for general diffusion processes. A continuous-time ELBO is developed that extends the ELBO of discrete-time diffusion processes. They use this to show that optimizing the score matching loss corresponds to maximizing a lower bound on the log marginal likelihood.\n\nMore specifically, the paper makes use of the Fokker-Planck/Kolmogorov forward equation along with the Feynman-Kac formula to express the marginal density of the generative diffusion process. Since the resulting expression is intractable, a change-of-measure is applied, introducing a variational approximation $\\mathbb{Q}$ to $\\mathbb{P}$. $\\mathbb{Q}$ is chosen such that the Radon-Nikodym derivative $\\frac{d\\mathbb{Q}}{d\\mathbb{P}}$ can be expressed using Girsanov theorem. From this, a continuous-time ELBO is obtained. Finally, through a specific choice of the drift coefficient $a$ in the inference SDE, a connection between the ELBO and score matching is established.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides a variational framework for general continuous-time diffusion processes, which to the best of my knowledge is novel. The continuous-time ELBO is shown to consistently extend the usual ELBO for discrete-time diffusion processes, which is very elegant. Further for a specific choice of the inference SDE, the ELBO is shown to correspond (up to a constant) to a score-matching loss with a specific weighting ($\\Lambda = gg^T$). This provides a connection between the training of continuous-time score-based diffusion models [1] and maximum likelihood training, concurrent with [2] which uses a different approach.\n\nFor the negatives: The experimental results are quite poorly presented. The plots in Fig. 3 and 4 and difficult to read. I would suggest 1) zooming in and 2) considering a log scaling also of the x-axis. Furthermore, a proper discussion of the results is missing (especially for MNIST/CIFAR-10), making it more difficult for the reader to get the take-aways from the experiments. Finally, details on the experiment setups appear to be missing.\n\nThis is a technically strong paper with somewhat heavy, but elegant theory. It adds to our theoretical understanding of score-based diffusion models and opens doors for further theoretical and experimental work. I would therefore like to see this paper accepted.\n\nMinor questions/comments:\n- A reference or appendix section on the relations in Eq. (3) would be useful.\n- Line 71: You refer to $\\mathcal{I}(q) = \\mathbb{E}\\left[ \\| \\nabla \\log q \\|_{\\Lambda}^2 \\right]$ as the Fisher information. Isn't the Fisher information commonly defined as the matrix $\\mathcal{I}(q) = \\mathbb{E}\\left[ (\\nabla \\log q)(\\nabla \\log q)^T \\right]$?\n\n[1] Score-based generative modeling through stochastic differential equations (Song et al. 2020)\n[2] Maximum Likelihood Training of Score-Based Diffusion Models (Song et al. 2021)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In the checklist, the authors claim to describe limitations of their work, but with no reference to where. A dedicated paragraph to this could be beneficial. Potential negative societal impacts are not discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main pros and cons that came up during reviews and discussions are:

pros:
* excellent paper with a elegant approach (k6w1, dz2y, xxp9, 99PV)
* new results opening up many interesting future works (dz2y, xxp9)


cons:
* framing in the context of prior work can be improved. (k6w1, dz2y)
* The advantages and disadvantages of proposed bound over other bounds could be clarified.  (k6w1)
* notation could be clarified. (k6w1)
* Section 6 can be improved in terms of clarity and can be expanded.(dz2y, 99PV)
* presentation of results can be improved. figures are hard to read and a discussion is missing. (xxp9)
* limited experimental results, although these are not the focus of the paper. (99PV)
* paper re-states too many existing theorems. Authors disagree, state it is important for a self-contained paper. (99PV) 

The authors and reviewers engaged in discussions during the discussion period and most of the concerns raised by the reviewers were addressed. 3/5 reviewers raised their scores, leading to a unanimous recommendation of acceptance. The recommended decision by the meta-reviewer is to accept the paper.
Decision: Accept (Spotlight)