Score-based Generative Modeling Secretly Minimizes the Wasserstein Distance
Authors: Dohyun Kwon
Ying Fan
Kangwook Lee
Venue: NeurIPS 2022 Accept (2022)
Keywords: score-based generative models
optimal transport
Wasserstein distance

Abstract: Score-based generative models are shown to achieve remarkable empirical performances in various applications such as image generation and audio synthesis. However, a theoretical understanding of score-based diffusion models is still incomplete. Recently, Song et al. showed that the training objective of score-based generative models is equivalent to minimizing the Kullback-Leibler divergence of the generated distribution from the data distribution. In this work, we show that score-based models also minimize the Wasserstein distance between them. Specifically, we prove that the Wasserstein distance is upper bounded by the square root of the objective function up to multiplicative constants and a fixed constant offset. Our proof is based on a novel application of the theory of optimal transport, which can be of independent interest to the society. Our numerical experiments support our findings. By analyzing our upper bounds, we provide a few techniques to obtain tighter upper bounds. 

Score-based Generative Modeling Secretly Minimizes the Wasserstein Distance
Dohyun Kwon, Ying Fan, Kangwook Lee University of Wisconsin-Madiscs
Abstract
Score-based generative models are shown to achieve remarkable empirical per- formances in various applications such as image generation and audio synthesis. However, theoretical understanding of score-based diffusion models is still in- complete. Recently, Song et al. showed that the training objective of score-based generative models is equivalent to minimizing the Kullback-Leibler divergence of the generated distribution from the data distribution. In this work, we show that score-based models also minimize the Wasserstein distance between them under suitable assumptions on the model. Specifically, we prove that the Wasserstein distance upper bounded by the square root of the objective function up to mul- tiplicative constants and fixed constant offset. Our proof is based on a novel application of the theory of optimal transport, which can be of independent interest to the society. Our numerical experiments support our findings. By analyzing our upper bounds, we provide a few techniques to obtain tighter upper bounds.
Introduction
The idea of gradually perturbing the data distribution to define the forward process and learning the reverse process has recently demonstrated remarkable performances generative modeling. Score-based generative models [33 34 35) and diffusion probabilistic models [30 17 have been
used numerous applications, including image generation 226 12 and audio synthesis 21].
Both model families have the following two processes. The given data distribution is incrementally perturbed by adding small noise in the forward process. This forward process can be understood as the standard diffusion process (35 The reverse process requires the so-called score function (2]. the gradient of the log probability density function. We minimize weighted combination score matching losses to approximate the score function. Once the score function is approximated, one can
generate samples from randomly generated noise by applying learned reverse process.
Note that score-based models do not explicitly minimize any sort of similarity between the model distribution (the distribution of the generated samples) and the data distribution. Therefore, is unclear whether or not scored-based generative models can minimize divergence (or distance) between the model distribution and the data distribution. In very recent work [32], the authors show that optimizing the score matching loss is indeed equivalent to minimizing -Lebler divergence (KL divergence). Therefore, is expected that the model distribution similar (in the sense of the KL divergence) to the data distribution if the score matching loss is small.
upper bound on the
natural question follows do score-based models minimize any other divergence or distance than the KL divergence? ? At first glance, this seems unlikely. For instance, Generative Adversarial Networks (GANs) [6] were shown minimize different divergences depending on the choice of the loss function. The original GAN loss function turns out to capture the
Emails: dkwon7 @ wisc.edu, yfan87 @ wiu, kangwook.eee@wisc.ell
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

ittedd 6i 93
0.5
125
644
180
0.4
02
2.50
0.5
0.00
Eoochs
(a) 1 cluster in ID
(b) 2 clusters in ID
(c) 1 cluster in 2D
(d) 4 clusters in 2D
Figure I: Training curves of loss function JDSM (eq. (20). Wasserstein distance, and KL divergence
during training. The training data is sampled from mixture of ID and 2D Gaussian clusters. (a): samples from one cluster (V
(b): samples from two clusters (O 0.1) and (0.55, 0.1) with equal weights. (c): samples from one cluster/V(C 0.010..0I. (d): samples from four clusters (((00.5, +0.5) 0.01I) with equal weights.
divergence [16. while the W-GAN loss function captures the Wasserstein distance 3 We refer the readers to 10) for more details.
On the other hand, a unique aspect of score-based models is that the score function is defined in the same space where data lies in. In stark contrast, all the other popular generative models, such as GANs [16. VAEs [19, and Normalizing Flows (20), use loss functions that have nothing to do with the underlying data space.
To gain more insights, we train score-based models on few synthetic datasets. Shown in Fig. ] the training results. We plot the (scaled) training objectives in blue. Shown in green are the KL divergences, and one can note that the KL divergence decreases as the training objective decreases, as predicted in [32]. What we also observe that as the training objective decreases, the Wasserstein distance also decreases. This is quite surprising as there is no direct relationship between the KL divergence and the Wasserstein distance, general. Therefore, this phenomenon is not immediately implied by the fact that score-based models minimize the KI. divergence.
Inspired by this observation, in this work, prove the following property of score- score-based modeling:
Score-based modeling minimizes the Wasserstein distance.
More specifically, using the theory of optimal transport, we investigate the evolution of data distri- bution based on partial differential equations (PDEs). This allows us to estimate the Wasserstein distance between the given data distribution and the distribution of the generated samples. Our work theoretically guarantees that the score-based generative models approximate data distributions in the space of probability measures equipped with the Wasserstein distance.
Our main contributions and findings are summarized follows:
(Theorem T & Corollary D Using the theory of optimal transport, we prove that the Wasserstein distance between the data distribution and the model distribution is bounded from above. The upper bound is given as the square root of the score-matching loss JsM up to multiplicative constants and fixed constant offset. The main idea is to estimate the time-derivative of the Wasserstein metric by using the form of the continuity equations.
(Theorem 2 We prove Jsas S JDSM (as defined in eq. (5) and eq. (0). respectively) under suitable assumptions, which gives estimate in practice and is widely used as a loss function to train score-based based models.
more practical upper bound since JDSM is casier to
We analyze how the model distribution changes if the data distribution is perturbed initially. This guarantees that the score based-mode is robust under noise.
(Theorem 3)
(Section 4 We corroborate our analysis by conducting numerical experiments. We also explore various techniques tighten up the upper bound in practice.
== (21,22, xZ) E IRd and E [0, denote the spatial variable and the time variable, respectively. Here, Rd is the d-dimensional Euclidean space and > 0 is the time horizon. The gradient of a real-valued function p with respect to the spatial variable and the time-derivative
Notations:
of p are denoted by Vp- =
Op Up Sma 021
Up
and Oep, respectively. The Laplacian of is
given as Ap- V (V)). Here, V
aF
ar
aF denotes the divergence of Dzg

---

= (2,, ... Fa) with respect to the spatial variable r. The Hessian of p is d x d matrix 8p
given by (D²p)is 0z, for 1 S i.j I d. We indicate Gaussian distribution with mean μ and covariance E ((,, Σ), and L2 norm by || ||. We denote the data distribution and the distribution of the generated samples by Po and qo-
2 Preliminaries and related works
In this section, we analyze the forward process and the reverse process in the score-based model through Partial differential equations (PDEs). In particular, we investigate the time evolution of marginal distributions through the two processes.
2.1 The forward process
As in [35,, the forward process be modeled by the standard diffusion process {z(t)){= for t E [0, [] such that z0) ~ po and
dx = f(x, t) + g())dw.
(1)
Here, w is the standard Wiener process, f: IR [0, IRd is the drift coefficient of x(t), and g: [0, 7 + R is the diffusion coefficient of (=))).
pr p(-()) be probability density function of the random variable sr() Rd time E [0, T]. Then, the function p Rd [0, 7] + [0, +o0) satisfies the following partial differential equations, namely the Fokker-Planck equation or the forward Kolmogorov equation as 5 Theorem 2.2]:
Oep + V (pf) g² Ap 0, p(:,0) Po-
where Po represents the given data distribution. Under suitable assumptions on f and g. solution to the above initial value problem uniquely exists. We refer the readers to (6 for more details.
Understanding the reverse process through PDEs
To generate samples from t = T to 0, the score-based model uses the reverse-time time SDE 2
dx = (f t) g(t)²v log p)dt + g(f)dw
(3)
Here, w is standard Wiener process when time flows backward from t = T to t = 0.
As in the previous section, for given xTT Pr, a probability density function of the random variable (t) from the above process 3) satisfies the final value problem:
g²
Oap + V (p (f g²V logp)) + Ap O, p(-TT) PT.
(4)
2
Using this simple identity, Ap (Vp) = V (p = V (pV(logg)) it can be seen that the differential equations in 2 and are indeed identical.
2.3 The objective function
The score of the probability distribution, V log Pt, is approximated by a function se : gd x [0, 7] + Rd For instance, in [32, the authors consider the weighted mean squared error (MSE):
/"
JSM (0; = //1 A(t)Bp, [[ (l so(a, t)] di
(5)
where x: [0, >> (0) is positive weighting function.
Replacing V log (4) with sg. we deduce a new reverse process. For given noise distribution gr,
probability density function g. = q(.,t) t) at time E [0, 7 solves
9² Ag O, q(T,-) qr.
0.g + (q (f (²se)) +
(6)
2
Here, we introduced new function g Rd [
+ [0. +00) to illustrate the reverse process.
3

---

2.4 Related works
Optimal transport The Wasserstein distance has a solid connection to the Fokker-Planck equation. In particular, the Fokker-Planck equation can be understood as the gradient flow in the space of probability measures equipped with the 2- 2-Wasserstein distance [ 18 277. For two probability distance
measures i and on R, we define the 2- Wasserstein
W)(u, = inf
|1 v1l : E M(,,))
v)
Here, [[ 2/) is the set of all couplings of μ and where a Borel probability measure π on IR x R satisfies T (AA x TRd) p (A) and (R. B) BB) for all measurable subsets A, B C Rd.
Under suitable assumptions on the drift coefficient and the diffusion coefficient, a solution to the
Fokker-Planck equation satisfies the so-called Wasserstein contraction property 7
8 29], which
plays an important role in our analysis. More discussions can found in Section 3.4 and Section
Score matching and diffusion probabilistic models Score matching models B3 use the score function to reverse the noise corruption in continuous time range by solving stochastic differential equations (SDEs). Diffusion probabilistic models [30 [7] adopt similar idea of modeling the tractable denoising process reverse each discrete step of noise corruption with probabilistic models. In fact, 355 show that diffusion models are score matching models with certain forms of SDE, which unifies the whole framework. Diffusion models are gaining increasing attention: recent advances 31 26 [2] show that diffusion models can generate even better images than GANs while enjoying more stable training process, which encourages it to be widely applied in various tasks, such as text-to-image and image editing [25 24, audio generation 9 21], etc.
3
Approximation in the Wasserstein space
In this section, we estimate the Wasserstein distance between a given data distribution pa and the marginal of the generated samples du obtained by the score-bar based model. The proofs of the main results are deferred to Appendices A B and C The key ideas are illustrated in Section
3.1 Assumptions
We present our main results under the following assumptions.
(A1) The drift coefficient f Rd [0, T] R is Lipschitz continuous in the space variable there exists positive constant (t) (0, oo), depending on t [0, ('. such that for all x, y E Rd
[lf(z,t) ((y,t)l (t)l vll-
(8)
(A2) se : Rd x [0, + RRd satisfies the one-sided Lipschitz condition B
Definition 2.1]: there
exists a constant (t) IR, depending on E [0, []. satisfying for all x, y E Rdd
(sa(z,t) so(y. f)) (x y) L. (t)llx y11².
(9)
The usual Lipschitz constant of so satisfies the above inequality 9 Unlike the usual Lipschitz constant, the one-sided Lipschitz constant (t)( is not necessarily nonnegative (see Appendix D
(A3) For the initial distribution Po of (2) and the final distribution fr of (6). the following quantities finite:
Ep, log Poll]. Ep. [A()]] Egg [I] log Irlll. and Egr [A())]
(10)
where A(x) := log max max 1).
(A4) There exist constants C₁ and C2 such that
f(x, t) S C| + O2 for all x e Id and t E [0, T].
(11)

---

(A5) There exists positive constant C3 such that
C3 < g(t) < C3s for all t e [o,
(12)
(A6) Jo Ep. [/ and So' By |C وووو Jdl finite.
The above assumptions guarantee the integrability of the score function logp.
Proposition
16 Theorem 4.2. A solution Pt to satisfies Vpt E L (IRA) and
% Ep. (o) dt t00.
(13)
We give the proofs in smooth settings.
(A7) Po. qr E C² (RR)), and g E ((O, and se are C in both R² and E [0,
(A8) A solution pt to 2 and a solution to (6) are positive C probability densities decaying rapidly
in RR there exists & 0 such that pe (z) [0, T.. Furthermore, the vector fields v(pt] f- 2 log and vlur = (J -g"sa) 2 log qr
(ccxp([[==|[))) and qı(x) -((xp(--||=)) -==[|==|)))) for all
grow at most linearly as || + c0.
3.2 The main results
We now present a general result.
Theorem 1. Let po and qo be the data distribution and a pdf obtained by the score-buseds model as in Section 2. Then the Wasserstein distance between po and qo is bounded as follows:
W2(po. 9o) S ( (())) [[[V logg)(((
- 50(x,0)13] de + ((T))2(pr,,n)) (r). (14)
Here, a function 1: [00,7] + (0, +o0) defined by
T(t) exp (J, (1+(0)++ dr)
(15)
an integrating factor of the differential inequality (28) given later in Proposition The main idea of the proofs is to estimate the derivatives of the Wasserstein distance along with the time evolution, which is illustrated in Section
The first term on the right-hand side of (14) naturally arises from the estimation of W2(Po, qo)- It is similar to loss function JSM commonly used in the score-based models. Applying the Cauchy-Schwar inequality, we obtain the upper bound in terms of the weighted MSE JSM-
If X(t) = g(t) for all t E [0,T], then the Corollary 1. Let po and qu be given in Theorem 1 following inequality holds:
W2(po. go) S / (J. ((t)2I(t))dt JSM + I(T))
(16)
We defer the proofs of Theorem [ and Corollary to Appendix
Remark 1. From the above result, the Wasserstein distance converges to zero the loss function
JsM goes to zero if pr = qr. However, due to the tension between I(T) and Wz(pr. qr), a general convergence result remains open (see Section Gand Appendix B).
Remark 2. 12x-1 is integrable in [0, T), the similar inequality as in 34 holds for other choices of X. See Corollary 4 in Appendix A
Remark Note that the above upper bound depends on the time horizon Numerical experiments
for different choices of T are provided in Section 4.3
5

---

We further investigate the offset (2 Denoising Diffusion Probabilistic Models (DDPM) 17: for 8 [, + (0, +oo). and
(Pr. Qr) in Corollary I Consider the following case as in
f(x () = 8(1)
I and g(1) = av 3())
(17)
In the above case, the stationary solution to 2 given as the probability distribution of (0 I2):
|
((x) V
exp
(18)
2x
Corollary 2. Under the same setting in Corollary
let QT Φ. For f.g given in 17 we have
W2(po, g0) - 2 (L. B(t)o2I(t) JsM + exp (L" B(t)o² W2 (Po. Φ). (19)
The proof of Corollary and the analysis of L, can be found in Appendix Remark If Sy is close enough to FlggPr (x)). then we have
(t) (-o + ( h lloo)
Here, hu ==po/ó exponentially converges to I as oo (see Lemma sufficiently large t 0, L, converges to -1, as observed in Fig. 4a term in 19 decays T increases (see Fig. 4b
3 As consequence, for
This yields that the second
3.3 The relation between JSM and JDSM
The above results give the guarantee that JsM upper-bound Wasserstein distance. However, JsM generally intractable due to the unknown quantity V. log (x) [32], and is rarely used in practice. Alternatively, there is a conditional version of the score-matching loss JDSM. which is equivalent to JSM up to a constant, as shown in 36 JDSM is also widely used as a loss function to train score-based modcls:
JDS (0, = - /
T
X(1) (cee
V log Por (se
(20)
where Po: indicates the conditional probability of x at step given (0)) Now we present our new result, which shows that JDSM is also an upper bound of JSM under certain assumption.
Theorem 2. If por satisfies
(Vr [cx = 0,
(21)
then we have
{f[ 0(0))1()))))) JDSM + ((T)Wa(pT.r))
JSM S JDSM and W2(po, qo) s
3
(22)
It is worth noting that Denoising Diffusion Probabilistic Models (DDPM) 17 satisfy 21
Corollary 3. Let p(æ(t)|z((- -ore(t-))) 1)) N(VI- 3,z(t - 1), B,1) in Denoising Diffusion Prob- abilistic Models (DDPM) (I7], which corresponds to f - Brx and g(t) = vB.. Here,
3t B(1) 0 is the noise schedule for all I, and let A(t) 2 Then, (22) holds.
The proofs of Theorem 2 and Corollary can be found in A Appendix results which show JSM & JDSM for DDPM in Appendix F
C We also provide numerical
Remark S. Using Theorem 2 and Corollary 3 can present upper bound of the Wasserstein distance terms of JDSM, where both Wa (po, q0) and JDSM are tractable. This enables us to empirically verify this upper bound via DDPM in Section 4

---

3.4 Perturbation and sensitivity analysis
Let Po be a small perturbation of the original data distribution Po- For given corrupted distribution Po. denote the probability density functions in the forward process and the reverse process by Pr and It respectively. We investigate how this perturbation changes the score-based model. particular, we are interested in the upper bound on (2 ÿo) with respect to (2
It is well known from the Wasserstein contraction property [8] that the Wasserstein distance between two probability distributions in the forward process can be estimated as follows: for all t E [0, 7]
W W2 pc) S exp L((r)ar) W2(Po. Po).
(23)
In the reverse process, the score function in the drift coefficients may vary, and thus the above inequality does not directly give us the upper bound on (2 ()). Instead, our analysis in the previous sections allows us estimate W2( Qo).
is worth noting that the Wasserstein distance satisfies the triangle inequality, unlike other statistical measures, including the Kullback-Leibler Leibler (KL) divergence. Combining main results with
W2(go- 10) W2(po.po) po) + W2(po, go) W2(0o,9o):
conclude the following theorem.
Theorem 3. Under the same setting in Corollary D let Po be the corrupted data distribution and to a probability density function obtuined by the score-based model starting from Po- It holds that
W2(0o; 40) S W2(Po. Po) + V ((" ((1)T((PP
Jsas + + ( ((0))(t))))))
Jsm
((T))a(Pr, 2r) + (fr)).
(24)
Here, JsM and i denote the weighted MSE and the integrating factor from the perturbed model. Remark 6. The Wasserstein distance between the original data distribution po and the model distribution to starting from the corrupted data Pu can also be estimated similarly.
4
Experiments
Here we present experiments on simulation datasets to verify the upper bound in eq. www.ersseeeaaeeeeiennnnn.e
22 Code is
available at httts: com/ com Mon
4.1 Experiment settings
Datasets Here we adopt three 2D datasets for simulation: One cluster Gaussian A ((, 0.1I), two moons in (28) and four clusters Gaussian mixture (..5. +.0.) 0.01I) with equal weights for each cluster. Visualization for these datasets is Fig 2
For forward dynamics, we use discrete timesteps with T = 10. For E [1, 10). p(x(t)=((( - 1)) 3,x and g(t) vBr. For Br,
M(V1 - Bt 1), 8,4) as in DDPM [17], i.c., f(z
adopt sigmoid schedule from 31 10 to Br 10-2.
Training and evaluation We use 4-layer neural network as the score matching model, with ReLU nonlinearity and skip-connection at the final output. Each layer composed of linear layer with 64 hidden neurons and an embedding layer for 10 timesteps. For optimizer, we use Adam W [22] with learning rate 0.001 and weight decay coefficient 0.01. To evaluate Wasserstein distance, we use POT [1] to compute W2(Po. qu) at the end of each training epoch. To avoid the term of (2 qr) eq. (22) for verification, we first perform forward pass on sampled z(0) to get =(T), and perform data generation on the same x(T') to fix qr pr and W2(PT, (T) O. For loss function, we use 128. To get both lower and higher losses to verify our JDSM with X(t) = g() and batch size =
bound, we first maximize the loss for 10 epochs and then minimize the loss till convergence.
7

---

4.2
Results
4.2.1
The upper bound
To see the upper bound in eq.
22 we take the log on each side of the equation with W2 qr) 0:
log (L" ((0))100))))
log W2 (2 g0)
log JDSM
(25)
2
2
skipe
intercepi
We present a log-log plot to verify the slope and the intercept in the upper bound respectively while fixing Pr gr see F The log-log plots are obtained by evaluating Wasserstein distance and the loss function during training. The red dots are the sampled data from each dataset, the green dots are generated data after loss maximization, and the purple dots are the generated data sampled at convergence. From the log-log plots, we can observe that the theoretical line (the black line) is valid upper bound of the empirical one (the blue dots).
Oo (start)
log-log plat
Oo (start)
log. loglloggglll plat
O: (start)
log-log plat
Upper bound Erriinical
Upper bound Empincall
Upper bound Emriinccl
convergerice)
convvrr perice)
converr genic))
Reqwin
Pg
2
(og(icw)
(nglisw)
(a)
(b)
(c)
Figure 2: Log-log plots of the loss function and Wasserstein distance, with the corresponding generated data at the start of the loss minimization and at convergence for each dataset. The darkness of the data points corresponds to training epochs: the darker ones are closer to convergence.
4.2.2 Effeet of weight regularization
Here we explore weight constraints that can affect the tightness the theoretical upper bound while fixing Pr 4r- As shown in Fig 2 the gap between the theoretical bound and the empirical one
mainly results from the intercept in eq. 25 Recall our intercept is 1 log (5o I(t)2 for [(t)
given in (5)). Since gt)) and L, are fixed, only the one-sided Lipchitz constant (r) of the model so is subject the training process, which is estimated via grid search in experiments.
Notice that the loss function JDSM aims to minimize the distance between s0 t) and V_ Pog
Poe (r (0)). Thus, L, at convergence is determined by dynamics V. log Por (0). If the one-sided Lipschitz constant for V, poe(poo((()))) is small, then the intercept would be ex- pected to be small (see example in Fig 2 of Gaussian cluster with moderately high variance). If the one-sided Lipschitz constant for V log Por (z(=(()) is large, to minimize the loss, the intercept would also be large (see example in Fig 2 of Gaussian clusters with lower variance). However, it can be possible to make the upper bound tighter while sacrificing the training loss by constraining L..
To constrain Lg, we adopt spectral normalization 4 weight clipping with threshold = 0.1 As we can observe in Fig. loss at convergence will increase when we impose very strong regularization on the weights while the intercept decreases. Importantly, the intercept approaches the empirical one without not exceeding it. However, the regularization too strong, the training loss at convergence would also become larger, resulting in worse quality of the generated data. See more examples of weight decay Appendix G where we also present the log-log training plots with different weight decay coefficients.
8

---

Upper bound Empirical
Upper bound Empirical
Upper bound Empirical
roghtt M00
4
2
-40
2
4
-4
4
2
log()ss)
logUess)
log(lss)
(a) Vanilla
(b) Spectral normalization
(c) Weight clipping
Figure Effect of weight clipping and spectral normalization.
4.3 Effect of the number total time steps: the decay of the offset
In our Remark 4 we argue that the offset in our upper bound would decay to when choosing the appropriate forward SDE and sufficiently large T. Here we provide empirical evidence for our claim without fixing Pr gr. We use the four clusters Gaussian dataset and fix
be a sigmoid schedule in the range of [10- 10 for all choices of T. In Fig. 4 we plot L, (t) for T = 100 to show that (t) would converge to when I + T. Moreover, since W2(PT, (r) decays exponentially (see eq. 39 we also show that T increases, our offset (( (PT, (r) would also decay in rate that approximately exponential when sufficiently large, as the model converges the actual data distribution (W2(po. go) 0).
Lt(t)
KT) 4r) W(po, qn)
-1
3-
2
40
20
0-
-1
0
60
70
80
90
100
50
100
150
200
(a) L. for T' 100. We verify that when T as suggested in
(b) W2(r, 9r) and W2(pa, go) for T 50, 100, 150, 200 respectively at the convergence Remark 4 (t) when S 50 is cut to visibility of of training. while 3 to be sigmoid schedule from
sufficiently large, (T'))
the threshold and -1. The full figure is in Appendix
81 = 10 to Br = 10-2 for all choices of T.
Figure Effect of T. We can observe that (T) would converge 1 and ((T) 9r) would decay almost exponentially when T is sufficiently large (2 100).
The main idea of the proofs
We investigate the time evolution of probability measures arising in the score-based models. Using the optimal transport theory, we differentiate the Wasserstein distances, W2(P:, 4:) with respect to
time. The main idea is to express these equations ( and (6) in the form of the continuity equation:
Opp + V (pv)
(26)

---

where u is a so-called velocity field. Let us denote the velocity fields of p, and by
1 =2 V logp, and vlq] (S g 2
u|p] =
logg-
Then, the equations 2 and (6 can be written in the form of (26 with v epp] and e(g). respectively.
Using these vector fields, we compute the time derivative of the Wasserstein distance between two evolving probability measures. For velocity fields v(p(o)] depending on p!t), let pl)) for i = 1,2 be two solutions of the continuity equations
0ip V. (p u(p
Then, the relation between the Wasserstein distance and velocity fields arises from the theory of optimal transport:
d
(p: P 22) Er,, (x y) u(pp
(27)
where Ty is optimal transport plan between of and p!.h, See [I] Theorem 8.4.7] and 22 Corollary 5.25]. This equality plays an important role when we estimate the Wasserstein distance. The following proposition is consequence of (27) and Lemma i Appendix
Proposition 2. Let and be solutions to (2) and 6). respectively, in [), Then,
d W2 W2 91) s (Ly + L.g")) 9:) 9
(28)
dt
where b(t) E., [1 [ (o))((x) so(2r, 1)
6 Discussions
Convergence of the Wasserstein distance If = Qr, then our result yields that the Wasserstein Jsw goes to zero. However, if pr is not equal gr and I and are not explicitly given, then we find that it is not easy to obtain such a convergence result. This is due the nature of the Langevin dynamics in the reverse process. Typically, additional assumptions such as the log-concavity are required to guarantee the Langevin dynamics convergence, for instance, [. In addition, the dependence of the neural network sø makes it hard evaluate the precise upper bound. We expect that appropriate assumptions on so and the initial distribution Po may resolve this issue, but we do not pursue them this paper.
distance (Po 9o) converges
Estimating I. Estimating our theoretical bound requires estimating the one-sided Lipschitz con- stant Lg. which NP-hard (37), making it difficult estimate in high dimensional data. Also, estimating the one-sided Lipschitz constant does not enjoy the semi-definiteness of estimating the two-sided one in [4]. Note that the two-sided Lipschitz constant an upper bound of the one-sided one, which can be more efficiently estimated as shown in [14]. However, it would result in a looser theoretical upper bound if we adopt the two-sided Lipschitz constant.
7 Conclusion
In this paper, we mainly discuss the relationship between the loss function of score-based models and the Wasserstein distance. With the help of the theory optimal transport, we present novel upper bound on Wasserstein distance in terms of the loss function used in practice. We also provide numerical experiments to verify our theoretical bound with various simulation datasets. Moreover, we explore factors that affect the estimation of the upper bound and show the corresponding influence on the training process.
Acknowledgments and Disclosure of Funding
Support for this research was provided by the University of Wisconsin-Madison Office of the Vice Chancellor for Research and Graduate Education with funding from the Wisconsin Alumni Research Foundation, and NSF Award DMS-2023239.
10

---

References
[1] L. Ambrosio, N. Gigli, and G. Savaré. Gradient flows in metric spaces and in the space of probability measures. Second edition. Lecture notes in Mathematics ETH Zürich. Birkhäuser Verlag, Bassel, (2008).
[2] B. D. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, ((3):313-226, 1982.
[3] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
[4] J. Behrmann, W. Grathwohl, R. T. Chen, D. Duvenaud, and J.-H. Jacobsen. Invertible residual networks. International Conference on Machine Learning, pages 573-582. PMLR, 2019.
[5] R. N. Bhattacharya and E. C. Waymire. Stochastic processes with applications. SIAM, 2009.
[6] V. Bogachev, N. V. Krylov, M. Röckner, and V. Shaposhnikov. Tokker-Plancke Kolnogooww Equations, volume 207. American Mathematical Soc., 2015.
[7] Bolley and A. Carrillo. Nonlinear diffusion: geodesic convexity is equivalent to wasserstein contraction. Communications in Partial Differential Equations, 39(1)): (000088001888000 2014.
[8] A. Carrillo, R. J. McCann, and Villani. Contractions the 2-wassersteinn length space and thermalization of granular media. Archive for Rational Mechanics and Analysis, 179(2):217- 263, 2006.
[9] N. Chen, Y. Zhang. H. Zen, R. J. Weiss, M. Norouzi, and W. Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.007 00000900073,... 2020.
[10] I. Csiszár and P. C. Shields. Information theory and statistics: A tutorial. 2004.
[11] A. S. Dalalyan. Theoretical guarantees for approximate sampling from smooth and log-concave densities. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 79()):651-
676. 2017.
[12] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
[13] T. Donchev and E. Farkhi. Stability and euler approximation of one-sided lipschitz differential inclusions. SIAM journal control and opfimization, 36(2):780-796, 1998.
[14] M. Fazlyab, A. Robey, H. Hassani, M. Morari, and G. Pappas. Efficient and accurate estimation of lipschitz constants for deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[15] R. Flamary, N. Courty, A. Gramfort, M. Z. Alaya, A. Boisbumon, S. Chambon, L. Chapel, A. Corenflos, K. Fatras, N. Fournier, L. Gautheron, N. T. Gayraud, H. Janati, A. Rukotomamonjy. I. Redko, A. Rolet, A. Schutz, V. Seguy, D. J. Sutherland, R. Tavenard, A. Tong, and T. Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021.
[16] I. Goodfellow, J. Pouget-Abadis M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[17] J. Ho, A. Jain, and Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 2020.
[18] R. Jordan, D. Kinderlehrer, and F. Otto. The variational formulation of the fokker-Planck equation. SIAM journal Mathematical Analysis, 29(1):1-17, 1998.
[19] D. P. Kingma and M. Welling, Auto-encoding variational bayes. arXiv preprint arXiv: 141..6114, 2013.

---

[20] I. Kobyzev, J. Prince, and M. A. Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 43911):3964- 3979, 2020.
[21] M. W. Lam, J. Wang, D. Su, and D. Yu. Bddm: Bilateral denoising diffusion models for fast and high-quality speech synthesis. arXiv preprint 2tt.:2203./33066 2022.
[22] I. Loshchilov and F. Hutter. Fixing weight decay regularization in adam. 2018.
[23] A. Markowich and C. Villani. On the trend equilibrium for the fokker-planck equation: interplay between physics and functional analysis. Mat. Contemp, 19:1-29, 2000.
[24] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. International Conference on Learning Representations, 2021.
[25] A. Nichol, P. Dhariwal, A. Ramesh, Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.1074 2021.
[26] A. Q. Nichol and Dhariwal. Improved denoising diffusion probabilistic models. In Interna- tional Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[27] Otto. The geometry of dissipative evolution equations: the porous medium equation. 2001.
[28] Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 0122825-28300 2011.
[29] Santambrogio. Optimal Transport for Applied Mathematicians. Calculus Variations, PDEs, and Modeling. Birkhäuser, Oct. 2015.
[30] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning. pages 2256-2265. PMLR, 2015.
[31] Song. C. Meng, and Ermon. Denoising diffusion implicit models. arXiv preprint arXi::0010.02502, 2020.
[32] Y. Song, C. Durkan, I. Murray, and S. Ermon. Maximum likelihood training of score- based diffusion models. Advances in Neural Information Processing Systems, 1414111-128,, 2021.
[33] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances Neural Information Processing Systems, 32, 2019.
[34] Y. Song and S. Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438 13:44488-12488, 2020.
[35] Y. Song, Sohl- Sicl-Dickst,in, D. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representationss 2020.
[36] Vincent. connection between score matching and denoising autoencoders. Neural compu- tation, 23(7):1661-1672 2011.
[37] A. Virmaux and K. Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "pR7NfrlRMRL", "reviewer": "Reviewer_Y7JZ", "summary": "This is a theoretical paper on score-based generative models. The main result states that the Wasserstein distance between the data distribution and the generated distribution of such models is upper bounded by the square root of their objective function up to multiplicative constants and a fixed constant. This suggests that such models tend to minimize the Wasserstein distance between the target distribution and the estimated one, and complements another recent similar result on the Kullback-Leibler divergence [31]. Another theorem connecting this result to diffusion models is also provided. The findings are supported by numerical experiments on toy data (Gaussian mixtures).", "questions": "- The analysis is conducted in the continuous time domain, whereas in practice, diffusion model are always trained using a discrete, finite number of time step. A comment on how this theoretical analysis translates to the discrete setting would be welcome. \n- In a similar vein, it is not clear how the various continuous-time integrals were numerically calculated in practice by the authors in the various experiments.\n- The loss function defined in eq. (10) is not very well explained or motivated.\n- Please check the bibliography, it contains many mistakes. For instance, bibliographical reference [31] appears to be incorrect (it seems to include a long list of \"editors\"). Moreover, many bibliographical references are duplicate, eg., [31] and [32],  [33] and [34], [36] and [37]. \n\nTypos:\n- L29: the upper bound => an upper bound\n- L59: neither variables J_SM or J_DSM have been properly defined at this point of the paper\n- L94: from in the above => from the above\n- L269: makes us hard => makes it hard\n", "limitations": "The limitations of the proposed theoretical analysis are adequately addressed in Section 6.", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n- The paper provides a strong contribution towards a better theoretical understanding of the dramatic empirical success demonstrated by score-based and diffusion-based generative models in recent years.\n\n## Weakness\n- The paper is hard to follow for a reader who is not deeply familiar with PDE formulations of scored-based models, diffusion models, and Wasserstein metrics.\n\nUnfortunately I do not have this background nor the time to acquire it, and hence was not able to check the details of the proofs in the paper. My assessment of the paper is based on the assumption that the proofs are all correct.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iS9Zuu5tYmS", "reviewer": "Reviewer_v4Sv", "summary": "This article uses optimal transport theory to prove that score-based generative models not only minimize the Kullback-Leibler divergence but also the Wasserstein distance between the generated samples distribution and the data distribution. In addition, it establishes that the Wasserstein distance can be upper-bounded by the square root of the objective function (up to multiplicative and additive constants) for both score-based models and denoising diffusion models. \nThis paper then discusses the impact of weight decay and the number of timesteps $T$ on the upper bounds. Theoretical results are experimentally checked and illustrated on simple synthetic datasets. Beyond that, the article also discusses the robustness of score-based models to noise perturbation on the datasets using the Wasserstein distance.  ", "questions": "- The article discusses casts light on reducing the intercept $I$ by applying weight decay (since it allows to reduce the Lipschitz constant $L_s$). Considering that $I$ also depends on the SDE used through $L_f$ and $g^2$, did the authors study the effect of the choice of the SDE on the intercept value given a fixed dataset? \n- On figure 6 (Appendix E), we can notice that the denoising score matching loss does not decrease much during training. Can the authors explain why the observed gap is equal to the theoretical one and not the consequence of the neural network of the denoising score matching model underfitting the distribution? (since it is a very small network).\n", "limitations": "There is no direct potential negative societal impact of the article proposed in this article.\n\n[Edit] - I raised my score", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The results proved in this article bring new interpretations on the score-based models learning properties, which were, to the best of our knowledge, not known before. The results are clearly stated and discussed thanks to numerous remarks sections. \n- The article is well written and easy to follow, despite some paragraphs that could be made clearer. Previous works, both in the field of generative models and optimal transport, are referenced and well-integrated in the narration.\nNotations are clearly defined and respected, even if prior knowledge about the related works is sometimes needed. Also, note that objective functions $J_{SM}$ $J_{DSM}$ are referenced (line 59) before definitions, or that the variables with respect to which variance and expectation are taken in Eq. 15 should be precised.\n- The experiments and figures help with the comprehension of the article and provide illustrations of the theoretical statements.\n\n- The motivation of the article could be stated more clearly. Besides the theoretical result, the article could mention the benefits of minimizing the Wasserstein distance between the model and the data distributions.\n- My biggest concern is about the Experiments part: This is mostly a theoretical paper and the presented experiments look somewhat superficial or confusing. The fact that the upper bound of Eq. 19 depends on the training procedure is rather unusual. This could have been controlled before training by devising a dedicated architecture. There are numerous resources on this subject, so that controlling the network's Lipschitz constant only via weight regularization does not seem convincing in this case. Also, it would have been interesting to discuss if the upper bound of Eq. 19 could suggest new ways to select the forward SDE. In the paper, only the dependence on the network's Lipschitz constant $L_s$ is discussed.\nThe assumption $W_2(p_T, q_T)=0$ depends on the choice of the forward SDE and is only approximately true. Completely removing it from Eq. 19 does not show the balance between low Wasserstein distance at $T$ and small $I(t)$.\n- It is not very clear how the theoretical results of this article can motivate improvement in the case of score-based model training on more complex datasets in practice.\n- Some details could be added regarding the training and evaluation of the network: T=10 timesteps seems like a too small value compared to what is regularly used in the literature. The number of samples used to compute $W_2$ is also missing.\n- Some adjustments regarding the overall organisation of the paper could be made: \"The score matching and diffusion probabilistic models\" subsection in §2.4 could be included in the Introduction. This could help understanding that there is not much distinction between score-based generative models and DDPMs, as suggested in the Introduction, and would help understanding Sect. 2. The sketch of the proof could follow the theorem.\n\n- l. 219 $L_s(t)$ instead of $L_s(r)$\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LV_ddj7_wmP", "reviewer": "Reviewer_EPR5", "summary": "This paper shows that score-based models minimize the Wasserstein distance between them and that the Wasserstein distance is upper bounded by the square root of the objective function up to multiplicative constants and a fixed constant offset.", "questions": "1. The main question is whether or not the theorems in this paper lead to the finding that score-based models minimize the Wasserstein distance between them, i.e., $W_2(p_0,q_0) = \\min W(p,q) = 0$. The theorems in this paper are based on the upper bounds of $W_2$. Remark 1 indicates that Corollary 1 implies that $W_2$ converges to $0$ under some assumptions. Hence, it is not guaranteed that score-based models minimize the Wasserstein distance between them.\n2. The assumptions in the main theorems seem to be strong. For example,\n- Remark 1: the conditions $J_{SM} \\to 0$, $p_T = q_T$, and $\\lambda(t) = g(t)^2$ seem to be strong. Provide some practical examples satisfying all the conditions. \n- Theorem 2: Provide some practical examples of $p_{0t}$ satisfying (15). \n3. As a result, I cannot understand what the contribution of this paper is and why the results in this paper are significant. In particular, I cannot understand why the upper bounds of $W_2$ are needed for deep machine learning and its related areas. ", "limitations": "1. There is a gap between the claim in the abstract and the main theorems in this paper. \n2. The assumptions in this paper seem to be strong.\n3. I do not know that numerical results in section 4 are appropriate since the condition $W_2(p_T, q_T) = 0$ is assumed. Also, I do not know why the setting in Page 6 (e.g., a 3-layer neural network, AdamW with lr = 0.001) are appropriate. The numerical results are based on factitious setting, and hence, limited results. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: This paper shows that score-based models minimize the Wasserstein distance between them and that the Wasserstein distance is upper bounded by the square root of the objective function up to multiplicative constants and a fixed constant offset.\n\nWeaknesses:\n1. The assumptions in the main theorems seem to be strong. \n2. The abstract indicates the claim such that score-based models minimize the Wasserstein distance between them. However, the main theorems cannot follow the claim. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "E54goC9XYda", "reviewer": "Reviewer_QQUJ", "summary": "- This paper discusses properties of the PDF realized by the data generating process of score-based models (denoising diffusion models), which have received a great deal of attention recently.\nThe three main theorems are as follows\n  1. If the true density is $p_0$ and the density of generated data is $q_0$, then its Wasserstein2 distance $W_2(p_0, q_0)$ is bounded above by \n$$W_2(p_T, q_T) \\times \\text{(a constant dependent on Lipschitz constants)} + \\sqrt{\\text{(training loss)}} \\times \\text{const}.$$\nTherefore, under ideal conditions such that $p_T=q_T$, the loss function is an upper bound of the Wasserstein2 distance between the true and generated densities.\n  2. The loss function commonly used in the training of score-based model is just an approximation of the true score-matching loss. In this paper, a theorem that gives the relation betweem them is provided. This gives a theoretical backborn  that the approximate loss function used in practice can be used as a surrogate function for the true objective function which is difficult to evaluate exactly.\n  3. Perturbing around the initial value (eliminating singularities at the time origin) is a fundamental technique in practical situations. A theorem is provided to evaluate the upper bound of the Wasserstein distance between the true density and the perturbed density. This is a direct consequence of the triangular inequality of the Wasserstein distance and the above theorem.\n\n- It is then demonstrated numerically using a toy model and toy data that the upper bound given by these theorems actually valid. It is also argued that the main factors of the gap between the upper bound and the loss are related to the the Lipschitz constant of the neural network.\n\n- The proof begins with a review of the fundamentals that the Fokker-Planck equation is analogous to the equation of continuity (or the law of mass conservation) which is the fundamental relationship between density and velocity field. Based on that argument, a equation is presented that states that the time derivative of the Wasserstein2 distance is expressed by the velocity field, the proof of which is based on outside literature.\n", "questions": "- The discussion in this paper is based on a continuous stochastic process, but the numerical simulations appear to be dealing with discretized processes with the step size of $t = 1$. Is my understanding correct that that this is justified by Cor 2?\n\n- (The issue that interested me most while reading the text was the behavior in higher dimensions rather than the 2-dim toy data. However, this was discussed in the appendix, albeit to a limited extent.)\n", "limitations": "- It is argued in section 6 that more general conditions make theoretical evaluation more difficult.\n", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- These theorems are interesting because they guarantee fundamental properties of the score-based generative models. (However, I did not follow all the technical details of the proofs, and am not still confident in my understanding.)\n\n- It is an interesting claim that there is a theoretical guarantee that the same algorithm minimizes KL and Wasserstein distances simultaneously, as opposed to GAN, which leads to different algorithms depending on its distance measure.\n\n- These evaluations are strictly valid only for special cases, and for more general cases, theoretical evaluation is an open question. This is discussed in Section 6.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept