Score-Based Generative Models Detect Manifolds
Authors: Jakiw Pidstrigach
Venue: NeurIPS 2022 Accept (2022)
Keywords: diffusion models
generative models
score-based generative models
stochastic differential equations
score matching
generalization
memorization

Abstract: Score-based generative models (SGMs) need to approximate the scores $\nabla \log p_t$ of the intermediate distributions as well as the final distribution $p_T$ of the forward process. The theoretical underpinnings of the effects of these approximations are still lacking. We find precise conditions under which SGMs are able to produce samples from an underlying (low-dimensional) data manifold $\mathcal{M}$. This assures us that SGMs are able to generate the "right kind of samples". For example, taking $\mathcal{M}$ to be the subset of images of faces, we provide conditions under which the SGM robustly produces an image of a face, even though the relative frequencies of these images might not accurately represent the true data generating distribution. 
Moreover, this analysis is a first step towards understanding the generalization properties of SGMs: Taking $\mathcal{M}$ to be the set of all training samples, our results provide a precise description of when the SGM memorizes its training data.

Score-Based Generative Models Detect Manifolds
Jakiw Pidstrigach
Institut für Mathematik
Universität Potsdam
Karl- Barl-Liebknechh-SS0 24/25 14476 Potsdam
pidstrigachèuni -potsdam.....
Abstract
Score-based generative models (SGMs) need to approximate the scores V log Pr of the intermediate distributions as well as the final distribution pr of the forward process. The theoretical underpinnings of the effects of these approximations are still lacking. We find precise conditions under which SGMs are able to produce samples from underlying (lw-dimensional) data manifold M. This assures that SGMs are able to generate the "right kind of samples". For example. taking M to be the subset of images of faces, we find conditions under which the SGM robustly produces an image of a face, even though the relative frequencies of these images might not accurately represent the true data generating distribution. Moreover, this analysis is a first step towards understanding the generalization properties SGMs: Taking M to be the set of all training samples, our results provide precise description of when the SGM memorizes its training data.
Introduction
Score-based generative models, also called diffusion models (Sohl-Dickstein et al. 2015 Song
and Ermon 2019 Song et al., 2021b Vahdat et al. 2021]) and the related models (Bordes al.
2017| et al. 2000 Kingma et al. 2021 have shown great empirical success in many areas, such
as image generation [Jolicoer-MMattnnau
al. 202 Nichol and Dhariwal 202T Dhhrriaa and
Nichol 2021 Ho et 2002)), ), audio generation (Chen et al. 2021 Kong et al. 2021 Jeonn et al.
2021
Popov et al. [021]] []]]])>)))) as well as other applications (Batzzlls al. 2021 De Bortoli
2021 Zhou et al. [222
C Cai et al. 2020 Luo and Hu 2021 Meng et al. 202 Saharia et al. 2021
al.
2022 Sasaki et 2021 Recently some progress has been made to bridge the gap between
the different approaches (Song et al. 202 Huang et al. 2021 through the framework of SDEs
and reverse SDEs.
In generative modelling one is given samples {a J.N. from a measure Adata- The task is to learn measure jssmmlee which approximates Pdata- The performance of generative model can then be measured by the distance from Msample to Pdata- In practice however, the true data generating distribution Mdata unknown. All that is known are the samples {x, (r:)7-1 which can be used define the empirical measure Adatas
Pdata == Unif(z²
Any sample from the empirical measure Adana will be equal training example. Hence, while plample being close to Adata is the final goal, /Asample being close to Adar implies that the generative model has memorized the training data. To summarize, good generative model will output measure Psample which is as close to Adaa as possible, while keeping some distance from Adata even though it only knows Adata through Adata
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Adata
Intermediate densities Ft
P1
Mdaia
Meomple
AYL
2
91 Psamole
Intermediate densities
9o Mprior
Figure 1: Top left: The leftmost plot shows the true data distribution Pdara which is Gaussian mixture. The heat maps show the intermediate densities of Xx, followed by line plots of fo Bottom left: The rightmost plot shows ppriors which standard Gaussian and differs from P1. We start the reverse SDE error and use sax, t) P1 + with a constant error of 3. Again, the heat maps show how the densities of Y evolve backwards in time. The leftmost plot shows the resulting distribution Q which is used as sample distribution, psample = Q1-
2 in Mprior But instead of using the real score, we introduce an approximation
Right: The densities Adaan and μsample are shown for direct comparison. We see that the approximation
errors in Mprior and the drift lead to an incorrect sample distribution psample * Jdluta Nevertheless, Msample is supported in the same area as Adata. For details on the numerical implementation see Appendix B
Given target measure To, score-based generative model (SGM) employs two stochastic differential equations (SDEs). The first one is called the forward SDDE
dX;
((X))dt odW,
(1)
Xo
The marginals of X, are denoted by Te. The forward SDE is run until some terminal time T. Furthermore, the reverse SDE is defined by
dy, = -((Y)) oo logpr- + adB,
Yo
go.
We refer to the marginals of Y, as qr- The samples are generated from the final distribution Qr, i.e. /ssample 47. The reverse SDE has the property that if go is chosen to be equal to Ty, then de = Āパ- In particular, this implics that qr = To- Therefore, we have samples from TT, we can run the reverse SDE them to create new samples from mo-
In the following we will denote by Pe the marginals of the forward SDE when started in the true data generating distribution To = Adaas- We will denote by P. the marginals of the forward SDE when started in To Pdau- Optimally, we would like run the algorithm using To Adatas i.e. with marginals = Pt. This is however not possible, since Adata itself is unknown.
To circumvent the problem of not knowing pr, the forward SDE is chosen such that forgets its initial condition Po- At time T, the marginal pr is then well approximated by proxy distribution Mprior 23 pr. independently of Po- Additionally, the marginals Pt and therefore the scores V log Pe cannot be evaluated for Po = Pdata Therefore, the scores are replaced by a neural network se (x t), which is trained via score-matching techniques [Vincent 201T Hyvärinen and Dayan 2005].
As a result, SGMs make two approximations. The first one in approximating Pr by Mprrior The second one is the approximation of V log Pt by the neural net (z, t). We illustrate this in Figure is important to understand how these approximations translate to the distance μsample to Adata or A
Some early works already deal with these questions. In (De Bortoli et al. 2021 the total variation distance between psample and Adata in bounded, whereas [Song al. 202 derives bounds with

---

t- 0.5
t- 0.9
t- 0.95
t- 0.99
2
1
2
(a) Here fatm is chosen as the uniform distribution on the unit sphere S². We use the Brownian motion for the forward SDE. We can compute the exact score V log (z:). We perturb it with the vector v (æ 0, y = -1) and define the approximation s(x, V log (z + Furthermore,
purposely adopt poor approximation Merior 78 Pı by setting Mpriar (m, f). where = (xr 1.5, y = 0). We then run the reverse SDE (8) The figure shows heat maps the intermediate distributions z the reverse SDE. At time = we reach = Msumple- We see that Msampte is distribution on M = S albeit not the uniform one. Furthermore, we can observe how the emror in the initial conditions and the drift influence the skewed distribution Mample- The initial conditions where chosen have large z-coordinate on average, whereas the drift chosen prefer lower y-coordinates. The distribution phamll concentrated in areas with high æ: and low y-coordinates.
t=0
t.5
t=
t= 0.95
t= 0.99
t=1
22 1
0
1
2
2
0
1
2
2
1 0
2
0
1
(b) The above experiment is repeated but with pam chosen to be the uniform distribution on M {x;)²- where the are 9 evenly spaced points on the unit sphere 8 Notice that while the approximation errois cause non-uniform distribution the training examples, Mample still supported solely on M and will not generate novel samples.
Figure 2: Perturbing V log
respect to the KL.-Divergenee The work [De Bortoli et al. [021] furthermore derives result similarly to the second part of Theorem [ but also treating the errors that are introduced by discretizing the SDE. However, both these works assume that the initial distribution Jaaa rather well behaved. In particular, it is assumed that Jaaaaa (a: > 0 for all We shortly discuss this assumption now.
Assuming that (2)
for any means that one postulates that every æ a possible sample from Mdata- For example, this implies that even if all samples {=1,..... )n, of Adata consist of images of human faces, we say that pdata can possibly also generate images of for example furniture, animals or pure white noise. Even though it might have low probability, any combinations of pixels is possible sample from AAtata- The assumption that (Aaaaa actually supported on some lower dimensional substructure M is well known under the name manifold hypothesis, see for example [Bengio et al., 2013 Pope al.) 2021). In practice this means that (4aam
for many x. This also leads to exploding scores V log Pt: as / + 0 (see Section 5). behaviour which also observed empirically and further underpins the relevance of the manifold assumption. We will from now on denote the support of /ddata as data manifold M.
fundamental question is then: What is the support of (sample and how does it compare to M. This is interesting for multiple reasons. On the one hand, if we for example assume that M is the set of all images of faces, then the knowledge that psample also has support M implies that, regardless of how close Hsample actually is to Adat. it will at least always produce an image of face. On the other hand, we can also compare the support of psample that of Âdua- The measures Msample and Mdau sharing their support translates to Psample memorizing the training data and not being able to generalize. Both of these are very valuable insights into the qualities j'sample on its own. Furthermore, statistical distances like the KL.-Divergence or the total variation distance are only meaningful if the support of the measures overlap some degree, as otherwise they will equal the maximum distance value.
If two measures have the same support, they are said to be equivalent. Our main result is the following:
Main result. We identify conditions under which Prample is able to learn the data manifold, that supp (pampe)) (pdaa() Applying these results to different settings, find precise conditions
3

---

under which an SGM memorizes its training data or under which it is able to learn the right data manifold M.
In particular we find that for SGMs to be able to generalize, the approximation error made when approximating the training drift 4 log Pt has be unbounded.
A first illustration of these results is given in Figure 2 We use simple example, where we can perfectly evaluate the true drift V log Tt. We then choose an incorrect initial condition Ppeiue which is far from Pr, and also add a constant error to V log (ar). The initial measures To are given as the uniform distribution on the unit sphere and the uniform distribution on samples from the unit sphere in Figure 2a and Figure 2 respectively. We see that the final distribution of the reverse SDE,
Msample. not the uniform distribution anymore. This is due to errors in the initial conditions and drift. Nevertheless, Psample is still supported on the exact same subset as To-
When applying our main result to the empirical measure Adata one gets the following corollary. supplying precise conditions under which SGM has memorized its training data.
Corollary 1. Denote by X, the forward SDE when started in the empirical measure mo Patata- Let 16' [|,o( t) v log (X, jtt be drift approximation error along a path of the forward SDE. For a given weighting function wt),, the training objective of an SGM can be written as
L2 = /
((t)))so(X..t)) - log (X.:))ddt
Section Simultaneously, the exponential integral of the drift approximation error is integrable in the following sense:
Lexp E Ex -0 S. [)so(X(,t) - - Vlog
oo;
(4)
the SGM has memorized its training data. Therefore, while training an SGM one should aim to minimize the mean squared error while ensuring that the mean exponential error stays infinite, Leep = CC.
In particular, if (X,, 1) - lo is bounded, then Loxp is finite. Therefore, the the generalization capability of SGM crucially depends the training error being unbounded.
We now proceed as follows. In Section 2 we will summarize some of the most popular forward SDEs that are applied in SGMs. Then, in Section 3 we discuss how the drift approximation sa(z, t) for SGMs trained in most implementations. We are ready to state our main results in Section 4 In Section [ we show how the empirically observed drift explosion is related to the the manifold hypothesis. Most of the paper discusses the error the drift approximation. But as we have discussed, we also have an error in the initial conditions. In Section 6 we will discuss how large the error in the initial conditions will be in practice.
2 Popular SDEs used in SGMs
In this section we introduce some of the most popular SDEs that are used when implementing SGMs. The first works on SGMs studied discrete forward and backward processes. Nevertheless, the transition kernels and algorithms proposed in those works can be seen as discretisations of some well-known SDEs. More recent works have studied this connection and state the algorithms in terms of SDEs (Song et al. 2021b. Huang et al. 2021)
Brownian Motion: The works [Song and Ermon SDE
2019 2020
be seen as discretization of the
dX, = (t)
Denoting I<(1)= = fo (s)ds, the solution to the above process can be explicitly stated as a time- changed Brownian motion, X, Wa(O- The time-change can help in the implementation but does not alter the qualitative behaviour of the reverse SDE. In our following analysis we therefore set (t)) = 1. Nevertheless, our results still hold for any positive (t).

---

Ornstein- Uilihne-UUebbbck Process: as discretization of
The works (Soht-Dickstein et al. 2015 Ho et al. 2020 can be seen
1 dx, - + va(()W,
which is an Ornstein-Uhlen Uhlenbeck process. Again, the parameters are time-change and do not influence the properties that we are investigating in this paper. Therefore, to simplify notation, we again set (tt) =
Critically Damped Langevin Dynamics (CLD): The work [Dockhorn al. 2021] studies second order SDE. Here artificial velocity coordinates V, are introduced and the system under consideration
dX, V/2 dV. -X, - 2V, + 2dW
where Xo ~ Pdatn. Vo (0, I). For generation one runs the reverse SDE in X, and V2 but discards the V coordinate at the end. The work [Dockhorn et al., [021] also includes the parameters M and y. We set both they do in their numerical experiments.
3 Score approximation with a finite number of samples
We now quickly discuss how the neural network is trained to approximate V log and what implica- tions this has. The score log Pe is approximated by minimizing a variant of
L(#) [0
w(t) Ep,(a) (r)) - s0(x, dt.
(5)
for some weighting function w. The optimization is done via matching techniques (see HHyvärinen and Dayan, (2055] Vincent 2011 [Song et al. (020). However, the above expectation
depends on pt, which cannot evaluate since depends Po Mdaata Nevertheless, we
evaluate the approximation Pr.
N
(x) = E E (zg) (2oo(t/
1
Σ
(=o(( so pe(x) = Exaan (e
(6)
N 1=1
The surrogate loss
i.(0) / w(() Ep.( [I] (.)) - s0(x, 1)),,
be evaluated and is used for training. The equation 7 equivalent to since X, has distribution Pt. If we would minimize this loss perfectly, then so(x, t) would be equal to V log P.. The reverse SDE started in appropriate initial condition with drift V log P. however, will produce samples from Adata. which are training examples. This raises the question in which way or to which degree the loss should actually be minimized.
4 Effects of the approximations
This section contains our main results. We first state the assumptions we have to make and then the Theorems.
4.1 Error in the initial condition
The following assumption is needed for the reverse SDE to be defined even the case when the initial distribution No has degenerate support. In Lemma T we will show that all the SDEs from Section 2 satisfy the Assumption.
Assumption 1. There is constant C such that
(i) 3 is globally Lipschitz, i.e. Le.1][(x) B(x))l < Cl y/l
5

---

Distance CIFAR-10 Dataset
40
30
20
Trained sylx, t)
Vlog Pt
(b)
0.0
0.2
0.4
0.6
0.8 1.0
Figure 3: (a): Both lines correspond to the same experiment for different drifts in the reverse SDE. For both lines we started N 1000 paths in the zero vector in 323223233 For the blue line we used the pretrained CIFAR-10 10 DDPM++ model from Song et al. [2021b), whereas for the orange line we used the true drift V log P., which is mixture of 50000 Gaussians, one for each training example in CIFAR-IO. We then saved the distance from Y, to the CIFAR-I0 training examples, by calculating the distance to the closest example. Above we plot the average distance. We see, that while the reverse SDE run with Dr will have distance of 0 to the training examples in the end, the SDE with the pretrained drift keeps some distance to the training examples and therefore produces novel images. (b): We evaluate V log P. in (a). We do the analogous experiment to Figure 2b on CIFAR-10 and perturb the empirical drift V log ft with a constant error vector. The first row shows the samples generated by adding the constant eπгог vector e(x, t) V Vog ft. In the second row we searched for the closest image in the CIFAR-10 dataset (with respect to the Euclidean 2-distance on 322x32x3 and plotted it. We see that all the sampled images nearly equal to corresponding image in CIFAR-10. The distance the images to their closest image in CIFAR-I0 is around 0.07 for all plotted images. Similar to Figure 2 we can observe the effect of adding the one-vector. The sample distribution jsample got skewed to prefer images that have high pixel values. This corresponds samples which are mostly white for the human eye. In the third and forth row we repeat the experiment of the first and second row, but add the negative one-vector e(x, (-1, -1,-1,.... -1) and get black images.
.......... 1) R 332x32x3 to
(ii) 3 grows most linearly, i.e. ||3(x)] C(1 + |
(iii) X, has density n, C² for every > 0 and Sto Ji <n (=(()) T (x) < oo for any and < to T.
Furthermore, for each S E (0, and all x. for which || Ilvll N there is constant Cs.N such that
(iv) log ne locally Lipschitz, V log (x) V log n. (y)ll S Cs.N all E [S,
Conditions (i)-(i are technical conditions the forward SDE. They ensure that if solution to the forward SDE, X,, backwards in time, then X := Xy- will be a solution to the reverse SDE ( [0, T). The last condition then ensures that the solutions to the reverse SDE unique, therefore we will be able to transmit the properties of X to any other solution Y, of (2). The following result shows that Assumption [ can be expected hold in practice. To simplify the calculations we assume that the data manifold M supp()adta) is contained in ball of diameter M. This is natural assumption for many data sets. Nevertheless, we that this assumption could be weakened by additional technical effort.
Lemma Assume that the data manifold M is contained in a ball of radius M. Then all the methods introduced in Section fulfil Assumption 7
We are now ready state first main result.

---

Theorem 1. Denote by the marginals of the forward SDE started in No. Assume that Assumption 7 holds and that Pprior absolutely continuous with respect to WT. Then the following hold.
(i) Let Yt be solution to 2 on (), The limit YT = limm- YT exists almost surely. We refer its distribution Psample- The distribution Hsumple absolutely continuous with respect to Pdoi- If WT and Uprior are equivalent. then so are psampte and To.
(ii) Furthermore, for any f-divergence Dj.
D (Psample TT) ) I D TT) and Dy(Too (sampie) S Dy(TT (ppioo)
Applying the above theorem with कт pr tells us something about the equivalence and distance between /ssample and Idataa since me = Po = Pdinta Applying the theorem with ft tells us something about the generalization capabilities of SGMs.
The measures тТ and Hprior are normally both supported on all of E2 and therefore equivalent. Since Assumption T also holds in most situations (see Lemma D. the requirements for Theorem are satisfied in practice. From item (i) with ग, Pe we can conclude that the error we make in the initial conditions is not responsible for the generalization capacities of SGMs.
The second item then shows that the f-divergences between Hsample and Adata are bounded by the f-divergences of Pprior to pr. The total variation distance and the KL-Divergence are both special cases of J-divergences.
4.2 Error in the drift
Given a forward SDE with marginals ne and an approximation s(x, t) to V log Te, we define the reverse SDE for the approximation
dř, = -3(()) + oo (V ()dl + odB.
(8)
Y
Assumption 2. We assume that the reverse SDE Y, has a solution on [0, Fort < T. we define the Girsanov weights
Z. =exp exp (f'""(((0.))
1 10 (s(2,,) Volgo--oogmmoooooo Vlogm.(F)))] logm T. (Y.)) aB. 2
and assume that the Ze are uniformly integrable martingale.
We shortly discuss this assumption. The assumption that Zr is a martingale is equivalent to the expectation of Z, being equal to 1 for all f. The assumption that it is uniformly integrable is more technical (see Appendix A.2. but is fulfilled for example for some p > R 7.|²] < oo for t.
For example, if the Z, have bounded variance, then they uniformly integrable.
condition that ensures that Z, is both, martingale and uniformly integrable, given by Novikov's condition Novikov 1980). It states, that if
exp ( (oTT((()))))) log (o)))( ]
NT Ey
< 00,
(10)
then Z, is uniformly integrable martingale.
Using Assumption 2 we can now state
Theorem 2. Assume that Assumption V log ni replaced by s(x,t), t).
2 holds. Assume furthermore that Assumptiont Tholds with
Then Yr = lim,-T Y, is well defined. Moreover, its distribution equivalent to the distribution of Yr. In particular, if [[, t) V fo is bounded, then Assumption 2 holds, and the SGM has memorized its training data.
Putting Theorem [ and Theorem 2 together, we can conclude that, if both Assumptions hold, /samppe equivalent to mo- Therefore, Assumption 2 holdds for n, = Pt, we have a positive statement and know that Msample will have the exact same support Mdaas i.c. it has learned the data manifold M.
7

---

If the Assumption would hold for Pt. we would however just memorize the training data, see Figure o or Figure for visualization. However, empirically has been shown that SGMs are able to create novel samples (see, Figure 3a or Dhariwal and Nichol [022 Therefore, we can deduce that Assumption 2 is violated in practice.
We evaluated N from (10) on CIFAR-I0, once for the difference between the sy s t) from Song al. 20211b and log Pt. and once by just using a perturbed drift with constant error, s(x 0
is equivalent Adaa. as we know from Theorem 2 and have also already observed in Figure 3b Figure
v Vlog + (1, 1), see Figure 4 The reverse SDE using the drift V log Pr + - (1, 1,.......
confirms that Z is indeed uniformly integrable martingale and therefore fulfils Assumption 2
Corollary 1 can be deduced by exchanging the roles of Y; and Y,. We then get an equivalent condition to Assumption 2
which is
1 T f. 2 %/500 (o
] oc,
N7 Ey
exp
v log (
where the expectation is now taken over the reverse SDE with the correct drift V log πt instead of the approximate drift sg. However, Y is just the time reversal of X, therefore we can also write the above expectation over X2 instead of Y, If we treat the case where start X, in the empirical measure Auaat Te will be equal to Te Pe by definition and
02 2 (''
] 20
Ex
exp
Ioo V Vlog
a sufficient condition for Adata and psample having the same support. We have here assumed that Theorem can be applied. However, this can be assumed in practice, the discussion after Theorem
In future work we believe that further understanding the characteristics of Ze how they relate to the minimization of L(#) and generalization is crucial to the understanding of SGMs and their empirical success. Recent works also study the question on how the neural network architecture
and parametrization related to the boundedness of the output Kim et al. [022]. The relationship between these choices and the properties of Z. is also an interesting research avenue. Lastly, the distribution of Ze is very heavy-tailed. Most the samples are very small with a few extremely large samples in between. Therefore needs many samples from Z, to understand its characteristics. Finding robust estimators for Z, or its expectation could help their usage the training or evaluation procedure.
5 Drift Explosion under manifold hypothesis
In practice it is often observed that the drift of the reverse SDE explodes as t + T. see for example Kim et al. [2022, Section 3.1]. We now show how this observed behaviour related to the manifold hypothesis.
First, note that all SDEs in Section linear SDEs. Therefore, their transition kernels Gaussian (Pavliotis) 2014 Section 3.7]):
Te = X, = mb) = N(x; (2)))
The explicit form of m, and Σ, differ for each of the SDEs and can found in Appendix C.T We remark that Σ, does not depend on the initial condition Zp. The transition kernel above is the distribution of the SDE started a single point wo. Since we start the SDE in Pdata we need to average over Maaa to get the marginal at time t:
(x( = / N(z; mz(zo), Eg Hamaa (dnaa (zo)dro-
(11)
JRd
We can also compute the additional drift the reverse SDE (see Appendix C.2
log)i(x)= = (pr(x) M (x (Xm)) = =x).
(12)
(z(
8

---

10/000
10a
==
olog 6x
5 Vlogg
10*
10
Dnft error
101
10
10a
10
10
10a
10a3
10a
0.00
0.25
0.50
0.75
1.00
Figure 4: Left: We simulated the reverse SDE on CIFAR-10 10, once with the pretrained CIFAR-10 DDPM+ model sy from Song al. 2021b and once with perturbed drift s(x; t) V log Pr (1, (1.1.....1)) 1). drift, the integral does not seem to explode
We then evaluated the integral 10 numerically for varying = T. For the perturbed - 1, implying that Z, is martingale. We see that for the DDPM++ drift, the integral explodes, therefore we can not infer that Z, is a martingale. We used N 12000 simulations from both of the SDES to generate this plot. Right: We again the two SDEs with the drifts as in the left Figure. This time, we measured the average distance to the empirical drift []s(Y,, t) V log Pe (Y))l along a path of the reverse SDE. We repeated the experiment N 2560 times and plotted the mean distance. For the constant perturbation we also of course get a constant distance. The distance of the true drift to V log P. is initially very small but explodes as + 1. From our results we know that this explosion is necessary for the SGM generalize.
We now want to evaluate V log pt along a typical path of Y, i.e. we are interested in E[[ po p.gg:.())) (Y+)] The distribution of Y, however depends the drift approximation Sg we will assume that we are able to run the reverse SDE with the true drift sæ sa(x, t) = (x()). Then however, since Y, then just X; run backwards, they have the same distributions and we can calculate
For this calculation
||ll V P. P. (Y] E log P. (X,) = E|] X,
= villl-
If the manifold M is not to badly behaved, can expect that for small and almost all x, B[X, X; ai] to be very close to the data manifold M. Especially, ||X,- - [X = =]]| will be larger than dist(Xe MM) in that case. However, the distribution of X, can be represented as m, (Xp) + VEgg where & has standard normal distribution. For the SDEs treated in Section my mx(,,) is either equal or very close to (X,) = X, for small values Finally, if we assume that M is a subset of relatively low dimension in a high dimensional space, we can expect that with very high probability / E points away from the data manifold. Therefore the distance of X, to M can be approximated by v /Etg. Putting these approximations together
calculate
B| (Xo), vill] ž E[dist (X,, M)] 2 |2, / | Vd.
where d the dimension of the data space which the samples x; lie. Therefore we can conclude that
d//2
Vlogg 2
|Σ
For the Brownian motion for example, Σ, = and therefore the right hand side scales like 1 V Σ, is the covariance Pt, then we can expect V log (Yi) to be of order [[. Furthermore. V logg Pt PYi) will point towards the data manifold for small t. The drift V log Pt (Y₂) then acts like support matching force, where the force grows to infinity as - 0, absorbing the SDE paths onto the manifold.
6 Distance from Pr to /'prior
We have seen in Theorem I that the distance between Msample and Plaata directly related to the distance between pr and Mprirr if we neglect the errors made in the approximation of the drift. For both, the OU-Process and the CLD, there are plenty of results on the distance of pt to (0, Ta). In general, one can expect this distance to grown exponentially in time

---

The Brownian motion however does not converge to a stationary distribution and therefore one has to choose a different Marior for each T approximate pr. practice, one normally chooses normal distribution pr = N (m,, C+) (see [Song et al. 2021b Appendix CJ) The following Lemma is derives the optimal values for m, and Ct.
Lemma 2. Let pr be the T-time marginal of the Brownian motion process X, of Section The following minimization problem
min LLppr (m,, C))
m.C
minimized by mr Eijtaall and Cr Cov[Uaduu) + TIg- If restrict the covariance to be a multiple of the identity matrix, the problem solved by choosing m as above and c as c E|||X, =
This result a slight variation on the well known fact that the K L-roojectionn in the second argument matches its moments. We prove it in Appendix [2.2 The following result shows that the distance I between pr and prrorr decreases with time and also gives a rate. It justifies using the Brownian motion and normal prior distribution for SGMs.
Lemma 3. Let Pt be the time t-marginal of Brownian motion with initial condition Pdate- Denote by C, i = I,...,d, the eigenvalues of Cov((daaa) Let perior be the normal distribution with mean mr = ElPdato] and covariance Cr = --ovv (Cov/Adual] + Tla. Then
[[:=n(es +T)
K KT (
S 2
log
T
The proof can be found in Appendix
E.2
7 Broader impact
The results deepen the understanding of score-based generative models. As such, they can be seen as step towards improving the quality of generative models. Therefore the possible negative societal impacts are the same ones that apply to generative modelling in general. First, generative models can be used to create synthetic data that is hard to distinguish from real data (for example images or videos), see [Mirsky and Lee [021]. Second, generative models can learn and reproduce biases that are prevalent in the training data (LEsser et al., 2020). Last, depending on the application, generative models might be used to do creative work that was previously done by humans.
Conclusion
We conducted theoretical study of some properties of SGMs. We found explicit conditions under which the sample measure /sample is equivalent the true data generating distribution (Aaaa- Under these conditions we can guarantee, that the SGM generates samples that could also be samples from Adata- Furthermore, each sample that can be generated by Platn also has positive probability under Msample: meaning that the full support is covered.
Since one can not actually access the full support of Mdata, but only finite number of training examples [I. our results can be applied to find conditions under which the SGM memorizes its training data. We believe that this observation provides first step towards understanding the generalization capabilities of SGMs.
Funding
The author has been partially supported by Deutsche Forschungsgeemeinschae (DFG) Project-ID 318763901 SFB 1294.
References
G. Batzolis, J. Stanezuk, C. Schönlieh, and C. Etmann. Conditional image generation with score-based diffusion models. CoRR, abs/2111.13600 2021. URL /ttxx: org/abs//////oooooooooo.oooooo abs/2111. 13606
10

---

Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798- 1828, 2013.
Bordes, S. Honari, and P. Vincent. Learning to generate samples from noise through infusion training. arXiv preprint arXiv: Ftxxiv/703.069977 2017.
N. Bou-Rabee, M. Hairer, and E. Vanden-Fijnden. Non-asymptotic mixing of the MALA algorithm.
IMA Journal of Numerical Analysis, 33, 08 2010. doi: 10. 1003//manuu//rr003....
R. Cai, G. Yang, H. Averbuch-Elor, Z. Hao, S. J. Belongie, N. Snavely, and B. Hariharan. Learning gradient fields for shape generation. In A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, editors, Computer Vision ECCV 2020- 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III, volume 12348 of Lecture Notes in Computer Science, pages 364-381. Springer, 2020. doi: 10. 10.11077978--3-00--20 978- 3- 030 58580-888
22. URL https://doi. org.//oo..rr/////// 1007/
Y. Cao, J. Lu, and L. Wang. On explicit I==convergenne rate estimate for underdamped Langevin dynamics. arXiv preprini arXiv: (at:::998..044444. 2019.
N. Chen, Y. Zhang, H. Zen, J. Weiss, M. Norouzi, and W. Chan. Wavegrad: Estimating gradients for waveform generation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL/httss://openeeeeee
M. Costa and T. Cover. On the similarity of the entropy power inequality and the Brunn- Mounn-Minkowski inequality (corresp.). IEEE Transactions Information Theory, (0(6))877-839 1984.
V. De Bortoli, J. Thornton, J. Heng, and A. Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 34, 2021.
Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
T. Dockhorn, A. Vahdat, and K. Kreis. Score-based generative modeling with critically-damped Langevin diffusion. CoRR, abs/2112.07066 2021. URL http:://arxxvoorr/bb//1111 07068
A. Eberle, A. Guillin, and R. Zimmer. Couplings and quantitative contraction rates for langevin dynamics. The Annals of Probability, 47(4): 2019.
Esser, R. Rombach, and B. Ommer. A note on data biases in generative models. In NeurlPS 2020 Workshop on Machine Learning for Creativity and Design, 2020. URL hRtpss/////ax org/ abs/2012.
U. G. Haussmann and E. Pardoux. Time reversal of diffusions. The Annals of Probability, pages 1188-1205, 1986.
J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
C.WW. Huang, H. Lim, and A. C. Courville. A variational perspective on diffusion- generative models and score matching. Advances in Neural Information Processing Systems, 34, 2021.
A. Hyvärinen and P. Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
M. Jeong, H. Kim, S. Cheon, B. Choi, and N. S. Kim. Diff-its: A denoising diffusion model for text-to-speech. In H. Hermansky, H. Cernocký, L. Burget, L. Lamel, O. Scharenborg, and P. Motlicek, editors, Interspeech 2021, 22nd Annual Conference of the International Speech Com- munication Association, Brno, Czechia, 30 August 3 September 2021, pages 2021. doi: 10.21457/hntesspeechh 2021-469. URL https:ttp... http://ii.oo//g///////
ISCA, 10. 21437 2 Interspeech
2021

---

A. Jolicoeur-Martnnauu R. BaiheeTa, I. Mitliagkas, and R. T. des Combes. Adversarial score matching and improved sampling for image generation. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. Utt.hattps://ppeneviiiee
net/ http//oormmidddeee httee/oorrrmmddd....g1.3.33..3.
I. Karatzas and S. Shreve. Brownian motion and stochastic calculus, volume 113. Springer Science & Business Media, 2012.
D. Kim, S. Shin, K. Song, W. Kang, and I. Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 11201-11223 PMLR, 2022. URL htlLhhttps/pppp htttps://proceedingg
mlr. (reess/v622kim2222 html.
D. P. Kingma, T. Salimans, B. Poole, and J. Ho. Variational diffusion models. arXiv preprint arXiv:2107.006300 2021.
A. Klenke. Probability theory: a comprehensive course. Springer Science & Business Media, 2013.
Z. Kong, W. Ping. J. Huang, K. Zhao, and B. Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL http:s:://ppeerevier.. net/
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
C. Léonard. Stochastic derivatives and generalized h-transforms of Markov processes. arXiv preprint arXiv:1102.3172. 2011.
H. Li, Y. Yang. M. Chang. S. Chen, H. Feng, Z. Xu, Q. Li, and Y. Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputiing, 479:47-59, 2022. doi: 00.1016/j.neucom.220 2022.01..22222000020 URL https://dooi...... http://............./....
1016// 1016/j neucom. 2022. 01.0190.099
Liese and I. Vajda. On divergences and informations in statistics and information theory. IEEE Transactions on Information Theory, 52(10):439444412, 2006.
S. Luo and W. Hu. Diffusion probabilistic models for 3d point cloud generation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, vir- tual, June 19-25, 2021, pages 2837-2845. Computer Vision Foundation IEEE, 2021. URL https: http::s//ppeaacesss thecvf com/ http///nnnee////PP00///////////s Sittsooooiifffiooooo 3D SD__Ponnt_Caiieeeeeeee Let_ecooooo_eeeeaaeeee
Probabilisticc_odellsss
CVPR_ 2000110ppppeee .ttml
P. A. Markowich and C. Villani. On the trend equilibrium for the Bokker-Plance equation: interplay between physics and functional analysis. Mat. Contemp, 19:1-29, 2000.
C. Meng, Y. Song, J. Song, J. Wu, J. Zhu, and S. Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. CoRR, abs/2108.01073, 2021. URL/attss://arxx org/ abs// 2108.01077
Y. Mirsky and W. Lee. The creation and detection of deepfakes: A survey, ACM Computing Surveys (CSUR), 54(1):1 2021.
R. M. Neal et al. Mome using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):
2, 2011.
A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
A. A. Novikov. On conditions for uniform integrability of continuous non-negative martingales. Theory of Probability & Its Applications, 24(4):820 824, 1980.
G. A. Pavliotis. Stochastic processes and applications: diffusion processes, the Fokker-Plande and Langevin equations, volume 60. Springer, 2014.
12

---

S. Peluchetti. Non-denoising forward forward-time diffusions. 2021.
Pope, C. Zhu, Abdelkader, M. Goldblum, and T. Goldstein. The intrinsic dimension of images and its impact learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. Net//orrmmid=xKJ1393
V. Popov, I. Vovk, V. Gogoryan. T. Sadekova, and M. A. Kudinov. Grad-tts: diffusion probabilistic model for text-to-speech. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8599-8608. PMLR, 2021. URL //procesdings.lt .ww.gg.s.......... attess/v139/pppov2222 htmm
O. Rioul. Information theoretic proofs of entropy power inequalities. Information Theory, 57(1):33-55, 2010.
IEEE Transactions on
G. O. Roberts and R. L. Tweedie. Exponential convergence of langevin distributions and their discrete approximations, Bernoulli, pages 341-363, 1996.
C. Saharia, Ho, Chan, Salimans, Fleet, and M. Norouzi. Image super-resolution via iterative refinement. CoRR, abs/2104.07636, 2021. URL htt.asss///axx...ogg//aa//////// 07636
H. Sasaki, C. G. Willcocks, and T. P. Breckon. UNIT-DDPM: UNpaired image translation with denoising diffusion probabilistic models. CoRR, abs/2104.053366 2021. URI. https://rxx org/abs/2104..o
J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, pages 2256-2263 PMLR, 2015.
Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché- Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference Neural Information P Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, cc/pppp/////ppppp///
Canada, pages 1197, 2019. URL htRp:ttt////rrr
haaeh//001ee22777777777777726662 wtt.://sae7cccc----tttttttt..... wwwttaac.t......... html
Y. Song and S. Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 11122448-24455 2020.
Y. Song, Garg, Shi, and Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pages 574-584. PMLR, 2020.
Y. Song, C. Durkan, Murray, and S. Ermon. Maximum likelihood training of acore-based diffusion models. Advances in Neural Information Processing Systems, 34: 14:1411-1428, 202
Y. Song, J. Gchl-Dickstein D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representatiinss ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b. URL https:://operee///
A. Vahdat, K. Kreis, and J. Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34, 2021.
P. Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):166-11674, 2011.
M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICMI-II), pages 681-688. Citeseer, 2011.
L. Zhou, Y. Du, and J. Wu. 3d shape generation and completion through point-voxel diffusion. In 2027 IEEE/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 5806-5815. IEEE, 2021. doi: 10. 10.1109/ICC449922222....0
URL
https: org/ 10. 1109 ww110///CCV3922.... ott:://..1109/110044422..0000000 000.0000044222222220000..7...7 00577
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] The assumptions are followed by
a discussion of their strength.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 7
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [No] There are error to report, we just have small illustrative numerical experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix B
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [N/A] We did not use any existing assets.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A] We did not use crowdsourcing or conduct research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "vnLnwNy88T", "reviewer": "Reviewer_3Xuq", "summary": "This paper is a theoretic paper that solidifies the well-definedness of sample measure and proves the equivalence of the sample measure and the learnt measure under Assumption 2.", "questions": "- Is $Q$ in the proof of Theorem 1 well-defined? Does $\\frac{\\mathrm{d}Q}{\\mathrm{d}P}(w)=\\frac{\\mathrm{d}\\mu_{prior}}{\\mathrm{d} p_{T}}$ enough to define $Q$? It seems that I am missing some parts, and it would be greatly appreciated if the authors let me know what I am missing.", "limitations": "- I am a practitioner in the community of diffusion models. In my perspective, a paper in this venue is better to contain intuitive illustrations and related explanations. In the current form, the submitted manuscript is flawless in its solidness, and I believe this is good work, but I doubt if this version could be valued in this field simply because it is difficult to understand the contents for the audience. Please remember that the expected audience of this paper is mostly the practitioners, and most of them are short-lesson-seekers. For instance, what is the meaning of uniformly integrable martingale? NeurIPS is not Annals of Mathematics, and sufficient interpretation of mathematical concepts is required. With the kind and satisfactory illustrations and insights, practitioners could find the true value of this paper.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "$\\textbf{Strengths}$\n- Until now, there is no guarantee that the reverse diffusion is well-defined at $t=0$ if $p_{data}$ is embedded in a low-dimensional manifold. This paper is the first to prove that we can safely solve the reverse diffusion up to $t=0$ when we use the previously suggested diffusion strategies (VE, VP, CLD).\n- On top of that, this paper shows the well-definedness of the generative measure if Assumption 2 holds. Theorem 2 additionally proves that the sample measure and the trained measure have identical support.\n\n$\\textbf{Weaknesses}$\n- It would have been much novel work if the author proved that Assumption 2 is the necessary and sufficient condition in Theorem 2. Theorem 2 only proves one direction, and it is difficult to conclude that uniform integrability is the key diagnosis of indicating the overfitting of the trained network.\n\n- It would be much better to move the proof of theorems to the appendix and put more explanations and experiments. In particular, this paper significantly lacks the empirical validation of their claims. Although I believe the claims are novel, the one-point lesson is missing to the community of diffusion models. What would be the \"concrete truth\" observed from the high-dimensional experiment?\n\n$\\textbf{Notes}$\n- This is not a weakness, but I should note that Theorem 1-(ii) is nothing but Theorem 1 of [**Song21Maximum**]. In Theorem 1 of [**Song21Maximum**], they used the data processing inequality for KL divergence, but the data processing inequality is well known for its $f$-divergence extension, and Theorem 1-(ii) is merely a straightforward generalization of the work of [**Song21Maximum**]. With all this, nevertheless, proving Theorem 1-(i) is a significant improvement, so I value Theorem 1.\n\n- Again, this is not a weak point, but I think it would be preferable to describe the meaning of equivalent measure for the general audience. A measure $\\mu$ is equivalent to $\\nu$ if $\\text{supp}(\\mu)=\\text{supp}(\\nu)$, and this can be framed in an everyday language as \"support matching force\". The general audience will much more value this paper if Theorem 2 is well-understood.\n\n- Not a weak point, but it seems that Assumption 2 is akin to the assumption of Theorem 1 of [**Bortoli21Diffusion**]. Theorem 1 of [**Bortoli21Diffusion**] explicitly bounds the total variation distance between the data distribution and the generative distribution under the uniformly bounded score estimation. Under such previous research, it would be very interesting to compare Theorem 1 of [**Bortoli21Diffusion**] and the authors' Theorem 2.\n\n- To get a generalizable model, the authors claim that Assumption 2 should be violated. [**Kim22Soft**] introduced Unbounded NCSN and Unbounded DDPM as $\\mathbf{s_{\\theta}}(x_{t},\\eta(t))$ with $\\lim_{t\\rightarrow 0}\\eta(t)=\\infty$ instead of NCSN and DDPM that parameterize the score estimation as $\\mathbf{s_\\theta}(x_t,t)$, to enable the score network to successfully estimate the unbounded score function. But it seems that NCSN and DDPM are more appropriate to violate Assumption 2, from their network design. It could be an interesting topic to investigate the network architectures with respect to the generalization power.\n\n[**Song21Maximum**] Song, Yang, et al. \"Maximum likelihood training of score-based diffusion models.\" Advances in Neural Information Processing Systems 34 (2021): 1415-1428.\n\n[**Bortoli21Diffusion**] De Bortoli, Valentin, et al. \"Diffusion Schrödinger bridge with applications to score-based generative modeling.\" Advances in Neural Information Processing Systems 34 (2021): 17695-17709.\n\n[**Kim22Soft**] Kim, Dongjun, et al. \"Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation.\" International Conference on Machine Learning (2022).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "uXTWY2bNj_Y", "reviewer": "Reviewer_ifjq", "summary": "The paper considers the approximation errors in the sampling process of score-based models. Specifically, the errors are in two folds: the estimation error of neural networks and the approximation of priors. The authors study the terminal distribution of the reverse SDE under such errors. They first show that the terminal measure is absolutely continuous w.r.t the data measure if we have an imperfect prior. Next, they show that the terminal distribution has the same support as the data distribution, under mild assumptions of the estimation error. Toy experiments demonstrate the validity of the theoretical results. ", "questions": "I copy and paste the questions above. \n\nBefore going into the concrete questions, I would like to double check some terms and the overall proof ideas with the authors. Please correct me if I'm wrong. (1) In line 71 \"its distribution is equivalent to the distribution of ..\": Does the equivalence here only mean that the two distributions have the same support? (2) The general proof idea of theorem 2: To my understanding, the paper uses the Girsanov theorem with $Z_T$ as the ratio $\\frac{\\mu_{sample}}{\\mu_{data}}=Z_T$. The uniformly integrable martingale assumption is to ensure the boundness of $Z_T$. Together the results manifests itself.\n\n", "limitations": "The authors have adequately addressed the potential negative societal impact of their work.", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The problem of approximation errors in score models is interesting and the theoretical understanding is lacking. The paper takes a step in this direction and investigates the terminal distribution of reverse sampling with the existence of various sources of errors.\n\n- The paper verifies the correctness of the main results on toy experiments, and examines popular SDEs to see whether they fulfill the assumptions. \n\n**Questions**\n\nBefore going into the concrete questions, I would like to double check some terms and the overall proof ideas with the authors. Please correct me if I'm wrong. (1) In line 71 \"its distribution is equivalent to the distribution of ..\": Does the equivalence here only mean that the two distributions have the same support? (2) The general proof idea of theorem 2: To my understanding, the paper uses the Girsanov theorem with $Z_T$ as the ratio $\\frac{\\mu_{sample}}{\\mu_{data}}=Z_T$. The uniformly integrable martingale assumption is to ensure the boundness of $Z_T$. Together the result manifests itself.\n\n## Weaknesses\n\n- The main result (theorem 2) in the paper is straightforward. It's a direct implication of Girsanov theorem. \n\n- The results seem tangent to the actual score-based generative models. For the image score-based generative models, the terminal $T$during sampling is set to $1e-3\\sim1e-5$ due to discontinuity at $T=0$. Critically, the support of $p_{T}(x), T>0$ is $\\mathbb{R}^n$, and the results trivially hold in this case. In addition, I ran some quick experiments on CIFAR-10 by adding 1 to the predicted score (the paper also does similar things), and the final results are noisy samples, with very large pixel values.\n\n\n\nDue to the limited impact of the theoretical results and the gap to practical models, I think the paper has a lot of room for improvement.\n \n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nbCUtZAA89T", "reviewer": "Reviewer_YBBA", "summary": "This paper is a theoretic investigation on how to learn a manifold with a score-based generative model (SGM). SGM uses an approximated score of the distribution in the middle of the reverse diffusion. This score estimation is only an approximation by a neural network, and this paper provides the study on error bound for this approximation. Also, this paper analyzes the impact of the limited sample population in learning the score function. Another story line is the limitation of diffusion in T because the diffusion step needs to be infinite to make the diffused distribution be a standard Normal distribution. This becomes infeasible in actual implementation, so there should be discrepancy between the final timestep distribution and the prior distribution. This discrepancy is analyzed, and its bound is suggested.", "questions": "See the weakness section", "limitations": "I think that this paper would need further toy example experiments and plain explanations to secure audience who may use diffusion models without enough knowledge on authors' discussions background.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1.\nThis paper provides a theoretic discussion on many mathematical assumptions suggested by SGMs.\n\n2.\nThis paper clearly shows the necessary and sufficient assumptions to make the SGM identify the low-dimensional manifold.\n\nWeakness:\n\n1.\nThis paper is very difficult to understand because of its eccentric structure. I partly understand the authors' effort because this paper is dedicated to the theoretic analyses. However, this paper is really needs a restructuring to draw more attention from possible audience.\n\n1) Assumptions are referred before their appearances. \n2) The motivation of Theorem 1 and 2 need to be further provided by formally throwing a research question.\n3) Need a further clear statement on error bounds. Assumption 2 shows the characteristics of e(x,t), but I expected the explicit error bound for e. \n\n2.\nThe score approximation error will grow as we bring the diffused distribution to the standard Normal distribution, i.e. t->T. This is briefly mentioned in line 206-211, and I think that authors might produce further discussions and plain explanation on the interaction of score approximation quality and closeness between the prior and the diffused distributions. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "78tZDOvgs2", "reviewer": "Reviewer_dRbJ", "summary": "This is a pure theoretical study of the score based generative model. The authors discover the conditions under which the sample distribution is equivalent to the data distribution.", "questions": "Move Theorem 1&2 to later sections, especially after assumption 1&2 are introduced, may make it flow better.", "limitations": "No societal impace for a theoretical work.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The theorems provided here could help researchers better understand those score based generative models, which is significant to the whole community.\n2. The assumptions proposed here can guide future design of score based models.\n3. As a pure theoretical work, the authors provide detailed proofs, which is helpful. But the lack of experimental verification could make it hard to digest. Although, I would not penalize the authors for this, since numerical verification on high dimension is a challenge on itself.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept