Score-based generative models are provably robust: an uncertainty quantification perspective
Authors: Nikiforos Mimikos-Stamatopoulos
Benjamin Zhang
Markos Katsoulakis
Venue: NeurIPS 2024 poster (2024)
Keywords: Score-based generative modeling
uncertainty quantification
Hamilton-Jacobi equations
generalization
Primary area: diffusion_based_models
TL;DR: Using techniques from analysis of partial differential equations and Hamilton-Jacobi equations, we show score-based generative models are robust, and also yield generalization bounds.

Abstract: Through an uncertainty quantification (UQ) perspective, we show that score-based generative models (SGMs) are provably robust to the multiple sources of error in practical implementation. Our primary tool is the Wasserstein uncertainty propagation (WUP) theorem, a *model-form UQ* bound that describes how the $L^2$ error from learning the score function propagates to a Wasserstein-1 ($\mathbf{d}_1$) ball around the true data distribution under the evolution of the Fokker-Planck equation. We show how errors due to (a) finite sample approximation, (b) early stopping, (c) score-matching objective choice, (d) score function parametrization expressiveness, and (e) reference distribution choice, impact the quality of the generative model in terms of a $\mathbf{d}_1$ bound of computable quantities. The WUP theorem relies on Bernstein estimates for Hamilton-Jacobi-Bellman partial differential equations (PDE) and the regularizing properties of diffusion processes. Specifically, *PDE regularity theory* shows that *stochasticity* is the key mechanism ensuring SGM algorithms are provably robust. The WUP theorem applies to integral probability metrics beyond $\mathbf{d}_1$, such as the total variation distance and the maximum mean discrepancy. Sample complexity and generalization bounds in $\mathbf{d}_1$ follow directly from the WUP theorem. Our approach requires minimal assumptions, is agnostic to the manifold hypothesis and avoids absolute continuity assumptions for the target distribution. Additionally, our results clarify the *trade-offs* among multiple error sources in SGMs.

Score-based generative models are provably robust: an uncertainty quantification perspective
Nikiforos Mimikos-Stann Stanaloopsta Department of Mathematics Université Côte d' Azur nmimikos@unice... fr
Benjamin J. Zhang
Division of Applied Mathematics
Brown University
junnjaii__zaan@brroo
Markos A. Katsoulakis
Department of Mathematics and Statistics University of Massachusetts Amherst
markos@umass. edu
Abstract
Through an uncertainty quantification (UQ) perspective, we show that score-based generative models (SGMs) are provably robust to the multiple sources of error practical implementation. Our primary tool is the Wasserstein uncertainty propagation (WUP) theorem, model-form UQ bound that describes how the L error from learning the score function propagates to a Wasserstein-1 (d1) ball around the true data distribution under the evolution of the Fokker-Planck equation. We show how errors due (a) finite sample approximation, (b) early stopping. (c) score-matching objective choice, (d) score function parametrization expressiveness, and (c) reference distribution choice, impact the quality of the generative model in terms of di bound of computable quantities. The WUP theorem relies on Basionn-Jaobi-BBllle
Bernstein estimates for (PDE) and the regularizing properties of diffusion processes. Specifically, PDE regularity theory shows that stochasticity is the key mechanism ensuring SGM algorithms provably robust. The WUP theorem applies to integral probability metrics beyond d1, such as the total variation distance and the maximum mean discrepancy. Sample complexity and generalization bounds in d, follow directly from the WUP theorem. Our approach requires minimal assumptions, is agnostic to the manifold hypothesis and avoids absolute continuity assumptions for the target distribution. Additionally, our results clarify the trade-offs among multiple error sources in SGMs.
partial differential equations
Introduction
Score-based generative models (SGMs) 2 highly effective (3). producing high quality samples, with more stable and less computationally intensive training methods than generative adversarial nets and normalizing flows (4) The models are empirically robust approximations and errors in learning the score function. While SGM generalization properties have been studied for idealized conditions [5]
6,77 analyses of their robustness in practical settings remains underexplored. This paper analyzes SGMs through the regularity theory of nonlinear partial differential equations (PDEs) [] specifically Hamilton-Jann Jaacoi---oB-eaee
(HJB) equations [TO. Our main result the Wasserstein uncertainty propugation (WUP) theorem (Theorem 3.T a versatile model-form uncertainty quantification (UQ)
bound which we use to theoretically explain the robustness of SGMs to approximation errors in
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

practical implementation. Generalization bounds for integral probability metrics (IPMs), such as the Wasserstein- (dj) and the total variation (TV) distance follow directly from the WUP theorem.
In the context of SGMs, the WUP theorem shows how an I,2 neighborhood around the true score function propagates to IPM neighborhood around the true data distribution. By relating how various approximations in SGMs contributes to L eπгоr with respect to the true score function, we establish how well the resulting SGM generalizes. Theorem 3.2 shows how the error in the learned score-function with respect to the explicit score-matching objective, or uniform-in-time L error, and the choice of initial condition define the radii of d1 and TV neighborhoods, respectively. Additionally. Theorem 3.3 addresses the case where the score function is learned from finite data using the denoising score-matching (DSM) objective and incorporates early stopping.
Our bounds capture trade-off's among the errors. The ability of SGMs to generalize depends on choices such as the early stopping time and how overtraining to the DSM objective and neglecting early stopping lead to SGMs that memorize and overfit to the data. Notably, our approach relies on minimal assumptions on the data distribution, being agnostic to the manifold hypothesis. This contrasts with existing convergence results which assume the hypothesis [8] or not 7 a priori. Furthermore, unlike previous work, we obtain our generalization bounds with respect the dj distance directly, withoul appealing to the Girsanov theorem, the Kullback-Leible divergence, the x² divergence, or Pinsker's s inequality [ 7. This suggests our bounds may be sharper than those which bound stronger norms and divergences. A notable feature of our DSM generalization bound is that the error bound is a computable function of the DSM objective, contrasting with prior results typically assume the learned score is close to the truth with respect to the ESM objective (5)
The WUP theorem also enables robust uncertainty quantification (UQ) for score-based generative models, a rare capability in generative models. Robust UQ recognizes that learning complex models involves multiple sources of uncertainty due to modeling choices and imperfect data. The distribu- tionally robust perspective [12 quantifies a uncertainty set based on divergences and probability metrics [13] 15 16 to measure the impact of model uncertainty around a baseline model. The brawback is that these sets are typically difficult to find computationally. The WUP theorem is an example of robust UQ bound for IPMs that is computable.
1.1 Contributions
We introduce the use of regularity theory of nonlinear PDEs for analyzing generative flows p 10..
Our key idea is using the Kolmogorov backward equation to study the evolution of IPMs under a generative flow, yielding generalization bounds. WUP is one particular application of this idea, and while we use it to study SGMs, this analysis is not limited only SGMs. We show that the regularizing properties of the underlying Fokker-Planck equation imply SGM's robustness to errors with few assumptions on the data distribution. An intuitive explanation of our main technical contribution is provided in Section 4
The WUP theorem (Theorem 3.1 which we state and prove, describes how an L ball centered around the true score function maps to neighborhood of IPMs (such as d1, TV, and MMD 17 WUP is a model-form uncertainty quantification bound, which maps uncertainties introduced by modeling choices when practically implementing SGM. The resulting generalization bounds WUP produces demonstrate that with properly chosen model parameters, SGM is robust to errors due to score- matching, finite sample approximation, early stopping time, and choice of the reference distribution. (Theorems 3.2 3.3, and C.I).
When applied to SGM, WUP produces generalization bounds under minimal hypotheses the data distribution and score function, and explains the impact of implementation errors has on generalization. Our approach is agnostic to the manifold hypothesis, applying whether or not the data distribution has density. Moreover, it is adaptable to additional assumptions about the target distribution to yield improved generalization bounds.
1.2 Related work
Convergence and generalization of SGMs have been well-studied. Many approaches 7 assume the target distribution is absolutely continuous with respect to a Gaussian, and obtain generalization bounds for TV, x2, and d, by bounding the KI. divergence, a stronger divergence, via the Girsanov
theorem [ 6 Pinsker's inequality (5), functional inequalities [7]. or other means | 19).

---

While (20] shows that kernel estimators for the score function are minimax optimal distribution estimators, they make no connections to deep learning-based SGMs, and they assume the target distribution is Lub-Gaussian and is in Sobolev space. Meanwhile, [ has comprehensive sample complexity results that considers neural net approximations in the finite sample regime. Their results, however, are specialized to distributions in Besov space, and, similar to other work [5] 11 19] rely on bounding the KL divergence, which breakdown under the manifold hypothesis.
Our results are similar to g which derives di bounds directly, proving SGM convergence under the manifold hypothesis. However, the analysis assumes a particular discretization of the SGM. 21I. an uncertainty propagation bound for the Wasserstein-2 distance is proven using similar approach this work. Their work, however, relies on strong, uncheckable assumptions on the score function and target measure, does not address the errors we study here, and is not extendable to IPMs.
2 Background and notation
Let dimension d E N, and terminal time T > O. Denote T C Rd to be the unit torus and RT Rd to be the torus of radius R > O. Let P(2) be the space of probability distributions on Q2 where Ω IRd or RT4. Let E (S2) be the target data distribution, known only through a finite number of samples J. π. Let f [0. 7] x Rdd I be a vector field and o R + R be a positive function. Score-based generative modeling M is based on considering two Stochastic Differential Equations (SDEs for short) whose flow maps are inverses of each other. Define Ys)). (t)) be the
forward and reverse diffusion processes over time interval s,t E [o, T that evolve according to
(s) (T 8, (s)) + (T s)dW(s), (0) π
dx = [f(t, + (()) (T i, (()) dt + (t) X(0) ~ mo.
(2)
where (()) ~ (s. ), and (t) is a standard Brownian motion in Rd Heres, 17" -) is the density
of (s) time where the initial distribution
æ. From 122). it is known that if mo 17 (T -). then X() ~ "7 (T (T-LL-). SGMs generated new samples from π simulating trajectories of the reverse process (2) using an approximate score function Sø s = log 1y" Typically and
chosen such that 17 (T, )) is well-approximated by a normal distribution, which is then used as the initial distribution.
The score function is Iearned via samples {2 from क by minimizing one of several score- matching objectives. Let sy be some function where the parameters 0 are learned via one of three score-matching objectives. For a path p: [0. T + P(2), we define the functionals
T / |se- V pl dp(s)ds J2(p 1-" / (Isol + 2V se) dp(s)ds, (3)
T(p
where the subscript L highlights the linear dependence of Jr. on the underlying measure p. The explicit score matching objective (ESM) is JFSM(17", 0) = I [n"", 0). As evaluation of the true score function V log 17t is typically inaccessible, the implicit (ISM) (1" .) J2(17, 23 or the
denoising (DSM) morrematchhnng objectives [24 I are computed in practice
/ So Vlog no Jo Jn J
p dif'' (s)dm(x')ds
JOSM(1)) (
,0)
Here 17" (s) denotes the probability transition kernel from x'' to of ① at time The DSM objective is most frequently used in practice as does not require computing derivatives of sg. It does, however, require the evaluation of 17 (s) in closed form, which is typically only accessible for linear SDEs.
Remark 2.1 (Choice of domain 2). While our approach will also produce bounds when f2 Rd we primarily focus on the torus RT which is equivalent to bounded domain with periodic boundary conditions. This choice ensures that the long-time behavior of the noising process ① converges to the uniform distribution on RTd, Therefore, we set f and at) = / instead of using the Ornstein- Unlenbeckk process. We make this choice for mathematical clarity, however our results generally apply, with minor modifications, to the entire space R and for other noising processes.
2.1 Equivalence of mcore- objectives in the finite sample regime
Finite sample approximations of ESM, ISM, and DSM are not generally equivalent. However, 24 25), show that DSM becomes equivalent to ESM and ISM in the finite sample regime when
3

---

(s) is replaced with its kernel density estimate (s), where 17 (0) = - - E Sz,- The kernel estimate, however, does not have well-defined score integrated only for E (,, T. an example of early-stopping in SGM [ this has the effect of morehnn for the mollified distribution N.. π transition probability kernel for + Then it can be shown that
so the DSM objective is often In continuous time,
26
* Pc. where Pe the
I "n 0) = (1) (,,0) JESM(nn Nee 0) = JIsM(1) +0)+ 4 V N
(4)
3 An uncertainty quantification approach to generalization in SGMs
We describe our UQ approach to generalization in SGMs and overview our main results. Discussion of the proof methods is deferred to Sections 4 and S Our primary goal is to study how practical and approximation errors made in implementing SGMs translate into errors in the generative distribution relative to the true distribution.
3.1 Source of errors in score-based generative modeling
We attribute errors of SGM the following six sources:
Data distribution is only accessible via samples e1: The target distribution is only known through a finite set of samples. In practice, the regularity of the score function and data distribution, such as Lipschitzianity or whether the distribution has a density, is unknowable.
Choice of score-matching objective e2: In practice, acore-matching objectives are approx- imated via samples. The DSM objective is most frequently used as it avoids computing derivatives of the score function. While DSM and ISM are equivalent given the exact density evolution 7/(1,, s), they differ when approximated with finite samples. Previous analysis typically assumes that the learned score function is close to the true score function within some 2 distance, i.e.. ball determined by JESM In contrast to previous work [T [5, we show in Theorem B.5 how training through DSM translates into bound for ESM. Moreover we prove Proposition B.8 which states that there exists a neural net that well-approximates the true score then that the target density is necessarily regular.
Expressiveness of the score function e3. The expressivity of neural net approximations to the score function will depend on the particular expressivity of the parametrization.
Choice of reference distribution c4. With access to the exact score function, the denoising process 2 produces trajectories that sample from π only if the initial condition starts at (,, (F.T)) T). practice, however, the initial distribution is usually a Gaussian approximation 1/² or, in the case of the OU noising process, its corresponding stationary distribution.
Early stopping of the denoising process eg. practice, the score- motehin objective is integrated from s E [c, T'. for small e, instead of s = 0 [ 26). This prevents the SGM from memorizing the training data (27 228 and is equivalent to rumning the denoising process for 1 c]. Early stopping is cracial when the data distribution is supported on
a low-dimensional manifold, where it has no density with respect to Lebesgue measure. Previous analyses of SGM often adjust to optimize generalization bounds [6] 20
Discretization error Eg- The denoising SDE must be solved via numerical scheme. Previ- ous work (5) T g have considered the impact of discretization error on the generalization abilities of SGM. While our analysis does not consider the discretization error, our approach can be extended to study discretization error through the use of modified equations [29].
3.2 Model-form uncertainty quantification
Let be an integral probability metric (IPM) between measures V1, 12 P(22)
/ (())d(122 - v1), DEX
d(v1,22) = sup
(5)
'Here, denotes convolution.

---

for some function space X. We assume X' = { Ω R. V = fb: : | S 1}. corresponding the Wasserstein-1 (d,) and total variation distances, respectively. Recall that M1 have densities, then their TVV distance is equivalent to the I,* distance between their densities.
Let be the target data distribution and take the stationary distribution of the noising process on the torus (0) = c as the initial condition. Given learned score function Sø, let m, (T') be the generative distribution produced by SGM. We study how IPMs between л and my (T) relate to errors from the first five sources of errors discussed above, i.e., we seek bounds of the form
d T) S F(e1, e2, E3, C1,
(6)
Our key insight is that for score-based generative modeling, we can derive bounds of the form (6) via a model-form uncertainty quantfficatin bound for the generative Fokker-Planck equation. The Wasserstein Uncertainty Propagation theorem formalizes this insight.
3.3 Wasserstein uncertainty propagation theorem
The WUP theorem is a general statement about how L neighborhoods the space of drift functions map to neighborhoods in ((2) defined by IPMs.
Theorem 3.1 (Wasserstein Uncertainty Propagation). Let Ω = RT or RR. Let bb, ,,", [0, x - Rd be given with || [|Vb'l 1| < se, and mı, m2 E P(2). If m for i = 1,2 are given by dem' Am' div(m'b') 0, m m,
then, up to universal constant C > O, we have the following:
(00(00--)((0))) (t,(t,z)) m² (t,)))ddr) 1/2
. If sup 051ST then
11 ((62-b')(tlll) ())(()))))())0))
sup USIST
Sen,
d + vTai).
1
-mn(t)-m'(T)llluu) CCVTTVVb'l> +1)
(
(8)
and
- CCVTIVbbll +1) - + vTei).
(9)
1/2
For Q = if 16 -6- (2)2)()) = 110 106 (6)(t))) (t, (t,
S
52
then
di (m) (T)) S CRE(1+
VIV6 (loo) (di (m), 52)
(10)
Equation
88 is particularly notable result as we bound the TV distance between m (T) and m mT
terms weaker d, metric between m and m2. This is due the regularizing effects of diffusion processes, which is showcased in the proof. See Section 4 and Section A.1 for full details.
3.4 Robustness of errors under ESM
We use WUP to produce generalization bounds when the only errors due to the choice of the initial condition and the approximation of the score function with respect to the ESM objective.
Theorem 3.2 (ESM bounds). Let π E P()) where Ω = RI for some R > O. Moreover let cnn 0. Assume that the learned score function So is such that T (17", 0) S enn- Then for by So t, t, the generated distribution my (T) = π where
1
O.m, - Amg - div(m,bo) = 0 in (0, T) x Ω m2 (0)
in Ω
(11)
vol(RTTT)
satisfies
di di( (T, (T)) CRå(1
(1+ vIVvollb>
Re WI ( voll RT
+
een Score function ervor (ea)
Error from choice of reference measure (E4)
(12)
5

---

If the stronger estimate sup 0S1ST
V Vlog(n") (12((7))22((7((1)) S Enno holds, then
RCc T di (T,
(mm (1) S C(VT) +1)
vol(RT4)) + VTemn) (13)
VT
Here, the constants C, w depend only on the dimension d.
Applying WUP for b² and b² be the and learned score function, respectively, with Proposition D.3 on the convergence of the noising process to the stationary measure immediately yields the estimates above. Notice that the TV estimate (13) is comparable Theorem 2 of [5]. which also assumes a uniform-in time accuraaee score function. Again, notice that we are able to bound the TV distance between my(7) and æ using the weaker d1 distance between æ and vol(
3.5 Robustness of errors under DSM
In practice, the score is learned through samples using the DSM objective with an early stopping parameter 26) SGM is effective at producing samples from distributions supported on (or near) low dimensional manifolds. Our di generalization bound describes how early stopping aids in generalization.
Theorem 3.3. (Pointwise DSM generalization) Let by = so and my [0,7] HT + R be N.e,
given by (TT). Assume the learned score function is such that (15 ,0) [,p 0 < cnn-
Let 0 < 6 < be such that 5 S a N.e, 5 n². Then up dimensional constant C C(d) 0,
di (T, (g))
Ve +R [0)
Re x² d1
+ Vven cnn
(14)
"ol((
Early stopping (eg)
Choice of reference measure (14)
where
log Ve
nn
Con
1+
+ T|
di(
(15)
DSM score function eroor (ea)
Finite sample evror (es)
Choice of SM objective (62)
The pointwise DSM generalization bound applies every finite training sample of size N. A crucial part this theorem relates the error in the practical DSM objective function to the ESM error needed to apply the WUP theorem. We connect the DSM objective early stopping to the ESM error with the mollified distribution ग in Lemma B.3 This result is agnostic to the manifold hypothesis for 0. Such bounds will blow up if the KL divergence previous results that use the KL divergence to bound the TV distance [5] 19 6 will produce vacuous bounds for the d, distance under the manifold hypothesis their d, generalization bounds are derived by bounding the KI. divergence.
π, long
used instead. In fact,
Remark 3.4 (Density lower bound). Similar to (6), our DSM generalization bound relies on density lower bound assumption. We note however, that our DSM generalization bound holds for any random sample { ~ w. This density lower bound assumption can be removed via Jensen' inequality if we consider the expected dj distance between π and mg over random empirical measures. See Theorem C.l cnd its proof in Section
Remark 3.5 (Trade-offs and memorization). Theorem 3.3 captures trade-offs when training SGMs and memorization effects. To minimize the error from early stopping, we can let e + 0. However, empirically, without early stopping, SGMs overfit to the kernel approximation and memorize the training data (27 28 25). The bound is vacuous when = 0 regardless of whether the distribution small implies that the
lies on now-dimensional manifold. As c <- O, training the DSM score function must approximate a rough function with large Lipschitz constant, which will increase the bound. This shows that overtraining the DSM objective may not necessurily yield a better generative model. Moreover, suppose that N
then as + O and ean - 0, we have that di (T, my (T)) > 0, indicating the model memorizes the training data. Our results corroborate those of (25y.

---

Regularity theory of Hamiloon-Jacoi-Bellaan PDEs enables uncertainty quantification in SGMs
A recent result in [30] established connections between generative flows with the theory of PDEs, more specifically the theory of mean field games. We continue investigating these connections and showcase how may study generative modeling algorithms by obtaining stability estimates for the Fokker-Planck equation. We provide proof sketch for our Wasserstein Uncertainty Quantification theorem (Theorem 3.1). Our strategy involves (1) constructing test functions for the IPMs using the Kolmogorov backward equation, (2) choosing the suitable function space for the terminal data depending on the desired IPM, and (3) obtaining gradient estimates of the test functions via Bernstein estimates. The theorem relies on the gradient of the test function remaining bounded for / E [0, T'), which is guaranteed by the regularizing properties of diffusion processes. See A. for full details about the proof.
4.1
Kolmogorov backward equation determines suitable test functions
From we aim to compute bounds for (m (T'), (T)) su)e(e)ex (mmmm (m) (T))
The measure À m m2 satisfies the PDE
0, A div + m² b²)) = 0 in (0, T) x Ω x() = m2 m2 in Q
(16)
Let o: [0. + be a test function in space and time. We integrate in space and time against the PDE T6) and integrate by parts to move the derivatives on to which yields
I n
(-' A(-0r0- Ao+ 6' Vo) dædt 1'"
x(T,)) X(O, 2))(0, x)d
(17)
T J. m (b² ba) b²)
Notice that if we choose the test function & to satisfy the Kolmogorov backward equation (KBE)
- AΦ + b' Ve = in [0, Ω ΦTT,x) (xx) in Ω
(18)
with terminal condition vo E X, then from (17) we have
[ X(T,
x(00 x))r S' m²(()Vo(())) (62 ( (I)
(19)
The equality above is valid for any terminal condition vb. Depending on the choice of function space X for v. we obtain bounds on different IPMs. Taking the supremum over X' we have
d(m m² (T))) S sup [ *0,#))(0, x)d VEX
['] V (b² bz)dded
sup sex
(20)
Recall that ò is related to & via the KBE (88) To obtain bounds for d, need to bound the two terms in 20). which depend on the choice of function space X' and assumptions on the drift terms.
4.2 IPM bounds depend on choice terminal function space and gradient estimates
We split our proof sketch into two parts; the first part focuses on deriving L estimates, while the second derives d, estimates.
L' estimates. We first note that because the regularizing properties of 18 we can obtain bounds on Jg x)00,))(00,1 r) with weaker nonms c. Notice that
[ A(0, (00,))(0,z))dz S [|A(0]]>~||(0))
(21)
where denotes generic space functions, and y" is dual. what follows, we show that regularizing effects of (18) takes functions in X' to functions with more regularity y such that ) is compactly embedded in X. This in turn implies that || y weaker than d and so we are able to bound the stronger norm d by the weaker norm.

---

To prove (8. we d = -| 1. (0), which corresponds with X' = { St - R. 11 S 1}.
Observe that we can take y = { Ω + R, V S 1}. in which case we obtain
J ((000))(00 dd = [V 20 / (m1 m2)
Φ(0, x))
(22)
V (0(0, 100
S <m m2)
Vo(0, xz Iloo
Notice that this bound crucially depends on showing that (60, is Lipschitz.
To prove ( we can simply take X' = y and obtain
[ 1(0 (0) x) v ||xO0, (x00,)))lccaall000==))).).
(23)
For 88 and 99. Schwarz on the spatial integral shows the second tern can be bounded as
/ m V (b² (')dddd s
!-" I1(bl
(24)
a
I1(b* (0)(()) / [[ llo
S sup 0S1ST
Notice here that crucial to produce estimates for VΦ.
Wasserstein-1 (d,) estimates. To prove (T0 we choose X' = = (b: |Voll S 1]}. the space of 1 1-Lipschitz functions. We obtain (22)
again, except the terminal data &p also has Lipschitz constant equal to 1. Applying Cauchy- -Schwar in space and time implies the second term is bounded
[/ m (62 (b--b'dade b')dzdt S (1." / | [ dodt
|6 - WM1-b21laaGE 1bllL2a(m)
(25)
ST sup V )1b= USIST
b² blll
We again see that we require estimates for V and we need them to be bounded.
4.3 Bernstein estimates from HJB theory provide gradient estimates
To obtain estimates for Vé, we could differentiate (88) to derive PDE for VΦ. The resulting
PDE, however, will have (V)) grow linearly with Vø, and so if we apply (reverse) Gronwall's inequality, the resulting estimates for Ve will grow exponentially time. To avoid this exponential time dependence, we first perform a Hopf-Cole transform u = log á on (18) ) to derive the HJB equation for u
O,u Au +
Vul² b Vu 0, u(T, 2 log((()z))
(26)
2
Here provide an example of classical Bernstein estimates (Proposition D.I and Corollary D.2 to
obtain bounds for V without using Gronwall's 's inequality [10]. The main idea is to derive a PDE
(60) for the function with Vu. Then by showing that the function w((, = Cu for sufficiently large C attains its
/ Vul2 by taking the gradient of (26) and then taking the inner product
maximum (o), can show that
2tt, x) I (T, + Cu(t, 2)) S 12 log 1/ + Col2 /o + CC|lulloo
(27)
2
Using the maximum principle for (18), we find ||u||>0 = |2 log 11 yielding z(t, x) S C'Il log el. Assuming that o is Lipschitz continuous implies boundedness of and therefore VΦ for all time. A similar result holds when d E _ only (see (59)). Detailed proofs and related boundes are provided in Proposition D.I Applications of the bounds to derive the WUP is provided in Section A.
Remark 4.1 (The regularizing role of stochasticity). In our analysis, the stochasticity in SGMs provides two types of regularizing effects. The first is early stopping, which adds a small amount of Gaussian noise of the data distribution. This, in effect, is equivalent running the noising process
8

---

for a short amount of time and immediately mollifies the initial empirical distribution so that it has a smooth density. Second, the Laplacian is the key mechanism that regularizes the test function in (8)) M9, which then allows to bound, for example, the stronger || 11 norm by the weaker di-norm, Recall that in PDE theory, the stochasticity of generative flow manifests as the Laplacian operator in the Tokker-Plance and Kolmogorov backward equations 19). Without the Laplaciann the regularizing the test functions in (18) would not be possible in general and, in fact, we would not have access to long time behavior results.
5 Proof sketches - Scoor-bbaeed generative models are robust to errors
We now provide sketches of the proofs for the main generalization bounds in Theorems 3.2 and 3.3 The full proofs are provided in Sections A.2 and A.3 respectively.
5.1 Theorem 3.2; ESM generalization bound
Theorem 3.2 is generalization bound with respect to error from the score function approximation with respect to the ESM objective (ez) and the choice of reference measure (e1). Assuming we have an (uniform-in-time) -clclose approximation of the score function, first apply the WUP Theorem 3.1 with mh = To[[ and m2 17" (T, .). The distance between mh and m2 can be expressed in terms of æ by studying the long time behavior of periodic solutions to the heat equation. Proposition D.3 shows the heat equation is a contraction under d1. Applying it to m and m² yields the desired result.
While Theorem 3.2 has no explicit assumptions on म, the assumption that the approximate score function Sg is close in L22 to the true score implicitly implies regularity of π. Specifically,
Sy is Lipschitz (which is true for neural networks in practice) and satisfies [Tz 0) < Cnn, then must have finite entropy. This implies that π necessarily has a density. See Proposition B.8 for the formal statement and proof.
5.2 Theorem 3.3: Pointwise DSM generalization bound
In practice, the score function is learned via a Monte Carlo approximation of the DSM objective (2 To avoid overfitting to the kemel estimator and memorizing the dataset [25), the time integral is taken only over E [e, T] where is the early stopping parameter. This is equivalent to training with the ESM objective with the true score replaced with the kemell approximation at time (20 27
To derive generalization bounds of SGMs trained via DSM, we establish the relationships between (1) the mollified distribution o' I(c) r and the true distribution π, and (2) the DSM objective JDSaS (1) 0) = [(n N.5, 8) and the ESM objective with respect to the score function of the mollified distribution J J(1 .)). Formally, this strategy involves bounding the following two terms
di(T, (T))
di TT)
( my (T)).
(28)
Generalization bound w.r. true distribution
Early stopping error (es) Generalization bound w.r.t. mollified distribution
For the early stopping error, we use the regularizing properties of the heat equation to show that di (IT, x²) S C-V,, where C' only depends on the dimension d. This implies that, measured in d early stopping only incurs a nominal C've error even if the π does not admit a density. This result would not possible if we were to study generalization error in terms of KL or TV directly.
bound for the second term is obtained by comparing the ESM objective value between the learned score function and the true score function of the early stopped distribution, I J(n"", O) the DSM objective I M, 0). We present Theorem B.5 and its corollary, which under the assumption that N, and 17 have lower bound 8, state that if J M,, (s)) < Cans then J(n .8) < 6nn with
é, = Can + C11 log( Ve
1
+
+ T (B= (I .T) VT
(29)
The main idea behind Theorem B.5 is to note that the difference between the ESM and DSM objective functions can be written as a difference in ISM objective functions plus the entropy difference between NyNie and 17"" Details for bounding this term is provided in Proposition B.2 Lemma B.3
To arrive at the final result, apply the WUP theorem to derive generalization bounds for dı (T., my (T)) under the assumption that [(n²", 0) < enny along with (29). Finally, combine this result with the error due to the early stopping d, (TT, x²). Full details of this proof is provided in Section A.3

---

6 Discussion: PDE regularity theory and UQ for generative modeling
Our main contribution is the study of generalization in score-based generative models from the perspective of uncertainty quantification. The regularity theory of nonlinear PDEs is the key technical tool that produces our results. We emphasize that the tools we use here can be used generative models beyond SGMs, and that we have not pushed our analysis to the limits of our tools in SGMs. Moreover, we also emphasize some downstream UQ applications for SGMs that may be of future interest.
6.1 The significance of the regularizing properties of SGMs
surprising result of our work is deriving bounds for the stronger L' distance in terms of the weaker
di distance (see (22) and Theorem 3.2 The key insight (Equation (2)) is that the evolution of observables defined by the KBE (8) regularizes the test function. We have not fully exploited the only focused on L and dı estimates.
regularizing effects of (18) in this paper,
Improved bounds in Sobolev spaces H To illustrate other extensions and choices of y in 21. observe that in the trivial case when b22 = 1 = 0. (99) simplifies to ['T x ())(())))))))))))))) (z) = J [ (TT() * +)) where I' is the heat kernel. If || lo S 1, then I' * (T)) E C and we have estimates of the form di (mm-(T),, (T)) S C(s, T, [| -m -=2l==- for all E N. When b², be are not identically zero we still expect such estimates, though they will depend on the regularity of b². To highlight the importance of regularizing effects, note that by [3I] if π E P(TT)) then for the empirical measure n,,, we expect di (T, x N S N However i 2' || - n I S v
This suggests that improved regularity may influence overcoming the curse of dimensionality.
6.2 A connection to likelihood-free inference
Computing expectations with respect to posterior distributions is key task in Bayesian inference. Generative modeling, in particular, has a key role in future developments of likelihood-free inference 35). For generative models to be trustworthy for inference, they must to be shown to be robust. The WUP theorem provides error bounds for approximating expectations with respect to some true unknown distribution, and may be significant for SGMs in likelihood-free inference.
32 33
34
For example, suppose we wish to estimate F for some distribution æ and observable h, and an SGM ma (T) is constructed to approximate the expectation. Bounds of the form (6), such as the WUP theorem, translate into guarantees on the expectations:
Erth (ryh S d(m,((T), x) S F(ee, E2, c3, 61,
(30)
In the context of SGM, the incqualities are a posteriori bounds, meaning they can be computed after learning the model. Additional regularity on h may yield improved guarantees.
6.3 Enabling distributionally robust optimization (DRO)
Robust UQ methods are based on the perspective that learning any complex model will typically involve multiple sources of uncertainty due to modeling choices, model reduction or learning from imperfect data. These uncertainties are not just in parameters but are inherently present in the mathe- matical model itself and will propagate to any predictions. There is substantial related work in recent years using distributional robustness perspective, [12), using divergences or probability metrics and their variational representations to quantify the impact of model uncertainty around a baseline model that may be either learned (e.g. generative model) could be just an empirical distribution from an unknown true distribution. The approach can generally be described as quantifying an uncertainty set around the baseline model that the worst-case distribution belongs to via some neighborhood defined in terms of a probability divergence [3 14) a Wasserstein distance [15] or maximum mean discrepancy [16). There are, however, drawbacks to each of these approaches: a divergence ball contains only distributions with the same support as the baseline, while the uncertainty set may be hard to determine practically. The WUP Theorem is related robust UQ notion where the uncertainty ball is in an IPM, e.g. 1-Wasserstein, MMD, or TV.. However, it allows us to bypass the robust UQ for stochastic processes relying on restrictive, path-space probability divergence- baasdd approaches or Girsanov's 's Theorem, [36] 377. Furthermore, the WUP Theorem bounds use PDE theory to provide a computable uncertainty set, as we also demonstrate in the case of DSM, see Theorem 3.3
Acknowledgements
M.K. and B.Z. are partially funded by AFOSR grant FA9550-21-1-0354 M.K. partially funded by NSF DMS-23077 L15 and NSF TRIPODS CISE-1934846
10

---

References
[1] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13466, 2020.
[2] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[3] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances neural information processing systems, 34:8780-8794 2021.
[4] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs, arXiv preprint arXiv:21 2021.
[5] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy learning the theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11211 2022.
[6] Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models minimax optimal distribution estimators. In International Conference on Machine Learning, pages 26517-26582. PMLR, 2023.
[7] Holden Lee, Jianfeng Lu, and Yixin Tan, Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems, 01-22870-22888 2022.
[8] Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. arXiv preprint arXiv:2208.00 14, 2022.
[9] Lawrence C Evans. Partial differential equations, volume 19. American Mathematical Society,
2022.
[10] Hung V Tran. Hamilton-lacobi equations: theory and applications, volume 213. American Mathematical Soc., 2021.
[11] Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning, pages 4735-4763. PMLR, 2023.
[12] Hamed Rahimian and Sanjay Mehrotra. Distributionally robust optimization: A review. ArXiv, hhs/1908.05663 2019.
[13] Kamaljit Chowdhary and Paul Dupuis. Distinguishing and integrating aleatoric and epis- temic variation uncertainty quantification. ESAIM: Mathematical Modelling and Numerical Analysis, 47(00):6335666 2013.
[14] Jun ya Gotoh, Michael Jong Kim, and Andrew E.B. Lim. Robust empirical optimization is almost the same as mean-variance optimization. Operations Research Letters, 46(4):448-452, 2018.
[15] Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations, Mathemati- cal Programming, 1151)1155-166, Sep 2018
[16] Matthew Staib and Stefanie Jegelka. Distributionally robust optimization and generalization in kernel methods. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[17] Erich Novak, Mario Ulirich, Henryk Wožniakowski, and Shun Zhang. Reproducing kernels of Sobolev spaces on Rd and applications to embedding constants and tractability. Analysis and Applications, 16(05):693-715, 2018.

---

[18] Giovanni Conforti, Alain Durmus, and Marta Gentiloni Silveri. Score diffusion models without early stopping: finite Fisher information is all you need. arXiv preprint arXiv:2308. 12240, 2023.
[19] Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of score- score-based generative modeling for general dat a distributions. International Conference on Algorithmic Learning Theory, pages 9669885.M0 PMLR, 2023.
[20] Kaihong Zhang, Heqi Yin, Feng Liang, and Jingbo Liu. Minimax optimality of score-based dif-- fusion models: Beyond the density lower bound assumptions. arXiv preprint arXiv:2402.15602 2024.
[21] Dohyun Kwon, Ying Fan, and Kangwook Lee. Score-based generative modeling secretly minimizes the wasserstein distance. Advances Neural Information Processing Systems, 35:20205-20217, 2022.
[22] Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982.
[23] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pages 574-584 PMLR, 2020.
[24] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-1677) 2011.
[25] Jakiw Pidstrigach. Score-based generative models detect manifolds. Advances in Neural Information Processing Systems, 35:35852-35865 2022.
[26] Yang Song. Score-basede generative modeling through stochastic differential equations (sdes)- Py Porcch implementation. htttps://githu: ottpata..ccc.o...oooooooooooooor 2020. Accessed: May 2, 2024.
[27] Sixu Li, Shi Chen, and Qin Li. A good score does not lead to good generative model. arXiv preprint 1eXiv:2401.048855 2024.
[28] Benjamin J. Zhang, Siting Liu, Wuchen Li, Markos Katsoulakis, and Stanley J. Osher. Wasser- stein proximal operators describe score-based generative models and resolve memorization, 2024.
[29] Assyr Abdulle, David Cohen, Gilles Vilmart, and Konstantinos C Zygalakiss High weak order methods for stochastic differential equations based on modified equations. SIAM Journal on Scientific Computing. (4(3):A1800AAB
2012.
[30] Benjamin Zhang and Markos A Katsoulakis. A mean-field games laboratory for generative modeling. arXiv preprint arXiv:2304. 13534, 2023.
[31] Nicolas Fournier and Arnaud Guillin. On the rate of convergence in wasserstein distance of the empirical measure. Probability theory and related fiedss, 162(3):707-738, 2015.
[32] Ricardo Baptista, Bamdad Hosseini, Nikola B Kovachki, and Youssef Marzouk. Conditional sampling with monotone GANs: from generative models to likelihood inference. arXiv preprint 27Xiv:2006..06755, 2020.
[33] Ricardo Baptista, Lianghao Cao, Joshua Chen, Omar Ghattas, Fengyi Li, Youssef M Marzouk, and J Tinsley Oden. Bayesian model calibration for block copolymer self-assembly: Likelihood- free inference and expected information gain computation via measure transport. Journal of Computational Physics, page 1 112844, 2024.
[34] Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnad Doucet. Conditional simulation using diffusion schrödinger bridges. In Uncertainty in Artificial Intelligence, pages 1792-1802. PMLR, 2022.
[35] Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with bcore-based generative models. arXiv preprint arXiv:2111.08000 2021.
12

---

[36] Paul Dupuis, Markos A Katsoulakis, Yannis Pantazis, and Petr Plechác. Path-space information bounds for uncertainty quantification and sensitivity analysis of stochastic dynamics. SIAM/ASA Journal on Uncertainty Quantification, (1):80-111, 2016.
[37] Birrell, Jeremiah, Katsoulakis, Markos A., and Rey-Bellet Luc. Quantification of model uncertainty on path-space via goal-oriented relative entropy. ESAIM: M2AN, 55(1):331-16, 2021.
13

---

A Proofs of main results
Throughout the proofs we will make frequent of the fact that I': |0, 00) Ω IR denotes the heat kernel f Rd 52 RT then there exists dimensional constant C C(d) 0 such that
vT * I |1g1|== VE
(31)
Furthermore, recall that constants subject to change from line to line but always maintain variable dependence as in the corresponding statement.
A.1 Theorem 3.1
The proof of Theorem 3.1 follows the strategy outlined in Section 4 and uses the different gradient estimates from the HJB equations provided in Section D.1
Proof. Let X = m - -m², which satisfies
( 0₂₂ - AA - div + m²( - b1) = 0 in (0,T) x RT4,
{ (A()) - m. m2 in RTd.
(32)
For fixed sb : Q + I, for which we will specify bounds latter, let Φ: [0, x 82 + R be given by
- - Ao +b V in (0, ?
(33)
(o(Tx)= (xx) in f2
Testing against in 32 and using
obtain
/
x(Txx)v(z)dr X(0)0(0, x), - L." / (m) (62 - (s)ddds...
(34)
Proof and 99) Assume that || S which then implies
-1 5 Φ 1.
Since [d)(t) = 0 for all / E j0, 7] up adding a constant we can assume without loss of generality in 34). that
S v s 3 and 1 S Φ S 3.
Then from estimate 63 in Corollary
D.2, we have
(T - ( & Vbllo + 1)
|V S C V +1
VT-t
Next, we note that
f ((T,x))(()dd x(0)0(0,z)dz - 1-) ( (6> (s))
< di dm m²mmm1111 -31111/0011+++///// /'' ( (
(f
ds
11(b2 - b1) /' ||Vo(s)lloods
S ( + sup oSeST
d (mmm mm²)
<C +1)
+ VT
sup 051ST
|1 - b²) (())l))))))
VT
Taking the supremum over 1/ 1 yields
Illm I (T) - m (T))l S C(VTTVVb' blo0 + 1)x
di d1 m²)
+ VT
||(b2 (6))())l
sup 0S:T
VT
14

---

Finally, if instead we used the bound
/ x(0))(0, x)d I [lllll I C m2 (2))
we obtain
Proof of 10 Let ? = RT and consider db with V Io S 1. Without loss of generality
we can assume (0) R + and thus
((z) = (xz) - (0)) + (0) 2 - 0 + R 2 R+
and similarly
(xx S 2R + 1.
Therefore by maximum principle have that
S @ x)) 2R 1,
and so
lblle = 11 Vi c R).
From
34 we have
f A(T,z) < d, (m, m ( + sup
Vo(t)]l<]] - (mm))
OStST
Thus by Corollary D.2 estimate
62
obtain
J XTT S <CL |b ||01|1 (d (mi, -lbb' b¹ (1))))))) -o'l--((
CC+
Vb (d) (m), m2) |bb 6² (b
which after taking the supremum over ||V I yields (10).
A.2 Theorem 3.2
The proof of Theorem 3.2 follows by application of Theorem 3.1
Proof. Let = Sa, mi
[) and 12 = V log(1" m2 = 17 (T). Note that m i = 1,2
solve
Ommm Am' div(m'b') = 0 in [0, x RT (m) m, in [0, x RT
(35)
then m² my. m²) (T t). Thus, by application of Theorem 3.1 we obtain
(T)m)(T),x) CR (1 | (dl vol(ReTa)) (d((vo[(/TT) "7") + ||so Vlog(1")ll<a0n)
< CCR V (Re _े vol((TT)) venn)
where in the last inequality used Proposition D.3 3.1 and the fact that
The second claim follows again by Theorem
d (2),
sup Villas
gd(n vi) s R| ll...
mma
15

---

A.3 Theorem 3.3
The proof of Theorem 3.3 follows the same strategy as in Theorems 3.1 and 3.2, The main technical
difference is that we only have a bound between our generated score function Sy and the score function generated by the sample (*) of the form
V /
S Cnn
and so we need to produce a bound between Sy and the true score function V Iog(("). This last step shown in Section and in particular Theorem B.5
Proof of Theorem 3.3 Consider the function 17t [) x RT + R given by t
Pen An in (O,T) RT
(36)
(0)= - RTT.
Define the drifts
bg. (t, x:) 80- (T' t,
and
bz ( )T - (
x)),
and let m² (t, (T x, which satisfies
{ 0 - Am - (iv b²' (t, = O, m 17T
(37)
Finally, consider the distribution of our generated sample my :00,T] x S2 + R which is given by
0mmg Amg do-(mybo-) = 0 in (0, x 22 (0) R in Ω2
(38)
We have the following
di (T)) < 7 + di(T', my (T))
(39)
We will bound each of the terms separately.
1. We recall that x = ((e) where I' is the heat kernel. Therefore, by the dual formulation di we have that
d (T, = sup
/ )(z)d(m +)= sup
(((()) * x g(x))())))))))) (d)
||
Vall
Although estimates on (T(c) g(x) - g(x)) for a function | 20 S are classical for the
readers convenience we provide quick proof. Let u(t) [(t) * g. which solves
00.0 Au= (0)) = g-
(40)
Then for all x E Ω
E [' [a,((s,)) x)lds= = /"' |au(s
ff ( (V) s V / Vi s Sove. Ové.
Where the above we used estimate
31. Thus, we have d T < OVe,
where C C(d) 0.
16

---

2. First we note that x" 3mmmmm00 (T). Thus applying Theorem 3.1 for
m Mg b¹ b
m² = =mmmm ,,,² = ba
yields
(T),,(T), = di(m)(T); mi (T))
R (1+ + VV (m (0),
vol(HT44) ) + vT
ba- ll (l(m)))
By Proposition D.3 we have
di (m (imm((0) (0), voll)) di (1)11=(()), vol(RTT)
SCRe
R di (nT
vol( ROT()
SCR
R²
Finally, from Theorem
B.5
l
ba- [122(m)) [(1" , S e'.
Cnn + c(I+ [og(())
VT += d1 d1(TT
Ve
Therefore, if 2 1, putting everything together, we have shown that up to dimensional constant
dI(T, dI(T, s C V R² (1+ v1lVb'll2)> x
R² = + + T fon+t(1+ (1+ log(o) Ve
T|
di (T m7)
B Analysis of Score Matching functionals
In this section, we gather all the technical results regarding the connections between ISM, DSM, and ESM score matching functionals. For the readers convenience we recall some of our notations. Let c dd and mo E P(2)). We will denote by p [0, T x Ω + [0, oo) the solution of
@0.p Ap t = 0 in (0, 7) x Ω (m (0) my in Ω
(41)
and may drop the superscipt mo when it is clear from context.
First we state a simple result that justifies the formula
J0p0 0) JD(p" 0) + 4 mmo
(42)
which formally follows by
[(. = (-) / (Isoldp(()
25g V log(p) + V log(p))3) dp(s)ds
II(op, 1-' /
P drds
= -I(p) + Vpl3.
However / may not be finite, thus we record the exact statement along with some technical facts for later use in the following Proposition.
1/2
17

---

Proposition B.1. Let mo be probability density in Qt such that log(u E L' (22 and p: [0,T] x Q R be given by (41) Then, the following hold:
1. There exists universal constart C O, such that
|p(()) (0)()(()))) Vpll S CTimmo tE|O.T]
sup
and thus 42 holds.
2. V Sa mo log(mo)) p(T)) log(p(T))ddr
Proof. Testing
against (p) for smooth function f, we obtain
J ((o(Txz))de + ['' I P' f(mo(z))d
and the choice of f(x) x log(z) yields
I log(e)) x))dx ! _ Vpl dedt = mo(()) log(mo(z))dr;
which shows the second claim. Finally, mo log(mo) E L this yields estimates on sup l1o(t) (og((())( V since (E(,,]]
I (og(o())))g(((
/ (((T)))))(((( d) + 2 (O(IT da $ p(T, log(p(()) 2e 112
where we used the fact that x log(x) 2 !
The rest of the subsection justifies the steps outlined in Subsection 5.2 estimate of the form
Namely starting from
J(T N.e .e ,00 S Cnn
our goal is to obtain estimate of the form
J(n 0) S é'.n'
For this we use formula (42). In particular we first use the linear dependence of Jr with respect to the underlying measure to bound
JE 0) Jn (n))),
Then, instead of comparing the gradients on the difference
|| log(17 N. 112 - V log(n*)
we use Proposition B.I from which it is enough bound the entropy terms. The next two Propositions provide these results along with some other technical facts.
Proposition B.2. Let T and n² for = 1, 2 denote two probubility measures in Ω such that | log( " [[ < oc and of the corresponding solutions to 41). Then, there exists dimensional
constant > such that
(JL(P 0) (L(p),0)1= CT sup di (t), p()) |0 (0))
[
S CT'( a =00
In the above indicates the score function approximation.
18

---

Proof. We recall that
I //¹ T
div(so) ds (-' / 1 |sol + div(so)dp) (s)
Bpt B
sol 2
Therefore, if
1
2 div(se)
90
2
T
1 10 2 + div(so)d(( p² (s))
Trlp, Jri('', ,80]
= =16%' / goodlp -p') (s)dss
VV x / d1(p) (s))
sup 051ST
S T sup
V oc
sup 0515T
d1 (0 (1),
0SIST
Since
V S |
to conclude we need to show that
sup d (p) s d, ( 0StS
However this follows from our Lemma 3.1 applied for = 0 and m = n² for i = 1,2.
Lemma B.3. Let x²; NN O < < be as in Theorem 3.3 with c < 1. There exists a dimensional
constant C (dd) 0 such that
Ni x-) S di( (T)
(43)
|| - 1 S cc dl dI x
(44)
and
d1(x .))
|lo((=)) Jx - log((* (NN (llll1 C(1 (lo(())
(45)
Ve
Moreover, let denote the constant from Proposition D.3
and T R2 large enough such that
2CR
WT RZ
2 voll RT4)
then to a dimensional constant C' > we have
I (ogg("" log (T))
C ( (1+ dlog(R) (T,,,,
S
(46)
VT
Proof. Lee ,p: [0,T] x 1 + R, be solutions to
0.p- Ap = 0 i (0,e), (0(0) P0o
(47)
for Po = 70 x respectively. Since n' = [(c) NN = x(c)+T* we note that x² = (c,
and # N.e p (c, .). Using the same adjoint method as in the proof of Theorem 3.1 for function
bb : Ω + R we consider the function Φ: [0, c) x Ω + IR given by
0,, Ao 0
(48)
(e())= = v.
Testing against in the equation for \ p² p yields
f [(*Ne - nt 7²)((x)drr = / $(0, 2)(.z) x (x))
19

---

First we consider |/ I which from (31. yields
| C
[Vo(O)l I
VV VE
Thus, for any 1l S we obtain
/ (AMi0 - 5 Cdi(
Ve
which implies
Cdi x vt
Next, by considering Lip(t) which implies
we obtain
di (1 x²) S d1
For estimate 45 we note that
= ///0 (1+ log( + (1 - ()- (dd x
| (og( x log((=))
$ (1 + log ()) [og(())) - 77/-
Therefore,
d2(x .T)
||*No Ne log(@ (1)-((NN))== x² log(T")) S log (0) (0)))) S ((1+)log(0))
Ve
For estimate (46) we note that for any convex function h
h(x) h(y) 2 Vh(y) (z y) = h(y) h(x) - Vh(y) -v-
Hence by applying the above to l(z)) log(z)) we obtain
f |lognn Mi (T))) N:) - (T) - / fa++log(11 Ni (T) (T))
S 1 + log(1) N. (T)) - -f
We now note that for any g: RI + R, |1 1
I glx)d(n - (T) -
/ r ((T) (all*N-=--m |VT((T) (
C
di (*Nis,,))))) 77))
VT
Taking the supremum yields
I| No -f (T) S C (i ( S di 79).
VT
Finally, from Proposition
D.3 above applied to p = N. vol(
we have
1 1
" N.. (T, x) 2 -Ip((T) + vol [)T
2 (oT(
as well as by our assumptions on T > 0. Therefore,
1
No (T,x) S vol(HTTa) + (IT( $2 vol(RTT)
|| + Vig(n)Ni((T)) S C(1 + dlog(R))
and the result follows.
20

---

Remark B.4. It is worth noting that the empirical sample x could have been regularized by any smooth kernel p. However the choice of the heat kernel which provides the same result as early stopping behaves quite nicely with respect to the metric d1 since it is a contraction. This is evident in Lemma B.3 above where we see that we do not pay any additional cost due to the mollification in
In fact the above techniques show the following general result regarding ESM to DSM bounds. Combining the above we can state the following general result, about ESM to DSM bounds.
Theorem B.5. Let x.. x² E P(2)) denote two probubility densities, such that
|| log(=")llzaaa) )[l2)1))) < fo i 1,2
(49)
and 8 such that
x'(x) 2 8.
Define p² [0, Ω + as the solutions
Saip' Ap' = 0 in (0, T Ω
(50)
(0(()) T' in Ω.
Then, up dimensional constant C C(d) 00
J(P²,0) - J() c [1]]so x²) + (1 +1+ -
-x'112). (51)
B.1 Proof of Theorem 3.3
Now we can prove Theorem 3.3
Proof. From Proposition B.1 we obtain
I J ( ) (n) ; s0- ) +
Ji s0-) 2 V 77 12 || v 1621 (JL("), so-) JJ JnN ,,()),,,s000))) .80))):
By assumption we have the bound
[np ( S Cnn-
By Proposition B.I
V || Vg
= =/ n" (x) log(T" (x)) Ni log(( No (x))dxx= / (T) log (T)) log(n
52
From Lemma
B33 we have that
|| log(n*) * Ne log(# ( S CC(1 (lo(())) dI(m),m)
Ve
Using the same argument and the fact that from maximum principle of (T) > 8 > 0, we can show
II) log(n(T)) Ty log(n N.e
(T))
s (1+ (T
C d (- Nie)
C
di (,,
VT
VT
Furthermore, by Proposition B.2 we have
[n( so. J- N.. (a.) s CTd Me, n ()so I CTd, 11)0 "/
Putting everything together we have
I (11, Sp-) 5 + C 1+ C(1 log(5)) 1
+ T sso 150122((07 ) di (T m))
T
21

---

As an immediate Corollary we have the following. Corollary B.6. Using the same notation as in Section 3.5 if een > 0 is such that
0 S (1 0*) < Cnn
(52)
then
0 S [nn" ,0) < e''
(53)
where for a dimensional constant = C(d) >
é'n Enn + C(1+ [log(d) Vc
1
+
+ T T solca xx)) di d1 .T).
(54)
vT
Remark B.7 (On regularity implications of NN approximation). It is important to note that the norm of our NN approximation Vseello is in fact dependent
as
from the following
s |= 2 (op))) 2 VV "12 Venn
and (ll(r"" blows up + O.
We finish this subsection of technical results with an observation about the implications on the existence of smooth NN-approximation.
Proposition B.8 (ESM implies regularity). Let n, Sg and enn > 0 be above for Ω = RT Assume cc. Then, π = (x)) admits a density and in fact || log(()))lzz)) < co.
moreover, that /
Proof. Since / < 00, for some constant C = C(R. d, Cnn) > 0 it holds
2 2 - J Vlog(r")) (s, x)ds S C.
Moreover, by the regularizing properties of the diffusion we have
n" log(n" (T))dz oo
and the result follows from Proposition B.1
C Average DSM generalization
Using the previous results we can in fact show result about the average error Theorem B.5 The main observation here is that by Jensen's inequality when take expectation with respect to the sample N we no longer require a lower bound on our densities. However, as we will see we require very restrictive assumption the norm of our function approximation
Theorem C.1. (Average DSM generalization) Let enns A > 0 and assume that for each sample ft from N there exists a Sg. such that N., 00) S enny
with
||s- ll A.
(55)
Let mig (T)) be the generated distribution from N (which also random since A is random). Let C. > 0 be the dinensional constant appearing in Proposition D.3 and T 2 R² large enough such that
2C
12 S
voll RTTd)
then
E d1 (M,,,,((T)) mT(T)) < CRi VR) # /Tenn),
where
C
e'm = c Cnn +
TRA² + vT
(1 + dlog(R))
No
22

---

Remark C.2. It is important to emphasize in the above that we are assuming that we may pick both an enror Cun 0 and a bound A 0 on the gradient of Sy globally for all random samples #N Moreover, is a weak approximation of V log(nf.*) which although smooth for all >> 0 it does exhibit a blow-up as > 0 tends to zero. Therefore, since by triangular inequality up to a constant C = CC(, >
CA2 [solcc 2 (1) 2 1D Enn
the constant A > O will also exhibit a blow-up which can make the approximation worse. Therefore there could be a potential trade-off between NN approximation cun becoming small and the value of the norm A.
First we require a preliminary lemma.
Lemma C.3. Under the same assumptions as Theorem C.1
we have that
E [( enn Nà
C
TA
1
dlog(R))).
(56)
/T
Proof. We note that
s00) [n ( se. + D17
= Jie sa-) + 2 11V D N. 112:111111)) + - ssp()
-
ss.)
By assumption have the bound
Je S Cnn.
N.c,
By Proposition B.I
2 D 11 IID 1731)113)
- -/A n' (o)(T((T log (x) (x))
[ (TT) Ni Ne dT)))dx
Taking expectations with respect to the sample AN we get
E
/ n' xx)
(og(T(()))) N.r log(# No)NNN
/
(x)) [* (o) (o))
J n (x) (x) (ogg(((((()) (x)) E (x)) log
Log(E((M)) [:Nie] )dr
In the above we used Jensen's inequality along with the fact that
Nie x..
Moreover, by Lemma B.3 we have that
E B (og(T)) - / (f((T) (o(T))og(f(())))
C
+ d dlog (I #M)
VT
VTN+ (1 + llog
Finally, from Proposition B.2 we have
E JI - Je( Nor ss-.)
< CTE
|| N 11 (TT, *N)
<
CT
A²
N
Thus,
E [int s enn +
C
TA² +
1
(1 + dlog(R)))
N
VT
and the result follows.
23

---

Proof. By Theorem 3.1 we have that
di ( S Ve + di( my
m (T)).
And
di (
CR. (1 + v Di pa (d (m
vTIlb" ba-
1bo-ll1au (())))))
vol RTM))
S <C (1 + vA) R&e के VTJ (0))
Taking expectations and using the Lemma above yields the result.
D Technical results
D.1
HJB estimates
In this subsection, we gather all the regularity estimates for the HJB equations. The results are for the most part standard, we include however detailed proofs for the readers convenience. For further study into the techniques and topics we refer to 9 [0 Although mentioned in the introduction
in the current work we assumed (t) = /2 and f = 0 in the OU processes (T), here we prove the regularity results for general non-degenerate diffusion (tt). The addition of may be incorporated in the drift b.
Proposition D.1. Let Ω Rdd v: _R R and b [0, x Ω + IR be given with | oo. Consider the solution u [0, Ωt R of
-,u (u) + a(t)
2ul3+bb Vu = 0 in [0, T) x Z
(57)
(TT,x) v(x) in ?
Assume moreover that for some M >
1
0
M at)).
Then up to a universal constant C > 0 the following holds
[v((t) & C(1 + [Vblo
(58)
(T --))[[ - CM + 1)llo/
(59)
Proof. Let [0, Ω R be given by
2 Vul2.
Differentiating the equation for u in i = 1, d yields
-ū,u, att Vt) Vu, + Vu, + b, Vu =
and thus multiplying by u, and summing over i, we obtain the following equation for
{- - 0, a(t) v(t) V a(t)Vu V2 + (VuVb, Vu) 0, (T()) v
(60)
In the above Vb is the matrix with entries
[bbba = 0b
for
b b= (b) ,.. .12).
Let [0,T] Ω IR be given by
wit, x)) = = Cu
where C> is a constant to determined later. Assume that the function wo achieves its maximum at (So, To) [0. x Ω. We will look at the following cases:
24

---

Case 1: Assume that 0 S < T and we will show that for C' > 0 large enough this leads a contradiction. At (sq, zo) we have the optimality conditions
Vz(so, zo) (Vuu(so,zo) xo)
Az(s0. _ CA (s. 2o)
0,2(su, - Cd 80, To).
Hence,
C(-Oru(so, 20) - Am(s,, + (u mn)) (- mo) ( Az(50, + + V2(50,
and using the respective equations for u, we obtain that at (su. mo)
(((00 Using the optimality conditions yields
Vul² v - a aa0os (Vu Vb, Vu) a(s0) Vu. Vz.
a<(0)) C Vn + aass)) - (Vu Vu) S /Vbllo/ 2
which is contradiction if C 2 Vbllo. Thus for
Co = vb|
the function
2 Cou
achieves its maximum at some point ro).
Case 2: Assume that Se = Then, for every (t, E [0, x 12 we have
z(t, 2) S w (T, mo) Cu(t, S 2
Vul² + C/lvll + C1|
and by maximum principle we also have that
||u|| S ||0|000
Thus, up to a universal constant
Vu(t,z)) S (1+ [bb||M]|e|lo
which proves estimate (58)
Now we prove estimate (59) The proof follows the previous strategy only this time we consider the function
wl(,x)) = (T l)= Cu,
where again C 0 will be determined later. Assume that the maximum occurs some point wo).
Assume that 8q < T. Looking again the corresponding optimality conditions and using the equations for u and we have at (so, mo)
Case 1:
Ca((s) 2
1) vml < -T so) (Vu Vb, Vu) < Vbllo Vul,
which is a contradiction if C 2TM + 2M.
2. Case 2: Assume that = T, then for all (1,2) E [0, 7] x Ω we have
x(t,x) < w/ (T, zo) + Cou(t,
(T, x) Cou(t, x) s 2Co
Thus up dimensional constant we have
(T - t) Vtl x) s M 1)]le/
25

---

Corollary D.2. Ler Ω C R. v RR - R and b [0. x Ω - R be given with Vb cc.
Given v : with consider the solution ø: [0, r] f R of
- att) + V in [0. T) x f2 (o(Txx) (x) in S2
(61)
Assume moreover that for some M > 0
1 M att).
Then up to dimensional constant C' > 0 we have the following estimates
|(V(()) I Cl (1 + [V
(62)
TT - V)(() < C M + 1)
(63)
Proof. Since vb 2 by the Maximum Principle we have that
1 1 o(t, x) S 11o.
Thus we can define [0, x Ω IR by the formula
<tt,z) = u(t
u(t, -2 log((((tzz))
We then have
0.6 - VΦ
Vu
A6
Vul2 Φ
Au
and so by substituting the equation for ø we obtain
- a())a())Au a(t)
2 Vul2 +b. Vu = 0,
(64)
(TT log (1/ (x)).
The results will follow by Proposition D.I for v(x) -22 (o)( First we need the following estimates
[e(x)] 2| log (6 (x)) < log([ - 20 since & 2 1,
Vu(x)) = [v((z)]
2
[x(x)
= | s 2|
Moreover, note that
(ut(t = Vo((t Φ²
[o(t,x))
Therefore, by Proposition
D.I
estimate (88
we obtain that up to a dimensional constant >
|V < C (1 + V M) & 1 (1 /V
Similarly by Proposition D.I
estimate
(59)
again up to dimensional constant C > 0 we obtain
T
1 < Cl M(T]Vbbll> 1).
D.2 Long time behavior of periodic heat equation.
The following result is classical and establishes exponential rate of convergence in dj for solutions to the heat equation in the uniform distribution on the torus. We provide a proof for the readers convenience.
26

---

Proposition D.3. Let Q = RT" and m, E P( 1,2 be two probability measures. Define p' 10 - [0, oc) by
Seep' - Ap = in (0, oo) x 2 (p((0) m, in S2
Then, there exists constants C = ((d) > 0,w = w(d) > 0 such that
(p) 1 CRR में d1 (m2, m1 for 2
In particular by choosing m2 i
di di (t), (eT ) S CRRe #3 d, (m,,
vol(RTd) fort 2
(65)
Proof. First assume that R and recall that if [t, æ) denotes the heat kernel on the unit torus T then for constant C C(d) >> 0 we have the normalizing effect
(T(()) gl S C |191|
Vt
Fix 15 : T + R such that Lip(v) < 1 and (0) 0. Such function yi will also satisfy
|| clo S
Fix > 0 and let o: TT [0,T] + be given by
{- 0. = (), x T
(66)
(TT,2)) = (xx).
By testing against e in the equation satisfied by x p² p¹ we obtain
f (Z)) (T)) = o(0)d(ma-m) -m)) { (l) S didimma ( ml)
VT
Taking the supremum yields
C ((,()
d d(( <
VT
Repeating the same argument above with initial conditions m, pp obtain
d di (2T), s / (TT = d (m2, mz)
2
and by induction for all l N have
di d1 P'T)) s
di d1 m1) = C*T d1
Let 2 and fix a such that
KT S 0 < (k: + 1)T.
Since the heat operator defines a contraction dj we have that
di (p(((()),p(())) (d(P'(kT), < (CVT) - (1).
m1).
Moreover, we have that
0 -k < T
+1
thus
d and so choosing T' large enough so that
(CVT)- - 4
dj (m2,
CVT> 2
27

---

yields the result for R = 1. For a general R > 0, we simply apply the above to the functions X [0, oo) T defined by
2(t,2) R (R Rx))
which yields
(1) - Ce- -tt dj (A)()),
Since
/ (p² Rx) - o (ttl, Rx))
di((?(() =
sup
/ RT
11 (0 (tR u) p (tR? u)) (
sup
= R
sup
(())(p u) 0¹ (1R u)) - Rd (1Pt) (126)
R... Re74
the result follows.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All our claims, both formal and descriptive explanations of our theorems carefully written to appropriately reflect the analytical results we obtain. Each theoretical
claim is backed by proof and accompanying supporting arguments in the appendix.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of the analysis are clear based on the assumptions in each theorem. We have also discussed throughout where the analysis has not been optimized to exhaustion relative to the power the analysis tools we use.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The main three theorems in the paper are carefully formulated and stated. The proofs are descriptively outlined in the main body, and the full details with supporting propositions and lemmas are provided in the appendix along with their detailed proofs.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [NA]
Justification: Our paper does not include experiments.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification: Our paper does not include experiments requiring code.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [NA]
Justification: Our paper does not include experiments.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Our paper does not include experiments.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
29

---

Answer: [NA]
Justification: Our paper does not include experiments.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes. The paper is also written to preserve anonymity as much as possible.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: While paper involves analysis of generative artificial intelligence algorithm, there are no immediate pathways a malicious actor could use our analysis to produce more harmful disinformation deepfakes. Our analysis primarily exist understand existing generative AI algorithms.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper is analysis paper, and poses none of these acute risks.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: No individual or entity owns the theory of PDEs.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsoureing nor does it involve research with human subjects.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
30


Reviews:
[{"review_id": "Fxw7AyVvbi", "reviewer": "Reviewer_PXMJ", "summary": "This paper studies the generalization error of diffusion models. The major tool is the Wasserstein uncertainty propagation theorem. With such a result and the regularity analysis in PDE, the authors establish robust analysis for diffusion models with respect to various errors.", "questions": "I have some minor comments and questions:\n1. The results established in this work are usually referred to sample complexity bounds or distribution estimation error. The authors use the name robust analysis while the meaning of \"robustness\" seems different from the one in robust optimization. I suggest the authors clarify this notion in the context.\n2. The paper leverages the pathwise characterization of probability distribution generated by the forward process. I suggest the authors add more explanations and/or refer the readers to the literature when a PDE is introduced, e.g., Eq. (11). The same applies to all the FK and HJB equations.", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors examine the robustness of diffusion models from an uncertainty quantification perspective, that is not well-understood in prior work.\n2. By leveraging the Wasserstein Uncertainty Propagation theorem, the authors provide the generalization error of diffusion models w.r.t. various error sources.\n3. The paper is well-written and easy to follow. In particular, I appreciate the presentation of math derivation.", "weaknesses": "1. The first major concern is the connection and comparison to the literature. The bounds in Theorems 3.2 and 3.3 are not explicit. I suggest the authors provide clear sample complexity results w.r.t. problem parameters. Furthermore, the authors should discuss how the bounds improved the existing results in the literature. \n2. The analysis in this paper heavily relies on the PDE theory and regularity analysis. I suggest the authors discuss relevant prior work in Section 1.2. Also, I suggest the authors briefly introduce UQ used in other ML problems beyond diffusion models. Furthermore, the score approximation and estimation theory should be mentioned as that is one source of the errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to weakness #1**\nYes you are correct that for the majority of our results, the bounds are more qualitative than quantitative. We refer the reviewer to our rebuttal summary, that our main goal  is to form a bridge between analysis of generative flows and PDE theory. We aim to illustrate our methods' strengths with specific applications such as the ESM and DSM bounds which, as you have highlighted, are not sharp. Establishing sharp constant bounds is important and beyond the scope of this work. See our response to reviewer gUwF about how we may investigate computing the bounds. To address your concerns, we highlight our novel quantitative and qualitative contributions:\n- **Qualitative:**\n  - ESM Bound (Theorem 3.2) and DSM bound (Theorem 3.3) show the following improvements:\n    1. We obtain estimates in norms better than the KL divergence and, in fact, are able to bound the stronger total variation ($\\|\\cdot\\|_{L^1}$ norm) by the weaker Wasserstein-1 ($\\mathbf{d}_1$) norm.\n    2. We obtain estimates for the $\\mathbf{d}_1$ norm *without* bounding the KL divergence and applying Pinsker's inequality.\n    3. Theorems B.5 and C.1 show it is possible to relate an *a priori* unknown error in the ESM objective with the error from the DSM objective (at least on average).\n\n  Our results are *agnostic* to the manifold hypothesis, i.e., they apply both in the case when the distribution is degenerate, and when it admits a density. We highlight this fact throughout the paper (Sections 1.1, 1.2, and Theorem 3.3). Estimate (12) in Theorem 3.2 exists in previous literature only under the assumption that the data distribution is absolutely continuous with respect to a Gaussian [1,2,3]\n\n- **Quantitative:**\n While we track the major quantities in our Theorems, you are correct that there are important constants which are not explicit. In particular using the same notation as in our Theorems we use:\n  1. A constant $C>0$ which depends only on the dimension.\n  2. A constant $\\omega$ related to the exponential rate of convergence to the stationary measure in the heat equation.\n  3. A constant $\\delta>0$ which captures the lower bound a measure obtains when we apply some diffusion of level $\\epsilon>0$.\n\n$C$ and $\\omega$ are related to classical problems in PDEs and should be computable in the future when our framework is further refined. We chose to focus on the core connections between generative modeling and PDE theory, as explicitly computing these constant would make the paper far too technical in PDE theory.\n\nA major focus of future work is to compute these constants more explicitly. For example, it is important to understand if the dependence of $C>0$ on the dimension is linear, polynomial or something worse. For $\\delta$, this appears to be more difficult to compute and would require more assumptions on the underlying measure $\\pi$. While adding some noise $\\epsilon>0$ (at least on the torus) guarantees that the measure will gain support everywhere the exact size is a challenging problem.\n\n**Response to weakness #2**\nThank you for your feedback. We will include discussion about relevant aspects of PDE theory that are most useful in our work. Moreover, while the use of generative modeling for UQ has been explored, to our knowledge, the UQ perspective for studying generative modeling is uncommon. We highlight [4] which derives a type of Wasserstein Uncertainty Propagation bound for the W2 distance, although they do not refer to their work as a UQ perspective. We discuss this work in Section 1.2. Regarding score approximation and estimation theory, we describe this error as source #3, score expressivity, in Section 3.1. We will rephrase this source of error in the next version of the manuscript.\n\n**Response to Question #1**\nThank you for your comments. There are multiple senses of robustness we use in our work. We will clarify these notions further in a future version of the manuscript. A subtle but key result of our work is that our sample complexity bounds are *worst case* bounds for *any* random sample of size $N$. As the worst case bound is finite and controlled, it demonstrates that score-based generative modeling is robust. See the discussion after Theorem 3.3. This sense of robustness is actually similar that of robust optimization, where the method produces the correct value for worst case choices of the parameters or initial conditions.\n\nIn contrast to previous analysis, our results are *agnostic* to the manifold hypothesis. Past work assumes the data distribution has a density in $\\mathbb{R}^d$, i.e., not supported on a low-dimensional manifold. As our results are independent of the manifold hypothesis, it explains why SGMs are robust, i.e., work well, even when the data distribution is degenerate. The regularizing properties of the Laplacian are what enable this robustness. \n\n**Response to Question #2**\nThank for for reminding us to include appropriate references to PDE theory for the generative modeling community. We will be sure to include references that will help both the PDE and generative modeling communities more easily explore each other's previous work. One goal of this paper is to showcase the connections between the two communities and we hope it may initiate such collaborations. We will add important sources, and provide some extra discussion to guide the reader. \n\n[1] Lee, Holden, Jianfeng Lu, and Yixin Tan. \"Convergence of score-based generative modeling for general data distributions.\" 2023.\n\n[2] Chen, Sitan, et al. \"Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions.\" 2022.\n\n[3] Chen, Hongrui, Holden Lee, and Jianfeng Lu. \"Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions.\" 2023.\n\n[4] Kwon, Dohyun, Ying Fan, and Kangwook Lee. \"Score-based generative modeling secretly minimizes the wasserstein distance.\" 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4DQmBdL3oo", "reviewer": "Reviewer_gUwF", "summary": "The paper studies the robustness of score-based generative models (SGM) to different sources of error that are relevant in practice, such as the limited expressivity of the score function representation or the choice of reference distribution. Specifically, they upper bound the Wasserstein-1 and L1 distances between target and approximate distribution (given by an SGM) in terms of these different sources or error for both denoising and explicit score-matching objectives.", "questions": "- In Section 6.2., the authors discuss an application of their bounds to likelihood-free inference. To that end, could the authors elaborate on how difficult it is to estimate their bounds in practice? Is accurately estimating the Lipschitz constant of the score function the main challenge?", "limitations": "This is mainly a theoretical work and I cannot foresee any direct societal impact stemming from this research.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Score-based generative models are extremely relevant within the machine learning community, and quantifying their uncertainty in terms of how well they capture the target distribution is an important research problem.\n- The results are, to the best of my knowledge, new and it is impressive that they manage to isolate different sources of errors in their bounds while placing no restrictive assumptions on the target distribution.", "weaknesses": "- The main weakness of the paper is its dense presentation. The authors did a good job of motivating and describing their contributions in the introduction, but the rest of the paper is not easy to follow. I am not sure, there is much room for improvement within the limited space of 9 pages, but I would suggest including a discussion on actionable insights one can get from these bounds. Throughout the paper, the authors comment on how important robustness is for the reliability of generative models in general, and I agree, but I could not get an intuition about when one can expect these bounds to be tight so as to ensure the resulting model is trustworthy.\n- While I respect the authors’ decision of going for an entirely theoretical paper, I cannot help but feel that some small scale experiments illustrating the tightness and usefulness of the bounds would be enlightening.\n\n### Minor Issues\n- Line 35: “contributes” should probably be plural here.\n- Line 53: “recognizes” is misspelled.\n- Line 61: no need for “of” after “study”.\n- Line 100: if I’m not mistaken, the acronym SDE was not yet defined at this point in the text.\n- Line 102: Albeit clear from the context, $W$ and $\\eta$ have not yet been defined by this point in the text.\n- Line 121: “however our results are generally apply”\n- Line 331: Word missing after “used”, probably “for”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to weakness #1**\n\nThank you for the insightful feedback. As stated in our rebuttal summary, our goal is to introduce a PDE framework for error analysis of SGMs that are comparable to previous bounds, while being agnostic to situations where the data distribution is supported on a lower-dimensional manifold (see our response to reviewer PXMJ Weakness #1). To that end, the paper is organized to best understand the PDE framework for analyzing generative flows, i.e.\n\n1. Study the evolution of test functions under the Kolmogorov backward equation of the true and approximate SDEs.\n2. Bound the resulting integral probability metric using regularity estimates of the KBE.\n3. Apply the resulting bound to the appropriate SGM setting.\n\nTherefore, the major actionable item the paper aims to convey is to consider PDE regularity theory for analyzing *any* generative flow—SGMs are just one choice.\n\nWe will revise the paper to better highlight the important insights one may derive from our results. One reason we decided not to conjecture on actionable items is that our bounds are worst-case bounds that hold for any random sample of size $N$. While this shows that the bounds imply SGMs are robust, i.e., even in the worst case, the errors are bounded, we are not confident in the sharpness of the bounds.\n\n**Response to weakness #2**\nWe appreciate the reviewer's point, and we believe the ultimate goal and impact of our approach is the ability to produce *a posteriori* bounds that can quantitatively capture the *confidence* a practitioner may have in a trained generative flow. In our rebuttal summary and the response to Reviewer PMXJ's Weakness #1, our primary objective is to showcase the connections between PDE theory and generative flows. Therefore, we make no claim in sharpness. In our response to the next question, we illustrate the usefulness of computing sharp bounds, which, however, requires further research. We defer numerics to future work, when the bounds can be sharpened and applied to more useful applications.\n\n\n**Response to Question #1**\nThis is a great point, and it is an exciting topic we will study further in future work.\n\nWe describe how one may think through how to compute the bounds. As you point out, the regularity bounds on the score function are critical. The Lipschitz constant of the score function will be dictated by the error on the ISM approximation we choose $e_{nn}$. To this end, a few factors need to be balanced.\n\n1. If the underlying measure $\\pi$ lies in a lower-dimensional manifold, its score function is undefined, since the term $\\log(\\pi)$ does not have meaning when $\\pi$ is, for example, a Dirac mass. Therefore we require the addition of some noise with $\\epsilon>0$, i.e., early stopping.\n2. If $\\epsilon >0$ is small and $e_{nn}>0$ is also chosen very small, then we are learning a potentially very rough function, and thus the Lipschitz norm will be large.\n3. Moreover, we also have the regularizing effect of the Laplacian, which is highlighted in the choice of the test function. To bound stronger norms by weaker norms, we need more regularity on the score function we learn. For example, to bound the $\\mathbf{d}_1$-norm by, say, some $H^{-s}$ norms, our bounds would require derivatives on the learned score function of order roughly $s>0$. The higher the $s$, the better the exponent on the size of the sample $N$, i.e., the rate. However, as pointed out in the previous item, the higher the $s$, the $ H^{-s}$-norm of the learned score function might also explode.\n\nWith all of the above in mind, it seems that to answer your question, one would need a far more detailed analysis of the exact growth of these quantities, which is in fact the content of a work in progress. Finally, although we are not in a position yet to answer these questions, we believe that in the framework we introduced they can be posed as standard PDE questions.\n\nOnce these questions are resolved, they may be most impactful for likelihood-free inference applications, e.g., [5]\n\n[5] Song, Yang, et al. \"Solving inverse problems in medical imaging with score-based generative models.\" 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YlGviITWV7", "reviewer": "Reviewer_DZY4", "summary": "This work studies the influence of different error terms for diffusion models from a continuous perspective under $W_1$ distance. They explain the reason why the early stopping parameter $\\epsilon$ would lead to a memory phenomenon of diffusion models. To achieve these results, this work proposes a WUP theorem to explain the robustness of SGMs.", "questions": "Please see the Weakness part.\n\nQuestion 1: This work does not consider the influence of discretization error. It would be better to discuss the challenge when considering this error.\n\nQuestion 2: For me, one interesting point is the balance of different terms when considering $\\epsilon$. As shown in Theorem 3.3, $e_5$ term has $\\sqrt{\\epsilon}$ dependence and $e_2$ term has $1/\\sqrt{\\epsilon}$. It seems that there exists a balance between these terms when $N$ is finite. It would be better to discuss this balance in detail.", "limitations": "This work does not discuss the limitation and societal impact in a independent paragraph.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe WUP theorem is novel since it can analyze generative flows instead of linear SDE and will arise independent interest.\n2.\tThe analysis of the early stopping parameter can deepen the understanding of the memory phenomenon.\n3.\tThe analysis of different objective functions is interesting.", "weaknesses": "1.\tThe abstract section mentions that stochasticity is the key mechanism ensuring SGM algorithms. However, this work does not discuss it in detail in the main content. It seems that the WUP theorem does not hold for the deterministic sampling process. It would be better to discuss it in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Weakness #1**\n\nThank you for pointing this out, we should have been more clear in our PDE terminology for the generative modeling community and will make sure to fix it. The *stochasticity* in the generative flow appears as the \\emph{Laplacian operator} in our PDEs. Without the Laplacian, the Fokker-Planck equation is a continuity equation, which describes the density evolution of deterministic flows. There are two aspects of SGMs where stochasticity/Laplacian provides regularizing effects. The first is early stopping, i.e., adding some level of noise $\\epsilon$ to the data distribution. This, in effect, runs the noising process for a short amount of time and immediately mollifies the initial distribution to have a smooth density. \n\nThe second aspect is that the Laplacian is the key mechanism that provides the regularizing effects to the test function that then allows us to bound, for example, the stronger $\\|\\cdot\\|_{L^1}$ normal by the weaker $\\mathbf{d}_1$-norm. Without the Laplacian this effect would not be possible in general and in fact we would not have access to long time behavior results. We will make sure to add a detailed remark on this.\n\n**Response to Question #1**\n\nYes, discretization error is indeed a source of error that should be looked into based on our analysis. This type of error can definitely be addressed by our framework; we however, defer this analysis for future work as it is a rather technical undertaking of its own. As we mentioned in Section 3.1, the most likely avenue for analysis is via the so-called modified equations [6]. That is, the numerical solution to the SDE is, in effect, the exact solution to a different *modified*  SDEs. The difference between the drift terms in the modified SDE and the approximated score can then be included when applying the WUP theorem. This further highlights the capabilities of the WUP perspective and our UQ angle.\n\nWe refer the reviewer to our rebuttal summary we chose the errors we thought were most impactful and relevant to score-based generative modeling and PDE regularity theory. As raise by Reviewer gUwF Weakness \\#1, the paper is already rather packed with a lot of new results and insights, and we believe adding the discretization error would diminish the readability and understanding of the main message of the paper.  \n\n**Response to Question #2**\n\nThis is a great point; in Remark 3.5 we briefly discuss how the bounds in Theorem 3.3 balances between the various sources of error or different properties of the data distribution. Moreover, it appears possible to optimize the bounds, which may further improve or inform the balance between the errors. This seems to be only possible as we work with integral probability metrics directly, which allows us to obtain sharper bounds than using a KL based approach. \n\nMoreover, roughly speaking our bounds (and all the previous bounds in the literature) obtain estimates on \n\n$d(m_{g},\\pi)$\n\nwhere $m_g$ is the generated distribution and $\\pi$ the true data distribution. However, we only have access to $\\pi$ through the empirical distribution $\\pi^N$, and we essentially estimate $d(m_g,\\pi)$ through \n\n$d(m_g,\\pi)\\leq d(\\pi,\\pi^N)+d(\\pi^N,\\pi^{N, \\epsilon})+d(\\pi^{N,\\epsilon} ,m_g)$\n\nwhere we further interpolate these distances through the addition of noise $\\epsilon>0$. Notice that $d(\\pi^N,\\pi^{N,\\epsilon})$ is finite when $d$ is an IPM, but not when it is a divergence. \n\nHowever, we believe it is currently rather premature to optimize the bounds as they are still qualitative for the most part, and are not fine-tuned (see our discussion to Reviewer PXMJ Weakness #1).  We will add a short discussion on how obtaining sharper bounds and then optimizing them in Remark 3.5 in a future version of the manuscript. \n\n\n\n\n[6] Abdulle, Assyr, et al. \"High weak order methods for stochastic differential equations based on modified equations.\" 2012.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their careful reading, time, and insightful comments on our work. They will be invaluable for improving the current manuscript and for future work. 

We emphasize the organizing principle behind our paper. **The primary goal of our work is to establish connections between PDE theory and flow-based generative models.  We create a proper framework where various sample complexity and error bounds can be the derived from existing PDE stability results and regularity estimates.** In particular, the current paper employs PDE regularity theory and the regularizing properties of the Fokker-Planck equation to analyze score-based generative models. Moreover, the Wasserstein Uncertainty Propagation theorem we derive is motivated by the model-form uncertainty quantification problem that naturally arises in score-based generative models.  Many presentation choices and trade-offs in our work are made with the organizing principle in mind.

Meta-review: This paper gives theoretical analysis of score-based generative models from an uncertainty quantification perspective, showing the impact of errors from different sources. They use a Wasserstein uncertainty propagation theorem using PDE theory, which is potentially of broader interest beyond SGM's. Reviewers received the paper positively and thought it was generally well-written and mathematically sound, with solid technical contributions, though noted a lack of tightness in the bounds (including some unspecified constants) and discretization analysis. I recommend acceptance, with the reviewers' suggestions to improve clarity of presentation.
Decision: Accept (poster)