Approximation and Generalization Abilities of Score-based Neural Network Generative Models for Sub-Gaussian Distributions
Authors: Guoji Fu
Wee Sun Lee
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion models
Score-based generative models
Approximation
Generalization
Primary area: theory
TL;DR: We study the approximation and generalization abilities of score-based neural network generative models

Abstract: This paper studies the approximation and generalization abilities of score-based neural network generative models (SGMs) in estimating an unknown distribution $P_0$ from $n$ i.i.d. observations in $d$ dimensions. 
Assuming merely that $P_0$ is $\alpha$-sub-Gaussian, we prove that for any time step $t \in [t_0, n^{\mathcal{O}(1)}]$, where $t_0 > \mathcal{O}(\alpha^2n^{-2/d}\log n)$, there exists a deep ReLU neural network with width $\leq \mathcal{O}(n^{\frac{3}{d}}\log_2n)$ and depth $\leq \mathcal{O}(\log^2n)$ that can approximate the scores with $\tilde{\mathcal{O}}(n^{-1})$ mean square error and achieve a nearly optimal rate of $\tilde{\mathcal{O}}(n^{-1}t_0^{-d/2})$ for score estimation, as measured by the score matching loss.
Our framework is universal and can be used to establish convergence rates for SGMs under milder assumptions than previous work.
For example, assuming further that the target density function $p_0$ lies in Sobolev or Besov classes, with an appropriately early stopping strategy, we demonstrate that neural network-based SGMs can attain nearly minimax convergence rates up to logarithmic factors.
Our analysis removes several crucial assumptions, such as Lipschitz continuity of the score function or a strictly positive lower bound on the target density.

Approximation and Generalization Abilities of Score-based Neural Network Generative Models for Sub-Gaussian Distributions
Guoji Fu School of Computing National University of Singapore guo guo nus, edu.
Wee Sun Lee School of Computing National University of Singapore I leews0comp.. nus edu. sg
Abstract
This paper studies the approximation and generalization abilities of score-based neural network generative models (SGMs) estimating an unknown distribu- tion Po from n i.i.d. observations in d dimensions. Assuming merely that Po is q-sub-Garme Gaussian, we prove that for any time step E [oo n()), where to > Oaa2 22/d log there exists a deep ReLU neural network with width < On log2 n)) and depth < O(log" n) that can approximate the scores with O(n-1) mean square error and achieve a nearly optimal rate of Ö(n-1 for score estimation, as measured by the score matching loss. Our framework is universal and can be used to establish convergence rates for SGMs under milder assumptions than previous work. For example, assuming further that the target density function Po lies in Sobolev or Besov classes, with an appropriately early stopping strategy, demonstrate that neural network-basse nearly minimax convergence rates up to logarithmic factors. Our analysis removes several crucial assumptions, such Lipschitz continuity of the score function strictly positive lower bound on the target density.
SGMs attain
Introduction
Score-based generative modeling (SGM) (15), also called diffusion modeling, has emerged as powerful tool of generative models, demonstrating exceptional performance in wide range of applications, such as image and text generation encompasses two Markov processes: forward process that gradually adds noise to convert samples drawn from a data distribution, denoted as Po. into noise (e.g.. Gaussian noise), and a reverse process that effectively reverses the forward process to recover the samples from noise. Specifically, SGM uses score functions (i.e., gradients the log probability density functions) to transform the Gaussian noise into the target data distribution via solving stochastic differential equation (SDE). Implementing the reverse process requires accurately estimating the score functions, which is typically accomplished through training neural networks a finite number of samples using score matching objective
6 video generation + SGM typically
Despite their remarkable empirical success across wide range of applications, the theoretical understanding of SGMs remains in its infancy. In particular, the following fundamental questions remain inadequately addressed in the literature:
How effectively do diffusion models approximate the data distribution? What is the optimal number of diffusion steps required for high high-quality generation? How many training samples are necessary for diffusion models estimate the true distribution accurately? In which scenarios do diffusion models excel, and where do they encounter limitations?
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To address these questions, existing theoretical analyses of SGMs primarily focus on two aspects: (i) the convergence rates of SGMs, which aims to quantify how quickly SGMs converge to the target distribution, assuming access to accurate score estimators; (ii) the generalization bounds of SGMs, which, on the other hand, investigates the score estimation error bounds throughout the diffusion process given finite number of observations.
Early works on convergence analysis literature either often relied on strong structural assumptions about the data distribution, such as requiring it to satisfy the log-Sobolev inequality (LSI) 9 be log-Concave [I]. or they exploit exponential convergence rates 12 L3
Subsequent research [14 16 achieved polynomial convergence rates under milder assumptions, requiring that the data distri- butions have finite second moments and Lipschitz continuous score functions along the diffusion process. More recently, [17 [8] have established nearly linear convergence rates in data dimension, requiring only that the data has finite second moments or finite Fisher information with respect to (w.r.t) the Gaussian distribution. Notably, the best-known convergence rates for Langevin Monte Carlo (LMC) under various functional inequalities [19 21
also scale linearly with the data dimension, up to logarithmic factors, thus matching the rates achieved by [17 [8] This observation shows that when arbitrarily accurate score estimators are available, SGMs can approximate the data distribution effectively without imposing stringent regularity conditions such as isoperimetry, log-concavity, LSI, or even smoothness on the target distribution.
However, assuming perfect score estimation in the convergence analysis of SGMs is highly restrictive and generally unattainable in practice, especially when only finite number of observations are available. Recent studies have delved into analyzing score estimation errors and investigated how these errors influence the final distribution estimation. 222 have studied the statistical guarantees of neural network- based score estimators, showing that neural network-based SGMs are effective distribution learners for distributions on bounded support [22 or smooth low-dimensional mani- folds 126] (27 with lower-bounded densities; distributions on low-dimensional linear subspace 223 or manifolds which are the images of Hölder smooth maps (28]; distributions on bounded support with Lipschiz continuous score functions 25:;: and sub-Gaussian distributions with Barron class of density [24]. Additionally, [29 32 have focused on kernel-based score estimators and demonstrated that kernel- SGMs achieve minimax optimal convergence rates for distributions that sub- Gaussian with Sobolev class of density (29] or Lipschitz continuous score functions (30 as well as for distributions bounded support with Hölder smooth density (31] 32 investigated the sample complexity results for scenarios where the target distribution is either a standard Gaussian or has bounded support, and discussed challenges related to the potential memorization of training samples when using KDE-based score estimators. Table summarizes some recent studies on generalization analysis for SGMs. Despite these advances, some assumptions on the data adopted in existing work remain quite restrictive. For example, the assumption of Lipschitz continuous score functions used in [25] 30 excludes many distributions of interest, such as those supported on a submanifold. Additionally, the Lipschitz constant can conceal additional dependence on the data dimension in some cases, especially when the data are approximately supported on a submanifold. Moreover, as highlighted in [29], the density lower-bound assumption, as employed in 22 26 27.. . prevents their results from being applicable to many natural distribution classes, such as multimodal distributions and mixtures with well-separated components, significantly restricting their ability to explain the practical success achieved by SGMs. Under such stringent requirements, the Holley-Stroock perturbation principle
[33] allows to conclude that the true density satisfies the LSI. When the LSI holds, it is well-established that Langevin dynamics are sufficient to achieve statistical efficiency [34]. However, diffusion models are designed to be effective for broader range of distributions by incorporating smoothed versions of the data distributions. On the other hand, several studies, such as [1]. have been focusing on kernel-based estimators, whereas neural network-based estimators are more widely used in practice. These works do not address the theoretical challenges associated with neural network-baat based score estimation, leaving a gap in understanding the practical effectiveness of SGMs.
1.1 Our contributions
In this paper, we develop a new theoretical framework for analyzing the approximation and gen- eralization capabilities of neural network-based SGMs. Assuming that the data distribution is a-sub-Gaussian on Rdd, we first establish a bound on the score matching loss between the true score
functions, Vpc()) and regularized empirical counterpart vp.()) with regularization Pa.: > 0. pc()

---

Table A summary of recently developed generalization bounds for SDE-based SGMs. Bounds are expressed in terms of the distribution estimation error in total variation (TV) distance. (Po, Po: true and learned data distributions; po: true density function; KDE: kernel- density estimator; DNNs:
deep neural networks; (.) kemel function; smoothness parameter, to: early stopping time)
Paper
Assumption
Estimator
Metric
Bound
Sub-Gaussian Po
TV(Pto: Pto
Omn-1/2 d/41
KDE:
Vic()
1 (.)-)pm 9c()
Sub-Gaussian Po
TV(Po, Po)
Ö (n
Sobolev class
30
Sub-Gaussian Pa
KDE:
VPc() Pr(-VPP
TV(Po, Po)
L-Lipschitz score
(P) 00, 1]
31
KDE:
TV(Po, Po)
O (n
Hölder class Po
Bx(-IVP(-.)
supp Po) = [0. 1]
22
lower bounded po
ReL.U DNNS
TV(Po, Po)
Besov class yo
Theorem
Sub-Gaussian Po
TV(Pt, Pho
Ó (/14) 4/4)
Sub-Gaussian Po
Corollary
|
ReLU DNNs
Sobolev class
TV(Po, Po)
supp(Po) = [0, 1]
Corollary
2
Besov class yoo
which is the KDE-based estimator introduced in [30]. We then derive the approximation and es-
timation rates of ReLU DNNs for learning the true score function Vpe(-) (see Theorems and 3
Pc())
respectively) by approximating and estimating the surrogate pc(-)Vpn..'
Vp.(-)
Score estimation via empirical Bayes smoothing. Inspired by 330,
we employ empirical Bayes (o V v1) for the estimator
smoothing techniques to establish score estimation rate of Ö
vp.(-) under only sub-Gaussian assumption (see Lemma Notably, we improve upon (30 by
P.(-JVP..t
removing the Lipschitz score requirement, demonstrating that regularity assumptions are unnecessary Vfe(-) which matches the result obtained by 29] for the for achieving minimax optimal rates for P:())....'
truncated score estimator
VEL(- px(-) (pp())ppa)
Neural network score approximation. We demonstrate in Lemma of width Ologg n)) and depth On log, n) that approximates p+(-)VPa,t
12 that there exists ReLU DNN
vp.()) with O(n=1) rate for
time steps t in 2/d oo). While DNN approximation rates for smooth functions have been well studied (e.g., (9))))). a naive application of existing results, e.g., assigning one sub-network per exponential component in KDE, would cause the network size grow linearly with the sample size n, making practical estimation infeasible. In contrast, our proof constructs a more compact DNN architecture, which not only prevents the size from blowing up with n but also yields nearly optimal estimation error bounds. Moreover, unlike (22), our approximation results do not require the density
lower bound assumption. The approximation rate for the true score function Vp((-) (i.e., Theorem Pc(.)
follows immediately by combining Lemmas aand 12
28 recently identified a flaw in Theorem C.4 of [22). which invalidates the proof's for the convergence rates claimed in that work and in subsequent papers (26. [7 that rely on it. In contrast, (28 establish a corrected score estimation rate of Ott (t. 12) 2ota' for data supported
Neural network score and distribution estimations.
d' dimensiiona manifold (d' < d), where the manifold is the image of s-Hölder smooth map. In this paper, we aim to derive optimal convergence rates for SGMs under broader conditions. Specifically, provide new proof strategy that removes Vv((+) the need for density lower bound condition. key observation that the surrogate px(-)Vpa.. can be uniformly bounded (see Lemma 5 which allows us to verify Bernstein's condition for the excess risk associated with learning VorC_ Instead of directly deriving high probability bound for
3

---

the true score function, we first establish uniform bound for the constructed DNN class trained to learn Vp+(+) PrO) using Bernstein's 's inequality and a E-net argument. Combining this with the above
empirical Bayes score estimation bound (Lemma I and the score approximation rate (Theorem [
we obtain a neural network-based score estimation rate of Ö( 06 3/3) (see Theorem 20. This approach enables to avoid the need for the density lower bound assumption (see Appendix E.
for more details). Finally, applying Girsanov's theorem [5 17 yields a Onn-1/2 bound
in total variation (TV) distance for the distribution estimation error at the early-stopping time to (see Theorem 3). Moreover, if the target density belongs to a Sobolev or Besov class, controlling the truncation error at to allows to achieve nearly minimax optimal rates up to a logarithmic factor.
To summarize, we remove the Lipschitz score assumption used in (3) and establish minimax optimal rates for the empirical score function under a sub-Gaussian assumption. We derive score approximation and estimation error bounds without the density lower bound condition as used in [22 26 27 and show that neural network- -based SGMs can achieve nearly minimax optimality in TV distance, even under mild regularity assumptions.
The remainder of the paper is organized as follows. Section 2 introduces the notation and definitions used throughout the paper, as well as the background of SGMs. Section 3 presents our main results concerning the error bounds for score estimation and approximation as well as distribution estimation in total variation distance for SGMs. Section 4 offers proof sketches for the score estimation and approximation errors. Finally, we conclude in Section
S
We defer all proofs to the appendix.
2 Preliminaries and Background
2.1 Notations and definitions
We use R == fo E R|x= 2 0} to denote the space of non-negative real values. Denote by
N = 1......... the set of natural numbers and N+ N 1/ O. We denote by (0 La) the
Gaussian distribution with mean vector 0 and covariance matrix o² and write &a for its density. The standard Gaussian distribution on Rd is represented by Ya == N Ia). For any function or distribution on S2 supp(S?) denotes its support. Let (Xe) (o.7] be a process with law(X,) and corresponding density Pt- We refer to (Po. Po) as the target data distribution and density. We write a V = max(a, (a, 6] and a A = min(a, b). The notation a O(b) means I Cb for universal constant C > 0 and we use Ō-) to hide logarithmic factors. Throughout, S suppress constants that depend on the dimension d.
P
Definition Buu--Gasssian Distribution 40 ). We say a probability distribution P on Rd a-sub- Gaussian for some 0 < a < 3o if for all 0 E Id::
Ex Ex-P [exp(( (X - Ex [X)] S cxp 011/222.
(1)
Deep ReLU neural networks. We follow the notation used in 38 for ReLU neural networks, please refer to Appendix D.I for a more detailed introduction. We say that neural network (architecture) with width N and depth L if the maximum width of any hidden layer in the network is at most N, and the total number of hidden layers does not exceed L.
2.2 Score-based generative models
In this section, we introduce the background of SGMs. A SGM typically encompasses two Markoy processes: a forward process (Xt) 0.77 that starts from the target distribution Xo ~ Po. the model gradually adds noise transform the signal into noise X, - X2 - Xx ~ Pr and a reverse process Y, = XT-1; 0 S 1. < T starts with the noise Yo ~ Pr, and reverse the forward process to recover the signal from noise Yo + Y, > >.... Yr ~ Po.
OU process. We consider the following Ornstein-Ulhenbed (OU) process as the forward process:
dx, X,dt + V2dB, (0StST), X ~ Po. where B, denotes d-dimensional standard Brownian motion and we have X, ct V1 - e 21 Z, with Z ~ Td. The OU processis well-defined and has reverse process (Yi)
dY = (Y; + 2V log (X)) dt + 22dB (0&+<T).
Yo Pr.
(3)
where B¹ denotes another d-dimensional standard Brownian motion and V log called the score function. The noise distribution and score function are unknown. Using the fact that the OU

---

process Eq. 2 converges to standard Gaussian distribution "aa exponentially [66 17 for sufficiently large T. we can replace Pr by "d-
Score matching. Given a finite set of samples, we can train a neural network (score( approximate v Vlggro pe(ppe)))))) for E [0, 7] by minimizing the score matching loss 7
(ss(oscore)
Ex, [Il (x,,) 1) - V Po
L=
which is equivalent to the denoising score matching Exo [E(Oscore: Xo)] up to constant 7 8 where
Ex,,xo [110hcone (x.() V Px
E(Ouoree Xo) =
(5)
1=0
We replace V Prggrr-(()) by 60(x, T t) and obtain the score-based process:
di. = (8, 200 (Y, T - t))de VZdB' (0 (0S1ST),
Y0d Pr.
(6)
We replace Pr by Ya in Eq. (6) and obtain reverse process (Y, 10.77 that starting from Yod Ya- Early stopping. Instead rumning Eq. (6) back to the start time f 0, we stop early at small time to > Hence, the diffusion model will approximate Pt, rather than Po. i.c. we want P se Pt. Problem statement. Given the ground-truth data distribution Po and set of n i.i.d observations (x( J pon, we learn the score function V log VI E [lo, 7] via the empirical risk minimizer:
T E arg min dENN
Σ (()xx!
and plugin ē to the process Eq. 6 to generate samples X' P where P7d is the modeled distribution of SGMs. Our goal is to study the estimation error in total variation distance TV(Po, P.
Main Results
In this section, we present our main results regarding the error bounds for score estimation and approximation and establish the nearly optimal convergence rates for diffusion models.
3.1 Assumptions
Here, outline the assumptions imposed on the target data distribution Po and density function Po our analysis. In particular, our score approximation and estimation results are not specified to Assumptions [n where they used only in Corollaries and control errors induced by early stopping, respectively. We follow [29,22] for the definitions of the Sobolev ball and the Besov space, respectively, which are deferred to Appendix due due space limitations.
Assumption (Sub-Gaussian Distribution). The target data distribution Po is aasub-Gaussien
Assumption 2 (Sobolev Class of Density). The density po belongs to the Sobolev ball with 0 < S 2. Assumption 3 (Besov Class of Density). The density po LLO (]4) (B ([0, )]d): C) for some C 0, where S 9 I 00,0 < of S xc, and 0 < s < 2.
(B
3.2 Score estimation by regularized empirical score functions
Given set of i.idd. observations (2.1 drawn from an unknown target distribution Po. let p(n) = 1 -m Egto be the empirical measure. For OU process Eq. 2 we have p(() - N(e
(1 -2)) and p.(+) - P.. ()) where m,
exp(-1),ot = -1-expí exppp22 21).. [30] showed that for an a-sub-Gaussian distribution Po with L-Lipschitz continuous scores, the following regularized empirical score function, with bandwidth
h (a log 2/(d+1) and regularizer Pn (2th) (2/2) e/2 -2.
VPA(.) = VHE Pp (()) px(:) Pr. " D P (1) V Pr.
(8)
5

---

achieves nearly minimax optimal rate of (da2 2f4 log n) score estimation error in term of score matching loss. However, the Lipschitz continuous score assumption excludes many distributions of interest, such as those supported on submanifold and the Lipschitz constant can conceal additional dependence on the data dimension in some cases, especially when the data are approximately supported on submanifold. On the other hand, (2 demonstrates that similar truncated score V 1)) under merely sub-Ga Sab-Gaussian distribution, indicating that regularity conditions on the score are not necessary to derive optimal rates. We resolve this question with the following lemma, demonstrating that the estimator in Eq. (8) can achieve minimax optimal rates as long as the data distribution is sub-Gaussian:
estimator VAC(2
pe(-) (p<())>pp) attains a minimax optimal rate of On-to,
Lemma 1. For any 2 1. 2 3, Let be a a-sub Guu--Gassian distribution on R and P be its empirical distribution associated to a sample {- For any a Z an 1/d log 1/2 n, let
Po P * o2 pb = p(x) * o0 with density functions Po. R2 - R4. Fix
0< D<P. Pm S (2TT²) --ae)))--3--eee e then we have
E (z2()) In
(x)) Po
(ir(F)
I Po (x)
S So- 20- + +a (
log" (2 $
log n
(x) Pn
Pn
We provide proof sketch for Lemma I in Section 4.1 and defer the detailed proof to Appendix C.2 Remark 1. To establish our neural network approximation and estimation bounds, we adopt the empirical regularized score estimator rather than the truncated estimator as our surrogate. The regularized estimator globally smooth and stable, ensuring compatibility with established approxi- mation theories for smooth fumctions, while the truncated estimator's 's discontinuities at low-density regions violate these assumptions and complicate both approximation and generalization analyses.
3.3 Score approximation and estimation by deep neural networks
3.3.1 Neural network score approximation
Lemma 12 shows that the regularized empirical score function Eq. ( can be well approximated by a ReLU DNN in Lz-distance. Combining Lemmas T and 12 we obtain the neural network score approximation error by the following theorem. For the proof, please refer to Appendix D.4]
Theorem 1 (Neural Network Score Approximation for Sub-Gaussiaa Distributions). Suppose that Po satisfies Assumption [ For any 1 < d S Vlog n, n 2 3 and any - lo/ < to 1 and T = n o1), let (EEtEE be the solutions of the process Eq. 2 with density function logn Pr: Rdd + 12.. Fix ks E N4, with d/2 < k S logn' Then, there exists a ReLU DNN &ueone with width I On log, n) and depth I Ollog" n) constructed from from
i.i.d. samples {2(0) ME such that
log 4/2+3 n
E (1ot1))71 Ee P. [I[V (ox.)(())) @score ( []]]]] S o, (o a²) and we have l1 (,, llo $ o, Vlogn. n. Moreover, let T 01),, have
n
E E yum
E BR V log - desooe (x (1, 01 i a 1-22
4/2 n
log +
3.3.2 Neural network score estimation
Vlo Hence, we limit the According to Theorem I o(x,t) t) can be taken so that [|0(, S o, So, neural network class of Theorem iinto
:= == to E N'N S On logz n); depth & (log? 2 n)) [)((, S Vog
Together with the score approximation error bound from Theorem 1 applying Bernstein's 's inequality and an e-net argument, we obtain the following neural network score estimation error bound in terms of score matching loss. A proof sketch is provided in Section 4.3 and see Appendix for details. Theorem 2 (Neural Network Score Estimation for Sub-Gaussian Distributions). Assume that the 6 conditions of Theorem | hold. For 3 S d S Vlogn, n. fix N+ with (d-2) log(to 55 V d/2 < k S
logn Tog log n in Eq. (9). Then, for any 8E (0, 1), with probability at least 1 --5, 8, the excess risk of an empiricul risk minimizer Eq. (7) over the neural network class NN satisfies that
1" Ex,
2 V Vlogpe X, dl S to d/2 log $ n + 1o n log log 8

---

Our rate in Theorem 2 matches the rate for regularized kernel based estimators in Lemma and aligns with the nearly minimax optimal rate derived in 29 for similar truncated estimator.
3.4 Distribution estimation errors of SGMs for sub-Gaussian distributions
Here, we evaluate the distribution estimation error of neural network- -base SGMs in TV distance. The following theorem aims to bound the TV distance between the true marginal distribution Pt. at time to 2 a²n -2/d log n and the learned distribution Pla by SGMs. Proof refer to Appendix E.2
Theorem (Distribution Estimation Error of Pto). Assume that the conditions of Theorem hold. Then, for any å E (0, 1), with probability at least 1 6,
E ( (V Pit0] S a 2/22 4/4 1/2 log 2 +2 + n 1/2 1/2 log 1/2 n Vlo((2/5).
Somewhat unexpectedly, by choosing å 1/n, Theorem 3 shows that with only sub sub-Gaussian assumption, neural network- network-based SGMs achieve a TV distance bound of 2g 2/////////// -d/4,
n 1/3) for the distribution estimation error at the early stopping time. By triangle inequality, TV(Po. Pt,) < TV(Po, P...) + TV(Pto. P. indicating that to bound TV(Pa, Pt.)). it is necessary to control the error introduced by stopping early. To this end, we introduce nonparametric class assumptions.
Bounding the early stopping induced error. Assume that the target density function Po belongs
to Sobolev space W (IR)) (Assumption 2) or Besov space ( 1)d) (Assumption 3 with
to d+20 can be upper-bounded by Lemma 26 and Theorem respectively:
TV(Po, Pt.) S n af2e.
(10)
Theorem 3 Lemma 26 iimmediatlly imply Corollary [
Corollary 1 (Distribution Estimation Error for Sobolev Class of Density). Assume Assumptions 7 and hold. Let to = n 242. T = n (1)) Then, with probability at least n, it holds that
E (2
[TV(P., PT X polylog(n)) .ी..
Notice that Po on [0, 1] is (d-sub-Gaussie Corollary immediately follows by Theorems and Corollary (Distribution Estimation Error for Besov Class of Density). Assume that Assumption holds. Let to xiं T ==m- n (1), Then, with probability at least 1/n, it holds that
E [a 12 TV(Ps, S n -n-u..
Therefore, we have proved that neural network- based SGMs achieve minimax estimation rates for Sobolev and Besov class densities in TV distance up to logarithmic factors, even without the Lipschitz score assumption (as opposed to (30) and the lower bounded density assumption (as opposed to
22 26 27
Remark 2 (Dependence of Network Width Dimension d and the Variane Trade-Off). The ((n3/d) network width appearing in Theorems Aand suggests that the required width decreases with dimension d, which may seem counter- intuutvvee This is a direct consequence of the interplay between the smoothness of Pt,, and the network architecture in our analysis, which has clear theoretical motivation: (1) Higher dimensions force more smoothing. Our condition on the early stopping time (to 2 OŌn-2/4) in Theorems I and or to = = गंं in Corollaries and 2) forces larger for higher d to ensure theoretical guarantees. (2) More smoothing simpliftes the Tearning of Pto- A larger to means that Po, is convolved with Gaussian of higher variance. This makes Pto inherently smoother and less complex. Its score can thus approximated by a network whose size scales less severely with the sample size Hence, while higher dimension usually implies increased statistical difficulty (as seen in the convergence rate n -I suffering from the curse of dimensionality), the necessity of stronger smoothing to achieve uniform control in high dimensions makes the intermediate learning problem architecturally less demanding in terms of network size. This a reflection of the bias-variance trade-off: we reduce variance (by smoothing), but incur bias, which is ultimately what limits the rate.
7

---

Proof Sketch
4.1 Proof sketch of Lemma
We use the following two ingredients prove Lemmal 1
(1) From score matching to Hellinger distance via empirical Bayes smoothing. The follow- ing lemma shows that the score matching loss can be upper-bounded by the Hellinger distance H2 (P,,,, p(()) plus the Lz-norm of the score over low-density regions:
Lemma 2. Given any distributions P.Q on IRd with density functions p.q : Rd - RR,, respectively. ; For all d 2 1, 2 1, let 0 Pa (2mo2)-d/2e--
Fix > O, let Pa (O, o² Ia), Qu = Q+A(O, o² with density functions Po, qa Rdd - Rt.
and let == {xr Rdd pp( S pn)) then
there exists universal constant C > 0 such that
Jaal Po(2)
Vpo(#)
(a)(
2
P (z)ddrr
(0(x) V Pn (2To²)- 4/2
S c a max log max(log@ 02
(HP, 26)) (o Qo) Qo) + [1l Vp (xx) (x) Pa (z))d))
||
log (H log(H
Pn
2 is the L2-norm of the score function over the low-density region {xr E Fd Po S pm.). By Assumption P is a-sub-Gaussian and once convolved with Gaussian noise, the resulting marginal Po will become va² + -usuu--aassiann Using Lemma to bound the L2-norm of the score function and leveraging the sub-Gaussian property of Pa. we find that this term bounded above by Olo 2pn log(o 1o-pppp. "p, (see Lemma 7) For the first term, recall the fact that H²(PP P.) S KL(P!") (IP:) (see Lemma 32
For the proof, please refer to Appendix A.I The second term in Lemma
Remark 3. Lemma 2 does not require the density functions to exist for P and Q.
Remark 4. Notably, 130 Lemma 1] employs a rescaling argument to adjust the random variables and apply 44 Theorem E. yielding a bound similar Lemma 2 However, careful examination reveals that a term h d/2 has been missing in their proof. Simply Jollowing their strategy results in bound that scales
2-2 instead of a 2, On the contrary. we employ the rescaling argument to get a generalization result (see Lemma 4 of [41] Lemma F.2], and adopt similar proof strategy of M1 Lemma E. derive a bound that scales 2
(2) KL-ddvergence rate smoothed empirical distribution. The following lemma is a restatement of 42 Theorem 3). in which we explicitly demonstrate the dependence of the bound on the Gaussian parameter ar In particular, following the original proof [42]. one obtains a bound that scales expo- nentially with 1. i.e, E (L
< O(cxp( ) log" "), which blows up when is sufficiently small. We refine their proof and establish rate that scales polynomially in 1. given below:
Lemma 3 (Convergence Rate of Smoothed Empirical Sub-Gaussian Distributions). Given d 2 1, 2 1, o > O. Suppose that P is a d-dimensional a-sub- Gaussian distribution. Let Pa = P * N 0² and P be the empirical measure of an i.i.d. sample of size drawn frrwwnfrom P and Pa = P*N Ia). Then we have
Epgn [L(P). s Co (2) dlog2/2 n
(11)
To upper bound the first term in Lemma notice that I + x log x concave and is increasing in (0,e-1) by Jensen's inequality we have
For the proof, please refer to Appendix B.2
E p (o P(W))) P(m))
< log
1
Epan (P.P..PM))
Epaa (PP(P,,
d "log" d/2
log 4/2
(
log
(12)
where the last inequality holds by letting -aa 1/2 n S e1 which can be satisfied when (a/a)) S log d/2 n, i.e., a Z on -1/d log 1/2 n. Combining the sub-Gaussian tail Lemma an
Eq. (12) we complete the proof for Lemma Please refer to Appendix for more detailed proof.
8

---

4.2 Proof sketch of Theorem 1
The approximation rates of DNNs for smooth functions have been extensively investigated in the literature 39 43). At first glance, one might try to construct a sub-network for each exponential component of the KDE, leveraging these existing results. However, such approach would require sub-networks, causing the overall network size to scale at least linearly with n. In contrast, our proof constructs more compact DNN architecture, which not only prevents the DNN size from blowing up with n but also yields an optimal estimation error bound as shown in Theorem 2 Moreover, unlike [22], our approximation results do not require the density lower bound assumption. Recall that the regularized empirical score function with regularizer Pr. (2To²) ) d/2e n for the OU process Eq. (2) can expressed as
Vpr(y)
(3) 1 my x Jdd (y. 1) Sude yv.t)
P3( V Pa. of
Adde (y, V Ve-In-
where we denote by
1 Fide (y, (), Siae y. 1) = n
exp
lly-m, uy
kde (y.
(3)
exp
My 1y-mu mi. (9) 113 (8)
20
207
Thus, we conduct two steps to construct ReLU DNN to approximate Vpr(y)
Approximating Gaussian kernel density estimators (Lemma 8). Let h 11 m.
2n
and h(y t) C+ ( ))// for some s E N4. Whenever ( C where
Č = O(v sl)g(e-1)) for some 0 < e < 1, the term exp(-1(() becomes small enough to be trivially approximated by DNN. Consequently, we only need to approximate fkde for h E [0, CJ.
We further divide [0, Ö] into K subintervals. For each 3 {0, (0..,.... K 1}, define
QB = ( E R::::E
BC (B+1) K
K - 8. 18SK 1185-221}
For each 8, we define hg == K E of the Gaussian kernel density estimator at hg up order - is given by
E E0, I,..., K - 11 as the vertex of Qg. The Taylor expansion
frdely,
ha3 1)* exp( (app-hss)
1
n
(100 -hs)*
n
(- 1y expl 000) (1 hn)² (a) (b),
E
n
The second term (b) can be upper-bounded by 1 1 ha (c.f. Eq. 62 ). We can approximate
this term by constructing sub-network to learn hg such that 1h ha e for some 0 < e
fc.f. Proposition D. For the first term (a), some elementary calculations lead to
(a) exp(- h4)
(k leali "4mm (mmll hi gg g2n +2r 2)
n
(1) 22
of 3(6-04) 21 tilgleginn!
17
O= [ tor=k
we construct each sub-network to approximate of by o/ (c.f. Proposition 2) y" by open (c.f. Proposition 3 My t) by oū (c.f. Proposition 4h. ho(y, t) by &h, (c.f. Proposition 5). (v.
by hh (c.f. Proposition 60. exp(-ha(y,,) t)) by Ph (c.f. Proposition 7). By combining the sizes and approximation errors of all the sub-networks described above, we obtain sub-networks that approximate H(l fxaet), respectively. A similar argument applies to the approximation of ((3
Approximating regularized empirical score functions (Lemma 10). After constructing sub-
networks that approximate f + Mc, o, we can compose them into a single ReLU
DNN approximate the regularized empirical score function. For more details, see Appendix D.3
The approximation rate for the true score functions follows immediately by combining Lemma and the above approximation rate for the regularized empirical score functions.

---

4.3 Proof sketch of Theorem 2
Denote by Φ0 (: = V (-) and ((.,t) == vpr()
pil./Pnn. For any & E NN from Eq. (9) we have I. Ex, | x,,) - (x, 1)]3)d = Exxo (e(0, Xo) E(0", xx) Ex.
[R (2t0)) [L( Xo)
l(o Xo)]] + E E [( Xo) - =((6", ;o)]], where the second term can be controlled using the empirical Bayes score estimation result from Lemma [ For the first term, a key observation is
that the surrogate o(-,t) .) can be uniformly bounded (see Lemma 5), allowing us to verify Bernstein's
condition for the random variable R [ X - E(0, Xo)]. This enables us to apply Bernstein's inequality together with an E-net argument to obtain a uniform high-pree probability bound for the first term over AN. Combining these two terms and applying them to the empirical risk minimizer Ø trained over NN, together with the score approximation results from Theorem | we obtain an upper bound on ff Ex, (X,, oxx,,, t),)) 1l] This approach allows to establish the desired generalization bound without requiring lower bound on the density (see Appendix E.1 for details).
5
Conclusion and Discussion
5.1
Conclusion
In this paper, we introduce theoretical framework for studying the approximation and generalization abilitiees of neural network- based SGMs for estimating uub-Gaussian distributions on IR.. Using empirical Bayes smoothing techniques and neural network approximation theory, we established nearly minimax optimal convergence rates for SGMs without requiring strong regularities, such as Lipschitz score and density lower bound assumptions.
5.2 Limitations and future work.
Curse of dimensionality (CoD). One of our limitations is that our current bounds still suffer from the curse of dimensionality (CoD). Incorporating structural assumptions, such as manifold assumption explored in [44[55, or low-rank structures [46]) is crucial next step to mitigate the CoD. This would require non-trin mon-trivial extensions to our analysis, particularly in adapting our empirical Bayes smoothing techniques and neural network approximation theory to efficiently exploit low-dimensional geometry. In particular, our analysis relies on an isotropic Gaussian kernel for the empirical score estimator. To effectively leverage manifold structure, this would need to be replaced with an estimator that respects the underlying geometry, such as one using anisotropic or manifold-intrinsic kernels (e.g.. heat kernels), to prevent smoothing data off the manifold. A manifold- the path of [44 [4]. would require establishing new bounds for score estimation and approximation that depend on the intrinsic dimension of the data, rather than the ambient dimension.
analysis, following, e.g..
Going beyond sub-Gaussian distributions. Our current analysis leverages the aub-Gaussian assump- tion in two essential components: controlling the score function in low- density regions (Lemma 7) and bounding the KL divergence the Caussian-smoottee
empirical distribution (Lemma (3). Re-
laxing this assumption to encompass broader tail behaviors, such as sub-exponential or lub-Weibull distributions [47 488, poses both technical and conceptual challenges, requiring finer control of tail integrals and stability properties of the score. We believe that developing such extensions would substantially deepen the thcoretical foundations of score-based models, bridging the gap between idealized sub-Gaussian settings and the heavy-tailes distributions often encountered in practice. In particular, leveraging recent progress on the convergence of under heavy- assumptions offers a promising pathway toward this goal.
empirical measures
Higher-order smoothness. Our nearly minimax optimal rates in Corollaries D and 2 currently apply to smoothness parameters 0 < 8 < 2. This restriction arises because our analysis employs an isotropic Gaussian kernel-based score estimator, which is a linear estimator and thus cannot fully exploit higher-order smoothness (49 [5)] Exploring nonlinear or adaptive estimators that can leverage
higher-order smoothness represents another compelling direction for extending our framework.
A Acknowledgments
G. Fu is grateful to Jonathan Scarlett for insightful discussions on Lemma and to Taiji Suzuki for valuable feedback on this work. We thank the anonymous reviewers for their helpful comments and suggestions, which improved the quality of this paper. This research is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 1--0001814-00--0
10

---

References
[1] Sohl-Dickstein, Weiss, N. Maheswaranathan, and S. Ganguli, "Deep unsupervised learning using nonequilibrium thermodynamics." in Proceedings of the 32nd International Conference on Machine Learning, ICML. 2015, Lille, France, 6-11 July 2015, vol. of JMLR Workshop and Conference Proceedings, pp. 2256-2265, JogR. 2015.
[2] Y. Song and S. Ermon, "Generative modeling by estimating gradients of the data distribution," " in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11895-11907, 2019.
[3] J. Ho, A. Jain, and P. Abbeel, "Denoising diffusion probabilistic models." in Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 12, 2020, virtual, 2020.
[4] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, "Score-based gen- erative modeling through stochastic differential equations," in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, OpenReview.net, 2021.
[5] A. Vahdat, K. Kreis, and Kautz, "Score-based generative modeling in latent space." in Ad- vances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 11287-11302. 2021.
[6] Dhariwal and A. Q. Nichol, "Diffusion models beat gans on image synthesis," " in Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 14, 2021, virtual, pp. 8780-8794, 2021.
[7] A. Hyvärinen, "Estimation of non- normaazzdl statistical models by score matching," " Journal of Machine Learning Research, vol. 6, pp. 695-709, 2005.
[8] Vincent, "A connection between score matching and denoising autoencoders," Neural compu- tation, vol. 23, no. 7, pp. 1661-1674, 2011.
[9] H. Lee, J. Lu, and Y. Tan, "Convergence for score-based generative modeling with polynomial complexity," Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[10] A. Wibisono and K. Y. Yang. "Convergence in KL divergence of the inexact langevin algorithm with application to score-based generative models," CoRR, vol. abs/2211.00 1.01512, 2022.
[11] S. Bruno, Y. Zhang, D. Lim, Ö. D. Akyildiz, and S. Sabanis, "On diffusion-barrt generative models and their error bounds: The log-concave case with full convergence estimates," CoRR, vol. abs/231 1.3584, 2023.
[12] V. D. Bortoli, J. Thornton, Heng, and A. Doucet, "Diffusion schrödinger bridge with appli- cations to score-based generative modeling," in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 17695-17709, 2021.
[13] V. D. Bortoli, "Convergence of denoising diffusion models under the manifold hypothesis," Transactions on Machine Learning Research, vol. 2022, 2022.
[14] H. Lee, J. Lu, and Y. Tan, "Convergence of besre generative modeling for general data distributions," " in International Conference on Algorithmic Learning Theory, February 20-23, 2023, Singapore, vol. 201 of Proceedings of Machine Learning Research, pp. 946-985, PMLR, 2023.
[15] H. Chen, H. Lee, and J. Lu, "Improved analysis of score-based generative modeling: User- friendly bounds under minimal smoothness assumptions," in International Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, vol. 202 of Proceed- ings of Machine Learning Research, pp. 4735 4735-4763, PMLR, 2023.
11

---

[16] S. Chen, S. Chewi, J. Li, Y. Li, A. Salim, and A. Zhang, "Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions." in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, OpenReview.net, 2023.
[17] J. Benton, V. D. Bortoli, A. Doucet, and G. Deligiannidis, "Nearly d-linear convergence bounds for diffusion models via stochastic localization," in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net, 2024.
[18] G. Conforti, A. Durmus, and M. G. Silveri, "Score diffusion models without early stopping: finite fisher information is all you need," arXiv preprint arXiv:2308. 12240, 2023.
[19] A. S. Dalalyan, "Theoretical guarantees for approximate sampling from smooth and log-concave densities." " Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 79,
no. 3, pp. 651-676, 2017.
[20] X. Cheng, N. S. Chatterji. Y. Abbasi-Yadkori, P. L. Bartlett, and M. I. Jordan, "Sharp conver- gence rates for langevin dynamics in the nonconvex setting," arXiv preprint arXiv: 1805.01648, 2018.
[21] W. Mou, N. Flammarion, M. J. Wainwright, and L. Bartlett, "Improved bounds for discretiza- tion of langevin diffusions: Near-optimal rates without convexity," Bernoulli, vol. 28, no. 3, pp. 1577-1601, 2022.
[22] K. Oko, S. Akiyama, and T. Suzuki, "Diffusion models are minimax optimal distribution estimators,"" in International Conference on Machine Learning, pp. 26517-26582, PMLR, 2023.
[23] M. Chen, K. Huang, T. Zhao, and M. Wang. "Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data," in International Conference on Machine Learning, pp. 4712-4712, PMLR, 2023.
[24] F. Cole and Y. Lu, "Score-based generative models break the curse of dimensionality in learning a family of sub-gaussian distributions," The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net, 2024.
[25] Y. Han, M. Razaviyayn, and R. Xu, "Neural network-based score estimation in diffusion models: Optimization and generalization." in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net, 2024.
[26] I. Azangulov, G. Deligiannidis, and J. Rousseau, "Convergence of difffusion models under the manifold hypothesis in high-dimensions." arXiv preprint arXiv:2409. 18804, 2024.
[27] R. Tang and Y. Yang. ''Adaptivity of diffusion models to manifold structures," in International Conference on Artificial Intelligence and Statistics, pp. 1648-16566 PMLR, 2024.
(28) K. Yakovlev and N. Puchkin, "Generalization error bound for denoising score matching under relaxed manifold assumption," in The Thirty Eighth Annual Conference on Learning Theory, 30-4 July 2025, Lyon, France, vol. 291 of Proceedings of Machine Learning Research, pp. 5824- 5891, PMLR, 2025.
[29] K. Zhang, H. Yin, F. Liang, and J. Liu, "Minimax optimality of score-based diffusson models: Beyond the density lower bound assumptions." " Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, OpenReview.net, 2024.
[30] A. Wibisono, Y. Wu, and K. Y. Yang, "Optimal score estimation via empirical bayes smoothing." in The Thirty Seventh Annual Conference on Learning Theory, June 30 Jull 3, 2023, Edmonton, Canada, vol. 247 of Proceedings of Machine Learning Research, pp. 4958 4991, PMLR, 2024.
[31] Z. Dou, S. Kotekal, Z. Xu, and H. H. Zhou, "From optimal score matching to optimal sampling." " arXiv preprint 20900020099 2024.
[32] S. Li, S. Chen, and Q. Li, "A good score does not lead to A good generative model," CoRR, vol. abs/2401. 2024.
12

---

[33] R. Holley and D. Stroock, "Logarithmic sobolev inequalities and stochastic ising models," Journal of Statistical Physies, vol. 46, no. 5-6, pp. 1159-1194, 1987.
[34] F. Koehler, A. Heckett, and A. Risteski, "Statistical efficiency of score matching: The view from isoperimetry," The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, OpenReview.net, 2023.
[35] D. Yarotsky. "Error bounds for approximations with deep relu networks," Neural networks, vol. 94, pp. 103-114, 2017.
[36] I. Daubechies, R. Derer, S. Foucart, B. Hanin, and G. Petrova, "Nonlinear approximation and (deep) relu networks," Constructive Approximation, vol. 55, no. 1, pp. 127-172, 2022.
[37] J. Schmidt-Hieher, "Nonparametric regression using deep neural networks with relu activation function," 2020.
[38] J. Lu. Z. Shen, H. Yang, and S. Zhang, "Deep network approximation for smooth functions," SIAM Journal on Mathematical Analysis, vol. 53, no. 5, pp. 5465-5506, 2021.
[39] 7.. Shen, H. Yang, and S. Zhang, "Optimal approximation rate of relu networks in terms of width and depth," Journal de Mathématiques Pures et Appliquées, vol. 157, pp. 101-135, 2022.
[40] R. Vershynin, High-dimensional probability: An introduction with applications in data science, vol. 47. Cambridge university press, 2018.
[41] Saha and A. Guntuboyina, "On the nonparametric maximum likelihood estimator for gaussian location mixture densities with application to gaussian denoising," " The Annals of Statistics, vol. 48, no. 2, pp. 738 762, 2020.
[42] A. Block, Z. Jia, Y. Polyanskiy, and A. Rakhlin, "Rate of convergence of the smoothed empirical wasserstein distance," CoRR, vol. abs/2205.00 0002220000222,.. 02128, 2022.
[43] T.-Y. Zhou and X. Huo, "Classification of data generated by gaussian mixture models using deep relu networks," Journal of Machine Learning Research, vol. 25, no. 190, pp. 1-54, 2024.
[44] P. Potaptchik, I. Azangulov, and G. Deligiannidis. "Linear convergence of diffusion models under the manifold hypothesis." arXiv preprint arXiv:2410.09046, 2024.
[45] J. Liang, Z. Huang, and Y. Chen, "Low-dimensional adaptation of diffusion models: Conver- gence in total variation." " arXiv preprint arXiv:2501.12982 2025.
[46] P. Wang, H. Zhang, Z. Zhang, S. Chen, Y. Ma, and Q. Qu, "Diffusion models learn low- dimensional distributions via subspace clustering," arXiv preprint arXiv:2409.02426, 2024.
[47] M. Vladimirova, S. Girard, H. Nguyen, and J. Arbel, "Sub-weibull distributions: Generalizing sub-gause gaussian and nub-exponential properties to heavier tailed distributions," Stat, vol. 9, no. 1, p. e318, 2020.
[48] A. K. Kuchibhotla and A. Chakrabortty, "Moving beyond sub- sub-gaussianity in high- nigh-dimensional statisties: Applications in covariance estimation and linear regression." Information and Infer- ence: A Journal of the IMA, vol. 11, no. 4, pp. 1389-1456, 2022.
[49] T. Suzuki, "Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality," in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019.
[50] T. Suzuki and A. Nitanda, "Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic besov space," Advances Neural Information Processing Systems, vol. 34, pp. 3609-3621, 2021.
[51] Y. Polyanskiy and Y. Wu, Information Theory: From Coding to Learning. Cambridge University Press, 2024.
13

---

[52] P. L. Bartlett, N. Harvey, C. Liaw, and A. Mehrabian, "Nearly-tight ve-dimension and pseudodi- mension bounds for piecewise linear neural networks," Journal of Machine Learning Research, vol. 20, no. 63, pp. 1-17, 2019.
[53] M. Anthony and P. L. Bartlett, Neural network learning: Theoretical foundations. cambridge university press, 2009.
[54] J.-F. Le Gall, Brownian motion, martingales, and stochastic calculus. Springer, 2016.
[55] D. Bakry, I. Gentil, M. Ledoux, et al., Analysis and geometry of Markov diffusion operators, vol. 103. Springer, 2014.
[56] M. Wainwright, High- dinensionns statisties: A non-asymptotic viewpoint, vol. 48. Cambridge university press, 2019.
[57] H. Amann, J. Bourguignon, K. Grove, P. Lions, H. Araki, F. Brezzi, K. Chang, N. Hitchin, H. Hofer, H. Knörrer, et al., "Monographs mathematics vol. 99," 1983.
[58] Z. Ditzian and V. Totik, Moduli of smoothness, vol. 9. Springer Science & Business Media, 1987.
[59] A. B. Tsybakov, Introduction Nonparametric Estimation. Springer series in statistics, Springer, 2009.
14

---

Broader Impacts
This work advances the theoretical understanding of score-based generative models (SGMs) by providing approximation and generalization guarantees under minimal assumptions. By removing restrictive conditions such Lipschitz continuity or density lower bounds, our results broaden the applicability of SGMs to more complex and realistic data distributions. These insights may inform the design more robust and sample-efficies generative models, particularly in high dimedsiionnal or structured data settings. While primarily theoretical, our findings contribute to a deeper understanding of SGMs, which essential for their safe and responsible use real-world applications.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions, focusing on the theoretical analysis of diffusion models, particularly in providing generalization bounds. The claims accurately reflect the scope and nature the work without overstating the results. The limitations of the theoretical analysis and its assumptions also acknowledged, ensuring the claims are appropriately framed.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work are discussed in the conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
15

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All theoretical results are accompanied by the full set of assumptions, which are explicitly stated in the main paper. The complete proofs are provided in the appendix, and the main paper includes proof sketches to provide intuition.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
All assumptions should be clearly stated or referenced
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [NA]
Justification: The paper does not include experimental results, focusing solely on theoretical analysis of diffusion models.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
16

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [NA]
Justification; As the paper is purely theoretical and does not involve experiments or datasets, data and code are not applicable.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7/nips. //nips.cc/publl. wwtpc//ggs/e//eeee ppbllicgguudess/dddeuubiiii wwweesuubmisssooPe..
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [NA]
Justification: The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
17

---

Question: Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp/////nnrpppss.cpppppt////t/
Answer: [Yes]
Justification: The research presented conforms with the NeurIPS Code of Ethics. It purely theoretical, does not involve human subjects, and does not involve sensitive data or models with potential for misuse.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
18

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper includes discussion on the broader impacts of the theoretical analysis. We outline the potential positíve impact in advancing the understanding of diffusion models' generalization properties, as well as possible negative societal impacts if the findings are misinterpreted or applied without consideration of their limitations. This discussion is included in the broader impact section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks, it does not release data models that could have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
19

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: The paper does not use external datasets, models, or code assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpppppppsataeea thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not introduce datasets, models, code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
20

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: The paper does not rely LLMs methods presented.
non-standard component of the
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy /ttp:::////urriie cc. fc//oonferencee/22255///// for what should or should not be described.
21

---

Appendix
Contents
1
Introduction
Our contributions
2
Preliminaries and Background
Notations and definitions
2.2
Score-based generative models
3
Main Results
3.1
Assumptions
3.2
Score estimation by regularized empirical score functions
3.3
Score approximation and estimation by deep neural networks
Distribution estimation errors of SGMs for sub-Gaussian distributions
Proof Sketch
Proof sketch of Lemmall
Proof sketch of Theorem
Proof sketch of Theorem
10
5
Conclusion and Discussion
10
Conclusion
10
5.2
Limitations and future work.
10
From Score Matching Hellinger Distance via Empirical Bayes
24
Proof of Lemmal
27
Convergence of Smoothed Sub-Gaussian Distribution KL Divergence
34
B.
Rényi divergence and Rényi mutual information
34
B.2 Proof of Lemma
36
Score Estimation by Regularized Empirical Score F Functions
38
Sub-Gaussian tail bound for score functions
38
Score estimation error for sub-Guusseee Gaussian distributions
39
Score estimations along OU process by regularized empirical score functions
40
C.4 Score estimation along Brownian motion process
42
D Score Approximation by Deep Neural Networks
44
D.I Deep ReLU Neural Networks
44
D.2 Neural network approximation for Gaussian kernel density estimators
44
D.3 Neural network approximations for regularized empirical score functions
61
22

---

D.4
Score approximation errors by deep ReLU neural networks
69
D.5
Auxiliary approximation lemmas
71
D.6
Existing approximation results
80
Neural Network Score Estimation and Distribution Estimation
81
Score estimation errors by deep neural networks
Distribution estimation by deep neural networks
Useful lemmas for estimation
89
Convergence of SGMs
91
Girsanov's s theorem
F2
Girsanov's s theorem for SGMS
92
Controlling Early Stopping Induced Errors
94
G.I Sobolev class of density
94
G.2
Besov class of density
94
Other Lemmas and definitions
98
Fourier analysis
98
H.2
Distribution inequalities
98
H.3
Taylor's theorem
99
H.4
Nonparametric classes
99
23

---

A From Score Matching to Hellinger Distance via Empirical Bayes
The following lemma extends 41 Lemma F.2] by generalizing it from densities convolved with standard Gaussian distribution (0,, Ia) to densities convolved with general isotropic Gaussian distribution (00 o² Ia) for any a >
Lemma 4. For every pair of probability distributions P and Q on I& with density functions p. q R + R+ respectively. Let (y == Po z)p(z)d and qo ( == SRa Po =)q(x)) For S j S d and l 2 1, we have
/ (o)po(a)) (0 dx 4o -0 2k+1 (2TO))
inf
2 2k (P. Qo)
-a 2k- e-
22
Proof. For X P.Y Qu we let
X
p' law
* NO Ia),
Q' law
* *N(0, Ia),
and their density functions p', d' Rd + R, respectively. Let
X, = X + (00 o² Ia) ~ P rala),
Y, Y + +N(000 21a)a ~ Q+ Q*N(O,oTTa)
X' X
+ +N(0,Ia) p
о Y' Y
+N Ia) Q'
Then we have
X, oX',
Y, oY
and po (z) =0 (x) =0-)((3)) Therefore.
=
| (0)po(t)) - (2) dx =
1 - - d 0 (=)) / o kgpp
2 x
- / 10:0' 2
1 0 (
d
By 41 Lemma F.2], we have
[ (0pp' 2 -
of
4(2T)
4/2 inf e2v2k-1
{ 22 Q²)
2 /
-
e a²
By the senle-ivvarianee of the Hellinger distance, i.e.,
-P((''Q0), H2 (c) -
we obtain
J. 00 dx < 40 22++1 (2TO2)-
inf
a 22
2 2x) 16)
a 2
a2
The following lemma extends 41 Lemma F.1] by generalizing it from densities convolved with
standard Gaussian distribution N (0, Ia) densities convolved with a general isotropic Gaussian distribution N NO,0
o² o2)a) for any 0.
24

---

Lemma 5. Fix a probability distribution P on I with density function R - R. For all o > 0,
let Y = μx + o oZ where of Denote by Pa the marginal distribution of Y with density function Po (y) SE+ Po (y μad pp dd the density function of Y. For all y E 12 we have
P (V 2
(y( & Tr (T( 02 G2
V((po(u))
2
log (Pmo²)-af?
(13)
Pu(
0²
Po (
Moreover, we have
(V Po(y) vpl
2 S
o log (2xT²) 4/2
if0 ps (2To²) e 1/2
(14)
if (2nT²) d/2 1 1-12 1/2
Proof. If X ~ Pand Y ~ N(xX, Ia), then by Tweedie's formula, for every E IRd
(Pr(.)
1 Elux Y = y).
(.(+) 1=
=
(15)
Then we have
(p)(
= /
V (po (a:))
Po (2x2 )ddx
(V)
(1(2))
1=9/
J Txxp
||Y-
11
oz (Y - -xX)(Y - -xx)=
exp
I 1X-xX113
20
2
=
|Y-
exp
20
= -xiY=
1
Ia + go
( -xx(xxxxx)) =
1
02
1
la +
Blux X) - X) = y).
Then we have
1 0²
(P(poy)))
Ia +
Eluux Y) - Y =
(y)
7 (FLuX ")y=)))(B(ux YIY vi) + Cor(wx-Yi - VIY -v1Y= =y)
a (Blux -1 - Y|Y= - + Cov(uX = =y)
1 (22 (p)
0² (VDo(u))
+ 1 0o
-Co(LuXY =)) (by Eq. 15
4 Po (1)
(y
VPo(y) (y)o(y))
+ 11
Cor(X|Y = =))
(y)
Po(y)
Hence,
(P) (VPo
0² Iat (2(p(()))
12 Cov(XlY = =))
(16)
Po (y) Pv(
Pz(
which gives that
Vp)
(Vpo T Ia (P(p
Po
Po(y)
- Po(y)
(17)
25

---

By the convexity of x - expl 2 Tr( (f
we have
o² (po
exp
+
Po
YY = v1)) Mux -
exp
Tr
[
EE
exp
Tr
(UX - -
(by Jensen's 's inequality)
E
exp
|UX
Y
12
1 = v)
20²
11
Ex|Y=
exp
20²
|
311
exp
Pa (x: y)dæ
2n²2
exp
|| 2n
1
Po (y) [ po (x)
dæ
Po (
f exp
| 3
Po y x Po (x) da
2
exp
| 1
(2TT²2 d/2 exp
I 2
Pa )da
20
20
(8 (IXx~(UXX0'1
(3)
(2mo²)-af2
Po (3)
Therefore,
Tr Ia + v2(oo())))
2
(2TT²)- -d/2
log
(3
02
Po (y)
together with Eq.
17 which gives
VV.
(2
2
(2TT²)-aa11
S Tr
Ia+
log
(3)
Po (z)
σ²
P3(
Then, have
(1)ll2
- Po(
(2T02)-a11 d/2
log
Po (3) Vp
Po (3) vpl
o2
(3)
V 3 log
(2TT22 d/2 (M(
2 log (2m02) d/2
if po (y) P:
p=(w)
2 log
(2m02)-444 4/2
if Po (y) S p.
Po (1)
When (D) S p < (2mо²) -2/22 e1/2-11/2 1/2 have
since x + z log (9) is non-decreasing on (0, -el. we
/ir(l))l P- (3 Vp
2 V o
log (2To2)-d4 -d/2
Therefore, if (2mo2)-d/2) d/22 1/2 we conclude that
IV
2
(2TO2) -d/2
log
(18)
P7 (3) Vp
P
26

---

pf (2n02)-d/2e-1) 1/2, since I vo ( S (2e)-112, 1/2, where the equality obtain when x V
Then.
VVro (2)ll2 Py vp
= Po(y)
(2
log
Po V p 0²
Po(
(2mo²) d/2
- 11/22-111 1/2
I
0
Po( vp
A.1 Proof of Lemma
Lemma Given any distributions P,Q on Rdd with density functions p. q RRd + Rt, respectively. Fix > let P. (O, o² Ia). Qo = Q0N(0, with density functions Po 90 Fd + Rt. and let = for Rd Po po(x)) S Pn.. then
For all d 2 1, n 1, let 0 Pn S (2m02)-df22e""
there exists universal constant C > O such that
Jual Vpo(
(2)
2
_ (x()d
(x(
(0(x)) V Pr.
< C( 02 maxflog"
(2TT²2) d/2
1og(H- Qo)) Qo) + /
VPo (x:) 2
|| Po (x))dx)
On
Proof. With 0 Pr. I (2022) On > (x)) such that Rd Gi U G2.
1/2 let G1 = for Rd po(x)) 2 pn),92 for Rd:
f11
Vpo
Vya
2
Up
V
Up
V 12 po-
2 Po
Po
Po
g V Pn
Po
lo
Pr.
Po
go Pm
:= (I)
Case 1: 91 = fu E IRd po) 2 pn).
Vp
Vg.
2
Po
11
Po
4o V Pn.
2
Vp
2Vp
2(VPo Vqg)
2V
V
2
+
Po
11
Pa + Oa
Pn
Po + 4a Pn
+ Yo V Pm
V
2
40 V Pm
Po
Vp
Vga
1,2
2
2(Vpo
V46)
Po +
Po
91
+ qo V
Pn
Po
V
91
9o V Pr.
Vpo +
V
(go V }2
Vpa V
< 2
Do +4
Po
61
Po
4o
Pn
(po 4o
p2
Po go Pm.
(1-))
(1.b)
Bounding (I-a)
27

---

VPo
2
V
1)) (40
Pn. Pa P
(l-a)
$4
91
Po
2
10 V Pn
(Po + 90 V pm,)
Vp
V
(v V vpn
/Paa yo (v
vpn + /Po \Po)
-4
+
91
Po
V Pn
(po + V pn)²
(1)
VP
2
Vga
(V V V /Pa)>
+
91
Po
90
V
Pn
Po + 90 V Pn
(1)
Vp
V
(vo
Pp
8
+
(by Po 2 Pn)
91
Po
90
V
Pn
Po 90 V Pn
(1)
Vp
Vq
+
(VTa )
6
Po
12
V Pn
32
(2TO²)-a12
4/2
log
(VDo
-
(by Lemma
0²
Pn
91
Bounding (I-b)
(I-b) =4 4 //
Upo - Varlé
|V - Vyz
9. (Po 9o pm)) Po S
(19)
0 Po + 90 V Pn.
Case 2: 92 (xr IRd Pn Po (x))
Bouding (II)
Vp
V
(g0 V Pn VPo V 1
[ - Po
Po
Po
To Pn
00
Po lg V PPn)
(go PP,)) Po Vp Po Vp
Po V
2
J01
Po V pr))
Po
(go Pn - Po Vo) Po (Vpo Vgb) 12
Po
Po Pn)
(go Pn Po) Upo
/
Upo
Vq 2
$
Po +
Po
G2
Qa V
6o
V Pn
(1I
(lI
Bounding (II-a)
28

---

Vp
40 V Pn -oo)2
(II-a)
Po
0 ph
Pr
VPo
(1 V VPn - vPa) VPn + vpa)²
= 2
13 V ph
Pa
02
Po
Ve (V90 V VPn /Pa) 2 V Pa + Pa)
< 4
Po
d V Vh
Pr
(Vir - /Dr) 2 + (vPn. - vpa) )
(4r
Pn + Po)
VP
< 8
92
Po
2 V Ph
Vp
(2(V -vPo)+ + (vPn
vDo) (4o V Pn Ppa)
< 8
PPo)
Po
12
9 V PM
(by 90 V Pn 2 Pr 2 Po 20)
VP
2(Ver /po)2 (pn + Po
pa)
pn Po)
8
Po
902
Pe
23 p
Vpo 2
Pe 12 ( (VVr /pT)²² + +2p0)
16
(by do V pn
Pa Pa
82
32
(2Ta2)-a122 / (V9o /pb)² + 32
/
Up
<
log
90
Po.
(by Lemma
Pn
62
Po
Bounding (II-b)
Vvo Vg
(II-b) = 2
2 R
Po
|V Pa
gz 12
< 2
(by 40 2 Pr. > Pa)
92
4e V Pm
VP - Vg
12
< 4
(20)
82 Go V Pn + Po
Combining Case 1 and 2 Bounding (I-a) (II-a)
32
(2no?)-a11) dd2
Upo 2
(I-a) + (II-a) <
-² log
(v -vPo)² + 2
+23 32
Po
On
Ru
G2
Dr
32
(2TO2)-a11
4/2
/
Up
< σ2
log
(2 32
Po.
(21)
62
Bounding (I-b) (II-b)
(l-b) + (II-b) 54 f Upa - Vga 13
J Go Pn Po
For S < d and k 2 0, let
A3 j.k /
(0fpo(x) 8 (
8k Day Du
dæ, with 83
Po + do Pn
Then, we have
VP - Vgrl1
(I-b) + (II-b) <
4 2
(22)
3.1
Rd Po + Go Pr.
j=1
29

---

(1) Bounding 4 3.0"
(po (x) 9o (x))² dæ
4 3.0
JB Po
(x) + 90 (x) V Pn
v 20
( (z) ( Po ( + v 40 (x:))
dx
JR
Po (x( + 90 (x) Pn
(VPr( 4n (x) (po (x) + (x))) dx
2
JRd
(x) + 90 (z) Pr.
$ (VPo(x) V [=(x)))
Rd
< 2H Qo).
(23)
(2) Bounding 43,4 for 2 2.
Note that
/ (z)))(z) - (x)))
dx (
(0fpo(z)) - (x)))
Jg Po(x) to xx)
Pn Jac
Pn
By Lemma we have
2 2h
3 S
2k+1
inf
a 22 Qo) + -
(24)
Pro Pn(22o2)a/2
2
(3) Bounding 2311
[ (0f (x) 0 (x))²
A3 j,k
dæ
(x( + (x(x)) On
(f 4o xz) 04 (x) (x)))
dx
(x + g (x) Pr
Jgd 0r Crh (p= (p-
- go (x: ) (po (2:) 2o (z))8 (x()) + go (x) v pn dx
1
f 8h (po (x) go (z) )0 (po(x) (x(x) dx.
Pa (x) + 4a (x) Pn
Note that
1
(x) 0,40 (x)
0
Po(a)) + (x(x) V
(po + 9o (xx)) pm )
V (x) 2|2 + V (a:) l12
1
Po (x) go x Pr.
Pa( go Pn.
2
(2
log
(x(x) + go (x) V
σ
(by Lemma 5 and go 2 P.))
30

---

Therefore,
AJ.e S - Rd
yo (x))0 (v) / of -
(x)]0 (10 Po( + 40 (2)) Pn.
dæ
(po (po - go (z)) (x) go (x)))
dx
(x 90 x V Pr
2/2
(2T02)
10 (po 20 (x))) 10 (po (x(2))
V log
da
0
Pr
(x( + 2a (3)
Pr.
104-1 (p)
To (a)) 10 (po (a:) 40 (a:)
+
dæ
(x( + do (x) V On
Rd
2
(2mo2)-a/1 d/2
(0) (vo 4o (x))))
dæ //s (P 4o (x))))
$
V
log
Po + (x( v dx
σ
Pn
Rd (x) + go (x) V Pn
(po (0
4o (2))))
dr V (p= (x(2))))
+
dæ
Po(x:) + (x) Pn
Po(ac) (x(x)) V Pn.
2/2
(2TO²) d/2
log
Ai + AJ.4-
Pn
Divide both sides of the above inequality by 1 and denote by
2v2 V
(2nT22) 4/2
Cp.o,d
log
Pn
obtain
Aj,h Ajæ-1
Aja+t
< Cpord
for all k 2 1.
(25)
Aj,k
Suppose first that there exist an integer S S kn such that D S BA Then applying Eq. 25 recursively for 1,... k, we obtain
Di,1 S kCppaat B.
A.o
Then we have
4 < kC (kCnnmt + 3) A5,0
22 p. + 3 3) H(P.,.
(by Eq. 23
S 2 Ko H Qo).
(26)
On the other hand, suppose that Aju+i > BAj,t for every integer S t S ko. Then, by
Eq. 25 we have
Aj.
S Cp. + Aist1 < (1 Caord
A
for every 1,..., kn-
AJA-1
Dj.
B
Aju
Recursively applying the above inequality obtain,
D < ( +
Cpord
& *1
for every k = O...., Ko-
3
Taking the geometric mean of the above inequality for 1, 1,..., ku, we obtain
Dj,1
Gpadd
Aix+1) 1/(ko+1)
11
<
3.0
B
k=0
1
Cp.odd)
ko/2
1/(ko+1)
A -1/(ku+1)
3
jibtz
31

---

which implies that
1 $ (1+
Cp.add
ko/2
A
1/(00+1)
A Ka/ (k)+1)
B
j,b+1
3.0
Therefore, with k = ko + and Eq.
24
we have
1 - (1+ ( 4 AA(2TT2)-a))
d/2
inf
22 Qo)
8
Pn
a 2kk
An 250
+ ako aa²
(2H Qa))
1 (2TO²)
22 (1 (1+
Gpord
21 20o
inf
(2 Qo)
2
Pn
B
a2v2ho+1
a
2ko -a²
2ho
H Bot
(Pr
1 (2TO²) -
22 (1
Gpord
40
25
2k
inf
(H²((r, Qo)
2
Pn
B
a2v2ho+1
+ a²"
2kp
H hot
4o
(Pal
(27)
Choose k,,
Choose 3 ko kp.o. Eq. (26 becomes
,₂ S 2V C(p Qo).
(28)
and the term in Eq. 27
(
Cpadd
4p
=
=11+
1n)
Ag
S Ve.
(29)
B
Ko
Choose ki so that we have
(2x0?)
Skg 12
Ve and thereby the term in Eq. 27 will
Pn
kg
be
(2xo")
1
Cood
< This requires that
Pa
(2TO²) d/2
2hin + 2 log (2T0²) d/2
26042
exp
S Ve.
(30)
On
Pn.
Hence, for all 1, we can choose kn
1 be the smaller integer such that
(2TT²) 4/2
(2TT²)- d/2
log
+1 ko S log
Pn
Pn
Consider the term in Eq.
28 we obtain,
$(10 log (2T02)-
3/2
2 2 220 &
Pr
which gives that
8
(2ma²)-a/2
d/2 3/2
Dr.1 S 2 D.
log
(31)
Pn
Choose a² (2k + 1, log (H² Qo)) Notice that a 2 1 and
e a² <HH Qo).
32

---

Consider the term in Eq.
27
we have
(H'(Pa, (o)
Qn)) रहकंग S 2H (Po, Qo).
(32)
Combine Eq. (30p and 32 we obtain that Eq. 27 is upper-bounded by
A I cad
H(Po, Qa) I ea max(o= Qo)
(33)
llog( (2n0²) 4/2 ) we obtain
Combining Eq. and 333 and
Ko
(2TO²) d/2
د S max
3/2
ea, Eao eao- H((c. Qo).
On
Hence, by Eq. 20 we obtain
(I-b) + (II-) I
23 S 4d max {3z
64
log"
(2TO²) d/2
edaa?
c²a (PP Qo). (34)
Pn
o²
j=1
Therefore, by combining (I-a), (I-b), (II-a), (II-b), i.c. Eq.
and
34
we obtain
(x)
Vqa (x)
2
Po (x)dæ
1R (a)
(ar) V Pn
12
< 4d max
0² log")
64
(2TO²) 1-/0)
4/2
e2 22
(2T0²)-a/2
+1)
e²
log(H
log (H- (ogg(H2((o,, (P.)
On (2T0²)
On
32
H (P. +32 /
Vp
σ² log
Po (x))d
On
(x)
288d
(2T0²)-aaf d/2
Vp
max log
log
Log(H-2 (Po) Qo) Qo)+ +
Pa (x dx.
σ²
Pn
2
Gz (x
33

---

B Convergence of Smoothed Sub-Gaussian Distribution in KL Divergence
B.1 Rényi divergence and Rényi mutual information
Definition 2 (Rényi Divergence and Rényi Mutual Information [51). Assume random variables (X, Y) have joint distribution Pxy. For any À E RR
\ {0, the Rényi divergence of order
between probability distributions P and Q is defined
dP ((0)1)
1))
Dx(P)1Q)
X = log
EE
The Rényi Mutual Information of order are defined
(x; = Dx @ Pr);
where Px: Py to denote the marginal distribution with respect to X,Y, and Px @ Py denotes the joint distribution of (X', Y') where X' ~ P ~ Py are independent each other.
The following lemma is restatement of 42 Lemma 5], in which we explicitly demonstrate the dependence of the bound on the Gaussian parameter and relieve the exponential dependence of
Lemma 6. Fix a > let P.Z NN a² Ia). X 1 Z and Y + Z. Fix < <<< If
P is a-sub-Gaussian distribution, we huve,
AX;Y)) 1 (oo (2 - 2)d/2) + d log a d)
1
Ca
(35)
for some Ca > O depends only on d.
Proof. By the definition of Rényi divergence Definition
2 we have
1
dPx,
IxX;Y)
2-1 log
ExxOrY (a(Px @ Pr)
1
(( (dprix) y) dPy
log
EXXOPY
X
p (Y - -X)
log
C'EExPPy
1
(Ex (Po (Y - x))) (by Y X + to ~ No0, L.)
for some positive constant -X) [3x (Y X]| dy for sub-Gaussian distributions.
C
0. Therefore, only need to upper-bound
Ex
Decompose Rd =U, G as a union of cubes of diameter 2o. For any X E Cr we have
Y
- 112 }
Y
X12 [EE]]
Ex
exp
2 Pr(X c.) Ex exp
20²
20²
Fix any (non-random) X' E Ci by |X - X' < 20 for all X,X' X' E ci we have
X
|
X
X | - -X11))
< Y
X1
Y X'112 |||22 + 40²
3
1
Y
X'
(| X'112 + 40) 2 12o²
2
2
3
Y X 1 + 12o2.
2
34

---

Then, for any X, E C₁2 we obtain
Y 1
} 2 Pr(X q) Ex exp (- 3 - X'
12
6
Ex exp
x E ci
20
402
0²
3/ Y
x1
= exp(-6) Pr(X E ci) Ex
exp
402
Y - X
X11
exp(-6) Pr(X E c)) exp
(36)
402
which indicates that
X 20²
exp
(3 - A) - X
(Ex (Y x I exp (6(1-1) Pr(X E Eci) exp
402
1
Y -
S exp(6) Pr(X E ci)= cxp
(
402
Therefore, for any X E Rd we have
A|
exp
20²
/ Pr(X exp
I
X
(ExIeo x)])a=1 dy S
J&c
dy
402
= exp(6) EG) / exp
I X
dy
40²
S exp(6)(470))d/22 Pr(X Ec)
Taking expectation over X, we obtain
2 Y X)
Σ P)(Xx))) /
(Ex x
p (Y X)
Ex
dy)
(Y x(1)=-1 dy
Rd
[x
(Po (Y-
X)]
-P((XE Ec)) / (2TT²)- -d/2 exp -11Y-XXI
20
dy
(Exipol (Y - -X]])~-1
exp(6)22/2 Σ P(((XE))2A
L,t = (al(r-1)o= < ||si||2 < ra) denote the set cubes whose centers 8; belong to ((r-1)o S ||se|l2 < ra). Then we have |C,]| Card-1, We further let Pc, = Ec,ec, Pr(X E c.). If P is a-sub-Gauut Gaussian, we have for all e₁ E Cr.
Pr(X & Pr[[ - s:ll2 S σ)
(by e is a cube of diameter 2n)
Pr[[I[sel22 al < 2 S (1s||12+0)
s Pr([]]s:ll2 - o < X 22)
< eeexp (- (- [a/
(7000²
2²02
(by {(r - 1)0 < ||sall2 < ra) and P is (auuGussiian)
35

---

which gives that
00
8
c|C, exp (- (2 (2- 02 (2-
Σ Pr(X (x)
>>> Σ
Pr(X E c=)=²x -
Σ
a²
r=1,EC,
$ Σ Card-1 exp (- (2 - 2)r202
a²
s SCa (0 pd-le
12
p2 dr
(2 1107 12=2
d/2
CaT(d/2)
a²
(by So rd exp(_arr))dr
r( -
with > 0,T(d) = (d (d- 1)
a²
d/2
= 07
(2 (2-
2)o₂
Hence, we obtain for all 1 <>>> 2 2,
A(X
& d --1 log (//2//2cxp(0) Σ Pr(x Pr(xX cc))2-2
a² 4/2
log Cill
< x 1
(2 - N)o2
1 log
Cill
+ log -))
(2 x)d/2
B.2
Proof of Lemma 3
The following lemma is a restatement of [42 Theorem 3], in which we explicitly demonstrate the dependence of the bound on the sub-Gaussian parameter a while removing the dependence on the Gaussian smoothing parameter a.
Lemma 3 (Empirical Convergence of Gaussian Smoothed Sub- Lub-Gaussian Distributions in KL- Divergence) Given 2 1,n 2 3. a > O. Suppose that P is a d-dimensional distribution. Let Pa = P (0, o² II) and P be the empirical measure of an i..dd sample of size n drawn from P and P. = P (0, of Ia). Then we have
d llog
4/2 n
E KL((P s Ca
7
Proof. Let X ~ P,Z ~ o20, X 1 Z and Y = X + Z. Then we have Y X ~ \(X, o²),), which indicates that PY|x P ~ N0, La). Therefore, adopting 42 Lemma 4] and Lemma 6 we obtain that for any < 2,
E Epsn [L. IPa]
1 log(1 + exp((X (A - 1)(T - llog n)))
(2 - 22-2//12 ())) - (A - 1) logn)
Ca
<
A log [1 + exp
log
(by Lemma 6
log [ Can-(1-1)
(2 2/d/2 (0) d
)
A-1
Cd
a
5
(by log(1 + x) xx, Vr > 0)
1) 1) - 122 2)d/2
36

---

Choosing A A= 2
logn by n2 2 3 we have 1 < X < 2 and
1
n A1-11 logn 1 eexp - -llg v1
n log
(37)
We have
Esan (L(PolPo)) $ Cae(logn)u/2
(9) C2 (
d "log
4/2 n
(1 1/ log n)m log
37

---

C Score Estimation by Regularized Empirical Score Functions
C.1 Sub-Gaussian tail bound for score functions
The following Lemma follows similar proof from 30 Lemma 5], while our results do require the assumption that the parameters satisfy a S o:
Lemma 7 Subb-Gaussian Tail Bounds for Score Functions). Given a P. let Pa = P * NO o² La) with density function pa. Fix 0 < p S (2 dd22 e let = { R PP((() S p). Then.
distribution
1.1
(x) 2 (x) (x))dx
2p
(2
(32(a 2 +02) logn)) 2d3/2 n²o²
log(
o²
P
Proof. For some A 0, set A = μ + A]s. where y Ex-P, (X]. Then we have
/
Vpo
2 Po (x)dx
Po (2:
/ Jana
VP (x))
Po (x)dx + ( Ionu-
Vp (ax)
1|
Po dx
Po (2)
Po (x)
2
Vpa
Vpo
Po 1p S p)dx +
Po (z) 1 1 (x) S p)
Po
m
Then by the sub- Gaussan tail bound of Po with parameter va² + 02
Pr[X A = f Po (a))d s 2dexp
A2
(38)
2( + 02)
Let Xo P.X P., Z
NO0, Ia) and we have X|Xo ~ No2o, 0² r2Ia).
Up (x)
2
(II 5
Po (x)dxx
(39)
Po (x)
1
8
x|X 20| Po (x))x
(by Tweedie's formula Eq.
15
A
|
1(x $ 1/}
(by Xo +Z)
E
B 12 X 1(x $ {}
(by Jensen's inequality)
21
Pr A
(by Cauchy-Schswarz)
1
A2
(2d + +d2)2d exp
(by Eq.
and E 1 2d + d2)
0z
2(a² + 0z)
3/2
A² 2(a² + 02)
0²
V
eexp
(40)
Let A = / 18(a2 + 02) log then
8(a² + (o02+02)lop) ooogoogn
Pr[X A < 2dexp(
2dn
2(a² + 02)
which gives that
(II < 2/3/2
(41)
n²o²
38

---

By Lemma 5 and notice that 2: loog( ()) is monotonously increasing on [0, E) we have
(2TT²) 4/2
1
log
Po (o(x) ((po(w) - p}dc
(by Lemma 5
A x²
(x)
4
/ log
(2T0²2=
-d/2
pdx
4/2 e (by (x S sss 121202
0²
P
4p
(2T0²)-a/22
-d/2
log
(2A)a
4p
(2TO²)
(32(a² + log) 4/2
log
(42)
Combine Eq.
41 and 142
we obtain
Po (r)
2
2p log
(2T02)-0/2 32((a² + 22(00²00))ogg)) log
d/2 + n²0²
2/3/22
Po (x) v r²
(x)
G
C.2 Score estimation error for sub-Gaussian distributions
Lemma 1 For any d 2 1,n 2 3, Let P be a
distribution on Rd and P be its empirical distribution associated to a sample {a For any a 2 an -1/d log" 1/2 n, let o² Po P PO, Ia),, p(m) p(o) * o0, with density functions Po Po Rd - R4 Fix
e 0 Pm S )2 )
then we have
(x) (p)(2)
13 Po (x)
S So- 0d-2 + a²) log (2T02) - loga/2 n
Po (x)
(x) v
Proof. By Lemmal
2 we have
(p)(
2
Po (x)
(x)
(ar) On
< Cdo (lgg'
(2mo²2)-414
Vlog (H- Vlog(H HP(P,, (P. 32
Pa(())
Po (x)ddx.
Pn
92
Po (x
2
By Lemma 3 for all
0,
1,
E pen KL(PP.IPo)) C. a d 4/2 n
E E
HH2 Pa)]
Notice that x +
+ogg x1 is concave and by Jensen's 's inequality we have
Epsa
log (2'' P(i)]] $ log
E Epon
Epon [H2(P,, p(m)
Note that x + x log(2 1) is increasing in (0,e-1) If C -1 log e1, which can be satisfied when (a/o)² 5 n log d/2 n, i..., o 2 an 1/d log 1/2 n. Therefore, we obtain
log
Epen H (P, p(n)) )) Epen (P P()))
d 4/2 g Cd
s log Co n
n
(Iog(C7
nlog
4/2 m)
Cd
llog
Co (x)
d dlog/22 d/2 log
n
n
log
d/2
1
log"
d/2
<
logn
Ca
( log(() 1/2, Vr 0)
2
a < c/
"log"
d/2+1
n
39

---

Hence, for any d 1, 2 3,0 2 an
1/d 1og /2
(p)
Vpo(a)
2
E (2 12 ~Pn
Po (x))
Po(x)) Pr
(2TT2) 4/2
I Cdo -2 (log²)
Epen (H2 )( + Ep
log H (Po. H2 (2 P(
Pn
Vpa
2
+ 32
Po (x))d
Po (x)
62
(log*)
(2TT 2) d/22
log4/2n
(
+ log
po(ad)
Pn
92
Po (x)
loge/2 (2T0²)-a/12
dd/2
log 4/2
Vpo
So 2
log
Po (1:)da.
Pn.
G2
Po (2)
(by < Pn S -12e=))
Notice that log (2 ) 4/2
is nondecreasing E [0, (2na²)-d/22 e1. By sub-Gaussian tail
bound Lemma for all 2 1, n 2 3, 2 an 1/d log 1/2 n and < Pn S (2mT²) d/2e e -in we obtain
Vpo(a)
Vpo
E (x )2 N
Po (x))da
Po(x)
(a(x)) V On 1
(2ma²)- -1/2
log" log4/2
(2502) d/2 ((a2 + +02) (02))ogi log d/2
6312
S a 20-0---2 d-2 log
Pu log
+
On
n
Pn
1100
(2TT²) d/2
4/2 log d/2 n
(log + 1) +00) 4/2 log d/2 n
S aa -d d-2 log
d-2
Pn
(by 0 < S (2Ta²)- 2 12
(2T02)-a/2
log2/2n
log d/241
S a d-2 log3
d-2 (0- 00)
+ o
Pr
n
n
(2T02)-
d/2 1/g
5 20-2 (a o00) log'
Pn
C.3 Score estimations along OU process by regularized empirical score functions
Theorem 4. Suppose the target distribution Po satisfies Assumption Aand let Po be the empirical distribution associated
sample (xx(0 y For any d Z 1, 3 and any - log n < to S 1 and = n (1)), let ( be the solutions of the process Eq. 4 with density function Rd + R+. Let (z) Ei=1Por Po. (y - (x(()))) be the empirical density function. Let
Pn,t (2 -e- -2)) -d/2-11p Then.
f / Vpr(ar))
(x.( 2
1 pe 1 up8
pr (r S d/21
log
+4 n.
(xx.)) (x( V Pn.
Proof. For OU process, we have X, = e X, vI - N/ I2). With Assumption
- X1 e- -a-sub--aaussinn To use Lemma we need of =1 1 exp( (xt) 2 a²n 2/d log n, i.e.
t t² -log(1 a² logn and -2/d logn <
L. Notice that T' 2 to 2 2// 22/d log have for
n2/4
40

---

all t E [tu, T)
1 annn
+ a² log n
1
t >
2/d
log 2 2
log
log
2
n2/d
2
logn 27d
2 log a² log n
1
n/2
n2/d
n2/d a² logn
1
a logn n n2/dd
at log"
log
2
2/ (2/d(1) (n2/2 2/d + a² logn) logg n 2n 2n A a log n) a log"" n
a² logg
2
log
2
n2/d
1
a² logn
log
(by a2n log E 1)
2
n2/d
2nd 22/ddllogn) logn)
1
1 a log
log
2
n2/d
which gives that
- exp((-21) 2 a²n2 2/d log
and it follows from Lemma and Pn,t (2T(1 E =)) - that
E (( M pr 1/c0
Vpr(at)) Po(x)))
(pt( || (xx))dax
log 4/2+3
Se Sc v v))0-
pa( V Pn,
Case 1: When 2 1, we always have 0i. vt 0.
Vpr(
[ ~P8"
Po P dxx;
(x()
(x(e) V Pnit
Se dt a
-2 211-d/2-11
d/2-1 log d/2+3 a
2
d/2
1
log
d/2+3
T-
2-2
Note that
exp(-22
1
1
1 exp(-22 I ^
for any >>> 0.
(43)
To see this, let f(t) = - exp( 2t) 1 (exp(-20)) 2f), Vt E [0, 0o). Since f(0) 0 and we have
(t) 2exp(-222 exp(-2t) ( + 2/ eexpp2220 exp( 22) 24 exp(-22p 0, for any 1 0,
which indicates that f(1) 2 f(0) = for any > 0 and validate Eq. 43 Then, for any T 2 ln > 0, have
1 (111- (1--0-0
e-2
d/2
T dd//
dt $
+ +1)dt
(by Eq.
c- 2t
1 - c
10
dd2 dt +
t
d/2
o1
d/2
|=
2
dd//2+1
t=T
1=to
d+2
1=
1 d/2
2
4/2+1
d/2
<
to
d
which gives that
Vp.(
(p((
2
Ps (xx))x
1=1o Rd
(((xx)
p((xx) Proc
S a
log
logsto
n
T
e 2t
d/2
dt S at
dd/2 2
$ +3 log n.
tmty
1 e -2
1
21
41

---

Case 2:
When 0 < 1
Low noise region: When ot = V1-expp 2(), which indicates that to <<< it follows from Case that
log(1 -a?).
1og((_aa²)
(x.)
Vpr(
11 (d
E [( ~
R Pe(az)
pr(ax) V Pn,
$ La4 2/2 - log
+3
S to
4/2
n
log +3
High noise region: When a S or, which indicates that - loool(1 - a Sts we have
Vpr(ar))
(a()))) V Pnie | [ ] (1- 20) log
Vpe(z))
d/2+3
E Be pon
Po(a))
11
which implies that
t=- 1/ log(1 a22) Jec Vpr(xt))
Vir(x))
V Mn,t 1| Dr (ort dar
(r(()))
(r(
log" 4/2+3
e dt
log" 1oga/2- d/233/ n
T
12
} 122) 1 e dt -
E dt
t=
t= loR(( a0))
1 1
log 4/243
T
log
d/2+3
T
1
el e- dt 5
el df
2n
$ log(1
2n
1=
1/2 log(1 (4²) a²)
log2/24 4/243
T
1 dt
(by exp(1)-1 St-1, 1/ 0)
2n
-= log(1 -2)
t
log d/2+1
12
C.4 Score estimation along Brownian motion process
Brownian motion process Consider the Brownian motion process as the forward process of diffusion models:
dx, = ddB (0SIST), Xo ~ Po-
(44)
It has an explicit solution
X = Xo + Vtz (0 STS)), Z ~ ~~N(0, T Xo.
(45)
which implies that Xo N(xo, La). The reverse Brownian motion process is given by
dy, V loypr- dB. (0SIST), Yo Pr.
(46)
Corollary 3. Suppose the target distribution Po satisfies Assumption Aand let Po be the empirical distribution associated to a sample (0))) For any d 2 1, 2 3 and any Ja²n -2/d n &
to < 1 and T To (1)), let {wr [1E).00) be the solutions of the process Eq. (44) with density function pt _dd 18.,. Let pr(y) Ei iPo (y (a) be the empirical density function with at /1,m, = Lel Pa,t = (2ml) -d/2 1- Then, we have
E (201)) 1010)n ~Pn (TT Vp((z))
1 (o s C2a "'oo""" Rd Pr( pe(xx. Pn,t
Vp((z.))
Proof. For Brownian process, we have X, = Xa VIZ, 2 No0, Ia). With Assumption X 2/d
is a-sub-Gauusian. For any d 2 1, 2 3 and any T > to
2 1 - log n, by Lemma we have
(p.( (20.) (x() V Pn,
1 (1:))dd
log 4/2+ n
E (2(0))) >pp
< Ca(aat d/2 1)t
n
42

---

Low noise region: when a > Vt, For any 0 < la S a², we have
t 1-/22-1tt dt
_
2 d/22 d/2
t=a²
d
2 -a- to d/2
aa
I 0
2 d/2
d
t=to
Therefore, we obtain
semto Jzal Pr((tt) ()())) V Pn,t || (1
Vpr(o.)
Vx((
E (a
I a log 1og
n
f-d22-1ds S 20 log +3
7
ttt
High noise region: when a S Vt.
T
12 Dr (axe) Pnit
Vpr()))
E (xx(t)) (7
(r( ddx
t=a? JRs
log $ +3
n
tdt
n
t=a²
oog 43 n
log 3
(logT 2 2loga) x
n
n
43

---

D Score Approximation by Deep Neural Networks
D.1 Deep ReLU Neural Networks
We follow the notation used in 38 for ReLU neural networks. For a function & E (#iinpp
d; widthvec [Ni, ................................ N #output 1). if we No and NL+1 Then, & can be represented in a form of function compositions as:
Lr. ReLU o Lr.-1 ReLU Li ReLU o Lo:
where ReLU : + R denote the rectified linear unit, i.c. (eLU(()) max(0, x) and C is the i-th affine linear transform in & with weight matrix W; E IR. N++1 x.N; and bias vector b, E RN i.e.
Ui+l W; U. + Li,))), for 0,1,...,44
and ün E RRd u. (u, for 1, ... T... We say that neural network (architecture) with width N and depth L if the maximum width of any hidden layer in the network is at most N, and the total number of hidden layers does not exceed L.
For simplicity of notation, we write ||
ll as | throughout this section.
D.2 Neural network approximation for Gaussian kernel density estimators
D.2.1
The main result
Assumption 4. Given sample (xx(0)
Ji=1 of size there exist E N4, 1.000<<<1 such that sup iEin]
a ( S /2as log(c-)..
Lemma 8 (Approximation of Gaussian Kernel Density Estimator). Given a data set (x((t In let me = =exp(-t), = VI-exp(-200 exp(-26) for any t E [tu, For any E Rd, t E [to, oo), the Gaussian kernel density estimators is given by
Diz
n
y m2x m
frae 1)
exp
(47)
20
Fix any 0 < <6<, 11/2 there exist N, T,,& E N4 such that N-21, S and Assumption holds.
Then, there exists a function @ude implemented by a ReLU DNN with width S (g N' (N) V g 1ot+:logg (eg"(e-1)) and depth & O(L3 logz(L) V s² log" such that
(B, fude S 3 (2s)ss log
log (E ()",,, for any y E RR,, [to. 00),
and 0 S Oeac t) 1.
D.2.2 Proof of Lemma 8
We decompose Rd = B B, where
B = (y Rd ly| < 2 2as
2as logle 1)}
(48)
(y E Rd ly! > 2as log(c bog(c=1) 1)}.
(49)
We approximate faa on y E B, E [to, oo) in Part I and B,t E [to, 2o) in Part II.
Part Approximating /ude on B
Fix any N, L,, E N_, here aim to approximate fua for B. By Assumption 4 we have
y ( 1/
(V2as log(c-1))
2s log(e-1) log((->), e
(50)
2
207
which implies that
feac -ssslsllleee 1)) =E", for any y E B, (0, oo).
(51)
44

---

Therefore, (3. Vy E By.t& E simply setting the output of the neural network be zero. Therefore, we only need consider the approximation error of neural network for 1B, E (0).
Ito. so) can be well approximated with an error within by
Part II: Approximating fkde on B
In the following, we prove the neural network approximation results for Gaussian kernel density estimator Eq. (47) for y E B, E [tu, so). Given { N for any y E B, [tu, 20), denote by
h 1 man
for any i E [ns].
(52)
20
Then we have
2 2 + 2m (1x
0 0 h
100, as log(t S 10 log(e
(53)
20
Step 1: Domain decomposition
Let 0% be defined as in Eq. 138 By Eq. 142 we can find universal constant Co depended only on d such that
0 S 05 () S Cato as log(c for any y E B, [to, 00].
(54)
Set K = TA where 5 E (0, Cato as log(e -1 /K). Let S( Cato log(e ())((- K, 6) partition
[0, C as log((- 1)] into K cubes Qs for {0, 1,..,, K 11,, where
Q₃ = { E ::
Bost loslog(e-)) (8+1)Ent
'aslog(e-1) (a+1))aaaasllo((("))
6 1(88k-2)]]
(55)
K
For each 8, we define
h8 = 3Calo aaal-taslgghe los(og(e-1)
B E
K
(56)
K
Clearly, [0, Cat5 as log(e-)) )] = S2[10, C (og((e))))),BB K,)) (UBE(U.1...... KK- Qa) and hB the vertex of Q with minimum -norm.
Step 2: Taylor expansion of the Gaussian density kernel estimators
For all ( ()) H
denote by
(u. (..)))) 1/-
My, 1) =
(57)
Clearly, we have h(,, t) E [0, 10 as log(c [)] for any B,t E [to, 00). By Eq. (141).
1o (v. t) h(y,t) t) S a s!s 2s+2 log 2c ))"
We choose hg for B E {0, 1, ..... K 1} such that for some y, t, we have
(y. ho(y, - ta log(e
(58)
K
In what follow, we use h hly f) and hs ho(y,t). For any y E B,t [to, 00), we have
h - hol < hh - o5 (v, (1,, - hal
$ Eq. 141
S s!g 20 20+2 log 2++2 2 ))²² + to tas 1og(c-1)
K
to to'aslog(c NAIA
aaslg
log 2042
)
S 202gs 20t2
log
20+2
(c
)0 + as log((
(by N- S S (t)
S 2888 20+2 log 20+2
)c
(59)
45

---

The Taylor expansion of the Gaussian kernel density estimator at ha up to order s 1 is given by
j m (1) !?²
exp
20
n
exp( exp(-1 (11
1
exp(-hha (1100 ha)* * + [-11) 111)exx((000) -000 (1(00 - h13)
Ei
ol
i=1
k=
expl
h2g 1
(n)) - hg)
1
1)" -000) (1:(0) - hg)²
(60)
k=
1=1
for some real number 8(0) that is between (1 and hg. By Minkowski's incquality, for all s 2 k 2 0,
(E00))))
1/k -
( Σ (2()))) 1/0 =h. h.
(61)
Eq.
59
and
66
give that
exp( (())) (x))
(h - -hg)
s!
(n))) & (-ha))
(by (o - y)a = k== (-
s!
K-!
k=0
1
( |
(-1)
s
(-hh) 3 k7p
| -
(h h1)*
s!
M
7 i=1
h
s!
k=
5h* by
Eq. 6
< h hgg 5 2 ss
a s!s2
20+2 log
2st2 (c
a² 20+2 log
2 (c-))c-
(62)
Denote by a = (ai, 02 uz, u4] N2 and
O
1
(z)
us +2
(63)
1=1
we have
1
(h
hg) &
n
1
m2x 10
hg)
71
20²
1
ll
11 -2oshs)* *
2 2
1=1
k! |lll
201
(-2m,y T T x mp
20s ar
11
203 x (-20ūh1)) )as
oF 2k
n
1=1
||a||l=k
k!(-
2)
ta
my
a2t2a3
Jaa
yity
20
Σ
a2!
1
(0) 22
es!
( (a.(i) 201
a!200?
2k
224
201
Ds!
es!
|
M
| 1=02
|| wwsl
Σ
el(-2)asta,
ml
22
Σ
201 Σ
12
(ao m2tt
||l|| A
0z
2h 201
M 11: m1
mg! y
1 li mag
Uz!
||sa||1 as
Ui!
71 1=1
O k![ - 1)
12 ta tacm!!
ml!
||22|| +
Σ
Tig y yo0
(64)
[e + [Iall +ll tata
a4 || as
0x
2(k as)
21 leng
46

---

Denote by ú= = H2,
12, Us} E N44 we obtain
k expl - ha)) 1
(h()) - hg)
k!
n
1) exp(-hhx)
C k1(-1)
1|22 1i +211ssll
E
hgg x yamtes)
k!
|| ll |elllt lallealli ta 2 1122111 ass
2(k- 22)
29 rgleglezsaa!!
10e 1|l 22|pssl1
Oua m,
exp(-hag)
hg z221--))
(65)
k= |ال| ta (-2) 1ls 1: iasa 2(2 2(k-as) as) en lnlllllgaa!!
as
Combine Eq.
660 and 665 gives that
1
y mx 11
exp
207
t
exp( - -hs) 1
n (1) - -ha)*
2 (-1)* exp(-0(()
(1(()) hn)
k!
n
s!
8-1
Cza 2/ M || 11: +2|lesll
(xp(-hs)
ng y 2v +22
(-2) -asa
2(k as) en !rlglaugaa!!
k=
|1l|| +ar=
1) exp(-001
(h(() - h1)
s!
where the second term is bounded by Eq. (62
(-1)² exp(-o00 (())
(n - ha) S a 2 22 2og 2 (c-)(c)))
s!
In what follows, we aim to construct a ReLU DNN Økde to approximate the first term.
Step 3: Construction of the ReLU DNN dhde
For each & E N3d, 04 E N such that ||@|| + a4 S 1, have 0 ||2||1 2| ||: <
2(s - 1), 2/| 1l +
| < 2(s - 1). For each A: = 0, 1, 1, 1, we have 0 S k - 04 S 8 1.
Then, by Propositions and 7 there exist
das hg
E.N
22 (N) v g 3d42 log² -t); depth x L logg (L) V 8 log (c- 1)
(66)
o ha
ENN(Wiitth S 23-22 (og V 3d42 ()-); depth X L log (L) V 32 log 1)
(67)
such that
1 (y.t t) hg (y. 1) S 22222 log 20 (E- (6-1))) 1-N--44-- N'N')NN---
L44,
E [Lo, 00),
(68)
oNp he
(y, t) (ap(-ho)((a t) < N T- AA E B, E (to,
(69)
and
0 < S (g, 1) S 10 a4 (Eg"((--))
(70)
0
(y. 1.
(71)
Moreover, by Eq. 132). (133) and 139 for each i N3d, a4 N such that |||||| + as 5 k k
0, B.as (WiWhh S s Nogg (N) V 3 (c depth S s² L v 2 (c (72)
s - 1, there exists
47

---

such that for any y E B, E [to, 00),
bime (1, le ll +2ljesll1 Φ1
m,
2(( you to Cm. S (2s)ss 10g"((
(73)
and
S 0 plas y. 1) a log"
(74)
By Lemma
13
and Eq. (70 and 74 there exists
(3) @ malti
E NN (width S 99V + 1) + 1, depth - 14sL),
(75)
such that
(p, (y, - -h0 (y, x a.a0 (v.1)) I 6l k a ah log"(e (E () "00 log"" (N + +1)-1bo1
P Pilli ha (y, 1), 19 p.00
x 10 25 220 log -e
(by Eq. (31
S a 20 20 log (E -)020
(76)
For each ú E Nad, E N such that | 1 +
S k. k = 0.1...., 1, define
p.as p2 (y.
y. 1) = Pabs ( (0gg (g. (), 2 ma (y, 4)), for any y E B, E [lo, xc),
(77)
where Paaa (I) ReLU(x)) (ollU())+eeLU--) |.c/ for any x E R. By the size of he P1 Ommll Pabs
we have
B.a E @WW(ditth S 8 32 logo V 3d+2 2og
$ 1,2 logz (L) V 8² log (c-))
(78)
and for any y E B, E No, 20),
2 p.as (y.t) t) hgg x mi
|| +2 22ll
- 2(k 2(k y tes c
0f
< mmit (0%g (y, @1 3 (y, t))
(y, x 01
B.as (y, t) + 00
(y, t)
hgg
01 ( t)
2.04
ha
$ Bq. 76
I Eq. 68
X Eq.
74
(1 (y, t)
m,
p +/|
2(kk y two
t Caa
S Eqg 73
S 2n 28 20 log 25 (c (c1))) + a 20 log (e N
4s I, 40 "r"" log"(6
+ t 00"00" "00""og"' log" 1)- a (28)
log (E (6 -
S $a²*(3s)ls" log (c ))²
(79)
which gives that
0 < 2
0,04 (..4 1) S l x m
|ual +2llesll
0 -2(k-ae) you Cun.us
+ 222"(2s)l (e ))
S to as ktas (slog(c-1)
|mn +1llslllesll +1 +1e
to 1| - log" s-1 1c-
(80)
Similar to Eq. (134) for any i E N7d. as N4, we have
Σ
1=
k + 3d
(k + +k)3d)!
S (A: + 1)3d
3d
(3d)!K!
|P +llt ENEN
which implies that for any 8 E N4.
Σ
Σ
S
Σ
+ 1)3d
3d
< s.
(s 1 1+ +)33 = 33d+1
(81)
k=
| +o
k=
48

---

With Eq.
80 and 81 and the face that ||2 + 112l + |1sll1 + S s - 1, we have
2 2k aas)
2.as (y,,t) S 13d+11 (0 a- 20=10" "log" 1og"~1(E") (E-1)
92
Vi lelllagga!!! tla
10 ( a 1 3d log 2 (6 (6
(82)
Again, by Lemma 13 and Eq. (1 and 82)
there exists
E WWN(width
S 9(N + 1) + 1, depth S 7sL),
(83)
multi
such that for any x E [0,1] (c.f. Eq.
71 and E 10 (5-1))))))01110000 13-1 g3d+ log (c-") (c.f. Eq.
82
2- (18-11ll1--4
mmum
O
(y,
piar (2.
nlllglzgla!!
k=
||]|| ta,=k
2 (k- well -os)
(y, t)
02 02 (z.t)
hg
km
|ell ta,=k
S to (s-1) a 4-1 3dt log" x- (c
(c (N + 1) 1 776
(84)
exp Given h2 02 p.as
Pmutu above, for any y E Rd E [tu, oc), we define
2 (k-llel11 -as)
(v. (
Cred (y, dabs
m
o
oxp
(
t),
viluzlezlas! 02
0.04
(85)
0=
|||||
+āy=k
By Eq. 67
78
83
88
and (III)
we obtain that
chde (Wi(tih $ 6d+3 N3 (N) V s 6d+3 (e-)))-)) depth 1 (L) V v sooog (c-)))
(86)
Step 4: Approximation error.
49

---

For any y E B and any E [to, 00),
1)ah expl (1--h, hs) 1
@kdo
Σ
(h(0) - hn
k!
km0
2m1
2 2-(1-1u
Pmutt
hpp
(xy, (y,
1 1
02 02 (B)
0= [elli +a,=
2) (K- (k-11e2111
expl (hpp
bg m
| o -((k-as) you tes 0 Cp
V !llgluslan!
k= [|]] tta==
2 (-(--1u11--0)
Pmulti
hp
(gy, t),
1 1lleleslag!
02
B.04 (g.
k= |ell1 +a,=k
2 (k- [al))-a)) uiluzlugla!!
exp ( 1)
3.a4 (y,
1)0
($ Eq. (84)
hg
0= ||]|=
2 (k- ||lll -as)
+
eNP ho
yU expl c
v1lusluglay!
922
20.04 (2)1)
k= || || ta.=k
S Eq.
$ Eq
2 ( ||l|| -44)
+ exp exp(-ha)) ha)l
0/2 12.00(y (y, 1) hig m || - 22(k-au
2 ten Curz.
v v1l22lugla!!
k=o |1lll k
Fq.
S t0 (1-1)
a 1 go to (o (C + 1) 1)-=o
TuL + +N 48 L NN-4
.t ty (0-1) a= x- 3d+ log - (c-1)
+ sdd+1 (2s)la S0 log (c 1)²²
I a ds (2s)) 3dd99++4 log 90 (E (c²
(87)
Therefore, for any y E B,t E [L,, 20).
1 (B, t)
(ude(y,
1) exp( h2)
Σ i=1
(h
h1)
+
1
(-1) exp(-O(i))
(n h0)
k!
n
s!
kmm
$ Eq. 57
$ by
62
(88)
S a (2(2)1133+9941 log 00 "c (6-))) + a2222022 log log (c
S a 30 (2s)1s 33+)5533+ log 95 (c1))))
(89)
which gives that
S (B, S fude +
26+1 log (c"))"" z
(90)
Combine Part and II
Combining the approximating results from Part I (c.f. Eq. 51 and II (c.f. Eq. 89 we obtain that
(1, - frae 1)) S a (2s)) log 9 (E )e"" for any y E Rd E (lo, oo), (91)
and
Okde (g, S
Therefore, finish the proof.
50

---

D.2.3 Neural network approximations for basic functions
In the section, we give the approximation results used in approximating the Gaussian kernel density estimator functions:
Approximating my, of by Pm: 022, respectively (c.f. Lemma 9
Approximating m. by @t (c.f. Proposition 1]
Approximating of 22 by ///-= (c.f. Proposition
Approximating y" by oott (c.f. Proposition B
Approximating 7((,, 1) by Φ4 (c.f. Proposition
Approximating h 1) 0h (c.f. Proposition
Approximating h3 (yy, by of (c.f. Proposition 6).
Approximating exp(-ha t)) by (c.f. Proposition
sexp he
We give detailed derivations for the sizes of neural networks and approximation errors for approxi- mating each of the above functions as below.
D.2.4 Approximations of m. and of for OU process
Lemma 9 (Approximate me, of for OU process). For all [0, oo). let me exp(-t) and ot = VI ex)(-2t). For any N, L, s E N+ and < e < such that 2-2 S e, there exist some functions Φm Φa2 implemented by some ReLU DNNs with width 48 + 1) log, (8NV and depth 18 + 2) log, (4T.)
such that for all t [0, log(c")),
(0m(t) me S 8* log" (e-")e" 100= - of S s* lolo" (c 1c"
Proof. For OU process, we have m, = exp(-t) and of exp(-2t) for f [0, oo]. Therefore, both m,, of can be approximated by some ReLU DNNs that well approximate the exponential function exp(-t).
Step 1: E (s (slog(c=)), 20). For / > s log(e=>), we have
exp(-1) exp(-slog(e = =
which indicates that cxp(-t), Vt > s log(c-1) can be well approximated with an error within c by simply setting the output of the neural network be zero. Therefore, we only need consider the approximation error of a neural network for / E [0, slg((e)))
Step 2: E [0, log(e [)]]
Notice that for any L N4 such that N e, have
2-2 Se5 log(e-1)
log(e-1)
Then, it follows from Lemma 16 that there exists function ø implemented by ReLU DNN with width 48s2 (N + 1og, and depth (L+2)) log, (4L) + 2 such that for all E (o,slo)(c='))
(o(t)-exp(-B (45s+s" log" ((=)))44 2s 2- 1-2
2* S 5" log" (E (c 2s -- < 3" log (e ))""
2=====
Therefore, there exists function implemented by ReLU DNN with width 48s + 1) (8g2(8N) and depth (L + 2) logo (4L) + 2 such that for all E [0, oo),
(o(t) exp(-t) < x log (E (
51

---

Hence, there exists
6m E (Widdtth < (N + 1) logg(8N); depth < 118> 2) logg (4L) + 2)
such that for all & E [0, 00),
mm me| 88 <00'
Similar, by letting i = 21, there exist
Oea E ENW (width & 48s? (N + 1) 1og) < 18s + 2) logg (4L) + 2)
such that for all f [0),
(0) - -ex((())) lo ")e".
Define dt= = ) (2t). Clear, we have
Ogz ENN 48s²(N + 1) 1og)(8.);deettt < 18s 2) loge (4L) + 2)
and
(t)= exp(-22 2t) = (f) - exp(- & s* log" t)-
Proposition (Approximating m by (hm). For any k, E N+ with S s and 0 < <<> 1. Ler m, == exp(-t), Vt E o0). There exist N, L E N+ with N 2 S c, and
dm E (Wiwwihth S N loge (N); S 8 logg(L)
(92)
such that
p (t) m S 8² log (c (ee) for any 1 E [0,
(93)
and
s mm S 1 s log" )e $
(94)
Proof. Since m exp(- kkt). By Lemma
9 there exist
om E NNW(witth S N logo (N); depth & s² logg(L)
such that
3m (t) m, ||
S s log" for any t E [0)
Then, define
om (t) = ReL + ReI.U(-mm(()
(95)
and clearly, we have
om (Widdhh & 2 logz (N); depth & (L (L))
and by the fact that |x| ReLU(x)++++++ + ReLU(-x))
2||+2| @mmm (f) ml || +2| 5 s0 (e )e"" for any t E [0, oo].
(m)(1) m.
Therefore,
S cm (1) 8" log (c ) $1, for any E (0)
Proposition 2 (Approximating o; For any k. 8 E N4 with k S 8 and 0 < c < to < 1/2. Let
ot = VI exp( 21),, VI E [40, oo). There exist N, L E N4 with 2 2 and
oi E (Widtht S s²N (N) V 3² le
depth S 5² LVVooe- logo (LV Iog(e= 1)) V 5² log (E 1)),
such that
o (t)
o, 22 < kls log 20 (e) for any t [to. 00),
and
0 0 / (t) S to
52

---

Proof. When k. = 0, 1 1, which is trivial. In what follows, we focus on & 2 1.
of
Note that V1 expl 21) for all /. E [0, oo), we have
(-1) r! exp(-2rt)
ot - exp(-2tt)))
For each l = 1,...,,................... s and each r = 0, 0.l...., ....... k by Lemma 9 there exists
Dexp <NN(Wwitth z (3s) N (ogg((V)) depth X (3s)² L loga (L))
such that
(t) - -cxp(-2rt) S دند log (c )) for any E (0)
(96)
Define
1) Pexp(t). for any E (to 20).
8 (t)
r!
Clearly,
* NNW(witth S 8 N (Ng) S (L (L)
(97)
and
(t) -0²
Dexp (t) - -exp(-221)
ckls log 3/ "0(c 2c
(by Eq. 96 and Σ Σr=o S e)
2 h! 3s log (c -1 12
(98)
Note that for any 0 < to s 1/2.
o =;= 1=xlpp 2t) 2 1 - cxp(- 2 to. for any t E [to, 00),
(99)
which gives that
(+) 2 of kls log 36 (E
35 2 to kt log (E (c-1)e e 2 to.
(100)
(t) (t) S 0, + k!s log 38 (E 35 I 1 + kls 3x log 32 (E ) 30 S
2k
Recall that 0 < to By Lemma 23 there exists
@i E NNN (width S 3² (e-)( depth S s log
(101)
such that for any E [tb 1] C [e,,e e0= and x' E R.
1
1x' - x 22
(ze)
S c
(102)
x
For each l: 1, .,..., define
(1)== = RoLU Oro (0t: (1))) + ReLU (-oree Dree (o5 (o5-( for any t E [0, oo).
(103)
Recall the fact that |x|| = ReLU(x) + ReLU (==), we have 04 // (1) 2 0 for any E [0, oo) and
[0f)o(()) -a, 2k
|ree (g=((t)) 0, 2h
< Are _ (t)) -0 2h
S + de (1)
SEq 98
S c" + lel log (c (c-1)
2 30 log 30 (c )c" (c")
(104)
53

---

which gives that
0 (t)) S o
o, + k!s log (E- S tta
26
(105)
Moreover, Eq. (97 and (101) indicates that
0i/00 E WWWWwitth S s Nolo (N) (o
& s² Logg lo (106)
Proposition 3 (Approximating y"). Given k s N4, Nd with ||l| k I and < <<<
Let
y E (-2/ 1-22/2ax0 2as ()) 2/ 2as log( 1]
There exist N, L. E 14+ with N 2 < and
Spoty E NN(wwidth S 9(N + 1) + 1;depth S 7s(k - 1)L)
(107)
such that
(bot)(1)) - -y S ka 1/2 log 2/2 ")(N +1 1 for any y B.
TsL
(108)
and
(0po,(1)) & 12 1og (/2
(109)
Proof. By Proposition
9. there exists
poly
E NN S 9NN + 1) k 1;depth S 7a(k 1)
such that
(1) - y² S 3:30 20s log((-)) "(N 1)
I k/2 2/2 k/2 log 1/2 )N + 1)-,, for any E B.
which gives that
10poty (2)
E 10 122 4/2 log 4/2 122
(N +1)
S ka 2/2 4/2 log k/2 (2((( + ka k/2 h/2 2/ k/2 2/2( ( + 1)-7st
))
Tst
x ka k/2 log 1/2 (c ))
Proposition 4 (Approximating My 1)). Given & E N_ and h is defined in Eg. 57p
(1 (g.,)))) 1/s
h(y,t)
For any 0 < < 1, There exist N, E N4 with N N S and
0 E (Widthhh S 3d++2 N (N) V 3d+2 log (E 1) depth S L V s (eg2(e-!)
such that
(0,( - h(y,t)) S 20
28+2 log 2s+2 2 for any y E B, and E [to, oo),
and
0 db (y, S to as log(c
Proof. Step 1: Taylor expansion of h(
Recall from Eq. 663 that CV2.us
1 -E (x:) 2ut2 ( and i = U2, U2] E Nid. Similar to the derivation for Eq. 65 we
obtain
C s!m || 2allall
1/6
(yy (n [x()( 1/8
Σ
12 25 v luylay! y 201
(110)
1=1
54

---

Step 2: Approximating each base function. Notice that
| ReLU(=)) ReLU(-=),, for any R.
Therefore, we define abs to approximate |x]] by
abs (r) =
for any a E R,
and we have
Pabs
E WWWdWihdt = 2, depth = 1).
(111)
For each i E N3d such that || have < 12211 22 I 2s and 21/211111 ||22|1 $ 2s.
By Propositions I
3
there exist
6 2 E NW & ²² N log2 (ee) depth 2 (L
(112)
0i E .NWWwidth 2 8² N log, (N) 3 log (c-t);
depth 2 8² L log, (L) V 8² log ))
(113)
2 + poly
E NN (width - 9(N + 1) + 2s 1:depth S 7s(2s - 1)L)
(114)
such that
@llezal +2llessli (a (4) m, 12 11 +221+
2 2 log (E 2) for any E o0],
(115)
/0//02 (1) Jr 20
S م! log 35 (e ))" for any E (l. oc),
(116)
201 1e (z) y
2 +
S a² log (E (N + 1)--T for any y E B.
(117)
and
øllاz + (1) S
(118)
S 0/ to
(119)
Fix (m (
for any y E RRd, t E [fo, oo), define
opoly (y) = Cu us 22, (y),
Cug
poly
(120)
where Cur.us
1 n Ei= (z 2 220 Clearly, we have
Sooy E NNN (width 9NN + + depphh 7s(2s 1)L).
(121)
Recall from Assumption 4 that sup (00 S / 2as log(e-1)
pplly (y)
tus
< Cus.us |
222 (y) - yy 221
S a 22 log 1 N + 1) -7b
for any y E B.
(122)
which gives that
(oooly()) (2) S Us 201 t2
+ a 28 log 20 (c + (N) Tsl.
2|12 |1| +| a² log (E -1).
S as log(e-))
(123)
By Lemma 13 and Eq. 119 and [23 there exists
(1) multi E NNi(witth < 9(N + 1) + deptt < 7sL),
(124)
such that
d (0i) (t) Opony (v)) (1)))(1) Opily (y)Cue
" (e-"(e-') () + 177b,
(125)
55

---

Therefore, for any i E Nad such that || = | 11 + 122lla +
||sll1 S 8.
@mmi (09 (t), Opusy (3)) 0, y y2 Caai
s @ (0i/02 poly (z)) /2/02(t) Opuuty (3)
+ 1//02 ox 2s
Eq.
125
x
Eq.
TT6
125
+ 0
puly (3)
2m tu Cr0 y
Fq.
to a's log" (( + 1)- ToL + s/g 3x log (e (E-
l's' (c
+ t a² 2a (E + 1) 1)-
S a log (E ))= + slt a² 4 log (e- + a 22 20 log 20 (E-
(by Eq. (31
S a s/s log (E (E=l)e",
(126)
which gives that
Io (09 (1), ofory (y))
2 0, 28 y 2 t Cuu + a s1s log (e
3 to a log" (E- + a ss log (E-
2 10 a log" (c t)
(127)
Again, by Lemma
13 and Eq. (118) and 127 we have
multi E NWi(tth s 9(N + 1) + 1, depth S 7sL)
(128)
such that
o (dllessli +2lles li (t), o ( (B))
multa
& + (1) of mumi ( (1) ppoty
a0 log (E )) 1
(129)
Step 3: Construct the whole neural network.
Given dabs, alle (|ee)])+2((es) 09/002 opayy df ( (2) above, for all y E B, é [fo, 0o), we define oe
by
(1, = dars (0he) (Bllezl: | +2|loath (t), a (0 opoly (fo)(
(130)
For any N, S.L. N
(N WW+17 1) 75 = (N 1) 1-11)-411) -4. + 1) 3st. N 4x1.1 - NN 4s N -5 25
(131)
56

---

Then, we obtain
@ (3, t) - ml! +2112sl1 of y2 to
o ( ( (t), e (0 (t), (t), (pory(y))) (2)))
m 122 1l +2llusll of 2$ Exy yom C
(by dars (:) = 1.. || + (t) o (1) ( (t), Opary (3))
(2) @ multi
+ (t), o ommi (0i (t), Opoly (y))) - d
Eq
123
+
[ m
+lleall2+2 (t) ml! | +
multi
(0i/0z (t), palyy (y))
SBq. 115
Eq.
127
+ m
[xll 2/
@ matti
(0))0e(t),
Opuly (v))
222"" y2 + 0
<Eq 126
S a00008 log 'N + 1) 1)- 7at + log 22 (e )e2 w"x log"(e + a slos log (c-)""
S a0' lo"""log"( (c ()' eee az 80 (e ')e" + a s1s log (c (by to and Eq. (31
Sa's'a log de (c )6.
(132)
Therefore, for i E Nod such that ||ا|| and 0 < to S 1/2, for any y B, E [to, oc),
0 S (u, E m, 2 rell -lslll or 22 am
am tus Curi + a d log log (e-
S v0 log" (1-).
(133)
Notice that
(s/d+ + 1)--1 x
Σ
==
s ss+3d= 3d -
3d-1 1)
$ (s 1)30-1
(134)
||
which gives that
s!
0 <
lu !uy! (1,
2 |122|
|
2 (+- (--1lllll) s
12 11 +2|lesll1
m
a 5/0 40.4"10 logg (eg"(((-))e)
0
||
S at + 1)3d-1ssh s!s" log"
(135)
Additionally, by Lemma 17 there exists
SiTE E ENN (width < 48(2s)²(N + +) log, depth 18(22)) 2) log2(4L) + 2), (136)
and for any l: E N4 with A: < and for any a E 10, a (s + 1)3d sss= s1s" log (e]
| (x)) - 2///|| < (90s + +5 5) " (1 s!s" log (e 1)) $ N 4- 4s (by Lemma 17
< 95t 1/2 a 1/2 + ديي 3² log 1/2 (c 222
(by (s!) s (5*) = 38)
< to 1/2 1/2 1//12/1///// log 1/2 )
(137)
For any y E IRd, t [to, 20), define oj to approximate h as below
2- (- l:)
t (2, == 2root Σ
lilesey!! (no(v.)))
(138)
57

---

i) Network size. With the sizes of Dabs (lll==1l++2(1e)) 26 11/02, o poly D & (Eq. (111 to 1130.
121
124) and 128 ) for each i E Nad such that || = s,
Pi AWW(Witth x S² N (N) V S
s² (og2(L)) V s² log (c (139)
With Eq. 134),
138) and (139), we have
0% E NWW((with $ 2d+2 N logg (N) V ad+2 2 (c & s² Lo V s² leg
(140)
ii) Approximation error.
With |a-/k b2/b1 S &/2 max 12/kk-13aa 1,t/a///111111111111 b/. we obtain
(v. t) - Rl
6 roal
Σ
h 1h (2, -
Σ
2-
oh (v. 1/51
in ten t2n!
inseales!
| =
$ Eq. 137
Σ
2- 102l122
Mlegles!! 2 2h (v.
2/s
2) 11 in
slnn uo 11+11
y
2m teo Cresiess 1/8
|
|1:||1=8
60 1/2 a 3/2 2og' 1E2
+ 1- 20 + 1) 1, 8/1 20 log ( -))) 3-11
1
Σ
2 (- 1olll lulllans
m. ll
12lleng 11 20g 1ag C
||
S/ 1/2 1/2
a 1/2 log 16
1 +
(t a + 1) 3d-1 sls" log" (c ()=
Σ
(u luglltde! inggges!
(2) t)
m..22l Lu2 +alleal au tw2 C22219
||
<Eq 132
1/1 1/2
a 1/2 20 22 2
1 (t (s + 1) 3d a 20 s!s log" 28 (c (c-1)) (1)== :-1 (s + 1) 1)3d-1s! 1s. a s!s 1e
1//2 a 1/2 log 1/2 ) + (8+ 1) اييد 1-- a² 20+2 log 20+2 (e -e
$ 1 1/2 1/2 2 12 1 20 25 + a² s!g 2st2 log 20+2 (E (6- ))
(by (s + +1) 11) 2يي S 13dd-1), Vs E N+)
S 2a2ss 20+2 log log2** 2 (E - )
(141)
which implies that
0 5 (3 t)
2 2Λ $ |h| aa?sts a² log (E
to as log(E 1 a² sls 20+2 20 (E -))=
to as log(c ) for any y E B, E 0c].
(142)
Proposition 5 (Approximating ha(y. t)). Given & E N,.0<e < lo $ 1/2, set K = N Let
Ča is defined in Eq. (54) and hi be defined in Eq. 56 i.e.
hg = 3Cat as log(e
B E 10.1,...,KK K.....,- 111
K
Let tbj be defined in Eq. 138 Then, there exist N, N with N 21 < c, and
Vhg E -.NN(widrh S 3d+2N2 v 3d++2 log (E :): depth S L V s2 log (e (c-
58

---

such that for any y E B, E [to, 00).
The (y, h&g
BE { ,.,,,....,. K-1K -).
and
16, (v.t) 0r ( aslog(c- -
1-2
Proof.
For any y E B, E [tu. 20), we define
(1,1))
(y. = tas log((-1)
(143)
which indicates that S Φ5 (v, 1. For N by Proposition 12 DNN
there exists a ReLU
@stop E NWWiwit < 4N2 + 3; depth I 4L +5)
such that
@xeep (6x (2,, ki if oū (y, E
kC log(e
(k+1) logll 8. InSK-2), k= 0,1, K-1,
Kto
Kto
where å E (0, Calotas log(e 1)K))
Define
thp by
hhe (v, = Cata as log((=))
0 (y, t)
Ustep
for any E BB, E [lo, x0). (144)
K
Cata as log(e- log(t-1)
Clearly, by the size of 0y (c.f. Eq. (40h) and @step: we have
Phg (Wi(wit S 3d+2 22 V 3d+2 log (c) depth & L2 v v² sog?
(145)
and for any y E B and any E 20),
The (y. @Calo tas log((=1) hgg for E 10, 1,.., K - -}:
(146)
K
such that
as log(c-1)
as log(c=))
Ohe (v, 1) - (r, hhg (1(11))
S as log(e (oog(=_))N-- 1)N 2=
Klo
N4T.
Proposition 6 (Approximating (y, ()) ). Under the same settings of Proposition N, k S.S. s, there exist N, E N4 with N- 22 and
Given k E
si (widhhh $ 33d22 logg (N) V 3d+2 le depth S I,2 (,olgg(L) V 2 log (c))
such that
10 10h,(1s (y, - (2)( S a² (E (E 2= 2-2 for any y E B, E (0),
and
0 s of (y S t $ log" (e-
Proof. For 3 E 10. K 1] and k = 0.l...., 8, we define
Boato las log(e=)) 1) k
5 Chioaa log (c-1)
(147)
K
With K -,, we have t E [0. for any 8 E 10,1, K 1} and = 0, U......,
8. By
Proposition 13 there exists a ReLU DNN
point E ENN ENVNWwidtt (width S 16s(N2 + 1)
S 5(L2 + +2) log, (4I.2)
59

---

such that for any fixed & E {0.1.....s) we have
Ppoima (3) - 5.8 < SN for each 3 = -0,1, 1,1,,.., K
0 S Pooin (3)
I 1.
We define
KΦh (31 (21.))
of oh (y, = t0 * log" (e
Caty as (olo)(--))))
Vy E R. vt E (0), (148)
Clearly, by the size of th (c.f. Eq. 145 1) and oppin we have
hg CNN(width S NN logg
N) V 3d+2 ( depth & 1,2 log2 (T) V 2 ))
Then for all hg BCat "ast logle
B E [0, ...., K 13. we have
(y, 10, O2 * log (E-1)]
(149)
and for any 1: 0, 1,....,... and (0.1..... --
o (SCato as aassoogl loolll
BCatn'as as aosog(e"))) k
10% (y. hg)
K C llo" (c- )
K
Chink log" (c 15
C255 a log (6-1) (e))))/fooom
(poii (3)
$
to IN I
a ')N-200 22-2 L-20
(by N- < < (0)
Proposition 7 (Approximating exp(-hs(y, 1)), Under the settings of Proposition exist N, N+ with N S c, and
Then, there
2 (c -1); depth S T,2 (T.) V 8 llog 1) NNN(width S 3dd22 (N) V 3d+ log The
exp
such that
1
exp (g. 1exxeeepphhp...0 S N N- for any E B, E (l, 00);
and
0 の hg
(y: t) S 1.
Proof.
Similar to approximate Mtz for 3 E {0, 10..,...,KK
K 1). we define
C exp BButō as log(e-"))
(150)
K
Then we have &o E [0, 1] for any (0, 10.1...., K 1}. Again, by Proposition 13 there exists ReLU DNN
exp point
NN (witth S 16s(N62 + 11) log, (8N2); depth s 5( + 22) log, (4I.2)
N.N
such that
10 ppoot (3)
5 < - for each 0, 1....,K K 1,
< exp
(B) < 1.
60

---

We define
K Kong (2hn)(111)
exP
(y, t) =
exp point
for any E R t (t,,
(151)
hg
C loslog(e-1))
By the size of the (c.f. Eq. 145 and Pupin we have cop hg
E (Wi(whhth S s 3d+ N2 log)(N)) 8 3d+2 ) depth S L2 Fogog2 V leg"(E
Then for all hg = BCsto os log(e
B E {0, 1,.., K 1}, we have
K
p (ha) [0,
hg
and for each 3 E {0, 10, K 1]],
3Calo ars log(( 10g(e-1)) exp
B.alo 2 log(e
cesp (g. exp(-ha(y.tt)
point (3) eexp
48 I, 45
D.3 Neural network approximations for regularized empirical score functions
Recall that the regularized empirical score function at time is given by
(2TO?) 4/21
11 Ei= exp
|| m.m
(1)
-wx, m. (9)
pkde (ztt () == (e(( V Puat (2TO?) 4/21 Ei= exp
Vir(
20
of
| - m,
20
Pr.t
Li=1 exp
1lne-mer
(1
(x) m)x
20²
1 E
exp
|| (mmmmmmx))))) 20
Ve-In-1
where Po,t (2mO²) dd/2 s/n2--nn
D.3.1
Approximation of regularized empirical score functions in - -
Lemma 10 (L -pprrxxixiai of Regularized Empirical Score Functions). Given a set of sample J2 for any E TRd E [4o x.), let my == exp(-t), 0; == v1 exp( 21). Fix Pa,t = s E (2ma -I and 0 < to I 1/2, let N,
N+ such that N 2 < and 0 < E S to A n 1//5. Suppose Assumption 4 holds. Then, there exists a fumcction &xcore inplemented by ReLU DNN with width < O (56d433 N logg(N) v 5d43 log" (c-)) and depth S (1,3 log,( V 8 logg (c-1) such that
(ecooee () (adden((y (y, 1) S a 18s+ (12s) S 3d 54s+ 2 log 54++ (6 Vy E Rd V E (0),
and we have (0 (y. o, Mog n.
Corollary 4. Under the same conditions of Lemma 10 let a 22d log I to I 1/2. Fix k N. such that & 2 d/2. Then, there exists a ReLU DNN with width < O( log, n) and depth
< Ologg n) such that
@ (y. t) fide (y, t) S a 18h in -1 log 54k+2 n, Vy E Rd, Vt E 20),
and we have @econe (2, t)] So, Vlog
Proof. Apply Lemma 10 with 1/k k and N = [ L such that 2 <
5 to A - 11/ hold and we complete the proof.
61

---

D.3.2 Proof of Lemmal 10
For any y E IR t E [to, 2o), recall that the empirical score functions is given by
fldd (g. 1)
1
= exp(
mix
20
(9)
x
y
x
exp(
1y m.a 2
20
ii
7 Dimm exp
1/
Ve-in-1
2
1 m, x
(3)
kde ( t) - kdd (y,
Skde (y,t) 1) V e --
Scooee (y. t)
(152)
7
of
score y 1)
where we denote
Mide y (), 102 (v,1) = 1
3
m (1) 12
exp
(153)
207
1/ m, m2x11))
112
11 (y.t) t) =
exp
(1)
(154)
207
1=1
s (y, t) = 1( (y, t) VE - nn
(155)
n (y, me J (y. 1) y Ride (y. 1).
(156)
Similar to Eq. (88) and 49). decompose Fd B u B, where
B (y Rd yy S 3 2as
(157)
B (y E Rd ly 2as log(E - -1)].
(158)
We approximate fude on y E B, t E (to, oo) in Part and y B,t [to, o0) in Part II.
Part I: Approximating flede on B
For any y E B,t E [to, 00).0 0 S c exp( 1.), similar to the derivations for Eq. 511) that, by
Assumption 4 we have
y max
(2 2as log(e log(e-1))² 1)2
8ass log(c-1) 4as laslo 2 1,
20
20
2
which gives
m (5) yy
exp
<
4s
for all i= 1,...,n.
20
Noting that the function & exp( -x? /2) jr is monotonously decreasing in [1, oo), then
Σ
exp
(y-maz
11
(m,z) 3)
pkde (y, t) =
a7 n
2n7
E exp
1l 11y-maz(1))
V Ve-In-1
207
en
1ly
112
(m) - ya
cxp
20²
Tt
1=1
2en
V Vas log((e 4 4s
a:
(exp( -22 /2)22x is monotonously decreasing in [2 as log(e -1], oc))
< 2e as
(159)
Part II: Approximating fkde B
62

---

Step 1: Approximating / by Pscore
(1)
Notice that
RcLU(x)) ReLU(-x)), and |x| ReLU(x) + ReLU(-x).
Then, for any Y. E. we have
[2, = x + 1/ x y
1 RaLLU(x + v) ReLU(-xx y) ReLU(x y) RRLUU(-x+ + y))
which indicates that
max E NWW(witth = 4; depth =1),
(160)
and
Pmax (x, max( for any x. y E R.
By Lemma
8 there exists
Okde E (Wi(witth X (3s) (3s))d++3 log, (N) (ds) (c
depth x L3 logg (L) V log"(E
(161)
such that
1o (y.
kdo (3. 1) S a (12sss 3 54s+1 log 54g55 )e
(162)
and we have
0 S (y, 1) 11
(163)
For any y E IRd, t E [to, 20), we define
(1) (y.t) = Pmax ( 1n
(164)
By Eq. (160) and [61
we have
NN (width+ S 560+3 N3 (og)(N) V 6d44 (t=
Pscore
depth Z L logg(L) V log (E (c-1)),
(165)
and for any y E B,BL E (4, 00). have
1o (v.1) 1) - - (y. 0) S (od 1ull (v, 1) S a (125))) (dd log (6-)
(166)
which implies that for any y E B, [to, oc),
e'n o(1) (y, S 1 y 1) +a
log 5le ())e) S
(167)
Step 2: Approximating Merre by Sccore
Again, by Lemma g there exists
Auee E NN (width E s 6d+3 A loga (N) V od+3 log (c
depth s L logg( V 3² log" (c 1c-
(168)
such that
( t)
fle ( 1) S -h (6s)! (4+)4 log 278 (c 3c
(169)
and we have
0 < kde y t) 1
(170)
By Lemma
13
for each 1,..., there exists
d motii. E CNN < 9(N + 1) + 1, depth S 7sT.):
63

---

such that for any yj [0.3 2as / 2aslog((=_))) wh [0,
I mulli. (Us, hi) Uyh s Vaslog(e- 1)( + 1) 1)-7sh,
For all y E Rdd h E R, we define
mile (y. h) = =[ommli. (yI, . ( (ld, h)].
(171)
Then, have
(5) malti E NWW(witth & 9d((N + + + d, depth - 14sL),
(172)
and
1 (y, (y ()) yo (v, (y, x (og(e 1) 1)-14
(173)
which gives that
matti (y. A (3, S |y| lde y, t) + Vas + 1)-11 S V log((=
(174)
For any y E B,t E [to, 00), it follows that
I (v, 212 (v. f (y) < ( (v,, kde
(2)
12 (u, ()) you (2)
(v 1) +lyl
Rud
(y, 1)
Rdde y. L)
$ Eq. 173
2 Eq. 169
I as log(e= (N + 1) 14ab + a 904 1/2 (6s)l 3d+47+++3 log" 27s+1/2 -1)33
aa (0)++1/(6s)))) 3d427s 272 log
(T 2
(175)
Moreover, similar the derivations of Eq. 65
obtain
y m,
(3) (3, t) = Σ exp
x (6)
207
exp(-hig
Σ
(k- ( (-2) 2th 244 2n lesslaaa
-a) m leg I 12llugl1s
hgg 2 2
t2g
ax (1)
42+200
(1)
k= 1+al=
For each D N3d a4 E N such that |1:1| + a4 S 1, denote by
Cur 1 (0) 22t20s (1) E Rd.
(176)
Following a similar derivation for dhude (i.e., Eq. 86 and 9T) in Appendix D.2 there exists
we can obtain that
E <Wi(Widtt S .3 logz(N) 6d+3 log 1);
depth S L (ogg(L) log² (c))))
(177)
such that for any y E B. [to, 00),
(v. 1100 (y1 t)] S a (6s)) log 2Tx (c-))E3
(178)
and we have
1 (v, 1)1 S | (y, 1)) + a
2T )
I V log((-1) + 9 2d+2777+ log 27 (c (c 2
S Vaslog(e-))
(179)
By Lemma there exists
@m E WWW x S2 + 1) logp (N): (og)
depth L
(180)
64

---

such that for any t E [0, 00),
p0 (t) mel E 3"x log (E -1 e
(181)
and
0 p (t) 1
(182)
Similar to Eq. (172). there exists
E WWW(witth S 9d(( N + 1) + d, depth - 14sT.),
(183)
mulli
(2) such that for any om (1) [0, 1), Rade yy, E aas Vaslog( 1]]
as log(
| I em (1), kdl (y: ()) 6m (1) (3) y. 1) 2 v Vaslog(E-))(N + 1) 14st.
(184)
It follows that
@ Pmuln (m (t), (3) Pkde (y, t)) - m, mr kade (y. t)
10 (0m (1), kdle (y. 1)) 6m (1) (y,
mml(1) mm|
kile y ()
Fq.. [84
x Fq.
18T
Eq.
179
+ me Okde ( t)
kulc (y.t)
$ Eq. 178
as log(e - I) +
14st
3 (e
a log(e1(-
+ +a 95 (6s)) 3d 27 log 27 2og27 (c )e
S a 99 (6 log
27e c0
(185)
With Eq. 168). 172
18 and (83)
we define
scure (y)
of munii @m (t), Ride y. t))
Q mutti (v. (2) t)), for any y 1R E 10, oc). (186)
By the sizes of (2) (3)
kde &m @ p Pmihii have
ENN( NNN(width z 6643 N logz (N) V 6043 (c-
score
depth $ 1, (Lg)(L)) V 8² log² (c ()).
(187)
Moreover, for any E B,( 1 E 0o).
@bleur (y, 1) - /12) (y, 1)) @mmi (0m lide (y, t)) Q malii (y. Ado (y, t))
(2)
m, x (3) (g, t) + x Mid (y.
p (pm (f), 10 1 (y. t))
m, x (3) (y.t) t)
+ p (y, o Ke (y. t)) y x 1220 (z.t)
$ Eq. [83
$ Eq.
a 9* (Gs)))
log (
ee 30 + +a
a
( 2d427733333 log
96 9./1/2
1/ (E -
S a (090t//((ss)) 344477043// log 27 27041/2 -2)
(188)
which implics that (2)
sanoe (g, t) $ | (y, t)] + a (Iss
log 27++1/2 (c
S m. 1 (u, + ly
| (y, 1) + 90+1/2 2447743333 log log22211//7 27
aaslog(E-1)
(189)
Step 3: Construction of the neural network &score.
Recall from Eq. 167 that -n1 S Oseure (y, S 1 for any y B,t E [t0, oo) and < e S to A n 2/8, -2/8, By Lemma 23 there exists
(1)
@rse E <NW(Widtt x 5² log'' (e-)()-)))) )); depth S s²
(190)
65

---

such that for any n E (2) (v,1) S and x' E R,
(oroe (ploore (2/, t))
S 2* +
[ (y,t)
facore (3,,)
S (y,
x Eq.
T60
2 20 + 180 (12s)\s 3d 644+++++10
log 54% 1c
18 (12s)ls 3d+ +1 log
34s
(c e)2
(191)
which implies that
Órco ( (olldce (3/ t))
2
+ a 189 (12s)) (1 log 545 (E
(by Eq. (167)
f (y. t)
S a (12x)11 (l log 64% 1e S
(192)
By Lemma
13 and Eq. (189 and (922) there exists
multi E (width & 9(N + 1) + 1, depth & 14sL).
(193)
such that
(T) matti
(olooro (3. t), Pree (Poorc (y, t)) o. (3. t) x Proe (olobre (2, (v.
2 /n log(e (NV + 1) 14st 114st $
nnvaslog(e - 1) S Vaslogg(-
(194)
which indicates that
multi
@
score
(2)
(y, t), &ree (o (y, t))
(2) Score
(V, t), x drec (Dledic t))
0 scare
y
Prec ( y, t)) + tvsslog(c-1)
$ Eq.
[89
S Eq. 192
n log(e-1) + as log(e
2) Z Snvallog(e-))
(195)
By Proposition
2 there exists
Φ1/02 E NNN (Widthh z 3 N logz log2((2) 8 log"(c
depth z 3² L L log(E=1)) V s² log
(196)
such that
(1)0=(1) o, / x (2s)1s 6* log 00 (6-
for any & (l,, 00),
(197)
and
0 (1) 1.5
(198)
By Lemma
13
13and and Eq. 195
and 198
mudi E CN (width - 9( + + 1, depth S 14sL),
(8)
(199)
such that
① muuli (01 am @ y t), Proo (o (y, 1)))
- 02/02 (t) x & mani 1 score
(y, t), croo (o (y, b)))
S to n as log (c- (N 14ah < to n nn as log(e (e I V (oslog(c-))
(200)
With Eq.
165
187
190 193 196) and 1999, for any y Rdd E [to, oo), we define
Oxcore (y. 1) = u (01)02(1) aa (ile) (y, 1), Proc (ol0drc (2,, 11) 1)))
(201)
By the sizes of Oseore. Oscare: &ree. 01/00 mulls & Pmmlus we have
(1)
(2)
Éscore E .V S 6d+3 3 (ogg(N)) v 0d+3 (E-t( depth S 3 logg(() V s² log² (c-))
(202)
66

---

Step 4: Approximation error of Oscore.
For any y E B,t E [to, 00),
@score (y, 1) - Sscore (v, t)
malti 021/02 (1), Pmmnn ( (y, (), Proe (Dloor (v, 11))
<
- (1) x @mun (0 (eeieee (y,, t), dioo ( (3/ t))
(by Eq. (200)
+
(1)
( @ (y, 1), cree (olovme (y 1))
of
Eq
197
195
1
o mullti
(7)
Socore y, t). Oroe (ooc (.. t))
score (y, t) Osec ( (y. t))
0
Eq.
1
@ score (y, t)
f(3) (y, t)
rre (000 (2)
( 1))
Eq. 18S
Eq.
192
1
((2) (y. t) 0,
Preec (olodrc (y,+))
scare (2.1)
(1)
Eq.
191
S as loog(==))e= + (2s)1s log 69 (E
))²² ny as log(og(-))) to- Vas lostog(e-1)e tola 30
+ nly²t (Gs))
log log27 27 (E -)
5te + 1 as Vaslog(e-11) 18s (12s) I1 3d+5sst1 log 510 (e )0
18x+1/2 (12s)1 3d+54++3/2 50g 1/ (c
(203)
By Lemma
and Pn.t = (2TO,)) 4/2 In we have for any y E TRdd 1 E 00),
Vir(v)
Vpr(z)
v2 (2TO?)- d/2
ploee (y, /.,t]==]]
$ pc( V Pm.
or V
log
pe(3)) Pn.
Pn,t
22
logn S o, logn,
(204)
which gives that
Øscore Oscoro f) S pkile
(y,
+ a
18s+1/2 (12s))
2d455st333 3d454s+3/2
log 51s+1/2 (E Soz Vlogn.
Therefore, we have completed the proof of Lemma
67

---

D.3.3 Approximation of regularized empirical score functions for sub-Gaussian distributions
7 For any d,n N+.0 < to S 1/2, fix Px,t = (2x0,) ) - Let N, L, E C+, E R such that 2 to A n 1//,, Let my exp(-t),
Lemma 11. Suppose that P satisfies Assumption
v1- -eeeppp22 for any [to, Then, there exists a function &ecore implemented by a ReLU DNN with width I Os logg (NV V gd++ 3og+ log"(e
and depth
I 5O log2 V s² so1o (c=)) such that
(atti) y M p8n [kll Pc( V Pn,t
Vpr(y)
Eecore ( t) dy S a 74 ((12s)))) f64 ( log 108++1
(E - 1)e²0
and we have || (:, flo S 0f 'Vlogn. n.
Proof. For some A > 0, set A = +- AJ", where μ. = [x]_[[X]. With loss of generality, we assume that Ex-r% [X] = 0. Our results can be easily applied to Ex [X] # 0. By the tail bound of Po with parameter a,
Pr[X $ A] = / pr(x)dx & 2dexp(
(
A2
(-a=²
(205)
Let A O(V
a n), then
O(a²slogn))
Pr[X & Al S 2dexp(
S 2dexp(- (og 2dn 20())
2a²
By Lemma 5 and Pn,t = (2TOz) -d/2e we have for any y E IRd t E [to, 00),
(y)
122 S of
2 2 log
(2x00) 4/2
2 of
1
(log n + 1) x of
log n.
(206)
(v) V Pn.
Pn,
By Lemma 10 there exists a function
Pscore E <Wi(witth
S $ 6d+3 N logg (N) V s 0d+3 log (c
depth 2 L logg V s² log (c (c-1)
such that for any sup (x) S (000(=slog)
derone (y, 1) - pide (y, 0) S a (12())) 3dd54+++ log 34s+ & (E -2)6, Vy E Rd. Vi E (l., 00).
Thus, as we have shown in Eq. 204 for all y E Rd, E (0), we have la (y, S I1@scoro (y. t)ll S o, Vlog n, which gives that
| (v. 1)122 S o, 20 logn.
(207)
With Eq. 206 and 207
Vir(y) Oxcone t) dyp(x))d
A- Rd (ey) V Pn.t
< 2 A- (l) (1( Pn.t
(p)(t) 1 + +||0acone (M. t) p(z) dd
S n (()) (a, 2 llog + oi logn) S O(s) logn. n.
(208)
On the other hand, for x E A, we have
[1
Vpr(y)
awore (2, (v.tt) po(x))d
A Rd pr(y) V Pu,t
$ / d( a(a3To ((2s) ( 3
log log 54st (E- (e-))e²) p(z)dxx
A-
$ a 740 ((12)))² 6d+108s+3 log 108++1 (c 20.
(209)
68

---

Combine Eq.
208 and 209 we obtain
V (p)
Psooro (v: 1)) |/
B (xl1)) 12 -on RM Po(U) Pn.t
x a 745 ((12)))) 100+ log 1080+t (E-
(210)
Lemma 12 AA-ppoxxxmaaion of Regularized Empirical Score Functions for Sub- Gaussian Distri- butions). Suppose that P satisfies Assumption / For any d. E 4x, Pn. == (2ma²)) -n and 2/d < to - 1/2. Let my exp(-t),, el-exp(-X-00e
exp(-26) for any [/, oo). Fix E N with 2 d/2. Then, there exists a ReLU DNN &xcore with width S O( logz n) and depth < (log" n)) such that
E (2 1 P SAN
Vprly)
Oxcore (g.t) (3). (B.tt] dy S a log 1088+1 (n)m
74k
Pr( V Pm.t
and we have 11 (: flo S o. Vlogn.
Proof. Fix A N4 and k 2 d/2, apply Lemma with n1/k k and N [n1/(22)] such that N 2 - < E to 1/s hold and complete the proof.
D.4 Score approximation errors by deep ReLU neural networks
We are now able to prove the score approximation error bounds of deep RelU neural networks for Sab-Gaussian distributions by combining Theorem [ and Corollary
Theorem (Neural Network Score Approximation for Sub-Gaussian Distributions) Suppose that -2/d
[ For any 1 S d S Vlog ni n 3 and any log - < to 1 and T = n (1),, let {Tette(to.TT be the solutions of the process Eq. (2) with density function Rd 1.,. Fix k E N4 with d/2 S k S log logn.. Then, there exists a ReLU DNN &score with width < (n d log2 and depth S Olog" n) such that
Po satisfies Assumption
So, (0 + az)) log 4/2+3
E (aco yn pp"" ERI,P [1] log lox.)( dscone (Ir,, () 1)
and we have || (c, t) So,² Vlog n. Moreover, let 011), have
E p 188 1T Barp, [|[ log - (x), 11]] S at d/2 -1 log n.
t=to
Proof. By Lemma
with Pn,t (2TO?) - dd/2 In -!_ we have
Vpi
(ax( 2
E (110))2,,,00 EnixP,
(xta))
(x)e)) V Pritt
S o. 2---2 + a log (2ma²) -$
2/22/21
Pn
71
log d/2+3 n
S d j0 a
69

---

Therefore, together with Pscore from Lemma 12
we obtain
1 (x(0) [120))11110
ER | Po Pa (xx) - @score (z), t]]
Vpr(
Vp. (xte
< E ~P0
2 2k
Pr (3:4)
Pe(ea) V Patt
X Lemmnu]
Vpr(
+ E [m(1); pmon
2 Pi
dscond (Iee 1) 2
Pr. (2x)) V Pn.t
X Lemmna 12
log d/2-3 n
a 74k log
108K+ + (1)
S of 20 + aa
log 4/2-3
S 20 (ot + a²))
where the last inequality follows from the fact that its second term will be dominated by the first term. To see this, let's fix d/2 < & <
(1+2/d) log a log log n- logaa. then we have
74 log or+ 108 log nl2logg
74k log a + 108k log n S (1 + 2/d) log n + log log n 2 log a,
which gives
a 74k 108k+1 (n) S a 2n 1+ log" n
log
log d/2+3
2 0,
a
(by a -2/d 1//1logn S lo S of
n
log 4/2+3
a, d-2 ( + a
Notice that we also need
d (1 2/dd log + log log 2log
2
74 a 108 log log
which implies that S Vlogn. Assuming that ensure our results hold, we require d/2 x
universal constant, have verified that to
1o log <
log log n for S Vlogn.
log log
Moreover, following the same proof for Theorem
obtain
E [[(1))2
1 EuIr P.
|| log Axcor (x,, t)11 dt
log d/233
$
0,
d-
(0, + az)
dl S 4/2 -1 -1n--llog ++4 n.
1=1
Thus, we complete the proof.
70

---

D.5 Auxiliary approximation lemmas
Proposition (Approximate Square Function on (a,b)). b)). For any N.L N4 and b e with
< b, there exists finction & implemented by ReLU DNN with width 3N + 1 and depth L such that
[o(x) - x21 = for any (a, 6].
Proof. For any æ E [a, 6). let
which implies that I E [o, 1]. By Lemma 18 there exists function & implemented by a ReLU DNN with width 3N and depth such that
2- b-a"
[Ø(z) - 22 SN-
which gives that
(6 0) 22 a -ax α x² S (b a)² N-2
a
For any x E R, define
x(x) = (b - a)² (
+ 2a ReLURR + lal) - a² -2a /|
b
Φ x-
b a
b
Φ(x)
X |a|
ReLU(x |a))
Figure I: An illustration of the network architecture implementing for approximating x² on [a, b].
By 6 (width I 3N; depth S L) from Lemma 18 the network in Proposition & has width 3N + 1 and depth < L + 2. It follows that 6 can be implemented by a ReLU DNN with width 3N + 1 and depth L, since the two hidden layers have the identify function as their activation functions.
Since + |a| 2 0 for any x E [a, b). which indicates that
a x(z) = (b - a)²0 x =) + 2ax a², for any x E [a, B).
Therefore,
1ő(x) - 231 (b N) for any x E [a,
Lemma 13 (Approximate xy on [a,, bı] (2),
b2)). For any N, L E N4 and 01, a2 b1, by E R with a < by and a2 < b2, there exists function & implemented by ReLU DNN with width 9N 1 and depth L such that
10(x, 2) - xyl < 6(bi a1)(b2 - - for any x bi),y E [a2, b2].
Proof. For any bily y Jaz, by], let i 9ー61 9-02 which implies that ă, ÿ [0. 1]. b, b,
by b2-a2'
By Lemma 19 there exists function e implemented by ReLU DNN with width 9N and depth L such that
(ق 1) ij 6.
which gives that
(b1 a1 (b2 -2)/ - 01 02
ry + 011 + a2 - 0103 < 6(b - aj a1)(b2 N-)
by a1 b2 02
71

---

For any x E R, define
((x.)) = =b1
02)R x- 01 y
02
+ 01 + 02y a102-
(211)
01 01 b2
d2
By construction, @ can be implemented by a ReLU DNN with width 3N + 1 and depth L. Therefore, for all x E [a1, bi], E b2),
(6(x,)) - ryl S 6(b, a1)( - a2)
Lemma 14 (Approximate Monomial Function on [-R, 18) For any N, L, k. E N4, with 8 2 k >
2 and R E R,,, there exists function implemented by ReLU DNN with width 9(N 1) k: and depth [sL 1) such that
[0(x) 22
F&l
30(k 1) R* (N 1) for any [x1,
[- R. R)
Proof. By Lemma 13 there exists function Omahi implemented by ReLU DNN with width 9(N + 1) + 1 and depth 7sL such that
1mmuta (2, ryl 6(b1 a1)(12 .2 1)-7sL for any [a1, b1,,y E [ar, b2].
For all E RR R]. define Φ1 [-R, -2 + RR? RR which can be implemented by ReLU
DNN with width 9(N + 1) and depth 7sL. such that
Φ (211 x2 < ((2 + 177- Tal. 30R + +) TaL.
(212)
Next, we construct sequence of functions di [-R,, R2 + | R for E (1, 11 by induction such that
Φ can be implemented by a ReLU DNN with width 9(N + 1) + i and depth 7sLi for each {1, 2, ........
(ii) For any i 2,, ... 1} and 21; #2 ... T1-1 E [- R, RJ. it holds that
(2, I;- 22 ... 1i+1l < 30iR1+1(A + -TLL
(213)
First, consider the case i 1, it is obvious that the two required conditions are true:
9(N + + 9(N 1) + 1 and 7sLi 7s =;
iii)
Eq. 212 implies Eq. 213 for i 1.
Now assume o; has been defined; we then define
Φi+i (II,..., T1+2) = @mati (s): (2),....... (1441),4442) for any IT, , E RR
Note that Φ; E (widtht < 9(N + 1) + i; depth < Li and @mati E NNN(witth < 9(N + 1) depth 7k:L). Then di+1 can be implemented via a ReLU DNN with width
max 1) 1,9(N + 1 9(N 1) (i+1) and depth 7sLi + 7sL 7s 1). By the hypothesis of induction, we have |d:( 24+1 - 11 ... x4+1 S 30 (N + 1) 70t.
Recall the fact that
301R 11 (N + 1) 1)-11a. Tat. < 30kR 2-122-7. 78 30k 2-7 < 0.25
72

---

for any N, L.k. N+ with s 2 k and i E (1,2..... 1}. By 21 114 [-R#+1 g R
it follows that
01( 1-1.25/t* 1.25
for any 2141 1-R, R).
Therefore, by Eq. 211 and 213 we have
(11, Tit2) 21 ... lit2|
= (o) I ReRRRCUU 22 ... Fi+2
< 101 (41,.....,,.....
- 02 (Z1....... 2i+1 |1:22 + ( 2,1,,,44444 21 2i+2
I <6 x 2. x 2R x (N 1) -T ToL 30iR (N + 1) 1 |2142|
(by Lemma B3
30R (N + 1) -7-2 + 30 (N 1)
30(i + 1)R* (N +
-7st. 1))
for any I1, I2 .... Ti+2 E R-R, R, R). which means we finish the process of induction. Now let = - by the principle of induction, we have
for any ,1,-..; J.k E [- R, R). So the desired function implemented by ReLU DNN with width 9(N + 1) L - 1).
|ó(1)..... ... mk) T132 .x - 30(k: (N++)
and depth
Proposition 9 (Approximate Multivariate Polynomials on [-R, Rl Assume P(x) x² att x/2 ... for E Nd with ||ll < k E N_. For any N, I., N4 with 2 k and R E R+, there exists a function & implemented by a ReLU DNN with width 9(N 1) and depth 7s(k 1)L such that
[0(x) P(x)] S 30kR (N 1)-7T for any RR
Proof. The case to = is trivial, so we assume that k 2 2. Set k = ||||| S k, denote by [11, P2i,.......... va] and let [.1,..., zz] T e R be the vector such that
j-1
21 = Ejj
1/3
Vi for j = 1....,d. d.
i=1
That is,
Mi
Va
[21 22 ... 2q.1 F1,
T1. F2 22..... 22
Fd.
Fdl ERK
Then, we have P(x)= x² 21 ZR.
We construct the target ReLU DNN in two steps:
Step 1: There exists an affine linear map Olin : Rdd + RR that duplicates æ to form a new vector
[21, 22...., 2g. I....,................. 1] R i.c.,
Oin (x) = [21, ......, 26 1, 1,..., 1]_ R
Step 2: By Lemma 14 there exists a function Omon R&* - IR implemented by ReLU DNN with
width 9(N + 1) +
and depth 7sL(k- 1) such that
Jomnn (x) - x01 30(k 5 - 1) (N
For any x E [R RJ4, define
((x)) = Omon (shrin (2))
(214)
Clearly, we have
E (Width - 99NN + 1) + 1: dedtth S 7s(k 1)L)
73

---

and
10(x) - P(æ) = (o(x) - x²1
Amool (2)) -1 22 x.
Vd
Amon (11, 22,,.........................2 ....,........................... 2/, ,,...,1) 21 ...22 S 30(k 1)lt (N + +1) 1)-77h
(by Lemma 14
30Kl2 (N + +1)
for any 21, 22, Rd E [- Rj.
Proposition 10 (Approximate mie-Dimensional Step Function on (a, b)). For any N, L,d E N+, a E with a < b, and E (0, 33X2 with K = [N L there exists a me-dimensional function o implemented by ReLU DNN with width 4 | + 3 and depth 4I + 5 such that
Φ(x) = k, if a k(b-a) , +
(b+1)(b-a) - 8. ( for k = 00 0,...... K -1.
K
Proof. Let = b-a for all E (a+ k(b-a)
- 0, 1,.... K K-1 1, KK at (++1)(b-a) - 8. 11155kk 21 }. k=
which gives
b-a (egk-22/ 21):
z E
for k 0, 1- 12 there exists one-dimensional function & implemented by ReLUU DNN 3 and depth 4L + 5 such that
Then, by Proposition
with width [
((t)= = k for all ā E
A+1 8 b-a
1 1 for k = 0, o......, K -
K
Let o(x) = ( Then, can be implemented by ReLU DNN with the same sizes as a ReLU DNN for implementing á and we have
((x) = k, i E a
k(b-a) K
a a
(k+1)(b-o) 8- 1(scx -1) for k = 0, 0.1...., 1.
K
Definition 3 (Modulus of Continuity (38)). For any a, b R with b > The modulus of continuity of continuous function C(a,, bjd) is defined as
(r(r) == sup (lf(x) f(y)) || - y12 Sr r, E [a, b)4), for any 2 0.
Definition 4 (Trifling Region 38 Given any K E N+ and 8 E (0, k), define trifling region 2(10, 1]4, K,8) of [0, 1]d
K-1
2((0, K,
U 12 ...
Tal (.., [0, :
U
6,
}
k
K
In particular, 2(10, 1]d, K, = if K 1.
The following Lemma extends the approximation results of 38 Lemma 3.3] from [0, 1] to [0, R), VR 0:
Lemma 15 (Approximate Trifling Regions on [0, R). Given any ε > 0, K, R E N4, and 8 (0, 3k assume f C((1,, R)) and g R + IR is a general function with
1g(x) - f (x) < E, for any æ E [0, R] \( [], K, iee, g(x) E B(f())). any [0, R S2 RJ, K,ō). Then,
(215)
[x(x) f()) < و + ( for any E [0, R).
where
= 8), g(x) g(x + 6)), for any R.
74

---

Proof. Divide [0, R] into KR small intervals denoted by Za = I K
| for k 0, ......KKK KR 1.
For each & E (0,1,-.. KR- 1}. we further divide I& into four small closed intervals as
I = UI,, U UIKAA:
where Ix,1 x + +
3x.3
|
28, K -I, = 1 *1
Following a similar proof in 38 Lemma 3.3], we have for all Ix 2 In,
mid(((( (z= ().g(z). g(x + 6)) =
By 00, R| 1R-- (u)= which implies that
mid(g(2 (2),, + 6)) E ( for all R].
Therefore, we have
[0(x) [())] w for any [0, R1.
While the technique of approximating the univariate exponential function exp(-z) using ReLU neural networks has been explored in recent works, e.g., [22] 43 we develop new proof strategy tailored to our setting, which enables separate control of the approximation rates in terms of the network's width and depth.
Lemma 16 (Approximation of exp(-x) on [0, RD. For any N, E N+ and R > such that 2 < R-!, there exists function 6 implemented by ReLU DNN with width 48s² (N 1) log, (8N) and depth 18s (L + 2) (40) + 2 such that
22 (Ø(x) exp(p22) < (45s R + N) L-2s for any [0, R).
Proof. Set K 22 and 8 E (0, 1)). Let (20 K, 8) partition [0, R] into K intervals Ia for 3 E (0.1..... K 1}. For each we define xg = 3R and
Σ3 = {- E ::: E
OR
(8+1) 5 [00K
K K
x-1) and vg is vertex of Ia with Clearly, we have [0, R = 2[ R,, U
minimum || || -norm.
Step 1: Approximation on non-trifling regions: x E [0, R] \ 2( R), K,5).
Approximate Tg.
By Proposition 12
there exists ReLU DNN
Ssuep E (widhh I 4N + 3:peph S 4L + 5)
such that
@step (2/ R ki i E
8. 1 for k = 0, K - 1. TRR (k+1)R K
K
Define ReLU DNN Sste by
Tenep (x)
@stp (x/ R), for any x E [0, R).
Obviously, we have
Suwp E CNW(width I 4N + 3; depth < 4L + 5)
(216)
and
@xcp (x) = 3R
K rp, for 3 E {0, 1, K 1}.
By Taylor expansion of exp(-xx 2)) at 2 up to order 1, we have:
exp( x3)
(x xx)* (-1)* exp(-e)
exp(-x)
(x x
k!
s!
km
=:
=:
75

---

for some real number 0 that is between t and 2g. For all [0, R] (10 R.,, K, 6), the magnitude of F2 can be bounded by
P21 (- 1) exp( (x m3)" (s x8)
R K
1 (217)
s!"
Therefore, only need to construct ReLU DNN to approximate Fi.
Approximate exp(-xx)) For /3 E {0, ...., K 1}.
define
BR
Ea exp
K
K - 1}. With K N by Proposition 13 there Then we have &g E 10 for all | E {0, (0.,...., exists a ReLU DNN
depth Ppoint (WidWhitth 16s([N log, (8N) depth S 55(L+ + log (4L)
such that
1ppoin (3) - -Eal 2 for B 0, 1, 1-
S Opoint (3) S 1.
For any x: E R, define
Spoina (x) = Spoint
R
Obviously, we also have
@point ENN (width < 16s(NN + 1) 18g2 depth S 5(L+ +) 2) (4L)).
(218)
Then for all 2g 3 E (0,1,..... K - 13. we have 0 S Spaine (x)) S and
an K
(ppoin( (xz) exp(-xr)) Spoint (
(8R) (
exp( exp(1 (-
Paiil (3) E,3
N -2a
(219)
Approximate (xr - (x) R Let 2 = r xg, then ā E [0, R S [0. 1]. By Proposition a ReLU DNN
there exists
Opoly E E.N 9NN 1) s 1, depth s 7s(s 1)L)
(220)
such that for all 0 < kSs,
(ppol - =+1 < 9s(NN + 1) TaL 77L
(221)
Note that 0 < ã < R/K, which gives that i* [0, 1]. For
s.
9s(N + 1)- 7sL. & 9s2 7s < 0.1.
With Eq. (21 we have for all x E [0, R| \ 22 R2, K, 8),
Opoty (.) Opoty (r x3)) [-0.1,1.1]
Approximate exp(-z) for [0, R 2( R), K. 8). By Lemma 13 there exists a ReLU DNN
Ómulti WNN((witth < 9(N 1) + depth 2s(L + 1))
(222)
such that for any I [a1, b1]. y E [az, b2].
@mmul (I, 3) zyl < 6(bb (b2 a2)(N + 1) 11-22(++1))
76

---

For all E [0, oc), we construct neural network of the form:
@oxp(a) = Omuliii (ppoin| (extep( (x)),
hd Oplyy (o (sse((x))
(223)
By Eq. (216). (218). (220) and 222). it is easy verify that Sexp can be implemented by ReL.U DNN with width
max 4N + +3, 16s(NN + 1) 1og2(8N), s. (9(N + + 1), + 1) +)+1} & 16s + log,
and depth
4L +++5+5(4+2)ln) (4L) + 752 + +3
2 +(L+)))+(44 2) log2 (4L) + 75 +) 25 ( 2L+
s 2)) (0g2(4)) (4T).
Therefore, have
Texp E WWi(witth v 16s2(N (N + 1) log, (8N); depth & 18s 1o02((1))
(224)
Fix (0,1,..., K --13, for all æ E Iz we have
@exp(1) (x) exp(-xx)
exp( x3.)
Pmulti @pprinl @Sstep (z))
poly (x 0ste((z))
(x xz)) + +F2
k!
Q multi @ppaiin (Sxepp
poty (x (xee((x)))
@paine (Pxup(()) x
k! Polyy (z paeep (x))
k!
k=
F1.1.1
@poine (Ssnep(( (I))
Opoly (x @step())) (2)) exp((=xx)
Paly (x (ste((2))
=:F1,
(-1)*
+ exp( xg))
k! Ppoly (x )ssep(t))) - exp( exp(-xx)) x3
(o - (xx)))
+ R N 22
k!
s!
0=
k=0
=:F,,3
For Bounding F1,1- For all x E 00, R 2((0, R1, K, 8), we have Spoine (0xep (2:) [0, and
0.1
1.1
<
Opoly 20) $
Sex
k!
k!
k!
k=0
k=0
Therefore,
Fi,1 S 6s(3 (-0.3) (N + 1) 20((+1) < 20s(N + -22((+1)
(225)
Bounding F1.2-
F1, <
&point (dxsep():) exp(-xx)
Σ
Pplyy (ar Tsep()))
<3N 22 I,
(226)
kd
k=0
<Eq.
219
77

---

Bounding F1.3-
1)
Fi,3 S (exp(-rp))
Ppoty (x 0stop(x))
(x xz)
k!
k!
k=0
k=0
Opoly, (x )sup(())) (xr xzg) )
k!
<qq
1
9s(N + 1)-
7oh Ies(N 1) -77k 25s(NN + +1) 77b 1)-75b
(227)
k!
k=0
Combine Eq. 217 and 225
to 227
we obtain
1
10exp (x) exp( -t) S 20s(? N 1) 2e(L+1) 3N-22N 25 + 1) 1 + R² N- 1.
s!
s 20s(N + 10- (N + 1)- (3 R$ N
Note that for any N, L, E N+.
22 20L S N-26 ي 2- (N + 1) 1)- 7oL < (N + 1)-22(+++1 < (N + 1
which gives that
[dexp(x) exp( exp(-x>]] S (458 R +3) N-23)-221)
(228)
Step 2: Approximation the whole regions: [0, R)).
For all I. define dexp by:
@exp (x) = mid mid (oexp (x - 8), (exp(()), Pexp + 6)).
Then, by Lemma 15 we have for all 8 E (0,
R
3K
(x) exp( (-p- S (45s + R + +) NNN---- 25 L-2a + Wexp(=z)(0) for any x: E [0, R).
where Wexp(- (6) defined as
Wexp(-a) (6) sup { <xp((=>) exp(-x)) |1 5.
such that Choose 8 E (0, 3K
R
Wexpl (0) < S N22
which gives that for all R).
(T) exp(-x) < (45s + R²
NN-+NN-- 2 = (45s + R² + 4) 2-20 220
To determine the size of the network for implementing Pnxp, note that from Eq. (224 we have
Tesp (--)) 8). desp (.), (-5)) E (Widtht < (N+ S 18s² (L+2) loga (4L)).
Then, have
0exp E ACW 1; width < (N < 18s2( log, (4I) #output = 3). (229)
Recall that mid(3s.-) E (wi(wh < depth < 2) by Lemma 22 Therefore, Dexp mi)(:.')) @exp can be implemented by a ReLU DNN with width
max(48s* + (8.. 14) 48s (N + (og,((8]
and depth
18s² (L 2) log (4L) + 2.
78

---

Lemma 17 (Approximate k-th Root Function on [0, R), For any N. L. E N+ and R > 0, there exists function implemented by ReLU DNN with width 48s² (N + 1) (8g)(8N) and depth 18 2) log (4L) + 2 such that
l0(x) z²/h S (45s + 5) RR/ 2* 2s for any x E [0, R).
Proof. Let á = z/R R E [0, 1), we have x//k = R// ///h for any x E R].
Step I: Decompose the domain.
Set K T,2 and 8 10,1/K). Let 1), K,,) partition [0, into K intervals Ig for [0, 1, 10....... }]. For each 3, we define Fg = k and
I3 {a :R:E
(++1)R
8. (185K22)
K
Clearly, we have [0.] ((0.). K, minimum || nno-m..
and is the vertex of Ig with
Step Taylor expansion of ///k
By Taylor expansion of ///k at & up to order 1, we have:
1/k
1/k
al/K
2) +-
(1 2)
=1F2
for some real number 0 that is between é and Tz. Step 3: Approximation error and the size of the network
Following a similar proof for Lemma 16 we can obtain that for any s, k E N4. there exists function implemented by a ReLU DNN with width 48s² (N + 1) 1ogg(8N) and depth 18 + 2) log, (TL) such that
((T)
/1/K| < (45s + 5)N-2= 2=
for any I EE [0, 1].
For any x E [0, R). define @ by
<(x) Rl/k0(x/R)
Clearly, we have
& @WNN S (N + (8g);8N); depth S (1. + 2) logg (4I.)
such that
1Ø(x) 21/k| Rl/k 0(z/ RR) - (x/R) 1/k
$ (45 5) R NN 2* 25, for any x E [0, R..
79

---

D.6 Existing approximation results
Lemma 18 (Approximate Square Function on [0, 1| (3)). For any N, L E N4, there exists a function implemented by a ReLU DNN with width 3N and depth L such that
( S N= for any E [0. 1].
Lemma 19 (Approximate zy on [0, 38 For any N, L E Mt, there exists function & imple- mented by ReL.U DNN with width 9N and depth L such that
10(2.1) v) ryl I 6N-L, for any z, y. [0.
Lemma 20 (Approximate ay (a, b) (38) For any N, L. E N+ and a, R with a < there exists a function & implemented by a ReLU DNN with width 9N + and depth I. such that
10 v) - -yy S 6(b - a)2 for any 2, y E [a, b].
Lemma 21 (Approximate Monomial Function on [0, 1] (38)), For any N, L, k E N4 with k 2
there exists function 6 implemented by ReLU DNN with width 9(N + 1) k - and depth [kL(k 1) such that
(x(x) 21 ... rxl S 9(k - 1)(N + +) TKL for any a [21, =1 22, xx E 1].
Proposition 11 (Approximate Multivariate Polynomials on [0, 1]4 (38). Assume P(xx) x² Fi x²²x x..'''''' for v E N' with ||ا|| S. k E N For any N, L E N4. there exists a fffuction implemented by ReLU DNN with width 9(N + 1) + k- - 1 and depth 7k2 such that
(0(x) P(x)) S (N 1) 7kL for any E [0,
Our goal is construct step function V mapping æ E Q3 Fg k for any B E 10. 1]d. We only need approximate mi-dimensional step functions, because in the multidimensional case, can simply set (ax) [*)),,, ((rr),..., (b(a))] where vi is one-dimensional step function.
....
Proposition 12 (Approximate One-Dimensional Step Function on [0,1] (8). For any N L, d N. and 8 E (0, skel with K by a ReLU DNN with width \/d + 3 and depth 4L + 5 such that
there exists a one-dimensional function & implemented
k+1 - 8. ( for 1: = 0, 0001..... K-1
δ(x) kk k. if E
K
K
Proposition 13 (Point Fitting on [0 (3)). Given any N, N,L, d E N+ and & E [0 f.r]for: for i there exists function & implemented by ReLU DNN with width 16s(N
1......, "......22'" 22 -
1) lo (8N) and depth 5(L + 2) log)(4L) such that
(1) (i) & S N 1, for 0.1..... N2T2 1;
(2) 0 0 ((z) < 1 for any x R.
Lemma 22 (Approximate Middle Value Function [38 The middle value function mid(æ1, T2, TT)) can be implemented by ReLU DNN with width 14 and depth 2.
Lemma 23 (Aprroximating the Reciprocal Function 22 For any 0 < E < 1, there exists a function é implemented by a ReLU DNN
① NN (width & );
depth log2(c-1)
such that
((x') - <ct 1x'- -x| xl
1
for all E -c and x' E R.
80

---

E Neural Network Score Estimation and Distribution Estimation
E.1 Score estimation errors by deep neural networks
Theorem 2 (Neural Network Score Estimation for Sub-Gaussian Distributions) Assume that the conditions of Theorem / hold. For 3 S d S vlogn, fix k N4 with log n V d/2 S k z
() log(to") Eq. (9) Then, for any E (0, 1), with probability at least 6, the excess risk of an empirical risk minimizer Eq. (7) over the neural network class NN satisfies that
logn log log n.
I Ex, (I)( V log (X,) dt S to d/2
2 log log 8 n to'n'"nn
log +4 to
Proof. For notation simplicity, throughout the following, we write
NN = (Wi(wwitth 5O(n zie loga n): depth I O(log" n)) (lo(:,))lo & of VVlogg].
and
(vy = V Vlogp(b)), for E R t E [b,77].
Recall that
T { Ex.ixollo((A) - V log (X, Xo) d
(o) =
Ex, t)x,,)
X, m,
2
dt.
of
2
For all Ø 12 x |to, 77 + Rdd have
Ex, (olll1(X, (), 0² (X., (x,, Ex. Xo)] - Ex. 100", (o)l-
(230)
Given of i.idd samples S is defined as an empirical risk minimizer Eq. i...;
(x(1) Ji=1 where x(+) Po. the denoising score matching estimate
e E arg min cENN n
Σ (0)x
Given another set of i.i.d. samples 5 = (x(0) p where ((()
Po. we write the regularized as
empirical score functions associated with and Pu, = (2TTo?) (aro??-d//a
VDr(y)
t(y,t)= = pe(v) V Pn.e
(231)
For any & E NN, denote by
Ex, (() - -(0", .X)]
-))0
e((o", ((0))
71
E Ex. Eg (6 E(O, Xo)I]
Bs ((( x (0) e(ô, (1)]
where the samples x x (n) are drawn independently from the same distribution as X, and independent of S. We aim provide high-probability upper bound
Ex, [((,, Xo) l(0", Xo)]
(232)
for the empirical risk minimizer Z.
81

---

Recall from Lemma that
1 5
(2TT?)
d/2
(to. (10x (o. BD S (100:)))
oo
log
S 0r 2 logn, n,
(233)
o0
Pn,t
which suggests that the random variable Egle(Ø, Xo) Xo) C(Ō Xu)] can be bounded. Therefore, instead
of directly upper bounding Eq. (232), we first use Bernstein's 's inequality to upper bound the following excess risk:
Ex. Esle(d, Xo) E(o, Xo)l Ex.
Esle(s, Xo) l(0, Xo)]
for a fixed o NN. Then, we provide a high probability bound for (( - f(0", ,.). We conduct the following steps for our purpose:
Step 1: Bernstein's large deviation bound for [E(((().)) E(Ø, )).
We first verify Bernstein's 's condition for the excess loss class
= (Es(!(o.)) :
(234)
By Lemma 25 for any NN,, it holds that
(Es(((o, Xo) {(Ø, x "]
Ex.
< Ex, [s Xo) - ((0, xo))² '1]
(by Jensen's inequality)
Ex,,x. [| -xxx, x11
x 1 logn Exo Es
(by Lemma 25
= =o logn Es I' Ex. [1o((x.))) (1, t)]1 dt
S logn a. (f/0 Ex, [|x( - 0² (X.: 111 de + Es L' Ex, [1x((x..)) 1) - 0² (X,,,, 11/3
230
$ Theorem
S 1/6 log (Exx!((d) ((0), Xo)) Ex 11( Xo)l) + a d/2 n log +4
= to logn Ex. [( Xo) - l(00 Xo)]] + a 4/2- d/2-11 n log +5
(235)
First, we show that for all ái E N', the random variable Eg! Xo) (()) Xo). is bounded. By Lemma 25 for any ó E EN and any x E Rd, we have
[Eg [( x) x)),
Es (c(0, xx) x)))
(by Jensen's 's inequality)
2 to logn Eg
Ex, [1o(X, [[
1) dt
T Ex,|x |1 0) + | X 1) de
2 1/ logn Eg
S t log
2t (by || 1) 1 5 m)
to
$ 101 log" n elo
5 10" log
2n. n.
82

---

Applying the Bernstein's ' inequality (i.e., Theorem 5 with Eq. (235), we obtain that, for any fixed NN and any å (0, 1), with probability at least á holds that
Ex. Egle(d, Xp) C(O, Xo) Ex Eg[((d. Xo) E(0. XooI||
24 Ex, (Bgle(d) Xo) l(0, Xo))) 21 log(2/6) + to log n log log(2/6)
(by Theorem 5)
n
log n (Exo [((0, Xo) 1(00, Xo)l 1011 4/2 1 log 2 +4 n) log
+ to log n l2g(2/6)
(236)
n
Step 2: s-net argument.
Here, we use the standard s-net argument to derive uniform large deviation bound based on Eq. (36).
Fix parameter 0 to be specified later, and define
NN 1/V = (0y : ::ss S
be the minimal -net of N/ w.r.. the - -norm 1dd x x T', where where N+/VT = N of NN, ||- /T' (loo) is the cover number NN.
By the union bound and Eq. (236). we obtain that for any 8 E (0, 1), with probability at least 1 - 5, it holds that
Ex. [Ell(od Xo) C(0. Xo)]] Ex. [El[(dd Xo) E(Ø, Xo>il|
00 logn: (Ex, [ Xo) l(Ø- f(or,xx)) Xo)] + t d/2 nn log $ n) (2g,2,, /
log n log(2N, 18)
(237)
n
simultaneously for all Φ0 ENN 7.
Moreover, by Lemma 25 we have for any NN, there exists &e NN T/VT such that
(((6)
l(0) x))² S 1 logn
xx,=x
(oat))))xxxx.. (X(x,,)) [1x
dt (by Lemma 25
log n
72
dt
(by 0 ENN 7/VF)
T
7 logn
(238)
to
which implies that
Tog n to
10(6, x)- - e(0",
S TV
(239)
83

---

For any ø E NN, let 6² be its closest element in NN T/VT. Then, for any 8 E (0, 1), with
probability least 1 6, we have
E Ex.
Es (s) Xo) - C(0, xo)]] Ex, [Es[((o Xo) [(Ø, Xo)]]
E Ex
[() Xo) - E(Φ xo)] - Ex. [(,,, Xo) - 1(00, [(6x.x)]
+ Ex, Eg [0 Xo) -E(Ø, Xo)|| Ex. [s . - E(ô. xoo]]
log
+ Ex. Eg (( Xo) E(o, Xo)|| Ex. Eg| Xo) E(0. Xa)]
71 to
(by Eq. 233
logn- (Ex. [e(do, E(00, Xo)] + a 4/2 n log $ n)) log(2N, T/ 4/2
log
to
nto
log n log(2N /OV
+
(by Eq. (23
nlo
logn + Ex. [( Xo) e(0", XX)] + a4/7//2 n log +1 n) log(2N (2N T/ T/v7/6) logn. (rV 0
nto
logn
logn log(2N (
+ +TV
(240)
nto
For the first term of Eq. (24 we have
logn (rV to logn Ex, [e Xo) ((0b,Xo)) + a0 d/2 n- log +4 n) 1og(2N, T/T//)
nto
logn n log(2N, /VT//8)
(v
log
Exo [e(d, Xo) f(0" Xo)] + v 2
nlo
to
log n log(2N, 1
log n tu
logn log(2N 18) V /att d/2 nn - log $ n
nlo +1
néo
logn 1xx Xo)
ElO,, (o) log(2N, /
+
nto
log n log(2N, (V/2///VT/))
log
1 aa
1
n log $ +4 n
nlo
t0
logn: Ex, (( (()
l(t, Xx)
log(2N ( 15)
(241)
nta
Substituting Eq. 241 into Eq. (40 we obtain with probability least 1 6, it holds that [E (0, Xo) E(Ø, Xo)]] Ex,
E
[E[ Xo) e(0,, Xo)]]
log n
logn Ex, [l( Xo) E(p Xo) 1og(2N,,
log n log(2N (6)
60
nlo
níoo
+ d, d/2 0
log +4 n.
(242)
Step 3: High probability bound for ((3.-) - !(o".-)
Recall that 2 minimizes the empirical risk over the class NN, i.e.,
n
E arg min Ex, !l(,, Xo)] = arg min *ENN
Σ
l(d, x: (0).
@EANN"
84

---

Let o E NNV be the neural network from Theorem and constructed from ii.d. samples $ =
{t(1)) |= such that
= Es [ Ex, [1 X,, xx 1
Es
Ex. [o)
- -((0" Xo)]] [{)],,,)]]]]
to
2 a d/2 a n log n.
Then, with probability at least
holds that
Ex.
[s[[(e, Xo) - E(0, Xo)]]
< xx,
[f Xo) - E(², Xo)]]
(by the definition of 3
logn n Ex, (((o.X)) f(0 Xx)] 1og 2N T
2 Ex. [Bg[(( Xo) - E(0. Xo)l] +
nto
logn
log n log(2N, /6)
4+ n + a 10 d/2 - log
+ T to
(by Eq. (422)
nto
Es
Ex [e(o, Xo) f(0" [x)] -xx, Eg (6,0 Xo) l(00 Xo)]]
log n Exo [e(Ø, Xo) l(oi" Xo)] 1og(2N 2N 7
nto log logn. 1og(2N,//TT!)
+ taa1 d1/2 -1 log
+
nlo
Eg [/"
Eg Ex, [e(oxXo) Xo) l(0" Xo)]]
Ex, (1;)) -0² (1)
t0
20
log n Ex. [E(,, Xo) f(0", Xo)] log(2N, -
nto logn 1og(2N,,vTT/)
logn
+ T to
+ a tad dt22 an--log=* $log**1
++1
(by Eq. 230
nto
logn Exo [e(ō) Xo) - E(0", Xo) (
< Eg Ex, [e(,, Xo) l(bb xoo]] +1
nto
logn
log n log(2N, log(2Nr/v7/0)
+ 1/2//21 log n.
+ +
nlo
For the first term and the second term of the above inequality, recall from Theorem that
[
Eg Ex. 1(0, Xo) ((0"xXo))
Es Ex, (x, o" o'(X,,,) (by Eq. 233
to
S 4 d/2 n log + n.
(243)
Therefore, with probability at least holds that
logn Log(2N,/vT/H) 15) att d/2 4/2n-1 'log" 4+4
log
Ex [8 [ Xo) ((ô, Xo)]]
+ TT
nto
to
logn Log(22~//T/H))
4 d/2 n log $ n
nlo logn 1og(2N,/vTFF) 1
log n tg (244)
+ 4
d/2 -1 log 4 +1 n + VV 4/22
ntg
85

---

Noticing that
Ex. [[(o Xo) , Xo)
Ex. [((ō, Xo) - EE [Es!((, X + Es(((0 Xo)] l(0", Xo) Ex. [E]( Xo) E(o, Xo)I] Ex Eg (e( Xo)l - f(0",
(245)
For the first term of the above inequality, according to Eq. 242 with probability at least holds that
Ex, Eg (0 Xo) E(Φ, Xo)]]
logn: Ex, (e(ō, Xa)
{()" log(2N 1og(2N,/v7/0)
2 E x,
1& 1,, Xo) - E( Xo)]] +V
nto
log
logn log(2N 2N /
+ 4//4/2/
4/2 logg +
to
nlo
logn- Ex, [(Ø, Xo) - f(0,, Xo) log(2N, /
log tu
nto
log 1og(2N
+ a dd/
log +4 n.
(246)
nto
For the second term of Eq.
245 recall from Theorem 4 that
Ex, [g] X - e(0".
Xo)] = E3
Ex. [1 -0 (x,,)))))
(by Eq. (230)
S a 0
d/2
log n.
(247)
Combining Eq. (25) to 247 we obtain that with probability at least 5, holds that
log Ex, [e(,, Xo) l( Xo)] log(2N, //T///
log n
Exo (e(ox Xo) ((0,"oon $
+ Ty
nıoo
logn log(2N, + a a 1/22 n log ++1
nto
Recall the fact that the inequality x S 2a + implies that S 4a² 26 for non-negative a, and 28 Therefore, with probability at least 8, it holds that
Togn n Ex, Xo) Xo)] I ST
logn log((N, t0 log 4+g n. (248)
to
nto
Step 4: Covering number evaluation for N /
It is shown in Theorem 6 and 8 of [52) that the Pseudo-dimension of ReLU networks has two types of upper bounds: O(W L log W') and O(WU), where W, L, and are numbers parameters, layers, and neurons, respectively. If we let N denote the maximum width of the network, then W = O(N2L) and = O(N implying that
WLlog w ONN2 L Llog(N2L) = (oggN)))(( log(NL))
WU ONN L- NL) O(N
Recall that
WW= {0 e N OOnn a log, n); depth < Ologg n)) [|o(;)) So, Vloog..
86

---

With N & O( log2 n), L - (log² n)), the pseudo-dimension of N. satisfies that
Pdim(NW) _ O(N L
- Ont log" n log n log(n *े log" n))
- (na a log" n).
(249)
Futhermore, by Theorem 6
53
Theorem 12.2l), the covering number N, //VT satisfies that
Pdim(NN))
VT VTlogn logn ))
1og(N,, x log Σ
to
Vllo
Pdim(NVN) log
Tea
S n² log" n log(
(by = (()))
$ nd log" n. (logn + log
1
+ log 1 )
(by ja2n 2/d log to 1/2)
vto
S n log' n (logn + log +)
(250)
Substituting Eq. 250 into Eq. (48), we obtain
Ex. [((ō. Xo)
log
(log(N + log 2 ) + +
log
+ ta
log ++4 n
nto
to
log ntg
nn log" n. (logn log ) + log
2 5)
log
+ a at 4/22
log $ + n.
+ +TV
to
Step 5: Determining r and k
Choosing n we obtain
Ex.
Xo) - 1(00 Xo)l S 1n 'log n + 2n log n log + a logg n.
2 4/2
Noticing that when 2 3, (d log(t we have ty I s t 4/2 which ensures that the last
log
term will dominate the first term in the above inequality. Moreover, recall that ensure Theorem 1 require d/2 < k S
log log log
for d S nlgn.
Therefore, we obtain that for 3 < d S Vlogn, n, fix h E N+ with d/2 V
6log n (d 2)- log(t 1)
<
k
log 72
log log
with probability at least - 5, it holds that
Ex, [e(ox Xo) E(0" Xo)] S t0 d/2
"oog + n + to n loo n log
2
87

---

E.2 Distribution estimation by deep neural networks
Theorem @Distribution Estimation Error of Ptn) ) Assume that the conditions of Theorem hold. Then, for any 6 E (0, 1). with probability at least 1 - 8. it holds that
E (m [TV( PT:)] S to 4/4n 4//4/ 1/2 log 2 n to to
1/2 -12 log lo n. Vlo((2/6).
Proof. The distribution estimation error in the TV distance at time to into the following two terms: the score estimation error, and the Gaussian induced error.
[ [V P| s ( [TV(Pt, Poo) (z))) [V((Pt. P ).
1) Bounding the score estimation error. By Pinsker's inequality (c.f. Lemma 31 TV distance is upper
bounded by the KL divergence, i.c., TV(Pt, Po.) V (KL.(ll)PPt)))) Furthermore, by Girsanov's theorem (c.f. Theorem 7 and Corollary
we have
E (m))) [TV(Pto:PP..)
2 fariiy (T.(Ptto)) Pro)
E
(by Pinsker's inequality and Jensen's 's inequality)
//
$ V Bia00)
Empor [| logg:: Pscore (rt,, t)1
(by Corollary 6
t=to
$ V aatt d/2 n log 4 +4 + 1o0n n- log log 8
(by Theorem
2/2 4/1, log $ +2
to 1/2 log 1/2 log(2/8).
2) Bounding the Gaussian induced error. The error from the last term is induced by starting from the standard Gaussian "Aa instead of the marginal distribution Pr. The convergence of the OU process 55 22 gives that
TV(P Pi S
KL S ST-T
(251)
Given that n 01), this term decays exponentially to zero. Therefore, the overall bound is dominated by the first term, which completes the proof.
88

---

E.3
Useful lemmas for estimation
Lemma 24 ((56, Proposition 2.2)). Suppose that X sub-exponential random variable with parameters v, b. that is
[Xexp(A(X B[X]))]] S exp(v2X2/2)
for all X such that dkl S 1/b.
Then, for any t 2 0, it holds that
Pr(x P exp ( (
Remark 5 (156) Eamples 2.5)). The chi-squared random variable with d degrees of freedom is sub-exponential with parameters (2vd, d, 4). This yields that, i x2(d), then
Pr[X 2 d+ S exp (1((2n)) for all t 2 0.
Lemma 25. Let Ø Rd x too T Rd and ø' R [to: T - Rd be any Borel functions such that |lo(; (((,,))le (R+))
(R*) o, Vlogn n and
S oo Vlonn.
Then, for all a E Rd, it holds that
(e(6) l((', x))) S to dlogn I' T
Ex, ())
- x'(X,, 1)1[2 dt.
Proof.
((0,x)) - l( x.)
T
Ex,1x==0 [(x,,,)+ +
X,
m4 2
['(X,,,, +
X,
m,x
22
o;
o;
2.
Ex
+ +2(0((x,,))
X: - -m4
|X t)22 2 ('(
X1,,
X. m2a
df
1o
oi
T
Ex,1X==0 (x(x),1))(((x,,)))) (x(x,,))
+ (x,,,1) (x(x,)))((xx)1))))))
2(X, m2 )
+
of
Applying the Cauchy-Schwarz inequuual we obtain that
(((0,x) x) - E(0' x))
Ex [0((,,t) 6'(,, 6 ((x,, + x)
o'X,,) + 2(X, - m2x)
<
dt
of
< / Ex.1Xx=0 [10(x..1) - o'(,, 012 V.""" Ex,1x==0 || 0b (Xi,t) (X(,ttt) + 2(X, m,,)
2
o²
2
Noticing that for the OU process, we have X, Xu + J.Z, where Z NO Ia). Then, we have
2(X; - mxx) of
|01 = 4E X ||
1 Z
4
Ex, Xo=
E
of
of
Since Z (Z,,.., Za) and Z, N(0, 1), Vi 1, ,,..,,,, have Z7 (1) and E 1. Thus,
d
E
z
E[Z
d,
which gives that
2(
m4 Xa)
2
4d
E Ex.
o7
2
oo
89

---

Then it holds that
2(X: mxx)
Ex xo== + ('
dl
of
2
I
2(X; m4 Xo} 12 )
(xx1xo==olld(x3 1)11 (X t)13
Ex, Xo=
dt
oi
/ (60, 26o,2logn 12do; 3d
(by || (
0 Vlo
T
1 dl
6(d lod a + 2d)
V 1, 1 expl 21)
(e - 1)(cto +1)
3(dlogn + 2d) log
(eT 1)(c'0 -1) 1)
$ 3od + 2d) log eto +1
et -1)
$ V(dlogn n 2d) 2d) 1
el - S V to d log n
Thus, we have
(((0,z) x))
$ 1' Ex,1xo=R (1( -x, ()xxx,)))]a. [ [ ||0oa||||Xxx000_ +
T
2(X, m,,a)
dt
o² 2
$ 2 dlogn I' ( o' d'(x, x11 dt.
E.3.1 Bernstein's inequality
Theorem 5 (Bernstein's 's inequality for bounded distributions 40) ) Let XI,..., Xn be independent
random variables such that [X,l S K for all E [n]]. Then, for every 2 0, we have
2 Pr
f2/2
Σ (x, - B(X.)) 2 S 2exp
Li=1 E| + Kt/3
In other words, with probability at least 1 - 8, it holds that
# -xx,-E(X)) i=1
1 2K log(2/5)
D
E E (2/6)
3n
E.3.2 Pseudo-dimension and covering number
Definition 5 Eiseudo-Dimensin [52 Let F be class of functions from X to R. The pseudo-dimension of F, written Pdimn(F,, is the largest integer m. for which there exists (#1,... ........ Ims "1....,, E xm x Rmm such that for any (bi....,,b, bm). E 13 there exists f E F such that
Vi: f(zz)
Theorem (Covering Number Evaluation by Deaud--Dimensoon [53 Theorem 12.2]). Let F be of real functions from domain X C Rd to the bounded interval [0, B). Let and suppose that the pseudo-ding dimension of F is Pdim. Then.
Pdim
B
which is less than LedBB/PddimPeaiee for d 2 Pdim.
90

---

Convergence of SGMs
Girsanov's theorem
Given a filtered probability space (S, (Fi)ezo, Q) and a Q-Brownian motion (EtoTT under the probability measure Q. Suppose that (ee,)) is an adapted process w.rll filtration (Fr)eel0.1) generated by B such that the following Novikov's condition holds:
Theorem (Girsanov's Theorem [44
(1 / [Ib(Br,)) (Bdd))
]
EQ exp
2
< 00,
(252)
Consider the process:
4= 1. (.B.s)dB.
Then, is square-integrable Q-martingale. Moreover, define the Doléans-Dade exponential:
E(C), = exp (cr ( ((c, = exp (L. (B.., s)dB,
1
1
2 -2 [b(B,, +13dds)
(0$(ST),
and suppose that EQ (E()( = then E(C) is Q-martingale and the process
= I b ((B..s)d (0
is P-Brownian molion under the new measure P = E(
The following theorem, Corollary 5 provides an upper bound on the score estimation error in terms of the score matching loss, which restates the results from [16]. In analysis, we apply Girsanov's Theorem (Theorem to continuous SDE processes, whereas [16] utilizes discretized SDE processes.
Corollary (Girsanov's Theorem for SDE Processes [66). Let Po be any probability distribution, and let (Xt) X' (X))tejo,7 be two different processes satisfying
dx, = f(X,, t)dt + g())dBe
(0$LST) T),
X ~ Po:
(253)
ax: = f'(,,, D)dt + g())BB (0&LST),
X ~ Po-
(254)
Denote the distributions of X. and X by Pr, Pt and the path measures of X, X' by P, IP,, respectively. Suppose that the following Novikov's 's condition holds:
exp 2 [1 f'(x, (1Bddrd)
]
Ep
<00.
(255)
Then, the Rudon-Nikoddm'' derivative of w.r.t. IP'
dPp dlp (X) (X)= cxxp exp (1," g(t) - ((x,
1 1/ g2(t)
[f(x,))) ((x,,)))
2
and therefore we have that
dP
& <KL(PIF)) Ep log dPw
I / 52(1)) [f(x,)) (1 (12p))(()))))
Proof. Let X be the process of Eq. (25 we define
b(B,,t) = g(t)
(X())))) - f(Xe.t)), 5
91

---

Then, we have
- T
Ep
[Ib(B,,)) dt
(t) Ex, [X,, f(x, t) de co. (by Eq. 255
Define
Jo bBB. s)dB, and E(C), in Theorem then we have is P-martingale
By Theorem 7 we have that under Ip = E()),P, there exists Brownian motion (B))y
B / s)s Be ! //)00000 (f(X,) - s))ds,
g(0)
which is F'-martingale and we have
dB; = dB. b(B,, ((B, dB, g(t)
(256)
Recall that under P. we have
dX, ((Xx,t)+ St Xo~ ~ Po
Then, by Eq. (256). we have
dX,- J(X,, (). + g()dB,
= g(t)dB + (',,, f(X,, t)dt
(by Eq. (256)
f'X,, f)dt g(t)
under the measure P...
dP
KL(P|PP)
EE>
log
1E>
(lg(E(())
(by p' = EE()))P
s)B + 2 2/ /' (b((2,,) s))ds
E2>
Lr + {" [b(B,, 12dds
E=
Ep
| Bg s)
(Ep[Cr) = Lo = 0 by L P-matttning))))
T
g²(t) ExIoR [ll((Xe,t) - f'X,, dt. 2
E.2 Girsanov's theorem for SGMs
Corollary 6 (Girsanov's Theorem for SGMs). Let Pt. law((Y Pr law P be the law of the random variables time T - to for the two processes Eg. (3) and 16 respectively, i.e.:
dY, (Y, + 2V log + 22dB, (0S1ST-6))
Yo Pr,
di. (Pe + 23g (Y,, -)) + v2dB (0st5T- - to),
Yo ~ Pr.
Then, we have
kL (P.)) - /" / V logp((x)) (23
gd
92

---

Proof. By Corollary 5 we have
(L
f
T T-to
/
1
2V logPr-((1)) - 200 T -v
2
2
(T)
/
so(2e, so(2e, Vllgp((z))
(by It = vr--)
93

---

G Controlling Early Stopping Induced Errors
G.1 Sobolev class of density
The following assumption, adopted from 129 characterizes the Sobolev class via the Fourier transform of Unlike the usual definition (which restricts orders integer values), this allows each V; to take values not only as integers but also as positive real numbers.
Definition 6. For 8, K E IR.,, the Sobolev class of density is defined
(T( = {s: RR RRIR 0, 1= =1 with ||]| 8, WWPPT( S (2x)aR²)
Lemma 26 ([29) E.I.) Under Assumptions ladd i E [0,2), atニ and Pto = Po Pay where Po, is the density of Gaussian distribution in di d-dimension, (o,,tIa and denote the convolution operator, then there exists constant C that depends pos L and dimension d such that
TV(po,Pta) S Cpolylog(n)n मनक.
G.2 Besov class of density
To define the Besov space, we introduce the modulus of smoothness.
Definition 7 (Modulus of Smoothness). For function f (92) for some q E (0), the r-th modulus of smoothness of f is defined by sup |
Wrg(fft)
1lAllaSe
where
AT(f(=)) - Ei== (1-1)- + jh) Ex + rh E St),
(otherwise).
Based on the modulus of smoothness, the Besov space defined as in the following definition.
Definition 8 (Besov space (B* 4. (2))
49 22
For O < q, & oc,
> O. r=: = [*] + 1, let the
seminorm |B= be q.4"
(Io W(
(q' < x)), (q' = x).
fl and
supizot wn
The norm of the Besov space B Bd (I) can be defined
IIflIB = 11.fll Ifl
and we have Ba Bg (92 = (ff L*(S)) |Ifll < co
Note that q is also allowed. In that setting, the Besov space no longer Banach space but a quasi-Banan Banach space. If > d/q. B 44 (I2) is continuously embedded in the set of the continuous functions. Otherwise, the elements in the space are longer continuous.
Considering the Besov space, many well-known function classes, such as Hölder space and Sobolev space be discussed unified. The relationship between Besov, Hölder, and Sobolev space are well known [57:
For s E N, B,,, (22) - (12) - 3,, (Ω)).
. ( = Wg(
For (2, = o0 (?).
Theorem 8 (Marchaud inequality (88). Let (R(d) with 1 1595 oc. For any integer r 2 1 and 0 < k < there exists a constant C C(r, k. d, q) depending only on r, k. d, such that for all > 0.
W Wk.g(f,t) S Ctk f.e Wr.([f, u) x++1
du
94

---

Lemma 27. Let E [10, 11() n Bqqqr (10, ) be probability density with 0 < S and 1 f < 00,0 < d' I xc. Let Sa = (2 d/2 exp( 20
Then, there exists a constant
C C(s,d.q.0 such that for all a > 0,
||p - p * Polli S CoolllB
and hence it holds that
(VP..PxN(O,O'...) 11p p polloll1 S Coo1plB:
Proof. We proceed in the following four steps:
Step 1: Pointwise bound for the modulus from the Besov seminorm.
Recall from Definition 8 that the definition of the Besov seminorm is given by
(155 (t "n (p. t)) de 4 suPiso 1 un, (p,1)
(q' < o0),
pl n-
xx=xo).
For 0 q < oo, has the elementary estimate (averaging dyadic annulus):
sup k w (p. & (log 2) (.," 2tt 1gk$2t
(k dky
1/4
1/4 and for k [t. 20] we get (rg(p,, k) < Ck 1kl In partieular, we can take k to obtain the
I C''pl ClplBB "
pointwise bound
Wr.(pp,) t) S Ct ".0"
(257)
For q' co, the same conclusion follows directly from the definition in Definition g Step Relate diffferene to the modulus smoothness w1,1 I). By the definition of convolution,
(p * po)(())) = -fe p(x - -y)p
we have
p(x) (p x (x)) [ (p(xz) - p(x - ()))o((y)d)))
Take L' -norm and apply Fubini and Minkowski's inequality, we obtain
llp p* poll2 Soan (p(x)) p(a y)) yo (1) dy da
$ { Rd Jo.1e (p(-) - p(. y) ldxy (z)
Change variables = az and write f0 for the standard Gaussian density, we have
| 1 ) oodd (1(y)d2.
(y)dy = (2TTa²) 4/2 exp(
20
Then,
p* polli f [p(() p(- -(---2)) -0 (z)
(258)
Thus the problem reduces to bounding the 7.1 differrne Ilx(-) - ppp---- - h)]l for small shifts h oz in terms of the modulus of smoothness. By the definition of modulus of smoothness (Definition 8)
95

---

we have
1P - p polll
Rd [p(()) p(- - 02))1 (z
|| (
- L wn (
(259)
Step 3: Bounding (p, () by Wryg (p, t).
Recall from Definition 7 that the r-th modulus of smoothness is defined as
Wp, 1 (p.t) == sup 1Appl11,
||
where [s] + 1.
Case 1: S
For < 8 - 1 we use r = 1 and the first-order difference is given by (pp(-) p(- + h) p(-)
Since = [0, 1] and 1 2 & oo, by Hölder's inequality, for any shift h, we have || hell | |1 ||
which implies that
w1,1 t) - (p,
(260)
Further, by Eq. 257p. we obtain
w1.1pp t) S Ct'lplB;
(261)
Case 2: < 8 S 2.
For < 2 we use r 2 and the second-orde difference is given by
(p.(() = p(- + 2h) 2p( + h)) + p(.),
and the second-order modulus is
W2 = sup
1 ||a|
When >> 1, we have direct control only of the second-order modulus (2, t). We need a relation bounding the first-order modulus (P, t) by w2 Apply Theorem
we obtain
f. w2 du m²
11, (1, S
(p.t Ct
Further by Eq. 257 and 1 < we have
W1 (p. t) S Ct / n-- p
C t" pl
du
0.0"
8
Plugging into Eq. (260), we get
W1,1 (p. t) < Ct 1t
(262)
Combining Cases and 2, we obtain that for any 0 < 5
W1,(p, t) < Ct pp B
(263)
96

---

Step 4: Bounding the TV distance.
Plugging Eq. (263) into Eq. 259 obtain
1p - p poll1 - f w1,1 (11,
s Clplm ( (z)d2
Clpla E [
S ClppB B
E [Z12
(by 0 < s S 2)
where Z ~ NO, Ia) and we have |E|] 2 d. Therefore, for the law P with density p. we obtain
N(0, 1 / ppx)) (p * po) =1pp-p*pol C'dpplB o².
Theorem 9. Let s S 2 and 1 9 00,0 d' S cc. Consider probability density
E ("( nU ([0, 1]{,, C) where U': denotes the ball of radius C. Let {Ethe)t
be the solutions of the process Eq. 2 Setting to = n ais., we have that
TV(Xo, Xt²) S n at2a
Proof. From Lemma 27 have for all Tto > 0.
TV(Xo, Xi.) S o Ipl:
Since S Vlo. substitute tn = = 2t20 gives
TV(Xo, x1o) 1/22 11 242.
97

---

H
Other Lemmas and definitions
H.1 Fourier analysis
Lemma 28 (Fourier Transform and Inverse Fourier Transform). The Fourier transform of continu- ous function f E L. (R) is defined as:
(((())) 22 / f(x)e- -we dx
The inverse transform defined as:
f(x) = V2T 1 F(w)e dw, Vx
Lemma 29 (Fourier Transform of Derivative). Suppose f R is an absolutely continuous differentiable function, and both I and iss derivative I' are integrable. Then the Fourier transform of I' is given by
fw)).
More generally, the Fourier transformation of the k-th derivative s(k) is given by
(w0(w) = =FT d ( = (
Lemma 30 (Plancherel's Identity). For a square-integrable function (x) E I?2(R), Plancherel's identity is given by
|[ = [((x) = [[ww)
where ((w) is the Fourier transform of f(x).
H.2 Distribution inequalities
Definition For distributions P.Q E P(K.), and their probability density functions p, IRd + IR
The total variation (TV) distance is defined as
ACE (pAA ((A)) ( (p(x)) ((r))
TV(P,Q) sup
(264)
The Kultback-Leible dive)-iivereence is defined as
KL(PIIQ) = / p(x) log
p(x) g(x) dx.
(265)
The Hellinger distance is defined as
HP(P,Q)= = / gd
(VPlo) - vg(x)) 2
dx.
(266)
Lemma 31 (Pinsker's 's Inequality [11 same probability space, have
For any two probability distributions P and Q defined on the
TV(P.Q) & V (L.(())))
Lemma 32. For any two probability distributions P. and Q defined on the same probability space, have
(2(P S KL(PIIQ).
98

---

H.3 Taylor's theorem
Theorem 10 (Taylor's Theorem). Let k E 14, be an integer and let the function f be times differentiable at the point E R. Then there exists function hy IR + R such that
f(z) Σ ( 1=
a)¹ (x-aa
(x a)
=Ra(2)
and limz-sa ha (a) O, which is called the Peano form of the remainder.
Lemma 33 (Lagrange Forms of the Reminder). Let f IR + R be k + times differentiable on the open interval with f(k) continuous on the closed interval between a and x. Then
((x+2)(ET..) (k: 1)!
Re(x))
(xc a)&t1
for some real number &r. between a and x.
H.4 Nonparametric classes
Definition 10 (Hölder Space). times differentiable funciioo
559 For
R+ Zz and Q C. Rd the Hölder space is set of [s]
1 f(x) 00 f(y)
(*( = f 8:Q- R:
::
Σ 1000 o1
Σ
sup aveReyy
S 00
x y/
a:l|ol <=
a:||a - [x]
where 8 ges with a = (a),...,aa) E N
99


Reviews:
[{"review_id": "RMZDfTdDZ5", "reviewer": "Reviewer_oUBr", "summary": "This paper investigates the approximation and generalization abilities of score-based (ReLU deep) neural network generative models (SGMs). The existing related theory work has two main limitations: some assumptions regarding the data remain quite restrictive, and kernel-based estimators are considered instead of neural network-based ones. To tackle the above issues, this paper provides a universal framework to characterize the approximation and generalization abilities of SGMs for sub-Gaussian distributions. Further, it also applies to the case where the target density function lies in Sobolev or Besov classes, with an appropriately early stopping strategy, demonstrating that neural network-based SGMs can attain nearly minimax convergence rates up to logarithmic factors.", "questions": "1. For the score estimation via empirical Bayes smoothing, what is the technical challenge to remove the Lipschitz score assumption compared with [30]?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. As a dense theoretical work, this paper is overall well-written with clear notations. The motivation is clear that the existing theory work has quite restrictive assumptions on the data, and neural network-based estimators are less considered. \n2. The results are promising and seem right. The proof sketches make sense, although I have not carefully checked the detailed proofs in the appendix since the proofs are rather long. Compared with related work, this work removes several restrictive assumptions. More specifically, compared with prior work [30], this work removes the assumption of the Lipschitz continuity of the score function under the sub-Gaussian distribution. In comparison with previous work [22, 26, 27], this paper derives the approximation and estimation error bound without the assumption of a strictly positive lower bound on the target density.\n3. This paper has several technical insights or contributions. For neural network score and distribution estimations, to remove the density lower bound condition, this paper proposes a new proof strategy (Lemma 5).\n\n# Weaknesses\n1. As the authors discussed, the obtained results can have the issue of the curse of dimensionality, and a low-rank or manifold assumption on data can solve this. Besides, more discussions about additional technical challenges can be added, where the related work [1*,2*] might be helpful.\n2. The tightness of these bounds (e.g., Theorem 1 and 2) are not discussed.  More discussions can be added.\n3. Although this is a pure theoretical work, experimental results can be added to illustrate the validity of these bounds.\n\n[1*] Low-dimensional adaptation of diffusion models: Convergence in total variation\n\n[2*] Linear Convergence of Diffusion Models Under the Manifold Hypothesis", "was_revised": true, "final_justification": "The authors have addressed most of my concerns, including the discussions about the technical challenges for tackling the curve of dimensionality, the tightness of Theorems 1 & 2, and the technical challenge to remove the Lipschitz score assumption compared with [30]. The remaining concern is that there are no experimental results to illustrate the validity of the established bounds. Therefore, I keep my score.", "rebuttal": "We sincerely thank you for your insightful comments and constructive feedback, which have helped us improve the quality of our paper.\nWe address your comments in detail below.\n\n>**W1:**\nMore discussions about the technical challenges for tackling the curve of dimensionality; Suggestion to discuss [1*, 2*]. \n\nWe thank the reviewer for this constructive suggestion. \nIndeed, our current theoretical guarantees suffer from the curse of dimensionality (CoD), as we explicitly acknowledge in the conclusion. \nIncorporating structural assumptions, such as manifold assumption explored in [1*, 2*], or low-rank structures)  is a crucial next step to mitigate the CoD. \nThis would require non-trivial extensions to our analysis, particularly in adapting our empirical Bayes smoothing techniques and neural network approximation theory to efficiently exploit low-dimensional geometry. \nIn particular, our analysis relies on an isotropic Gaussian kernel for the empirical score estimator. To effectively leverage a manifold structure, this would need to be replaced with an estimator that respects the underlying geometry, such as one using anisotropic or manifold-intrinsic kernels (e.g., heat kernels), to prevent smoothing data off the manifold.\nA manifold-aware analysis, following, e.g., the path of [1*, 2*], would require establishing new bounds for score estimation and approximation that depend on the intrinsic dimension of the data, rather than the ambient dimension. \nWhile this is beyond the scope of the current work, we view it as a promising direction. \nWe thank the reviewer for this direction and will add a substantive discussion in the revision, referring to the cited works for context.\n\n\n>**W2:**\nDiscussing the tightness of Theorems 1 & 2. \n\nThank you for pointing this out. \nWhile we do not provide a full minimax lower bound for the neural network-based score estimation in Theorems 1 and 2, we establish the near-optimality of our framework in two key aspects:\n\n- Our rates in Theorem 2 match the minimax-optimal score estimation rate for regularized kernel-based estimators (see Lemma 1), up to logarithmic factors. \nIn particular, our results align with the optimal rate derived in [29] for a similar truncated estimator, indicating that this foundational part of our analysis is indeed tight.\n\n- When we supplement our sub-Gaussian assumption with standard nonparametric smoothness assumptions (i.e., Sobolev or Besov classes), our framework demonstrates that neural network-based SGMs can achieve nearly minimax optimal convergence rates for distribution estimation in total variation distance. This is detailed in Corollaries 1 and 2.\nThese results show that our framework is fundamentally sound and capable of achieving optimal rates when appropriate conditions are considered.\nWe will clarify this tightness explicitly in the revised manuscript.\n\n>**W3:**\nAdding experiments to illustrate the validity of the established bounds.\n\nWe appreciate the suggestion.\nAs our work is purely theoretical and aims to answer fundamental questions about SGMs under minimal assumptions, we prioritized analytical clarity. \nWe agree that illustrative experiments would be valuable for validating bounds empirically and will consider adding such results in future versions or follow-up work.\n\n\n\n>**Q1:**\nFor the score estimation via empirical Bayes smoothing, what is the technical challenge to remove the Lipschitz score assumption compared with [30]?\n\nThe primary challenge lies in controlling the score matching loss without causing the error bounds to depend sub-optimally on the noise variance $\\sigma$.\nIn particular, Lemma 2 of [30] established a convergence rate of the smoothed empirical distribution in Hellinger distance $H^2$ under the Lipschitz score assumption. \nAs discussed in [30] following their Lemma 2, one can instead bound the Hellinger distance by KL divergence by the fact $H^2 \\leq KL$ without the need of the Lipschitz score assumption. However, the SOTA convergence rate of the smoothed empirical measure in KL-divergence established in existing works, see [42], has a constant scale exponential in $1/\\sigma$, i.e., $\\mathcal{O}(\\exp(1/\\sigma)\\frac{\\log^dn}{n})$.  We improve this KL-divergence convergence rate in Lemma 3 for the constant from exponential dependence to **polynomial** dependence on $1/\\sigma$, i.e., $\\mathcal{O}(\\sigma^{-d}\\frac{\\log^{d/2}n}{n})$. This crucial improvement allows our bounds to hold even for small noise levels without needing the Lipschitz assumption to control the error. We believe this result on the convergence of smoothed empirical measures may be of independent interest.\n\n\n[42] Rate of convergence of the smoothed empirical Wasserstein distance. 2022"}, {"review_id": "bEp3nB4G90", "reviewer": "Reviewer_rjcC", "summary": "The paper studies the risk bound on estimation of score function in diffusion model via neural networks with i.i.d. samples and ERM algorithm where it is assumed that samples are drawn from a sub-Gaussian distribution. With estimated score function, the authors obtain an error bound on the total variation distance between the learned distribution and true distribution as a natural consequence. The main contribution of the paper is that the authors aim to remove some assumptions imposed by other works on data distribution while keep the error rate optimal.", "questions": "Same problem exists in all of the main results.\nFor example, in theorem 1, the conditions for stopping time is $t_0 \\ge 1/2 \\alpha^2 n^{-2/d} log n$. If you substitute this into the bound, we have $\\alpha^d t_0^{-d/2} n^{-1} log^{d/2+4}n \\lesssim log^4 n $.\n\nIn my understanding, the main problem is the assumption. Authors would like to obtain the risk bound without smoothness assumption, which seems unrealistic. In lemma 1, where they claim that they achieve the optimal rate with only sub-Gaussian assumption, the bound diverges with the sample size. Their results align with the results in [29]. If we bring the smoothness parameter $\\beta$ to 0 in [29], we obtain something similar to the bound in this paper. But $\\beta $ has to stay positive to guarantee the convergence.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: the paper is well-written with a clear structure. They present rather a complete picture of issues in learning via diffusion model. Also, it is fairly self-contained since they include most elements of the problem, such as how to utilize OU process to learn a target distribution, how to obtain estimation error and generalization error of score function via neural network, and how to compare learned distribution and target distribution once we have the former and etc. \n\nWeakness: the error bound divergences with sample size n.", "was_revised": true, "final_justification": "As the authors mentioned in the response, they would clarify that the convergence result would only hold for some proper stopping time without extra assumptions. In that case, their main results remain valid.", "rebuttal": "We sincerely thank you for this valuable feedback. \nYour observation is correct and highlights a key feature of our framework: it accurately reflects the fundamental trade-offs in nonparametric estimation. \nThe behavior you noted is expected and **does not undermine our main results**. \nWe address your concern in detail as follows. \n\n>W1/Q1: Divergence of bounds with sample size $n$\n\nWhile the reviewer’s observation is correct, this behavior is expected within our framework and does not undermine our main results. Specifically, \n\n- **It is expected with only a sub-Gaussian distribution assumption.** \nUnder only the sub-Gaussian assumption, we establish a convergence rate of $\\tilde{\\mathcal{O}}(t_0^{-d/2} n^{-1})$  for any $t_0 \\geq 1/2\\alpha^2n^{-2/d} \\log n$.\nThe rate indeed becomes vacuous (or diverges) if $t_0$ is taken too close to this lower limit, which is expected and consistent with the findings in [29].\nIn fact, [29] also shows that without assuming smoothness, the kernel density estimator cannot achieve meaningful convergence rates either. \nOur framework correctly captures this. \nIt reflects the fundamental difficulty of estimating an arbitrarily \"rough\" distribution. \n\n- **The bound converges to zero if we select an early stopping time $t_0$ that decreases with $n$ slightly slower than $1/2\\alpha^2n^{-2/d} \\log n$.** \nNoticing that we only require the early stopping time $t_0$ to be in a admissible range $t_0 \\geq 1/2\\alpha^2n^{-2/d} \\log n$ and the bound converges to zero when the early stopping time $t_0$ is set slightly larger, for instance, $t_0 = \\mathcal{O}(n^{-2/(d+\\epsilon)})$ for any $\\epsilon>0$. The convergence rate would be of $\\tilde{\\mathcal{O}}(n^{-\\frac{\\epsilon}{d+\\epsilon}})$. \n\n- **$t_0$ can be tuned to achieve nearly-optimal rates with additional smoothness assumptions.**\nIn particular, as detailed in **Corollaries 1 and 2**, when we add standard nonparametric smoothness assumptions, we can tune the early stopping time $t_0$ to balance the bias $TV(P_0, P_{t_0})$ and statistical error $TV(P_{t_0}, \\widehat{P}_{t_0})$. \nFor instance, with a Sobolev density, we choose $t_0 = n^{-2/(d+2s)}$, which satisfies the admissible range $t_0 \\geq 1/2\\alpha^2n^{-2/d}\\log n$, and we recover the nearly optimal convergence rate of $\\tilde{\\mathcal{O}}(n^{-2s/(d+2s)})$.\n\nSo in summary, our framework provides a general bound valid for all $t_0$ above the threshold, and when smoothness is available, the early stopping can be tuned to achieve the nearly minimax optimal rate. \n\nWe will clarify in both the abstract and main claims that minimax-optimality holds for Sobolev/Besov or otherwise regularized classes, while for the pure sub-Gaussian case we provide the convergence guarantee for $P_{t_0}$ when $t_0 > \\mathcal{O}(\\alpha^2n^{-2/d}\\log n)$."}, {"review_id": "I3rd5r1qZa", "reviewer": "Reviewer_tHVn", "summary": "This paper provides a theoretical analysis of score-based generative models (SGMs) under the mild assumption that the data distribution is sub-Gaussian. It shows that ReLU neural networks can approximate score functions with nearly optimal error rates, even without assuming Lipschitz continuity or lower-bounded densities. The authors establish generalization bounds for neural network-based SGMs and demonstrate that these models achieve near-minimax optimal convergence in total variation distance for Sobolev and Besov class densities. This work broadens the applicability of SGMs to more general and realistic data settings.", "questions": "See weakness", "limitations": "See weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1 The paper establishes approximation and estimation error bounds under weaker assumptions, achieving minimax optimal rates, improving upon a previous result that requires an additional Lipschitz assumption.\n2 It offers a technically refined analysis of the regularized empirical score estimator, showing that it attains nearly minimax-optimal rates under the score matching loss, improving theoretical understanding in a general setting.\n\nWeaknesses:\n1 Aside from the empirical Bayes smoothing analysis, many of the technical tools employed—such as Bernstein’s inequality and Girsanov’s theorem—are standard and well-established in the literature.\n2 While the relaxation of assumptions is valuable, the novelty of the contribution may be somewhat limited given the existing literature on the theoretical analysis of SGMs.", "was_revised": true, "final_justification": "Thanks for explaining the technical novelty. While I am not changing my score, I am leaning toward acceptance.", "rebuttal": "We sincerely thank you for your thoughtful feedback and acknowledgement of the value of relaxing assumptions. \nWe agree that many of the tools we employ, such as Bernstein’s inequality and Girsanov’s theorem, are well-established. \nOur primary contribution, however, lies in the novel synthesis and application of these tools to remove critical and restrictive assumptions from prior work, which required new, non-trivial technical developments. We address your comments in detail below. \n\n>**W1 \\& W2:** Use of standard tools and limited novelty of the contribution\n\nWhile the individual components are standard, their integration within our framework is what enables our main results. Specifically, our key technical contributions are:\n\n- **A refined KL-divergence bound (Lemma 3):** Prior work provided a KL-divergence rate for smoothed empirical distributions that scaled exponentially with the inverse of the noise standard deviation $1/\\sigma$. As detailed on page 8, we refine their proof to establish a rate that scales **polynomially** in $1/\\sigma$. This improvement is crucial for deriving meaningful bounds in the low-noise regime. \n\n- **New application of Bernstein's inequality:** To remove the density lower-bound assumption from [22, 26, 27], we introduce a new proof strategy. Instead of analyzing the true score function, which may be **unbounded**, we analyze the excess risk for learning a **surrogate** regularized empirical score function. This surrogate is uniformly bounded by construction (Lemma 5), allowing us to verify Bernstein's condition and obtain a uniform high-probability bound without requiring a density lower bound.\n\nThe combination of these non-trivial technical steps that allows us to provide the a theoretical framework establishing nearly minimax optimal rates for neural network-based SGMs under the general sub-Gaussian assumption, without the restrictive requirements of Lipschitz continuity or a density lower bound. We believe this significantly expands the applicability of SGM theory and represents a valuable contribution to the literature."}, {"review_id": "gbU5pdhP07", "reviewer": "Reviewer_XhMA", "summary": "The paper studies the approximation, generalization, and distribution estimation guarantees of score-based generative models with neural network estimators, under the sole assumption that the data distribution has sub-Gaussian tail decay. In particular, they remove two common assumptions made in previous works: the Lipschitz continuity of the score function (across the whole time domain) and a strict lower bound of the density. The key technical insight of the paper is that the regularized kernel score estimator, proposed in [30], approximates the true score function even without Lipschitz continuity of the score function. This allows them to prove two important results:\n\n1. Neural networks can approximate the score function with only the sub-Gaussianity assumption. This is proven by constructing a neural network which approximates the regularized kernel score estimator via local Taylor expansions of the kernel.\n2. Neural networks trained by empirical score matching generalize well. To prove this, they reduce the generalization error to a difference between losses of the ERM and kernel score estimator built from independent ghost samples. This random variable can be controlled using elementary techniques from probability theory, whereas direct approaches to bound the generalization error often require the density lower bound.\n\nUsing these two results, the authors bound the distribution estimation error of score-based generalization errors as a sum of the score matching error, which is minimax optimal up to log factors, and early stopping error. For some common nonparametric function spaces, such as Besov spaces, the early stopping error can also be controlled, leading to end-to-end bounds.\n\n[30] A. Wibisono, Y. Wu, and K. Y. Yang, “Optimal score estimation via empirical bayes smoothing,”\nin The Thirty Seventh Annual Conference on Learning Theory, June 30 - July 3, 2023, Edmonton,\nCanada, vol. 247 of Proceedings of Machine Learning Research, pp. 4958–4991, PMLR, 2024.", "questions": "- In Lemma 12, I believe the weights of the network $\\phi_{score}$ actually depend on the sample points $\\{x^{(i)}\\}$ being averaged, because the construction of $\\phi_{score}$ relies on approximating the kernel density estimator associated to the samples. This needs to be addressed in the proof of Theorem 1, where you drop the expectation from the bound. Essentially, I think you should clarify in which results the neural network constructed is random and in which results it is deterministic.\n- Can you explain where the $O(n^{3/d}))$ depth arises from, and whether this dependence on $n$ can be improved? Also, can you explain why the depth of the network decreases with dimension? Intuitively, a higher-dimensional distribution should require a more complex network to approximate its score.\n- In Lemma 2, I believe the set $\\mathcal{G}$ should be defined in terms of $\\rho_n$ rather than $\\rho$?\n- Is it possible to extend your results to allow for distributions whose tail decays slower than Gaussian? For instance, it seems the main use of the sub-Gaussianity is in bounding the second term in Lemma 2, but this term could still be controlled for distributions with fatter tails (e.g., sub-exponential or sub-Weibull).\n- At the bottom of page 8, why do you introduce both $f^{(1)}$ and $f^{(2)}$ if they are equal to the same expression?\n- Typo on page 43, Step 2, “estiamtors”\n- On page 22, what is the meaning of “standard Gaussian-convolved densities” and “general Gaussian-convolved densities”?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: The paper makes important progress to remove commonly-made assumptions in the analysis of score-based generative models. It is the first work, to my knowledge, to provide generalization guarantees under only sub-Gaussianity of the data distribution. The intermediate results, such as removing the Lipschitz continuity assumption in the generalization error of kernel score estimators and approximating KDEs via neural networks, may be of independent interest. The proof strategies are well-explained considering the technicality of the work.\n\nWeaknesses: Since the results apply to such a wide class of distributions, the convergence rates are quite pessimistic and probably fail to explain the real world successes of diffusion models. However, for a theoretically-focused paper, this is not a major concern. Additionally, the depth of the score network grows polynomially in the sample size $n$, which is quite large compared to related works (e.g., [22] constructs a score network with depth $O(polylog(n))$) and does not seem realistic.\n\n[22] K. Oko, S. Akiyama, and T. Suzuki, “Diffusion models are minimax optimal distribution\nestimators,” in International Conference on Machine Learning, pp. 26517–26582, PMLR, 2023.", "was_revised": true, "final_justification": "I maintain my original assessment, which is that the paper comfortably meets the criteria for acceptance. The paper is a technically impressive theoretical work which unifies and improves several previous results on error analysis of diffusion models and develops independently-interesting approximation and generalization error techniques along the way. The only limitation is the lack of experimental validation, but in general it is extremely hard to validate such worst-case results numerically, and it is common for these kinds of theoretical papers to be accepted to the conference when the contribution is strong enough. I recommend acceptance.", "rebuttal": "We are grateful for your extremely thorough and constructive review. \nYour detailed questions have helped us identify several areas for clarification.\nWe address your comments in detail as follows. \n\n>**W1:** \nPessimistic rates. \n\nWe agree that our convergence rates are conservative, especially for high-dimensional settings. As our work aims to offer worst-case guarantees under minimal assumptions, this is somewhat inevitable. Nonetheless, we believe our results fill an important gap in understanding what is theoretically achievable under weak regularity.\nWe see breaking the CoD through further structural assumptions as an important future direction.\n\n\n>**Q1:**\nSample-dependence of the score network\n\nThank you for this sharp and important question. You are correct that the score network constructed in our approximation results is sample-dependent. We clarify how this is handled in our analysis below.\n\n1. **The approximating network in Theorem 1.**\nThe neural network $\\phi_{\\text{score}}$ in Theorem 1 is indeed a random network. \nIt is constructed to approximate the regularized empirical score function (a KDE-based estimator), and its parameters therefore depend on the specific sample used to construct the kernel estimator. \nThe statement of Theorem 1 should be interpreted as follows: for any given sample $\\bar{S} := \\bar{x}^{(1)}, \\dots, \\bar{x}^{(n)} \\sim P_0^{\\otimes n}$, there exists a network with the specified architecture constructed from $S$ that achieves the stated approximation error.\nThe expectation in the theorem's bound should be understood as an expectation taken over both the test point $x_t$ and the randomness of the sample $\\bar{S}$, i.e., $E_{\\bar{S}}[E_{x_t \\sim P_t}[\\cdot]]$.\nWe will revise the statement of Theorem 1 to make this sample dependence explicit.\n\n2. **The learned network in Theorem 2.**\nThis sample dependence is a critical point that we carefully account for in our generalization analysis for the learned network $\\widehat{\\phi}$ in Theorem 2. \nOur proof strategy is as follows:\n\n   - We introduced an independent \"ghost sample\" $\\bar{S}$ to construct the target regularized empirical score function, $\\widehat{\\bar{\\phi}}$ (Line 1689-1690 Page 79), as noted by the reviewer.\n\n   - The \"ideal\" approximating network $\\tilde{\\phi}$ (Line 1725 Page 83) from Theorem 1 can then be considered to be the one constructed based on this ghost sample. This makes $\\tilde{\\phi}$ a random target that is independent of the actual training data. \n\n   - The final network $\\widehat{\\phi}$ is obtained by performing empirical risk minimization on the actual training data. \n\n    Our proof correctly bounds the excess risk by properly handling the randomness from both sample sets. Therefore, our generalization bound and proof of Theorem 2 remain valid.\n\nWe will revise the manuscript to improve clarity on this crucial point.\n\n>**W2/Q2:**\nWhere $\\tilde{\\mathcal{O}}(n^{3/d})$ depth arises from, and whether it can be improved. \n\nThis is an excellent and constructive point. \nWe agree that a network depth that grows polynomially with the sample size is a practical limitation. \n**This can be improved within our framework**.\n\nThe depth of $\\tilde{\\mathcal{O}}(n^{3/d})$ (for $k \\geq d/2$) arises from a specific choice of parameters $N = 1, L = \\lceil n^{1/(2k)} \\rceil, k \\geq d/2, \\epsilon = n^{-1/k}, s=k$ in the proof of Lemma 12 (see Line 1447-1448 Page 67), which are made to satisfy a technical condition from Lemma 11 and 12 (i.e., $N^{-2}L^{-2} \\leq t_0 \\wedge n^{-1/k}$). \nHowever, this choice is not fundamental. \nOur framework allows for separate control over network width and depth. \nFollowing the reviewer's suggestion, we can revise our construction to achieve a more realistic architecture. \nBy swapping the roles of width and depth, we can choose $N = \\lceil n^{1/(2k)} \\rceil, L = 1$ and construct a network with width $\\leq \\mathcal{O}(n^{\\frac{3}{2k}}\\log_2n)$ and depth $\\leq \\mathcal{O}(\\log^2n)$. \nThis new architecture is more practical and, crucially, our main theoretical guarantees remain valid: \n\n- The condition $N^{-2}L^{-2} \\leq t_0 \\wedge n^{-1/k}$ required by Lemma 11 and 12 is still satisfied.\n\n- The generalization bound in Theorem 2 remains valid since the new network architecture's pseudo-dimension remains of the same order, which is bounded by $\\tilde{\\mathcal{O}}(n^{\\frac{3}{k}})$. \n\nWe are grateful for this suggestion and will update the manuscript to reflect this significantly improved architecture.\n\n>**Q2:** Dependence on dimension $d$. \n\nThank you for the excellent question. \nThis counter-intuitive relationship is a direct consequence of the interplay between the smoothness of $P_{t_0}$ and the network architecture in our theoretical analysis, which has a clear theoretical motivation:\n\n- **Higher dimensions force more smoothing.**\nOur condition on the early stopping time (i.e., $t_0 \\geq \\tilde{\\mathcal{O}}(n^{-2/d})$ in Theorem 1 and 2 as well as $t_0 = n^{\\frac{-2}{d+2s}}$ in Corollary 1 and 2) forces $t_0$ to be larger for higher dimension $d$ to ensure theoretical guarantees. \n\n- **More smoothing makes the marginal distribution $P_{t_0}$ easier to learn.**\nA larger $t_0$ means that $P_{t_0}$ is convolved with a Gaussian of larger variance. \nThis makes $P_{t_0}$ inherently **smoother** and less complex, as fine-grained details are averaged out. \nAs a result, its score can be approximated by a network whose size scales less severely with the sample size $n$. \n\nIn summary, while higher dimension usually implies increased statistical difficulty (as seen in the convergence rate $n^{-s/(d+2s)}$ suffering from the curse of dimensionality), the necessity of stronger smoothing to achieve uniform control in high dimensions makes the intermediate learning problem architecturally less demanding in terms of network size. This is a reflection of the bias–variance trade-off: we reduce variance (by smoothing), but incur bias, which is ultimately what limits the rate. \nWe will clarify this point in the revised manuscript. \n\n\n>**Q3:**\nUse of $\\rho_n$ in $\\mathcal{G}$ Definition. \n\nYes, this should be written in terms of $\\rho_n$. \nWe will correct this typo.\n\n\n>**Q4:**\nExtension to sub-exponential or sub-Weibull distributions\n\nThis is an excellent suggestion for future work. \nOur current analysis relies on the sub-Gaussian assumption in two critical places:\n\n- Lemma 7: To bound the score function's norm over low-density regions.\n\n- Lemma 3: To bound the KL-divergence of the Gaussian smoothed empirical distribution, which is fundamental to our main error bounds.\n\nExtending these results to broader tail classes like sub-exponential or sub-Weibull would require more refined control of tail integrals, but we agree it is plausible and an exciting avenue for future work. \nIn particular, we believe that the extensions may benefit from the convergence rates of Gaussian smoothed empirical distributions under heavy-tail distribution assumptions. \nWe will include the discussion in the revision. \n\n\n>**Q5:**\nUse of $f^{(1)}, f^{(2)}$. \n\nThese notations correspond to different components in the DNN decomposition (e.g., numerator vs. denominator in the score approximation), and are treated separately for modular construction. \nAn alternative approach would be to construct a single sub-network for this expression and reuse its output via connections to different parts of the network. \nSince both methods result in a final network of a similar size and complexity, our notation was intended to reflect this separation.\n\n\n>**Q6:**\nTypo \"estiamtors\"\n\nThank you. We will correct the typo.\n\n>**Q7:**\nClarification on “Standard” vs. “General” Gaussian-convolved Densities\n\n\"Standard Gaussian-convolved\" refers to convolution with a standard Gaussian distribution $\\mathcal{N}(0, \\mathbf{I}_d)$, whereas \"general Gaussian-convolved\" refers to convolution with a general isotropic Gaussian distribution $\\mathcal{N}(0, \\sigma^2\\mathbf{I}_d)$ for any $\\sigma > 0$. \nWe will improve this terminology for clarity."}]

Author rebuttal:

Meta-review: This paper studies theoretical properties of score-based generative models, and establishes nearly optimal rates under weaker assumptions than existing work. The paper is technically dense but well-written, as noted by most reviewers. Several comments came up during review:
- The assumptions that are relaxed (Lipschitz, LSI, density lower bound) are typically made out of convenience, and are not fundamental road blocks in existing papers. To quote one expert reviewer: _""While the relaxation of assumptions is valuable, the novelty of the contribution may be somewhat limited given the existing literature on the theoretical analysis of SGMs"_
- The techniques used to remove these assumptions are standard: Empirical Bayes, Bernstein’s inequality, Girsanov’s theorem, etc
- Although the authors comment that "the theoretical foundations of SGMs have been less thoroughly explored", their own literature review (citing 20+ theoretical papers) suggests otherwise. Indeed, this problem is extensively studied.
- One reviewer raised a substantial concern about the potential vacuosity of the bounds, however, during the discussion, this was clarified. In particular, without smoothness assumptions (eg only subgaussian), the rates must degrade out of necessity, and the authors show that under smoothness conditions this can be corrected (consistent with existing results on density estimation).

During the discussion, the latter concern was addressed. In the end, all reviewers support accepting the paper. The concerns about techniques and assumptions remain, but the successful application of these ideas to SGMs should be of broad interest to the NeurIPS community.
Decision: Accept (poster)