Title: Approximation and Generalization Abilities of Score-based Neural Network Generative Models for Sub-Gaussian Distributions
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion models

Score-based generative models

Approximation

Generalization
Abstract: This paper studies the approximation and generalization abilities of score-based neural network generative models (SGMs) in estimating an unknown distribution $P_0$ from $n$ i.i.d. observations in $d$ dimensions. 
Assuming merely that $P_0$ is $\alpha$-sub-Gaussian, we prove that for any time step $t \in [t_0, n^{\mathcal{O}(1)}]$, where $t_0 > \mathcal{O}(\alpha^2n^{-2/d}\log n)$, there exists a deep ReLU neural network with width $\leq \mathcal{O}(n^{\frac{3}{d}}\log_2n)$ and depth $\leq \mathcal{O}(\log^2n)$ that can approximate the scores with $\tilde{\mathcal{O}}(n^{-1})$ mean square error and achieve a nearly optimal rate of $\tilde{\mathcal{O}}(n^{-1}t_0^{-d/2})$ for score estimation, as measured by the score matching loss.
Our framework is universal and can be used to establish convergence rates for SGMs under milder assumptions than previous work.
For example, assuming further that the target density function $p_0$ lies in Sobolev or Besov classes, with an appropriately early stopping strategy, we demonstrate that neural network-based SGMs can attain nearly minimax convergence rates up to logarithmic factors.
Our analysis removes several crucial assumptions, such as Lipschitz continuity of the score function or a strictly positive lower bound on the target density.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the approximation, generalization, and distribution estimation guarantees of score-based generative models with neural network estimators, under the sole assumption that the data distribution has sub-Gaussian tail decay. In particular, they remove two common assumptions made in previous works: the Lipschitz continuity of the score function (across the whole time domain) and a strict lower bound of the density. The key technical insight of the paper is that the regularized kernel score estimator, proposed in [30], approximates the true score function even without Lipschitz continuity of the score function. This allows them to prove two important results:\n\n1. Neural networks can approximate the score function with only the sub-Gaussianity assumption. This is proven by constructing a neural network which approximates the regularized kernel score estimator via local Taylor expansions of the kernel.\n2. Neural networks trained by empirical score matching generalize well. To prove this, they reduce the generalization error to a difference between losses of the ERM and kernel score estimator built from independent ghost samples. This random variable can be controlled using elementary techniques from probability theory, whereas direct approaches to bound the generalization error often require the density lower bound.\n\nUsing these two results, the authors bound the distribution estimation error of score-based generalization errors as a sum of the score matching error, which is minimax optimal up to log factors, and early stopping error. For some common nonparametric function spaces, such as Besov spaces, the early stopping error can also be controlled, leading to end-to-end bounds.\n\n[30] A. Wibisono, Y. Wu, and K. Y. Yang, “Optimal score estimation via empirical bayes smoothing,”\nin The Thirty Seventh Annual Conference on Learning Theory, June 30 - July 3, 2023, Edmonton,\nCanada, vol. 247 of Proceedings of Machine Learning Research, pp. 4958–4991, PMLR, 2024.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper makes important progress to remove commonly-made assumptions in the analysis of score-based generative models. It is the first work, to my knowledge, to provide generalization guarantees under only sub-Gaussianity of the data distribution. The intermediate results, such as removing the Lipschitz continuity assumption in the generalization error of kernel score estimators and approximating KDEs via neural networks, may be of independent interest. The proof strategies are well-explained considering the technicality of the work.\n\nWeaknesses: Since the results apply to such a wide class of distributions, the convergence rates are quite pessimistic and probably fail to explain the real world successes of diffusion models. However, for a theoretically-focused paper, this is not a major concern. Additionally, the depth of the score network grows polynomially in the sample size $n$, which is quite large compared to related works (e.g., [22] constructs a score network with depth $O(polylog(n))$) and does not seem realistic.\n\n[22] K. Oko, S. Akiyama, and T. Suzuki, “Diffusion models are minimax optimal distribution\nestimators,” in International Conference on Machine Learning, pp. 26517–26582, PMLR, 2023.\n\nQUESTIONS:\n- In Lemma 12, I believe the weights of the network $\\phi_{score}$ actually depend on the sample points $\\{x^{(i)}\\}$ being averaged, because the construction of $\\phi_{score}$ relies on approximating the kernel density estimator associated to the samples. This needs to be addressed in the proof of Theorem 1, where you drop the expectation from the bound. Essentially, I think you should clarify in which results the neural network constructed is random and in which results it is deterministic.\n- Can you explain where the $O(n^{3/d}))$ depth arises from, and whether this dependence on $n$ can be improved? Also, can you explain why the depth of the network decreases with dimension? Intuitively, a higher-dimensional distribution should require a more complex network to approximate its score.\n- In Lemma 2, I believe the set $\\mathcal{G}$ should be defined in terms of $\\rho_n$ rather than $\\rho$?\n- Is it possible to extend your results to allow for distributions whose tail decays slower than Gaussian? For instance, it seems the main use of the sub-Gaussianity is in bounding the second term in Lemma 2, but this term could still be controlled for distributions with fatter tails (e.g., sub-exponential or sub-Weibull).\n- At the bottom of page 8, why do you introduce both $f^{(1)}$ and $f^{(2)}$ if they are equal to the same expression?\n- Typo on page 43, Step 2, “estiamtors”\n- On page 22, what is the meaning of “standard Gaussian-convolved densities” and “general Gaussian-convolved densities”?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-13"}

comments: SUMMARY:
This paper provides a theoretical analysis of score-based generative models (SGMs) under the mild assumption that the data distribution is sub-Gaussian. It shows that ReLU neural networks can approximate score functions with nearly optimal error rates, even without assuming Lipschitz continuity or lower-bounded densities. The authors establish generalization bounds for neural network-based SGMs and demonstrate that these models achieve near-minimax optimal convergence in total variation distance for Sobolev and Besov class densities. This work broadens the applicability of SGMs to more general and realistic data settings.

STRENGTHS AND WEAKNESSES:
Strengths:
1 The paper establishes approximation and estimation error bounds under weaker assumptions, achieving minimax optimal rates, improving upon a previous result that requires an additional Lipschitz assumption.
2 It offers a technically refined analysis of the regularized empirical score estimator, showing that it attains nearly minimax-optimal rates under the score matching loss, improving theoretical understanding in a general setting.

Weaknesses:
1 Aside from the empirical Bayes smoothing analysis, many of the technical tools employed—such as Bernstein’s inequality and Girsanov’s theorem—are standard and well-established in the literature.
2 While the relaxation of assumptions is valuable, the novelty of the contribution may be somewhat limited given the existing literature on the theoretical analysis of SGMs.

QUESTIONS:
See weakness

LIMITATIONS:
See weakness
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the risk bound on estimation of score function in diffusion model via neural networks with i.i.d. samples and ERM algorithm where it is assumed that samples are drawn from a sub-Gaussian distribution. With estimated score function, the authors obtain an error bound on the total variation distance between the learned distribution and true distribution as a natural consequence. The main contribution of the paper is that the authors aim to remove some assumptions imposed by other works on data distribution while keep the error rate optimal.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: the paper is well-written with a clear structure. They present rather a complete picture of issues in learning via diffusion model. Also, it is fairly self-contained since they include most elements of the problem, such as how to utilize OU process to learn a target distribution, how to obtain estimation error and generalization error of score function via neural network, and how to compare learned distribution and target distribution once we have the former and etc. \n\nWeakness: the error bound divergences with sample size n.\n\nQUESTIONS:\nSame problem exists in all of the main results.\nFor example, in theorem 1, the conditions for stopping time is $t_0 \\ge 1/2 \\alpha^2 n^{-2/d} log n$. If you substitute this into the bound, we have $\\alpha^d t_0^{-d/2} n^{-1} log^{d/2+4}n \\lesssim log^4 n $.\n\nIn my understanding, the main problem is the assumption. Authors would like to obtain the risk bound without smoothness assumption, which seems unrealistic. In lemma 1, where they claim that they achieve the optimal rate with only sub-Gaussian assumption, the bound diverges with the sample size. Their results align with the results in [29]. If we bring the smoothness parameter $\\beta$ to 0 in [29], we obtain something similar to the bound in this paper. But $\\beta $ has to stay positive to guarantee the convergence.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the approximation and generalization abilities of score-based (ReLU deep) neural network generative models (SGMs). The existing related theory work has two main limitations: some assumptions regarding the data remain quite restrictive, and kernel-based estimators are considered instead of neural network-based ones. To tackle the above issues, this paper provides a universal framework to characterize the approximation and generalization abilities of SGMs for sub-Gaussian distributions. Further, it also applies to the case where the target density function lies in Sobolev or Besov classes, with an appropriately early stopping strategy, demonstrating that neural network-based SGMs can attain nearly minimax convergence rates up to logarithmic factors.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. As a dense theoretical work, this paper is overall well-written with clear notations. The motivation is clear that the existing theory work has quite restrictive assumptions on the data, and neural network-based estimators are less considered. \n2. The results are promising and seem right. The proof sketches make sense, although I have not carefully checked the detailed proofs in the appendix since the proofs are rather long. Compared with related work, this work removes several restrictive assumptions. More specifically, compared with prior work [30], this work removes the assumption of the Lipschitz continuity of the score function under the sub-Gaussian distribution. In comparison with previous work [22, 26, 27], this paper derives the approximation and estimation error bound without the assumption of a strictly positive lower bound on the target density.\n3. This paper has several technical insights or contributions. For neural network score and distribution estimations, to remove the density lower bound condition, this paper proposes a new proof strategy (Lemma 5).\n\n# Weaknesses\n1. As the authors discussed, the obtained results can have the issue of the curse of dimensionality, and a low-rank or manifold assumption on data can solve this. Besides, more discussions about additional technical challenges can be added, where the related work [1*,2*] might be helpful.\n2. The tightness of these bounds (e.g., Theorem 1 and 2) are not discussed.  More discussions can be added.\n3. Although this is a pure theoretical work, experimental results can be added to illustrate the validity of these bounds.\n\n[1*] Low-dimensional adaptation of diffusion models: Convergence in total variation\n\n[2*] Linear Convergence of Diffusion Models Under the Manifold Hypothesis\n\nQUESTIONS:\n1. For the score estimation via empirical Bayes smoothing, what is the technical challenge to remove the Lipschitz score assumption compared with [30]?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}