Title: Algorithm- and Data-Dependent Generalization Bounds for Diffusion Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion models

Generalization bounds

Score-based generative models
Abstract: Score-based generative models (SGMs) have emerged as one of the most popular classes of generative models. A substantial body of work now exists on the analysis of SGMs, focusing either on discretization aspects or on their statistical performance. In the latter case, bounds have been derived, under various metrics, between the true data distribution and the distribution induced by the SGM, often demonstrating polynomial convergence rates with respect to the number of training samples. However, these approaches adopt a largely approximation theory viewpoint, which tends to be overly pessimistic and relatively coarse. In particular, they fail to fully explain the empirical success of SGMs or capture the role of the optimization algorithm used in practice to train the score network. To support this observation, we first present simple experiments illustrating the concrete impact of optimization hyperparameters on the generalization ability of the generated distribution. Then, this paper aims to bridge this theoretical gap by providing the first algorithmic- and data-dependent generalization analysis for SGMs. In particular, we establish bounds that explicitly account for the optimization dynamics of the learning algorithm, offering new insights into the generalization behavior of SGMs. Our theoretical findings are supported by empirical results on several datasets.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes both the sample complexity (data-dependent) and the algorithmic generalization ability (algorithm-dependent) of the score-matching error that arises when training score-based generative models using practical optimization algorithms. The theoretical analysis investigates the generalization behavior of using SGLD and ADAM to solve the empirical score-matching objective, offering insights into how hyperparameters and topological complexity measures influence the generalization error in the context of training score-based diffusion models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper is clearly written and provides a helpful overview of prior work, effectively situating its contributions within the existing literature.\n\n2. The theoretical results are well presented and, to the best of my knowledge, offer novel results in the context of score-based diffusion models.\n\n---\n\n**Weaknesses:**\n\n1. Based on my understanding, Theorems 4.1 and 4.2 are adapted from existing results. However, the required assumptions are not explicitly verified or discussed in sufficient detail.\n\n2. The topological complexity quantities introduced in Section 4.3 are not intuitive, and the paper does not provide adequate interpretation of these terms within the context of diffusion models. It would also be helpful to have some discussions on how the result differs from standard settings where ADAM is used to minimize generic training objectives.\n\n3. The theoretical findings offer limited insight into the empirical success of score-based diffusion models or practical guidance for their training. \n\nPlease see the questions related to above three points in the Question section.\n\nQUESTIONS:\n**Q1:**  \nIn Theorem 4.1, the loss function $\\ell_{\\lambda}(w, Z)$ is assumed to be $\\tau^2$-subgaussian. Is this assumption directly implied by Assumptions 2.1 and 3.1? Similarly, Theorem 4.2 assumes that $\\ell_{\\lambda}(\\theta, z)$ is uniformly bounded with respect to both $\\theta$ and $z$. Does this automatically follow from Assumptions 2.1 and 3.1?\n\n---\n\n**Q2:**  \nHow should we interpret the three topological complexity measures introduced in Section 4.3 in the context of score-based diffusion models? Do these quantities provide practical guidance for model training?\n\n---\n\n**Q3:**  \nFigure 1 suggests that using a large batch size makes training less sensitive to the choice of learning rate and also improves generation quality. Can the theoretical results offer any insight into this empirical observation?\n\n---\n\n**Q4:**  \nBeyond the choice of training hyperparameters, the noise scheduling in the forward process [1] is known to significantly affect the performance of score-based generative models. Is it possible to incorporate the scheduling of the forward process into the current theoretical framework to offer guidance on how to choose or optimize the noise schedule?\n\n---\n\n**Q5:**  \nThe results in Theorems 4.1 and 4.2 appear to be independent of model complexity. Could the authors comment on why this is the case? Is it correct to interpret that model complexity only influences the empirical explicit score-matching loss (i.e., the first term in the error decomposition in Theorem 3.1)?\n\n---\n\n**Q6:**  \nCould the authors provide additional numerical results illustrating how the empirical explicit score-matching loss depends on the hyperparameters and the other complexity measures introduced in the paper? For instance, similar to Figures 2 and 3, but tracking the score-matching loss instead of the generalization gap.\n\n[1] Karras, Tero, et al. \"Elucidating the design space of diffusion-based generative models.\" Advances in neural information processing systems 35 (2022): 26565-26577.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
This paper delves into the statistical analysis of diffusion models and is the first to take the Dirac sampling from the dataset and optimization generalization error into account. They further conduct experiments to show the effect of optimization dynamics on the generalization ability of the DMs.

STRENGTHS AND WEAKNESSES:
**Strengths**

1. The mathematical derivation is rigorous.
2. The experiments and figures are illustrative to show the effect of optimization dynamics on the generalization ability of DMs.

**Weaknesses**
1. The generalization analysis of the optimization algorithms like SGLD and ADAM directly follows the previous works, but only adopts them into the diffusion models’ score matching analysis.
2. I am not sure if the ‘Dirac sampling from data distribution’ is both the source of the data-dependent constant and the score generalization gap in your analysis. I think a more heuristic explanation of the source of the different error parts presented in your analysis is needed.

QUESTIONS:
See weakness

LIMITATIONS:
no
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 1
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical bound on the Kullback-Leibler (KL) divergence between the data distribution and the generated distribution of score-based generative models based on Ornstein-Uhlenbeck process.\nThe paper presents a key decomposition of an approximation error term of the score function, enabling further analysis of generalization performance based on empirical datasets and optimizers.\nUsing this decomposition, the paper derives a gradient-based generalization bound for stochastic gradient Langevin dynamics (SGLD) and a topological generalization bound for ADAM.\nExperiments on image datasets demonstrate a correlation between the generalization error and the derived upper bounds.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\nThe paper is well-structured. The introduction clarifies the target issues of previous work and provides a clear overview of the theoretical results. The logical flow is easy to follow, and the purpose of each section is clearly defined. The limitations of the work are well-discussed.\nThe paper demonstrates the usefulness of Theorem 3.1 and Proposition 3.1 by extending algorithm and data-dependent generalization bound analyses based on these derived theorems.\nThe experimental results on image datasets support the presented theoretical claims.\n\n### Weaknesses\n1. In the introduction (around L73), the lack of consideration for practical model architectures such as UNet is raised as an issue of an existing approach. However, the algorithm-dependent generalization analysis in the paper focuses primarily on optimizers, and the effects of model architectures do not appear to be discussed.\n2. In Figures 4 and 5 in the Appendix, the vertical axes of the bottom right figures appear unusual. They repeat the same value as the tick labels.\n3. There are some minor typos and missing information. I have listed some of them below:\n    * There appears an unnecessary parenthesis ( within s(T-t, \\overleftarrow{X_t}) in (3).\n    * The order of arguments of s_{\\hat{\\theta}^{(n)}} in (6) seems to be swapped; it is supposed to be (s, t) instead of (t, s).\n    * Just above (6), t\\in[t_k, t_{k+1}] is supposed to be the subscript of (\\hat{X}_t^{(n)}).\n    * The definition of B(0,D) is missing in Assumption 3.1. It likely refers to the ball centered at 0 with radius D.\n    * \\sigma^2 around L669 does not appear to be introduced, and it should consistently be 2 throughout the paper.\n    * There is a word duplication: \"have have\" in L696.\n    * In the equations below L680, d\\nu(x) is supposed to be d\\nu(z).\n    * In L715 and L716, the paper states that Hölder's inequality was used to derive inequalities. However, it seems more likely that Jensen's inequality was used.\n    * \\zeta around L754 is supposed to be \\delta.\n\nQUESTIONS:\n1. Could you please clarify the connection between the presented theoretical results and the limitation (i) mentioned in L73 regarding practical classes of score estimators?\n2. Could you please address the errors in Weaknesses 2 and 3?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes in more detail the generalization error of learning a score function. They propose a new decomposition of the score error term that is commonly assumed to be small (under large hypothesis classes) and analyze each term of the decomposition separately, providing concentration bounds and ending with progress towards end-to-end analysis of the error of learning a score. In particular, the score error is broken up into 3 terms, one involving the inherent loss of the score function trained on the data, and the other two representing errors that occur from using an empirical dataset rather than the true distribution. These terms represent the generalization gap. These latter terms are then analyzed and bounded. They also show numerical experiments that support their theoretical claims.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe breakdown of the score error into three (somewhat) interpretable terms is great, especially since these terms can be bounded. The more careful analysis of the score matching error term allows one to more clearly understand what is going on with the error term. The writing is also clear and seems well-focused on conveying the intuitive meaning of the work.\n\nWeaknesses:\nAlthough the paper does bring up a new way in which to analyze the score error, the analysis of one of the terms is from previous results, and merely borne out in experiments in this work, and another term is (understandably) left as inherent. The third term is bounded using a (somewhat coarse, as admitted by the authors) concentration inequality, which seems to be the main theoretical contribution of the work. Still, the decomposition is novel in itself and serves as a useful tool for further analysis.\n\nQUESTIONS:\n1. Can assumption 3.1 be weakened to be bounded with high probability? being strictly bounded is a little strong, considering Gaussians have exponentially small (but nonnegative) infinite tails. Perhaps some subgaussian assumption instead? \n2. Is Figure 3 just evidence that supports the theorem by [MWZZ]? Is there anything represented by the graphs that is meant to represent theory found in this paper? \n3. In general, is there anything in section 4 that is particularly novel to this paper?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}