Generalization through variance: how noise shapes inductive biases in diffusion models
Authors: John Vastola
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion models
generalization
inductive biases
theory
infinite-width neural networks
generative models
path integral
Primary area: generative models
TL;DR: Diffusion models generalize well partly because of noise built into their training objective

Abstract: How diffusion models generalize beyond their training set is not known, and is somewhat mysterious given two facts: the optimum of the denoising score matching (DSM) objective usually used to train diffusion models is the score function of the training distribution; and the networks usually used to learn the score function are expressive enough to learn this score to high accuracy. We claim that a certain feature of the DSM objective—the fact that its target is not the training distribution's score, but a noisy quantity only equal to it in expectation—strongly impacts whether and to what extent diffusion models generalize. In this paper, we develop a mathematical theory that partly explains this 'generalization through variance' phenomenon. Our theoretical analysis exploits a physics-inspired path integral approach to compute the distributions typically learned by a few paradigmatic under- and overparameterized diffusion models. We find that the distributions diffusion models effectively learn to sample from resemble their training distributions, but with `gaps' filled in, and that this inductive bias is due to the covariance structure of the noisy target used during training. We also characterize how this inductive bias interacts with feature-related inductive biases.

Published as a conference paper at ICLR 2025
GENERALIZATION THROUGH VARIANCE: HOW NOISE SHAPES INDUCTIVE BIASES IN DIFFUSION MODELS
John J. Vastola
Department of Neurobiology Harvard Medical School
Boston, MA 02115, USA
john.vast ola@mms hww.aaaraae..... edu
ABSTRACT
How diffusion models generalize beyond their training set is not known, and is somewhat mysterious given two facts: the optimum of the denoising score match- ing (DSM) objective usually used to train diffusion models is the score function the training distribution; and the networks usually used to learn the score function are expressive enough to learn this score to high accuracy. We claim that a certain feature of the DSM objective the fact that its target is not the training distribu- impacts
tion's score, but a noisy quantity only equal to it in expeccationnstrongge whether and to what extent diffusion models generalize. In this paper, we develop a mathematical theory that partly explains this 'generalization through variance' phenomenon. Our theoretical analysis exploits a wwhysics-inspppppe proach to compute the distributions typically learned by a few paradigmatic under- and overparameterized diffusion models. We find that the distributions diffusion models effectively learn sample from resemble their training distributions, but with 'gaps' filled in, and that this inductive bias due to the covariance structure of the noisy target used during training. We also characterize how this inductive bias interacts with meaturerrelate inductive biases.
path integral ap-
INTRODUCTION
Diffusion models (Sekt-Dieksteinn al., 2015 Song & Ermon 2019 Ho et al. 2020 Yang et al, 2023) have proven effective at producing high-quality samples (e.g., images) like those from some training distribution, but not overwhelmingly so. This ability to generalize somewhat surprising for two reasons. First, the optimum of the denoising score matching (DSM) objective usually used to train them is the score function of the training distribution (Vincent 2011 Song & Ermon 2019). and sampling using this score only reproduces training examples (see Appendix AA. Second, the network architectures usually used for score function approximation are highly expressive. Two near-SOTA models developed by Karras al. (2022) have 56 million (CIFAR-10 10, trained on 200 million samples) and ~ 296 million parameters (ImageNet-64, trained on 2500 million samples), respectively. Sufficiently expressive models can fit even random noise (Zhang al. 2017
A body of empirical work bears on the question of when and to what extent diffusion models gen- eralize. Training data is more likely be memorized when training sets are small (Somepalli al.)
2023a
Stein et al. 2023 Dar et al. 2024 Kadkhodaie et al. (024), contain duplicates (Somepalli
al.
2023a| Carlmni et al. [0233 Somepalli al. (023b), or feature low 'complexity' (Somepalli
et al. 2023b Stein et al. (023). The specific training examples more likely to be memorized are ei- ther highly duplicated or outliers (Carlini al. | 2023). Whether generalization happens also strongly depends model capacity, with Yoon et al. (2023) and Zhang al. (2024) observing a sharp tran- sition from memorization to generalization as the number of training examples used somewhat out- strips model capacity. However, the relationship between model performance (e.g.. FID score) and model size, given a fixed number of training examples, is not monotonic; Karras et al. (024) observe that their ImageNet models strictly improve (and hence generalize better) as model size increases.
At present, there is arguably no theory that describes when diffusion models generalize and charac- terizes how the associated inductive biases depend details like training set structure, the choice

---

Published as a conference paper at ICLR 2025
of forward/teverse processes, and model architecture. Most existing theoretical work focuses on orthogonal questions: given a known ground truth, can one mathematically guarantee that in some limit (e.g., large or infinite number of samples from the ground truth distribution) diffusion mod- els recover the ground truth, and bound how score approximation error impacts agreement (Bortoli, [202 Chen al., 2023a c HHan al.) 2 The question we interested in is qualitatively dif- ferent: given M 2 =1 examples from a data distribution Pdato. how do samples from a model trained on those examples differ from them? For example, does the model effectively interpolate training data? If so, when, and what details does this depend on? Concurrent work (Kamb & Ganguli 2024]
Niedoba et al. 2025) addresses these questions at the level of phenomenology, but not mechanism.
In this paper, we argue that six factors substantially impact how diffusion models generalize.
1. Noisy objective. The target of the DSM objective is not the score of the training distri- bution, but a noisy quantity only equal to in expectation. This quantity, which we call the 'proxy score', introduces additional randomness to training. and has extremely high variance at low noise levels (infinite variance, in fact, at zero noise). Intuitively, this makes score function estimates, especially at low noise levels, inaccurate (this is well-known|JKan ras et al. (2022) remark on this when they discuss their choice loss weighting). Moreover, this variance not uniform in state space, but higher in 'boundary regions', e.g., regions of state space close to multiple training examples. This provides a useful inductive bias.
2. Forward process. Details of the forward process (e-g., when noise is added, asymmetry in how noise is added along different directions of state space) affect generalization through their influence on the covariance structure of the proxy score.
3. Nonlinear score-dependence. The learned distribution depends nonlinearly on the learned score function through the dynamics of the reverse process. This implies that the average learned distribution is not the training distribution, even if the score estimator is unbiased.
4. Model capacity. Models generalize better when # training samples ~ # model parameters.
5. Model features. Feature-related inductive biases interact with, and can enhance, inductive biases due to the covariance structure of the proxy score.
6. Training set structure. Nontrivial generalization (e.g.. interpolation) is substantially more likely when large number of training examples are near each other in state space; outliers are less likely to be meaningfully generalized.
Hence, details of training (1, 2), sampling (3), model architecture (4, 5), and the training set (6) all interact to determine the details of generalization. Other aspects, like learning dynamics. also almost certainly play a role, but we mostly neglect them here. The first factor is particularly important, and without it we will see that diffusion models do not generalize well; for this reason, we refer to the phenomenon enabled by (1) and affected by (2-6) as generalization through variance.
We support this claim using physics-inspired theory. The Martin-Siggia-Ros: (MSR) path integral description of stochastic dynamics (Martin et al.] 1973). which has also been exploited to charac- terize random neural networks (Crisanti & Sompolinsky 2018) and learning dynamics (Mignacco plays a pivotal role in our analysis. First, we use
et al., 2020; Bordelon & Pehlevan the MSR path integral to derive the generie form of ggenrraiization through variance'. and then we discuss in specific, analytically tractable cases of interest (e.g., linear models, lazy infinite-widie neural networks) how the details change and the role of each of the aforementioned factors. To keep our theoretical analysis tractable, we focus unconditional, non-latent models.
2 PRELIMINARIES
Data distribution. Let Paato (ao) denote a data distribution on R.D. We are especially but not exclusively interested in the case that Pdata consists of discrete set of 1 S M < examples (c.g., images), so that Paato (aata(Fro> = -M
M 5(250 - pmmm M,, where 8 is the Dirac delta function.
Forward/reverse diffusion. Training a diffusion model involves learning to convert samples from normal distribution Pnoisa (#:) = (wr; ; 0, S7) to samples from Pdata (3Fo) via processes
t. = -B,x, + Gin
t= 0 -
forward process, Pdota to Pnosse
a. -B, t)
t= T -
reverse process, Proise to Pdata
(2)
2

---

Published as a conference paper at ICLR 2025
Table 1: Popular forward processes in our parameterization. For these, G. := I and S = of
gr
Qx
end time
VP-SDE EDM
Bmin +
Bat
23
e
Sá By
de'
V1 - e So Bo de
1 T
0
21
b
where 7/: E & Gaussian white noise, G. E BRD is a nonnegative matrix that controls the noise amplitude, Dj == G.G, /2 2 is the corresponding diffusion tensor, Bt 2 0 controls decay to the ori- gin, e <>>>>>> a time cutoff that helps ensure numerical stability, and (xx t) == V a log p(æ: t) the score function. We allow G2 to be matrix so we can study how asymmetries affect generalization later. The forward process' marginals are p(æ = J p(ar an 1) drg. The transition probabilities are p(æ æo, 1) Nax;; 04 St), where Or, :- e fo Br dt' and S, == So 2D di'.
The forward process assumed here fairly general, and includes popular choices like the VP-SDE (Song et al. 2021) and EDM formulation (Karras et al. 2022) (Table 1). This choice of reverse process is called the probability flow ODE (PF-ODE), and has been shown to have both practical (Song et al. 2021) and theoretical (Chen et al. 2023b) advantages. Since s(x, t) is required to run the reverse process but a priori unknown, "training" a model means approximating s(x, t).
Denoising score matching. Given P > independent samples from p(æ, æu t) (note: P is differ- ent than M. the number of points in discrete Pdata), one could use a mean- -quared-uuuroooooe objject
( = Er, {
Aee [0( t) - s(x, t)113
/ X 2 || t) - s s(x, t) 12 p(zlt)p(() dædt (3)
to learn a parameterized score estimator So (zx, t). Here, A > 0 positive weighting function and p(t) is time-sampling distribution. The DSM objective (Vincent 2011 Song & Ermon 2019)
Ae [10 5(x0 (; 21)
Ae
Ji (0 = Er, m
86 a| p(æ, æo, 1) dadwood (4)
where p(z, Zo. == p(zz xo. tapaata (z)) ((t),, is usually used instead. While the folklore justifying this choice is that the score function is not known, this is not true: both Jo and J1 are optimized when & equals the score of the training distribution (see Appendix (A). which known.
We will argue that the real difference between Jo and J1 is that J1 generalizes better, and that this is in part because the proxy score (xx, t; mo) := V x log p(x: z.. t) = S, (a,20 x) is used as the target instead of the true score. It is 'noisy' version of the true score (see Appendix B since
Ex, (xx, 1; æo)] = s(x,t)
Cx,(x, t) = Cov [5; (,,55, 3,] S + 0₂ logp(z](). (5)
Although the proxy score equal to the score of the training distribution in expectation, neural networks trained on J1 empirically learn different distribution and generalize better. We claim that this fact is closely related to the covariance structure of the proxy score. Two relevant observations about its form large where the log-likelihood (ogp(2//) ()) has substantial curvature. In the typical case, where Pdate consists of discrete set of M examples, regions of high curvature correspond to the location of training examples and the boundaries between them (Fig. 1; see Appendix ( for more discussion).
follows. First, it is large small times, since S. - 0 as [ - 0. Second, is
Generalization and inductive biases. In typical supervised learning setting, one trains a model on one set of data and tests it on another, and defines 'generalization error" as performance on the held-out data. Here, we are interested in a different type of problem: given model trained on samples from p(ær, No. l). to what extent does the learned distribution differ from Pdate, and what are the associated inductive biases? Of particular interest whether models do three things: (i interpolation (filling in gaps in the training data), (ii) extrapolation (extending patterns in the training data), and (iii) feature blending (generating samples which include both feature X and feature Y even when training examples only involve one of the two features).
In our setting, a subtle but important point is that there is generally no ground truth. For example, the smooth distribution that CIFAR- 10 or MNIST images are drawn from does not exist, except in a 'Platonic' sense; we are interested in the extent to which diffusion models learn a distribution plausibly like smoothed version of the training distribution.
3

---

Published as a conference paper at ICLR 2025
t: 0.05
0.1
0.2
0.5
0.75
0.05
0.1
0.2
0.5
0.75
line polygon
random L-shaped
Figure 1: Visualization of proxy score variance (tr(C)/[ur(C) + || for four example 2D data distributions. Each example data distribution is supported on small number of point masses (red dots). As changes (left: small right: large f), boundary regions at different scales are emphasized.
APPROACH: COMPUTING TYPICAL LEARNED DISTRIBUTIONS
The distribution < ()) learned by diffusion model depends the learned score éø nonlinearly through PF-ODE dynamics; importantly, we are less interested how well the score is estimated, and more interested in how estimation
impact q. The learned score be viewed variable, since depends on the samples (+)), Th /(0) ~ p(æ, To, 1) used during training. In order to theoretically understand how diffusion models generalize, expression for the 'typical' learned distribution by averaging q over sample realizations.
random
aim to obtain analytic
How do do the required averaging? One of our major contributions is to introduce theoretical approach for averaging ((xo) over variation due to s. Below, we describe our approach.
Writing PF-ODE dynamics in terms of path integral. How does average the result of an ODE given that, the case PF-ODE dynamics, there is generally no closed-form expression for the result? To address this issue, we use novel stochastic path integral representation of PF- ODE dynamics that makes the required average easy to do. If q( (T;; 0) denotes the distribution of PPFOOE outputs given a score estimator $0 ne,t t) and fixed noise seed æT.
exp ((oo|=r;:0 pr: = / D(p, D(p,)D(a)) exp {
ip: ( + B,0, + Diso(zx,t)) t)] dt
(6)
where the integral is over all possible paths from ØT Vo. (To avoid technical issues, we assume a particular time discretization in all calculations. See Appendix D) This type of path integral is time-reversed version the Martin-Siggia Rose (MSR) path integral (Martin et 1973).
Averaging over possible sample realizations. Because the argument of the exponential depends linearly on the score, the required ensemble average is now easy to do. Using [...] to denote
[g(0o FT)] f (Dpp. exp Mi
1
1 M2 +...}
Mi 1: ip: - + Brxr + DrSang (x,, di
M₂ = [."
J. pp. (a 1; ve, ('DD didi'
where Sang (2e, == (x), is the ensemble's average score estimator, and V(x,, t; we, f') D,Cove 6(x,,)) t), 8(xe, Dr measures ensemble variance. Assuming higher-order terms can be neglected and hence that the estimator distribution is approximately Gaussian- one can show (see Appendix D that sampling from [g(2o|=>)) is equivalent to integrating an (Ito-interpreted) SDE:
Proposition 3.1 (Effective SDE description of typical learned distribution). Sampling from [g(2,) (r) is equivalent to integrating the (Iro-interpreted) SDE
i. -3,,, Danang( (I,, {
t=T
(8)
with initial condition ær, where Saug (2), = [ 1)) and where the noise term E( 1) has mean zero and autocorrelation V V( xv, = D,Cove ((x,,,), s(xv,t'))
& is unbiased and M is finite, then the noise term solely responsible for the difference between true PF-ODE dynamics (which reproduces training examples) and model's 'typical' sampling

---

Published as a conference paper at ICLR 2025
dynamics- ie.. generalization occurs if and only if V + O. This makes characterizing V. which we call the V-kernel since it reflects ensemble variance, crucially important for understanding how diffusion models generalize. Our remaining theoretical work is to complete two tasks: first, to compute Sang and V for a few paradigmatic and theoretically tractable architectures; and second, to study how their precise forms affect [q(2)))
DIFFUSION MODELS THAT MEMORIZE TRAINING DATA STILL GENERALIZE
It is instructive first consider extreme do diffusion models generalize in the complete absence of any model-related inductive biases? Perhaps surprisingly. the answer is yes. In this section, we make this point using a toy model in which training and sampling are interleaved.
Suppose the PF-ODE is integrated backward in time from an initial point ET until / = using first-order Euler updates of size A. At each time step, suppose samples For p(azo ( () p(a (ata (edaa constructs the 'naive' score estimator $(x,, t) == s(z,, t)+ ( (((1, t; xoi) p(m,
s t)), and uses this estimator as the score for that update. Assume this process continues, with new samples drawn at each time step. Despite this approach using the proxy score directly (so that training data is 'memorized") ) one obtains a nontrivial V-kernel, and hence generalization:
Proposition 4.1 (Naive score estimator generalizes), Consider the result of integrating the PF-ODE from T to t using first-order Euler updates of the form (Eq. 2
Ft-1 T, At B,0,+ D. (s(022))
s(x t)l 111)
The ~ p(zolzz, t).
At
Then [g( is described by effectivv SDE (Eq. (x 1') == xD, D, o(t-t').
8 with Sang and V-kernel
(9)
See Appendix E for details. Notably, the effective SDE is noisier when the covariance C' of the proxy score is high, e.g., in boundary regions between training examples. Next, we will see that this also true for less trivial models, but that the proxy score's covariance interacts with feature-related biases in order to determine the SDE's overall noise term.
5 FEATURE-RELATED INDUCTIVE BIASES MODULATE GENERALIZATION
Model architecture is known to produce certain inductive biases, with spectral bias being a well- known example (Rahaman et al. 2019 Bordelon al., 2020; Canatar al. 202T) How do model- feature-related inductive biases affect the V-kernel? We answer this question below in two interest- ing but tractable cases: linear models, and (lazy regime) ffinite-width. neural networks.
5.1 THE V-KERNEL OF EXPRESSIVE LINEAR MODELS
In what follows, we may write = (x, to case notation. Consider linear score estimator
(0 t) = wp + (),
(10)
where the F feature maps Φb (oi.-.,,rr) Φr) are linearly independent, smooth functions from R x [) to R that are -quare-integrabee with respect to the measure App(æ, t). The parameters to be estimated are 0 (wo, W). with wo E R and W E DxF Note that this estimator is linear in its features, but not necessarily in or t. The weights that optimize Eq. 4 are (see Appendix F
W - J'
wi J E (0) + (8)
(11)
where we define (...):= ;= En .e ... 1/E, [N:] and matrices
J ((0(#, ( (5 t:) (5) Σs ((0(1, (x)) (xx,t) (
When averaged over De sample realizations, the estimator 8., w6 W (axx t) is unbiased as long as the set of feature maps is sufficiently expressive. Interestingly, this is true regardless the or samples used, provided F < P. The following result characterizes (4(=)) for linear models:
5

---

Published as a conference paper at ICLR 2025
Proposition 5.1 (Expressive linear models asymptotically generalize). Suppose the parameters of an expressive linear score estimator (Eq. [70) with F features are perfectly optimized according to the DSM objective (Eq. 4 using P independent samples from p(x, t). Define the feature kernel
kzzz²) VF
(0)')) -
(12)
Let := F/P. Provided that the limit exists and is finite, in the P + oo limit, we have =
{
Xz. E.AA, (x("" Dr
V(z;z') lim De Eg" P0000
(13)
Note that if the number of features F does not scale with P, V 0. See Appendix For the details of our argument. The V-kernel for linear models differs from the naive score's V-kernel (Eq. D via the presence of feature-related factors-in particular, the effective SDE is noisier where features take atypical values. One expects that these factors can either enhance or compete with noise due to the covariance structure (e.g., noise is higher if features take atypical values in boundary regions).
5.2
THE V-KERNEL OF LAZY INFINITE- WIDNTTE-IDTTII NEURAL NETWORKS
Neural networks in the neural tangent kernel (NTK) regime (Jacot et al. 2018 Bietti & Mairal provide another interesting but tractable model. Such networks exhibit 'lazy learning (Chizat 2019) in the sense that weights do not move much from their initial values. Moreover, it is known that they interpolate training data in the absence of parameter regularization or early stopping (Bordelon al.] 2020). If they precisely interpolated their samples, we would expect to recover V-kernel like the one we computed in Sec. more generally, we expect something similar modified by the spectral inductive biases associated with the architecture (Canatar et al. 202Tp.
2019)
al.
For simplicity, we consider fully-comnected networks whose hidden layers all have width N, which taken infinity together with P (see Appendix forr details). The associated NTK has Mercer decomposition with respect to the measure >.p(ar a. (),),, so K can be written in terms of F orthonormal features {
Σ Se t)ox((t))
E[A du (x. t)oe(x, t) p(x, t) dxdt dge (14)
We assume training involves full-batch gradient descent on P samples from p(æ, 2o. f), so that the learned score function after training for amount of "time" has the closed-form solution
8(z) (z) + S -500 (1 e ArkK// K-1(())
where &p the network's initial output, 5 E DxP contains proxy score samples, S. E DxP contains the network's initial outputs given the samples, K PxP is the kernel Gram matrix, Ar E IR PxP is a diagonal matrix containing the weighting function > >r/E[A,] evaluated on samples, and (<x,,) 1) is an input-dependent vector whose i-th component is K ((x(()) (()))))((()aa)) t). We have:
,(0);,,,))
Proposition 5.2 (Lazy neural networks asymptotically generalize). Suppose the parameters of fully-commected, widhh neural network characterized by rank F NTK are optimized ac- cording to the DSM objective (Éq. 4 using P independent samples from p(æ, xo, t) via full-batch gradient descent for training 'time" Define the feature kernel
J /F -
x'2;z')= =
(15)
Let K = F/P. Provided that the limit exists and is finite, in the P + ao limit, we have
X².
V(z;z') lim RD, Eg" P-000
k(z: ((x:"")))""
Dv
(16)
E[A.}²
In the infinite training time limit, we recover the Sec. 4 result with a prefactor x s(Az) const.:
v(z;²')= =<())))(())))))
(17)

---

Published as a conference paper at ICLR 2025
increasing F/ P
true dist. PF-ODE avg. learned dist.
1/4
1/2
0.01
probs density
0.001
increasing
0.0001
0
Figure 2: Average learned distribution (N- 100) for linear model with Gaussian features trained on different sample draws from a ID data distribution {-1, 0, 1}. Red: average learned distribution; black: true distribution; gray: PF-ODE approximation of true distribution. Different values of the time cutoff and ratio F/ P are shown. Note that there is more generalization as both become larger.
See Appendix G for the full details our argument. Interestingly, although the network is not assumed to be in the feature-learning regime, this result interpolates between our pure memorization (Prop. 4. and linear model (Prop. 5.1) results as we change the value of the training time т. The relted inductive biases that appear precisely the well-known spectral biases.
See Appendices H and [ for discussion of how to obtain analogous results for diffussin models with slightly different training objectives, like those for which 'denoiser' rather than a score approxi- mator is learned (see, e.g.. Karras et al. (2022)).
6 GENERALIZATION THROUGH VARIANCE: CONSEQUENCES AND EXAMPLES
In this section, we briefly discuss salient consequences of generalization through variance.
Benign properties of generalization through variance. In what sense might generalization through variance provide 'reasonable' inductive bias? Its key driver is the proxy score covari- ance, which is large primarily in boundary regions between training examples (Appendix [C. and this fact greatly constrains the way this type of generalization point (so that distribution is primarily supported variance tends to be nontrivial only along that manifold, and hence generalization through variance preserves the dimensionality of the data manifold.
occur. A data set with one data 1) is not generalized, since the proxy score covariance is trivially zero. If the data
some oow-dimensional 'data manifold', the proxy score
Very far from training examples, the proxy score covariance approximately zero, so there is no generalization through variance. Finally, the effective PF-ODE both follows deterministic PF-ODE dynamics on average, since the V-kernel-related noise term has mean zero, and is also most likely follow deterministic PF-ODE dynamics, since the probability of paths that deviate from can be shown to be somewhat lower. This means that, although effective PF-ODE dynamics differ from the deterministic PF-ODE's s dynamics, they do not substantially differ. meaning that regions near training data will still tend to be sampled most. See Oppendix]]oo more details and discussion.
Memorization and the V-kernel the small noise limit. Our characterization of the typical leamned distribution [g] in terms of stochastic process is somewhat unsatisfying, in part because it remains unclear how the V-kernll affects the way [q] generalizes Pdata- One can make some progress on the issue by making small noise approximation, which is valid (for example) when models are somewhat underparameterizee. so that x = P/P P is somewhat less than 1. When the effective PF- ODE's noise term is sufficiently small, one invoke semiclassical approximation of the relevant path integral. We find (Appendix(K) K) that, at least this limit,
[g())) p(æolee
(18)
V det
SP (2no (mro) Berg
7

---

Published as a conference paper at ICLR 2025
where Set quantifies the (negative log-) likelihood of the most likely path that goes from noise seed 2r to a sample Zy- In words: [g] equals the (e-noise-corrupie data distribution, times a curvature factor that quantifies the likelihood of small deviations from deterministic PF-ODE dynamics. It through the V-kernel's influence on this curvature term that it affects generalization (although unfortunately it appears difficult to be more explicit about how it does so, at least analytically).
Gap-filling inductive bias. Given that the V- kernel is especially sensitive the 'gaps' be- tween training examples, one expects that gen- eralization through variance works by effec- tively filling in these gaps. This appears to be often, but not always, true. First, in its naive form (see, e.g., Sec. F generalization through variance can actually reduce the probability as- sociated with boundary regions, since the addi- tional noise those regions makes the dynam- ics spend less time in them. If there is nontrivial temporal generalization, for example via time- dependent features Φ the V-kernel may have a nontrivial temporal autocorrelation structure; we speculate that these autocorrelations may be a key mechanism that allows the dynamics to spend more time in boundary regions.
00
22.5
450
Gaussian features
Fourier features
Figure 3: Generalization of 2D data distribu- tion depends on features used and data orientation. Heatmaps of samples from N 100 linear mod- els are shown in different conditions, with train- ing data (red dots) overlaid. Notice that which gaps are 'filled in', e.g,, whether a square shape or cross shape made, depends on both factors.
Second, the details of generalization are strongly modulated by two numbers: the time cutoff e, and the ratio F/P that determines the extent to which a model over- or underpa- rameterized. Fig. 2 depicts an illustrative one-
dimensional example where there are three training examples {-1, 0, 1}, and where the model is linear (see Prop. 5.1 with Gaussian features centered at different values of and t, each with the same width. The average learned distribution (red) tends to differ from both the true distribution (black) and its PF-ODE approximation (gray) in the size of peaks near training data, and in the gions between training data. These differences are larger when F/P and c are larger. Taking both large produces the largest difference, but not obviously the 'best' generalization of training data.
Feature-noise alignment affects generalization. Different feature sets interact with the structure of the proxy score covariance differently, and hence produce different kinds of generalization. Fig. 3 shows how the same 2D data distribution (four examples, which together determine the vertices of a square) is generalized differently depending on its orientation, and depending on which linear model feature set (here, either Gaussian or Fourier features) is used.
DISCUSSION
We used a novel path-integral approach to quantitatively characterize the 'typical' distribution learned by diffusion models, and find that generalization is influenced by combination of factors related to training (the DSM objective and forward process; Sec. 2 and 4). sampling (the learned distribution depends nonlinearly on score estimates; Sec. 3 model architecture (Sec. 5) and the
data distribution. Below, we use our theory to comment on various previous observations.
DSM produces noisy estimators, but stable distributions. Various forms of score 'mislearning' are well-known. At small times, scores are hard to learn due to the noisiness of the proxy score target, leading authors like Karras et al. (2022) to suggest a p(t) that emphasizes intermediate noise scales. Chao et al. (2022) discuss how score estimation enrors affect conditional scores. Xuu et al. (2023) explicitly study the mariance-near-me boundarissss issue we discussed, and propose a strategy for mitigating it. On the other hand, it is well-known that despite noisy score estimates, diffusion models generally produce smooth output distributions (see, e.g., Luzi et al. (2024).. Moreover, two diffusion models trained on non-overlgp subsets of data set are often highly similar (Kadkhodaie al.) 2024). These facts are due to noisy score estimates contributing to sample generation through the PF-ODE, which effectively 'averages' over estimator noise. Our theory is consistent with these
8

---

Published as a conference paper at ICLR 2025
observations: even the interleaved training-samme sampllig procedure discussed in Sec. produces well- behaved, smooth distribution.
DSM produces a boundary-smearing inductive bias. This has been previously pointed out by authors like Xu et al. (2023). Where we differ from previous authors is in considering this issue a potential strength. Integrating the PF-ODE using the true score reproduces training examples, so is in some sense beneficial to 'mislearn' the score. This particular kind of mislearning is useful for several ways of generalizing point clouds, including interpolation, extrapolation, and feature blending. Moreover, producing this inductive bias is an interesting way diffusion models differ from something like kernel density estimation: boundary regions across different noise scales are smeared out, with different scales linked via PF-ODE dynamics, which may provide better generalization than convolving the training distribution with any single kernel.
Architecture-related inductive biases play a role. As we showed in Sec. feature/archiiecture- related inductive biases interact with DSM's boundary-smearing bias in order to determine how diffusion models generalize. This appears to be consistent, for example, with the Kadkhodaie et al. (202)) finding that diffusion models effectively exhibit 'adaptive geometric harmonic priors'; their hinding is specifically in the context of score estimation using a convolutional neural network (CNN) architecture. It is plausible that this choice encourages a harmonic inductive bias, since CNNs more generally exhibit inductive biases related to translation equivariance (Cohen & Welling 2016).
Generalization through variance harmful and helpful. It is important to note that this kind of generalization not always helpful. trivial example is that unconditional models trained on MNIST digit images tend to learn to produce non-digits output in the absence of label information (see, c.g., Bortoootoii et al. (202I). ). More generally, blending modes may or may not be desirable, since it can produce (e.g.) images very qualitatively different from those of the training distribution.
Other forms of generalization are possible. Factors we did not study, like learning dynamics, most likely also partly determine how diffusion models generalize. For example, the use of stochas- tic gradient descent introduces additional randomness that disfavors converging on sharp local op- tima (Smith & Le 2018, (mmth al., 2020). It would be interesting to utilize recent theoretical tools (Bordelon & Pehlevan 2023) to characterize how learning dynamics impacts generalization, especially in the rich (Geiger et al. 2020 Woodworth et al., 2020) rather than lazy learning regime.
Comment on memorization. Determining whether diffusion models memorize data (Somepalli jet al. 2003a Carlini et al. 2023). and if so how to address the issue (Vyas et al. [003). has become significant technical and societal issue. Our theory suggests that since generalization through variance happens primarily in boundary regions, diffusion models are unlikely to substantially gen- eralize outliers. Since conditional models involve distributions of much higher effective dimension, one may expect that more training examples are 'outlier-like" and hence memorization should hap- pen more often; this is consistent with the observations of [Somepalli also suggests why duplications increase memorization: the existence of a strong boundary between modes, which requires modes to have comparable probability mass, degraded.
al. (2023b). Our theory
Limitations of theoretical approach. Our theory is simplified in at least two ways. First, only a simple formulation of training (via DSM) and sampling (via the PF-ODE) from diffusion models is considered. There exist alternatives to DSM, like sliced score matching (Song al. 2202)) and alternative ways of sampling, including using auxiliary momentum-like variables (Dockhorn et al.) 20226). Also, our theoretical analysis neglects variation due to numerical integration schemes, even though these may matter in practice (Liu et al. 2022. Karras et al. 2022 Dockhorn et al. 2022a).
Second, we study only unconditional models for simplicity. This means that in particular do not consider diffusion coupled to attention layers, which enables the (ext-conditionig behind many of the most striking crffusionnmodel-relltte successes (Rombach et al. 2022. Blattmann et al.) 2023).
Finally, we do not consider realistic architectures (like U-nets) and rich learning dynamics due to theoretical tractability. However, these challenges are not unique to the current setting. Despite our contribution's 's simplicity, we hope that it nonetheless provides a foundation for others to more rigorously understand the inductive biases and generalization capabilities of diffusion models.

---

Published as a conference paper at ICLR 2025
REFERENCES
Sanjeev Arora, Simon S Du. Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ru- osong Wang. On exact computation with an infinitely wide neural net. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garett (eds.), Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL httpp::/pproceeedngg..s www.wneeriipc. cc/ wcppaper.ppp..e..ppppe fi files/paper/2019/
fi http:///o66.tttt22222eeeeeeeeee0
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https:://proceedings..c.e www.ips..eeuuiu.ccmm.c cc/papr files/paper/2019/ Fi File/co # 9c3 04449993993330311133333333333333
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhom, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patfern Recognition (CVPR), pp. 22563-22575, June 2023.
Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of ker- nel evolution wide neural networks. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.). Advances in Neural Information Process-
ing Systems. volume 35, pp. 12240-32256. Curran Associates, Inc., 2022. URL https::/prrceediin nwwr cc http://cccccpppepe.eee///p/rr//a d02
file/
Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution in wide neural networks*. Journal of Statistical Mechanics: Theory and Experiment, 2023(11): 114009, nov 2023. doi: 10.10888/742--4668ada0 1088/1742 48/ //44/////0bbb
URL https: //cps::/dxd.. doi iooi..ce wwt.ddoooo..o........
Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In Hal Daumé III and Aarti Singh (eds.), Pro- ceedings of the 37th International Conference on Machine Learning, volume 119 of Proceed- ings of Machine Learning Research, pp. 1024-1034. PMLR, 13-18 Jul 2020. URL https: html
//proceedings ml wttps/e11l/ooeeeeeeeeeeee.
Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothe- sis. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https: Expert Certification.
//openreview.nee
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL bttps /ttp/ttpss/////eee//eeeeeee http://wwwwee.eeeeeeeooooecceaaa
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature Commu- nications, 12(1))2914, May 2021. ISSN 2721-1723 doi: 1111008/441477-022223100 https://dof org / 10. 1038 444444467722 0000070-2211111000 23103-1
URL
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253-5270, Anaheim, CA, August
2023. USENIX Association. ISBN 978-1-9331133-7-33 URL https:/ /ttp:://wwwwwww.. www. wswnix.ei...... orq/
Chen-Hao Chao, Wei-Fang Sun, Wun Cheng. Yi-Chen Lo, Chia-Che Chang, Yu-Lun Liu, Yu-
Lin Chang, Chia-Ping Chen, and Chun Yi Lee. Denoising likelihood score matching for con- ditional score-based data generation. In International Conference on Learning Representations, wwt.://wwwweeeeeeeeeeeeeeeeeeeee wtwp:eeeteoorrm.tetttemmmmm d= -FF..FFFFFFFFCC
2022.
10

---

Published as a conference paper at ICLR 2025
Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In Andreas Krause, Emma Brunskill. Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 4672-4712. PMLR, 23-29 Jul 2023a. URL wwweeeeiingg.......pp.... wtt.:mm.....sss..... htt.0///hhe2...ccccccm
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations.
Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- -isa-Bianchi, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL WtLLittps//pppoeeeiiiiii w http://aaarrpuuuuusessoopoppppre cc/paper files/ Pap.saapppppp...
Sitan Chen, Sinho Chewi, Holden Lee, Yuanzhi Li, Jianfeng Lu, and Adil Salim. The probability flow ode is provably fast. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Process- ing Systems, volume 36, pp. 68552-68575. Curran Associates, Inc., 2023b. URL
d8 oa4.7776665cc222200003335------0
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference Learning Representations, 2023c. URL BeLLhttpss//openzeeeet /oppnreview..... net/ ftre/forum?iii id http:oormmmdd-22yyyyy222222 ZyLVMgs20U
Lénaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable pro- gramming. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019. URL https: ptttps://ppoceeediing
wwwprrppppcccpppppreeeee files/ pdf.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 2990-2999, New York, New York, USA, 20-22 Jun 2016. PMLR. URL
wtw.aiiiiiinn.mm...s.....ee press/
v48 V448/chhenc66.. html
A. Crisanti and H. Sompolinsky. Path integral approach random neural networks. Phys. Rev. E, 98:062120, Dec 2018. doi: (113//hysReEE..8.662... URL httph:tt://// https://iing /tt.///inkkk//// aps org/ doi/10 1wt.11103/PPyyeee.e 98.98888 062120
Salman U1 Hassan Dar, Arman Ghanaat, Jannik Kahmann, Isabelle Ayx, Theano Papavassiliu, Ste- fan O. Schoenberg, and Sandy Engelhardt. Investigating data memorization in 3d latent diffusion models for medical image synthesis. In Deep Generative Models: Third MICCAI Workshop, DGM4MICCAI 2023, Held in Conjunction with MICCAI 2023, Vancouver, BC, Canada, Octo- ber 8, 2023, Proceedings, pp. 55-65, Berlin, Heidelberg, 2024. Springer- Verlag. ISBN 978- 3-031-53766-1 doi: 978 978-3- 378-3-031-53777-
URL https:://ci.orgggeeee 1007/
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. GENIE: Higher-order denoising diffusion solvers. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022a. URL http://pppeevvvieee ew net / f forum?
ld LKEYuYNOqx
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically- damped langevin diffusion. In International Conference on Learning Representations, 2022b.
URL http:tttss///ppeeee///
net hww.://eeeeeooommooooooeoeo
Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020 113011 nov 2020. doi: 10.1088/1742 10.108//774.5546bbbbbee URL https: // //dx. doi. orq/1C.
1088/ 7088/1742-5468/220
11

---

Published as a conference paper at ICLR 2025
Yinbin Han, Meisam Razaviyayn, and Renyuan Xu. Neural network- -based score estimation in diffu- sion models: Optimization and generalization. In The Tweifth International Conference on Learn- ing Representations, 2024. httpthttss///oeeeeeeeeee
net/
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: https::/ htttps://pproceeddiggs wwwr:iis.....eurrs............ ccpppe 4be
Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representations. In The Twelfth Inter-
national Conference on Learning Representations, 2024. RRL[tttppp:/openrviieee nett
Mason Kamb and Surya Ganguli. An analytic theory of creativity in convolutional diffusion models. arXiv e-prints, art. arXiv:2412.20292, December 2024, doi: 10.48550/arXX.2211222229
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
https://openrevit wtwp:rreiiieeeee/fooruuiooo wtte:/forummdd=xkPPcst
Tero Karras, Mila Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyz- ing and improving the training dynamics of diffusion models, 2024.
Hagen Kleinert. Path integrals in quantum mechanics, statisties, polymer physics, and financial markets. World Scientific Publishing Company, 2006.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv: 09778, 2022.
Lorenzo Luzi, Paul M Mayer, Josue Casco-Rodriguse Ali Siahkoohi, and Richard Baraniuk.
Boomerang: Local sampling on image manifolds using diffusion models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL httpps://eppmrevieww.nee.
forum? N-NYdTkk-NWW
P. C. Martin, E. D. Siggia, and H. A. Rose. Statistical dynamics of classical systems. Phys. Rew A, 8:423 437, Jul 1973. doi: 10. 101103/PhysRevA...422 URL htttt:t/t///ikk aps /oo/
10. 1..1103/PhyReeA. 8. 423
Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborová. Dynam- ical mean-field theory for stochastic gradient descent in gaussian mixture classification. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 9550- Curran Associates, Inc., 2020. URL httppss//pprccceedngg. neuri cc hawpapper_fllesspppp////000 Fil flle/cclceccccdbb8880556oood3333 Pappeaa.ppppppppppppp
Matthew Niedoba, Berend Zwartsenberg, Kevin Murphy, and Frank Wood. Towards mechanistic explanation of diffusion model generalization. arXiv e-prints, art. arXiv:2411.193333 February
2025. URL htt.t:ttp:://tt www.awwwwwi.....o..oaooa.aa abs 241 19339
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5301-5310. PMLR, 09-15 Jun 2019. URL lttpt::///// http:///pccceeeeeeeeeeee /ttp:////tts///ooeeeisssg
mll. www...press//ee v hat//aaaaaaaaaaaa 19ann99... html
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Patlern Recognition, pp. 10684-10695, 2022.
12

---

Published as a conference paper at ICLR 2025
Haozhe Shan and Blake Bordelon. A theory of neural tangent kernel alignment and its influence training, 2022. URL TtRLhttp::// hattts://arxiv..oo org/ abs/2105 14301
Samuel Smith, Erich Elsen, and Soham De. On the generalization benefit of noise in stochastic gradient descent. In Hal Daumé and Aarti Singh (, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp- 9058-9061 PMLR, 13-18 Jul 2020. URL httpt::tp///........ btttps://proceedl: ngg.mlrs......................m. ml http./........//.......... Sm
press/vis
html
Samuel L. Smith and Quoe V. Le. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018. URL https://
openreview n iidimiiddyyyyy
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Pro- ceedings of the 32nd International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 2256-2265, Lille, France, 07-09 Jul 2015. PMLR. URL htt.a///smmmm.pppps///////////// http///hhhliiistteiinnin.n..s html
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 6048-6058 June 2023a.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Under- standing and mitigating copying in diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL https: https:://ppeneettt /ttp//ppeevvieeeeeeeeeeeee/eeeee net/ forum? d=
HtMXRGbUME
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distri- bution. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL 1ate5//000C70111119aa4444411aa Paper. pdf
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Ryan Adams and Vibhav Gogate (eds.), Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pp. 574-584. PMLR, 22-25 Jul 2020. URL https: //tps///ccccee..... mlr. mlr. wtt.peess/v11//ongg22222
html
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021. URL.
net.
Forum?id-PxT1112R22
George Stein, Jesse C. Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Leigh Ross, Valentin Vil- lecroze, Zhaoyan Liu, Anthony L. Caterini. Eric Taylor, and Gabriel Loaiza-Ganem. Expos- ing flaws of generative model evaluation metrics and their unfair treatment of diffusion mod- els. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https::/cpperrviiie net. wtw.leeww.e///toreeeeeee feee:/forummidoe http:/orrummi10000fff710o1oo
Pascal Vincent. connection between score matching and denoising autoencoders. Neural Compu- 2011. doi: 10.1 10.112//EEO...00014
tation,
Nikhil Vyas, Sham M. Kakade, and Boaz Barak. On provable copyright protection for genera- tive models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. v202 wttp://aa23bb.. html
PMLR, 2023. URL ht https: httss://pocceeeiiiioo mlrr. htt.p//gsgm......ssseeee press/
13

---

Published as a conference paper at ICLR 2025
Binxu Wang and John Vastola. The unreasonable effectiveness of gaussian score approximation for diffusion models and its applications. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Binxu Wang and John J. Vastola. Diffusion models generate images like painters: an analytical theory of outline first, details later. arXiv e-prints, art. arxiv:2303.02490, March 2023. URL
https:/ //riiv.//rxvvv..//a org/ wwtg//a//////// /2033. 02490
Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Jacob Abernethy and Shivani Agarwal (eds.), Proceedings Thirty Third Conference on Learning The- ory, volume 125 of Proceedings of Machine Learning Research, pp. Jul 2020. htts::::::
PMLR, 09-12 html
Yilun Xu, Shangyuan Tong, and Tommi S. Jaakkola. Stable target field for reduced variance score estimation in diffusion models. In The Eleventh International Conference Learning Represen- tations, 2023. BtLhttt:::// /ttp::///ppeeeeeeeeeeeee net wow.:oooomndiwwwwiooo
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: comprehensive survey of methods and applications. ACM Comput. Surv., 56(4), nov 2023. ISSN 0360-0300. doi: 10.1145/3626355.
URL
TaeHo Yoon, Joo Young Choi, Sehyun Kwon, and Ernest K. Ryu. Diffusion probabilistic models generalize when they fail memorize. In ICML 2023 Workshop Structured Probabilistic Inference & Generalive Modeling, 2023. URL httph:ht///oo htttps://openreveew net forum?? 2id=
sheiCbSk9h
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentations, 2017. URL httt:hhts///ppott //tpr://////errreeeweeee net/ htte//ffummmicooocc (
Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Peng Wang, Liyue Shen, and Qing Qu The emergence of reproducibility and consistency in diffusion models. In Forty-first International Conference on Machine Learning, 2024. URL https::/ /ttt:::://ppnreview net/ forum?id=
Hs Hsli0qZke0
14

---

Published as a conference paper at ICLR 2025
Appendix
See
for code that produces Fig. 1-3.
Table of Contents
Optimizing objective reproduces training distribution
16
Denoising score matching preserves optima of naive objective
16
A.2
Training distrhution reproduction)
17
Covariance proxy score
Computing covariance of proxy score
2
Connection Fisher information
18
B.3 Explicit covariance for isotropic Gaussian mixture training distribution
18
C
Boundary regions: definition and Bayesian interpretation
20
Path-integral representation learned distribution
21
Warm-up: Deriving path-integral representation the PF-ODE
21
D.2 Deriving path-integral representation of more general process
22
D33 Averaging learned distribution over sample realizations
23
E Naive score estimators generalize: details
24
F
Linear score estimator: details
25
Definition of linear score mode|
25
F2
Optimum of DSM objective for linear score model
25
Optimum DSM objective given a finite number of samples
..........
26
F.4
Linear model estimator unbiased]
27
F55
Computing the V-kernll of the linear score model
28
G
Neural network score estimator in NTK regime: details
29
Detiniion of neural network model
................................
29
G.2
Learned score after full batch gradient descent
................................
29
Computing the V-kemel of the NTK model
31
H
Results for noise prediction formulation
33
Results for denoiser formulation
34
Benign properties of generalization through variance
35
J.1
Single points are not generalized]
35
J.2
Dimensionality of data distribution preserved
35
J.3
Variance is added far from data distribution examples
............
35
J.4
Following average field is most likely
36
J.5
Training data are more likely to be sampled when noise is small
36
Memorization and the V-kernel in the small noise limit
37
Setting the semiclassical approximation
37
Semiclassical approximation the learned distribution
38
K.3
Quantifying memorization semiclassical regime
39
15

---

Published as a conference paper at ICLR 2025
A OPTIMIZING OBJECTIVE REPRODUCES TRAINING DISTRIBUTION
In this appendix, we characterize the optima of the naive and DSM objectives introduced in Sec. 2 and in particular show that one (naively) theoretically expects diffusion models to reproduce the training distribution in the absence of expressivity-relate constraints.
A.1 DENOISING SCORE MATCHING PRESERVES OPTIMA OF NAIVE OBJECTIVE
First, we reestablish the well-known fact that the optima of the naive objective
/
X
(0) Er {eelso(< s(x, +)112
(z. s(x,t))l t)ll p(z)t)p(() dadt
(19)
and DSM objective
(1) = Br Bi, (2el(5((2,1)) a(x f; 2
(20)
de | s(x, t: 2 p( æo,
drdwodt
2
the same Vincent 2011 Song & Ermon 2019
Song et al. 2021 Assume that æ, 2gg
R
and that E The gradient of Jo with respect to is
OJo
/ X (s, t)
(.0( () s(x, ()] p(ar ()(()) dadl
(21)
00
00
where 0s (x, t)/00 is the D x Jacobian matrix of the score estimator. The gradient of J1 is
OJ1 - / /
(x. t)²
(., t) $(x, æo)) p(æ zo. t))ta (zo))(() dæddyddt.
(22)
00
At this point, we make two observations about the gradient of J1. First, the term the left does not depend on æo. so we can marginalize over Øo. Explicitly.
f A1
0sg (x. (x,
00 se (x, t) p(z|zy (xo))() dodxodt f Ax 0āo(x,t)) so(x t) p(æ t)p(t) dxdt.
00
Second, the term on the right only depends on Bg through the proxy score target. Moreover,
f
5( [x, t; wo) p(x|200 Pdaata(())) dæ
Va log (xp t) p(a) æo, t))data (xo)) dæy
V xxp(a (00, t)paara ( (xo) dæo
(23)
V / p(æ|oo t)patta (xt)) dwo
Vxp(x(t)
s(x, t))(xx(t).
Hence, the gradient of Ju is the same as the gradient of J1. so they have the same optima. If the score approximator is arbitrarily expressive and smooth in parameters, we particular have that the true score (a global minimum of Jn) is an optimum of the DSM objective.
This optimum also the global minimum of Jj. Note that J1 can written as
Dee | 1) - s(x, 1) + s(x, 1) 8(x,, 1; 2;)
E2,00,00
A (1 (x, t) - s(x, 1) + ( t) - s(x, t)] [s(x - 5(x, t; vo)] [| 8(x, 6; 200)131)
Er x
The first term is precisely equal to Jg. The second term vanishes, since (as shown by Eq. 23
E ((x t; xo)] = s(x, t).
(24)
Hence, we have that
J1 Jo Be, {he ir( Covaola.1 (8) In words: J, is equal to Jn up to a o-independent term that is weighted combination of proxy score variances.
(25)
16

---

Published as a conference paper at ICLR 2025
A.2 TRAINING DISTRIBUTION REPRODUCTION
In practice, the training set consists of 1 S M < ao examples (e.g., images) which together define
M
Pdota (xo)
Σ
5
pm)
(26)
M m=
The corresponding 'corrupted" distribution, given our choice of forward process (see Sec. 2 is
M Σ m=1
p(zlt)
N(x;
a:p4: St)
(27)
M
Usually, model updates utilize batches of samples from p(a, æo t) (Song et al. 2021 Karras et al. 2022). As training proceeds, the model sees an ever larger number of samples from this distribu- tion, making the empirical objective
P Σ A(( 1000 (2x(1) ((0)) ((x) (()))a))))) (m)) )
J1 P)
(28)
where the superscripts index different (independent) samples from p(x; vo. t) p(æ] Papana (tt). For P sufficiently large, by the central limit theorem, we expect the em- pirical objective extremely close to the true objective, and hence share its global minimum. But the global minimum is the true score, i.e.,
Nax St) Σ S. (ar x) Em N(x; Ore Oa Si) m=1
s(x,
Since integrating the PF-ODE using this score produces samples from Pdata (Zn)-as as t O, S, + Op, so the asymptotic 'force' pushing 2; towards an example becomes infinitely strong- we expect expressive diffusion models trained the DSM objective using large number of samples to reproduce training examples.
17

---

Published as a conference paper at ICLR 2025
B COVARIANCE OF PROXY SCORE
In this appendix, we compute the covariance of the proxy score (( = V log zo. t) with respect p(æo) x. t). We also show how this covariance is connected to Fisher information, and explicitly compute the case that Patta is an isotropic Gaussian mixture.
B.1
COMPUTING COVARIANCE OF PROXY SCORE
Note that
0²
p(æ (00
[-S
8t8g p(ar/ar0 1)
(29)
Or,
Using this fact, we can write
p(a wo. t)pdata (200
Cov (6,,, Kj) =
865g
dxo 8185
(()
0
+
p( æo, t) (xo) dæg - 8,88
p(=((t ()
dx,0x, 1,30
f p(a](t)
0²
p(ac aoo t) t) (2:0) dxo - 8,85 (30)
1,39 +
p(a=(t)
Our,O.r
p(x((1)
pp((tt)
Stjj + p(a=(t)
845,
dx,0xi
8²
S Or,0n logp(0>11
B.2 CONNECTION TO FISHER INFORMATION
By definition, if p( x. 1) is viewed as a distribution with parameter vector æ, and è is viewed as hyperparameter, the Fisher information Ip defined as
Ô log p< t) Ô log ((xol(0 t) (10 dæt
[=((())
dr.
/
a log p(æ Zo. t) Dx;
a Olog
① log p(2 t) Oxz
alog p(a))) Oz,
p( xx t)
das
(31)
[ si) [8 sj] p(zcolæ, t) dao Co 3z)
B.3 EXPLICIT COVARIANCE FOR ISOTROPIC GAUSSIAN MIXTURE TRAINING DISTRIBUTION
Suppose that p(æo) and p(=]() are
Pdata
1 M
Σ Ni pm I)
p(alt) 1 N Waxaaammm arpm ajo Si)
M
(32)
Note that the delta mixture case is an example (o = 0). Define the softmax distribution
N(x; axp of
S.)
p(m| t) Σm Ni amplmm ajo I S.)
(33)
on M (1, [I., M). This distribution, whose moments determine the proxy score covariance, has a Bayesian interpretation: it corresponds to an ideal observer's 's belief about the outcome Tp. given that said observer is in state æ time t.
The first and second derivatives of p(zlt) can be written in terms of expectations with respect to this distribution, since
0p(zlt)
Σ (afo8/ S.) (aapum x)p(mlee t) - - x)
p(x](()
dr
m
18

---

Published as a conference paper at ICLR 2025
and the Hessian matrix (Hız (()p()))) is
H (1 -ar -))0a)aammmmm00000000 S.) p(m|2,
p(al()
-aaio Si) (.)o'+ (000001++0000000000))0)+)+++01)0 -( - x)² (a/00 + S.)
Then we have
8 log p(z)t)
- + a - + S.) (ov S.) (34)
Oz, 0z,0z,
and hence that
S.) Co = -6 (a.o + (o S.)
For delta mixture training distribution, since og = 0, the covariance simplifies to
Cov S (ov,
(35)
The above equation implies that the covariance the proxy score is, up to scaling, the same as uncertainty about given and t.
19

---

Published as a conference paper at ICLR 2025
C BOUNDARY REGIONS: DEFINITION AND BAYESIAN INTERPRETATION
key concept used throughout this paper is that of a boundary region, which we informally define as a region of IR between two or more training examples in the case that Pdata is discrete. Intuitively, these regions correspond to the "gaps' in the training distribution, and reasonable generalization strategy fill them in.
In this appendix, we briefly comment that the notion of a boundary region can be made more precise via Bayes' theorem: for given reverse diffusion time t, boundary regions are sets of values for which uncertainty about the endpoint zo is particularly high. For discrete data distributions, such regions coincide with sets of states between training examples, since one maximally uncertain about the endpoint when æ is equidistant from two or more training examples.
An illustrative dieediiensional example involves two training examples at xn =u. The noise- corrupted data distribution at time l. is
1 p(=lt) = 2 Ni; (x; 1 N(x (aix;;a
(36)
so the posterior endpoint estimate given z, is the softmax distribution (see also Appendix B
plz| zo: Pdtt (zo)
p(zoo/a, Eno p(z t) (zo)
(2; asp,,
N(:; arp, oz)
N(1 of) N(;;; -4);,0 of) 5(20 11) N(r;ooy,, N Nix; oulk, ot)
5(20 11)
The mean E[ lr, E of this distribution
E E|zo xr, 1] = Σ p(rolar t) = μ tanh
asff T
(37)
The interpretation of this quantity is interesting in light of the 'Bayesian guessing game' metaphor for score learning (see, c.g., Kamb & Ganguli (2024)) One imagines that one starts at an unknown To (here, either +4 -p),, and then noise added according to the forward process until time t. Given that an observer is state x at time f, what was the likely starting point znp The quantity E[rolz, t] is the Bayes-optimal solution this problem.
In the context of this toy example, it has the following form. If is very positive, one tends to believe To +u if is very negative, one tends believe zo -v. For intermediate æ, especially near = 0, uncertainty is highest, and B[ro jr, is near zero, since the observer could have plausibly started at either za + or 2g -v.
This high uncertainty allows us to formalize the idea that the region between + and -4, especially near x 0, is a boundary region. Quantitatively, we have
var(xolor,
Σ
2op(xolZ, t) E[pola, 1/2 222 1 tanh2
(Asj o x)
)]
n
(38)
cosh? 2
zo
or equivalently
V var[[oo|
/
(39)
cosh
Dell x
At x = 0, the standard deviation of p(20|2, f) equals 14, i.e., there maximum uncertainty about of the starting point zi- Moreover, it fairly high until x As axp which also shows that the effective size of boundary region is smaller at smaller noise scales. Said differently, the basins of attraction surrounding each training example become increasingly sharp o, - 0.
20

---

Published as a conference paper at ICLR 2025
D PATH-INTEGRAL REPRESENTATION OF LEARNED DISTRIBUTION
In this appendix, we derive a path-integral description of the 'typical' distribution learned by diffu- sion models. We do this in three stages, First, we derive a path-integral description of the PF-ODE. Next, we derive a path-integral description of a more general kind of stochastic process. Finally, we show that averaging the path-integral representation of the PF-ODE over sample realizations pro- duces a path integral whose dynamics correspond to those of the aforementioned stochastic process.
D.1 WARM-UP: DERIVING A PATII-INTEGRA REPRESENTATION OF THE PF-ODE
general ODE can be written as
c fax,t)
(40)
where 2, E RD and & E [c, We will assume that f is smooth to avoid technical issues. If we discretize time, and slightly abuse notation by using and T refer integer-valued indices instead of real-valued times, we can write the trajectory (Tr, TT-1, 11....... 11, and the corresponding updates in the form
2++1 f t 1)
(41)
Note that our discretization corresponds to first-order Fuler update scheme. In the small Al limit, this specific choice does not matter, even matters in practice; we use to slightly simplify our argument. Conditional on the initial point ær, the probability of reaching another point wo after T backwards-time steps is
p( / 8(2r0 x f11 1)41) 8( or + f( ( day ... ...-r- (42)
where ô is the Dirac delta function. Here, we will employ a well-known integral representation of the Dirac delta function:
dp (2T)
8(x x1)=
exp ( (x x')}
(43)
where p is integrated over all of RD. Our expression for p(zo becomes
p(Zo|Tr)) = dpo dæ dør-idpr-1
(2m) (2m)
(2T)
exp
ip: [mt x1+1 + f( + 1)
(44)
Schematically, we can write this path integral as 'sum paths'
p(æo|oT) / Dlp.ID(aa) exp { S" ip: 1- + f ()] dt
(45)
although explicitly using this form is unnecessary for our purposes. (This is good, since remaining in discrete time allows to avoid various thorny mathematical issues.) For the particular choice of f associated with the PF-ODE, we have discrete and schematic forms
p(x0oo07) /
dpo dx dp dxt-idpre ET 1 -ip--le: 20+1 (Be. 18:41 Des Pe0e:e//211100 ((2)))
( (2r)
(2T)
p(Zoo(ET) / "pp "p//D(xd) exp I' ip. [ + Blz + Dis(a,, t)] dt
21

---

Published as a conference paper at ICLR 2025
D.2 DERIVING A PATH-INTEGRAL REPRESENTATION OF A MORE GENERAL PROCESS
Consider a more general type of backwards, discrete-time stochastic process. Once again, suppose that a variable 24 RD evolves backwards in time from initial point æT. But this time, suppose that the transition between 2++1 and & depends upon some set of K independent standard normal random variables {Ex}. In particular, suppose that discrete-timg updates have the form
K
Tt r4+1s - ff 1)
Gj Σ
At,
(46)
Le., updates the same before except for the new noise term. In general, the noise term is quite complicated: G is a D x K matrix which can depend explicitly on both the current state and the current time. The process described by the above updates is generally not Markov, since noise added at different time steps can depend on some of the same &a variables, and hence the amount of noise added at time step can be correlated with the amount of noise added some other time step.
What is the distribution of zo. the result of 'I' steps of this process, conditional on starting point Ør? ? We know that each update depends only on the previous state and the noise variables, so
p( = p( 121, (Ex (p(p))((p))))) |222, [Ex)) ... p(#r-1lag) (Ex) }) p((Ex)) dr ... ---d---dd [Cx
In particular, conditional on the previous state and the noise variables, updates are deterministic. This allows us to write the above transition probability as
K Σ
Ggllanitt1) Ex st) ) p((Ex)) dx -ddd--idEE)
II 8 -=
Ftis ( I1ti 1 + (1 1)
Using the same integral representation of the Dirac delta function that we used above, this becomes
f e -ip sest. (=4+1 G)u(mee1,t11) &a Ad] p((Ex)) dpo dxidpı dxr drr-idpr-2 dl
(2T) (2x)
(2x)
Although this appears to be extremely complicated, it can be considerably simplified by doing the integral over the noise variables. Since the noise variables are all independent and standard normal,
1
&
S
p((E+)) (2T)a/2 exp
(47)
2
Hence, the integral over the noise variables is typical Gaussian integral with linear term We can save time by recognizing the integral as essentially computing the characteristic function of standard normal; more precisely, we have
T-1 D
-t/2 c
I = exp
iEk
Σ
Pc. Ga + 1)
dEp
22TT
t=0 j=1
(48)
T-IT-1 D D 1E t= :=0 j=1ا'1
cxp
Σ
Pc. Gr.Gatt
1)Gyn(ev41, 1)pe Atdt
for each Éx. Putting everything together, we find that p( (T) can be written
En Σ Pt. G, (eet1 (11)))1+++)44 ( +1lee Aide (dttd(pp)
(2 (T
where we have used the shorthand (( == dpo da ... dxr- -dp--1 This is our final answer, although it more enlightening write its schematic continuous-timg form. We obtain
(()))o)) / D D.la] exp {T" [.--'" -ip 1- + f(x,, t)] dt
T
["
2 2.
P (ai, 1 l'r didt'
where we have defined the state- and time-dependent D x D V-kernel Viy(a,, t; xt, via
K
Viz 1; (') = Σ
Ga (av,),)
(49)
equivalently via V V( 1; my, 1') G( () GO ('). Note that it is positive semidefinite.
22

---

Published as a conference paper at ICLR 2025
D.3 AVERAGING LEARNED DISTRIBUTION OVER SAMPLE REALIZATIONS
What is the 'typical' distribution learned by an ensemble of diffusion models which differ only in the samples each used during training? In this subsection, show that the net effect of averaging sample realizations is to contribute noise term to the PF-ODE. The path-integral representation we obtain is of the class we discussed in the previous subsection.
Suppose a diffusion model is associated with a parameterized score approximator $ t). The distribution learned by the diffusion model is then
q ar;; 0) D(p.)D(os) exp {[" ip. [ Br0r, (asa () di }
(50)
where have used the schematic form of the PF-ODE path-integral representation for clarity. (Moving to discrete time does not affect our arguments, but only makes notation more cumbersome.) Averaging over sample realizations is mathematically equivalent computing the characteristic function of the score approximator. The sample-averaged g. Eg (2 ær; 0) = (g(=ol<<))) is
[g<2g|<T)] / Dip,)D[a.] exp {1. ip; ( + 3,x,] dt Eg e sp. Di di)
(51)
Assuming the score approximator ensemble is well-behaved, its characteristic function can be writ- ten cumulant expansion. Here, we have
log Eg (e." ip! (m))( dt
T = =f "p,D,(sa(a,,)) ()] dt {"² J.¹
(52)
PÊ D,Cov (5 So ((xe,t))) Depr
where the dots indicate higher-order cumulants and (x, f)] indicates the ensemble-averaged score approximator. In this work, we neglect the higher-order terms. Often, they are suppressed by some factor (e.g., the number of model parameters divided by the number of samples).
We obtain dynamics of the class described in the previous subsection. Here, the D D V-kernel is Dr,sa Cove Sa (Iz, t), 8 De.
Viy t; Fy, t') = Σ
(53)
a.b
equivalently V( ky, 1') D,Covo (# t), s0(xe, Dr.
23

---

Published as a conference paper at ICLR 2025
E NAIVE SCORE ESTIMATORS GENERALIZE: DETAILS
In this appendix, we show that integrating the PF-ODE using naive score estimates yields a specific kind of generalization (Prop. 4.1). Suppose that we are integrating the PF-ODE from some initial point æ7 using 7' first-order Euler updates (or some other integration scheme; the choice does not matter in the continuous-ting time limit), so that
xt I++1 (84410441 D At.
(54)
But suppose that we do not use the score function directly in our updates, but at each time step construct a noisy version of it based on sample æo: p( x. t). In particular, consider the naive score estimator
8(x,,t) s(x,,
(z moo) s(x,,)) t)]
(55)
where 2 0 is a constant that controls the estimator's variance. Note that new, independent sample zo is drawn at each step t'. We are interested in studying the extent to which this scheme produces distribution different from Paato P
Using the result from Appendix D the typical learned distribution [g(0o) is (approximately) characterized by the average and V-kernel of &. Since Emg [$(x; t; æo)] = s(x, t), this score estima- tor is unbiased, i.e., [$] = s. The V-kernel (( t; Fe, t') is
V == DoCove ( mr, t), 8(xr,, Dr D,Covo (s f). $(x,, D; 8(t - 1')
(56)
since samples generated different time steps are independent of one another. Moreover.
Covolš(zz,,)) At Covelš(zz,
(57)
since the only random part of the estimator is the proxy score. Finally,
Vag 1: ve, 1') = K. Σ
1), (a Da,s 5(1 - (')
a,b
(58)
K Σ
Di.ia [s + 0g log p(zi(t)) D1, 8(t t')
a,h
Using C(x,t) t) as shorthand for the proxy score covariance matrix, we equivalently have
V ( 1; Rv. (') = DDDC L) 8(1- (0--)) 1')
(59)
As final technical note, note that the naive estimator must scale like 1/VAt in order for the V. kernel to be nontrivial in the At - limit (and indeed, for the continuous time limit to make sense). This is easiest independent, the V-kernel picks up factor dkc, which equals one when k = f and is zero otherwise. In continuous time, this looks like ö(t At) not 6(t-t') (Eq. 56). This problematic At factor can be canceled by a corresponding 1/(At) factor in the estimator covariance, which motivates making the estimator scale like 1/ 1/VAt.
in discrete time: since samples generated at different time steps and & are
24

---

Published as a conference paper at ICLR 2025
F LINEAR SCORE ESTIMATOR: DETAILS
In this appendix, we compute the sumplerealization-avergg distribution learned by a linear score estimator (Prop. (.T). Whether it generalizes not depends strongly whether the number of features J' scales with the number of samples P used during training. First, we must compute the optimum of the DSM objective for linear model. Then we will determine the average and V-kernel of the optimal linear score estimator.
F.1 DEFINITION OF LINEAR SCORE MODEL
Consider linear score estimator
F
8 (x, t) wo + (
8, t) = 100 +
(2,
(60)
j=1
where the feature maps Φ- (Ø),... "r) are linearly independent, smooth functions from RD [0, T) to R that are square-integrable with respect to the measure A.p(a, t) for all t. The parameters be estimated are = {wo, W). with wy R D and W RDP.
F.2 OPTIMUM DSM ORJECTIVE FOR LINEAR SCORE MODEL
For this estimator, the DSM objective reads
(0) = / d Ww + WΦ(x, t) 5(x,t==)) 2o) p(a/zo. t)pdata (zo)p(() dædzodt.
(61)
2
Note,
0s:
08:
8in
8ia
(62)
Owcn
oW ab
Using these take the gradient of the DSM objective, we have
0Jı
Ex
A
+
Σ
5a to t00) W (x,
Owos
j=l
(63)
0J1 Exe æu.it
A
Woa
+
W (2, -3, 200)
Φb(2z t) s.
aWab
Setting these equal to zero, we have
Enn, {de) Woa
+
Wa Way En (aco)(ar Em (4,8a 1;20))
F
Em,.o.. (aecho 1)} Woa
En. (26 (x,)( = Em (4.8 (2 f))
The first row tells that
Woa
Ex
E no.t [A.]
E2 [vr]
1
Σ W a Br. (x,))(
(64)
or cquivalently that the optimal bias term satisfics wi = (3) W (Ø), where we have used (...) to denote averages with respect Xep(2,, wa, t) [x.] and where we have defined the vectors
(8) = Ex.... 3 L; [00]
E, Ax] / A, $(x; t: xa) p(z, 2a. t) dx
E, [1.]
(65)
(6) = Ex, De ( (x, (x,))) t)]
Xe sl t) p(æ, t) dxdt.
E, [1c]
E, [x(
25

---

Published as a conference paper at ICLR 2025
Using the first row result, the second row can be written
F Way
Ex
{do0, (x, (x,
Be.w.. {Ae5a (a, t; au) (x, t)}
(66) (5o) Σ (og)
Ee[A,]
E, (x
and hence the second row can be written in terms of matrices
EΦ = Ex, {de (Ø(x,t) t)
(Ø) ( (Ø)]² }
E, [1.]
Ed / de (0(2,) (6)) [0 t) (6))² p(a,t) dædt
(66)
J = Eg, wa.t {he (0(x,t) (क)) ((x t; vu) }
E, [xr]
/ de (o((,, (Φ) (((xt;20)) (8) p(a, xo. t) dxdagdt.
E, Ae
In partieular,
W W
W =-==J'>
(67)
where we have assumed that Es is invertible. This ought to be true, since the feature maps are independent and p(z]t) is smooth distribution supported on all of R (especially since we are technically only considering as small as the nonzero lower bound, for regularization purposes).
The optimal score is
(x w + W (xx (b) b(x, t) (8)
(68)
As side comment, omitting the bias term just removes the mean corrections from the definitions of J and Eg, as well as the (Ø) and (s) offsets. Without it, the optimal score is 8. (x. W ((x,t) t) J ((x t), where J and Eg are instead defined to be
Eg Ex,; {he Φ(x, 1) 1)=
Ec[Ac]
(69)
J E Egg. the ((x, t)5( t; x0)"
[A,]
In the rest of this appendix, we will assume that the bias term is present.
F.3 OPTIMUM DSM OBJECTIVE GIVEN A FINITE NUMBER OF SAMPLES
Assume we have access to P > 1 samples (0),, To (()) p(æ, Fo, t), and that we estimate the parameters of the linear score model using naive sample mean estimators
1
Σ
(m)
P
Σ
(m) 8( (n) (1)) ((0))ar(u)) (
= X, P
A
X X,P
1 1
(m) (1))m))(()))))
(70)
E = 1 1
Σ (m) ()) ((00) - :10) (n) (1000) im/b)
X, P
3 = Xe P Σ n
1
(m) (2))) ((0)) ms0) (8(zoln) ((0) x0) (n)) - _)=
where have used (m) about using Bessel's 's correction in the covariance estimators, and we will see below that ø is actually unbiased for finite P even if the covariance estimators are not. Our learned score estimator is then
slightly less cumbersome shorthand for Aytn). We will worry
S (z. 1) j i Pp $xx,1)] b.
(71)
26

---

Published as a conference paper at ICLR 2025
Note that if the number of samples P is less than F, Eg is not invertible; in this case, can either use the Moore-Penrose pseudoinverse, or explicitly include a weight regularization term in the objective, ie., add to J1 term of the form
Jreg
Σ
2 wot
Σ
W
(72)
where the parameter E 2 O controls the importance of this term. Including this term changes the score estimator (Eq. 71 by modifying the inverse that appears:
4 E +
(73)
In what follows, if one wants results in the case that such a regularization term is present, note that this replacement of the inverse of the empirical covariance matrix the only necessary change.
F.4 LINEAR SCORE MODEL ESTIMATOR UNBIASED
We are primarily interested in variance due to Zo (for reasons that will become clear), so we will
consider ensemble of systems for which the (n)) and g(m) sample draws are the same, but the ax"))
draws are different. Our estimator depends linearly on 8, the quantity through which it depends on the By samples. In particular,
j " Iio Φ0x, t)] = P
1
Σ
(n)
(1)) 5(x ((x(11),
(()))),)))) (00) (00))))) 6] ( - : ES [0(x, t) - P
X
n
1
(m1)
((x: {(m) (()))))))))))cc
(0((011)) ((a)) mes] ST [0( - 1
=
P
so
P Σ (m)
(2 xx t)
((x(m) {00) ; x,
((x("), {(m); ((m) =()); (xn))
(74)
X
where we have defined the kernel function
( 1; x', (') = 1 + (0(a,, - 20) E (1 8) A
(75)
To that this estimator is unbiased (when the model is sufficiently expressive), suppose the true score has the form of our linear estimator, i.e.,
s az, t) w W ((x,tt W*
W [<( t) (Φ)
(76)
where we have used the fact that E.n is] (s) O. Next, note that
(m)
p Σ X (m) (()) x. t) = 1.
(77)
Averaging our estimator over Mo sample draws yields
E Si (x: t)] = P
1
Σ
(n)
(x)
t(m); (x)
x, t) W ()(((1)) ((()))))))))))))) - fp + ft (
n
1
(m)
d (r) (,)),,
;x,1) 1) W (x) ,()))- (i] + W -())
P
1
Σ (m)
d (' ((m)) - 41] | ((0)) a T SS1 (Ø(x, t) ()) w (Φ))
P
W (Φ(x,t) (t) W ( (
= w (Φ(x, (6))
i.., is unbiased. What worth emphasizing is that this is exactly true, and does not require taking any kind of large limit. In other words, as long as P is large enough that Eg is invertible, one recovers the true weights w and W independent of the and t sample draws. This is why variance due to Tn sample draws matters, and variance due to the other draws does not, at least for this linear model.
27

---

Published as a conference paper at ICLR 2025
F.5 COMPUTING THE V-KERNEL THE LINEAR SCORE MODEL
Computing the V-kernel amounts to computing the covariance of the score model with respect to æo sample realizations. In the previous section, computed the of our estimator; the covariance calculation will be fairly similar. Note that
A (n) A (m)
Cov($(=),
;)) (z))))) 5(2 (12(m)) xmm))
p2
X
A (m)
Q(2(m;; ; ;01((())))))))))))))))))))))))() (0)
p2
where we have used z as shorthand for {æ,t), and the fact that zo sample draws are independent of one another. Now we will invoke the central limit theorem. Using C(z) == Cov(š(z; zo)] as shorthand, when P is very large, to leading order in 1/P we have
8(()) P.
Q(=": (<=" C(z") p(z") dz"
(78)
Ae B
((=x=)))(²) C(z") p(z") dz"
p
where we replace the estimates j and Es that appear in the kernel function with the true quantities, ie., we redefine Q be
Q(x", x,; () = 1 [0(@" (6)) EΦ [0(x 1) - (6)
(79)
If the number of features F does not scale with the number of samples P. then we are done: in the P oo limit, the score estimator covariance, and hence the V-kernel, approach zero. Alternatively, if the number of features F does scale with the number of samples P. nontrivial result is possible.
The second term of Q, a quadratic form involving the model's feature maps, is the only place in Eq. 78 one can get nontrivial scaling with F. Motivated by this observation, define the feature kernel
1 16(0)) (00==) -0)) (0' (0(2') (
k(z;2')= ==
(80)
Provided that the limit exists and is finite, in the oo limit (where F may scale with P), the asymptotic V-kernel is then
"(="""""""" } Dr.
X
V(z;=') = lim D. Bg" P P
(81)
Note also that, in the large P limit, the V-kernel also does not depend on the x and sample draws.
28

---

Published as a conference paper at ICLR 2025
G NEURAL NETWORK SCORE ESTIMATOR IN NTK REGIME: DETAILS
In this appendix, we prove Prop. 5.2 which means computing the V-kernel of fully-connected, widd neural network in the "lazy" learning (Chizat et al. 2019) regime. Although we focus on extremely specific type of network here, note that argument can be straightforwardly adapted to compute the V-kernel of other architectures with NTK limits, like convolutional neural networks (Arora al. 2019).
G.1 DEFINITION OF NEURAL NETWORK MODEL
Consider a neural network score function approximator (0 1) trained on the DSM objective (Eq. 4p. As elsewhere, we may use as shorthand for (x, t). For concreteness, assume that the network fully-connected, has L 2 layers and N, trainable parameters, and that each hidden layer has N neurons and an identical pointwise nonlinearity G:
a (z) = (1(z))
(4T)) (2) = G
W
(((+)) oj (x)
= L.....--22
VN
(82)
(z)
1 VN
Σ w
aj
-- (z)
80(z) = (z)
The (non-trainable) initial feature maps + == (v), .... NN,))² 1 account for various preconditioning- related choices. For example, in practice, diffusion models receive time/noise as input only through some time/noise embedding (Ho et al. 2020 Song et al. 2021 Karras et al. 2022).
Although characterizing the gradient descent dynamics of may be difficult in general, the initial network weights are sampled i.i.d. from standard normal (i.e., in ~ N/ 1) for all i, j. and E), as N is taken to infinity the network output becomes independent of the precise values of the initial weights. Moreover, the network's output throughout training can be written in terms of kernel function- the so-called NTK defined by
(8a(2) 08 (2') )
K (z, (z,z')
Σ Eo
(83)
ab;
00,
where and c' index different network outputs. In the anfinitewwitth (N - 2o) limit, Kee (z,z') de K (z, z²), i.e. the off-diagonal kernels are identically zero and kernels along the diagonal are the same (Shan & Bordelon 2022).
G.2 LEARNED SCORE AFTER FULL-BATCH GRADIENT DESCENT
Computing the learned score. For simplicity, we assume that our neural network model is trained via full-batch gradient descent P samples from p(æ, Fp t). Although this assumption does not reflect standard practice (Song et al. [022 Karras et al. 20022.. it makes our computation substan- tially easier. If we let the dimensionless parameter denote training time, the output evolves via
d s(x', t') EE t.00
A
0s(x', 08(x, t)T
[5(z, t; $(x, s(x,0) 011
(84)
JF
E
00
80
In the infinite-widht limit, we can replace the outer product that appears with the NTK:
de
dr 8(x' 1') = F. 2000
K(x,, 11: a. 1) [8(xr, mo) - s(x,1)]
(85)
Ex(Xc]
29

---

Published as a conference paper at ICLR 2025
Define the Gram matrix K RPxP, the time-weighting matrix Ar E RPPP, the target matrix S PxD, and the output matrix $ PxD via
abb == K ((x (1()) (1)) ((0)
ab == Sah Apin)
Ex(
(86)
Sai (((((1), ,ta)) )
S.t = (,((()) ((a)).
Eq. 85
implies that
d $ 1 KAT (S S)
(87)
dr
Hence, after training, the network's 's output on the set of samples is given by
(88)
where is the total training 'time' and So is the P D matrix containing the network's initial output on the samples. Let k(x,() 1) denote the P-dimensional vector whose i-th component is ((x(0), x00 t). The network's output given other inputs evolves according to the ODE
dr ((x,t)) = t)² AT (S-S).
(89)
whose solution is
8(x, 1)7 = 80(x, t)T + k( k(xx (SAArr/P)( Sp)
(90)
where $o(x, t) is the network's initial output given far, input. If the Gram matrix K is rank- deficient, we must use its Moore-Penrose pseudoinverse. Alternatively, one can avoid this issue by including a weight regularization term in the objective.
Expressing the learned score in terms of eigenfunctions. We will find it useful consider Mercer decomposition of with respect to the measure A.p(2,, t)/ [De,, so that can be written
K(æ, L; t;
Σ k
Aadul (2x ,(')
(91)
where the features are orthonormal and complete, i.e.,
/
de
E. de (æ,, () (ar, () p(n, dadt 8
(92)
d p(x,t) (x ())x(2)) 1') -6xx x')8(t (t)
Σ
EAA
If we assume K has rank F not necessarily equal to P, we can write Eq. 90 in terms of the eigen- functions associated with the Mercer decomposition by defining the P x matrix & with
Pak dz (x( (a) sla))
(93)
and noting that K oA where A is the F x F' diagonal matrix of associated eigenvalues. It useful to observe that
8k = Exx.
d
cx t) (2,
_ (m)
E, [r] Øn (m), (ox) (x'") ((m) +O(L/VP).
[x
which implies I
s" A to leading order. Similarly, the completeness relation becomes
ΦT T
AT
T
Ip 2
(94)
P
P
30

---

Published as a conference paper at ICLR 2025
to leading order. Using these identities, rewrite Eq. 90las
AA -1p"" Ar
8(xe1)" = 80(2,, 1)ad + [ 1)" A
Ar) AT ($ - So)
p2
= =x, + t) -
- AT So) -So))
(95)
Equivalently,
= (2, + p (S So) Ay 4 (M =
(96)
Let S denote the Px D matrix whose entries are the true score evaluated on the set of input samples {(x11) )) When averaged zo sample realizations, our estimator is
B[s(0, t)] so(x, t) + 1 (S - (6)" AT - -- (0)(()
(97)
which implies
1 ( S)" (I
& ( t) E[$((, t)] =
- - --
(98)
p
To make things slightly easier, define the feature kerne
1 k( t;, J --
1
Σ
(z f) e Aet jon("',
1m
(99)
In terms of this kernel, can write
VF (m) P Re(Ac
8(x, () - E[$(x, 0)
Σ
1100;; (0()))
,(0)()),((0)) ki(as
[ba))]
( ,,00 x0, 1)
(100)
We will use this result in the next subsection to compute the V-kernel of this model.
G.3 COMPUTING THE V-KERNEL OF THE NTK MODEL
The covariance of the learned score estimator with respect to zu sample realizations is
Cov($o(2),8ol 6z('))
F
(mm)
A (im)
Cov
[8(2(m);
(m); mr (m)) 5(2 (m) ;a) (m)) } () (=)
P2
E.
[
[Ac]
F
(m))
[((2 (x) (m)) (n) (2)
Cov
p2
[A
(=);)) ((==""")))<>) dz"
P
EA
when P is large, where exploited the independence of the samples in the first step, and the central limit theorem in the second. As elsewhere, we have used C(z) : Cov [š(2; 2o)] as shorthand. Finally, the V-kernel is
F Dr Ey" { Eg"
XB.
} De
Viz;z') lim P-000
(=x""""""""""""""""
(101)
provided that the limit exists and finite. Since F N, this can happen if N x
Also note that the form of this V-kemel is identical to that of the V-kernel for linear models (c.f. Prop. 5.T. with the only difference being the feature kernel that appears.
We have run out of letters, and unfortunately will use & to denote this quantity. Note that it is different from both K and k.
31

---

Published as a conference paper at ICLR 2025
The infinite training time limit is of particular interest, since in this limit expect the model to interpolate all its (noisy) samples. In this limit, we have
1
/
X
P
Φ(z)² (0==(((")))))"""")))))"))))))")" Ae
"(x") p(z") dz"
E, [,
1
I
Ed(A 0xz)=((=")
Aem
B.(Ac] 0 (((=")(()))()))))) Cis(z") dz"
P
P Ec(Ai ((=))(("") - -2") dz"
1
/ de"
(102)
1 de
P PE o(z) 25 (22')((((z²)
where we have exploited the completeness relation. Now we encounter subtle technical point. Since F * in general, in the F, P oo limit the quantity
1 A ((2)²((2') PEA
d(z, x')
(103)
is not quite equal to the Dirac delta function, but is instead proportional to it. We need to work out the constant of proportionality. To do this, observe that
1 Σ
(m) [A
A
d( (n)) = n
((
)x(z))x(z) p(z) dz F.
Ex[Ac]
On the other hand, for the Dirac delta function, we would have
P
Σ
5(0) Az
(104)
where A= is some small bin size. This implies
FAz 8(z 2²) d(z, 2')
(105)
we define x := (FAx)/P, and assume x remains constant as both parameters approach infinity, finally obtain
= 6(2 - z²)
(106)
32

---

Published as a conference paper at ICLR 2025
H RESULTS FOR NOISE PREDICTION FORMULATION
In the main text, we present the problem of training a diffusion model in terms learning a param- eterized score function estimator (see Eq. 4 In practice, one often does try to directly estimate the score, but the quantity (æ,, t) == ( t) (Rombach et al. 2022), since it can be somewhat better-behaved. this appendix, we note that it straightforward port our results this slightly different setting.
For simplicity, assume that forward process involves isotropic noise, so that D. In and S, = o. The DSM objective (Eq. becomes
gi. 2
d |o, - of t: 2
(1) Er,
2
(107)
A (10 E12
Ex, 20.00
2
Since := (ay - (//aao/)//o o, by definition, is normally distributed with mean zero and variance onc. The function & can be viewed as taking a noisy sample æ as input and outputting the (standardized) noise that was added to the original sample zy.
Importantly, the objective has the same form as before (i.e., a mean eeran-s error objective compar- ing an estimator to a target), but the time-weighting function is now X := \z/o?.
In this formulation, the PF-ODE reads
Φ; -3,0
20
(108)
B,x
D.E (x,
where we define
92
D.
(109)
20.
Hence, the PF-ODE also has the same form as before. We conclude that our results apply once one makes these identifications, and also makes the slight change
C(x, t) == o² o/
(110)
since the learning target is now a scaled version a, of the proxy score.
33

---

Published as a conference paper at ICLR 2025
I RESULTS FOR DENOISER FORMULATION
In the main text, we present the problem of training a diffusion model in terms of learning a param- eterized score function estimator (see Eq. 4). An alternative approach formulates the problem in terms of learning a 'denoiser' function, which takes a noise-corrupted sample æ as input and outputs a noise-free sample Zv. These formulations are mathematically equivalent, but their implementa- tions slightly differ in practice; in this appendix, we note that is straightforward to port our results this slightly different setting.
Define the 'optimal" denoise P D in terms of the true score $ via
((x,t) t) = a2
[St s(z, 2
s(x t) S (a, x
(111)
Parameterized estimators of the denoiser and score, which we will denote by D and 8, are related in the same way. This means that the DSM objective (Bq. becomes
Ae | (a ((x, m:)01
J, (0) 2
de || aa (o(o,, () -a] -0 - =0112
mo.
(112)
toaa (D( - xo) s; 3D2 t) mol
2
For simplicity, we will assume the diffusion tensor of the forward process is isotropic (D, = 5/ In) which implies S. = of Ip. Then
Axaa
(0) = my
Da(a,, () x0l12 Be, 20
2011
(113)
where A, Xra /ot Hence, the objective has the same form before mean-squared error objective comparing an estimator to a target, with particular time-weighting function).
In this formulation, the PF-ODE reads
2 [zs
91
t -Bax;
xc]
20
g
(114)
Br
20
Dt
20²
04 D(xr.
B,0,
D,D(xrt)
where we define
gi
of
B, = Br
20
D. =
20 aq
(115)
Hence, the PF-ODE also has the same form as before. We conclude that our results apply once one makes these identifications, and also makes the slight change
C( () == Cov Cov 2 (wo)
(116)
since the learning target is now 2g rather than the proxy score.
"There is unfortunate collision of notation here, with D being used to denote both the diffusion tensor and the denoiser, but it should be fairly clear from context which is which.
34

---

Published as a conference paper at ICLR 2025
J BENIGN PROPERTIES OF GENERALIZATION THROUGH VARIANCE
priori, one may worry that generalization through variance can happen in an 'uncontrolled" fash- ion, and hence produce generalizations of the data distribution that somehow 'bad' 'unrea- sonable'. In this appendix, we collect properties of generalization through variance that help ensure that this does not happen.
Most of these properties relate to the proxy score covariance, which when the data distribution consists of M examples has the form (see Appendix B
C(., t) == Covenla ( = S, CovM(u)S;
(117)
where
p( = pm xx,t) = Nax; ar S,)
(118)
Em N(x; aeptmms Si)
For the sake of this appendix, we will focus mainly on the case where the forward process is isotropic, which means G: Ir D; g8 ID and S. = IfI In this case,
a? C(x,t) ot
(
(119)
N(a; Wilflm: of In)
[_1x (-lee (-110-0tftim On (m
p(2o- = 1mm (onlxr,))
softmax
Em N(x; aeppmee Ip)
20
J.1 SINGLE POINTS ARE NOT GENERALIZED
Suppose that the data distribution consists of a single point ag = μs, so that M 1. How do diffusion models generalize single point? Since
N( IiIp)
p( = 1a ar, ()
(120)
M(x; Ib
the covariance of zo is zero. This makes sense, since given pair (x, there is no uncertainty about the training example 2g that generated it.
This implies that the covariance of To given and it is zero, and hence (if the forward process is isotropic) that the proxy score covariance is zero. For the naive score estimator and NeK-me neural network, this implies that the V-kernel is always zero, and hence that generalization through variance does not occur. This is reasonable, among other reasons because there no reason to introduce anisotropy if is not present in the data.
J.2 DIMENSIONALITY OF DATA DISTRIBUTION PRESERVED
Suppose that the data distribution lies entirely within a subspace of dimension < D, so that the components of each pm along the non-subspace dimensions are equal (to 14 along dimension k, say). It seems intuitively reasonable not to substantially modify probability mass outside of this subspace. We clearly have
Exg mr. = = =
(121)
for all non-subspace directions k. This has the following consequence. Let denote some other (possibly within the subspace, possibly not) direction of state space. The k-L covariance is
Cov (T - (xoe - l [ 1)] =]=00
(122)
since Fok is always equal to 1h. Hence, Cty (and by extension, Viy) is only nonzero if i and j are directions along which the data distribution varies, which means generalization through variance only happens along the 'data manifold'.
J.3 VARIANCE NOT ADDED FAR FROM DATA DISTRIBUTION EXAMPLES
Since p(zn/2; t) has the form a softmax function, taking x extremely large is analogous to taking the temperature parameter of typical softmax to be extremely small. For far from the support
35

---

Published as a conference paper at ICLR 2025
of the data distribution, to good approximation p(æ [m| (m..)) where mc (2: t) denotes the index of the data point closest (in terms of Euclidean distance) to æz Since p( ar, t) collapses to Kronecker delta function, its covariance goes to zero, and hence the V-kernel also goes to zero.
J.4 FOLLOWING AVERAGE SCORE FIELD MOST LIKELY
straightforward consequence of the effective reverse process (Fq. 8 is that
d (z) = (f(x))t)
(123)
dt
where the angular brackets here denote averages with respect to [q(x, t)]. Hence, although the effective reverse process involves noise, that noise has mean, average the system follows PF-ODE dynamics that use the ensemble-averaged score estimator. the estimator is unbiased, this just the usual PF-ODE. In other words, even in our setting, on average PF-ODE dynamics will reproduce training examples.
There is a slight technical subtlety here, which is the fact that (f(æ ()) + ((x)),)) () in general, but this distinction becomes unimportant for very small noise scales, which most strongly influence whether memorization occurs.
J.S TRAINING DATA ARE MORE LIKELY TO BE SAMPLED WHEN NOISE SMALL
Similar to the previous point, if the prefactor that controls the size of the V-kernel is not too large, trajectories that follow PF-ODE dynamics are more likely than trajectories that do not. Note that there is distinction between the most likely trajectories of stochastic process, and the average trajectory associated with that process, although here one expects them to be fairly similar.
This idea can be formalized in the context of semiclassical analysis (see Appendix ??), which shows that the probability of a given path goes like
where
T 1'
[") [ f() Q(x,, t: xe t') [Ev f(xe,t)) dtdt'
S|x =
(124)
for some matrix Q that functions as the inverse of V. The point we would like to make here corresponds to the observation that this action attains its smallest value when क; = fa,, t) for all times t. That is, trajectories which follow PF-ODE dynamics more likely than those that deviate from it, with deviations being penalized more harshly as k made smaller.
36

---

Published as a conference paper at ICLR 2025
K
MEMORIZATION AND THE V-KERNEL IN THE SMALL NOISE LIMIT
It's clear how the V-kernel affects a given sampling time step, but how does it affect the overall learned distribution? This question is difficult answer analytically, since in general the effective reverse process is not exactly solvable. Even without the difficulties related to state-dependent noise and nontrivial temporal autocorrelations that the V-kerel introduces, PF-ODE dynamics are only exactly solvable in special circumstances,, e.g., when the data distribution is Gaussian (Wang & Vastola 2023 2024).
It slightly easier to relate the V-kernel to the (average) learned distribution [(())) when the overall magnitude of the V-kernel is small, since in this limit one can invoke semiclassical approximation (Kleinert, 2006). which is the path integral analogue of a saddlepoint approximation. This limit plausibly applies if the prefactor (which is proportional to the ratio F/P for the latter two kinds of models we consider) is small, equivalently if the model is somewhat underparameterized.
A proper treatment of this topic involves the careful manipulation of notoriously tricky mathematical objects like functional determinants; to sidestep these issues, and be somewhat more brief, will proceed in a somewhat heuristic fashion here. one of the following steps appears unclear, we advise the reader examine in discrete rather than continuous time, where the associated issues are less severe.
K.1 SETTING UP THE SEMICLASSICAL APPROXIMATION
First, define the prefactor-divided V-kernel V as
V(z=²') = V(z;2')/K.
(125)
and define the 'inverse" Q of the V-kernel as the vector-valued matrix satisfying
Q ti aer, 1') (xve we, di" di" f( t"x, avv, di" In 8(t-('.
Our path integral expression for [(())) has the form (see Appendix D
[a(aro)= = DDox) exp (-S(_ p.)) N:)) 0, Sr)
(126)
where the 'action' is
Slar pc) :=)= [" -ip! [. f dt + NN f.
1" [' (an, deddt' (127)
2
and where the functional integral is over all paths x(t) which have x(0) zo and xT) xr, and all possible p(t) paths. The explicit form f is
f(x, 1) = - 3,x - DiSavg (x, f)
(128)
although for this analysis not relevant.
Since the action is quadratic in the 'momenta' variables Pt. we can perform the associated Gaussian integrals exactly obtain reduced path integral with
[a(==)) = / D (lass/(pp)) exp (-S(mi)/n) F((x:)) N N(Tr; 0, Sr)
(129)
/" T /" ( f)) Q trr,,tzr,t t') [+te f(xr,t') dtdt'
S(x;]
where the factor F involves functional determinant
F((x+)) = del
(130)
due to the Hessian the action. Note that Q is positive semidefinite since V is, and hence the minimum possible value of 8 is zero
37

---

Published as a conference paper at ICLR 2025
K.2 SEMICLASSICAL APPROXIMATION OF THE LEARNED DISTRIBUTION
One can now invoke the semiclassical approximation of this path integral assuming x is sufficiently small. Relevant to this approximation is the 'classical' action (Z Tr), which quantifies the likelihood of I following the most likely (i.c., 'classical') path from 27 to Zoi
So TT) := (1), S
(131)
Also relevant is the Hessian of this quantity evaluated at the least action path, which reads
H 82S
(
T J,, Q di
dt J,
additional terms
(132)
K. 82x
where y is the Jacobian f. The additional terms will vanish after we make one more approxi- mation, ignore them. The (functional) determinant of this Hessian is
{(d JJ J J,
det H( DT) = det
Q Jz) }
additional terms
(133)
Although notation here is deliberately somewhat vague sidestep various technical details, the above quantity depends on the entire path { and the values of the Jacobian and Q along The semiclassical approximation says that
[d(xx)]
/ exp (-Sa 2T)/k) det H(wo &r del V V(Do, wT)
dær ((2T)D
(ar 0, ST)
(134)
We can invoke Laplace's method in order to approximately evaluate the ør integral, and hence obtain an expression for the (average) learned distribution that does involve any integrals. The classical action S can be expanded with respect to (wo), the most likely (in terms of minimizing Sat) noise seed or given the endpoint æoi
1 (wo, æT) as Sat (xo)) 2 aT w/!" 0 ( (7
(TT
(135)
But the classical action takes its minimum possible value-i.e. i.e., when Dr is chosen be the unique noise seed that corresponds to deterministic PF-ODE dynamics (i.e., d; = f(x t) at all times t), just have
Sa (x0))
(m xT) Rs [aT x71²
(TT x77
(136)
Özrōzι
By Laplace's method, which is also usable due to the smallness of K, obtain
(xx) 0, Sr)
[a(0))] ~ S. (ang an (mo)) det H(wo, (2x)) det (wo, a. (xa))
(137)
V det (a
we only consider 'elassical' paths with de = f(x,, t), as in this approximation, the additional terms Eq. [32 vanish. This means
J,
det H(W 27 (wo) det
Jj
(138)
Q
d
det
det
J
dt
where this manipulation can be more formally justified if one works in discrete time. But since Q/k the inverse V, and since
det ( (d- dl ()-s))
T
ss det exp { {-
{- J5(() di
so det
|[
J:D(]
(139)
38

---

Published as a conference paper at ICLR 2025
where we have used discrete time argument to compute the determinant, have
-2 [- Jz(1) dt
det Ha (2ro) det (( (f)())) = det V det V-1 det exp
detexp
(140)
{ -2 Jz(t) at
= det exp
Note that this determinant can be simplified somewhat:
dt dt log det exp -2 J. Jt(t) dt = -2 f. dt
f Va. f(x,,t dt (141)
where the æ; that appears in the expression above follows deterministic PF-ODE dynamics. Finally,
exp log W(g.(( {"
T
[g(20))
V f(x dt
det (1. (=) Swr@eT
(142)
At this point, we can make crucial observation: the argument of the exponential is precisely the instantaneous change variables formula that can be used to compute the log-likelihnn eeglikelihood p(molee (Song et al. 2021 Chen et al. 2018). This immediately implies
[a(x))] ~ p(æole)
(143)
det (1.
1102 22 (aeg ae (wo)) Un Bar
We conclude that, at least in the small x regime, the learned distribution is equal to the memorized (e- noise-corrupted) data distribution, times the determinant of Hessian that quantifies the likelihood of deviating from PF-ODE dynamics. This Hessian depends on the V-kemne,, since the classical action depends on its inverse Q, is precisely here that the V-kernel can influence generalization.
The required Hessian appears difficult to compute in general. Incidentally, since the relevant action (Eq. 129 generically local in time, is also hard derive a Haanilton-Jacobi-tte differential equation satisfied by this Hessian.
K.3 QUANTIFYING MEMORIZATION IN THE SEMICLASSICAL REGIME
Suppose we quantify memorization by computing the Kullback-Leible (KL) divergence between the data distribution Pdota and the (average) learned distribution [q]:
Pdtata (xo) dx [q(x0)]
Emem DKL (la))
Pdata (x:) log
(144)
By our semiclassical approximation result (Fq. 143 this is just
1 Frnem = DEL. (pdatallpa)) lp:)) + 2 Panta (#o) log det
1 S2 (wo 2 (2)())))))
das
(145)
where Ps p( pa(aul e). Hence, the curvature of the classical action near x (mo) strongly controls the extent to which the data distribution is memorized, especially when e taken to be small, in which case the first term is negligible.
39


Reviews:
[{"review_id": "zvMXLaeCz3", "reviewer": "Reviewer_DoAH", "summary": "The main idea of the paper is that diffusion models generalize since the training step of estimating the score uses the “proxy score” as its target, which is a noisy version of the true score, or more accurately, a pointwise version of the score, conditioned also on the training data point. This introduction of this randomness into the nonlinear estimation process, and then into the diffusion process is claimed to lead to randomness in the generation process, and in turn, to generalization.", "questions": "1. Line 67: What is exactly meant by “boundary regions” ? One example is regions that are close to multiple training points, but how these are defined in general ? As high likelihood regions ? This term is used repeatedly afterwards so it would be good to accurately define it. \n\n2. Line 75: How does the claim “models generalize better when they are somewhat underparameterized” agree with the (Karras et al, 2024) paper mentioned on line 48 ? \n\n3. Line 78: Why interpolation considered a non-trivial generalization ? \n\n4. Line 138: It would be better to introduce the proxy score before (4). \n\n5. How is the operator \\mathcal{D} is defined on (6) ? \n\n6. Section 4: It is explained that even using the generated samples to estimate the score leads generalization. But what is the starting point of the process X_T (the noise point). Isn't this point generated from the training data in a noisy manner, and this also contributes to generalization ? \n\n7. Line 207: Once the higher order terms in (7) are neglected, it is mentioned that this implies that the estimator distribution is approximately Gaussian. However, due to the Gaussianity of the forward process, doesn't this tacitly also approximate the data distribution as Gaussian ? In other words, if we assume that the data distribution is Gaussian, would the result be the approximated integral of (7) ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Understating the empirical success of diffusion models in creating new samples that are similar to the training set is a timely and central question. The analysis made in the paper is dedicated to the generative question, and does not impose generalization measures from other problems (e.g., the MSE of the score function). The intricate details of the diffusion models considered are clearly discussed, and how they affect the generalization process. The paper is well written and conveys its ideas in an interesting way. \n\nAn analytic stochastic path integral technique is used to derive closed-form expressions for the sampling distribution of new samples, with focus on the covariance structure, which highlights the role of the variance of the score proxy in the generation of new samples. It is also a direct performance measure compared to the MSE of the score. This technique is exemplified on two analyzable architectures.", "weaknesses": "1) The paper assures that the generated samples are different from the training samples due to the noise in the score proxy, but this does not seem to explain why the generated samples are meaningful in some sense (e.g., have similar features to the training samples). If one assumes a ground truth distribution then this is obvious, but the authors emphasize that they refrain from that, but it is not clear what replaces this assumption (beyond the generated sampled being different from the training samples). \n\n2) Except for the appearance of a non zero covariance matrix in noise of the reverse diffusion process, the expressions in Propositions 5.1 and 5.2 are somewhat implicit. For example, it is not obvious how easy it is to compute them, or how its form affects generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MKF7tOsMtK", "reviewer": "Reviewer_hd21", "summary": "This paper investigates the generalization capabilities of diffusion models through the perspective of \"generalization through variance,\" a novel concept where the inherent noise in the denoising score matching (DSM) objective used in training significantly influences model generalization. The authors develop a mathematical theory using a physics-inspired path integral approach, which reveals how the covariance structure of the noisy training target impacts inductive biases. The findings suggest that diffusion models are not merely reproducing training data but are capable of filling in 'gaps' that enhance their generative abilities.", "questions": "**1.** On the bottom of page 1 “At present, there is arguably no theory that describes…” It seems there exists a bunch of papers analyzing the generalization ability of diffusion models including their generalization time and its relation to data structure, see [1], [2].\n\n[1] Li, P., Li, Z., Zhang, H., & Bian, J. (2023). On the generalization properties of diffusion models. Advances in Neural Information Processing Systems, 36, 2097-2127.\n\n[2] Fu, S., Zhang, S., Wang, Y., Tian, X., & Tao, D. (2024). Towards Theoretical Understandings of Self-Consuming Generative Models. arXiv preprint arXiv:2402.11778.\n\n**2.** It would be clearer to add more description to Figure 1, including the scale of the variance heat map, etc.\n\n**3**. How does the \"generalization through variance\" compare with other known mechanisms of generalization in diffusion models, such as those related to model capacity and training set structure? In particular, is there a specific formula between V-kernel with any kind of generalization error (or its bound)? If so, the generalization could be more quantitively measured based on V-kernel. \n\n**4**. Could the authors provide empirical evidence or simulations that specifically illustrate the effects of the covariance structure on generalization as hypothesized in the paper?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper's originality lies in its novel theoretical framework, which is a significant leap in understanding the mechanics behind diffusion models' ability to generalize. The quality of the theoretical analysis is high, and the writing is clear. The significance of the work is evident as it gives a new understanding of the generalization of diffusion models.", "weaknesses": "1. The paper's main weakness is the lack of empirical validation of the theoretical claims. While the theoretical intuition is clear, the paper would be greatly strengthened by including simulations or experiments that demonstrate the practical impact of the \"generalization through variance\" phenomenon.\n2. There is a lack of theory directly bridging the gap between variance and generalization error.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XJcWpCgQYo", "reviewer": "Reviewer_AyYq", "summary": "The authors study the generalization in diffusion models. They highlight six key factors influencing generalization: noisy objective functions, forward process dynamics, nonlinear score dependencies, model capacity, architectural features, and the structure of the training set. They \nattribute the generalization of diffusion models to the fact that the training objective is in fact equal to the ground truth score function only in expectation. The key statement is that generalization occurs if and only if the V-kernel is not zero. Then they characterize the generalizability of diffusion models by analyzing the structures of V-kernel under several simplified conditions.", "questions": "See my questions in the weakness part.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Expressing PF-ODE in terms of the path integral is novel and offers important insight i.e, generalization occurs if and only if the V-kernel is nonzero.\n\n2. The paper provides comprehensive study on three distinct cases through the V-kenrl: (i) memorization (ii) linear model (iii) NTK model. These results demonstrate V-kernel as a useful tool toward understanding generalization.", "weaknesses": "(i) The authors claim that generalization occurs if and only if V-kernel is nonzero since otherwise the reverse sampling process can only produce training examples. However, this definition seems restricted since when diffusion models generalize, they not only differ from the PF-ODE dynamics, but generate \"high quality\" samples as well. In other words, the V-kernel should have certain benign properties. This point is not adequately addressed in the current paper.\n\n(ii) In section 5 the authors derive the analytical form of the V-kernels under three circumstances. However, the resulting equations (9), (13) and (17) are really hard for me to interpret. Can the authors provide more explanation on the unique properties of these different V-kernels and how they shape the distributions corresponding to the reverse sampling process?\n\n(iii) As a researcher focuses on empirical works, I don't find the results of this paper very useful. I think the paper could benefit from a further discussion on how the results can help improving practical diffusion models. \n\n(iv) Overall, despite the weaknesses I listed above, I think the results are interesting and worth being published.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)