On the Interpolation Effect of Score Smoothing in Diffusion Models
Authors: Zhengdao Chen
Venue: ICLR 2026 Poster (2026)
Keywords: diffusion model
empirical score function
norm-bounded neural network
barron norm
function smoothing
subspace recovery
Primary area: learning theory
TL;DR: Diffusion models can avoid memorization when the neural network learns smoother versions of the empirical score function, which causes the generated samples to interpolate between the training data along the data subspace.

Abstract: Score-based diffusion models have achieved remarkable progress in various domains with the ability to generate new data samples that do not exist in the training set. In this work, we study the hypothesis that such creativity arises from an interpolation effect caused by a smoothing of the empirical score function. Focusing on settings where the training set lies uniformly in a one-dimensional subspace, we probe the interplay between score smoothing and the denoising dynamics with analytical solutions and numerical experiments. In particular, we demonstrate how a smoothed score function can lead to the generation of samples that interpolate among the training data within their subspace while avoiding a full memorization. Moreover, we present theoretical and empirical evidence that learning score functions with neural networks - either with or without explicit regularization - can indeed achieve a similar effect as score smoothing, including when the data belongs to simple nonlinear manifolds.

ON THE INTERPOLATION EFFECT OF SCORE SMOOTH- ING IN DIFFUSION MODELS
Zhengdao Chen
Google Research
Mountain View, CA 94043, USA zhengdac c3 @mmall com
ABSTRACT
Score-based diffusion models have achieved remarkable progress in various do- mains with the ability to generate new data samples that do not exist in the training set. In this work, we study the hypothesis that such creativity arises from an inter- polation effect caused by smoothing of the empirical score function. Focusing settings where the training set lies uniformly in a one-dimensional subspace, we probe the interplay between score smoothing and the denoising dynamics with demonstrate how
analytical solutions and numerical experiments. In particular, a smoothed score function can lead to the generation of samples that interpolate among the training data within their subspace while avoiding a full memorization. Moreover, we present theoretical and empirical evidence that learning score func- tions with neural networks either with or without explicit regularization can indeed achieve a similar effect as score smoothing, including when the data belongs to simple nonlinear manifolds.
INTRODUCTION
Score-based diffusion models (DMs) have become an important pillar of generative modeling across variety of domains from content generation to scientific computing (Sohl- -Sohl-Dickstein et al. 2015 Song and Ermon 2019: HH et al.) 2020; Ramesh et al.) 2022) Abramson al., 2024,] Brooks et al. 2024). After being trained on datasets of actual images or molecular configurations, for instance, such models can transform noise samples into high-quality images or chemically plaussible molecules that do not belong to the training set, indicating an exciting capability of such models to generalize beyond what they have seen and, in a sense, be creative.
The theoretical mechanism of the creativity of score-based DMs has been topic of many discussions. At the core these models is the training of neural networks (NNs) to a series of target functions, often called the empirical score functions (ESFs). which drive the denoising process at inference time. The precise form of the ESFs is determined by the training set and can in principle be computed exactly, but when equipped with the exact ESF instead of the NN-leamed version, the DM will end up generating data points that already exist the training set (Yi et al. [023 Li et al.) 2024a), a phenomenon commonly called memorization. This suggests that, for the models to generalize fresh samples beyond the training set, it is crucial to have certain regularization on the score function through NN training that prevents the ESF from being learned exactly. For example, for image generation, Kamb and Ganguli (2024) showed that DMs with convolutional NN (CNN) as the score estimator can gain their creativity through CNNs' inductive biases including equivariance and locality. One may ask: can there be more general mechanism behind the creativity of DMs beyond image generation and CNNs?
Our work takes inspiration from an interesting hypothesis by SSarvellis et al. (2023) that smoothing the ESF allows DMs to generate samples that interpolate among the training data, which motivated their proposal of closed-form score estimator for DMs based on local convolutions. In this work, we bring the hypothesis further by studying how score smoothing can happen through NN learning and how it quantitatively affects the denoising dynamics. To gain concrete insights, focus simple setup with uniformly-spaced training data and analyze score smoothing as well as the denoising dynamics mathematically, and main contributions two-fold:

---

1. We give theoretical and empirical evidence that regularized two-layer ReLU NNs tend to approximately learn a smoothed version of the ESF (named the "Smoothed PL-ESF"). The theory is based analyzing a non-parametric variational problem involving the score matching loss and a non-smoothness penalty known as equivalent to NN regularization;
2. Through analytical solutions of the denoising dynamics, we show that a DM under the Smoothed PL-ESF produces non-singular density that interpolates the training set. In particular, when the training data lie in a one-dimensional (1-D) subspace, the denoising dynamics is able to recover the underlying subspace without collapsing onto the training set.
Together, these results shed light on how score smoothing can be an important link for understanding how NN-based DMs avoid memorization.
The rest of the paper is organized as follows. After briefly reviewing the background in Section we examine the smoothing of ESF in the 1-D case and discuss its connections with NN regularization in Section The trajectory of the denoising dynamics under the Smoothed PL-ESF is derived in Section 4 In Section 5, we generalize the analysis to the multi-dimensional case when the training data belongs to hidden subspace. In Section 6 we present empirical evidence that NN-learned SF exhibits interpolation effect similar to that of score smoothing, including when the data belongs to nonlinear manifold. Due to space limitation, we defer the discussion of related works to Appendix
exp(=2²/(2o") Notations For x, > O, we write PAr a) = -Vx2)))eepp(22
for the 1-D Gaussian
density with mean zero and variance o², 8g for the Dirac delta distribution centered at R. and sgn(z) for the sign of æ. We write [n] == {l,... nu for n E N4. For a vector æ = [x] Ia] E Rd.
write (zr): x; for E [d]. The use of big-O notations is explained in Appendix
2
BACKGROUND
While score-based DMs have many variants, we will focus on a simplest one (named the "Variance Exploding" version by Song et al. 2021b) where the forward (or noising) process is defined by the following stochastic differential equation (SDE) in Rd for t 2 0:
dx:
dw:, , X po
(I)
where w is the Wiener process (a.k.a. Brownian motion) in Rdd The marginal distribution of x,, denoted by Pt, is thus fully characterized by the initial distribution po together with the conditional distribution, Paoo (xlaaa) = II= p><( - [a']:: vf): specifically, Pt. is obtained by convolving Po with an isotropic Gaussian distribution with variance (t)² t in every direction.
A key observation is that this process is equivalent (in marginal distribution) to deterministic dynamics, often called the probability flow ordinary differential equation (ODE) (Song et al. 202
dx: = - {x:)
where st (x) V log (x) the score function (SF) associated with the distribution Pe (Hyvärinen and Dayan 2005) and is also fundamentally connected to the denoiser function via Tweedie's 's formula (Efron| 2011). l). In generative modeling, Po is often a distribution of interest that is hard to sample directly (e.g. the distribution of cat images in pixel space), but when T is large, pr is close to Gaussian distribution (with variance increasing in T'), from which samples are easy to obtain. Thus, to obtain samples from Po. may first sample from pT and follow the reverse (or denoising) process by simulating (2) backward-in in-time (or its equivalent stochastic variants, which we will not focus on). A main challenge in this procedure lies in the estimation of the family of SFs, V log Pt for t E [o, In reality, we have no prior knowledge of each Pt (or even Pu) but just a training set S = (yx kejjj usually assumed to be sampled from pn- Thus, we only have access to an empirical version of the noising process, where the same SDE (1 is initialized at i = 0 with not po but the
uniform distribution over S (i.e., X p(") 1. C and hence the marginal distribution of
(n) (z: == Ek (1) yk), called the noised empirical distribution at time t. To obtain x₁ is Pt.
proxy for V log one often uses an NN as a (tme-dependent) score estimator, se (æ, f), and train its parameters minimize variants of the averaged score matching loss (Song et al. 202
("
I (m) [so( ,t)ddt
min 0
T
2

---

Figure 1: From the noised empirical distribution (p)"), middle), denoising with the ESF (V log p. leads back to the empirical distribution of the train- ing set (po ()) top), while using a smoothed SF (e.g. the Smoothed PL-ESF, 8 NN-learned SF) produces distribution that interpolates among the training set on the relevant subspace (e.g., Pb =) in the case of Smoothed PL-ESF; bottom). Defini- tions are given in Sections 2
Add
Denoise vin Empirical
Denoise vin Smoothed SF
where
(1) [ xx=p!"))
[1I((x)
logp:" (n) (x) [x)
measures the L distance between the score estimator and V log (*) - which is the ESF at time with respect p The scaling factor of 1/E V log (x, serves to balance the contribution to the loss at different (Song et al. 2021b).
Though in practice the minimization problem is solved via NN optimization and Monte-Carlo sampling (Vincent 201T), we know the minimum is attained uniquely by the ESF itself, which can be computed in closed form (e.g. see Section B). we use the ESF directly in the denoising dynamics (2) instead of an NN-learned SF, we get an empirical version of the probability flow ODE:
dx; = 2 log (m) (x:)
(5)
which exactly reverses the empirical forward process adding noise to the training set, and hence the outcome at t 0 is inevitably Po (xs) In other words, the model memorizes the training data. This suggests that the creativity the diffusion model hinges on a sub-optimal solution to the minimization problem 3 and imperfect approximation the ESF. Indced, the memorization phenomenon has been observed in practice when the models have large capacities relative to the training set size (Gu et al.) 2023 Kadkhodaie al. 2024), which results in too good approximation to the ESF. This leads to the hypothesis that regularized score estimators - in particular, those that learn to smooth the ESF give rise the model's ability generalize beyond the training
In Sections 3 we provide theoretical evidence for this hypothesis in the 1-D setting. We first argue that NN regularization biases the score function towards smoothed version of the ESF by analyzing variational problem involving the score matching loss and the non-smoothness measure associated with the weight norm of two-layer ReLUU NNs. Then, we show analytically how the smoothed score drives the denoising dynamics to generate distributions that interpolate between instead of memorizing the training data.
SCORE SMOOTHING IN ONE DIMENSION
Let begin with the simplest setup where d 1 and S {y1 -1 1} consists of
points (whereas in Appendix C we prove extensions all results in Sections 3 5 to the setup where S consists of n uniformly-spaced points). At time the noised empirical distribution is (n) Pi (x) = (px(( 1iv() + pv L;))), and the (scalar-valued) ESF takes the form
d log (m) (z) (= x)/t,
(6)
where
(x) = Eoj X X2 = x] px(a- LWD pr
(7)
px IVA))
lies between +1 and has the same sign x. Ast - O. the Gaussians sharpen and a.(²) (a:) approaches sgn(z), allowing us to approximate the ESF by piece-wise linear (PL) function, named the PL-ESF:
(x) (sgn (x) x)/1,
(8)
which explains the attraction the backward dynamics to +1, corresponding to memorization. We will show below that this behavior crucially be avoided by smoothing the ESF at small t.
3

---

glm) 5t
20
0
0ogs
-20
-2
0 X
-2
-1 65
0
1-
2
Figure 2: Similarities between NN-learned SF (xx))) under increasing strengths of regularization, À (left) and the Smoothed PL-ESF (8 with decreasing values of (right) in the d 1, case with fixed t. Details of the experiment setup are discussed in Section J.1
3.1 SCORE SMOOTHING VIA NN REGULARIZATION A THEORETICAL MODEL
To gain intuition, we perform experiments to fit the ESF (6 at a fixed t using the loss by two-layer ReLU NNs that are regularized by weight decay (additional details given in Appendix J.I). As shown in Figure 2 for various strengths of regularization (denoted by A) the NN-learned score estimators nearly PL, and remarkably, well well-approximated by the following ansatz parameterized by E (0, 1]:
- fa + 1)/t. 4.6 (x) = - (a 1)/6,
if < 8 - 1, if 2 1 6
6/(1 8) x/t,
if (6 1).
In particular, a stronger regularization corresponds smaller 8. We will refer ((x) Smoothed 62,5
PL (S-PL) ESF. As illustrated in Figure (right), it is PL and matches s5()) except on the interval [6 - 1-,11 -1,1-8 (hence ((x)) = =(")), In Appendix I.1 we show that (9) can be obtained by averaging the PL-ESF over local windows.
Why do the regularized NNs learn score estimators that are so close to being expressed by 9 While it is difficult predict exactly what function an NN will learn due to the nonlinearity and stochasticity of the training dynamics, below we will provide an argument based on a non non-parametric view of NN regularization. Specifically, for function fitting in one dimension, it is shown in (Savarese et al. (0199) that regularizing the weight norm of two-layer ReLU NN (with unregularized bias and linear terms) - equivalent to applying weight decay during training - is essentially equivalent to penalizing non-smoothhess measure of the estimated function defined as:
Rif /
1f" (d)
(10)
where f" is the weak second derivative of the function f. Note that even without explicit weight norm regularization, this term has also been associated with the implicit regularization effect that gradient-based training induces in two-layer ReLU NNs (Chizat and Bach 2020).
Inspired by this connection, for e, 0, we consider the following family of variational problems in function space as a proxy for NN learning:
rī. = inf
R[f]
S.t. L. I <c.
(11)
f
with the infimum taken over all functions f on R that are twice differentiable except on a finite set (a broad class of functions that include, e.g., any function representable by finite-width NN). Heuristically, we seek to minimize the non-smoothness measure among functions that are e-close to the ESF according the score matching loss (3). When
only the ESF itself satisfies the constraint and hence attains the minimum uniquely; if is small but positive, the feasible set is infinite and we need study how the con-smotthness penalty biases the score estimator away from the ESF,
Due to the non-differentiabiiiiyy of the functional R. the variational problem (IT) is also hard to solve directly. Nevertheless, we can show that near-optimality can be achieved by the Smoothed PL-ESF uniformly across small when choose ă depend proportionally Vi:
4

---

1k
With the motivations discussed above, we now study the effect on the denoising dynamics of ((n) Figure 3: Phase diagram in the
(A)
(B)
16 t
v plane for the flow solution (4)
(C)
of the dynamics (T3) in the d
2 case analyzed in Section
1
X
Proposition 1 Given (0, 0.015). for any 2 (c), where F is computable function that decreases strictly from 1 to O on [0, oo), there exists t > 0 (dependent on k) such that 1,6, with 5. NVL satisfies the following two properties for all (0,11):
s(n)
1. Lt" [m] [ < and hence the function s/m) belongs to the feasible set of 11
2. R 1,6
< (1 + 8
Outline of proof (full proof given Appendix D The first property follows from the lemma below:
Lemma Let 8 = for some &. > Then 36, C > (depending on k) such that Vt E (0,t1).
f1).
- P F(x) (((K)CCV CVi.
(12)
Lemma (proved in Appendix relies on the insight that when small, pt (') is concentrated near +1, and hence _ is dominated by contributions from the neighborhood of =1. In particular, for
_(") remain constant level, we can afford to decrease 8, x Vt as +
For the second property in Proposition 7 we observe that when t is small, for any function belonging the feasible set with small enough e, its derivative near +1 needs be close d dloog (m)) (.o /dx z (m) near +1). Combined with the fundamental theorem
-1/t (again because of the concentration of Pt of calculus, this gives lower bound ri,--
Proposition and Lemma 2 thus establish that the Smoothed PL-ESFs with ò x Vt are nearly minimizers of the non-smootheness measure in the function space while maintaining small bounded errors across small In Section [6] we will further show empirical evidence that when we train single NN with regularization to learn the time-dependent SF through the time-averaged score matching loss the solutions are indeed also closely approximated Smoothed PL-ESF.
4 INTERPOLATION EFFECT ON THE DENOISING DYNAMICS
substituting the ESF (6) with 6,6 where d, kVT for some x >> O, that is, replacing (5) by:
X - (x,)
(13)
Thanks to the piece-wise linearity of O the backward-in-time dynamics of the ODE 13 be solved analytically in terms of flow maps:
Proposition 3 For $ 1/m2, the solution to
satisfies x, = &e(e((x)) where
(1 - 6,)/ -1)) x
if 16, 1, 8el if <
Φ.f((2)
VB/Vt-2 x - 1aat-==-11///1111 v3/Vi).
(14)
V8/Vt. x + vs/Vi)
if 2 1 - de
The proposition is proved in Appendix and we illustrate the trajectories characterized by dxj: in Figure 3 The differentiability profile of & divides the vt plane into three regions (A, B and C) with linear boundaries, defined by -1 &c. 2 5, and & S 8r, respectively,
[m)
5

---

Note that (8) allows us to prove KL-divergence bounds for p (((.fo) based on those of p. (2.1o) and an (n) which is and trajectories induced by 6aje do not cross the region boundaries. If at to 0, Xto falls into region (or B), then as t decreases to O, it will follow linear path in the vt plane to y1 (or 3/2 1). Meanwhile, if Xt, falls into region C, then it will follow a linear path to the r-axii with terminal value between -1 and 1. In other words,
x1(11 - 6c) Φ (x)
if E [6, 1, -dc otherwise
(15)
sgn(
Evolution of marginal distribution Suppose we start from some to E (0, 1/x²) and run the denoising dynamies (33) backward-in-time and we denote the marginal distribution of x; by Pr (f,.
(n) for t E [0, fol. We assume that pf sfm.to) Pto is the noised empirical distribution at time to. Since the map @aje invertible and differentiable almost everywhere when 0 < S S la. we can apply the Change-of-variabe formula of push- furh-forward distributions to obtain an analytic expression for the density pa to),
(1 11-51)/(1- - 52) p.m ((1 61)/(1,)1111) -5)) 52).2) x),
(n,
if E [6, - 1,1 5,1
p plntoo()))
6 D p(m (8r/dg x - (6)-6.)/6.) 6,))
if S 8, - if 2 6,
(16)
8,/5, (0) (5./6.
(6. 6,))
and its evolution decreases from to to 0 is visualized in Figure map). When = 0, Φnj invertible only when restricted to [ö, 1, 1 - 6t), and the terminal distribution can be decomposed
(the lower grey-colone, heat
(n,to) = aa+0 81 + a 8-1 + (1 a. a >/b (o) Po
(17)
ta) is a probability distribution satisfying where a+ E (m))0) + 6.p and (((m.aa
x
1n) (x) (()
( (1 dt.) a+ a-) (m.to) ((1 ((1-dt.)a)) )z),
if E [-1,1]
(18)
otherwise
In particular, since pl on [-1, 1], corresponding smooth interpolation between the two training data points.
(m,tt) has a positive density on [o., 1, 11,1-8a Se,l, Po
_((,,to) also has positive density
example is given in Appendix I.2 In contrast, denoising with the exact ESF results in po
fully singular and has an infinite KL-divergence with any smooth density on 11.
5 HIGHER DIMENSION: SUBSPACE RECOVERY WITH SCORE SMOOTHING
Let us consider a case where S = { 1,,, 0j. [1, ........] 0}} C Rd consists of two points on the Iaxis (and in Appendix
C we generalize the analysis to the case where S contains uniformly-spacee points in any 1-D subspace). In this case, the noised empirical density is Pi
(n) (x: (p=) 1; /t) + PN((=)) 1; v() I (M0l vt), and the (Vector-valued)
ESF is given by V log (1) (x)) |0 log (m) (x), .... Os log (m) (x)), where
a, log (n) (x) = ((e)) - (x)))//,
(19)
Vi f [2,....
0 log Pt (x: [t),/t,
where (()))) is defined in the same way as in Relative to the subspace to which the training set belongs the laxis- we may refer to the first dimension as the tangent component and the other dimensions as the normal component. The tangent component of the ESF behaves the same
'his can be viewed as starting from the noised empirical distribution at some large time I' (nearly Gaussian), initially denoising via the ESF until fo. then switching to Smoothed PL-ESF for the rest of the denoising process until t O. An equivalent interpretation is that we add noise to the training data for time to before denoising them with the Smoothed PL-ESF for the same amount (as illustrated in Figure l.

---

as in the 1-D setting, whereas its normal components are linear functions and explain the uniform collapse onto the 1-D subspace during denoising.
To understand score smoothing in this context, study extension of the variational problem to the multi-dimensional case through a gencralization of the non-smoothness measure (TO) by:
R
f + bl
(20)
(w. (B (B)
for f : - Note that ((d) reduces to 110 when d = 1 and is invariant to coordinate rotations
and translations
We can now consider analog of (11) for >> 1 by having (20) as the objective:
Ti,,d = inf
s.t.
L
[ <e,
(21)
f
with the infimum taken over all piece-wise twice-differentiable functions f Rd Rd.
Analogously to the 1-D case, we show that near-optimality can be achieved by vector-valued functions a(m)
(n) of the following type, 6,5
_d + Rd where 1,8 -R + R is defined the same way as in 99:
% (a) == (()) - [#/22/t,..... - [x]a//]]
(22)
22 can be viewed as defining generalization of the Smoothed PL-ESF higher dimensions.
Proposition 4 Given any fixed (0, 0.015), if we choose 6, kV² with any 2 F-) then there exists t > (dependent on k) such that the following holds for all t E (0,t)):
(n) <G Lt 2,6.
[8()) 2,61
(m)
< + 1++8)rrr..d
The proof is given in Appendix | Fnd builds on Proposition[ Tb leveraging two main observations: (1)
(m) is product distribution between the tangent and normal dimensions; (2) the normal components of the ESF are fully linear and do not incur "additional" non-smoothness penalty under (20).
Pt
Implication for denoising dynamics Motivated by Proposition 4 and similar in Section 4
consider a denoising dynamics under the smoothed score given by dxx noticeably decoupled across the different dimensions:
1
2 t.6, (x:), which is
di [x1] = - 1 1, ([x(l1),
(23)
Vi {2.....
de
[x,]= = [x/]/tt
(24)
Based on our analysis in the case, we know that this dynamics has the following solution:
Proposition 5 For 0 S 8 < 1/x2, the solution of 23
24 is given by x, = F (xx) =
(xx1)() /s/t|xx22.... /s/t[x]]d with dxit defined as 14 15). Hence, if run backward-
in-time with the marginal distribution of X1o being Pio PN : Voo) x ... PN(( ; Vooo, at t E [0, Lo), has marginal distribution p. ((,.t)) x PPN( :Vt) x x PN( Vt), where pi (n,,oo)
(n,to)
satisfies (16) when and (7 when i =0
Crucially,
distinct dynamical behaviors in the tangent versus normal dimensions. As the trajectory converges to zero at rate of vi. in the normal directions and results in a uniform collapse onto the aaxis (as in the case without score smoothing). In the tangent dimension, meanwhile, smoothing phenomenon happens similarly the 1-D case. In particular, if the marginal
distribution of [x4]1 has a positive density on jōg - 1, 5e), then so will [xo]1 1 on [-1 1,, 1]. meaning that Xa has non-singular density that interpolates smoothly among the training data on the desired 1-D subspace.
We note that our definition of R(d) differs from the complexity measure in function space associated with regularized two-layer ReI.U NN multi-dimensional inputs, which has more involved definition via the Radon transform and fractional powers of I Laplacians (Ongic al. 2020
7

---

0.5
log density
[x]
t= to
0.1
-0.5
[x]
[x]
8: V log (n)
(x)
NN (NN-learned SF)
1.6
(ESF)
(Smoothed PL-ESF)
0.5
[x]
-0.5
t- to/4
log density
[x] 0.0
-0.3
t= 0
log demsity
[x]
[x]
Figure 4: Results of the experiment in Section 6.1 Each column shows the denoising process under
(w) one of 3 choices of SFs, which starts from the distribution Pto at to and evolves backward-in-tiin following the respective SF. At t to. to/4 and tinn 10-5 we plot (a) the samples from the denoising processes in IR? and (b) the density histograms (log scale) their first dimension. In (b), the colored curves are the analytical predictions of Pt with the formulas given in Appendix C.2
.(n,fa) (for to. lo/4 and Po
_((,fa) (for = (min).
Contrast with inference-time early stopping The effect of score smoothing is different from what can be achieved by denoising under the exact ESF but stopping it at some tmin > 0. In the latter case, the terminal distribution is still supported in all dimensions and is equivalent simply corrupting the training data by Gaussian noise. Hence, without modifying the ESF, early stopping alone does induce proper generalization behavior.
NUMERICAL EXPERIMENTS
6.1 DENOISING SMOOTHED PL-ESF AND NN-LEARNED SF (d 2, 4)
To validate theoretical analysis on the effect of score smoothing the denoising dynamics,
we choose the setup in Section 5 with = and n = 4 and run the denoising dynamics (2) under
three choices of the SF: (i) the ESF (s; V log Pi (n)) ), (ii) the Smoothed PL-ESF (s: $..6. from
mm))
22 and (iii) an NN-learned SF with as an input (s; s NNN). All three processes are initialized at
(n) to 0.02 with the same marginal distribution Xt, Pto and run backward-in-time to 4min 10-5
The results are illustrated in Figure We first observe that, all three cases, the variance of the data distribution along the second dimension shrinks gradually to zero at roughly similar rate as + O, consistent with the argument Section 5 that score smoothing does not interfere with the convergence in the normal direction. Meanwhile, in contrast with Col. (i), where the variance along the first dimension shrinks to zero as well, we see in Col. (ii) that the variance along first dimension remains positive for all validating the interpolation effect caused smoothing the ESE
The detailed setup of all experiments is given in Appendix
8

---

train
t-
[- Emin
[x]2
[x]2
NN SF
[x]2
ESF
[x]1
[x]1
[x]
Figure 5: Experiment in Section 6.2 with training data spaced uniformly on the unit circle in R?. Top: Samples from the beginning and end of the denoising process with NN-learned SF. Middle and bottom: Visualization the NN-learned SF vs ESF at / 4n/8 as vector fields, with the length corresponding the magnitude and the color determined by their angular direction (red for clockwise, blue for counter-clockwise).
Moreover, the density histograms in Col. (ii) are closely matched by our analytical predictions of _(n,to) (the colored curves). Finally, we observe that Col. (iii) much closer to (ii) than
Pt ((m..o) and Pb (i) in terms of how the distribution (as well as the SF itself, as shown in Figures 7 evolves during denoising. This suggests that NN learning causes similar smoothing effect on the SF and supports the relevance of our theoretical analysis for understanding how NN-based DMs avoid memorization.
6.2 DATA ON CIRCLE
To show that the effect of score smoothing goes beyond linearly-spaced data, consider training sets spaced uniformly on a circle in R2 and train 2L NNs without weight decay to fit the ESF and drive the denoising dynamics, with results illustrated in Figure As shown in the last two rows, the NN-learned SF is evidently smoother than the ESF, especially in the polar (i.e., tangent) direction. Notably, this leads the denoising dynamics training data points a nearly linear fashion, which forms a regular polygon that approximates the underlying circle better as the number of training data increases. That this phenomenon occurs without weight decay in NN training also suggests that implicit regularization from the optimization dynamics can be sufficient induce the score smoothing effect (see discussion in Appendix A
generate samples that interpolate between nearby
In addition, we observe that NN-learned SF also tend to be smoother than the ESF when the training data are conuuniformly spaced in 1-D. The results are illustrated in Figure 10 Appendix [.2
6.3 CURVED MANIFOLDS HIGHER DIMENSIONS
this section, we empirically demonstrate that the interpolation effect continues to exist when training data belong to manifolds in higher dimensions. A challenge for this goal is the difficulty to visualize interpolation phenomena in high dimensions. To overcome this challenge, we consider training data belonging to 2-D spherical-type manifolds embedded in higher dimensions (up to 20), which amenable to visualization via stereographic projection on to a flattened latent space (Figure TT). Further details of the latent embedding and the experiment setup given in Appendix [.4

---

5
train
-1
-1.5 0.5
21
U2
U1
Figure 6: Results of the experiment in Section 6.3] where the training set is randomly sampled from a 2-D spherical-type manifold embedded in d 20. Under different coordinate systems and views, the plots each illustrate a) the samples generated by denoising diffusion with the score function learned by two-layer NN (green dots), b) their starting positions before denoising (gray dots), and c) the training set (black "plus" signs). Left: In the canonical 3-D subspace containing the manifold with coordinates 21,, #2, 23] T. Right: In latent coordinates produced by stereographic projection (Appendix T. T.4). i.e., u = [u1, u2 ua]* == ((). Bottom: 2-D-sliced views in the u1-12 plane.
We consider training sets both randomly sampled and regularly spaced and train two- and three-layer NNs (without weight decay) as score estimators that drive the denoising dynamics. The generated samples are visualized in Figures 6 and 12 |2-[14] We observe that NN-leamed score estimators produce samples that interpolate the training data along the data manifold despite its nonlinearity and the higher dimensionality of the data, suggesting that our analysis on the interpolation effect goes beyond linear subspace settings.
7 CONCLUSIONS AND LIMITATIONS
Through theoretical analyses and numerical experiments, our work shows how score smoothing can enable the denoising dynamics produce distributions on the training data subspace without fully memorizing the training set. Further, by showing connections between score smoothing and score learning using NNs, our results shed light on an arguably core mechanism behind the ability of NN-based diffusion models to generalize, hallucinate create. Additionally, NN learning is just one way to achieve score smoothing, our work also motivates the exploration of alternative score estimators that facilitate generalization in DMs, including the proposal by (Scarvelis et al. (2023).
The present work focuses on a vastly simplified setup compared to real-world scenarios. First, would be valuable extend theory
where training data are generally spaced, random or belonging to complex manifolds as well as to more general variants of DMs (De Bortoli et al. 2021 Albergo al. 2023 Lipman al. 20023 Liu al. 2023). Second, score estimators in practice typically have more sophisticated architectures than 2L ReLU NN, which can lead to more complex and nonlinear interpolation effects (Kamb and Ganguli, 2024). The connections between score smoothing and the implicit bias of NN training have also only been explored to a limited extent, especially the higher-dimensional setting (see Appendix A). Lastly, will be useful to consider alternative forms of function smoothing and other regularization mechanisms beyond smoothing
(Wibisono et al. 2024 Baptista et al. 2025).
Reproducibility statement The detailed setup of all experiments is given in Appendix and the proof of all theoretical results can be found in Appendices D
REFERENCES
Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L.. Ballard, A. J., Bambrick, J., et al. (2024). Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pages 1-3.
Achilli, B., Ventura, E.. Silvestri, G., Pham, B., Raya, G., Krotov, D., Lucibello, C., and Ambrogioni. L. (2024). Losing dimensions: Geometric memorization generative diffusion. arXiv preprint arXiv:2410.08727
10

---

Aithal, S. K., Maini, P., Lipton, Z. C., and Kolter, J. Z. (2024). Understanding hallucinations in diffusion models through mode interpolation. The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Albergo, M. S., Boffi, M., and Vanden-Eijnden, E. (2023). Stochastic interpolants: unifying framework for flows and diffusions. arXiv preprini arXiv:2303.08797,
Azangulov, I., Deligiannidis, G., and Rousseau, (2024). Convergence of diffusion models under the manifold hypothesis in bigh-dimensions. arXiv preprint arXiv:2409.18800
Baptista, R., Dasgupta, A., Kovachki, N. B., Oberai, A., and Stuart, A. M. (2025). Memorization and regularization in generative diffusion models. arXiv preprint 2xXv:2501.11785
Benton, J., Bortoli, V. D., Doucet, A., and Deligiannidis, G. (2024). Nearly SdS-linear convergence bounds for diffusion models via stochastic localization. The Twelfth International Conference on Learning Representations.
Biroli, G., Bonnaire, T., De Bortoli, V., and Mézard, M. (2024). Dynamical regimes of diffusion models. Nature Communications, 15(1):9957.
Block, A., Mroueh, Y., and Rakhlin, A. (2020). Generative modeling with denoising auto-encoders and langevin sampling. arXiv preprint arXiv:2002.00107.
Boffi, N. M., Jacot, A., Tu, S., and Ziemann, I. (2025). Shallow diffusion networks provably learn hidden low-dimensional structure. In The Thirteenth International Conference on Learning Representations.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. (2018). JAX: composable transformations of Python+NumPy programs.
Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L.., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., and Ramesh, A. (2024). Video generation models world simulators.
Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramèr, F., Balle, B., Ippolito, D., and Wallace, E. (2023). Extracting training data from diffusion models. In Proceedings of the 32nd USENIX Conference Security Symposium, SEC '23, USA. USENIX Association.
Chang, S.-.., Cosman, C., and Milstein, L.. (2011). Chernoff-type bounds for the gaussian error function. IEEE Transactions on Communications, 59(1 (9(11):293-2944
Chen, H., Lee, H., and Lu, (2023a). Improved analysis of score-based generative modeling: User- friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning, pages 4733-4763. PMLR.
Chen, M., Huang, K., Zhao, T., and Wang, M. (2023b). Score approximation, estimation and distribution recovery of diffusion models low-dimensional data. In International Conference Machine Learning, pages 4672-4712. PMLR.
Chen, S., Chewi, S.. Li. J., Li, Y., Salim, A., and Zhang, A. (2023c). Sampling is as casy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference Learning Representations.
Chen, S., Kontonis, V., and Shah, K. (2024). Learning general gaussian mixtures with efficient score matching, arXiv preprint arXiv:2404. 18893.
Chizat, L. and Bach, F. (2020). Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory, pages 1305-1338, PMLR.
Cole, F. and Lu, Y. (2024). Score-based generative models break the curse of dimensionality in learning a family of sub-gaussian distributions. In The Twelfth International Conference on Learning Representations.
11

---

De Bortoli, V. (2022). Convergence of denoising diffusion models under the manifold hypothesis. Transactions on Machine Learning Research. Expert Certification.
De Bortoli, V. Thornton, J., Heng, J., and Doucet, A. (2021). Diffusion schrödinger bridge with applications to based generative modeling. Advances in Neural Information Processing Systems, 34:1 34:17695-17709,
Efron, B. (2011). Tweedie's formula and selection bias. Journal of the American Statistical Association, 106(496),1600-11614
Gao, W. and Li, M. (2024). How do flow matching models memorize and generalize in sample data subspaces? arXiv preprint arXiv:2410.23594.
Gatmiry, K., Kelner, J., and Lee, H. (2024). Learning mixtures of gaussians using diffusion models. arXiv preprint arXiv:2404.1888 18869.
Gu, X., Du, C., Pang, T., Li, C., Lin, M., and Wang, Y. (2023), On memorization in diffusion models. arXiv preprint 2rX1v:2310.02664.
Han, Y., Razaviyayn, M., and Xu, R. (2024). Neural network score estimation in diffusion models: Optimization and generalization. In The Twelfth International Conference on Learning Representations.
Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851.
Huang, D. Z., Huang, J., and Lin, Z. (2024a). Convergence analysis probability flow ode for score-based generative models. arXiv preprint arXiv:2404.09730.
Huang, Z., Wei, Y., and Chen, Y. (2024b). Denoising diffusion probabilistic models are optimally adaptive to unknown low dimensionality. arXiv preprint arXiv:2410.18784
Hyvärinen, A. and Dayan, P. (2005). Estimation of non-oormllized statistical models by score matching. Journal of Machine Learning Research, 6(4).
Ji, Z. and Telgarsky, M. (2019). Gradient descent aligns the layers of deep linear networks. In International Conference on Learning Representations.
Kadkhodaie, Z., Guth, F., Simoncelli, E. P., and Mallat, S. (2024). Generalization in diffusion models arises from geometry-adaptive harmonic representations. In The Twelfth International Conference on Learning Representations.
Kamb, M. and Ganguli, S. (2024). An analytic theory of creativity in convolutional diffusion models. arXiv preprint arXiv22412.20292
Kamkari. H.. Ross, L.. Hosseinzadeh, R., Cresswell, C., and Loaiza-Ganem, G. (2024). A geometric view of data complexity: Efficient local intrinsic dimension estimation with diffusion models. In JCML. 2024 Workshop on Structured Probabilistic Inference Generative Modeling.
Karras, T., Aittala, M., Aila, T., and Laine, S. (2022). Elucidating the design space of diffusion-basee generative models. Advances in neural information processing systems, 35:26565-26577
Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y., editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Lee, H., Lu, J., and Tan, Y. (2022). Convergence for score-based generative modeling with polynomial complexity, Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K., editors, Advances in Neural Information Processing Systems.
Li, P., Li, Z.. Zhang, H., and Bian, J. (2023). On the generalization properties of diffusion models. In shirth-seventh Conference on Neural Information Processing Systems.
Li, S., Chen, S., and Li, Q. (2024a). A good score does not lead to a good generative model. arXiv preprint arXiv:2401.0488
12

---

Li, X., Dai, Y., and Qu, Q. (2024b). Understanding generalizability of diffusion models requires rethinking the hidden gaussian structure. In The ehirty-eighth Annual Conference on Neural Information Processing Systems.
Lipman, Y., Chen, T. Q.. Ben-Hamu, H., Nickel, M., and Le, M. (2023). Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations.
Liu, X., Gong, C., and qiang liu (2023). Flow straight and fast: Learning to generate and transfer data with rectified flow. The Eleventh International Conference on Learning Representations.
Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. In International Conference Learning Representations.
Lyu. and Li, (2020). Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations.
Oko, K., Akiyama, S., and Suzuki, T. (2023). Diffusion models are minimax optimal distribution estimators. In International Conference on Machine Learning. pages 26517-26582. PMLR.
Ongie, G., Willett, R., Soudry, D., and Srebro, N. (2020). A function space view of bounded norm infinite width relu nets: The multivariate case. In International Conference on Learning Representations.
Peebles, W. and Xie, S. (2023). Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205.
Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. (2018). Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
Peyré, G. (2009). Manifold models for signals and images. Computer vision and image understanding, 113(2):249-28) 2600.
Pidstrigach, (2022). Score-based generative models detect manifolds. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K., editors. Advances in Neural Information Processing Systems.
Potaptchik, P., Azangulov, I., and Deligiannidis, G. (2024). Linear convergence of diffusion models under the manifold hypothesis. arXiv preprint arXiv:2410.09046.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C.,, and Chen, M. (2022). Hierarchical text-conditional image generation with clip latents. arXiv preprint 200.:2204.061255, 1(2):3.
Savarese, P.,
Evron, I., Soudry, D., and Srebro, N. (2019). How do infinite width bounded norm networks look in function space? In Beygelzimer, A. and Hsu, D., editors, Proceedings of the Thirty-Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pages 2667-2690. PMLR.
Scarvelis, C., Borde, H. S. d. O., and Solomon, (2023). Closed-form diffusion models. arXiv preprint 2rXiv:2310.123990
Shah, K., Chen, S., and Klivans, A. (2023). Learning mixtures of gaussians using the DDPM objective. In seirth- Conference on Neural Information Processing Systems.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. (2015). Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning. pages 2256-2265. PMLR.
Song, Y., Durkan, C., Murray, I., and Ermon, S. (2021a). Maximum likelihood training of score-based diffusion models. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, W., editors, Advances in Neural Information Processing Systems, volume 34, pages 1415-1428. Curran Associates, Inc.
Song. Y. and Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32.
13

---

Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. (2021b). Score- based generative modeling through stochastic differential equations. In International Conference Learning Representations.
Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. (2018). The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822-2878.
Stanczuk, J. P., Batzolis, G., Deveney, T., and Schönlieb, C.-B. (2024). Diffusion models encode the intrinsic dimension of data manifolds. In Forty-first International Conference on Machine Learning.
Tenenbaum, J. B., de Silva, V., and Langford, J. C. (2000). A global geometric framework for nonlinear dimensionality reduction. Science, 29055500):3319-22225
Ventura, E., Achilli, B., Silvestri, G.. Lucibello, C., and Ambrogioni, L. (2024). Manifolds, ran- dom matrices and spectral gaps: The geometric phases of generative diffusion. arXiv preprint arXiv:2410.05898,
Vincent, P. (2011) 1). A connection between score matching and denoising autoencoders. Neural Computaation 23(7):1661-1674
Wang, B. and Vastola, J. (2024). The unreasonable effectiveness of gaussian score approximation for diffusion models and its applications. Transactions on Machine Learning Research.
Wang, B. and Vastola, (2023). Diffusion models generate images like painters: an analytical theory of outline first, details later. arXiv preprint arXiv:2303.024900
Wang, P., Zhang, H., Zhang, Z., Chen, S., Ma, Y., and Qu, Q. (2024a). Diffusion models learn low-dimensional distributions via subspace clustering. arXiv preprint arXiv:2409.02426.
Wang, Y., He, Y., and Tao, M. (2024b). Evaluating the design space diffusion-bassee generative models. In Globerson, A., Mackey, I.., Belgrave, D., Fan, A., Paquet, U., Tomezak, J., and Zhang, C., editors, Advances in Neural Information Processing Systems, volume 37, pages 19307-19352. Curran Associates, Inc.
Wibisono, A., Wu, Y., and Yang, K. Y. (2024). Optimal score estimation via empirical bayes smoothing. In Agrawal, S. and Roth, A., editors, Proceedings of Thirty Seventh Conference on Learning Theory, volume 247 of Proceedings of Machine Learning Research, pages 4958 4998-4991. PMLR.
Yi, M., Sun, J., and Li, Z. (2023). On the generalization of diffusion model. arXiv preprint arXiv:2305.14712.
Zhang. F. and Pilanci, M. (2024). Analyzing neural network-based generative diffusion models through convex optimization. arXiv preprint arXiv.2402.01665.
Zhang. K., Yin, Liang, F., and Liu, J. (2024), Minimax optimality of score-based diffusion models: Beyond the density lower bound assumptions. In Forty-first International Conference on Machine Learning.
14

---

A RELATED WORKS
Generalization in DMs. Several works have noted the transition from generalization to memoriza- tion behaviors DMs when the model capacity increases relatively to the training set size (Gu et al.. 2023 Yi et al. 2023 Carlini et al., 2023] Kadkhodaie et al., [024] Li et al. (024b). Using tools from statistical physics, Biroli et al. (2024) showed that the transition to memorization occurs in the crucial regime where is small relative to the training set sparsity, which is also the focus of our study.
To derive rigorous learning guarantees, one line of work showed that DMs can produce a distribution accurately given good score estimator (Song et al. 2021a Lee et al. 2022 De Bortoli 2022] Chen
cct al. 2023a oj Shah et al. 2023; Cole and Lu 2024 Benton et al. 20224] Huang et al. 2024a), which leaves open the question of how to estimate the SF of an underlying density from finite training data without overfitting. For score estimation, when the ground truth density or its SF belongs to certain function classes, prior works have constructed score estimators with guaranteed sample complexity
(Block et al., 2020 Li et al. 2023; Zhang et al. 2024 Wibisono et al.) 2024 Chen et al. 2024
Gatmiry et al. 2024 Boff al. 2025 Han et al. 2024h, including for scenarios where the data
supported on low- owwdimensional sub- uub-manifolds (further discussed below). An end-to-end error bound derived by Wang et al. (2024b) which covers both training and sampling and is used to inform the choice of time and variance schedules. Unlike these approaches, which concern the estimation of densities from i.i.d. samples, our analysis does not assume ground truth distribution. Based on a finite and fixed training set, our work focuses on the geometry of the SF when ł is small relative to the training set sparsity and elucidates how it determines the memorization behavior via an interplay with the denoising dynamics. For future work, it will be interesting to study the implication of score smoothing in the density estimation setting by potentially adapting our analysis to cases with randomly-sampled training data.
DMs and the manifold hypothesis. An influential hypothesis is that high-dimensional real-world data often lie in low-dimensional sub-manifolds (Tenenbaum et al. [2000] Peyré| 2009), and has been argued that DMs can estimate their intrinsic dimensions (Stanezuk et al. 2024, Kamkari et al. 2024), learn manifold features in meaningful orders (Wang and Vastola 2023 2024 Achilli et al. 2024). or perform subspace clustering implicitly (Wang et al. 20244))) Under the manifold hypothesis, Pidstrigach] (2022); De Bortoli (2022); Pooaptchik et al. (2024); Huang et al. (2024b) studied the convergence of DMs assuming sufficiently good approximation to the true SF, while Oko et al.) (0023); Chen et al. (2023b); Azangulov et al. (2024) proved sample complexity guarantees for score estimation using NN models. In particular, prior works such as Chen al. (003b): Wang and Vastola (2024); Gao and Li (2024); Ventura et al. (202) have considered the decomposition of the SF into tangent and normal components. Our work is novel in showing how score smoothing can affect these two components differently: reducing the speed of convergence towards training data along the tangent direction (to avoid memorization) while preserving along the normal direction (to ensure a convergence onto the subspace).
Score smoothing and regularization. Aithal et al. (2024) showed empirically that NNs tend to learn smoother versions of the ESF and argued that this leads to a mode interpolation effect that explains model hallucination. [Scarvelis et al. (2023) designed alternative closed-form DMs by smoothing the ESF, although the theoretical analysis therein is limited to showing that their smoothed SF is directed towards certain barycenters of the training data. Their work inspired our further theoretical analysis on how score smoothing affects the denoising dynamics and leads to a terminal distribution that interpolates the training data. In the context of image generation, (aab and Ganguli (2024) showed that imposing locality and equivariance to the score estimator allows the model to generalize better. However, in addition to being limited to the image generation setting with CNN-based score estimators, their result does not show how new samples are generated, but only that if such samples are created, then they obey the consistency properties enforced by the CNN architecture. In comparison, our work shows that the interpolation effect can arise from score smoothing with NN architectures as simple as 2L MLPs.
Recent works including Wibisono et al. (2024) Baptista et al. 2025 considered other SF regularizers such the empirical Bayes regularization (capping the magnitude in regions where ((()) is small) or Tikhonov regularization (constraining the norm averaged over pi "(")). In the linear subspace setting, these methods tend to reduce the magnitude of the SF in not only the tangent but also the
15

---

normal directions, thus slowing down the convergence onto the subspace and resulting in terminal distribution that still has a d-dimensional support. In contrast, the Smoothed PL-ESF preserves the (linear) normal component and hence is not prone to this issue.
Building on a convex program formulation of the training of two-layer ReLU-like NNs in finite-data settings, Zhang and Pilanci (2024) derive convex program relevant to score learning with two-layer NNs. Compared to the score matching loss (4). however, the denoising score matching objective therein has gone through an additional simplification step: the expectation over the noised empirical distribution is approximated by sampling one noise vector per training data point. This choice reduces the problem into finite-data setting order derive the convex program formulations which is indeed crucial for their technique to apply since the dataset size determines the dimensionality of the dual space but introduces variance to the problem in the mean time. Furthermore, whereas /Zhang and Pilanci] (2024) prove that the score predictor obtained by solving the convex program generates Gaussian or Gauss-Laplace distributions through Langevin Monte Carlo, we show that a smoothed score drives the denoising dynamics to produce distributions on desired subspaces that interpolate the training data.
NN regularization (explicit or implicit) as non-smoothness penalty Savarese et al. (2019) showed that the weight norm of an indthite-width two-layer ReLU NN on scalar inputs is equivalent to a complexity measure in function space given by (10), which was later extended by Ongie et al. (2020) to the case of minti-dimennional inputs. We use the result of Savarese et al. (2019) to motivate variational problem specific to score learning in diffusion models (11): minimizing the non- smoothness R[ subject to the constraint of the score matching loss [""] Our novel theoretical contribution on this front (Proposition D is the analysis of this variational problem, in particular proving that wear-minimizer of this problem small t is given by the Smoothed PL-ESF. This required a novel analysis of the interplay among the score matching objective, the smoothness penalty, and the geometry of the ESF, which is specific to our work. It will be very interesting for future work to adapt our analysis in the multi-dimensional case to the complexity measure defined in Ongie et al. (2020).
Though models in practice are not always trained with explicit regularization such as weight decay, a similar effect could still occur through implicit regularization: it has been shown theoretically that when trained with gradient-based algorithms for sufficiently long, certain classes of NNs can be viewed as implicitly minimizing some complexity measure while fitting the target labels (Soudry) et al. 2018; Ji and Telgarsky| 2019, Lyu and Li] 2020). Notably, when we consider infinite-width 2-layer homogeneous NNs trained with logistic-type losses (and under certain assumptions). the complexity measure agrees with in (1O) (Chizat and Bach) 2020). Though the argument does not apply directly in our setting as score estimation involves a different type of loss, it gives us reason to hypothesize that score smoothing can occur through NN training even without explicit regularization. would be highly relevant to investigate this further in future work.
B ADDITIONAL NOTATIONS
We use big-O notations only for denoting asymptotic relations as f - 0. Specifically, for functions f.g: 14, + RR.,, we will write f(() = (g((t) if 36, C. > 0 (they may depend on other variables such and A) such that V'' (0, it holds that ft) I Cg(t). In addition, in several situations where f decays exponentially fast in 1/t as t > but the exact exponent is not of much importance, will simply write f(t) =O(exp(-C//t (and the value of C can differ in different contexts).
which is intended to be interpreted as EC' > 0 such
that [(1) =
C GENERALIZATION TO n > 2
The analysis above can be generalized to the scenario where S consists of > 2 points spaced uniformly on an interval [-D, that is, y/ = 2(k 1)A - for l E [m]. where A == D/(n-1n
(ye+1 )/2. We additionally define 2ạ yk + A (ye 9k+1)/2 for k [n 1].
16

---

C.I SCORE SMOOTHING
In this case, we can still express the ESF as 6 except for replacing 7 by
Dx VLPN(1 yk: vt)
(25)
Ek=1 PA yk- Vt)
and its PL approximation at small is now given by
(y) 2)/l,
if x ø1;
(c)
Ukk - 2) x)/6.
if x E 1 2x] for ls E (2..., 02,...n-11 1},
(26)
(y/n /)
if x 2 2n-1.
For 8 E (0, A), we now define
(y) xt/tt.
= S + + i 2 Wn-1 5
8 (x)
(yn x)/t.
(27)
(yk z)/t,
if E yy 6, yv + 6], E [n] x E [yk + d, - -] [n 1 1]
6/(A - 6) -x)-t
and it not hard show that Proposition and Lemma 9 can be generalized to the following results, with their proofs given in Appendix E. and respectively:
Proposition 6 Let 8, = kVt for some O. Then 1t,, C > 0 such that Vt E (0,t1).
F(k) Cvi 1 ((.)) n
F(m) CVT,
(28)
where i and C depend only on K and F function that strictly decreases from [0, oo).
Lemma 3t, C> such that Vt (1, it holds that
t E x-pp
|| (n) (x)
- de
log p. (1) (z))] s <Cexp(-0*/99t)
(29)
C.2 DENOISING DYNAMICS
The backward- ti-time dynamics of replaced by
13 can also be solved analytically in a similar fashion, where
VE( V1) + V1
if < V1 + 6c
Vi(x Vn) + V/
if 2 Yn dc
(x)e(x)=
(30)
E((x yk)) + yk
i E [y òG Uk + 86), E {2, n 1},
(z 2x)
2k
if r E [y + 8c, yk+1 51), 3k [n 1].
The formula (16 is then generalized to
8./8. P (m,bo)
(5e/6. (6. 6,) v1) if S 111 + 6,
6r/5, Pi (5./5, I (ō. 6,)/6. yn)
(n,to)
if 2 8,
s(n.to)
82/5, pt (/ (8./5,
(ō. 6,)/5, yk)
if E [yx: 5,, yk + 5,]
(31)
(A (A 52) Pi
(nl ((A (A)
(.) x + (6, 5,)/(1 /A 5,) 2x)
if E [ye S., 9h+ 8x]
When 0, there is
(x) = [(A 2 - -8), if [y): + 8,, 2 - 6c), 3k [n - 1], (32)
yarg min, Iu m2|
otherwise.
As (x) is invertible when restricted to UkEin UkE(n-11lMn + dr, Uh+1 - ōz], the terminal distribution can be decomposed
Pb
(n,to)
Σ
agoy + (1 Σ bx PPo
(n,to)
(33)
k=1
17

---

((n,to)
where Pb
is probability distribution defined
Po
(n.
(x)
(A (-)5
Pi ((A 6:) A
(n.to)
x + 5./4 2x);
if E [yx M++1l otherwise
(34)
0,
and it holds for all t0, that
k = 1
1x2D-b, xxp!iito)
if k: n. if k E {2, - 1}
(35)
xx (n.te) L -iixSssn
C.3 HIGHER DIMENSIONS
Because the definition of R(d) is invariant rotations and translations of the coordinate system, we can assume without loss of generality that the training set lies on the [axis and is spaced uniformly on the interval [-D, D].
Thanks to the decoupling across dimensions, the denoising dynamics associated with the generalized Smoothed PL-ESF also follows (23) and 24). Hence, Proposition 3 still holds except with 114 and (T6) replaced by (30) (32 and respectively.
D PROOF OF PROPOSITION
The first claim is straightforward consequence of Proposition 6 threshold dependent on k), there is
when t is small enough (with
(n) (m) 1,0
n -
+(s)+ F(R) (()
(36)
Next we consider the second claim. On one hand, it is easy to compute that
8t
A 2(n 1) A(A 8c)
RI
I(A 5z) 1 Σ k=1
(37)
On the other hand, let f be any function on R that belongs to the feasible set of the minimization problem meaning that f twice differentiable except on set of measure zero and (
[f] e.
Define = f If(x)= dar log (1) (2))2 y&: vt for each & E [n]. By the definition
of p! we then have Σx=1 define fa()) Vi fii), there is
nc. If we consider a change-of vhriallof-variaale
variable ă (x vt)) V and
U/ 1f(x) ((() (x)) yks Vddr=
+
yk - -
f(x)
PN(c - yk: Vidr.
-
A/vi
(F((T) + Pp; 1)dE
A/Vi
(38)
Hence, using 662 with 5 A, obtain that
A/Vi
A/VE (
2
PN
1)dE <t
1f(x) (x) - yki Vtjdz
1f(x) de logp!" (n) (x) px yki Vt)dr
(39)
00
log (x) (x) 5 (a) 2e00pppxee Pvvlx yx; Vidda
+
du
< Eq + (expp( - A ( (9t)))
18

---

Thus, for small enough such that vt < A/3, apply Lemma from below to fi. from which we obtain (after reversing the change-of- that
inf
(x) - (-1+ 2//3////+++00 2//////
rr(v 1.50.u 1.50 /W
inf rEly vA +30
f(z) < - 0.5
(40)
sup
f(x) 2 0.5.
=E(yx-3o, 1.50]
Hence, 3ag E [yA 1.50, + 1.50] N, b E Like + 1.50, yu + 30] and by.- E Ly 30, yk 1.50] such that f'(x)) I (-1+ 2v ((exp(eexpp)))) (10t)) f(bx.-) I 0 and f(bx.-) ) 2 Furthermore, for t small enough such that vt < A/6, there is b < 1 for t: E [n 1]. and hence by the fundamental theorem of calculus, 3eg E [b,, bx+1,- such that (c 0.
Now, we focus on the sequence of points, <a < < < Cn-1 < an- By the fundamental theorem of calculus and the fact that f is twice differentiable except for on a finite set, there
(x) 2 (f'
aa+1 (x) 2 (f
2 (1-1 01
and hence it is clear that
R[ = f (x))x) Σ (((()) - f'(ax)1 +
Σ f((a))
k=
k=
n1
(41)
2 2 n 1- 1- Σ 2 /t 00eex((-22((100)
k=1
1(ex0(eexpP)))) 2 2(n - 1- (
Therefore, for 0 <<<< 0.015, as n 2 2, it holds for t sufficiently small that
R R(
(n 1)A
(n - 1n-1-2nn-4 11-1-2n - 5,) O(cxp(-A2/(11)
(42)
<1+7.//+
which is bounded by 1 1 8VE. Since this holds for any f in the feasible set, it also holds when the denominator on the left-hand-side is replaced by the infimum, ri.--
Lemma Suppose f is twice differentiable on R except on a set N of measure zero and 1 Ix (x)) 1)de < e with 0 < < 0.03. Then we have
inf x 1.5
f( -1+
(43)
inf
f(x) < 0.5
(44)
=E 3)
sup
f(x) 0.5
(45)
=E[- 1.5, -3
Proof of Lemma We first prove (43) by supposing for contradiction that inf 1.8).N fz(x) = -1 + kA with ka > 2 By the fundamental theorem of calculus, this means that the function [(x) is monotonically increasing with slope at least hi on [-1.5, 1.5). Hence, there exists
19

---

D1 R such that x + f(x)) > kalz 21 21 [-1.5, 1.5]. Therefore,
1.5
x + PAr;;; 1Id 2
Jr + [(x)) par(2; 1)
-15
1.5
2 (ka)²
x x (<1; 1)dx
1.5
1.5
(46)
(ka)²
(x² + 22 + (pN() 1)dx
1.5
2 (k&)²
x² Pp (2; 1)dx.
1.5
> 0.25 >E,
which shows contradiction.
Next, we prove (4 by supposing for contradiction that inf inf.s, S/NN holds that s3 (xx > and hence
S/NN f(x) -0.5, in which case
l"
3
x + pN=)) 1)dx 2
1
f(x)1 (1;
11.
3
(47)
PA 1)dr
1.5
0.03 >
which shows contradiction. A similar argument can be used to prove 455
E PROOF OF PROPOSITION 2
Below prove Proposition 6 which generalizes Proposition 2 to the case where n > 2.
Lemma 36, C such that VL (0,4). 11), it holds that
(x) ( log pt (w0) (=))| Cexp(-6'/99))
t E 2mppt
(m)
(48)
Lemma 9 proved in Appendix E. In light of it, we only need show that
(x) s ((") (x) (") (x)dr = n- F(x) + +0VI).
(49)
n
By the definition of Pi first evaluate the integral with respect the density pv(x Uks vF) for each [n]] separately and then sum them up. We define
(n)
-00,
i
00 yk
k
Vk.-
Uk++
(50)
Us 5,
otherwise
+ dc
otherwise
By construction, $2,5 is PL function whose slope is changed only at each yk.- and yk.+
Let us fix a E [n]. Since ((() (x) s(") (z) when x E 9t.+,, we only need estimate the difference between the two outside of [ym.- +.
We first consider the interval (Ji,+ Uk + A] (yk + 8r, yk + A] when & {I,..., n 1}, on which
holds that
1,6,
(m)
(x:)
86
(n)
(x)
A
A
(yk
5τ)
(51)
20

---

by the -eneeis---iieaiity of the two functions. Hence,
2k
t
2,62
(n)
(x)
5)
(x) - Uki vijdæ
+5
5, 2
I
2k
PN yks vtjda
A
dz
Vt
Vt
+8,
(52)
Uk
PA-( - yki vi)dæ
Ki
A
dz.
Vt
Uk
K
PN - yki vt)dx
VE
Ux+ +
Note that by change-of-variable ã x-
(xc
Wk// //V, we obtain that
yk
K
1 pN(x yks Vtjda
2 F(c).
(53)
Ue 14
Vt
where we define
F(x)
2 Iu plpp 1)du
(54)
It is straightforward to see that, as increases from O to co, F strictly decreases from to 0. Therefore,
Ve
(1)) - sx (x) yk; Vida
A
F(()) SAv Iu x|² "ar(l; 1)dr
(55)
A
F(r)) + O(vE)
2
Next, we consider the interval 1yk + 00), in which we have
A
(1 - s (x)
_.
(56)
Thus,
00
1 186,6 (x) - (x) yk: VDda St
A 2
pN
yki Vidr
ueta
- A
2
PNrl;; 1)du
(57)
A/Vi
(-
A
0
t- [teexp
2t
Hence, we have
ff (m)
1 yetōr (,6, (z) (x) Mbi Vtdda F(s) + +((V)).
(58)
2
Similarly, for E (2,..., n), we can show that
18 (x) - a65) (x)) - Mes Viyds - F(x) + +(VI).
(59)
Thus, there is
(8(m) (x) 5/ (x) yks Vtjdr
000
(60)
[F(K) +OVI), (FF(K)+((1)
if E 12,..., 1} if = nr.
21

---

Summing them together, get that
(n) B, (x)
sx)
(m)
n- (x)dr F(x) O(vi).
(61)
n
This proves the proposition.
E. PROOF OF LEMMA
Below we prove Lemma which generalizes Lemma to the case where n >
By the definition of Pt (m)
it suffices to show that vk [n]]
f dl logpt" (m) (x) s( (x) yk! Vt)dr 0(exp(-62/(9t)
(62)
Consider any & [n]. We decompose the integral into three intervals and bound them separately. First, when æ 2 + A > yk. noticing that that
de log (m) (x)), (s|") (x)] s (|x) + 2D)/t, we obtain
d
d
logp;
(2) (z') -- (a') PN yki Vtjda'
da
Ub +++44
log (n) (x') - 31 (m)) (z') ? exp
(x' (1''-91) y2))²
de'
de
21
4((21 + 2D)2
(x' [zx22))2
exp
dol
V 2Tt
Bot Bot
12
21
4((x' Val
+ 4D)²
(x' )
exp
dx
(63)
2T
12
8
2k
16D
(x' - )2
+
exp
dx
V 2
Vt
t
2t
8
00
16D2
(
(x)²
exp
di
1
2
8
((10=++))
16.D2
82
82
12T//3
2
+
2
exp
81
exp
9/
where for the last inequality we use Lemma 10 below.
A similar bound can be derived when the range of the outer integral is changed to between -oo and
y/ A.
Next. suppose & E [yke - A + AA), which means that jx - ykl S 1/4 while ja vil 2 2/2 A for
1 + k. Thus, holds for any * is that
px yk: vt)
y/
1x: x?
Ad
exp
2 exp
(64)
PN - 33; vT)
2t
Hence, writing (x) = pula- awla- w)i
there is (1,(xx))
1 (n
exp (-
A4 7
and for
Ee (
I * k. (x) < exp (66). Therefore,
1(r.a (x) 1)yul= + Eipp (x)
logp: (n) (x) - s(n) (x)
<
da
(65)
2(n - 1)D
A6
= =0
(-
A6
exp
t
t
22

---

Since p2 Yki vB)
for any z', then have
+ 3
log Td) (x') s/" pN - yki vi)da'
Pi
di
1/k 1
+ w
logp; (0) (x') - (o'))
/2
sup
de
(66)
AA J3
=0 t
240
exp
t
Combining 663 with (66) yields the desired result.
Lemma 10 For u 2 0,
x. 12 dx S
1
=
π -1212
(67)
x² 12 dx S
(v
π +
+u
12
(68)
Proof of Lemmal 10 It known (e.g.. Chang et al. 2011) that
f.
V dx v
(69)
2
from which 66 can be obtained by a simple change-of variable. Next, using integration-be by--nrtsss we obtain that
-hannel.f-varrablle
z² x² dx x(-e x0-e--133 1121 / 1 (-c
(-e----a o 4)dz dx
(70)
812 1 1t)
= aa0-0 -
12
Hence,
1 12 + ue-1212 22/2
Le 22 S
(71)
π
+ e u2212
V
2
F PROOF OF PROPOSITION
To make the dependence on d more explicit, below we add it as a super-script into (n.d) and L (m.dd)
Our first observation is that the score matching loss (m [f] can be decomposed
sum over com-
ponents that depend separately the different output dimensions of f: L." (n.d) [f] = Ei
(m.d)) i,a
[[]]]]
with L. (n.d) [f] == t IE x"> [xl θ logp!") (m) [x)] In particular, in each of the normal dimen- (1)
sions, I (n.d) can achieve its global minimum of 0 if we set [f()) O log (n) (x) = [x]: for
i 1. This motivates to focus candidate solutions of the following form:
Definition Given g + RR we define aflat extension of g along the normal directions to the domain Rd, fg Rd R, by
fx( s((x)1), Va E Rdd
(72)
We then further define
fa(m) [,((m))
[m]2/4,.., (0//)),__/2//,___ (mmaa/
(73)
23

---

Lemma 12 Given any g R R. it holds that L.
[fa] 1
(/m.1) [g] and R [Sa] [g)
Proof of Lemmal 12 For the first property, note that
1,, [Vl] = E x (m) [[x)) - O, log (m.)) (x)1
(n.d)
- E x
(i [1l9((x)1)
d
logpt (n. ([x]1)]1
d[x]]
(74)
d
=t B Xxpl m [xlx(x)
log Pi
(m.1) (x)]l²
dx
= [gl
Hence, L, I.( = (1
[fg] L
L 1. [g].
For the second property, let us consider any w E gd-1, There is
Vwlg(a)> - ((z(2) [w]1g' ((as(((=)))
(75)
(x)) (w) 1 ([=)1) ([w)))²g" ([=]))
(76)
Meanwhile, thanks the linearity in the normal dimensions, have ( (a:)) 0, Vi (2....,d] d}. Therefore,
V2 fo (we + b)]ldz
/ V Wi fwr b)]dr
([w)1) 19 [ [b]1)]dr
(77)
= [[w]] [gl S [g]
Since we maximize over all w E gd-1 in the definition R(()), we see that [(] [g]
Lemma 13 The,a Ti,,, Vd E Nd.
13 Consider any f IRd + Rdd in the feasible set (i.e., I.(n.d0] [f] < c), and let
Proof of Lemma
j= [f) denote its tangent dimension. This implies that I (n.dd) ( < I [f] <. On the other
(m.d)
It,11
hand, we have
27 21,
1, [ = t. E E Bar (ma))
X(x,, - dxi
d
log p. (n,1) (x1)
(78)
where we let i = inf (ar JRR-- F.
x-plaa) m [F (2,,,,,,)) (2,,
dx log (n.1)) (m.))) (x) For any given
to > O, we can find uz xt Rdd-1 such that
22 xz)) ai bogg(n.1// (x) (2
E
Amp mil))
<i+eo co
(79)
We now define ğ :::R RR bb
π(x) = 2i.,, x2)
(80)
Then implies that 7.( < i +i+00 60 S (.(nnd)) [ + 6g- Since 6g is arbitrary, we may choose
60 - T ( which then implies I.(n..) []] < e,dd hence ğ is in the feasible set for the 1-D
variational problem. On the other hand, we see that
R [ =
d I (x, 2 (..,,,),)))) dx
de
(81)
000
22 fllf +b) dx <
22 22 f(wr + +b) dr,
24

---

if we choose [1.,,...,0) and b [0, ..........
...,..... 23]. This implies that R - R( [f]
Since this argument applies any f in the feasible set, can take infimum over f conclude that Tia 2 "
Now we proceed to the proof the proposition. Let and x be specified as in the proposition statement, and let 1 be determined in the same way in Proposition We observe that [$ la is the
flat extension of 8t,6, the domain R4, and therefore, 8t,6,
(n)
((n) S (_m) By Lemma 12 this means that
L ( [s(2]] L,
_. is 1, and the right-hand-sidee smaller than when < by Proposition
(x))
Meanwhile, Lemma 12 also implies that R(d) [ (x1)
2.6, [[ and the right-hand-side is smaller
than (1 + / (r.rr when / < 4 Proposition which is further bounded above by (1 + 8VF)ri,e,d
by Lemma 13 This completes the proof of Proposition 4
G PROOF OF PROPOSITION 3
We consider each of three cases separately.
Case I: x E [6, 1, 5.]. this case, it is easy to verify that x, = 1-6, the ODE
1- x is a valid solution to
d
1 T+ 10-6, 8,
ds 2 2
(82)
on [0, E] that satisfies the terminal condition x, x. It remains to verify that for s E (0, t),
holds that vs E [8, 1, - 5.] (i.e., the entire trajectory during (0, t] remains in region C).
Suppose that x 2 0. Then clear that v, 2 0, Vs E [0. t). Moreover, holds that
(1 - 5.) 1 1- 8g (x, (1 - 6.)) 0 6e
(83)
Therefore, v, E [0, 8.] S [6, - 1, 8.). A similar argument can made if
Case II: < 6 1. In this case, is also easy to verify that Vs / + 1) is valid
solution to the ODE
1r 1
(84)
ds 2
on [0, t] that satisfies the terminal condition x, x. remains verify that for all s E (0, f). it holds that < 6, (i.e., the entire trajectory during [0, t] remains in region A). This is true because
- 8, - -
(o--v((+116)) 50
(x.. 1) 6,
6x)
(8.5
Case III: 2 8 A similar argument can be made as in Case above.
H PROOF OF PROPOSITION 15
The proof relies on the following lemma, which allows relate the KL-divergence between (n. and the uniform density via that of p (n.4))
P
Lemma 14 AL [0, to), KL (ww.lllllmmmm (n. (L(uu -, (1)
25

---

Proof of Lemmal 14
KL(ui (
[
1
(-log
log(Po (m.to) (x)))dr
1
/ log(po (x))
log 2
2
1
1-6.
(log(1 5,) mog (x))) de' (86)
log 2
2(1
81)
6c-1
1
1-6
log (gg(1/(2(111 --)1- 5,))) log(p! (m,to) (a' )dd
2(1 6τ)
6.-1
KL lIp (nitol)
In light of Lemma 14 we choose 1 = ln and examine the KL-diieegennce between un un-6 and pt, ((n.fo) By symmetry, we only need to consider the right half of the interval, [0, 1 - òtul. on which there is (m)
Pio
(n,5o)
(x) Pio (x: 2 1PN(( 1; Vo)) We have
61o
[ log
log (pN(x - 1; vio) dr =
1 exp(-x² dr
V2
1-bea (log(2T) -log(to))
1
/
6to
x²
(87)
to
dia (log(2x) + (oo(()))
34a
Therefore,
1-500 - -log(1 - 61)) - llg
1
1 160 (1pl10
p 1; vtt)) de
1 8to 30
2
< -log(1 1 5,) + logg()) 1
(-
1-big (-1-a (log(2ir) + log(to)) 1
I-org
3ta(1-8u. + +log (1 ) + log(2v211)
(88)
By symmetry, the same bound can be obtained for KL (1 which yields the desired result when combined with Lemma
I ADDITIONAL DISCUSSIONS
I.1 CONNECTION TO LOCAL SMOOTHING
Given f RR R and > O, we define locally-smoothed version of f with window size w as:
(w * fx) =
2w f(x')dx'
(89)
is then not hard to see that, for w E {0,1)).
(w (() (x) $111 (x).
(n)
V..R.
(90)
Therefore, ((n) & can also be interpreted as a socolly- version of the PL ESF with window size 1 ô, where smaller value of d means higher window size for local smoothing.
I.2 KL DIVERGENCE BOUND
18 allows us to prove KL-divergence bounds for ph (n,to) based on those of p (n.to) For example, letting un denote the uniform density [-a,a] have:
26

---

Proposition 15 Let 0 and 0 < 60 1/x2. xfx solves
13) buckward-in-time with Xto Pta
(n)
then there is KL(ui (10uil/Pomtt) ( S 3to(1_x//o) log ( I-x/ta
vio
+ 1og(22/2m) < co.
This result is proved in Appendix
H Although the choice of the uniform density as the target to compare Po ((n,ta) with is an arbitrary one (since the training set is fixed rather than sampled from the uniform distribution), the result still rigorously establishes the smooth component of .fm.to) that interpolates the training set. contrast, denoising with the exact ESF results in Pú which fully singular and has infinite KL-divergence with any smooth density on [-1,1].
1.3 COMPARISON WITH INFERENCE- TIME EARLY STOPPING
The effect of score smoothing on the denoising dynamics is different from what can be achieved by denoising under the exact ESF but stopping it at some positive min- In the latter case, the terminal distribution is still supported in all d dimensions and equivalent to corrupting the training data directly by Gaussian noise. In other words, without modifying the ESF, early stopping alone is not sufficient for inducing a proper generalization behavior.
J ADDITIONAL DETAILS ON THE NUMERICAL EXPERIMENTS
All experiments were hosted Jupyter Notebook service with single TPU (v3) backend. The code was written in JAX (Bradbury et al. 2018) and each experiment ran for less than 1 hour.
J.1 NN-LEARNED SF vs SMOOTHED PL-ESF IN 1-D
NN-learned SE. We trained a two-layer MLP with skip linear connection from the input layer to fit the ESF at 0.05. The model is trained by the Adam optimizer (Kingma and Ba 2015 Loshchilov and Hutter 2019) for 6000 steps with learning rate 0.0002, and consider four choices of the weight decay coefficient: A = 1.0, X2 = 3.0, A3 = 5.0 and A = 7.0. At each training step, the optimization objective is approximation of the expectation in (3) using batch of 1024 i.i.d.
samples from p
We considered four choices of X: t = 1.0, X2 = 3.0, X3 = 5.0 and X1 = 7.0.
Smoothed PL-ESE.
We chose 0.05 and four values of 8 (6; 0.648, 82 0.548, 63 0.453, 54 0.346), which were tuned to roughly match the corresponding curves in the left panel.
J.2 EXPERIMENT OF SECTION 6.1
The ESF is computed from its analytical expression 25 To ensure numerical stability small f,
truncate the sampled values of V log (m) based on magnitude. At to 0.02, 20000 realizations of Xty sampled from p(x) Then, the ODEs numerically solved backward-in-time to t 10-5 using Euler's method under the noise schedule from Karras et al. 2022 with 200 steps and p = 2.
For the Smoothed PL-ESF, we choose = nvtt with x 1.2.
For the NN-learned SF, after a rescaling by Vt (c.f. the discussion on output scaling in Karras al. 2022), y. we parameterize NN with three two-layer MLP blocks (MLP1, MLP2, MLP3): MLP, is applied to log(t) to compute time embedding: MLP; is applied to the concatenation of x: and the time embedding: MLP3 also applied to log(!) and its output modulates the output of MLP; similarly to the Adaptive Layer Norm modulation (Perez et al. 2018 Peebles and Xie 2023), MLP, and MLP: share the first-layer weights and biases. The model is trained to minimize a discretized version of (3) with T = to. where the integral is approximated by sampling from [10 6 to] with (n)
/1/3 uniformly distributed (inspired by the noise schedule of Karras et al, 2022) and then æ from p updated by the AdamW optimizer with learning rate 0.00005, batch size 1024
The parameters and total number of 150000 steps, where weight decay (coefficient 3) is applied only to the weights and biases of MI.P
27

---

[Vlog " (x)]1
[+|n] (x)]1
(x)]
100
to/2
-100
(xla
100
t//
-100
0.2
02 [x]e
100
t/8
-100
0,2
pxle
0.2
[x]
[x]
[x]]
Figure 7: Comparing three SF variants from Experiment 2 in their first (tangent) dimension at different t. We see close proximity between the Smoothed PL-ESF and the NN-learned SF, both of which are smoother than the ESF especially at small t.
28

---

[Vlog (x)])
[6 (x)12
(x)12
t- tof2
1100
0.2
[x]]
100
t= f//4
-0.2
0,2
[x]2
tp/8
1000
[x];
[x]
[x];
-1
0.2
[x]]
Figure 8: Comparing three SF variants from Experiment 2 in their second (normal) dimension at different We see all three SF are relatively similar in the normal direction, except for mild distortion of the NN-leared SF when is small and [x]2 is large (where has low density).
29

---

to/2
Vlog p(
25
S: [(n)
Se s
t = to/4
[sx[[x, )]
-50
to/8
100
-100
Figure 9: Comparing three SF variants from Experiment 2 in their first (tangent) dimension at different t when they are evaluated on the [x]] axis. Again, we see close proximity between the Smoothed PL-ESF and the NN-leareed SE, both of which are smoother than the ESF.
30

---

NN
S
Vlogp
A=100
2=2.0
-
2
X
Figure 10: ESF vs NN-learned SF with various strengths of weight decay regularization (X) when training data lie non-uniformly in 1-D. We see that the NN-learned SF becomes increasingly smooth as À increases.
J.3 EXPERIMENTS OF SECTION 6.2
Data circle The training data lies circle with radius centered at the origin in TR2, and choose tu = 0.08 and n = 4, 8 and 16. The NN model trained with learning rate 0.0001, batch size 64 and weight decay for 5000, 20000 and 80000 epochs respectively for the three choices of n. The rest of the configuration is the same as described in Appendix J.2]
Rapdedd in 1-D The training data (with n are randomly perturbed from uniform grid. We choose = 0.1. The NN model has the same architecture described in Appendix [.1 and trained for 15000 steps using the AdamW optimizer with learning rate 0. 0.00005.
EXPERIMENTS OF SECTION
J.4.1 DATA MANIFOLD AND STEREOGRAPHIC PROJECTIONS
We consider manifolds that are intrinsically 2-D and obtained by slicing the unit sphere in R.3 They can then be embedded in d > 3 dimensions by randomly choosing 3-D subspaces within Rd.
((z)
-2
2 0
-2
Figure 11: Illustration of the stereographic projection.
The stereographic projection is typically defined map R from the unit sphere (excluding the point the top) to the x-y plane. By considering concentric spheres with different radii, this can be extended to a bijective map from R (xc IR3 namely:
T 22 = 0, xz 2 0} to the lower half of R3
2 |||0||05
x
D2
(xx)
||>|||
(91)
D3
x
F3
31

---

fmin
X2
1
X1
-1.5
U2
-1
-0.
1
U1
train
S
-1
-1.5
-15
-1
1
-1
U1
U1
U2
where d and the training set forms uniform grid in the latent space and the score is learned by a two-layer NN. Top left: In the canonical 3-D subspace containing the manifold with coordinates produced by stereographic projection, i.e., ((z). Bottom: 2-D-sliced views in the latent coordinates.
Figure 12: Results of the experiment in Section 6.3
[2], 22, 22, zz] Top right: Latent coordinates
This map is illustrated in Figure I1
J.4.2 EXPERIMENT SETUP
The two-layer NN has the same architecture described in Appendix adds one additional layer to the MLP. We consider four scenarios:
1.2 while the three-layer NN
1. =
grid latent space, two-layer NN (Figure
12
2. d = 20, 36, grid in latent space, two-layer NN (Figure 13
3. d = 20, 36, grid in latent space, three-layer NN (Figure
14
4. d = 20, 64, uniformly random in latent space, two-layer NN (Figure 6
where "grid in latent space" means the training set form a uniform grid (6 x 6) in the latent space after the stereographic projection. The models are trained by Adam without weight decay and with learning rate 0.001 with various numbers of epochs.
32

---

[ to
1.5
Z2
21
U2
-0.5
U1
train
0.5
0.5
S
-1
-1.5
-1.5
-1
O
1
-1
0
1
-1
0
1
U1
U1
U2
Figure 13: Results of the experiment in Section
6.3
where 20 and the training set forms
uniform grid in the latent space and the score is learned by two-layer NN. Same meaning of the plots as in Figure 12
22
t
22
0
1
0
21
-1
115
-1
-.5
U1
train
1
0.5
0.5
S
0
-1
3
-1
-1.5
-1.5
-1
U1
U1
U2
Figure 14: Results of the experiment in Section 6.3 where d = 20 and the training set forms uniform grid in the latent space and the score is learned by a three-layer NN. Same meaning of the plots as in Figure 12
33


Reviews:
[{"review_id": "fQxoHxmAnE", "reviewer": "Reviewer_PEv5", "summary": "This paper studies the interpolation effect of NN-learned score estimator in diffusion models. Numerical experiments are conducted to support theoretical analysis.", "questions": "I have a minor question:\n1. Although maybe not easy, I would appreciate it if authors could show numerical results for high-dimensional data.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. By using a two-point 1-d example, the authors analyze properties of a smoothed piece-wise linear score estimator.\n2. The authors investigates the evolution of marginal distribution in sampling using the above estimator\n3. For multi-point case, the theory is also established.\n4. Numerical experiments are conducted for this special estimator with comparison to neural networks.", "weaknesses": "I have several major concerns:\n1. Essentially, the authors constructed a special score estimator and argues diffusion model sampling benefits from this estimator. However, it is not clear to me how this estimator is connected to neural networks. Even in 1-d two-point case, such a connection is missing.\n2. Seemingly the authors propose a new regularized score estimator to make it smoother and thus benefiting the sampling process. However, in practice, the regularized training objective (10) and (20) seem to be hard to compute. Could you elaborate this point further?\n3. In terms of experiments, the NN-learned estimator indeed behave similarly to the one proposed by authors. Is it because that NN learns exactly this piece-wise linear form or NN estimator enjoys some other smoothing properties?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zhATEAsFfM", "reviewer": "Reviewer_1ab2", "summary": "This paper studies why score-based diffusion models (DMs) with neural networks can generate new samples instead of simply memorizing their training data.\nBesides, the authors propose a mathematically tractable Smoothed Piecewise-Linear Empirical Score Function and show that this smoothing prevents the reverse-time denoising process from collapsing to training points.\nThrough analytical results and simple experiments, they demonstrate that both smoothed and NN-learned score functions produce smooth interpolations rather than pure memorization.", "questions": "See Weakness. Further:\n  1. Can the proposed smoothing framework be extended or tested on high-dimensional real data ？", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper gives a clear and rigorous mathematical formulation of how score smoothing prevents collapse and enables interpolation in diffusion models.\n2. It provides elegant geometric insight and connects explicit smoothing with the implicit regularization effect of neural network training.", "weaknesses": "1. The paper does not theoretically prove that neural network training truly realizes the same smoothing mechanism.\n2. All experiments are conducted on low-dimensional toy cases, and it remains uncertain whether the proposed mechanism can actually help reduce memorization in diffusion models trained on limited real-world data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pv0mwqYQeL", "reviewer": "Reviewer_GL3U", "summary": "This work studies how NN-based score predictor can help alleviate the exact memorization phenomenon when an exact score function is used. Specifically, the authors study the score function learned via 2-layer ReLU NN, and show that the learned score function has a smoothing effect that helps creativity.", "questions": "1. is there any clue how to extend the current analysis framework to deeper NN architecture?\n\n2. if it's hard to generalize current result to modern architecture, is there at least any clue that whether modern diffusion models' effectiveness also comes from such score smoothing mechanism? \n\n3. if the authors take a look at the prior work mentioned above, there is some deeper connections between these two (which might also be superficial), like the current work has sign(x) for 1d data, and the derived convex program also has sign(x) (see the paragraph below Theorem 3.1). Is this coincidence?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-organized and strongly motivated. Studying NN-learned score function is important in understanding the creativity of diffusion model. The theory is rigorous and the authors carry out well-targeted experiments to verify their theoretic results.", "weaknesses": "1. missing highly-related prior work [1]\n\nSpecifically, in the current paper, the authors mention \"while it is difficult to predict exactly what function an NN will learn due to the nonlinearity and stochasticity of the training dynamics\" and seeks to find the NN prediction by converting weight decay as a smoothness penalty, while in work [1], the authors already converted the non-convex two-layer NN to a convex program and solved it to optimality, where they showed the optimal function is piecewise linear.\n\nIndeed, Figure 2 in current paper is well-aligned with Figure 1 in [1], where the authors in [1] already theoretically showed why 2-layer ReLU will produce results as Figure 2 in current paper. The 2d circle experiment in current paper was also done in Section 5.2 in [1]. Also, formula (9) in current paper looks similar to formula (7) in [1].\n\nThe highly-correlated results and scientific problems being studied should be paid attention.\n\n2. the setting is limited to 2 layer ReLU, and is far from what's being used in real diffusion model\n\n\n[1] Analyzing Neural Network-Based Generative Diffusion Models through Convex Optimization: https://arxiv.org/abs/2402.01965", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RN9itfijPa", "reviewer": "Reviewer_8eUJ", "summary": "This paper studies the effect of score smoothing on the ability of a diffusion model to generate samples outside the training data set. Most of the analysis revolves around a simply toy problem with evenly-spaced data points, where the analytic score functions (ESF) are known exactly.  It is shown that the scores learned by a regularized two-layer NN trained on this dataset is closely approximated by a smoothed version of the ESF. In the 1D case, and when the data manifold is a 1D subspace of a larger data space, the smoothed score (which models the NN score) ascribes some probability to regions on the data manifold that lie between the training data samples. At least in this limited setup, smoothed score functions offer a compelling explanation to 'generalization'.", "questions": "In proposition 1, if I understood correctly, the function $F^{-1}(\\epsilon)$ grows as $\\epsilon \\to 0$. That means $\\kappa$ becomes larger at small $\\epsilon$. So $\\delta_t = \\kappa \\sqrt{t}$ is small only when $t$ is very small. But we also know that the actual diffusion model is trained by evaluating the denoising score-matching objective from $t_{\\rm min}$ to $T$. If the smoothed PL ESF in Eq. (9) is a 'proxy' for the NN score in your analysis, should you not restrict yourself to using this score only till $t_{\\rm min}$. Then, is $\\delta_{t_{\\rm min}} = \\kappa \\sqrt{t_{\\rm min}}$ necessarily small?\n\nI have given a tentative score based on my current understanding of the paper. I'm willing to update the score if my questions are addressed satisfactorily.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. It is always great to have an analytic model that closely approximates the NN behavior. The authors have done a great job in picking a very minimal setup, modeling the score in closed form, and analyzing it thoroughly to draw their conclusions.\n\n2. The problem being studied is pertinent, and the paper makes clear progress albeit with a simple example.\n\n3. The claims made in the paper are backed up with rigorous proofs.", "weaknesses": "1. The core arguments in the paper, which are contained in Sec. 3 and 4, were a little difficult to get through for me. Part of the the problem was that a great deal of notation was introduced in the course of developing the argument that I missed the forest for the trees. I think it would greatly help the paper to have a sketch of the content of these sections precede the more detailed discussion. On a more minor note, making Fig. 2 wider, and stating explicitly that $\\lambda$ is the weight decay coefficient around line 161 would also help.\n\n2. The authors use the fact that regularizing the weight norm penalizes non-smooth scores. Later, in Sec. 6.2 they show an experiment where the NN scores smoothly interpolate a circular dataset, even _without weight decay_. This is attributed vaguely to the optimization dynamics. But does this not undercut your main thesis? That is, can you still claim that your smoothed PL ESF reflects the NN scores in more general scenarios where the latter is smoothed by mechanisms other than weight decay? In particular, is there reason to believe that such smoothing will produce a score with smoothness close to that of $r^{*}$? To be clear, this is a question about the _relevance_ of the smoothed PL ESF, rather than the soundness. If there is an answer in the paper that I missed please let me know. Otherwise, it appears to me that this is the primary weakness of the paper.\n\n3. In the extension to higher-dimensions, it would have been interesting to see a data subspace with some curvature, even a mild one. Would some sort of 'locally-flat' assumption help generalize your arguments to such a subspace?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)