Diffusion Models and the Manifold Hypothesis: Log-Domain Smoothing is Geometry Adaptive
Authors: Tyler Farghly
Peter Potaptchik
Samuel Howard
George Deligiannidis
Jakiw Pidstrigach
Venue: NeurIPS 2025 poster ()
Keywords: diffusion models
generalization
manifold hypothesis
memorization
Primary area: deep_learning
TL;DR: We show that diffusion models adapt to low-dimensional data geometry as a result of how approximations are formed at the level of the score function. We show that smoothing the score function naturally produces smoothing along the data manifold.

Abstract: Diffusion models have achieved state-of-the-art performance, demonstrating remarkable generalisation capabilities across diverse domains. However, the mechanisms underpinning these strong capabilities remain only partially understood. A leading conjecture, based on the manifold hypothesis, attributes this success to their ability to adapt to low-dimensional geometric structure within the data. This work provides evidence for this conjecture, focusing on how such phenomena could result from the formulation of the learning problem through score matching. We inspect the role of implicit regularisation by investigating the effect of smoothing minimisers of the empirical score matching objective. Our theoretical and empirical results confirm that smoothing the score function—or equivalently, smoothing in the log-density domain—produces smoothing tangential to the data manifold. In addition, we show that the manifold along which the diffusion model generalises can be controlled by choosing an appropriate smoothing.

Diffusion Models and the Manifold Hypothesis: Log-Domain Smoothing is Geometry Adaptive
Tyler Farghly
Peter Potaptchik * Department of Statistics University of Oxford
Samuel Howard Department of Statistics University of Oxford
Department of Statistics University of Oxford
George Deligiannidis
Jakiw Pidstrigach
Department of Statistics University of Oxford
Department of Statistics
University of Oxford
Abstract
Diffusion models have achieved state-of-the-art performance, demonstrating re- markable generalisation capabilities across diverse domains. However, the mecha- nisms underpinning these strong capabilities remain only partially understood. A leading conjecture, based on the manifold hypothesis, attributes this success to their ability to adapt to low-dimensional geometric structure within the data. This work provides evidence for this conjecture, focusing on how such phenomena could result from the formulation of the learning problem through score matching. We inspect the role of implicit regularisation by investigating the effect of smoothing minimisers of the empirical score matching objective. Our theoretical and empirical results confirm that smoothing the score function-or equivalently, smoothing in the log-density domain- produces smoothing tangential to the data manifold. addition, show that the manifold along which the diffusion model generalises can be controlled by choosing an appropriate smoothing.
1 Introduction: Diffusion, manifolds and generalisation
Diffusion models (Sohl-Dickstein al., 2015) Song and Ermon, 2019 Ho al., 2020 Song et al.,
2021) have emerged as a powerful class of generative models, achieving state-of-the-art performance across domains (Dhariwal and Nichol, [02T] Kong et al., 2021] Liu et al., 2023) Ho al., 2022). Beyond their ability to generate high-quality outputs, they are also capable of producing novel samples not present in the training data, indicating a surprising capacity for generalisation.
The goal of diffusion models is to produce samples from a target distribution Pdat on Rd, given only finite number of samples. They do this by learning to reverse noising process, X', which begins with a random sample of the data distribution X, ~ Mdaaa and gradually transforms it into noise. This process defined by the stochastic differential equation (SDE).
dx, -ax, dt + v2dB, Xo Pdata:
for some a 2 0, where B, denotes the d-dimensional Brownian motion. It well known (Haussmann and Pardoux, 1986) that the time reversal Y, = Xy- of ① satisfies
dY aY, dt + 2V log + v2dB,
where p. denotes the density of X,. Therefore, the task of generating samples from Maaa can be solved by simulating paths of ( To that end, the unknown score function, V Vogg, in (2), is approximated
Authors contributed equally to this work. Correspondence to {last (ame))stats.ox.ae
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

0.02
o= 0.07
o= 0.12
Figure I: Isotropic smoothing of the score function identifies manifold structure. Training data (A) shown against generated samples (+) from a diffusion model using the smoothed score V log Py *Na. where the scale of the kernel is increased from a 0.02 to 0.12. As increases, generated samples begin to fill out more the manifold without having seen training samples in those regions.
by minimising the (population) score matching loss (see Hyvärinen (2005)
(am(() = /'" [l))(t. - V log P: (x.))l dt.
In (3. the expectation is taken over samples from X₁ (see !.) when started from the true data distribution Xo |Aaata- In practice, one has access only to finite dataset of samples { from Mdata and so fam must approximated empirically. Therefore, during training, the noising process is not started from the target distribution Adata but is instead initialised from the empirical measure, Aaasaa N Σg 8z,, This gives rise to the empirical score matching loss,
Rem(s) L'' E [ls(tt, X1) V log (x dt,
where P. is the density of the forward process, Xt, which is initialised from Paaata
One quirk of this objective is that possesses unique minimise lidentical to the empirical score function, V log p. (x). As result, if one were to reverse the noising process with this minimiser, one would arrive close to the empirical measure Ādaa. reproducing the training data instead of generating novel samples from the target distribution. In fact, it has been shown that any approximation sufficiently close to log P. will produce samples belonging to the training dataset (Pidstrigach, 2022). However, in practice, diffusion models trained with this objective perform well and avoid memorisation, suggesting that regularisation key to their generalisation capabilities.
AinimisseEddeniiall
The study of generalisation in diffusion models can be divided into three parts: (i) formulating the learning problem via score matching and its empirical approximation; (ii) the inductive bias of the training procedure and model architecture; and (iii) how regularising the minimiser of 4 affects the reverse SDE and generated samples. Of these, (ii) has been widely studied- -paanning architectures. regularisation, and optimisation and while far from completely understood, there are numerous studies into how neural network training promotes bias towards smooth functions interpolating the data (Rahaman et al., 2019, Mulayoff et al., 2021 Ma and Ying, 202T]; Vardi, (023). In contrast, (i) and (iii) are -specifc and comparatively underexplored, so we focus on these two parts.
To account for the inductive bias during score matching, we propose a simple model built upon smooth approximations to the minimiser of Ê.m. In particular, we consider the score function s which smooths the empirical score function log P., with a generic probability kernel k:
(1,
log (x Ki (dy)-
(5)
While this significantly simplifies the possible inductive bias employed during training, it succeeds in capturing a defining property of diffusion models: as result of the approach of score matching, any smoothing resulting from inductive bias occurs at the level of the score function-in the log-domain.
Beyond these considerations, understanding the generalisation of diffusion models also requires an analysis of the data distributions they successfully model. There is growing support for the
e Here mean in the L2 sense: any minimiser of l. is identical to v log Ór almost everywhere.

---

Figure 2: Density smoothing generates samples off-manifold, whereas score smoothing generates samples that retain manifold structure. Left: Comparing samples (o) drawn from a KDE (top) versus from a diffusion model with the smoothed score (bottom) from Figure (training data is
.). The scale of smoothing increases from left to right. Right: ID intuition for data-domain versus
log-domain smoothing. The left sub-figure shows the Gaussian (-) smoothed in data-domain (-), and the right sub-figure shows the Gaussian smoothed in log-domain (-) with the same kernel.
theory that diffusion models are particularly successful at modelling distributions adhering to the manifold hypothesis, wherein high-dimensional data concentrates on a lower-dimensional manifold (Tenenbaum et al., 2000) Bengio et al., 2012 Goodfellow et al., 2016). A standing conjecture is that generative models, including diffusion models and flow-based approaches, owe their success partly to their capacity uncover these hidden structures (Pidstrigach, 2022] De Bortoli, 2022] Ganiz et al., 2024; Farghly et al., (023). This raises a critical question: what mechanism allows diffusion models to effectively identify and leverage this underlying manifold structure? In this work, argue that the practice of smoothing in the logarithmic domain plays a key role.
2 Log-domain smoothing retains geometric structure
This work contributes small but growing literature the effect of scort-smoothing in diffusion models (Scarvelis et al., 2025] Chen, 2025) Gabriel et al., (025). In this section, we provide intuition for how diffusion models inherently perform smoothing in the log-density domain via their score matching objective. We then show that log-domain smoothing is crucial for preserving the underlying manifold structure the data. Finally, we revisit the manifold hypothesis and explore how specific characteristics of the smoothing kernel can guide the model to generalise along different geometries.
2.1 Diffusion models smooth in the log-domain
As outlined in the introduction, we model the inductive bias of neural network training by smoothing kernel & (see (5). Assuming that the nature of the inductive bias does not vary too rapidly over the spatial domain, can treat the kernel as locally constant. In this case, the convolution will commute with the gradient operation, and we obtain the following simple but consequential equation:
(6) Therefore, smoothing the score function corresponds to smoothing the empirical density P. in the log-domain, as opposed to smoothing at the density-level directly. Consequently, when trained diffusion model generates samples by following the reverse SDE in (2). effectively utilises scores derived from this log-smon fog-sheehhed version of the empirical density:
* (t, 1s * V (x) V (I+ log).(z)).
pt (dx) & exp (f (v)k dr,
where we use hy to denote the distribution of the smoothing kernel centred at 2.
Sampling from a diffusion model involves discretising the backwards process in (2) using the learned score function approximation. To correct for discretisation and approximation error, so-called corrector-steps interspersed between iterations (Song et al., 2021: Karras et al., 2022). This involves running Langevin Monte Carlo to correct the distribution of the diffusion model, maintaining correspondence between the diffusion model samples and the distribution associated with the score function. Furthermore, to account for instability near convergence, the technique early stopping is often used, where the reverse process is terminated an amount of time (Song et al., 2021]. With this, we arrive at our approximation of the diffusion model output as the log-de domain smoothed empirical measure, PE. Indeed, this is the density recovered by the diffusion model with score function s with sufficient correction steps and sufficiently fine discretisation.
before convergence
3

---

Figure 3: The smoothing kernel influences the manifold on which generated samples lie. The empirical score corresponding to training data (A) is smoothed with different kernels. We visualise the kernels with samples (+) from kg- We use the smoothed scores to generate samples (+). Despite using the same training data, different kernels generate samples that lie on different manifolds.
This characterisation of diffusion model output through smoothing in the log-domain identifies distinction between diffusion models and classical density-level estimators. For example, the classical kernel density estimation (KDE) (Tsybakov. (2009) approximates the underlying data distribution by (200))
smoothing the empirical measure Pdata with a kernel k, providing an estimator of the form,
4kDE (dx) =
Riz (y)) (dy)dr.
In words, the KDE also approximates the data distribution by smoothing the empirical data distribu- tion, but it performs its smoothing in the data-domain as opposed to the log-domain
2.2 Smoothing in log-domain preserves manifold structure
Capturing the geometry underlying the data distribution is critical aspect of effective generative modelling. We briefly provide some intuition for why log-domain smoothing plays vital role here. Consider a data distribution that is concentrated on manifold within the larger data space. Data-domain smoothing techniques, such as the KDE, yield a positive probability density wherever the smoothing kernel overlaps with the manifold, leading to a smearing of the density away from the manifold. In contrast, smoothing in the log- oog-domain offers a distinct advantage when we transition to the log-domain, locations where the original density is zero are mapped to -o. Consequently, if a smoothing kernel extends into regions off-manifold, the resulting smoothed log-density in those regions is effectively -00.
In Section 3 we make this intuition more concrete, theoretically showing that smoothing the empirical density in the log-domain approximates smoothing along the data manifold. We start by analysing the case in which the data supported on linear manifold (see Section 3.T. where we obtain a perfect correspondence between smoothing the empirical density in the log-domain and smoothing along the (linear) data manifold. Then, in Section 3.2 we state our main theoretical result that generalises this to the curved manifold setting- showing that smoothing in the log- log-domain approximates a geometry-adapter smoothing, which generates samples close to the underlying manifold.
2.3 Choosing an interpolating manifold via geometric bias
The manifold hypothesis traditionally assumes that data lies on low- www-dimnsional true submanifold. However, in many real-world scenarios, this assumption is too rigid: rather than adhering a single well-defined manifold, data likely exhibits geometric structure that different interpolating manifolds can approximate. This especially true when the size of the dataset is small relative to the dimension and curvature of the space. In such settings, the focus is no longer on recovering a frue manifold, but on choosing a plausible interpolating manifold. With this, we arrive at our next question of interest: how does the algorithm choose the interpolating manifold?
Returning to score smoothing, this reframes our central question to one of understanding how smoothing induces biases in the geometric structure of generated samples. As Figure illlsttrates, the geometry of the output distribution depends entirely on the directions of smoothing. By choosing a kernel that aligns with certain geometric structures (e.g., tangent to a circle), the diffusion model is biased to interpolate along the corresponding manifold. We term this relationship between sample geometry and the smoothing kernel-ov or more broadly, the inductive bias of the score matching algorithm-as as the geometric bias of the diffusion model. Section 4. we develop theory and experiments that identify key structural properties of the smoothing keree that dictate this bias.

---

3 aeometry-adaptiiity of log-domain smoothing
In this section, we provide theoretical results that aim capture and make concrete the intuition presented in Section 2.2 examining the smoothed density d in 7 as tractable proxy for the diffusion model output. We also consider manifold-adapter counterpart to p.. denoted by p. The kernel wy acts similarly to k. but restricts the smoothing to occur only along level sets of the manifold, spreading mass along the manifold without destroying the geometric structure. Note that a priori, one may not have knowledge of the manifold structure and thus could not construct such kernel here of a generative model: that interpolation identifies and preserves geometric structure. To show that smoothing with a generic kernel k is geometry-adaptin
it merely theoretical tool to represent a desirable behaviour
we wish show that & is close to its -anifold-adapted counterpart Pet In this section, we provide theoretical results analysing this relationship, and the properties of the data manifold and diffusion model that influence it.
3.1 Warm-up: linear setting
We first restrict our analysis to the setting where the data distribution is supported on d' -d-mensioonll
affine subspace M {a Rd Ar b}. where A (Rddd")) )xd is a row-orthonormal matrix and b (dd-")) This allows us provide intuition for our main result in Section 3.2 where we generalise the linear case. Consider the simplified setting where the kernel k is location indtionnnddeendeentt
h : law( + E),
where & is a zero-mean random variable independent of In this case, we have the following result. Proposition 3.1. The log domain smoothed density satisfies the property,
ph = P where k-M := law + PP),
(8)
where 1= AT is the projection onto Null(A) fxr E Rd Az = 0}.
The kernel k-M is modification of kg that smooths only along the plane parallel to A passing through z. From this proposition, we see that in the affine setting, smoothing in the log-domain with respect to a generic kernel is equivalent to smoothing with the geometry-adaue adapted kernel k.M In other words, log- domain smoothing is fundamentally geometry-adaptive.
We now provide brief exposition of the proof technique which also forms the basis of the proof in the more general setting. Given the training set { we can directly compute the noised empirical densities (x()) and the corresponding score functions. Recall that the LogSumExp (LSE) function is defined on any finite set {riks C IR and is given by LSE((rsh)) == log(E; (xp(r:)) Using this function, we can succinetly express the empirical log-density as
(r)) = LSE (( - - 2,,, - (2])(1)))) ) Ce.
(9)
for data- data-independent quantities Ce: Pt: ai given in Appendix B.1 We then use the following property. Fact 3.2. For any (ri}s C TR. and any constant e E TR, I.SE (ra + c)a) LSE((r:)) + c.
Using this fact, we can decompose the log-density into directions tangent and normal to the data manifold. Indeed, using the fact that x; E M we obtain,
log pz + E) = LSE { P x + - x2) + |A A(TA(T=)) A(x+ - 22)1)//220001 C.
L.S
{ [ [ (xx + -=))))1 + ||( + -11)))(00))) + Ce
= LSS
(1 P( + E x;))l² - ||A(z + E) b1| (20?) +Cr,
normal
tangent
In other words, interactions between the noise È and the data occur only in the tangent direction, and the normal direction is constant with respect the examples (2.).. Once taking the expectation of the above expression,
obtain that the log-density of P is identical, up constant, to the log-density of a which only applies smoothing in directions tangent to the manifold. We refer to Appendix for the complete derivation.
5

---

3.2 The case of curved manifolds
In this section, we state the main theoretical contribution of this work in which we show that smoothing
in log-density is fundamentally eeometry-adaptive Similar to the above analysis, we do this by
deriving relationship between p- using uninformed kernel k and pe using its manifold-adapted counterpart M We consider curved manifolds satisfying the following assumption.
Assumption 3.3. Suppose that pdata lies on smooth compact submanifold M RR, and that Adaaa restricted to M admits a density Pμ satisfying Sp == inf Pμ oF
Our approach to generalising the proof from the previous section is to use the defining feature of Riemannian manifolds- that locally the manifold behaves as if it were flat. The distance that one must be to the manifold depends on its curvature, which we control with an object from differential geometry called the reach. This object defines the maximum distance from the manifold for which the projection to the manifold, IL, is well-defined, i.e. a unique element of the manifold is closest. For example, if M were sphere, the reach would be the radius.
Assumption 3.4 (Manifold reach). The manifold M has a reach no smaller than r > 0, i.e. for all J E Rdd with dist(r: M) т, there exists a unique x² E . such that dist (x, M) || - x
The reach is inversely related to the maximum curvature of the manifold. Our assumption effectively requires that the curvature is globally upper-bounded. This assumption, as well as the lower bound on the density, have been used in several recent works and is common in the manifold hypothesis and manifold learning literature (Aamari, 2017 Potaptchik al., 2025 Azangulov et al., 2024). We refer to Appendix D.2 for further discussion and details regarding the reach of the manifold.
To generalise the manifold-adapted kernel in (8) consider the kernel's s projection onto level sets of (x, the manifold M, = {ir E R dist (x, M) = =. We define the manifold-adapted modification by
k (Ilp..Mar)) kek:
r(x) Ey wk (diist(y'' p
(10)
where McM is the element-wise scaling of M by jec and (T...M..s) )- denotes the push-forward by the projection mapping [.n. that is, the distribution (I...M)..,, Y ki The function r(z) approximates the distance of x to the manifold, but with some correction according to the variance of the kernel in directions normal the manifold. Therefore, similar the definition in (8. the kernel k-M is a modification of ki adapted to the geometry of M by smoothing only in directions tangential to the manifold. We refer to Appendix B.3 some additional details regarding the definition of k-M, including a discussion on the well- rell-posedness of the projection function.
The variance of the smoothing kernel k in directions normal to the manifold will prove to be an important object in our bound, leading us to make the following assumption.
Assumption 3.5. There are constants K, Kmax 2 0 such that for all x E Rdd Y ~ kz.
E[| distt(y, M)) dist(r, < K?, dist - dist(r: MM] S Kmaxx almost surely.
By measuring the change in distance to the manifold under smoothing, K and Kmax quantify noise in directions normal to the manifold and kernel adaptivity to the manifold structure. If & places most mass tangentially, then Y ~ ky stays near the level set through x and so K is small. For example, in the Gaussian case kg NNxz, o Ia). have that K z (d - d' ) whenever is taken small.
Unlike in the affine case, f.: and pt are not the same in general,
instead we show that these distributions are close. We consider the Rényi divergence, D,-a natural generalisation of the Kullback-Leibler divergence (which is the case g = 1). For the sake of brevity, we leave the definition and brief exposition on the Rényi divergence to Appendix B.2 and we state our main result.
Theorem 3.6. Under assumptions 3.3
3.4 and 3.5 and if Kmax T/96,, then for any q E [1, 1 Nmin(0) < Emax we obtain with probability at least - 8 that
T/ r/96K], 8 E (0, 1]. whenever N >
Dy( 11 S T
K
max { +1, (CEN) (-)))~--1 =c)
where the quantities Cmax and N (8) S (d' + log( (og(5-1)) ()) 20" are defined (53) and 45
3 'eere, we take Pa to be the density with respect to the volume measure the manifold M. which is itself inherited from the Lebesgue measure.
'Here, & denotes upper bound that ignores multiplicative logarithmic factors.

---

0- 0.1
d= 0.2
0.35
0.5
3.0
0.8
6.1
Smoottingg Parametee
Figure 4: Score smoothing can promote generalisation along curved manifolds, but too much smoothing distort the desired structure. Left: Training data (A) against generated samples (o) using isotropic Gaussian score smoothing with variance o². Right: Corresponding population negative log-likelhood, calculated for 1000 points the true circular manifold. See Appendix [..
For large N, the right-hand side depends only on dimension, curvature and kernel scale. This bound shows that for log-domain smoothing to become geometry-adaptive it is sufficient for the scale of smoothing normal to the manifold be small relative to the manifold curvature and dimension. When N small relative e-d", the bound becomes K/re Te, highlighting the role that early stopping plays in the data-sparse setting. The dependence on K also provides insight for how the behaviour of pt depends on the kernel's s manifold-alignment when & is already more aligned with the manifold structure, K is smaller, and the closer / is to its manifold-adapted counterpart P
We once again emphasise the key difference between the log- dogmanm smoothing that consider, and the traditional KDE approach which instead smooths the empirical measure in the density- -omain. As KDE bandwidth increases, samples rapidly leave the data manifold, whereas smoothed-scot score diffusion models produce new samples along the manifold structure without deviating far from it.
3.3
Log-domain smoothing and generalisation
So far, we have presented results pertaining to the similarity of D and its manifold-adapted counterpart ph While intuitively clear that smoothing with the manifold adapted kernel W-M will help promote generalisation, we provide two results to validate that this is indeed the case. The following result demonstrates that &ee preserves mass concentration around the manifold structure.
Corollary 3.7. Consider the setting of Theorem 3.6 then for any 5 E (0, 1J. whenever << 6max: > max(Nmiul =²,, we obtain that with probability at least 1 25 that,
P Fy (dist (Y,M) 2 + mm|) S 2exp(_r22/88
for all r 2 0,
where m > 0 satisfies m² x 22+ K max (dr +1, (c,N) -1/e" c1) (c/c,))
This corollary shows that the distance the manifold decays exponentially fast, at nearly the same rate as the noised empirical measure P., prior to smoothing. In other words, log-domain smoothing preserves concentration to the manifold. We note that when K is large, the concentration bound becomes less strong. Next, we show that smoothing with K- does indeed distribute mass along the manifold structure. We let T denote the space of vectors tangent M at æ.
Proposition 3.8. Consider the setting of Theorem 3.6 assuming further that Ey [dist(Y, (Y,M) 2 is constant in x E M. Let (0, 1] and suppose that N Nmin then, with probability at least 1 - 8, it holds that for any x E M,
16 Fl x;
px(x)) 2 De (x;) exp
x = argmin (2. | xill.
c2 1/(d++2) +2)
where F2_ = SUP
and I(x) E Radd is the Fisher information matrix of lig. Tols
The quantity F is an upper bound on the Fisher information of the kernel along the manifold. In the case where k is Gaussian kernel with variance o² we have that F 1/o. Thus, whenever is taken large, arbitrary points on the manifold receive similar density as the training data.
Together, propositions 3.8 and 3.7 show that log-domain smoothing distributes probability mass along the manifold while preserving geometric structure. While further work must be done to obtain rigorous generalisation bounds, the above results already suggest interesting relationship between
7

---

0.1 = 0.25
1.0
Figure 5: Different smoothing kernels can isolate alternative manifolds, given the same training data. Training data (A) against generated samples (+) using isotropic Gaussian score smoothing. By changing the smoothing variance o², different geometries are realised.
smoothing scale and generalisation: as smoothing grows, K increases, and once it becomes large relative to r/ed", the strength of the bound in Corollary 3.7 weakens; meanwhile F decreases, increasing the distribution of mass along the manifold. This suggests possible trade-off in the generalisation error that is governed by the scale of the smoothing and its relationship to c, d' and т. We explore this in Figure 4 where
plot population error (given by the negative log- log-likklihood) against scale of smoothing and observe a U-shaped curve, suggesting moderate smoothing can improve over the empirical measure P. while oversmoothing can worsen generalisation.
4 Rethinking the manifold hypothesis: geometry and inductive bias
So far, we have considered the traditional setting of the manifold hypothesis, where the goal is recover the data's geometry under well-defined ground-truth manifold. Yet with finite data-especially in high dimensions-mant plausible interpolations fit. In practice, "correctness" is task-dependent: if interpolation meets application-speccfic criteria, the generative model is deemed successful. Consequently, practitioners are not recovering a single true interpolation; network architectures and training algorithms implicitly inject biases that steer models toward desirable behaviours. This motivates shift in perspective in this section: rather than assuming a ground truth manifold, we study how inductive biases make the model choose an interpolating manifold. We refer to this form of bias as the model's geometric bias.
4.1 Geometric bias of log-domain smoothing
In Figure study toy case where data (red triangles) admit several plausible interpolating manifolds. When the smoothing kernel is aligned with the wavy-cirele level sets, the generated samples (blue) remain faithful to the wavy geometry, while aligning with the base circle yields circular samples. This indicates that smoothing the empirical score parallel or tangentially to a target manifold M induces geometric bias toward it. The right of Figure shows that tailored kernels can even alter dimension and connectivity. In Figure we consider isotropic Gaussian smoothing. Here, the scale of the smoothing controls the bias: small bandwidths preserve fine waviness, larger bandwidths recover the broader circular shape, and excessive noise leads to eventual sample collapse.
4.2 Geometry-adaptivity and geometric bias
The theoretical analysis of Section can also be extended to the present setting, allowing to further elucidate the relationship between the smoothing kernel and geometric bias. In particular, provide modification of Theorem 3.6 that quantifies how well log-domain smoothing adapts to different manifolds, without the requirement for the data to belong to that manifold.
We define the set of permissible manifolds M,, to be the set of all smooth compact submanifolds M C Rd with non-zero reach TM 0, satisfying the property CM := ess inf pasta Pn.M > 0 where Pr.M denotes the density of (IIM)s ) with respect to the volume measure on M. In other words, M,, consists of all manifolds with bounded curvature, such that the projection of faata onto M has full support. Furthermore, given any M E Ml,, we let diM denote the manifold dimension and define
K2 sup By-k, M di,
Kmax,M dist Mi,
zERd
8

---

Increasing Smoothing
4
4
4 4
1.2
Dota Point
Hestance to
4
0.0
Increasing KDE Bandwidth -
4
4
4
4
0.50 0.75 1.00 1.25
Figure 6: As smoothing is increased, generations from the score-smoothed diffusion model remain in the man- ifold structure. In contrast, samples from the KDE quickly deviate, leading to poor reconstructions.
Figure 7: Comparison of L distance to closest point training data and closest point in M. Arrows indicate increasing amounts of smoothing.
With this, we can state our second main result.
Theorem 4.1. Let M M,, and AM dist({z,) I. Then, for any 8 E (0, 1], whenever Maxx + Am S TM/96 and N,, is sufficiently large, with probability at least 1 ó we have
Ku(dmm +1) KMAM
K
AM
(11)
TM
TM
While Theorem 3.6jeonroo how closely log-domain smoothing adapts to an underlying true geometry, Theorem 4.1 instead bounds how well any given manifold describes the geometry induced by log- domain smoothing. manifold M E M,, makes the right-han side of this bound small, then smoothing under generic kernel similar to smoothing under the the M-adapped kernel k-M, Indeed, such manifold M effectively captures the geometric bias of the smoothing kernel k. Thus we can identify favoured manifolds by optimising the right-han side with respect to M E Mμ-
Analysing shows how the smoothing kernel drives geometric bias. The bound trades off curvature TM against interpolation error AM. M. modulated by KM. d' and c. For small the second term can dominate, optimisation favours manifolds with small A,M, even the expense of larger TM- As smoothing increases, KM grows and the KM term dominates, shifting preference toward lower-curvature manifolds (as in figures 3 and 5). Moreover, if k, emphasises certain directions in its smoothing, then choosing M tangent to them keeps KM small, yielding low-dimensional manifold M aligned with those directions that minimises AM and diy-- That is. low-dimensional manifold that is tangent to the smoothing directions while best interpolating the data.
5 High-dimensional experiments
So far, our experiments have focused on illustrative low-dimensional settings to complement our theory. In this section, consider nigher-dimensiondl settings to investigate the extent to which the identified phenomena persist in scenarios more representative of practical applications.
5.1 Generation in latent space
We begin with MNIST and define 32-dimensional VAE latent space (following Rombach et al. (2022)). We study the digit-4 manifold M, which comprises a lower-dimensional structure in the latent space. This ground-truth manifold is approximated using all samples of the digit 4, from which we use a subset 100 samples as our training dataset. We compare a amoothed-score diffusion model using isotropic Gaussian kernel to KDE, which corresponds to density-level smoothing.
To assess how well the manifold structure preserved, we visualize samples as smoothing increases in Figure 6 The top and bottom rows display samples from score-smoothe mmoothed diffusion model and KDE, respectively. different smoothing scales. With score smoothing, generations perfectly recover training examples (plotted on the left) at small smoothing levels. As the amount of smoothing increases, the samples become progressively novel images that are not present in the dataset yet nonetheless decode to resemble 4's, indicating mass is spread primarily along the underlying geometry. KDE, contrast, deteriorates more as bandwidth grows, as generated samples move substantially off- manifold. We provide quantitative assessment of this behaviour by reporting the average distance of

---

1.50
122
KDe
Cousriar
Adipiid
120
Cocmite Adapied
0.4
Casaoian Adaptes
80
Quueiian Adagted
1.00
2.15
Distance
0.4
60
0.5
1.85
20
1.00
0.5
1.2
1.5
3.5
1.0
1.5
0.2 0.4 Distannc to data
0.8
0.0
0.2
0.4
0.5
0.6
Distance to data
Distanre data
Distanow to data
Figure 8: Comparison of smoothing kernels for the synthetic 'bump' image manifold. Left: Comparing L2 distance to data and M. Right: Anisotropy of generated 'bump' samples.
Figure 9: Comparison of smoothing kernels for the MNIST image manifold. Left: Comparing L2 distance to data and M. Right: FID of gener- ated samples.
samples to the manifold against distance to the training set in Figure 7 As KDE smoothing increases, the distance to the manifold increases identically with the distance to the data, whereas the diffusion samples sometimes become closer to other 4s not used for training. See Appendix G.2 for details.
5.2 Generation in pixel space
While latent-space generation is common, diffusion models also succeed directly in pixel space. We therefore repeat the analysis there, focusing on 1d image manifolds where the geometry can be controlled and evaluated. Data is naturally more separated pixel space, yielding many permissible manifolds; we leverage this to test how kernel choice affects the geometry of the sampling distribution.
Synthetic image manifold We construct a closed 1d manifold ø : [0, - 64x66 that maps an angle an image of bump function centred the corresponding angle around circle, and form dataset of 16 equidistant points on this curve. For a visualisation of the manifold, see Appendix G.3 We compare isotropic Gaussian score smoothing against the KDE by plotting eestance-to-manifol versus distance-to-data across smoothing scales in Figure g We further test a manifold-alapte score- smoothing kernel that translates the manifold M to pass through the current point and smooths along it. Crucially, the sampler "knows" the manifold only via the smoothing mechanism; the empirical score itself uses only the training dataset. The adapted kernel produces samples significantly closer to the true manifold than isotropic Gaussian smoothing, supporting the discussion in Section 4 As measure of how visually man--anifold" the generations are, we also report the anisotropy of the generated bump samples in Figure [ The samples using the adapted smoothing exhibit lower anisotropies as the degree of smoothing increases, indicating that they appear visually closer the data manifold than those obtained using Gaussian smoothing.
MNIST manifold pixel space We repeat the pixel-space study on MNIST by constructing an explicit image-space manifold Φ J0, 1] R 32x32 by decoding curve in VAE latent space; the VAE only defines the manifold, and the diffusion runs entirely in pixel space. In Figure D we plot distance the manifold M versus distance the training set, again finding that manifold-adapted smoothing stays closer to M than Gaussian smoothing. As complementary measure of visual quality, Figure 9 also reports FID held-out test set and shows a consistent benefit for score-function smoothing. Gaussian smoothing is known to produce barycentres of training datapoints (Scarvelis et al., 2025). which in pixel space appears as increasing blurring and yields a steep increase in the FID at larger smoothing. The manifold-an manifold-adapted kernel mitigates this blurring and avoids the same increase in FID value. Full details and additional plots for different curves ø appear in Appendix G.4
Conclusion
In this work, we have investigated how implicit regularisation caused by smoothing of the empirical score function interacts with the manifold structure of data. In particular, we identify that smoothing at the level of the log-domain implicitly geometry-adaptive, behaving similarly to a manifold-adapter kernel when given enough samples. Beyond the data-rich setting, we observe that the choice of smoothing kernel can shape the generated distribution. Future work could examine how inductive biases in deep learning architectures and training influence the smoothing that occurs in practice.
10

---

Acknowledgements
Tyler Farghly was supported by Engineering and Physical Sciences Research Council (EPSRC) [grant number EP/T517811/1] and by the DeepMind scholarship. Peter Potaptchik is supported by the EPSRC CDT Modern Statistics and Statistical Machine Learning [EP/S023151/1]
Google PhD Fellowship, and an NSERC Postgraduate Scholarship (PGS D). Samuel Howard is supported by the EPSRC CDT Modern Statistics and Statistical Machine Learning [grant number EP/S02315 15111. George Deligiannidis and Jakiw Pidstrigach acknowledge support from EPSRC [grant number EP/Y018273/1). The authors would like to thank Iskander Azangulov and Arya Akhavan for stimulating discussions and Ioannis Siglidis for valuable feedback.
References
Aamari, Eddie (Sept. 2017). "Convergence Rates for Geometric Inference". PhD thesis. Université Paris-Saclay.
Aamari, Eddie, Jisu Kim, Frédéric Chazal, Bertrand Michel, Alessandro Rinaldo, and Larry Wasser- man (2019). "Estimating the reach of a manifold". In: Electronic Journal of Statisties 13.1, pp. 1359-1399.
Azangulov, Iskander, George Deligiannidis, and Judith Rousseau (2024). "Convergence of Diffusion Models Under the Manifold Hypothesis in High- Sigh--Dmensioos.. In: arXiv (stat.ML) arXiv: 2409
18804 [stat. ML]
Baptista, Ricardo, Agnimitra Dasgupta, Nikola B. Kovachki, Assad Oberai, and Andrew M. Stuart (2025). "Memorization and Regularization in Generative Diffusion Models". In: arXiv [cs.LG). arXiv: 2501 15785 [G]
Bengio, Yoshua, Aaron C. Courville, and Pascal Vincent (2012). "Representation Learning: A Review and New Perspectives". In: IEEE Transactions on Pattern Analysis and Machine Intelligence 35, pp. 1798-1828.
Biroli, Giulio, Tony Bonnaire, Valentin de Bortoli, and Marc Mézard (Nov. 2024). "Dynamical regimes of diffusion models". In: Nature Communications 15.1.
Bobkov, S G (2003). "Spectral gap and concentration for some spherically symmetric probability measures", In: Lecture Notes Mathematics. Lecture notes mathematics. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 37-43.
Brown, Bradley CA, Anthony L. Caterini, Brendan Leigh Ross, Jesse C Cresswell, and Gabriel Loaiza-Ganem (2023). "Verifying the Union of Manifolds Hypothesis for Image Data". In: The Eleventh International Conference on Learning Representations.
Carlini, Nicolas, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace (Aug. 2023). "Extracting Training Data from Diffusion Models". In: 32nd USENIX Security Symposium (USENIX Security 23). Anaheim, CA: USENIX Association, pp. 5253-5270.
Chen, Minshuo, Kaixuan Huang, Tuo Zhao, and Mengdi Wang (2023). "Score Approximation, Estima- tion and Distribution Recovery Diffusion Models on Low-Dimensional Data". In: Proceedings of the 40th International Conference on Machine Learning.
Chen, Ricky T. Q., Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud (2018). "Neural Ordinary Differential Equations". In: Advances in Neural Information Processing Systems.
Chen, Sitan, Vasilis Kontonis, and Kulin Shah (2024). "Learning general Gaussian mixtures with efficient score matching". In: arXiv [Cs. Xiv: 2404. 18893 [cs.DS]
Chen, Zhengdao (2025). "On the Interpolation Effect of Score Smoothing". In: arXiv [cs.LG], arXiv: 2502. 19499 [c]
Cheng, Ming-yen and Hau-tieng Wu (2013). "Local Linear Regression on Manifolds and Its Ge- ometric Interpretation". In: Journal of the American Statistical Association 108.504, pp. 1421- 1434.

---

Chewi, Sinho, Murat A Erdogdu, Mufan Li, Ruoqi Shen, and Shunshi Zhang (2022). "Analysis of Langevin Monte Carlo from Poincare to Log-Sobolev". In: Proceedings of Thirty Fifth Conference on Learning Theory.
Cohen, Samuel N and Robert James Elliott (2015). Stochastic calculus and applications. Vol. 2. Springer.
De Bortoli, Valentin (2022). "Convergence of denoising diffusion models under the manifold hypoth- esis". In: Transactions on Machine Learning Research.
Dhariwal, Prafulla and Alexander Quinn Nichol (2021). "Diffusion Models Beat GANs on Image Synthesis". In: Advances in Neural Information Processing Systems.
Erdogdu, Murat A, Rasa Hosseinzadeh, and Shunshi Zhang (2022). "Convergence of Langevin Monte Carlo in Chi-Squared and Rényi Divergence". In: Proceedings of The 25th International Conference on Artificial Intelligence and Statisties.
Erven, Tim van and Peter Harremoes (July 2014). "Rényi Divergence and Kullback-Leibler Diver- gence". In: IEEE Trans. Inf. Theory 60.7. pp. 3797-3820.
Farghly, Tyler, Patrick Rebeschini, George Deligiannidis, and Arnaud Doucet (2025). "Implicit regularisation in diffusion models: An algorithm-dependen generalisation analysis". In: arXiv [stat.ML). arxiv: 2507 2507.03756 03756 [stat.MI)
Fefferman, Charles, Sanjoy Mitter, and Hariharan Narayanan (2016). "Testing the manifold hypothe- sis". English (US). In: Journal of the American Mathematical Society 29.4, pp. 983-1049.
Gabriel, Franck, François Ged, Maria Han Veiga, and Emmanuel Schertzer (2025). "Kernel-Smootie Scores for Denoising Diffusion: A Bias- Variance Study". . In: arXiv [cs. arXiv: 2505 22841
[CS.LG]
Gao, Jia-Xing, Da-Quan Jiang, and Min-Ping Qian (2022). "Adaptive manifold density estimation".
In: Journal of Statistical Computation and Simulation 92.11, pp. 2317-2331.
Genovese, Christopher, Marco Perone-Pacifico, Isabella Verdinelli, and Larry Wasserman (2012). "Minimax Manifold Estimation". In: Journal of Machine Learning Research 13.43, pp. 1263-1291.
Goodfellow, Ian, Yoshua Bengio, and Aaron Courville (2016). Deep Learning. MIT Press.
Gray, Alfred (2004). Tubes. Basel: Birkhäuser Basel.
Gu, Xiangming, Chao Du, Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang (2023). "On Memo- rization in Diffusion Models". In: arXiv [cs.LG). arXiv: 2310-02664 [ca LG]
Handel, Ramon van (2014). "Probability in high dimension". In: Lecture Notes (Princeton University).
Haussmann, Ulrich G and Etienne Pardoux (1986). "Time reversal of diffusions". In: The Annals of Probability, pp. 1188-1205.
Heusel, Martin, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter (2017). "GANs Trained by a Two Time-Scua Update Rule Converge to a Local Nash Equilibrium". In: Advances in Neural Information Processing Systems.
Ho, Jonathan, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans (2022). "Imagen Video: High Definition Video Generation with Diffusion Models". In: arXiv [cs. CVJ. arxiv: 2210.02300 02303
[vs
Ho, Jonathan, Ajay Jain, and Pieter Abbeel (2020). "Denoising Diffusion Probabilistic Models". In: Advances in Neural Information Processing Systems.
Hyvärimen, Aapo (2005). "Estimation of Non- Normalized Statistical Models by Score Matching". In: Journal of Machine Learning Research 6.24, pp. 695-709.
Kadkhodaie, Zahra, Florentin Guth, Eero P Simoncelli, and St' speehe Mallat (2024). "Generalization in diffusion models arises from geometry -ddatiive harmonic representation". In: The Twelfth International Conference on Learning Representations.
12

---

Kamb, Mason and Surya Ganguli (2025). "An analytic theory of creativity in convolutional diffusion models". In: Forty-second International Conference on Machine Learning.
Kamkari, Hamidreza, Brendan Leigh Ross, Rasa Hosseinzadeh, Jesse C. Cresswell, and Gabriel Loaiza- Ganem (2024). "A Geometric View of Data Complexity: Efficient Local Intrinsic Dimen- sion Estimation with Diffusion Models". In: The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Karras, Tero, Miia Aittala, Timo Aila, and Samuli Laine (2022). "Elucidating the Design Space of Diffusion-Basee Generative Models". In: Advances in Neural Information Processing Systems.
Kingma, Diederick P and Jimmy Ba (2015). "Adam: A method for stochastic optimization.. In: International Conference on Learning Representations (ICLR).
Kingma, Diederik P. and Max Welling (2014). "Auto-Encoding Variational Bayes". In: International Conference on Learning Representations.
Kong, Zhifeng, Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro (2021). "DiffWave: A Versatile Diffusion Model for Audio Synthesis", In: International Conference on Learning Repre- sentations.
LeCun, Yann, Corinna Cortes, and CJ Burges (2010). "MNIST handwritten digit database". In: ATT Labs [Online]. Available: http://yaann. www.mn.eecc.c.om coccm..com/exxb/misstttt 2.
Li, Gen and Yuling Yan (2024). "Adapting to Unknown Low- Son-Dimensiona Structures in Score- Based Diffusion Models". In: The Thirty- Annual Conference on Neural Information Processing Systems.
Liu, Haohe. Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley (2023). "AudioLDM: Text-to-Audio Generation with Latent Diffusion Models". In: Proceedings of the 40th International Conference on Machine Learning.
Sraiza-Ganem Gabriel, Brendan Leigh Ross, Rasa Hosseinzadeh, Anthony L. Caterini, and Jesse C. Cresswell (2024). "Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections", In: Transactions on Machine Learning Research.
Ma, Chao and Lexing Ying (2021). "On linear stability of SGD and input-smoothness of neural networks". In: Advances in Neural Information Processing Systems.
Mironov, Iya (Aug. 2017). "Rényi Differential Privacy". In: 2017 IEEE 30th Computer Security Foundations Symposium (CSF). IEEE
Moscovich, Amit, Ariel Jaffe, and Nadler Boaz (2017). "Minimax semi-supervised regres- sion on unknown manifolds". In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics.
Mousavi-Hosseini, Alireza, Tyler K Farghly, Ye He, Krishna Balasubramanian, and Murat A Erdogdu (2023). "Towards a Complete Analysis of Langevin Monte Carlo: Beyond Poincaré Inequality", In: Proceedings of Thirty Sixth Conference on Learning Theory. Ed. by Gergely Neu and Lorenzo Rosasco. Vol. 195. Proceedings of Machine Learning Research. PMLR, pp. 1-35.
Mulayoff, Rotem, Tomer Michaeli, and Daniel Soudry (2021). "The implicit bias of minima stability: A view from function space". In: Advances in Neural Information Processing Systems.
Niedoba, Matthew, Berend Zwartsenberg, Kevin Patrick Murphy, and Frank Wood (2025). "Towards Mechanistic Explanation of Diffusion Model Generalization". In: Forty-second International Conference on Machine Learning.
Oko, Kazusato, Shunta Akiyama, and Taiji Suzuki (2023). "Diffusion Models are Minimax Opti-
mal Distribution Estimators". In: Proceedings of the 40th International Conference on Machine Learning.
Pavliotis, Grigorios (Nov. 2014). Stochastic Processes and Applications: Diffusion Processes, the Fokker-Planck and Langevin Equations. en. Springer.
Pidstrigach, Jakiw (2022). "Score-Based Generative Models Detect Manifolds". In: Advances in Neural Information Processing Systems.
13

---

Pope, Phil, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein (2021). "The Intrinsic Dimension of Images and Its Impact on Learning". In: International Conference on Learning Representations.
Potaptchik, Peter, Iskander Azangulov, and George Deligiannidis (2025). "Linear Convergence of Diffusion Models Under the Manifold Hypothesis". In: The Thirty Eighth Annual Conference on Learning Theory.
Rahaman, Nasim, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville (2019). "On the Spectral Bias of Neural Networks", In: Proceedings of the 36th International Conference on Machine Learning.
Raya, Gabriel and Luca Ambrogioni (2023). "Spontaneous symmetry breaking in generative diffusion models". In: Thirty-sevetth Conference on Neural Information Processing Systems.
Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer (2022). "High-resolution image synthesis with latent diffusion models". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 10684-10695.
Scarvelis. Christopher, Haitz Sáez de Ocáriz Borde, and Justin Solomon (2025). "Closed-Forn Diffusion Models". In: Transactions on Machine Learning Research.
Shah, Kulin, Sitan Chen, and Adam Klivans (2023). "Learning Mixtures of Gaussians Using the DDPM Objective". In: Thirty-seventh Conference on Neural Information Processing Systems.
Sohl- Sohl-Dickstein, Jascha, Eric Weiss, Niru Maheswaranathan. and Surya Ganguli (2015). "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" In: Proceedings of the 32nd International Conference on Machine Learning.
Somepalli, Gowthami, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein (2023).
"Diffusion art or digital forgery? investigating data replication in diffusion models". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (048-6058.
Song, Yang and Stefano Ermon (2019). "Generative Modeling by Estimating Gradients of the Data Distribution". In: Advances in Neural Information Processing Systems.
Song, Yang. Jascha Sohl-Dicksted Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole (2021). "Score-Based Generative Modeling through Stochastic Differential Equations". In: International Conference on Learning Representations.
Stanezuk, Jan Pawel, Georgios Batzolis, Teo Deveney, and Carola- Bibiane Schönlieb (2024). "Dif- fusion Models Encode the Intrinsic Dimension of Data Manifolds". In: Forty-first International Conference on Machine Learning.
Tang, Rong and Yun Yang (2024). "Adaptivity of Diffusion Models to Manifold Structures". In: Proceedings of The 27th International Conference on Artificial Intelligence and Statistics.
Tenenbaum, Joshua B., Vin de Silva, and John C. Langford (2000). "A Global Geometric Framework for Nonlinear Dimensionality Reduction", In: Science 290.5500, pp. 2319-23223
Tsybakov, Alexandre B (2009), "Nonparametric estimators". In: Introduction to Nonparametric Estimation, pp. 1-76.
Vardi, Gal (2023). "On the implicit bias in deep-learning algorithms". In: Communications of the ACM 66.6, pp. 86-93.
Vastola, John (2025). "Generalization through variance: how noise shapes inductive biases in diffusion models". In: The Thirteenth International Conference on Learning Representations.
Vempala, Santosh and Andre Wibisono (2019). "Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry Suffices". In: Advances in Neural Information Processing Systems.
Ventura, Enrico, Beatrice Achilli, Gianluigi Silvestri, Carlo Lucibello, and Luca Ambrogioni (2025). "Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusion". In: The Thirteenth International Conference on Learning Representations.
Wainwright, Martin J (2019). High-Dimensional Statisties: A Non-Asymptotic Viewpoint Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press.
14

---

Wang. Peng, Huijie Zhang, Zekai Zhang, Siyi Chen, Yi Ma, and Qing Qu (2024). "Diffusion Models Learn now-Dimensional Distributions via Subspace Clustering". In: arXiv [cs. arXiv: 2409.
Wen, Yuxin, Yuchen Liu, Chen Chen, and Lingjuan Lyu (2024). "Detecting, Explaining, and Mitigat- ing Memorization in Diffusion Models". In: The Twelfth International Conference on Learning Representations.
Weyl, Hermann (1939). "On the volume of tubes". In: American Journal of Mathematics 61.2, pp. 461-472.
Yoon, TacHo, Joo Young Choi, Schyun Kwon, and Ernest K. Ryu (2023). "Diffusion Probabilistic Models Generalize when They Fail to Memorize". In: ICML 2023 Workshop on Structured Probabilistic Inference & Generative Modeling.
Zhang. Huijie, Jinfan Zhou, Yifu Lu, Minzhe Guo, Peng Wang, Liyue Shen, and Qing Qu (2024). "The Emergence of Reproducibility and Consistency in Diffusion Models". In: Forty-first International Conference on Machine Learning.
15

---

Technical Appendices and Supplementary Material
A Extended Discussion
Related work In the manifold setting, Pidstrigach (2022) and De Bortoli (2022)
provide precise
convergence bounds for diffusion models. Recently, there has been surge of interest in providing
refined results in the manifold setting (Oko et al., 2023] Chen et al., [023] Tang and Yang, (2024] Li and Yan. 2024 Azangulov et al.. 2024 Potaptchik et al., 2025). or under other specific structural settings (Shah et al., 2023] Chen et al., 2024; Wang et al., 2024). Additionally, many works have focused on empirically validating the manifold hypothesis for data such images (Fefferman et al., 2016 Pope et al., 2021; Stanczuk et al., 2024 Brown et al., 2023 Kamkari et al., 2024). Our work also shares similarities with wider literature regarding how manifold structure interacts with learning tasks, such as Genovese et al. (2012), Cheng and Wu (2013). Moscovich et al. (2017). and Gao et al. (2022).
Recently, there has been an increased interest in understanding generalisation and memorisation in diffusion models. Memorisation of training data has been observed empirically by Somepalli et al. (2023) and Carlini et al. (2023) when the capacity of the network is large relative to the number of training samples. Other works investigate how inductive biases of neural network architectures aid in generalisation (Kadkhodaic et al., 2024; Niedoba et al., 2025 Kamb and Ganguli, (0225). The recent work of Vastola (2025) examines the role of noise in the objective. The dichotomy between generalisation and memorisation has been investigated in Yoon et al. (2023). Gu et al. (2023), Wen et al. (2024), Zhang et al. (2024), and Baptista et al. (2025). The works of Raya and Ambrogioni (2023), Biroli et al. (2024), and Ventura et al. (2025) examine the roles of distinct regimes in the generative process.
This work contributes to a small but growing line of research into the effect of score-smoothing in diffusion models. Scarvelis et al. (2025) previously studied isotropic Gaussian and Gumbel smoothing of the score function, as a training-free free alternative for running diffusion models, and show that this generates barycentres of training datapoints. Chen (2025) investigates the effect of score smoothing on generalisation in the Id linear setting. Concurrently, the work of Gabriel et al. (2025) also studies the effeet of kernel-smoothe score function and its relation to preserving manifold structure, though with different analysis techniques.
Limitations and further investigations Our argument that the log-domain smoothed measure p approximates the output of the diffusion model with score smoothing relies on the exchangeability of gradients (see (6)) a property that holds for location-independent kernels. Extending our framework to the more general case of leeation kernels is important next step. Similarly, our key theorems (theorems 3.6 and 4.1 currently require the scale of noise normal to the manifold, K, to be small relative to its curvature, T. A more complete characterisation of the geometric bias would therefore require relaxing this assumption. Furthermore, a more robust characterisation of the geometrie bias of log-domain smoothing would also benefit from a matching lower bound on the Rényi divergence. While we have demonstrated through heuristic arguments how our results show the generalisation potential of log-domain smoothing, our work stops short of deriving a formal generalisation error bound. Whether log-domain smoothing alone could produce optimal generalisation bounds is a question that we leave open for future investigation. Finally, we believe this theoretical framework could be valuable tool for the related literature analysing the memorisation and privacy properties of diffusion models.
The experiments presented here illustrate that the type of smoothing can influence the geometric structure of the generated samples. However, they also highlight the challenges posed by high- curvature manifolds, where smoothing the empirical score alone may not suffice. Real-world image manifolds tend be highly curved, yet diffusion models still generalize well from relatively few training samples (Kadkhodaie et al., 2024). Furthermore, in practical settings we do not have knowledge of the ground-truth manifold structure to explicitly apply manifold adaptive smoothing. If such adaptation occurs, it must arise implicitly from the model's inductive biases. This suggests that other factors must also be at play, such as biases induced by neural architectural choices like convolutions and attention (Kamb and Ganguli, 2025). We do not examine the behaviour of such practical architectures in this work, but remark that understanding to what extent architectural designs choices interact with the ideas presented here is an interesting direction for future study.
16

---

B Notation and omitted details
In this section, we include some technical details concerning the theoretical results of the paper that were omitted for the sake of readability.
B.1 Properties of the forward process
we introduced the forward process in Throughout the proofs, we use the following property the forward process:
In Section I
of a (a-1(1 pL), if 0, 21,
X, N
Pt =
(12)
otherwise.
When a O, this follows immediately from properties of the Wiener process and when a X becomes the CeattinnUhhlenbekk process and the result follows from standard analysis (e.g. see Pavliotis, 2014 Cohen and Elliott, 2015).
Using this fact, we have the following closed-form form expression for pr, the density of the empirical forward process Xx:
N px, xo (xl(z))
1
logı: = log
N
1=1
1
d log(2noo})
= log
exp( - || -1:0:11/220
N
1=1
N u 1|2 11/
|| {
= LSE
+ Ct.
i=1
where Ct - log(N) - & log(2TTo) and we recall the definition of the function LSE((r:)))
log(2: exp(r:)).
B.2 Rényi divergence
We provide brief exposition of the Rényi divergence, which is measure of difference between two measures. Given two measures p,, on Rdd and q E (1, 0o) we define q-Rényi divergence by
T-T log S ( (x)) u(dz)) if M v,
Do(ullu)
0C,
otherwise.
For the case of q = 1 we set Dg to be the Kullback- (KL) divergence,
[llo de (x) u(dx), if J < v, otherwise.
Di(u(l>)
00,
Indeed, whenever (u(l() < for some q 1, it be shown that lim Dy(u(ll) Di(pllv). Furthermore, whenever de is bounded p-almost surely, we obtain,
dp
lim Dy(ullv)= = log [SsS (csssup, du -000
which is taken be (ullv)) Thus, the Rényi divergence provides natural interpolation between the KL divergence and the worst-case regret, with D, increasing in This measure of distance recently gained popularity in the sampling (Vempala and Wibisono, 2019, Chewi et al., 2022 Erdogdu et al., 2022. Mousavi-Hossein et al., 2023) and privacy (Mironov, (017)) literatures as a stronger alternative to traditional divergences. We refer to (Erven and Harremoes, 2014) and (Chewi al., 2022) for further properties of this divergence.
17

---

B.3 Projections
Throughout this work, we frequently utilise the projection mapping M R + MM which maps E IR" to the nearest element of M. In cases where M is curved, we run in to the issue that the projection is not well-defined as there could multiple elements that are equally close to æ. In most places in the proof we consider quantities that are sufficiently close that the projection function is uniquely defined (see reach in Appendix (.2) but, for example, when we define the manifold adapted kernel in 10), use the projection for all E Rd.
Throughout the proofs of this work we do not utilise any property of the projection aside from the fact that it maps z to some element of the manifold that is of distance dist (M, M) away from æ. For that reason, II can be taken be any mapping onto M such that I| [I) dist(r; MM). Since M is taken to be closed set, such a mapping always exists and we will take this choice of mapping to be fixed throughout the work.
When a > 0 the samples generated at early stopping time are slightly biased due to contractions of the
process. For this reason, will frequently consider the contracted manifold,
M {uz M).
where Me is as defined in [12. and will frequently the shorthand M := M.M Given projection mapping 11M onto M, we take the projection mapping 11M onto M to be given by
IT (x) p.Il(p//p.)
C Madtiitd- in the affine setting
We begin with the proof of Proposition 3.1 concerning the affine setting. Recall the assumption that the support of Adata is restricted to the affine subspace M = {x E R Ax = 6}, where A E yx is cow-orthonormal and Rdd-d"), and we write the smoothing kernel in the following form:
he == law xx +),,
where & is centred random variable independent of Throughout the proof, we use the null space projection matrix P := I- ATA.
Proof of Proposition 3.1 posed as
Since is the projection matrix onto Null(A), any 2 E Rd can be decom-
|| = |||2|| + [I( - P)=|1 PP|2 || Az|1²
|P|1 + +1A=21
where the final line follows from the fact that A is row-orthonormal and so AAT = 1(d-a0). Using fact 3.2 and the assumption that Az, = for every i E [N], we obtain the identity
|| x.,13
log (x) LSE
5 C.
(13)
20?
P((z 21)112 1|A(2z -2:)13
}}
LSE
+ +
(14)
20
P((2 a;)113
}))
||
b11
LSE
+ Ct.
(15)
20
20
This decomposition separates the influence of 2 into normal and tangent directions with respect to M.
Now, let Rdd and define Y, = (xx + ~ hn and Y, = (x + PE) KM, Since P p2, we observe that
P(Y - x3)1l = P( - =z)]l
18

---

Furthermore, using the fact that & is centred, we also have,
BI|IAY, b = | bl|² E [| EiILAY, b11 || Bl1 + Bil|
In particular, substituting into 15 and taking the expectation, we conclude that
{{
1(Y 22)11
}.)1
Ax bll2
Bllog (.
E LSE
+ C,
20
20
and
{
[P(Y, xz x1)11
}.)1
Ax
b11
[e
LSE
+ C
20
20
for constants C,C independent of æ. Therefore, the log-density of p. and D. are identical up to constant.
D Lemmata
For proving the results for the more the general manifold setting, require several additional properties of the log-sum-exp function, smooth manifolds and tubular neighbourhoods. In this section we collect these results.
D.1 Stability of the LSE function
The following lemma provides stability bounds for the LSE function which we make use of throughout our analysis. It can be seen as generalisation of Fact 3.2 which is heavily used in the analysis of the affine case.
Lemma D.1. For any { C and ( R. we have
LSF((#; (s) LSF (fx)) L
N i-n explx; + rei)es dr.
(16)
-i=1 exp(x; + rei)
In particular, we have that
LSE({x, + - LSE((1,) s maxfci).
(17)
Proof. From the chain rule, we compute the partial derivatives,
a LSE((2a)))
exp(x;)
dx,
Σn exp(x.)
Therefore, by the fundamental theorem of calculus, obtain that
a Ej Dz,
LSE([x; +.)) LSE({2))))
LSE({2; + recl:)
+ e dr,
ΣE exp(x; + rc;)
completing the proof of (16). To obtain 17 we use the fact that the sum is weighted average of the sequence {e and so it has the property that,
exp(x; + re:lc
S maxe; EG.
E/=r exp(x) (2,, + re,)
We also note that if all e, are identical, readily recover Fact 3.2 from 16
19

---

D.2
Manifold reach
Next, we collect some facts about the reach of the manifold, which quantifies how far one can extend from the manifold before the projection onto it ceases to be unique. We refer to (Aamari, (007)) for more detailed exposition. We begin with the rigorous definition.
Definition D.2. The reach of set C Rd, defined by тΛ infpea dist Med((A) ()), where we define the set,
Med(A) = f R² =..g Ast. q, 1p 21| 119 z|| dist(2, A).
The reach defines the maximum distance at which the projection to the set is unique. In the case where the set A is a smooth submanifold of IR, the reach captures the curvature of the manifold and provides an upper bound on the distance at which the manifold appears approximately flat. The following lemma demonstrates this, controlling the curvature of paths along the manifold using the reach. We use the notation N to denote the normal space of the manifold M at M which consists of all vectors perpendicular to the tangent space at æ.
Lemma D.3. Suppose that the manifold M has reach TM >> 0, then for any 2, E M, v E N
||l|| 2TM
Ku, 28 - y) -
| yll².
Proof. Let 2, M, N and define 2 & Med(M). Since the projection uniquely defined with llm(2)) = x, we have,
+ re/llull for some (0, TM) so that
= - yl 2 dist(z, M) =
On the other hand, we have,
2r
|| 21 ||x y112
x2
y, v).
|
Combining these and rearranging, we obtain the bound,
||0|||
(y x. S
|
y/12.
2r
By replacing e with - which is also in the normal space, we obtain the opposite direction, hence obtaining,
|||||
1(x y, u)2)l= $
| y112.
2r
Since this bound holds for all r (M, we can take - TM to obtain the bound in the statement.
L
With this, we can control the geodesic distance on the manifold by the standard Euclidean distance.
Lemma D.4. Suppose that the manifold M has reach TM > 0. x. y E M such that || yl s Tm/2 and let "/ be a geodesic (shortest path) between x. on M. Then, we have the bound,
/ ||0crelldt 21|x vll-
Therefore, if we are close enough to the manifold, its inherited metric behaves roughly like the Euclidean one. The proof of this lemma can be found in Lemma I11.21 of (Aamari, 2017).
D.3 Concentration under the manifold hypothesis
We now turn to results concerning probability measures supported submanifolds with bounded reach. The next lemma controls the mass of a small ball centred on the manifold, showing that the rates are similar to the affine case.
20

---

Lemma D.5. Suppose that the measure Pdata is supported on smooth compact submanifold M with reach TM >> and dimension d', Then, for any S. TTM/2V2, we have
Pdata (B,la)BB()))))))) 2 Curd"
Cp = inf Pp-
(x)
where Py denotes the density of pdata with respect to the volume measure on M.
For the proof of this lemma, we refer to the proof of Proposition 4.3 of (Aamari et al., 20199 or Lemma III.23 of (Aamari, .017).
We use this bound, to obtain a result concerning the concentration of the empirical measure on the manifold. To this end, we recall bound the covering number of the manifold. Given the covering number Now (M, r) is defined as the minimum number of Euclidean balls of radius required to cover the subset of R.d defined by the space M. The following lemma is from Proposition I11.11 I of Aamari, 2017]]
Lemma D.6. Consider the setting of Lemma D.5 and suppose that Cp. > then for any (M, we have,
(M, 28) S
'ueds.
We now prove bound the concentration of the empirical measure. Lemma D.7. Suppose that the measure Pdata is supported with reach TM and dimension d² and Cp. > O. Then, for any r E (0, TM], 8 (0, 1), we have
compact smooth submanifold M
pel"
Pdata (B.(()) 2 cp 2d ZEM
inf
2 1 1
whenever,
N > 16
16 (14Ad" log(4/r) (og(4))) + 1441og(1/c,)) v log(8-1) (1/22)-20) 20 cx²
(18)
Proof. According to Lemma D.6, there exists a set C M such that (Br/2(c) c E C} forms covering of M with |C| S c 4d" r-d" Thus, for any E M there exists E C such that E (c) and therefore B, B2/2(c) C (z)x). From this we deduce the following bound
inf Pdata (B, (x)) 2 inf Pdata (ddat(B,,/2(0 (Br))(c)). Z
cEC
Therefore, suffices to lower bound this object on the right-hand side.
Next, using a rudimentary bound from the empirical processes literature (for example, see Section 4.2 of (Wainwright, 2019) or Section 7.1 of (Handel, 2014)), we obtain the bound,
log C
sup Mdata (Br/2(c)) (Be/2(()) Pdata (Brr2 cEC
12
+ Sexp(-2Ne2).
Thus, choosing E = (a//r///"//// /4, we obtain that under 18 it follows that
IP
|iatta (C) Pdata (C)l 2 (T(r22) 12 < 8. CEc
sup
(19)
To conclude the proof, we use Lemma D.5 to obtain that,
inf (daaaa (rr/(() 2 inf Paata (Br,2(e)) sup |iaata (B,;2(()) Mddata (B,r(() cEM
csc
cEC
2 cr(r/2)d - ssup
Paaata (B,r((( Udata (B,/((x))]
#CM
Combining this with 19). arrive at the bound in the statement.
21

---

D.4 Weyl's tube formula
The M, and M, are related the notion of tubes that have been investigated the differential geometry literature (Gray, 2004) Weyl, 1939). We borrow result from Weyl. 1939 that computes the volume enclosing these sets. Let M == { E F dist(z, M r).
Proposition D.8 (Weyl's Tube Formula). Suppose Assumption 3.3 holds, then for all 0.
/21
xMM = Σ pmo
+2p
for some quantities (22)) 2 0.
The quantities l are related to the integrated mean curvature of M and further details about these quantities be found in (Gray, (004)) where the result is stated in Section 1.1. In this work, develop upper bounds in such a way that the final result does not depend on these quantities.
Note that using this result, we can obtain estimates for the integrals of functions depending X(M using the expression,
[ f(dist(x, M M
f f(r) x(Mg,)dr
dr
[2]
(d d' +20) / -1-29drd-11t2
-1
(20)
E Proofs for the main results
This section of the appendix provides the proofs for theorems 3.6 and 4.1 These theorems establish the core result that smoothing in the log-domain approximately geometry-adaptive meaning that smoothing with generic kernel k behaves similarly to smoothing with a manifold-adapted kernel g-M We begin by proving some lemmas that are involved in the proof of both of these theorems.
E.1 Controlling the log-density ratio
To establish the proximity between pt and pt in divergence, must control the ratio of their densities. In this section, we fix a permissible manifold M E My, and use KM, K Max TM and d'M as in Section 4.1 We also fix ar E Rd and let Y ~ kig Y = MT (Y), so that ~ J.M. Using the expression in we can express the density ratio by,
dp log dfp-v (x)
{
Y p. N
LSE (f
I18 p, N
CMM
E LSE
+ log
2
20
i=1
(21)
where we define the normalising constants,
c= c exp (flos, (v))k=<())) dr,
C-- cM=fesx
( Io0 (o)) (d)) dr.
(22)
We proceed similarly to the proof in the affine case (see Appendix C. decomposing the LSE function into normal and perpendicular components. We use the decomposition,
lyy 14,2% 2 Il I (1) 2(y IT (v), M (v) p.x.)) I| (v) p.2.11
along with Fact 3.2 to obtain that,
LSE {
Y
20 (E(N)
LSE {{
M (Y) - / ? + 2(Y
II IM), - medi)) })
IY I (1)
20
20
22

---

It follows from the definition of Y and r(x) that,
|| 1... (M) = 1110-118---- -IIM
=r(x)²
= =EIIY- - -I.
Therefore we may apply Fact 3.2 once more obtain,
{{
||
2
E
LSE
20
KEIN
{{
IIM (Y) Mea: 112 + +((Y- II (M) ple }
LSE
20?
11 - 11. (1)1
S
20?
E LSE ({
[T M,,;;1 + 2(Y - II (Y) me2z))
+
20
1 II (Y)
},)
20
LSE {{
Y
M.I; Ai
j VEENN [
20?
where we define the quantity expression,
;= == 2(Y
Ýł 1L (X) - p.T.). Therefore, we obtain the simple
dp dpp.o
C.M
log
(x) E'ALEES LSALSE + log
(23)
A LS)M(()) = LSE {{
11 + A
M,
LSE
20
20
REIN]
Having expressed the log-density ratio in terms of A L.SEM our next task is to bound this quantity. For the sake of intuition, we can consider the linear setting: In this case, Y - Y is normal to the manifold and IT (TM=(Y)
14 tangent to the manifold, so it would follow that A; 0 and thus LSEE( = 0. In the case where the manifold curved, it is no longer necessarily true that A; is and control A LSEM using the curvature of the manifold and the stability of the LSE function.
We proceed with simple lemma. Lemma E.1. Suppose that AM := dist(( M) < and TM > O. Then, for any x E RR. [N] we have,
A S pdlS (( df) ^d + AM).
- -M d; =
where we define the quantity.
18 -EIII-IIIII - 1 --I
11 (Y)Il
Proof. Using the definition of Y, we obtain that,
Y Y = (Y - (I.x((() - (8- - (I)+((Y)
Y - -Il
B. ||| ()) 1/2
= (Y - (TM:(T)
|/ -TT (Y)
(Y)
||Y -I (V) [
|Y (IM((Y)
23

---

With this, we can write A, in the following form:
Y LYM:(Y)
A; = 20
||Y (M) IIM (Y) MoRi))
(24)
To control A;, we use Lemma D.3 as well as the Cauchy-Schward incquality to obtain,
| = 2/c Y - MV) TI (8) - McTlm(ass)
(25)
| (Y)
+ 1|Y -IIu McIIae(as) Hozi )
Y -I)
s 2pelsl [IIm(\/e) - IM(x))1 A [ILL((H/m)) IIM(x))l + +AM).
(26)
2TM
completing the proof of the lemma.
Since ( is random variable, we next find ways of controlling it using KM and Kmax.M-
Lemma E.2. Let Ç be as in Lemmal E.7 and suppose that M, Kmax,M < co, then we have that,
|||</118]122 2KM,
K Kmax KM.
almost surely.
Proof. For the first bound, we use the L -tiiniee inequality to obtain,
B|(ci & E |IIY ITM (Y) - 1| I (x)) [|s]
+ || 11 (x)] 1-0-----IIIII II (1)111871111
E ||| MY)I || II (x) (z)) [|s] 1/2
E (IX II 1IM -|== (x)))) ||s)
S 2
Similarly, obtain - bounds via,
K || (M) || II -m)(())
+ ||. -x)l Eilly II (1)11
S Kmax.M + KM-
We now state the bound for A ALSEM that use for two main theorems. Lemma E.3. Suppose that Am dist((r:)a= < and TM > O, then for any x E IRd, it holds that
E A MSEM(EE))
8KM
12AM M
+ 2log ( (TA)) + (AM TM + KM + Dz)
TM
o?
+ 5k max, K112 + Dz) 2P(E
of
+ inf { log (Er PAdata (II (M/x))/)))))=-s + +(1+ 0020
KM+D; 6
1
TM
o7
where we define En {KM + Do + + HoTM/4) Dz -IIM-(x))l
24

---

Proof. For this bound, we begin with (16 of Lemma D.1 to obtain
[ALSEM(S)) < L.¹ EiEIN exp( I |r.3; 1| 220 - -
exp( (10 - H,2; r dr.
We decompose this further as,
A ASLEM(E)) Eien, exp(- - M,z, 2 1/0 dr
Eiel, exp( A8 p. 12/0011/202 r /2,00
EiErf exp(-11 M. 1u..;112200 11220 r 20 )
dr
Eien exp(- p. /2.::11/2007 -4,
1 max sel:
20
Di exp( -|11- Me -
EiEO exp(-1|X 1,22;11/220 -22 dr
=: A + B,
where we define the sets,
To = {i E [ [TT -
S E0}, 1 = e -aao(//))-MM()))))
for some random quantities E > 0.
The quantity A can be bounded directly using Lemma E.1
From this, we obtain,
Mr oe
S
of + AM)
2PM
To bound B, we proceed with the following upper bound,
EjErp expppepp(ll 1118 u. 12 + 1
Eier, exp(- - Me 20 |. 1
< | 11 max
Σ exp((I - ue:; 1/:::11//20 - 11 - -.., + +.1./220 + (A,l/o?), (27) iElo .jErf
where we have used the fact that r S exp(r). To further control we control the quantity inside of the exponential function by choosing Eo sufficiently small and s sufficiently large so that the quantity in the exponential becomes negative. This allows for control of by taking sufficiently small.
Me 112 For any E Io.j [N], we have that We start by controlling ||Ý p2x; |
o Memell 12 - 1 - -M,2,,11
||I ( -,00; 1| + +228 - 11 (Y), II.--()))) mexa)) III (Y) -
- 2(8 - (Y),
- -1,r))
(28)
The first term is bounded using the fact that [ (Y) u.D. I MeED + H.A.M The second term is controlled using the technique from the proof of Lemma E.1 to deduce the bound,
(Ý IM M=) - (63a))
(29)
Sm.ll (IM=(())
T (M/K))))) - (I:())) A [[T((//))) - - + AM
See + Dz)
2
: A + Am).
(30)
where in the second line, we use that
Y = Ellly II (1 11)
< E |||X - II (Y) || -I (x))
|| II (z)ll
< K + Da-
25

---

Similarly, the fourth term of 28 is bounded by,
(V M (Y), IT (Y) MeJr))
S Su(KM D2) ( -
A - -MM(s) Am,)
(31)
and finally, the third term of (28 is controlled using Young's ' inequality to obtain,
[I (X) - M,
[ (Y) II.- (H,2z) Il ||IIII (HcTTz) - Hett,
+ 2 (Y) M (1. II.: (Me - He
3
M (Y) - -Il-IIl--e (u, )11 - MT (Hert)) - 1e,r,11
4
IV
+1
M (Y) M 21 ))| -3, AM:
(32)
Thus, substituting (30)
31
and
(32 in (28) leads to the bound,
||8 11 2 118 1 2
S 12(600 AM) + 2K.(KM + D2) ( ( :0) AoO + Am)
3 1 (V) (M;T((3;))) (
4
+ (KM D.)) ((2tm [TM (cII(I)))19) A - -MM-:) +AM)
1 Me (2p + TM
K - TM
Dz) e + Me (5H.AM + +KM + 4Dx) AM
- (x;)(
+ +((KM + +D))( (ل | (Y) - (=llm(z;))1)) A (X) - (x)ll(
(33)
Continuing with bounding the contents of the exponential function in (27). we next control |A;|+
2/ using Lemma E. to obtain,
| | < 2p-l5| 2TM
=0+
(34)
Therefore, combining 33 and (34), we obtain the bound,
|| 1 - | Me 11 I + 2|A,|
S Mr 2pe + -1
TM (151 + KM + Da)) &0 Me 5p + G|Cl + 4K + +4D)) AM
[1IIIm (Y) (x,II
+2(KM + Dz + (((l)( 2HeTM
$ Me (2pe + PM (15) + KM + D2)
20 + Me (5p.AM + 6l1l + 4KM + +4D:) AM
I1 (X) (x,))
21g, (KM + D2 + IT (eT
where the indicator function contains the event E. = {KM + Dr + 2[ 2 HcTM/4].
Using this bound, we choose a value o; that guarantees that the contents of the exponential function
in 27 is negative. By solving the quadratic, it follows that to have ||Ý - Mez; 122 1|8 12
A 2 I Mek, for some K > 0, is sufficient to have II (Y) II (H.Z,) )l z ueEt
with
2²= = 6jccK + +bue (2 + 1 TM (15) + KM + D2))= + 6pe. (5p.AM + 6|[| + 4KM + 4Dx AM
+8 (KM + Da + 2
(35)
Substituting this into
we then obtain the following bound for B,
191
B <
E
s
exp(-pux<200)
Tn
26

---

We now return to bounding A with this choice of E1 to obtain that,
E[A[S] S
Me KM
(6ULR + 6/z (2/le
3 KM + TM
2yD2)et
o?
TMH?
TM
+ 6/4e 55tcMM 16KM + +4Dz)
+ (P(E max.M K + D2))) + 224m)
where we utilise the bounds in Lemma E.2 to control [c]. We then optimise x by choosing,
20 log
E
To s
TM KM
H4
which produces the bound,
KM
(1+ + 12g (B|149//4o1)
6ed
E|| LSE <
IML KM
+ (2
KM + 1 Dz
TM
o
TM
6AM o?
8P(E)1S)1/2
(5Am+ (5AM+16KE) 16KM 4Dz)
(5 (5K
max, K2/22 + D2,))
o?
2KMAM o?
where we have used that mr I 1 to simplify the expression.
To conclude the proof, we use the fact that 4o = N Natata (Beo (LlM(//))) and |491 S N. Then, optimising over Eg leads to the bound in the statement.
E.2 Manifold concentration under log-domain smoothing
The bound on A LSE developed in the previous section depends the distance the manifold, Dz. Since, to control the Rényi divergence, must integrate A LSEM with respect P.. we must develop some bounds on the concentration of this measure to the manifold. Due to the complexity of log-domain smoothing, this non-trivial and relies on Weyl's formula for the volume of tubular neighbourhoods. In the following lemma, we develop such concentration inequality.
Lemma E.4. Let o.e >> 0 such that ess we obtain the bound,
(B((x)) 2 8, then for all r2 2 2o?d.
IP 2ugr. M (dist(Z, 2 r|s) S exp (doo((8) + 5(K3 + A²2AM²
+ 44222
20
(r v 2202d
28 +22
+
40
Proof. We begin by expressing the probability in integral form,
P(dist P(dist(Z,M)) 2) = C J exp (f (V)) (W) )dz
Jdist(.M)2r
From the formulation of log P. in p. we readily obtain that
log (vv) < min; yyy 1122112 + log(N) + C.
20
min - Ho. (x;))
[11. - -Mxxl))+
+lg((N + C.
20
(dist(y, - He A2
d
2 log(2To?))
(36)
20²
27

---

Letting Y ~ k. we use the fact that Y E EM Mrx)) to obtain the following lower bound:
dist(Y, M
= Ry [dist(Y" ,M)9111
dist(a,,M) - By [(dist(Y' .M) - dist(x, M
2 (ist(z,,M)) - KM.
(37)
Thus, combining (36) and 37
we obtain the bound,
( log Jog
exp
( (v)):
dr
(dist(x, Mx) - U.AM KMd
s (2TO?) d/22
exp
dr
dlat(-MM22r
20²
dist(x,MM)2 40?
(KM + +MAMM)
s (2 (2no?))a11 4/2 dlst(-.M)2r
exp
+
dr. (38)
20²
Next, we lower bound Cm. For this, we use Lemma D.I with the parameters,
(dist(y, M
lyy pe
6
o²
which produces the expression,
LSE {{
(dist(y. + (11y-11 (mxa) (ist(y, + +nAmi))
) Ce
log (3)
20
(dist(y. M + McDm)
i== exp(re))et + log(N)) + C..
o?
Zi=1 exp(re:)
We control this further using similar technique to that used the proof Theorem 3.6 We define the sets To {i E [N] 2 - -2,²2/ /o²), 1 = fi [N] : 4 2 2
/o?) for any quantity
E >
With this, we obtain the following bound.
log (v) (dist(y, M³) + Am)
+ /'' Eien exp(re)) dr + / Everp exp(re;)es Σien, exp(re;)
Eie exp(re,) dr
o?
d
log (og(2To0))
2
(y. + mM)
|f L/ min exp(r(e Deddr verf.jeo
mine + ieh
[oo Jo
o²
d
2/4
(og((2T02)
(1, + AM)
2p?e² o²
[7G
ruge2
2
exp
dr
o²
4n Jg
0²
o²
d
log ))
2
This is further controlled using the fact that In 2 (i E [N] [M(2.) - $ and
44 < N and hence
I Poo
N
[LM())) n B. (IM(U/H))) S (Ba 58-1
where the last inequality holds almost surely. In combination with the bound E[dist (2, (2, 2 < dist(x, + KM, we obtain that,
20 log (g (dy) 2 (dist(e, + KM + p.AM)²
25
log (2moo)) (39)
o?
o?
2
28

---

With this, we lower bound CM by,
( (dy) dx
CM
exp
z (2To²)) d/2 / exp (- 2 dis
+2(K meAm)²
2p2=² o²
d/2
26- dr.
o?
(40)
Before combining the bounds in (38) and (40) we further simplify their expressions using Weyl's
formula (see Section D.4p. Combining the bound in (38) with the integral formula in (0), we obtain the bound.
Jastt.Merr exp
(108 (u)) ( )ax
[d" /2]
[KM + He 2
(d- d' 2p)) f
- L -1
<2TOO)) 4/2 exp
Σ p=U
2
where we use the shorthand d" = d'u- The integral on the right-hand side can be analysed by relating it to the measure of related spherically symmetric measure (e.g. equation (4) in Bobkov, 22033)
With this we relate the integral to the concentration a Gaussian random variable:
f g - exp(p=> /400 = T(k/2+1 +1)
[[oER exp(-1|><1* /ocell///oo /40 dr
km/k/2
T(k/2 + 1) k
P
P (lE)l 2 r).
Thus, we obtain the expression,
Sames--ae exp
(floz (ds) dx
ld' /2/
= d/2 exp
(KMMtnAm)) 20
Σ
NpPPE-N0002000000---p.. (1KE) 2 r),
Wpp : [((d - d= + 2p)/22+ 1) (
(d-dd 42p
(d - d" + 2pp) (M)
d d= 2p
By similar argument, we also obtain
[d'
Cm 2 (2TO?) exp (- 4(KM + 2=) + 4p?2²
-20-1
64 Σ
(d +2p) 12 12
20²
p=0
Dividing the two, obtain the bound,
zmaai (dist(Z, M²) r)
ld /2]
Wp W PeNNNN
(lE 2 r)
5(K2, + 1²AM)² 2ppE2
+ 28-1 o²
exp
64 64-(d- (d- (2p)ddd-))22ppp +2p)/2
20
5(K2 + AAM)
2/2E² + 26
< 8d exp
P0--N00.00/44-0--1 (IEE) 2 r).
20
o²
(O....... /2]}
max
We then bound this further using the concentration of the chi-squared distribution (see Example 2.28 of (Wainwright, 2019),, obtaining,
(r v 20
Pg (lEl) 2 S exp
402d
(r- 22
< exp
40?d
completing the proof of the bound.
29

---

E.3 Proof of Theorem 3.6
With the pointwise bound on EJ M from the previous subsections, we are now prepared to derive the Rényi divergence bound in Theorem 3.6
Proof of Theorem 3.6
To bound the Rényi divergence, we begin with the following expression which
follows from
log /
p (x)
9-1
Pa
l1bee)
Pe (dæ)
(x)
9 log f ( ([BAASSE(()))S)) " (dx) log(CM (C),
where the normalisation constants, C' and C.M, are as defined in (22 Furthermore, we obtain the following relationship between the normalisation constants:
C c= exp (/ log Pe (v) - / log Pe (b) + (og (dy) dz
=CM / ee (EXP(EAALEE(())))))
*M (dx).
Therefore, using Jensen's 's inequality, we deduce the bound.
B log
LXp()) (dx)
2 log
(41)
where, for the sake of brevity, we use the shorthand | = (q 1) V1 and Z ~ p. (dr).
We proceed by applying the bound on A LSEA developed in Lemma Lemma E.3 hold with AM- = 0 and hence, we have the bound,
E.3
The assumptions of
8K (1 [++2 2log ( T x
K1/2 + Dz)² (5K1/2 max
E| (|))55 s 5
16K +D22N474
o²
+ inl <0>0
log (Bljdaata (Beo (TM((//2))) )) + +(1+ K+D2)
o?
(42)
where we have used the fact that I<| S 2Kmax (see Lemma E.2 and therefore, P(E2|S: Z) 1/2 <
15Kmax pp.//- To control the infimum term, we utilise the bound on balls of Adta given in Lemma D.7 In this lemma, it is shown that whenever Eo T, with probability 1 - 6, we have,
sup faantal Beg (v)) yEM
S 2 /4,
(43)
once N is sufficiently large that the condition in 18 is satisfied with Eu- If we set E d' o? and require that of S (T/64)3/d" then once is sufficiently large so that (18) is satisfied, we obtain the bound,
log Ey (Y [data (Be (I) + (1 K+Dz =
inf so>0
inf sup { log (Bdata(Be,, + (1 K+Dx) eo 20E(00TM /64| 9EM
2 (1 K+ K+Dz d
(44)
Indeed, this would require N 2 (d' (d)=" If N does not satisfy this, then we instead set Eo to be the smallest such that (18) is satisfied. We have that such a quantity exists and satisfies Eo T/44 as soon we assume that,
2d" c Nmin (d' T, 5, cp). N 2 16 16((11 (144d log(256/r) + 144 log(1/c))) v log(6 log(8"1) ( (7/128)
(45)
30

---

With this, we arrive at quantity with (c,,N) 1/d" 2 (c,,N)- 1/d"
Thus, we obtain the bound,
+ +(1+ K+Dz E
inf #a30
log
By Adata (II (I)
S1+ (1 k+ ( 2 N) 1/4"
(46)
o7
We can then combine the bounds in depends only logarithmically on structural parameters and satisfies,
(4) and (46) to obtain that there exists a quantity C2 > 0 that
(1+ K+D2) ojt l
2 inf =o
log EElidata (Beo (TT((IT((Y/)))))
o?
S C2 1 + (1 Ps ) d' ((
(47)
where we have also used the fact that S Kmax S T/96.
Returning to bounding (41
we use
(42 and (47 to derive the upper bound,
log E exp
(
2|][s]
88 E
2 <
log
exp
88K
log (x)
1/2 x2/2+D2)² 0²
1 5Kmax +D22n.7/4 + C2
d" V (:: d'- }s{s)
+ C2
[ + Dz
16K
(
1 + 2
log
+ C2 C2d V
of
7
1
405K CDD2 d' (GH,N) # of
+
log
E
exp
20,3
1
109K (5K K Dz 1
max
log E
exp
15K D22p.7/4
(48)
5
o²
where in the second inequality, we Hölder's inequality.
We now bound the last two terms, starting with the second. We do this by utilising Lemma E. which we apply with = 7/64. As result of (43) and the assumed lower bound on N, the assumptions of Lemma E.4 are satisfied with 8 cp (T/ 64)d /4, and so, for any x² 2 4o
P(Dz 2 r|S) < exp (c
(r- /202d)
C= 1 5K2 + 2- 10 22
+ Sc (T/64 -d"
4o
20
Thus, for any c, > 0, we have the bound.
E exp (cDz)[S} = / P(Dz 2 log(r)/c)dr
I exp(cR) + / Texp(eR)
P(Dz 2 log(r)/c)dr
c (log(r)/c
/ 2202d)²
I exp(cc) +
exp
dr.
4c²0²
eep((R)
This is simplified using the change of variables.
I exp (c- (log(r)/c v 20
dr
402
/ exp (c
(u+ R / 20
+ clu + R) du.
40
31

---

We then choose R == /202d + 16c0 + 2256co02 + 320,C to further simplify the expression, obtaining,
C (log(r)/c v 2222d22
eexp(oR)
exp
dr
32c2
412
R
exp
+u
c
+
/c/20
du
320
160?
u²
S cexpley 20
exp
du
320
= (32x) explev 2o
where the last line follows from the Gaussian integral. With this, we obtain a bound on the MGF:
log El exp (cDz)
= log ( (xp((cR) + explee/
S cR + (32 1200
o2d <cvo
2 c
de + c(K r) + co.. (7//4))))
Substituting values for e into the bound, we obtain,
(2 d o2
log
E
exp
409K C.Dz dt V
S
208
3 v ood +
KL
dº V
(C +
oe (Vd cp 1/2
+ 2
/
d'
72
oo
oz
< Vožd +
K
d'
V
(c (2) d
1+ Te
+ 2// (
K
(o::")
of
(49)
Thus, soon as require that,
K² 2 2
72
72
ce S
= o - 1 d", T, K, cp),
(Va+ cp (T/64) d' 12)2
(dt)
we obtain the bound,
1 2033
log
cxp
403K CoD2 d' V
(c - +
s S ((ar
(d' + 1) No
(50)
We now bound the third term of 48 obtain,
We once again use the integral formula for the expectation to
1/2 (5K'/2
15K1/2K-1/
+ Dz)
E
exp
103K
L5Km DD2H.7/4
5
o?
109K (5K1/2 12/2 + Dz)² l5KmexD222cch
P
exp
o?
2 )d
00
o,r log(r)
P
(5K D2) 18kmae DD2207/44
5 5)0r dr
30
1033
109K ožr (MeT/4)2) /4)2) - 1))P(D2 2tT/ /4 5K 5Kmax 15)
1
exp
o²:r T
(5K1/2K1/2 + 22)²² 2 103K
log(r) dr. (51)
+
103K ofr (M47/4)²)
exp
32

---

We further bound the last term using K S K max along with a change of variables, to derive,
0²²
P
(5 + D2))² 2
log(r) / drr
100K
T. (NL7/1)2)
103K
200K
1103K 0Z²
P(5K + D2 2 w|S) exp
u1² uddu
HcT/4
2
103K
exp
C-
4o? (u- maxx
v 202dd
+
oār u² Juu
o²²
HoT/1
Setting c = 1
408K
>
1/2, we can simplify this expression by,
203K
00
exp
(c-
1
(u-
5Kmax
/202d))+
+
103K oāT u² udu
o²T
4
Mc7/4
203K
exp
C
(5Kmmx 2202d)e
40?
Pc7/4
1 c-
(5Kmax + v202d)2) udu
4
$ o². (40 /c) 12 exp (c- 40? [ET/4-- m5K + v 2202d)c-1
203K
//22//. //c,
where the final inequality follows from the concentration of the Gaussian random variable. Using the fact that K maax S T/996, c- < 2, and assuming that o² <
K K 2d
203
00
exp (c-
103K
Ao2 (u 5K
2202d)
u² Judu
aēT
PC7/4
oēT
40v 2m
(c- 1
(1L)
exp
80?
Similarly, we can bound the second term of 51. in total, obtaining,
(5K1/2 1/2 + Dz)²
100K
5,8
logE
exp
o?
5K1/2K D2 2//T/4
S
exp
108K off
(MeT/4)9 - -1) 14T 5K ma)
x-/2 + D22²² o². (5x
+
TP
log(r) 5 dr
109K
(H.T/1)2)
1033
exp
exp
C
+
108K प.r
(M,T /4/2
1
(3ueT/16)²)
+
K
exp
(c-
807 18)
1
40
exp
C
80? (MT)
1
Thus, by requiring that,
of < 20 (og((S) Sc, (r/64 log(t/K) =: o (d",,, K, Cy),
we obtain the upper bound,
1 log E
103K 15K K1/23 + Dz)²
) s s -
K
58
exp
o²
5K//2///2 DDz //4
(52)
Thus, by substituting (50) and have,
52
into
48
we obtain that, with a probability of at least
5, we
2
K max { do + 1, (c
log [IS
B
o²
33

---

completing the proof. We collect the required upper bounds on o which when combined with the fact that o S e, leads to the sufficient condition,
K
72
c max, (d, i
T, K,K, cy) (dd,,,,d d. K, c,)
max
(d, (d, T, K, K maxx Cu))
2d
(64)²d
(53)
E.4 Proof of Theorem 4.1
Proof. Let M M, and let TM, d'u: KM, Kmax.M AM and G, as defined Section 4.1 and assume that Am < and M S 7/96. Using the same argument that produced 42 obtain the bound,
D2 (1:t) - log Elexp( LSEM (2)
where Z ~ p. (dx). Using Lemma E.3
we obtain the pointwise bound,
E]JLSEM((2JSS 2]
8KM
(5
max.
KM²²
+ D2)²²
1 1 2log ( (xM)) +
Dx
TM
124M
(AM+TM TM KM + Dz)
o?
log [Rljiaata Beo (Be (TMm(Y/p.)) 21) + KM+D2
inf 0o>0
To bound the term with the infimum, we proceed similarly to the proof of Theorem 3.6 bounding it using Lemma D.7 Given that N is sufficiently large, we obtain from this lemma that,
Paata (Be, (x)) < 4u Eu -dii /4, yEM
sup
with probability 1 - 6. With this we can choose =i = d' "²² to obtain the upper bound,
2 inf <o>0
log [y Ey>kz [daata BBo (II (1 21) + (1 + KM+ |s,
E
TM
(C5(1+ 1+ (1+ KutD2 ) TM TM
CCss
for some quantity C3 > 0 which depends only logarithmically on structural parameters. With this, we obtain the bound,
2 log [E| [[(exp(
(EASPPEEAALA AASEEE (Z)),
KM
AAM + TM KM
AM
+ din) + log E
exp
40K TM
Cadii TM
+
120
Dz
S
TM
0²
o²
10K (5K
1/2
K
Dz)
+ log E
exp
max
LsK Dz 2
5
(54)
o?
We bound the second and third terms similarly to as in the proof of Theorem 3.6 However, the concentration of Dz differs slightly due to the additional error from AM > O. We apply Lemma E.4 with K/2 obtain that for any ? 2 d
P(Dz 2 r|S) S exp (c
(r- 220000)
107
where the constant is given by
C = log(8)d+
5lK+u
+ + 8c, ( 1//44)-b5s
20²
log(8)d=
6(K+n,
&c (7/64) -dar. d'm.
20
34

---

To bound the third term of (54), we can directly use the argument in the proof of Theorem 3.6 that produces [ST] Indeed, taking of sufficiently small, we obtain,
log E
cxp
1 TM
(5K4/2 +D2)²
K 1 1DZSucTn/4
o7
KM
TM
Similarly, we can borrow the argument that produces
49
to obtain
log E
exp
40KM TM
Cs
Aig of
Jd
TM
S cv o.d + c:o + ccM M + co. (vd oz (T
x
KM
+ ميد dim
KM + AM oo c (22
KM
TM
TM
RM
1 + - )dn
TM
FM
With this, we obtain that as soon o is sufficiently small, we obtain,
KM
A 0²
log E Therefore, returning to (54) we obtain the bound,
exp
M0 TM
Ca
+
R TM TM
+
dzu.
TM
2 log R B[exp([E[4 (2)5
KM
AM +dy
K
1
A M
1+ AM Am++TM +T + KM)
+
+
d'm
TM
oZ
TM
TM
o²
Kmid" +1 + KNAM o²
KMd" Aud
1+
TM
TM
Proofs of other results
This appendix contains the proofs for Corollary 3.7 and Proposition 3.8 These results elucidate some
of the generalisation properties of log-domain smoothing, specifically regarding how it concentrates mass near the manifold and distributes mass along it.
F.1 Proof of Corollary 3.7
To prove this corollary, we utilise Lemma E.4 As remarked the proof of Theorem 3.6 there exists Eo //44 such that with a probability of at least 1 8, we obtain the bound,
PY (dist(Y, r|S) S exp (c (r-v (r-/20?d)?) C == log(8)d+ +
5K*+4432 + 8c, eo"
1o?
20
for any 72 Z 4o²d, where Ep must chosen according to the size of and the condition (8) In particular, we have the bound,
P PYmpLe (dist(Y, 2 200dd + 4402C²|S) <
exp
To?
When N is large enough, choose the optimal value of = oe
xー cp
-
that,
0²
Au353 20
+ 8e, 8c,2= "5 S
rr 0e
cn
If N is not sufficiently large, we choose Eo to be the smallest value such that 18 satisfied, yielding
(c2 (1// I = S (c2N) and also,
4u2c2 20
+ 8c, 50 Sc.'E0 )
a²
35

---

With these choices of Co. we obtain the bound,
V402Ca & Vold+ + K max 1/6
オ 00
Cp
Next we transfer this concentration property to the measure ji by utilising the Rényi divergence bound in Theorem 3.6 By utilissng Lemma 21 of Chewi et al., 2022 we obtain the bound,
Py (istt(Y, 2 22 A0
V 4o? D2 ()) exp (- y²
80?
Finally we use Theorem 3.6 to bound the 2-Rényi divergence with probability least 1 8.
F.2 Proof of Proposition 3.8
Fix x. E M and let o be the shortest constant-velocity geodesic on M connecting the points x and Tz- Then the density ratio at x compared with xy can be expressed as
(x) pe (Zx
= exp (/ logpe(w) (dy) ((ogRR(()))(())))
f logge(u) kr. kr, /ogg)(())x..()))) (dy)
cxp (/ (10g C. + -ll-llln II (2)11?/2o?) kr (dy)
-/
(lo P. (v) - -C, + l1l II (2)Il kn, (dy)
where we have used that Ey [dist MM²] constant in z. Thus, by the fundamental theorem of calculus applied along the path t + 7e we obtain
(x))) exp (%r)) (1og C. + lly - IT- (w)1P/22220 (V.rk (v) bmnz; Onii) dy dt (x,))
To control this term further, we introduce the Fisher information matrix
I(x) - / (V log (v)) (V log ko(v)) hi (dy).
and define
F2
sup sup v. I(x) u.
REM v
|0|==
From for each fixed we have
/ log (ogge(()) Ce + 11l - TM (v)ll /200) (V) (2))) dcti) dy
= -/ (logpc()) Ce + 1l - IT (v)12/20²) (V log his (u) lampp deri) Ring (dy)
(vz
s (ʃ KVo loggk 2= 2 (dy)
1/2
x (ʃ (1og - -c, + lly IM (w)11//20²) k (dy)
1/2
< ((ont)TI(AA)) (Brit) 1/2
1P( (/ (1188p(00) C. + 11y II (2)l1 (2))11//00) kn kn (dy)
1/2
s FFll0nil ( ((11g C. + 11y II.- (2)):/20)) k kn,((d))
1/2
Next we bound the term
(y) == (o) C. + lly 11 (2)111)1
36

---

Recall that
log -.SE. (f 11y - u.a,)1//00? N )=m)
so that
|b+( LSE {{ lly - 1ey-He,,12/205 )=1) + 11y II (v)112/20²
LSE {{ (1ly - (1)1 + 2 11 (v), 11 (1) me2zz)
N j=1) + l1 -IM (1)) + +I|ldM (1) - Me 11 120=
S LSE ({ (llly 1IM (v) (3) p. |.0 [1) p.2, )=1)
S LSE ({ [[TM - Me /0?
N )=1)
Using the bound
|| (TM-())) s Kinx S 7 and the same argument as in the proof of Corollary
3.7
we obtain
2u?e² o?
| S
+ 2 Adata (I(()/()))))
Since N Nmin we have that with
Pdaaa (I(y/Pe) 2 CuE
In particular, this implies the uniform bound
4"41
8p
(v) S
02 cp
2
Combining this with the previous estimate yields the following bound on the density ratio:
(x)
(-
SpFF
गーं [10nill [(
exp
(x:)
02e77-
Finally, we control the length of the path gÝ. Let at be nearest neighbour of x among the sample points, so that
| x3ll = inf ||ær azel--
iE(M]
By the probability bound, we have inf ||2 xzll S r/2 with high probability. Together with Lemma D.4 this yields
L lanill S 211x - x;1l.
which completes the proof of the proposition.
37

---

G
Experimental details
In this section, provide detailed descriptions of the experimental settings used in the paper. Code to reproduce the experiments is available at htttps://github. com/ smoothing
1og__
G.1 2-dimensional circle example
The plots in Figure illustrate the trade-off that score- smoothing can provide for generalisation, to complement the theoretical results and discussion in Section 3.3 We consider an empirical dataset of 12 uniformly spaced points on the umit circle, and generate samples using the smoothed score function with an isotropic Gaussian kernel. We use a variance exploding diffusion model with T = 9 and geometric noise schedule, an Fuler-Mand Marayama discretisation with 100 steps, and 1000 samples in the smoothing evaluation. In Figure F we show how the resulting samples behave for different smoothing parameter a. Too little smoothing generates only training data, while too much smoothing causes generated samples to move towards the centre of the circle. There is good choice of smoothing that balances between the two, promoting generalisation along the manifold (a phenomenon noticed by Scarvelis et al. (2025)).
To further illustrate this trade-off, we also plot how the population negative log-likelihodd changes as the degree of smoothing increases, averaged over 1000 points on the true circular manifold. We recall that one can calculate the log- log-likelihood of a point by integrating the divergence of the probability-
ODE drift function along the probability flow ODE trajectory (Chen et al., 2018). In our case, the drift of the probability flow ODE is smoothed empirical score function. As we are considering isotropic Gaussian smoothing, the divergence and the kernel convolution can be interchanged, allowing us to compute the log-likelihood by integrating the smoothed divergence of the empirical score function. The resulting plot exhibits U-shape, clearly demonstrating the generalisation trade-off that arises from varying the smoothing level.
G.2 Comparing Gaussian smoothing and KDE
We here describe the experimental setup used in Section 5.1 which aims to illustrate how score- function smoothing can preserve the geometry of the data, in contrast density-level smoothing which quickly loses such structure. In order to consider a well-structured manifold, we follow Rombach et al. (2022) and use 32-dimensional VAE latent-space encoding of MNIST digits, and perform generation in the latent space. We consider M to be the set corresponding to the digit 4, which comprises a lower-dimensional structure in the latent space. This ground truth manifold is approximated using all samples of the digit 4, from which we use a subset of 100 samples as our training dataset. We consider smoothed-scone -coore diffusion model using an isotropic Gaussian kernel, and compare with kernel density estimation which corresponds to smoothing at the density level.
We train the VAE on 10,000 samples from the MNIST database. The VAE uses 16 initial feature channels, with scaling multiples of (1, 2, 2, 2) during downsampling, convolutional kernel size 3, dropout rate of 0, 1, and 4 groups for group normalisation. It maps into a 32-dimensional latent space. It trained for 10,000 training steps with a batch size of 64, using the Adam optimiser (Kingma and Ba, 2015) with learning rate 1e-3 and default parameters 0.9, 0.999.
VAE
Dataset construction We use the remaining 60,000 points not used to train the VAE. The latent representations of the individual classes comprise lower-dimensional structures in the space, which we verify in Figure I0 To show this, for a particular class we map all points to the latent space. We then pick one of the latent points and look its 50 closest neighbours 2j. We perform PCA decomposition on the vectors 2 - 2; to analyse their local structure. In Figure [0] we plot the cumulative explained variance, and observe that most of the variance is captured by fewer than the full 32 dimensions, suggesting that the latent representations for a particular class lie on a lower-dimensional manifold within the latent space.
We restrict to considering the digit 4, of which there are 5842 samples. We use these points to approximate the 'true' manifold M, and randomly choose 100 points for use as the empirical dataset.
38

---

Cum. explated tasta220
Com. explutedd VerthatcM
Com explaiedd vertemson
0.6
0.6
0.4
022
20 Num componeents
20 Num. components
Num. componenta
(a) Digit
(b) Digit
(c) Digit
Figure 10: Verifying that latent representations of digit class lie lower-dimensional structure in the latent space, by performing a PCA decomposition on the differences between nearby points.
Increasing Score Smoothing
Increasing Score Smoothing -
4
4
4
4
4
4
Data Point
Data Point
4
4
Increasing KDE Bandwidth
lncressing KDE Bandwidth
7
4
4
4
4
4
(a)
(b)
Figure 11: Additional generations, as in Figure
Experiment hyperparameters We variance-exploding diffusion model with T 9.0, geometric noise schedule, and 100 generation steps with an Suler-Maruyama discretisation scheme. We generate 500 samples to calculate the Le distances reported in Figure 7 For the isotropic Gaus- sian kernel, we used smoothing with standard deviations o E {0.4, 0.45, 0.5, 0.6, 0.7, 0.8, 0.9}, and we use 1000 smoothing samples at each generation step. For KDE, (0.07, 0.1, 0.13, 0.16, 0.19, 0.22, 0.25). which were chosen to induce comparable average distances to the dataset from the generated samples (plotted along the æ=-axis).
To obtain the samples plotted in Figure 6 we use Caussian-smoothiin with standard deviations (..1, 0.4, 0.5). These were selected to induce
E {0.3, 0.4, 0.5, 0. and KDE scales comparable lateral distances along the manifold, computed (d(ar, (dataa22 2 ((xx and averaged over 500 generated samples, which corresponds to inducing the same degree of 'novelty' relative to the training set. We display the same plot showing more samples in Figure [I] The quality of the reconstruction provides a proxy for how well the manifold structure is preserved. For the score-smoothed diffusion model, the amount of smoothing increases, the samples become novel images that are not present in the dataset yet nonetheless decode to resemble samples from the class of 4's, suggesting that they remain close to the underlying geometry. For KDE, we see that the quality of the reconstructed samples deteriorates more, indicating that in order to induce the same degree of novelty and difference from the training set, the KDE moves significantly further off-manifold, thereby failing to preserve the geometric structure of the data.
G.3 Synthetic image manifold
We now describe the experimental setup used in the synthetic image manifold experiment in Section 5.2 As working in pixel space is more challenging, we will focus on a simple one-dimensional image manifold. Now that we are operating in the pixel space, the training datapoints are further apart and thus there are many permissible manifolds that interpolate the data. Section 4 we considered how the type of smoothing can induce different structures in the generated samples, and illustrated this effect with low-dimensional experiments. We therefore aim to assess to what extent this intuition transfers to higher-dimennsonal settings by also considering smoothing kernels adapted to the manifold structure, and seeing whether this can influence the geometric structure of the sampling distribution.
39

---

e-
e=F
8-
6 6-13"
Figure 12: Visualisation of traversing the synthetic image manifold.
(a) Isotropic Gaussian smoothing, a 2.6
(b) Adapted smoothing. d 5.0
Figure 13: Samples generated using the Gaussian and manifold-adapted smoothing kernels. The manifold-adapted smoothing generates samples that are visually more 'on-manifold", in that the samples appear more spherically symmetric. See also Figure for quantitative measure of the spherical symmetry of the samples.
Dataset generation We construct the synthetic image dataset using function Φ [0, 2T) + R64x64 that maps angle 0 to an 'image'. The 'image" is constructed as the density of a 77 -vrrinne Gaussian distribution centred on the point on the circle with radius 0.5 corresponding to the angle (where overall image corresponds to [-1, 1] [-1, 1).. The density is scaled take values between 0 and The resulting manifold in image space therefore consists of a closed curve of 'Gaussian bumps' that move around the 0.5-circle 0 moves from 0 2x. We provide visualisation of traversing the manifold in Figure 12
We use 17 = 0.2, and use 16 equally spaced points along the curve as the
training dataset.
The manifold -ndifoldddddpteed smoothing kernel defined as follows. For point in the generation procedure, the projection (1) is computed. We define shifted manifold as . (z (z)). which is a translated copy of the manifold that passes through æ. Gaussian noise of standard deviation is added to then project onto this shifted manifold. All manifold projections are approximated by generating 1024 equally spaced points along the manifold and taking the closest one.
Experiment hyperparameters We a geometric noise schedule, and 100 generation steps with an Euler-Maruyama discretisa- tion scheme. For the isotropic Gaussian kernel, used smoothing with standard devia- tions 0 E (1.0, 2.0, 2.4, 2.6). (1.6, 2.4, to the training dataset in the generated samples (plotted along the z-axis).
variance-exploding diffusion model with T 9.0,
These values For the manifold- adaated smoothing, we used e chosen induce comparable average distances
For the isotropic Gaussian smoothing, we took 50,000 kernel samples at each generation step. For the manifold-adapte smoothing, we take 1000 smoothing samples each generation step (note that this can be much lower than for Gaussian smoothing, as the manifold along which we smooth is only 1 l-dimensional). We generate 100 samples, and average the closest distances to the manifold and to the empirical dataset. As with the projections, the closest distance to the manifold is calculated by generating 1000 points on the manifold and taking the minimum L2 distance.
Assessing the visual quality of samples The plot on the left of Figure 8 reports the average T distance from the manifold M, relative to the average L2 distance from the training dataset. is clear that the smootheed samples deviate comparatively further from the manifold than the adaptive-smoou
samples according to this distance. On the right of Figure [ we additionally report an alternative measure of how 'on-manifold" the generations related to the visual properties of the generations.
Note that samples from the true manifold consists of renormalised Gaussian density functions. Visually, being 'on-manifold" therefore corresponds to the generated images being spherically symmetric. In Figure 13 we display generated samples for the isotropic Gaussian and manifold- adapted smoothing mechanisms (for the largest smoothing values that were used in Figure K). and see that the banifold-adaptes smoothing generates samples appear visually more spherically symmetric.
40

---

Projected
Projected
Projected
(a) Gaussian kernel, 1.4
(b) Gaussian kemel,, 2.2
(c) Gaussian kernel, a= 2.6
Projected
Projected
Projected
(d) Adapted kernel, o 2.4
(e) Adapted kemeel, 3.8
(f) Adapted kernel, 5.0
Figure 14: Plots showing the projected 0 values for the generated samples, for different amounts of smoothing. As the smoothing increases, the generated samples spread along the manifold structure, and populate the space between the points in the training dataset (indicated by red vertical lines).
This property is however somewhat difficult to assess by eye, as any such changes can be subtle, so in Figure we also quantitatively measure the spherical symmetry assess this visual property.
In order to do so, we report the 'anisotropy" of the generated samples. Namely, we consider the renormalised generated samples as a probability density function [0, 1] [0. 1]. and record the anisotropy of the corresponding distribution (that is, we compute the covariance matrix Σ E R?, and report Amaz for eigenvalues Amazi Amén). Samples that are on-manifold" will have values close to 1.0. the computation, we set values less than 0.1 to zero, so that the noise in the generations does not impact the calculation.
Ameen
The results consistent with the pattern of Le distances reported in Figure - as the degree of smoothing increases and the generated samples deviate away from the training datapoints, the generations using the adapted smoothing have lower anisotropy and are therefore more 'round' than those obtained from Gaussian smoothing. Indeed, we know from Scarvelis et al. (2025) that Gaussian smoothing will generate barycentres of training points, which will skew the generated samples away from being perfectly round: it appears that the manifold-adapted smoothing somewhat mitigates this effect by shaping the geometry of the generated samples towards different interpolation.
G.3.1 Additional plots
The results in Figure 8 indicate that an adapted smoothing kernel can induce different structure in the generations compared to isotropic Gaussian smoothing-as as the degree of smoothing increases, the generated samples deviate away from the training data for both kernels, but remain comparatively closer to the manifold structure when using the adapted smoothing kernel. We here include some additional plots that further elucidate this observed effect.
Spread along the manifold While the Le distances reported in Figure show that the generations have deviated away from the training data, it is not necessarily clear how the scale of such deviations corresponds to the degree of spreading along the manifold structure. We therefore also examine the extent to which the generated samples become spread along the manifold as the smoothing increases, to confirm that the generations do indeed deviate sufficiently far from the training points to reasonably be considered 'novel'.
In Figure 14 we plot histograms showing the projected 0 values of the generations, in order to see how far the generated distribution has spread along the 1 synthetic manifold. We provide histograms for three different smoothing values, for both types of smoothing. For small smoothing levels, we recover only training points as expected, but the smoothing increases we see that the generations do indeed deviate far from the training datapoints relative to the manifold structure, and spread out to fill the gaps in the manifold between the points in the training dataset.
41

---

1.50
Gausscion Adupted
1:25
Dieranoe to
1:00
0.70
0.50
0.25
0.5
Destance data
Figure 15: Comparing L2 distance to data and M, for 2d synthetic image manifold.
2-dimensional synthetic image manifold We now consider a similar 2-dimensional image manifold example, so see whether similar effects hold in this setting too. Now, rather than considering Gaussian bump images with the centres located around a circle, we consider the manifold induced by placing the centres of the Gaussian bumps covering the [-0.5, 0.5] x [-0.5, 0.5) square. As before we take a small training dataset, which now consists of 116 points positioned on a lattice of equilateral triangles covering the square, each with side-length 0.1. We run sampling as in the 1d case, now using smoothing with standard deviations a E {0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0} for the Gaussian kernel, and {0.4, 0.8, 1.0, 1.2, 1.4, 1.6, 3.0} for the manifold-ade -ddaptedd kernel. In Figure [5 we plot distance to the manifold (approximated with 2879 samples, on a triangular grid with triangle sidc-length 0.02) versus distance to the training dataset, and observe a similar effect in the 1-dimensional case.
G.4 MNIST manifold
We now provide the details for the MNIST manifold experiment in Section 5.2
Dataset generation Similarly to the synthetic case, we construct a manifold by defining curve 6: [0, 1] + 3222X32 in pixel space, which interpolates between samples of the same digit from the MNIST dataset (LeCun et 2010). To obtain such an interpolation, we train convolutional VAE (Kingma and Welling, 2014). We then choose three datapoints from the same digit class (in this case, the digit 4), and draw triangle between their latent representations. We construct Φ(t) by decoding this triangle, which results in a closed loop in pixel space. We use the decodings of 10 equidistant points along the latent triangular interpolation to define the training dataset. We emphasise that the VAE is only used to construct a manifold structure in pixel-space, and the actual diffusion procedure takes place directly in the pixel-space without any interaction with the VAE.
Experiment hyperparameters We use a variance-exploding diffusion model with T 9.0, a geometric noise schedule, and 100 generation steps with an Suler-Maruyama discretisation scheme. We used smoothing with standard deviations a E (0.0, 0.3. 0.6, 0.8. 0.9, 1.0, 1.05, 1.1] } for isotropic Gaussian smoothing, and a E (0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 4.0, 7.0} for madifold-adapted smoothing (which again were chosen to induce similar distances from the data points in Figure D). For the isotropic Gaussian smoothing, we took 50,000 kernel samples at each generation step. For the manifold-adapted smoothing, we take 1000 smoothing samples at each generation step (this can be much lower than for Gaussian smoothing, as the manifold along which smooth is only 1- dimensional). As before, we generate 100 samples, and report the average closest distances to the manifold and to the empirical dataset. The closest distance to the manifold is calculated by generating 1000 points on the manifold, and taking the minimum L2 distance to these points.
FID calculation As we work with a I-dimensional cuve in pixel space, neighbouring points in the empirical dataset look very similar. It is therefore difficult to visually judge the quality of obtained samples from both smoothing mechanisms, so we use FID (Heusel et al., (017) as measure of similarity to the true manifold that also provides an indication of sample quality. We compute the FID values using the Inception-v3 model, and we stack across the channels and resize to match the network input. We calculate the FID scores of the generated samples relative to the 1000 random samples from the manifold. As we use small dataset sizes for the computations, these values should not be compared to values reported elsewhere; nevertheless, they are still representative of the visible changes in sample quality.
42

---

XBE
50
IDE Caussian Adaptted
50
7 98
Gussian - Adappied
40
Adagled
04
0.4
0.2
0.3
10
10
0.8
0.0
075
0.A
(a) distances, for (b) FID, for curve 2s class.
(c) L2 distances, for (d) FID, curve 7s curve in 7s class. class.
curve in 2s class.
Figure 16: Comparison of Gaussian and manifold-adapted smoothing kernels, for alternative curves ④ in the manifold of digits 2 and 7. Arrows indicate increasing smoothing.
Different manifolds We also ran the same experiment with manifolds for different digits, and observe similar behaviour. Results for the curves for digits 2 and 7 are plotted in Figure [6 The selected points were generally chosen to the first three examples of that digit in the dataset (other than when these datapoints induced poorly-decoded manifold, in which case used the first that made the constructed manifold of good quality).
Licenses:
MNIST digits classification dataset (LeCun et al., 2010 CC BY-SA 3.0 License
43

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we state that we examine the interplay of smoothing in the log-domain with manifold structures present in the dataset, and explain how this relates to diffusion models. We provide concrete theoretical results regarding this phenomena in Section 3.3 and provide empirical explorations in Section 5
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include a discussion of the limitations of our approach in Appendix A and aim to place our contributions within the scope of the wider literature.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We carefully stated all assumptions in the main paper in sections 3.2 3.3
and included proofs the appendix. In the main body, we aim to provide an accessible approach to our proof ideas in Section 3.1 by outlining the main ideas from the proof's in simplified setting.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include detailed explanation of our experimental setups in Appendix G including hyperparameters and architectures used.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide link code for reproducing experiments, in easily repro- ducible notebooks.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include all such details in Appendix G
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars not appropriate for the experimental settings that consider.
8. Experiments compute resources
44

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include details regarding computing resources in Appendix
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the Code of Ethics, and confirm that our work conforms to
it.
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
This work advances the theoretical understanding of diffusion models, though it does not propose specific methodology. While the insights may inform future algorithmic developments, we consider potential negative impacts speculative and therefore beyond the scope of this broader impact statement- such considerations apply to most research.
Justification:
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We are neither releasing new data or newly trained models.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
Justification: We report the licences for data used the Appendix.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
45

---

Justification: The paper does not involve crowdsoureing nor research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, non-standard components.
46


Reviews:
[{"review_id": "1dpVeAwjPH", "reviewer": "Reviewer_bbTq", "summary": "Diffusion models generate data from a target distribution by propagating points along paths given by a vector field.  That vector field is given by the gradient of log(p_t) where p_t is the distribution at time t, as it moves from the source (Gaussian) distribution to the target distribution of the data. A neural network is trained to approximate the time-dependent field \\grad \\log p_t(x) (known as the score).\n\nThe authors argue that since the NN is trained to approximates the log-density, any noise in the training process implicitly smoothes out the log values of the density. Since areas with a low density have log(density) go to minus infinity, this has the interesting effect of smoothing out the data only in the directions that are tangent to the data manifold, as long as the effective smoothing width is small w.r.t. the curvature of the manifold.  The authors explain their reasoning very well and prove this claim for a manifold with bounded reach. They also included an empirical study with both synthetic data and MNIST digits that supports the claim.", "questions": "1. The phenomenon discussed in this paper follows from the fact that as the density goes to zero the log-density goes to minus infinity. It seems that one could pick any other monotone function whose limit is minus infinity at zero and get the same behavior. This suggests that diffusion models based on variants of score matching should still be adaptive to the manifold geometry. Could your results be easily extended to such generalized settings? Will you consider running an experiment with a different monotone function?\n\n2. By Conv(M) do you mean the convex hull of the manifold? If so it should be stated.", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "The main observation in this paper is important and novel as far as I can tell (but I am not an expert on diffusion models). The paper is written very well and I enjoyed reading it. Overall, I think this is an excellent work.\n\nMajor comments:\n1. It seems like the work could be extended beyond the log transformation. See the Questions section of the review.\n2. The synthetic and MNIST examples are very nice but both are one-dimensional manifolds. Consider adding an example with 2 or 3 dimensions.\n\nA few minor comments:\n* It is well-known that the reach of a manifold controls the difficulty of estimating it is well-known, see for example Section 1.1 of [1].\n* The concepts of smoothing, density estimation and local regression along the tangent directions / geodesics of a data manifold have been explored extensively in the literature years before generative diffusion models gained popularity. e.g. [2], [3], [4].\n\nTypos:\nPage 3, line 71: \"spacial\" -> \"spatial\"\nPage 4, line 141: \"subspace subspace\"\nPage 5, line 183: \"the the\"\nPage 8, line 277: \"16 uniformly points\"\n\n\n---\n\n[1] Genovese, Christopher R., Marco Perone-Pacifico, Isabella Verdinelli, and Larry Wasserman. \"Minimax manifold estimation.\" The Journal of Machine Learning Research 13, no. 1 (2012): 1263-1291.\n\n\n[2] Cheng, Ming-Yen, and Hau-tieng Wu. \"Local linear regression on manifolds and its geometric interpretation.\" Journal of the American Statistical Association 108, no. 504 (2013): 1421-1434.\n\n[3] Moscovich, Amit, Ariel Jaffe, and Nadler Boaz. \"Minimax-optimal semi-supervised regression on unknown manifolds.\" In Artificial Intelligence and Statistics, pp. 933-942. PMLR, 2017.\n\n[4] Gao, Jia-Xing, Da-Quan Jiang, and Min-Ping Qian. \"Adaptive manifold density estimation.\" Journal of Statistical Computation and Simulation 92, no. 11 (2022): 2317-2331.", "was_revised": true, "final_justification": "I have read all the reviews and rebuttals. No major problems were uncovered so I leave my scores unchanged. I am looking forward to see the final version with the additional 2D example and congratulate the authors again for this high quality work.", "rebuttal": "We would like to express our sincere gratitude to the reviewer for this rating and for the insightful, encouraging comments. We are very happy that you found our paper interesting! The references you provided are indeed highly relevant, and we have integrated them into the revised manuscript to better contextualize our work for the reader.\n\n## Other transforms than the logarithm\nThis is an excellent observation, and we agree that the core intuition extends beyond the logarithmic transform. We find this prospect very interesting. Our focus on the logarithm was motivated directly by the formulation of contemporary diffusion models: a neural network is trained to approximate $\\nabla \\log \\hat{p}_t$. Since the network's approximation (and thus its implicit smoothing effect) occurs at the logarithmic level, we felt this was the natural choice for our analysis.\n\nOur current paper was primarily focused on trying to help explain part of the success of modern diffusion models, and as such we haven't found a good experiment for more general functions $f$ in explaining this. However, we would greatly appreciate any suggestions or ideas that you may have in this direction.\n\nWe do, however, believe your point is especially prescient when considering the design of new algorithms. The principle that any monotonic function where $f(0) = -\\infty$ could theoretically confer a similar geometry-adaptive benefit is a powerful one. Training a diffusion model with a different transform would mean that one would need to learn $\\nabla f(p(x))$ and then transform this to $\\nabla \\log p(x)$, which is then used as a drift (which is possible if $f$ is an invertible function). This is an interesting direction to think about.\n\n## Synthetic and MNIST experiments\nThank you for the great suggestion. We are working on a 2D experiment to add to our camera-ready version.\n\n## Minor\nYes, we meant the convex hull and we have clarified this in the paper. Thank you very much for catching the other typos as well; they have been corrected.\n\n\n## Summary\n\nWe want to warmly thank you for your rating and your great questions! Adding a 2D experiment will be a significant improvement, and your observation about the broader class of applicable transformations is a very pleasing and interesting insight that we will continue to reflect on."}, {"review_id": "NW9LYmVSrq", "reviewer": "Reviewer_RumM", "summary": "The paper offers an intuitive explanation for why diffusion models naturally apply smoothing in the log-density domain as a consequence of the score-matching objective.", "questions": "Regarding the identified weaknesses, I kindly request clarification on the following points to improve the clarity and presentation of the paper:\n\n1. Illustration of Reach: Please consider providing a more intuitive and accessible illustration for the definition of reach. This request is purely aimed at enhancing the presentation and reader understanding, rather than questioning the validity of the concept.\n\n2. Motivation for Assumptions 3.3, 3.4, and 3.5: It would be beneficial if the authors could elaborate on the intuition and motivation underlying 3.4 (lower bound on reach), and 3.5 (control over smoothing in normal directions). A clearer justification of their necessity would aid in understanding how these assumptions influence the theoretical results.\n\n3. Role and Origin of Manifold Assumption in Theorem 4.1: Please clarify the role played by the assumption on the manifold family $\\mathcal{M}$ in Theorem 4.1. In particular, how does this assumption arise, and what purpose does it serve in the context of the stated KL divergence bound?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths.**\n\n- The paper offers a clear and accessible intuition for the connection between smoothing in the log-density domain and the score-matching objective in diffusion models. Despite not working directly in this area, I was able to develop a reasonable understanding of the core ideas.\n\n- The theoretical results and definitions are presented with rigor, particularly in the supplementary material. The related work and limitations are thoroughly discussed.\n\n**Weaknesses.**\n\nWhile the paper is generally well-written and its sections are individually comprehensible, its theoretical nature calls for a more structured presentation of core concepts. For instance, the definition of manifold reach should be included in the main text; if that is not feasible, the authors should at least provide a clearer and more intuitive illustration. Additionally, the motivations behind key assumptions—such as Assumptions 3.4 and 3.5—should be elaborated upon in greater detail.", "was_revised": true, "final_justification": "The authors have addressed the concerns I raised in their response.", "rebuttal": "Thank you for your positive assessment and for providing such constructive feedback!\n\nWe appreciate the opportunity to clarify the presentation of our work, especially related to Assumptions 3.4 and 3.5. We have incorporated your feedback (and utilized the additional page) to improve the camera-ready version. We have expanded Section 4 with more intuition and enhanced the presentation in several places. As we cannot upload a revised manuscript at this stage, we will outline the changes below, focusing on the points you highlighted.\n\n## Assumption 3.4: Manifold Reach\nWe agree that the definition of reach would benefit from greater clarity. A minor error in the current definition may have contributed to some confusion. In the revised version, we have replaced this with a more intuitive, equivalent definition: *the reach is the maximal distance from the manifold within which every point has a unique projection onto that manifold*. This is the precise assumption used in our proofs and we believe this more intuitive definition may help clear some confusion.\n\nTo offer further intuition on the concept and its necessity: for a linear submanifold, any point in the ambient space has a unique closest point on the submanifold. This is not guaranteed for curved manifolds. Consider a circle, for which the center point is equidistant from every point on its circumference. The reach of this circular manifold is precisely its radius. Our assumption is designed to exclude ill-behaved manifolds with rapidly oscillating curvature (e.g., the graph of y=sin(1/x) near x=0), where the reach can become arbitrarily small locally, posing challenges for analysis. We are excluding this case with this assumption.\n\n\nIn the camera-ready version, we have moved the revised, more intuitive definition of reach into the main paper and included some of this intuition.\n\n## Assumption 3.5: Smoothing tangential to the manifold\nThis assumption bounds the extent to which the smoothing kernel operates in directions normal to the manifold. The linear case is again instructive. Imagine the manifold is a linear hyperplane. If a smoothing kernel $k_x$ only distributes mass parallel to this hyperplane, then for any point $Y \\sim k_x$, we have $\\text{dist}(Y,\\mathcal{M})=\\text{dist}(x,\\mathcal{M})$. In this scenario, the assumption is trivially satisfied. However, if the kernel also smooths along directions normal to the manifold, the value of $\\text{dist}(Y,\\mathcal{M})$ will vary. The assumption specifies that this variation is bounded.\n\nIn the nonlinear case, the ambient space can be locally decomposed into a set of \"parallel\" manifold slices, at least within the reach (per Assumption 3.4). The same intuition applies: we require a global bound on how much mass the kernel assigns to these parallel slices, which is an important condition for our proof technique.\n\n\n## Theorem 4.1\nWe agree with the reviewer that the definition of the manifold family would have benefited from further discussion. A smooth compact manifold $\\mathcal{M}$ is a member of this family $\\mathbb{M}_\\mu$ if the projection of the data distribution onto $\\mathcal{M}$ has a density that is uniformly lower-bounded away from zero:\n\n$$\n    \\inf_{x \\in \\mathcal{M}} p_{\\mu, \\mathcal{M}}(x) > 0.\n$$\nThe main intuition having no areas with $0$ probability is that these areas would never carry any samples, and could not be identified even if we have an infinite amount of samples. The requirement that the inf is strictly bounded away from $0$  is necessary to guarantee that even with a finite number of samples we can recover the whole manifold with a lower bounded probability. This type of assumption is typical in the literature on diffusion models and manifold learning (e.g., [1], [2]). \n\n[1] Aamari. 2017. “Convergence Rates for Geometric Inference.” Université Paris-Saclay.\n\n[2] Azangulov; Deligiannidis; Rousseau. 2024. “Convergence of Diffusion Models under the Manifold Hypothesis in High-Dimensions.”\n\n\n# Summary\nWe thank the reviewer again for their concrete feedback, which has helped us identify areas where our presentation could be improved. We have already integrated these changes into our camera-ready version. If any of the above explanations was particularly helpful, we would be very happy if you point that out so that we can elaborate on it in the camera ready version. If you feel like these explanations address your concerns and raise the papers appropriate rating beyond a borderline accept, we would be delighted if you could change your score accordingly."}, {"review_id": "ed5EdajhTv", "reviewer": "Reviewer_sUDw", "summary": "The paper concerns diffusion models and the relation between inductive biases and the fact that the score is related to the log density of the data. Specifically, the authors consider a specific kind of inductive bias in the form of a smoothing kernel. Assuming spatial stationarity, the convolution with the kernel commutes with the gradient in the score, and the smoothing can be seen to happen on the log density, in contrast to more standard kernel density estimation that relates to the density itself. The authors use this to investigate how the smoothing happens if the data is supported on manifolds of different complexities, and how small sample regimes affect the results. The paper is mostly theoretical, but it contains some experimental validation of the results.", "questions": "- \"Consequently, if a smoothing kernel extends into regions off the manifold, the resulting smoothed log-density in those off-manifold regions will also equal −∞\" ... \"it will have exceedingly small values in regions distant from the data manifold\" Could this intuitive argument also go the other way around: at a data manifold with positive codimension, the smoothing will extend to -\\infty areas close by, and thus give very small values also at the data manifold itself. Essentially setting the probability at the data manifold to zero? (with the same argument) I think what saves it is that t>0 immediately smoothes the manifold so that the probability is positive in all dimensions and this case doesn't appear, but still one should be careful with the intuition in such cases\n- reach in assumption 3.4: should the distance be signed? otherwise, for opposing points of a hyperplane, the dist would be equal but the points not. Isn't the reach usually defined as for \\tau close points, the projection to M is unique?\n- Theorem 3.6: \"with high probability\". What is the precise statement here?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- the theoretical exploration of the relation between inductive biases in this specific form is relevant, theoretically interesting, and is useful in understanding the behaviour of diffusion models\n- the paper is well-written and clearly presented\n- the theoretical results are non-trivial, particularly the extension of the result from linear spaces to manifolds\n- the paper contains some experimental backing of the theoretical investigation \n\nWeaknesses:\n- the results rely on some assumptions, and it is not always clear how realistic they are in practical settings. E.g. eq (5) is an assumption, and it is exactly clear to me how realistic this actually is\n- more generally, one can always be in doubt how relevant theoretical investigations like this are to actual data and training settings where the assumptions may not be satisfied. In any case, I find the investigation interesting in themselves as guides to understanding the relation between log domain smoothing and data supported on low-dimensional manifolds", "was_revised": true, "final_justification": "The authors responded adequately to the questions and concerns I raised in my review. My initial rating was positive, and the response backs this evaluation. I therefore keep my score of the paper.", "rebuttal": "Thank you for the positive feedback on our paper. We are very happy that you appreciate our theoretical contributions. We will address your points and questions below.\n\n## Weaknesses\nWe agree that Equation (5) is an assumption. However, we want to note that it is essentially the assumption that the learned score is some smoothing of the empirical score, rather than a restriction on the type of smoothing that can occur. Since we allow the kernels to be position-dependent (i.e., the kernels $k_x$ can be completely unrelated for different $x$), this assumption posits that neural network takes the target function and at each point $x$, smooth the function arbitrarily and completely independent of other points. This allows for a very large class of smoothing operators (or even non-smoothing operators if $k_x$ is a dirac) to be encompassed in this setting. We then study which kind of assumptions we have to make for $k_x$ to recover a specific target manifold (in particular, Assumption 3.5). It turns out that smoothing in the log-domain is already a very powerful tool to recover manifolds; moreover, by smoothing along tangential directions of a target manifold $\\mathcal{M}$, one can induce a biases towards that particular interpolating manifold.\n\nRegarding the practicality of our work, we see its primary contribution as deepening the theoretical understanding of how diffusion models interact with manifold-structured data. Nevertheless, we believe there are two key practical takeaways. First, smoothing empirical densities in the log-domain is more compatible with manifold structures than traditional density-domain smoothing (as illustrated in Figure 2), a principle that could inform the design of novel algorithms. Second, we argue one can induce a geometric bias by smoothing along the tangential directions of a desired manifold. While engineering neural networks with such specific inductive biases is a complex challenge, we hope our work provides a guiding intuition for this direction, which we plan to explore in future work.\n\n\n## Questions\n- You raise an excellent point regarding the intuition of log-domain smoothing. In the theoretical limit where the log-density is $-\\infty$ off-manifold, any smoothing kernel with support extending beyond the manifold would indeed yield a value of $-\\infty$. However, for any positive time $t>0$, the noising process ensures the density $p_t(x)$ is positive everywhere. Consequently, the log-density is finite across the space. While smoothing with points far from the manifold will lower the log-density value on the manifold, the same effect occurs for all points. Since the final density will be normalized to mass 1, what matters isn't the absolute/specific values taken by the unnormalized smoothed log-density, but rather their relative differences. Our intuitive argument is that this procedure \"punishes\" points further away from the manifold more than points on the manifold, thus preserving the structure. We hope this clarifies the intuition. We have made sure to clarify this point in the paper and to emphasize that it is the relative effects that matter. We are open to making more changes if you think it would improve the presentation.\n- You are correct about the definition of reach. We made a small error when finalizing the paper; the definition in the submitted version is incorrect. The proper definition is indeed equivalent to requiring a unique projection onto the manifold. This definition is more intuitive and we have changed it.\n- When we say \"with high probability\", we mean that for any $\\delta>0$, there exists an $N_\\delta$ such that for $N>N_\\delta$ the statement holds with probability at least $1-\\delta$. We have revised the phrasing of the theorem to clarify this and to provide precise dependencies of the relevant quantities on each variable; please let us know if you would like more information regarding this point.\n\n## Summary\nWe want to thank the reviewer again for the great questions and we hope that our answers helped to address your concerns. We would be grateful for any final suggestions on which of these clarifications you feel are most helpful to incorporate into the final manuscript. If you think that our changes justify a change in your score, we would of course be delighted."}, {"review_id": "X1zN7nQ17T", "reviewer": "Reviewer_bgfQ", "summary": "This paper seeks to understand the success of diffusion models by relating them to the manifold hypothesis. After recalling the basics of diffusion models, they argue that such models implicitly perform smoothing of the empirical density.\n\nThey provide theoretical results showing that the smoothed density is well approximated by its manifold adapted counterpart and that this counterpart keeps mass near the data manifold. They complement their theoretical analysis with experiments on synthetic data (where they can directly control the curvature) and on simple images from MNIST", "questions": "In light of the supposed connection between diffusion and the manifold hypothesis, I was wondering if diffusion models be applied to biomedical data sets which satisfy the manifold hypothesis? If so, this could be a good way of generating synthetic data for testing ML algorithms while preserving privacy\n\nAlso, is there a way to related diffusion models to manifold learning algorithms such as Diffusion Maps of Laplacian Eigenmaps? E.g., can one diffusion in the coordinates induced by these algorithms?\n\nIn settings where the manifold hypothesis is justified, is it possible to replace B_t in the definition of Eqn's 1 and 2 with a manifold-valued Brownian motion.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe linear warm up is very helpful for gaining intuition\n\nWhile I did not have time to check in detail, the proofs appear to be carefully written and well organized (unlike many proofs I see in conference submissions with obvious mistakes that jump of the page)\n\nConnection between manifold hypothesis and diffusion is interesting as is the casting of implicit smoothing of the density in the log domain to preserve the manifold structure\n\nTheoretical understanding of diffusion is limited \n\n\n\nWeaknesses\n\nThe experimental set up is limited to images, in particular synthetic images and MNIST. It would be improved by considering experiments on non-image related manifolds or even on more complex image data sets\n\nFurthermore, the connection between the success of diffusion models and the manifold hypothesis would be strengthened if one were to establish that diffusion models perform poorly on data that does NOT satisfy the manifold hypothesis\n\nIt is not entirely clear how the projection onto \\Pi_{\\mathcal{M}_r(x)} would be implemented if the manifold is unknown\n\nMinor: Should explicitly state the B_t is a Brownian motion (Weiner process) in equations 1 and 2", "was_revised": true, "final_justification": "I stand by my earlier review, for the reasons outlined there. This is a good paper.", "rebuttal": "Thank you for the positive review. We appreciate that you found our theoretical contributions to be correct and that you agree we are tackling an important problem. Your main concerns appear to be centered on the experimental scope and the practical implications of our approach to the manifold hypothesis. We hope the following clarifications are helpful.\n\n## Manifold hypothesis\nThe classical manifold hypothesis assumes that there is a single, ground-truth manifold that one aims to recover (for example, a \"true\" manifold of images or proteins). This is how most of the theoretical analysis of diffusion models up to this point treat the problem, and also the way in which we analyse the setting up to Section 4. Afterwards, we offer an alternative perspective: we argue that in high dimensions there are many plausible interpolating manifolds given sparse observations. The choice of which manifold is learned is therefore determined by the inductive biases of the training procedure.\n\nPractitioners already leverage this kind of intuition, though often in an ad-hoc manner, by experimenting with different model architectures that encode certain invariances. Our work takes a step toward understanding how to design such a bias purposefully if the desired manifold structure were known. We demonstrate this on toy examples (Figures 1 and 3), where different biases lead to different interpolating manifolds for the same set of data points. For a more complex and realistic case like MNIST, we use a VAE to construct a manifold, and guide the smoothing to occur along the manifold structure to examine its induced effects.\n\n\n## Weaknesses\n- We think that designing the correct inductive biases for specific, complex datasets is a challenging task. We hope our work has helped make a step in this direction. Our primary goal with this work is to deepen the theoretical understanding of what an ideal bias would entail if one already knew the target manifold structure. We believe this is a necessary step before such principles can be used to design application-specific biases. For instance, in scientific machine learning, where data may be solutions to a given PDE, we are hopeful that this framework can be used to design more effective biases/models in follow-up work.\n- To clarify, we are not claiming that diffusion models work only on manifold-structured data. Instead, we argue that they remain effective even in this challenging setting, and we posit that this effectiveness is linked to the implicit log-domain smoothing.\n- You are correct that the projection operator is not implementable without prior knowledge of the target manifold. In our experiments, we use the knowledge of the true manifold (constructed explicitly in Section 5.1, and via a VAE in Section 5.2) to construct the smoothing mechanism, while the function that is being smoothed is the empirical score and thus unaware of the true manifold structure. These experiments are designed to illustrate that the *type* of smoothing can provide a mechanism that impacts the resulting interpolating manifold, as suggested by the results in Section 4. We do not propose this as a practical algorithm or argue that this is necessarily the right way to actually smooth your neural network. We discuss this in lines 316-320, noting that if manifold-adapted smoothing occurs in practice, it will have arose implicitly from the models inductive biases, via architectural choices such as convolutions and attention [1].\n\n[1] Kamb and Ganguli, 2024, \"An analytic theory of creativity in convolutional diffusion models\"\n\n## Questions\n- Biomedical datasets can vary greatly, but we do believe that many of them do satisfy some kind of manifold hypothesis, and that in these cases the manifold constraints might even be a bit more explicit than for images, making it possible to think about the right biases. We think an analysis in terms of privacy metrics is an important direction. However, it requiress a considerable amount of theoretical work which we plan to do in future publications. We are very interested in the reviewer's ideas for follow-up works on privacy in case the reviewer wants to elaborate.\n- If one could learn a manifold structure via a global coordinate chart $\\mathbb{R}^d \\to \\mathbb{R}^D$, with $\\mathbb{R}^D$ the ambient space, then it would be possible to run a diffusion model in these coordinates (see [1]). However, standard formulations of Diffusion Maps (and to the best of our knowledge Laplacian Eigenmaps as well) do not yield a global feature map $\\mathbb{R}^d \\to \\mathbb{R}^D$. While the inverse map $\\mathbb{R}^D \\to \\mathbb{R}^d$ (from the embedding to latent space) can often be extended, extending the forward direction chart $\\mathbb{R}^d \\to \\mathbb{R}^D$ is typically non-canonical and non-trivial. Still, the reviewer may be interested in [2], which uses diffusion maps in a generative setting with an algorithm similar to diffusion models.\n- Yes indeed, in cases where the manifold is known, one can also run the whole algorithm directly on that manifold which has been explored in prior work, see for example [1]. However, our theoretical model is focused on analyzing why standard diffusion models work so well even when this structure is not explicitly known or enforced. Our work aims to make the connection between an inductive bias (e.g., tangential smoothing) and the recovered manifold more explicit.\n\n[1a] De Bortoli; Mathieu; Hutchinson; Thornton; Teh; Doucet. 2022. “Riemannian Score‑Based Generative Modelling.” \n\n[1b] Huang; Aghajohari; Bose; Panangaden; Courville. 2022. “Riemannian Diffusion Models.” Universität de Montréal & University of Toronto & McGill University\n\n[2] Gottwald; Li; Marzouk; Reich. 2024. “Stable generative modeling using diffusion maps.” \n\n## Summary\nWe thank the reviewer for their questions, as clarifying these points has greatly helped improve the quality of the paper. We have modified the camera-ready version in several places accordingly. We would be more than happy to discuss any questions further or to hear if you feel that any of the points we mentioned would in your opinion help improve our camera-ready version. \n\nIn case you think that the above modifications have improved our rating beyond a borderline accept, we would be most appreciative if you would consider raising your score."}]

Author rebuttal:

Meta-review: This work provides evidence for the conjecture that the success of diffusion models is related to the manifold hypothesis. The authors' theoretical and empirical results show that smoothing the score function produces smoothing tangential to the data manifold. Diffusion models are not well understood theoretically, and this paper provides a nice contribution in this direction.

All of the reviewers' concerns were addressed to their satisfaction and they all recommend the paper be accepted.
Decision: Accept (poster)