When Scores Learn Geometry: Rate Separations under the Manifold Hypothesis
Authors: Xiang Li
Zebang Shen
Ya-Ping Hsieh
Niao He
Venue: ICLR 2026 Poster (2026)
Keywords: Score learning
Diffusion models
Manifold hypothesis
Uniform sampling on manifolds
Primary area: generative models

Abstract: Score-based methods, such as diffusion models and Bayesian inverse problems, are often interpreted as learning the data distribution in the low-noise limit ($\sigma \to 0$). In this work, we propose an alternative perspective: their success arises from implicitly learning the data manifold rather than the full distribution. Our claim is based on a novel analysis of scores in the small-$\sigma$ regime that reveals a sharp separation of scales: information about the data manifold is $\Theta(\sigma^{-2})$ stronger than information about the distribution. We argue that this insight suggests a paradigm shift from the less practical goal of distributional learning to the more attainable task of geometric learning, which provably tolerates $O(\sigma^{-2})$ larger errors in score approximation. We illustrate this perspective through three consequences: i) in diffusion models, concentration on data support can be achieved with a score error of $o(\sigma^{-2})$, whereas recovering the specific data distribution requires a much stricter $o(1)$ error; ii) more surprisingly, learning the uniform distribution on the manifold—an especially structured and useful object—is also $O(\sigma^{-2})$ easier; and iii) in Bayesian inverse problems, the maximum entropy prior is $O(\sigma^{-2})$ more robust to score errors than generic priors. Finally, we validate our theoretical findings with preliminary experiments on large-scale models, including Stable Diffusion.

Published as a conference paper at ICLR 2026
WHEN SCORES LEARN GEOMETRY: RATE SEPARATIONS UNDER THE MANIFOLD HYPOTHESIS
Xiang Li, Zebang Shen, Ya-Ping Hsieh & Niao He Department of Computer Science ETH Zurich
8092 Zurich, Switzerland {xiang. 1i, zebang.. shen, yaping. hsieh, nia.. hssieh.miao.heeeiiff.tttzz
ABSTRACT
Score-based methods, such as diffusion models and Bayesian inverse problems, are often interpreted as learning the data distribution the low-noise limit (a - 0). In this work, we propose an alternative perspective: their success arises from implicitly learning the data manifold rather than the full distribu- tion. Our claim is based
novel analysis of scores in the small-o regime that reveals a sharp separation of scales: information about the data manifold is (2=2) stronger than information about the distribution. We argue that this insight suggests a paradigm shift from the less practical goal of distributional learning the more attainable task of geometric learning, which provably toler- ates ()) larger errors in score approximation. We illustrate this perspective through three consequences:
diffusion models, concentration on data support can be achieved with a score eπтоr of o(o 3). whereas recovering the specific data distribution requires much stricter o(1) error; ii) more surprisingly, learning the uniform distribution on the manifold an especially structured and useful ob- ject-is also O( easier; and iii) in Bayesian inverse problems, the maximum entropy prior is 0²) more robust to score errors than generic priors. Finally, we validate our theoretical findings with preliminary experiments on large-scale models, including Stable Diffusion.
INTRODUCTION
Score learning has emerged as a particularly powerful paradigm for modeling complex probabilistic distributions, driving breakthroughs in generative modeling, Bayesian inverse problems, and sam- pling (Laumont et al. 20022] Saremi et al. 2023) Ho et al.) 2020 Song & Ermon 2019; [Song et al.)
2021
Let Pdata be data measure over 2at and define Gaussian-smoothed measure as
Ma law (X + oZ) or Pa law (VI- X + oZ). where X Pdatas Z NO, (1)
Let Po be its density function w.r.t. the Lebesgue measure over Rd. A key step in the score learning framework approximate the score function V log Pa and sample from the target distribution
Mo. possibly across a spectrum of different a values (Vincent 2011 H Hyvärinen & Dayan 2005).
A central challenge in this framework understanding the low-temperature limit, i.e., learning the score of Mor as σ + O, which encodes the most detailed information about the data distribution. Em- pirically, this regime is also the most valuable: -ow-tempertture scores underpin many probabilistic
learning frameworks (Laumont et al. 2022; Saremi et al.) 2023 Janati et al.) 2024 Kadkhodaie Simoncelli] (020), including the influential diffusion model framework (Ho et al. 2020) Song
| 2020; Karras et al.) (022). whose noise schedules specifically designed to emphasize low temperatures and often require substantial post-training engineering to stabilize the learned scores.
Despite its importance, accurately estimating the score function in the low-o regime remains noto- riously difficult (Song et al. 2021]] Karras et al.) 2022 Arts et al. 2023 Raja et al. 2025 Stanczuk et al. 2024). Motivated by this challenge, this paper establishes the widely adopted manifold hypothesis, which posits that the data distribution Mata is supported low-dimensional manifold, M embedded in high-dimensional ambient space.
qualitative phenomenon under

---

Published as a conference paper at ICLR 2026
learning larget density
recover the support
learning the gcometry
uniform sampling M
Baycaian inverse problems with maximum entropy prior
Rate Separation
o(1)
af>
leaming difficilty
afr
leamnng diflislly
(a) Existing Paradigm
(b) New Paradigm
Figure 1: Toy examples illustrating recovered distributions under different regimes, with the mani- fold represented as a one-dimensional circle embedded in R??
Our key finding, formalized in Theorem 3.1 is that in the small-o regime of score learning there sharp separation of scales: geometric information about the data manifold appears at order O(r=)) whereas density information of Hdata emerges only at order (1). As shown in Section this implies that distribution learning of [Aa (e.g., diffusion models) necessarily first recovers the support of the data distribution before any information about the density can be learned. This per- spective naturally separates score learning into two fundamental tasks: geometric learning, which targets the manifold geometry, and density learning, which targets the specific data density on that manifold, with the latter being order of magnitude more difficult. It also suggests that the practical success of based models (e.g.. diffusion models) stems from constraining generated samples to the manifold, thereby producing realistic data even without fully recovering the underlying distri- bution. According to our analysis, achieve this, score error even large o(a-2) ?) is sufficient.
However, our analysis reveals a critical limitation: unless the score learned to a stringent accuracy that is beyond (1), attempts recover the data distribution may yield arbitrary densities supported on the manifold. This amounts to only a partial recovery of geometry and can compromise the reli- ability of downstream tasks and analyses. Such an observation motivates to pursue full geometric tearning-thatin that is, learning to sample uniformly with respect to the manifold's 's intrinsic (Riemannian) volume measure, is well-known that uniform samples can best support tasks that depend solely and heat-kernel approximation, geodesic and
on the underlying geometry (e.g.. Laplace-Beltrnn diffusion distances) (Coifman & Lafon 2006 [Belkin & Niyogi| [2008] Jostt, 2005). addition, they also facilitate principled manifold exploration, yielding diverse samples while mitigating potential biases present Adata (De Santi et al. 2025)).
In this light, central contribution of this work is to show that simple, one-line modification to standard algorithms can provably generate the uniform distribution on the manifold-requiring only o(a 2 score accuracy, in stark contrast to the o(1) accuracy needed for exact distributional recovery. In summary, we advocate paradigm shift: from the demanding goal of distributional learning toward the more practical and robust objective of geometric learning.
We substantiate the aforementioned rate separation phenomenon by three key results (see also Fig- ure
Theorem 4.1 shows that, in existing frameworks, the score accuracy required to force concentra- tion on the data manifold is 02)-2) weaker than that needed to exactly recover Adata. Neverthe- less, the resulting distribution can still be arbitrary.
contrast, Theorems 5.T too to55 5.2 establish a new paradigm centered on extracting precise geometric information of the data manifold by producing the uniform distribution. Notably, we show that simple one-line modification of a widely used sampling algorithm suffices to obtain samples from the uniform distribution under the relaxed score error condition o(a (2-)), substantially weaker than the o() required for full recovery of jdata-
In the context of Bayesian inverse problems (Venkatakrishnan al. 2013). Theorem 6.1 estab- lishes a rate separation in posterior sampling depending on the choice prior. When the prior is uniform, posterior sampling requires only o(o (2) score accuracy. By contrast, when the prior is taken to be the commonly used data distribution Mdaaas substantially stronger accuracy guarantees are needed to ensure provable success in existing works (Laumont et al. 2022 Pesme al. 2025)
2

---

Published as a conference paper at ICLR 2026
We validate these theoretical results with preliminary experiments on both synthetie and real-world data, including an application of our algorithm to a large-scale image generation model (Stable Diffusion 1.5 (Rombach et al. 2022)).
1.1 RELATED WORK
Diffusion models for distribution learning. Prior theory shows that diffussion/score-basee sam- plers converge to the target law when the learned score is accurate, with error bounds that scale
directly with the score mismatch (De Bortoli 2022 Chen al. [023] Lee al. (023); related
works study other factors such as dimension dependence (Azangulov et al.) 2024 Tang & Yang
2024). However, these results do not separate geometry from density in the score error but instead consider them together, therefore they do not imply any scale separation.
Diffusion models detect data manifold. There is growing body of work probing whether dif- fusion models learn the full data distribution or primarily the underlying low-dimensional mani- fold. A number of studies suggest that these models often capture the data support while missing fine-grained distributional structure. However, these results are obtained under restricted settings: Stanczuk et al.] (2024) focuses on estimating the intrinsic dimension of the data manifold; Ventura it al. (2024) analyzes only linear manifolds (linear subspaces); and Pavlova & Weil (2025) provides primarily empirical evidence. Pidstrigach (2022) establishes sufficient regularity conditions under which high-ace accuracy scores concentrate mass near the manifold, but does not address how approx- imation errors scale with and therefore does not reveal separation of scales. By contrast, analysis quantifies how inaccuracies in the learned score propagate differently to geometry versus distribution learning, exhibiting distinct error rates that lead to sharp scale separation in the small-o regime. Furthermore, prior work does not address full geometric recovery via uniform sampling.
Asymptotic behavior of the score. It established that under the manifold hypothesis, the score function develops singularity the small-noise regime, becoming orthogonal the data manifold. Recent works characterize this behavior mathematically, showing that the score effectively acts as geometric projection operator onto the manifold (Lu et al., 2023 Lyu et al., 2025 Liu et al. 2025). This aligns with the oodder term in our expansion (Equation (6)), which governs geometric concentration. However, these analyses generally subsume the distributional information into generic bounded remainder term. Crucially, they do not explicitly isolate the higher-order terms involving Pásta and thus do not characterize the separation between geometry and density. Our analysis reveals that these missing terms are not merely residuals but essential for establishing the rate separation between recovering the manifold support and learning the underlying density.
Uniform sampling on manifolds. Classical approaches achieve uniform-on-manifee sampling via graph-based normalizations that cancel the sampling density that the limiting operator is the Laplace-Beltrami operator (Coifman & Lafon 2006; Hein et al. 2007). While foundational, these methods are designed to approximate geometric operators from neighborhood graphs and do not readily scale to high-dima nigh-dimensional, large-scale generative modeling. Recently, De Santi et al. (2025) proposed fine fineituning diffusion models to produce uniform samples. In contrast, our approach operates entirely at inference time, achieving uniform sampling without the cost of fine-tuning.
2 PRELIMINARIES AND NOTATION
In this work, we adopt the manifold assumption (Song & Ermon 2019 De Bortoli 2022 Loaiza- Ganem et al. 2024) as follows:
Assumption 2.1 (The Manifold Hypothesis). We assume that the data distribution pdata is sup- ported on compact, boundaryless C* embedded submanifold M C Rd., with dim(MM n.
Local coordinates and manifold geometry. Under the manifold hypothesis, the ne-dimensional manifold M can be described locally using coordinates from a flat Euclidean space. This is done via set of smooth mappings, parameters c R" to a patch on the manifold. For notational simplicity, we will work with a single chart, where u E U represents the local coordinates of a point (u) on M. The manifold's intrinsic,
charts, & U - M, where each chart maps open set of
3

---

Published as a conference paper at ICLR 2026
and generally non-Euclidean, geometry is captured by the Riemannian metric tensor, g(u). This tensor provides the means to measure lengths and angles on the curved surface. The metric gives rise to the Riemannian volume measure, dzM(z), which is the natural way to integrate a function f.M- + R over the manifold. In local coordinates, this integral is expressed as JM [(x) dM(x)) = Jo (d((u))
du, w.I.t the Lebesgue measure on U. Here, the term v (d)) is the volume correction factor. While we use single chart for clarity, integration over the entire compact manifold is handled by stitching together multiple charts via a partition of unity. The set of points in Rd that are sufficiently close to the manifold forms the tubular neighborhood: TM((E) = f E I dist(z, MM < <). For any point x within this neighborhood, there exists a unique closest point on the manifold, given by the PM (x) (c) + M. This projection allows us to define the squared distance function to the manifold, quantity of central importance to our analysis:
1
d (x) = dist (z, = min | ~1
(2)
2
seM 2
Further details and notations regarding the manifold hypothesis are provided in Appendix A
2.1 THE GAUSSIAN SMOOTHED MEASURE AND CONNECTION TO DIFFUSION MODELS
With Assumption 2.1 we define the corresponding density Pdatta of pdata with respect to the Lebesgue measure on U: (u) == d( ydatal (u), where 4" (SdataS) = Mdataa (D))) for S SCU. U, and assume the following regularity assumption:
Assumption 2.2 (Regularity and Converage of pdata). The probability density Pdata U R defined w.r.t the Lebesgue measure on U C2 (U) and strictly positive.
Recall the two Gaussian smoothhed measures Me introduced in Equation D..
We follow the naming
convention of Song et al. (2021) and denote by M the variance explooiigg (VE) smoothing and by UYY the variance preserving (VP) smoothing. Their densities w.r.t. the Lebesgue measure on Rd is
Po (x := S/ (2ma2)a/2
|| ( (o)((1)))
exp
Pdata du,
20²
where the densities are denoted p.y for VE with >(r) = 1 and p. for VP with 7(r) = vi r²
We take Pdata be the true population density rather than a finite-sample empirical approximation.
These smoothed distributions correspond to the marginals of the forward noising processes used in diffusion and score-based generative modeling. In SMLD or VE-SDE (Song et al. 202T). Gaussian noise with variance o² (t) R + R+ is added to the data at time t, a model is trained to progres- sively denoise, and in the reverse process the objective is to sample from p Pa(g) recovering Pdata as
t + 0 (equivalently, (t) + 0). Similarly, DDPM or VP-SDE (Ho et al., 2020 Song et al. 2021 VP corresponds the VP density Po(e) again with the goal of recovering Pdata in the limit + O.
Beyond the reverse process, one may also directly use the learned score to run Langevin sampler targeting y. (Song & Ermon 2019) or P.P. or combine Langevin sampling with the reverse pro- cess, as in the Predictor-Correctoor algorithm (Song et al.] (021). Since our results apply to both VE and VP settings, we adopt the unified notation Pa whenever ambiguity arises.
2.2 BAYESIAN INVERSE PROBLEMS
Another important algorithmic implication of our results concerns Plug-and- (PnP) methods for Bayesian inverse problems (Venkatakrishnan et al. 2013). Let x id be the latent signal and
y E Y C R the observation y = + where A R Rm is the measurement map and & e RTT is noise. Under standard assumptions A and & (e.g., A linear, e~ (0, s2 I)), the likelihood admits density p(y æ)) exp( - v(x; v)) (for the Gaussian case, v(z;; u) 2."h Ax()) > In the Bayesian framework we endow 2 with a prior Pptior. Inference is cast as sampling from the posterior p( v) = p(y z) Pprioc(x)/ (x)/ f p(y z) Pprior ( (i)dF.
Plug-and-Play (PnP). PnP methods address the case where the prior is (i) known up to a normal- izing constant, e.g. Gibbs measure (ii) only accessible via samples (common in ML). A unifying sampling paradigm is posterior Langevin with a (possibly learned) prior score $ ~ V log Pprior.
dX,- -V () dt 8(X.) dt V2dW,
4

---

Published as a conference paper at ICLR 2026
In case (ii), $ is score estimator obtained, e.g., by score matching prior samples. A common choice of Porior would be the density Ps defined in eq. (3) with small σ. this context, to ensure up- date 4 yields samples matching the target posterior distribution, existing works require the learned score 8 to be at least ((1) accurate Laumont et al. 2022). or even exact (Pesme et al. 2025
2.3 STATIONARY DISTRIBUTION FOR NON-REVERSIBLE DYNAMICS
In score learning, one typically learns score function s(z, é) for target density and then runs Langevin dynamics (equivalently, the corrector step in the Predictor-Coreecto algorithm for diffu- sion models (Song et al. 2021) until near stationarity to sample from that density:
dX, = s(X,, ct dt + v2dW.
If sx, (z) the stationary distribution proportional exp(-fe(z)), In practice, how- ever, the score is often produced by a parameterized model and need not be a gradient field (this is also the case for our proposed algorithms). The resulting Langevin dynamics then generally non- reversible, and its stationary distribution need not admit closed form open problem studied in, e.g., (Graham & Tél 1984 Maes et al., 2009 Rey-Bellet & Spiliopoulos 2015).
Several works have sought to characterize the stationary distribution of non-reversible SDEs. No- tably, Matkowsky Schuss [1977; Maier Stein (1997); Graham & Tél (1984); Boucher &
Reygner (2016) employ the WKB ansatz (Wentzel 1926; Kramers 1926 Brillouin (926), which
is commonly used in matched asymptotic expansions (Holmes) 2012). This approach posits that the stationary density takes the form
exp (- (-V(e) cz(z), with cx(x) Ea(t)e, 1=0
(5)
for some & E N. The functions V and } are then identified by inserting (5) into the stationary Fokker-Planck equation and balancing terms order by order in c. Importantly, prior analyses typi- cally focus low-dimensional special examples on drifts with single stable point. The difficulty of removing such restrictions turn out to be central to our analysis; see Section for details.
3 CENTRAL INSIGHT: GAUSSIAN SMOOTHING RECOVERS GEOMETRY BEFORE DISTRIBUTION
This section presents the central insight of the paper: While the proofs of our later main results are technically involved, they are all guided by a common intuition that is transparent and can be understood through a simple Taylor expansion of logpr
0:
Theorem 3.1 (Informal Theorem B.2h. Assume Assumptions 2.1 and 2.2 holds. For any E TM((),
log (x) = dm(()) - + Panta ( - d-n log((mo²) + +(x)+o(1)) o(1),
where (x) contains the curvature information of the manifold and is some sufficiently small constant; both of them are independent of a. The small o(1) term is uniform for TM(()..
From Equation (6) it follows immediately that the scaled log-density recovers the distance function the manifold in the small limit:
lim o² loggo -dv(x) uniformly for all TM((e. o=
The appearance of dm(x)) under the manifold hypothesis should not come as surprise; indeed, as Po + Pdata when o + 0, and since Pdata is supported entirely on M, any point æ with d'y (x) > must be assigned zero probability, which explains the divergent scaling factor o in the coefficient. What is more surprising is that only d (x) appears at leading order, with no dependence on Pdata: Information about Paata enters only at the higher-order tenms of (
This reveals fundamental rate separation: for any distribution supported on M, one must first re- cover (x) exactly before learning anything about pdata, any inaccuracy in (lM(a)) gets blown
5

---

Published as a conference paper at ICLR 2026
up by the diverging factor o=2 Moreover, coefficients encoding Pdata appear order O( 2) higher, meaning that extracting information about Pdsta requires a level of accuracy orders mag- nitude stricter than that needed to recover the manifold geometry, i... the distance function d.--
As demonstrated in Sections to 5 this observation entails several significant consequences for machine learning. Each of these can be understood manifestation of the fundamental rate separation between geometric recovery vs. distributional learning established Theorem 3.1
4 SCALE SEPARATION IN EXISTING GENERATIVE LEARNING: GEOMETRY VERSUS DISTRIBUTION
In this section, study the paradigm of existing generative learning where algorithms target to learn the Gaussian-smoothed measure Paa, such as the diffusion models discussed Section 2.1 We denote the corresponding perfect score function by s* (x,
Po (2:).
In practice, however, the generated samples may follow a different distribution due to imperfections such errors in training discretization of the reverse differential equation. We therefore let To (x): Rd + R denote the density of the distribution actually produced by an empirical algorithm, and define its associated score 8x (x) v log To (zz). Our analysis focuses on To in terms of discrepancies between ST,, (z) and the ideal score s" (z, o).
Before presenting our result, we impose the following assumption on the recovered distribution.
Assumption 4.1. We denote the log-density of the recovered distribution ff log (2), and assume that fo is (K). Furthermore, we impose the following conditions:
There exists a compact set C IRd with Tm(e) C such that the density concentrates on as + O, i.e. lo JK To (2) dx= 1.
2. K is uniformly rectifiably path-connected, meaning that for any two points x: y K, there exists a path in K connecting x and whose length is uniformly bounded for all x. y E K.
Remark 4.1. We believe our assumptions are already reflected practice: Since T, represents the effective distribution of the generated samples, it can incorporate standard constraints such as data clipping (e.g.. to 1)) used in many diffusion models (Ho et al., 2020, Saharia et al. 2022) compact set K as required. Furthermore, such
This ensures the generated density concentrates regular sets are naturally uniformly rectifiably path-connected.
We are ready to state our main result in this section; see Appendix B.3 for the proof.
Theorem 4.1. Suppose Assumptions 2.1
2.2 and 4.7 hold. Denote the score error as
Eg = s5, s" (.,))ll==())
Concentration on Manifold. If we have that Eg = 22),
then T, concentrates on M, Lee.
lim a a=1 dist(
Te (x))d for any 8 > 0.
2. Arbitrary Distribution Recovery. For any distribution स supported M with C2 density, one can construct fo such that E. = =1{ as a + O, and no converges weakly to
Recovering Pdata- we have that E,, = o(1) as a + 0, then Ta converges weakly to Pdata-
This result formalizes the intuitive fact that recovering Pdata requires log म. to match the true score to within o(1) accuracy + 0. The reason is clear from the expansion (6): the distribution Pdata only appears in the @(1) term, and any larger error would overwhelm this information. In practice, however, achieving such accuracy is extremely challenging, particularly in the small-o regime. However, recovering the manifold is simple-one only o(1/a²) accuracy is required such that 0, the density will concentrate on M- shape separation from recovering Pdata-
Implications to Diffusion Models. As we mentioned before, the paradigmatic example to which results can be applied is diffusion models. Our Theorem 4.1 then reveals a sharp scale separation

---

Published as a conference paper at ICLR 2026
in terms of the score emror: well before the true distribution Pdata fully recovered, one already recover a distribution supported on the same data manifold. In practice, this often suffices, as what truly matters capturing the structural features of the manifold-realistie realistic images, plausible protein conformations, or meaningful material geometries. This insight provides a potential new explanation for the remarkable success of diffusion models.
NEW PARADIGM OF GEOMETRIC LEARNING: RECOVER UNIFORM DISTRIBUTIONS WITH o(o SCORE ERROR
As shown in Theorem 4.1, while concentration on the manifold is orders of magnitude simpler, the recovered distribution can still be arbitrary unless the score learned with o(1) accuracy. In contrast, we show in this section the striking fact that even with score emrors as large as o)a--), with simple modification of the existing algorithm, one can recover the uniform distribution on the manifold-a fundamental distribution that plays key role in scientific discovery and encodes rich geometric information about the manifold (De Santi et al. 2025 Belkin & Niyogi 2008).
Unlike in Section 4 where we compared errors by evaluating a learned distribution TT against the ideal Po through their score functions, in this section we assume direct access to an estimated score oracle s(, o), such as those learned via score matching in diffusion models. Given access to such an oracle, our proposed algorithm consists of running the following SDE for some a > 0:
dXi σ s( dt v2dW,,
which we refer to as the Tempered Score (TS) Langevin dynamics. We claim that, under mild error assumptions, the stationary distribution of this SDE, denoted 7.. converges the uniform distribution on the manifold as - 0.
Our analysis proceeds two steps. First, we establish the result simplified setting where the score oracle s(. o)) is guaranteed to be a gradient field, with a proof analogous to Section Second, we tackle the substantially more challenging case which no a priori gradient structure is assumed. Full proof's are provided in Appendix B.5
Warm-up: Score Oracle is a Gradient Field. We use the same notation as Section namely s(z,o) f (z). In this case, the stationary distribution of Equation (8) admits the explicit form
# (x) x fx)))==(((())))
We then obtain the following result, using a proof technique similar to that of Theorem 4.1
Theorem 5.1. Assume Assumptions 2.7
2.2 and hold. with To replaced by 7.. Suppose
||s(;,)) 5 (- = =((0)) for some B > -2
Then for any max ( -3,,0} < < 2, as + 0, *o converges weakly to the uniform distribu- tion on the manifold M with respect to the intrinsic volume measure. More precisely, the limiting distribution ã with respect the Lebesgue measure U satisfies
d.M
7(u)=
x du
(u);
where (dM/du)(u) = (d)t(g(u)) is the Riemannian volume element on M.
General Non-Gradient Score Oracle. While theorem 5.1 already illustrates the rate separation phenomenon we wish to emphasize, it relies on the highly impractical assumption that the estimated scores s(, a) are exact gradient fields. To enhance the applicability of our framework, crucial to relax this stringent assumption.
As discussed Section 2.3 existing approaches to non- non-gradient scores (and hence non-reversible dynamics) typically assume the existence of unique point x: such that lim,- s0"("", " , 0, with the key consequence of collapsing the prefactor Co in 55) to a normalization constant cy cx(x)) Our framework. however, explicitly violates this assumption: we require that lim o" bilizes to a manifold rather than a singleton. Under this setting, the limiting behavior co is far from obvious, and the resolution of this issue turns out to be highly nontrivial.
o s(,, or) sta-

---

Published as a conference paper at ICLR 2026
To this end, central part of our proof is devoted to showing that cu nevertheless remains constant, albeit for an entirely different reason: we prove that the higher-order terms in the Fokker- Planck-Planck expansion enforce Co to satisfy a parabolic PDE the manifold, and by the strong maximum principle (Gilbarg et al. 1977). the only solutions on a compact manifold are constants.
With these techniques, obtain the same conclusion as Theorem 5.1
Theorem 5.2. Assume Assumptions 2.7 and 2.2 and eq. (9) hold, and further suppose Pdaca (C) For any max[-3.00 < a < 2. assume that the SDE admits a unique stationary distri- bution, denoted n. which locally admits a WKB form (Assumption B.2 with 02-a). Then the conclusion of Theorem 5.7 holds.
Setting 0 in eq. recovers the standard Langevin sampler the "Corrector" step commonly used in diffusion- sampling (Song et al. 2021). Our results in Theorems 5.1 5.2 therefore imply that simple, one-line modification of these standard schemes is enough to recover the uni- form distribution on the data manifold from samples Pdata- even when the score error as large as o(a substantially weaker requirement than the o( accuracy needed to recover Pdata itself. Remark 5.1. In Appendix(] [D] provide further diseussion on the convergence (mixing time) of TS Langevin compared to standard Langevin dynamics. While characterizing the general convergence rate is non-trivial problem left for future work, our analysis indicates that TS Langevin maintains comparable algorithmic efficiency. In fact, by analyzing the Poincaré constant, we identify concrete examples where TS Langevin converges provably exponentially faster than standard, untempered Langevin dynamics.
UNIFORM PRIOR IS MORE ROBUST BAYESIAN INVERSE PROBLEMS
In Bayesian learning, often sets the prior Pprior the Gaussian-smoothed data distribution Po defined in Equation 3 with some small smoothing parameter a. To ensure asymptotically correct posterior samples under this choice, the learned score typically must be exact (Pesme al.
2025). V log Pa. or achieve vanishing error, || V log p- o(1) (Laumont et al.) 2022 Proposition 3.3 and H2). In contrast, under our framework, if one adopts the manifold volume measure (i.e., the uniform distribution on .M) as the prior, then correct posterior sampling can be attained under substantially weaker requirement: it suffices that the score error scales as 020- The precise statement is given in the theorem below.
Theorem 6.1. Under the same assumptions as in Theorem 5.2 and suppose v : Rd + is bounded on RR, and C on (e). Then, as + O, the stationary distribution of the SDE dx. = -v(xz) dt ( dt + V2W,,
(10)
converges weakly to a distribution supported on M with density x exp(-e(((u)) dM (u).
du
Diffusion Models with Classifier-Free Guidance. The above result can also be applied to diffu- sion models. The drift term in Equation (TO) represents the effective score of a diffusion model with classifier- free guidance (Ho & Salimans, 2022p. In this formulation, - V fo denotes the uncondi- tional score estimate, while the guidance term Ve equals the guidance scale w times the difference between the conditional and unconditional score estimates. Our tempered score can be applied di- rectly CFG diffusion models with a Predictor Coorrecor sampler: in the corrector (Langevin) step, replace the score by its tempered version according to Equation (TOp (i.e., scale the unconditional score by o²), We will demonstrate the effectiveness of this modification empirically in Section 7.2
7 EXPERIMENTS
To empirically validate our theory, we present preliminary experiments on both simple synthetic manifolds and a real-world image-generation setting with diffusion models. On synthetic manifolds, we directly verify the claims of Section [5] demonstrating recovery of the uniform distribution on the manifold. In the image domain, we show that our proposed algorithm yields samples that are both more diverse and high-quality. Further experimental details are provided in Appendix
7.1 NUMERICAL SIMULATIONS ON ELLIPSE
In this subsection, we illustrate our theoretical results with numerical simulations. We consider simple manifold given by ellipse embedded in the two-dimensional Fuclidean space, M
8

---

Published as a conference paper at ICLR 2026
2.15
r3
633
6/55
Aage Un4)
Aaga One)
Angle (p4)
(a) L (ellipse)
(b) TS-1 (ellipse)
(c) (circle)
(d) TS- (circle)
Figure 2: Comparison of stationary sample distributions generated with standard L Langevin dynamics (L) versus Tempered Score Langevin dynamics Equation (8) with a 1 (TS-1). The circle and ellipse correspond to manifolds with (a, b) = (1, ) and (a, b) (1, 2), respectively.
Prompt
Furniture
Car
Architecture
Method
P-simf
[-sim.]
P-sim
I-sim
P-sim
I-sim
DDPM
29.56
80.78
26.23
87.30
27.36
81.53
PC
29.40
81.24
26.30
87.20
27.13
81.03
TS (ours)
30.20
80.76
26.62
87.14
27.32
80.76
Table I: Comparison of images generated by DDPM, PC, and TS. The prompts used are "Creative furniture," " "An innovative car design," and "A creative architecture." For PC and TS, the number of corrector steps and (for TS) are tuned.
1}, a, a,b O, and Pdata is chosen to be von Mises distribution supported on the angular parameterization of the ellipse. The score function is parameterized using transformer-har based neural network, trained with the loss function introduced in (Song Ermon 2019). After training, we evaluate the learned score function with a 10 10-2 and perform Langevin dynamics until convergence. Training hyperparameters are tuned to minimize the test loss.
{(z,)) E R | (z/a)²+
As shown in Figure 2 the stationary distribution produced by standard Langevin dynamics deviates substantially from Pdata; even in this simple elliptical setting, highlighting the difficulty of accurately learning the score function at small a. In contrast, our TS Langevin dynamics reliably recovers the uniform distribution on the manifold, in agreement with Theorem 5.2
7.2 IMAGE GENERATION WITH DIFFUSION MODEL.
To validate our theoretical findings in practical, large-scale setting, we conducted experiments on image generation. We demonstrate that a one-line modification to the widely-used Predictor- Corrector (PC) sampling algorithm (Song et al. 202I) can enhance both the quality and diversity of images generated by pre-trained diffusion model. These experiments serve as proof of concept, applying our proposed Tempered Score (TS) method off off-the-shelf diffusion models. Our modi- fication targets the corrector step of the PC algorithm, which uses Langevin dynamics to refine the sample at each stage of the reverse process. In TS method, we scale the unconditioned score prediction by a factor of oa, as motivated by our analysis and discussion in Section 6 The standard classifier-free guidance term, i.c.. Ve in Equation (IO), remains unchanged. Specifically, we com- pare Stable Diffusion 1.5 (Rombach et al. (022) with DDPM sampler (Ho et al. 2020). DDPM with PC sampler, and DDPM with our TS sampler.
We evaluate the performance using two metrics derived from CLIP scores (Hessel et al. 2021 which measure the cosine similarity between feature embeddings. Quality: We use the CLIP Prompt Similarity (P-sim), defined as the average CLIP score between the generated images and their corresponding text prompt. A higher P-sim value indicates better alignment with the prompt and thus higher image quality. Diversity: We use the CLIP Image Similarity (I-sim), which is the average pairwise CLIP score between all images generated with the same prompt. A lower I-sim value means greater diversity among the samples.
The experimental results in Table and Table provide empirical validation of our theoretical framework. Our proposed TS method consistently generates more diverse images than the DDPM and standard PC baselines across three distinct prompts, while maintaining very high image quality. In particular, Table 2 shows that, for all numbers of corrector steps considered, TS outperforms

---

Published as a conference paper at ICLR 2026
Num. Corrector Steps
5
10
15
20
30
Prompt
Method
P-sim²
I-sim]
P-sim
I-sim
P-sim
I-sim
P-sim
I-sim
P-sim
I-sim
Furniture
PC
29.40
81.34
29.30
81.24
29.32
81.64
28.98
81.72
28.67
82.33
TS (ours)
29.54
81.11
29.58
80.95
29.68
81.34
29.52
81.15
29.43
81.87
Car
PC
26.20
87.20
26.30
87.57
26.24
87.98
26.26
88.06
26.17
87.94
TS (ours)
26.23
87.14
26.37
87.42
26.32
87.88
26.28
88.07
26.20
87.87
Architect.
PC
27.13
81.83
27.13
81.81
26.92
81.64
26.87
81.60
26.60
81.03
TS (ours)
27.23
81.58
27.27
81.57
27.14
81.54
27.06
80.97
26.84
80.76
Table 2: Comparison of images generated by PC and TS across different numbers of corrector steps. For TS, a is used without further tuning. The prompts are the same as Table [
Figure 3: Top row: PC. Bottom row: TS (ours). Samples in the same column are generated using the same prompt, the same number of corrector steps, and the same random seed. As shown, TS produces samples that appear more authentic and contain richer details.
standard PC in nearly every case. Crucially, these improvements robust the choice of and are not merely the result of a larger tuning budget; as demonstrated in Table simply setting a = 1 without further tuning is sufficient consistently enhance both quality and diversity compared to the baseline. Examples of the generated images by PC and TS are shown in Figure
CONCLUSION
This paper advocates for paradigm shift in score-based learning, moving from the difficult goal of full distributional recovery more robust, geometry-fist approach. We demonstrate fundamen- tal rate separation in the low-noise limit, where information about the data manifold is encoded at significantly stronger scale (O(o 2)) than details about the on- no-maaoiold distribution (9(1). This finding explains why models often succeed at capturing the data support even with imperfect score estimates. Building on this insight, we introduce Tempered Score (TS) Langevin dynamics, simple one-line modification that robustly targets the uniform volume measure on the manifold, tolerating score errors up to a(a -2). This geometric approach not only provides more stable foundation for Bayesian inverse problems but also, shown in experiments with models like Stable Diffusion, empirically improves the diversity and fidelity of generated samples.
Limitations and future work. Key limitations and future directions include: a) The implications for diffusion models are presently limited: we do not track cumulative error along the sampling trajectory; instead, analyze simplified setting that assumes access to the eπro of the final generated distribution. b) Our I score-error assumption could potentially be relaxed to an I bound, thereby aligning theoretical framework with practical training objectives like denoising score matching (Fisher divergence) that minimize L error. c) It remains to generalize the rate separation in score estimation into corresponding results on statistical sample complexity. d) Our analyses on the uniform sampling are in continuous time; we do not quantify discretization error arising practical implementations. e) Our experiments are preliminary; we have not conducted large-scale study with -fate-of-thh-ase diffusion models.
ACKNOWLEDGMENT
The work is supported by ETH research grant, Swiss National Science Foundation (SNSF) Project Funding No. 200021-20734 and SNSF Starting Grant.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Marloes Arts, Victor Garcia Satorras, Chin-Wei Huang, Daniel Zugner, Marco Federici, Cecilia Clementi, Frank Noé, Robert Pinsler, and Rianne van den Berg. Two for one: Diffusion mod- els and force fields for coarse-grained molecular dynamics. Journal of Chemical Theory and Computation, 19118:6151-6115 2023.
Iskander Azangulov, George Deligiannidis. and Judith Rousseau. Convergence of diffusion models under the manifold hypothesis in high-dimensions. arXiv preprint arXiv:2409. /8804, 2024.
Mikhail Belkin and Partha Niyogi. Towards a theoretical foundation for laplacian-based manifold methods. Journal of Computer and System Sciences, 74(8):1289-133) 2008.
Thibault Bonnemain and Denis Ullmo. Mean field games in the weak noise limit: wkb approach to the fokker-planck equation. Physica A: Statistical Mechanics and its Applications, 523:310-322 2019.
Freddy Bouchet and Julien Reygner. Generalisation of the eyring-kramers kramers transition rate formula to irreversible diffusion processes. In Annales Henri Poincaré, volume 17, pp. 1499-3532 Springer, 2016.
Léon Brillouin. La mécanique ondulatoire de schrödinger; une méthode générale de résolution par approximations successives. CR Acad. Sci, 183(11):24-26, 1926.
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is easy as learning the score: theory for diffusion models with minimal data assumptions. In International Conference on Learning Representations, 2023.
Ronald R Coifman and Stéphane Lafon. Diffusion maps. Applied and computational harmonic
analysis, 21(1):5-30, 2006.
Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. Transactions on Machine Learning Research, 2022.
Riccardo De Santi, Marin Vlastelica, Ya-Ping Hsich, Zebang Shen, Niao He, and Andreas Krause. Provable maximum entropy manifold exploration via diffusion models. arXiv preprint arXiv:2506. 15385, 2025.
David Gilbarg, Neil S Trudinger, David Gilbarg, and NS Trudinger. Elliptic partial differential equations of second order, volume 224. Springer, 1977.
Yun Gong, Niao He, and Zebang Shen. Poincare inequality for local log- log-lolyyk-yy.. ojaseewwez mea- sures: Non-asymptotic analysis in low-temperature regime. arXiv preprint arXiv:2501.00429, 2024.
Graham and T Tél. On the weak-noise limit of fokker-planck models. Journal of statistical physics, 25(5):729-745) 1984.
Matthias Hein, Jean-Yves Audibert, and Ulrike von Luxburg. Graph laplacians and their conver- gence on random neighborhood graphs. Journal of Machine Learning Research, 8(6), 2007.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXi::2104.08718, 2021.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.
arXiv preprint
arXiv:2207. 12598, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:684006851, 2020.
Richard Holley and Daniel Stroock. Logarithmic sobolev inequalities and stochastic ising models.
Journal of Statistical Physics, 155--1144, 1987.
Mark Holmes. Introduction to perturbation methods, volume 20. Springer Science & Business Media, 2012.
11

---

Published as a conference paper at ICLR 2026
Chei Hwang. Laplace's method revisited: weak convergence of probability measures. The Annals of Probability, pp. 1177-1182, 1980.
Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score match- ing. Journal of Machine Learning Research, 6(4), 2005.
Yazid Janati, Badr Moufad, Alain Durmus, Eric Moulines, and Jimmy Olsson. Divide-and-conquer posterior sampling for denoising diffusion priors. Advances in Newral Information Processing Systems, 37:97408-97444, 2024.
Jürgen Jost. Riemannian geometry and geometric analysis. Springer, 2005.
Zahra Kadkhodaie and Eero P Simoncelli. Solving linear inverse problems using the prior implicit in a denoiser. arXiv preprini urXiv:2007.13644 2020.
Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-
based generative models. Advances in neural information processing systems, 35:26565-26577, 2022.
Hendrik Anthony Kramers. Wellenmechanik und halbzahlige quantisierung. Zeitschrift für Physik, 39(10):828 840, 1926.
Tomasz M Lapinski. Multivariate laplace's approximation with estimated error and application to limit theorems. Journal of Approximation Theory, 248: 105305, 2019.
Rémi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. Bayesian imaging using plug & play priors: when langevin meets tweedie. SIAM Journal on Imaging Sciences, 15(2):701-737, 2022.
Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of score-based generative modeling for general data distributions. In International Conference on Algorithmic Learning Theory, pp. 946-985 PMLR, 2023.
Gunther Leobacher and Alexander Steinicke. Existence, uniqueness and regularity of the projection onto differentiable manifolds. Annals of global analysis and geometry, 60(3):559-587, 2021.
Zichen Liu, Wei Zhang, and Tiejun Li. Improving the euclidean diffusion generation of manifold data by mitigating score function singularity. arXiv preprint arXiv:2505.09992,, 2025.
Gabriel Loaiza-Ganem, Brendan Leigh Ross, Rasa Hosseinzadeh, Anthony Caterini, and Jesse C Cresswell. Deep generative models through the lens of the manifold hypothesis: A survey and new connections. Transactions on Machine Learning Research, 2024.
Yubin Lu, Zhongjian Wang, and Guillaume Bal. Mathematical analysis of singularities in the diffu- sion model under the submanifold assumption. arXiv preprint arXiv:2301.07882 2023.
Yang Lyu, Tan Minh Nguyen, Yuchun Qian, and Xin T Tong. Resolving memorization in empirical diffusion model for manifold data in high-dimensional spaces. arXiv preprint arXiv:2505.025008 2025.
Christian Maes, Karel Netočný, and Bidzina M Shergelashvili. Nonequilibrium relation between po- tential and stationary distribution for driven diffusion. Physical Review F F-Statisticall, Nonlinear, and Soft Matter Physics, 80( (0(1001111 1221,111,1110 2009.
Robert S Maier and Daniel L Stein. Limiting exit location distributions in the stochastic exit prob- lem. SIAM Journal on Applied Mathematics, 57(3):752-790, 1997.
Piotr Majerski. Simple error bounds for the multivariate laplace approximation under weak local assumptions. arXiv preprint 1rXiv:151.000300 2015.
Bernard Matkowsky and Zeev Schuss. The exit problem for randomly perturbed dynamical sys- tems. SIAM Journal on Applied Mathematics, 33(2))365-33) 1977.
Georg Menz and André Schlichting. Poincaré and logarithmic sobolev inequalities by decomposi- tion of the energy landscape. The Annals of Probability, 1885); 2014.
12

---

Published as a conference paper at ICLR 2026
John Willard Milnor and James D Stasheff. Characteristic classes. Number 76. Princeton university press, 1974.
James Raymond Munkres. Topology. Prentice Hall, 2nd edition, 2000.
Elizabeth Pavlova and Xue-Xin Wei. Diffusion models under low-noise regime. arXiv preprint arXiv:2506.07841 2025.
Scott Pesme, Giacomo Meanti, Michael Arbel, and Julien Mairal. Map estimation with denoisers: Convergence rates and guarantees. arXiv preprint arXiv:2507.15397 2025
Jakiw Pidstrigach. Score- based generative models detect manifolds. Advances in Neural Information Processing Systems, 35:35852-358666 2022.
Sanjeev Raja, Martin Sipka, Michael Psenka, Tobias Kreiman, Michal Pavelka, and Aditi S Kr- ishnapriyan. iettoonmiiimizztiion meets generative modeling: Efficient transition path sampling with the onsager-machlup functional. arXiv preprint arXiv:2504. 18506, 2025.
Luc Rey-Bellet and Konstantinos Spiliopoulos. Irreversible langevin samplers and variance reduc- tion: large deviations approach. Nonlinearity, 28(7):2081, 2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 15:36479-3649/1 2022.
Saeed Saremi, Rupesh Kumar Srivastava, and Francis Bach. Universal smoothed score functions for generative modeling. arXiv preprint rrXi:2303.11666 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
Jan Pawel Stanczuk, Georgios Batzolis, Teo Deveney, and Carola-Bibiano Schönlieb. Diffusion models encode the intrinsic dimension of data manifolds. In Forty-first International Conference on Machine Learning. 2024.
Rong Tang and Yun Yang. Adaptivity of diffusion models manifold structures. In International Conference on Artificial Intelligence and Statistics, pp. 1648-1656. PMLR, 2024.
Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. In 2013 IEEE global conference on signal and information process- ing. pp. 945-948. IEEE, 2013.
Enrico Ventura, Beatrice Achilli, Gianluigi Silvestri, Carlo Lucibello, and Luca Ambrogioni. Man- ifolds, random matrices and spectral gaps: The geometric phases of generative diffusion. arXiv preprint arXiv:2410.05899 2024.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- lution, 166117666 2011.
Gregor Wentzel. Eine verallgemeinerung der guantenbedingungen für die zwecke der wellen- mechanik. Zeitschrift für Physik, 38(6):518 02992, 1926.
Hermann Weyl. On the volume of tubes. American Journal of Mathematics, 61(2):461-472, 1939.
Stephen Willard. General topology. Courier Corporation, 2012.
13

---

Published as a conference paper at ICLR 2026
A ADDITIONAL NOTATION AND PRELIMINARIES
In this section, we provide some notation and preliminaries complementary to Section
2
We denote by W, standard Brownian motion, with its dimension clear from context. The Gaussian density with mean i and covariance Σ, evaluated at I, is written as N(x f4, E). The symbol denotes the convolution operator. We use to indicate proportionality, i.e. that the left-hand side and right-hand side are equal up to a constant factor. For a set S, we write S for its closure, OS for its boundary, and 5e for its complement. Throughout the paper, by the term limiting distribution or by convergence of distribution/density function, we mean convergence of the corresponding measures in the weak sense.
A.1 THE MANIFOLD HYPOTHESIS
We outline few notations and standard results from differential geometry. By the tubular neigh- borhood theorem (Milnor & Stasheff 1974, Weyl, 1939), there exists such that the normal tube
TM(E): == {a E Rd. dist(a, M) < <]}.
admits local C+ coordinate
:: x R R Tm(()
where UCR R (r
TR-"" |||||| <c),
such that & is a diffcomorphism mapping from local coordinates to ambient Euclidean space. With this result, we can then work with local coordinates describe the manifold. For notational sim- plicity, we work with a single chart and suppress indices: u E U denote tangential coordinates and r R denote normal coordinates. The slice r = corresponds to points on M, and we write ((u) := (uu, 0). Let J(u, r) denote the Jacobian of ((u.r) with respect to (u, r), i.e., Ju,rr = @b(u, ((,, r). Furthermore, let g(u) denote the Riemannian metric tensor of the manifold M, defined as g(u) := J(u, 0) J(u, 0). Intuitively, the Riemannian metric tensor gives way to measure lengths and angles of the manifold geometry.
B PROOFS OF MAIN THEOREMS
In this section, we prove the main theorems of the paper. We begin by developing a general frame- work for characterizing the limiting distribution when the density admits a specific form. This framework will then be applied to establish the results in Section 4 where such density form was assumed.
The results in Section a different approach, since no explicit form of the density available. In this case, we employ the WKB approximation to obtain approximate stationary distribution, which we then substitute into the general framework to derive the limiting distribution.
B.1 A GENERAL FRAMEWORK FOR THE CONVERGENCE OF THE LIMITING DISTRIBUTION
In this subsection, will establish general framework for the limiting distribution of density proportional to
exp (- (=)(()))))) /0), with fo(x) = fo(x) (0)
(11)
where fo's minimizer is on the manifold M and f(x, 0) is a perturbation that is uniformly o()) so that it does not affect the limiting distribution. This general result stated in Theorem B. Our main results fall into this framework by letting 0 = r² for Theorem 4.1 and 0 = σ2-a for Theorem 5.2
In all cases the theorems we will prove later, the density will concentrate on the tubular neighbor- hood of M. i..,, TM((). Therefore, we will discuss the lemmas and intermediate results in such a neighborhood and use local coordinates (u, r). The notations used can be found in Section 2 When we use local coordinates, we assume the discussion is in the closure of TM((). We define the local coordinate versions of the functions: fo(urr) = fo(((u, folu, r) = fo( (1, r) == ( r)), and f(,, r, f(((u,)), Our assumptions are stated follows.
14

---

Published as a conference paper at ICLR 2026
Assumption B.1. We assume that
1. M C R is a compact C+ manifold without boundary with dimension n. << d.
2. .M = arg wigminger sET fo(æ)). In addition, we assume that there exists 0 < ē < such that inf [ETM(e)TTM() fo(z) minzeTT(le) fo(z) is bounded away from zero.
3. The absolution value of f(u,r, r. 0) is o(0) as A + 0 uniformly for all E U and ||r|| <c
4. fo 2 is C fi is C², and fe is continuous coordinates (u,r) for all U and ||||| S e, i.e. in the closure T((e).
5. Further, we assume that the smallest eigenvalue of zero for all E and ||||] <.
L (u,r) r) is uniformly bounded away from "
Remark B.1 (Compactness of the manifold implies boundedness of gradients.). Consider
Ok (T). In local coordinates (u,r) induced by a tubular atlas, we write fu, == (()( Since M is compact, one can choose a finite atlas with precompact coordinate domains. Let the cover be {U.}. By the Shrinking Lemma (Munkres (2000) Theorem 32.3) combined with Willard 2012 Theorem 15.10)), there exist open subsets {V:} with V. U, such that {Vi} still forms cover. We use these }Vi} as the new atlas. The transition maps & and their derivatives are then bounded on these sets (since V; is compact), and by the chain rule the same holds for f(u, r) and its derivatives up to order k. Thus, throughout our arguments we may freely assume uniform bounded- ness of such derivatives without loss of generality. The same reasoning applies to Pdata, we can use the same constructed atlas such that Paaaaa is uniformly lower and upper bounded, and gradients of Pdata are uniformly upper bounded.
During our proofs, will frequently use Laplace's 's method for integrals. We adapt the error estimate from Lapinski (2019) as follows.
Corollary B.1 (Theorem of Lapinski 2019)) Let C R an open set and let S C St be closed ball. Let Vol(S2'. Let F S2 R with the following assumptions:
1. Fla E (S') and F 2 0 on Ω. There is a unique minimizer x² E Eint(52') of F on Q. Define
mi ==
inf z
{F(x) F(x*))) >0,
m2 inf Amin (V²F(x)) 0. zEl
Let
2 = sup
F<x)l
C3 = sup [(x(()
xE
=en
glar e (2') and Ja [g(x) dr < xo. Let
c = 1 [s(x))
64 = sup
gg(()), zeW
c5 = sup
Vg(()))
#e0
Then, for every 0,
[ (l(x)e-F(o))) de exp(-F(x*)/0)
(270ym/2
\VF((>)) (o(a*) (o(a")+ +(0)).
where [h]) can be upper bounded by a function (f ... Cb Mi, m2). Moreover, h(0) = O(v0) as 0 - The O(v0) uniform over any class of pairs (F, g) for which .I,...; CB are bounded above and ma, m2 are bounded below by strictly positive constants uniformly over the class.
Proof. The result follows directly from Lapinski 2019
Theorem 2).
To show the convergence of the distribution out the normal direction so as to obtain a distribution on u, such as what [wang (1980) did. The following lemma proves Laplace's type of result for integrating out
distribution the manifold, a key step is to integrate
15

---

Published as a conference paper at ICLR 2026
Lemma B.1. Assume Assumption B.1 and let h(æ): R. - R be C¹ and uniformly bounded in
TM(e). Define h(,,r) == h(0(u, r)). Then we have
J.rIee exp
fo(u,r)
h(u, r)dr
0
=exp ( [o(u,0)) exp (-fi(u. (2 (d-w)/2
(-Lotuu
(h(u, 0) + (1)),
88
dr (u, 0)
where the o{1) term is uniform for u.
Proof. We have that
solu,r)
exp
M(,,r)dr r)dr
Ir|lee
0
-I /
ro(ur))
(osp
([(urrs))
) dr
exp
exp (-fi(u,r)) h(u,r)
exp
0
= -Ii J
r)(u,r)
exp
exp (-f h(u,
r)( 0
f (u,r, -1) dr. 0
exp
exp
For the first term, we can directly apply Corollary
B.1 with F(r) = folu, g(r)
(xp(=fi(urr))((,,,) and S' being the ball {r ||]l| S fJ. Define
0)(2, exp (-fi(u, 0)) (2T0) (2 (d-n)/2
J exp exp
82 8F (u, 0)
The first term can be approximated as (h(u,, 0) + o()). The boundedness of the quantities in Corollary B.I will be discussed later. The second be upper bounded by
Flu,7.0)
1 exp (-folusr)
fo((,r))
sup/h(u, r) sup exp
exp (-fi(ur)) dr
=()) = o(1)J,
where used Corollary B. for the integral. The lower bound can be obtained similarly. The result follows.
{ch is uniformly bounded by the compactness of the manifold. The constant cg is uniformly bounded by our assumption on h. The uniform lower bounds of m, and m2 is guaranteed by Assumption B.
Regarding the uniform boundedness of the quantities in Corollary B.
Next, we will prove that the support of the limiting distribution will concentrate on the minimizers of the leading term. Previously, we considered fo consisting of fo + +(0) o(0). Next, we will show that as long as fo is fo + o(1), the concentration on fo 6's minimizers will happen.
Lemma B.2. Let fo(x) fo(x) + f(z, 0), such that exp(-fe(z))0) is normalized density function on Rd. Suppose M is a connected and compact Ca manifold without boundary. Assume that:
fo(x) is continuous with arg min 2eT fo(x) = M and min
min 2EXETM(() fo(x) = 0.
2. f(z, 8) is continuous and uniformly o(1) 0 + 0 for all E Tm(e).
3. The density concentrates in [M((c, i.e..
fo(z) de
lim exp (e(())
16

---

Published as a conference paper at ICLR 2026
For any n > O, define the C {x fu(x)) m). Then,
Jopurete exp(-fo(x)/80 dx - 0 - 0.
If in addition, exp(-fo(z)/0) converges weakly to distribution as 0 + O. the support of the limiting distribution is contained M.
Proof. Since we have that (T) exp(-fe(z))0)dr + 1, for the first result, it suffices to show that JTM(t)c,, exp(-Jo(z)/0)ddr + O. According to the assumptions, we have that for any 6, 30. such that 48 80 (f(zz 0) Therefore, we have
J Ju(e)nn, exp((-n+ I exx(Tx))))
Tilonn, exp(- exp(-fo(x)/O)ddt
We choose 5 n/2, then the right-hand side goes zero Let the limiting measure be P, and Pg be the probability measure corresponding to the density exp(=fa(z)/0), Since Cy an open set, we have that
P(C,)) & lim inf Po Po = O. 800
We also have that
P
[m(()
lim Po (T((t))
lim Po (TM(e))) 0. 0-0
Denote C == M We have that C = U C1/m U UTM(c)
Then we have
P(C) <
p(C + [Tulc)) O.
m=1
which concludes the proof.
Theorem B.1. Assume Assumption B. Define
To(x) x exp ( (-Lole)
Assume that 1 - SzeTmie) (o(x)d - 0 as 0 + 0. Then have that as 0 - 0, converges weakly the following distribution:
T(U)= exp(-fi(u,0)) 0))
foluo)
1/2
dM(u)/du
dr
SM exp(x=f11( 0))
fo(w.)) Gr
d.(u))/du
where d.M is the intrinsic measure on the manifold M, i.e., d.((u) = //(u))=/2du, and du the Lebesgue measure the local parameterization domain U.
Theorem 3.1). The only difference is that replace the estimate of Hwang (1980) Equation (3.2)) with our Lemma B. Note that Q Hwang 1980. Theorem 3.1) is assumed as a probability measure, thus f (in his notation) integrates to one. However, the proof technique of Hwwang (1980. Theorem 3.1) remains valid even if is not probability density, so applying to our case.
Proof. The proof follows the same as the proof in Hwang 1980
B.2 PROOF FOR THEOREM 3.1
The remaining the proof is to expand the true log-density w.r.t. a, analyze the error of the learned log-density, and then plug in the result obtained from Appendix B.1
17

---

Published as a conference paper at ICLR 2026
Theorem B.2. Assume Assumptions 2.1 and 2.2 holds. Suppose TM(c). Then have that
log (x) = 20
1| - PMM(2)] + log Pdata ( (PM((()))
n1
log 1og(2To²))
2
log
[((0 (V,(),)),
- PM)(( + log Pdlata ( (PM d d n2
logpo VP (1) = 20
log (og(2To²)-
2
(H [P())())()))..)) _ (P PM(x)) + 1
log
where (x, o) and v (x, are functions that are o(1) uniformly for E Tm((). The matrix H(, a)) is such that
(PQ()) D(u) - x + Ou,
( @D(()) @4(u)
H(u
Ou;
Ouj
Proof. We can apply Corollary B.1 as an error estimate for Laplace's method, the integral in Pa- The minimizer of F(u) is $ (x)( for both VE and VP.
We first consider the case of VE. By letting F(u) || (2u)ll g(u) Paata (u) and a²
we can obtain that
(2TO²) (n-d)/2
| PMM
(x())) exp
(pdata ( (f=)) h(a2))
20
(((0-1(MM(())
(12)
where [/[a2)] is O Now we take logarithmic and use the fact that log(A+ B) log A) +
log(1 B/A), we obtain
log (x)
1 (1)
n- d log(2To²) log M (PM()))) -1/2
20
log (pdata ( 1-1-ppp (PM(()))) + h((22))
(x)
+ n-
log(2TTo²) ) + log Pdata (
20
2
112
+ log (1+
1(022)
log A( (PM()()
Pdata ( 1 (P
Therefore, we have
h((22)
p(x, 0) log 1+
Pdata 1 (PM((M
The remaining is to show that 1((02)/ //rt)a/ (o-aaaaa-1110 (P-1(PM(x)) (x))) is uniformly o(1) for all æ E T (e)(e).
Since the manifold is compact, Pdats (u) is uniformly bounded away from zero (see Remark B.
The remaining is to find suitable S! and upper and lower bound the constants in Corollary B. We will discuss this later.
Now let us look at the case of VP. The only difference is the exponential, we changed from ||æ 2 to
| -
v(u) + (1- v1 01- @(u))
we do a Taylor expansion
1- 1-VI- x2 _ + o
18

---

Published as a conference paper at ICLR 2026
Using this expansion, we have that
| - +(u) (1
(1 1--v--o9 $(u)]l2
|| - +())1 + (Q(u), x (()) + o Q(u)).
Then we use the same argument as in the proof Lemma B.1 to show that the o(o²) does not affect the approximation. Specifically, let
J exp (- |1 PM(x))1
(2 (n- 12/4
20²
(P
and
(-Ike | - {(u)11?
K =
22 exp
exp (-2 ( (w()))
panta (w).
20²
We have
Jma (2m02)a/22 exp
| VI vI-o= 02
Pastaa (u)
202
S. Kdu / K (exp(o())(Q(u))))) du
< I Kdu + .S Ko(1)du
S J ( (T(((()))))))))))) exp ( + +o(1))
Pa(()))
The rest of the proof follows similarly to the proof of the VE case.
Then we need to discuss the upper and lower bounds in Corollary B.1 For the upper bounds,
since the manifold is compact, there exists such uniform upper bounds for {c:] (see Remark B.1
For the lower bounds we first consider Amiu (fr(u,, æ²). The part @b(u) aa(w)) is positive def- inite and uniformly bounded away from zero for all u. The eigenvalues of other part, i.c., @0(x))
Ou
Ju,duyy ((u) - ))) may be negative. However, long its eigenvalues are small enough, by Weyl's inequality, we can still lower bound the smallest eigenvalue of (u, x). The eigenvalues of a0(x)) Ju,duy (()) - x). can then be bounded by ([))l()))l ([ - z/|. Therefore, as long as the (H(u.x)) For- 8u au Let C2 be the uniform upper bound of ["))l, and Ci be that of [|V((u)]]- Those constants are uniform for a fixed finite atlas since the manifold is compact. Let the radius of Q be To. We have that in L
tubular neighborhood and the set S! is small enough, we can lower bound Amin
mally, let G > 0 be the lower bound of the smallest eigenvalue of @0(u)
T aw(w))
Amin (u,,a) 2 G (2) (M)())) (x) x=|) 2 G C2(Ciro + +e. There-
fore, we can choose To and small enough (but away from zero) such that Amin (A x1) 2 G/2, e.g., is the minimum of G/(4C2) and the original in the tubular neighborhood definition, and ro G/(.C.C2). This way, my can be lower bounded by Gr
B.3 PROOFS FOR SECTION
The results in Appendices B.I and B.3 consider only points in (M( Therefore, to use the results,
we need first show that the density outside the tubular neighborhood becomes negligible as a + 0. In the following two lemmas, we will show the concentration of the density for po and exp(-fo).
Lemma B.3. Assume Assumptions 2.1 and 2.2 holds. We have that limg fZETMate Po (z)dr 1.
19

---

Published as a conference paper at ICLR 2026
Proof. We have that
J.CRY (Tm(()
Po Po(d)dr
I / JIEM (2T02)a/2 exp
(_11z- [110-0((u)]) (
Pdata (u)dudr
20²
-INEM (u)ta(u) I JEERE/TMLte) (2To2)d0/2 exp
|| (1))13
drdu
20²
1 Paata (ua
f [n- (2nr22ya/2 exp
|| {12))12)
drdu,
2r
where the exchange of the integral is justified by Tonelli's theorem with the non-negativity of the integrand. The last inequality holds since any point in R /TM((c on the manifold. Now the inner integral is the integral of a Gaussian density with distance to the Let be standard Gaussian random
least c away from any point
origin at least c. will decay exponentially fast as variable of dimension d, and then the above integral is equivalent to
J. (u) [ 2 2) du = PAP((111
(1 2 )
The RHS can be shown to decay exponentially fast by the Gaussian concentrations. 2.7 and 2.2 holds. Further assume that
Lemma B.4, Assume Assumptions
V (z) + log (o-
sup zek
have that
lim o (EE((T(()
exp(-fo(z))d = 0.
Proof. For x & TM((), the points are at least away from the manifold. Therefore, have that
p=(x)= < f
1
1
2
JEM (2mo²)d/2 exp
2x Pdata (2T02)a/2
exp
2x
Pdata is density function. Therefore, have that
1
20 + +0
2
exp(-fo(z)) (2no?)af2 exp
There exists do. such that for all < do. the o2) term is upper bounded by e² 2/442.. Then we have that
JEEKYTAtO exp(-fo(z))dz & Vol(K) (2xo2)d/2 exp (-
e²
4g
The RHS goes to zero as + 0 as Pdata is bounded.
Now we are ready to prove our main theorems.
Proof of Theorem 4.1
First, since both fo and log are C functions on K, we have the that L norm their gradients is the same as the supremum. First we will show that for any 7) -2,
sup (x) Vlogpo o(o") as 0.
*EK
implies that
supl fo + log (x) = o(o") - 0.
=EK
20

---

Published as a conference paper at ICLR 2026
Given assumption, for any two points 2, y E K, there exists finite length path, say Ya.y (+) [0, 1] - K with and ||>l|| being upper bounded uniformly. Consider an arbitrary point zo E K, then we have
Ax(x) = - (a(x) log Po (2:
=ooox - logpo(zo) + L. ((-Vfo(((()) v log (o((t)))) (()ddt
-v0( + )z(2,0);
where sup,lg(z, o) is o(o")) uniformly for e according to the assumption. Further, have that
/ exp(-fo(x)) = f JeEK Po exp((x
f DEK
(xp(((o))) + g(r, o))da,
which then imply that
/ JEK
(a() 2 log
exp(-fo(z))dz llog
Po (o(z)dc-suplofe
suplg(z o),,
2EK
øek
and
(a())) S log
exp(-fo(I))de - log
Po (x)
zEK
zEK
æek
The first two terms on the right-hand side is o(1) as o
+ 0 as our assumption about fa and
SK Po Po (x)dx 2 (Ty Po (z)dx 1 according to Lemmal B.3 Thus, (xo) is o(o"). Therefore,
(x) is o(a") uniformly for x E K. Further we can apply Lemma B.4 to conclude that the density of exp(-fo) concentrates in (M(() 0.
Then, we can prove the first conclusion that the support is on the manifold. By the expansion of log Po Theorem B.2 we have that
fo(a) = - P
Then we can apply Lemma B.2 with fo(z) = o2) 0² and 17 = 32/2 to conclude the claim. To prove that the limiting distribution is Pdata the manifold. we have
1
fa(x) 202 ||. PMM(x)11 log Pdata (a (P=))))
log | |
d log( log(2TTO²) + o(1).
Then we can apply Theorem B.1 Then the fo becomes the distance function (changed local coordinates), and fi is log Pdtta + log / [L(w), 2(u, ())r)) In addition, note that for 0.
(M), ((,, r)) dM(u) and therefore, we recover Pdata- The (d n) log(2no²) term
is simply constant and does not affect the result after normalization. One can replace fo with fo + d=n log(2mo²) and then apply Theorem B.1 and this does not change the distribution after normalization.
What remains is to ensure Assumption B. holds, especially the second condition, i.e., to ensure that the Hessian of [ r) 22u))l² /2 w.r.t r is uniformly bounded away from zero. We can write ((u, r) as d(u) + (W(r,, where (u)) is the normal vector field the manifold M at point
(u) Wey! 1939). We have that
| (,, r) 22))l
(4,u)))
-N(u)
dr
2
dr
21

---

Published as a conference paper at ICLR 2026
since the columns of (u) are orthonormal. Therefore, the Hessian of || r) +buu)1 w.r.t r is simply the identity matrix, which satisfies the assumption.
To construct s(o, x) such that the limiting distribution is arbitrarily, say 4t we let s(a,, x) being the gradient of
2x | PM log + (P= log / d1).
The difference between fo and log Pa is then S2(1).
B.4 MANIFOLD WKB ANALYSIS OF THE STATIONARY DISTRIBUTION
A key difference between our theorem in Section 5 and the results in Section that, in the for- mer, the density does not admit an explicit form. When s(x, o) gradient field, a closed-form expression for the density is readily available; however, this property not guaranteed for most parameterized models, such as neural networks. We therefore resort to the WKB approximation to approximate the stationary distribution. Similarly to Appendix B. I we first present a general framework and then apply it to our specific setting. We will show that SDE with the following form admits a stationary distribution of the form Equation (IT). Interested readers may refer to Bouchet & Reygner 20166; Bonnemain & Ullmo (0199) for more details on WKB applied on Fokker-Pland' equation.
We consider the following SDE:
dz,- = by(2z) + 20dW,, with -
or the following SDE with the same stationary distribution,
bo(xx)
dx, 0
dt V2dW..
(13)
We assume that b(x, 8) is uniformly (0) in (c)() as 0 + O. Also, we have arg min fu(x) M. This framework is general enough cover the that in these two cases, the function fo is the distance function to the manifold, and 0 will be chosen differently in different cases. We make the following assumptions about the SDE.
of Theorems 5.2 and 6.1] We will later
Let (T) be the stationary distribution of the SDE Equation (13 First we assume the WKB ansatz: Assumption B.2 (Local WKB ansatz). We assume that lime- STante) To (e(ddd and that te admits local WKB form within compact set (e):);
V(x)
Te (x)
OC exp
co(z) with co(x) = co(z) + 0(x,
where co E (T)) is positive, and co in (TTM(())) We further assume that V E (T)()()) admits a unique solution.
The normalization constant can be explicitly written
/
V(x) (-V 0
J to(x)dr/
(xETMLE) exp
co co(r)dr,
since we have for TM((),
To = Te(x) ( (T)(() = Tz( E (T
co(x) (-V(e) V(x)
V((x)
no (T((m())).
SxETAIe) co(x) exp (-V
dr
Our goal would be to solve for V(x) and cu(x) with the Fokker Planck equation. Once solved, to study the limit of Te, we can use results Appendix B.I as
Te( x exp V(x) 9 log co(x) + o())
22

---

Published as a conference paper at ICLR 2026
Theorem B.3. Consider the SDE described in Equation 13 we have that
Assume Assumption
B.2 holds. Then
V(x) - fo(z), co(x) = C Cexp(=A(())
for some constant C.
Proof. By Fokker-Plance equation for the stationary distribution, we have that
0 = div (bb + @TT((T)
dx
By plugging in the WKB ansatz, we have that
( bu,
dca
10V
Oco θV
-iiv(bo)co
dr
8x
Go
0 Tr
0r
2
Ox
dx
(14)
TOV 2x
0V
Tr
Co
+
= 0.
dr
Next by the method of WKB, will equate different orders of 0 in the above equation solve for (xx and (z),,, starting from the lowest order f-1 is easier to show function is constant, therefore, for co, will define (x) exp(Si (z)))c(<x) and try show that it is constant.
Order 0 1 In this order, we have that
0V 00
0V 2 Ox
0x dr
This corresponds the Hamilton- Jacobi equation typically appears in the WKB approximation. The equation gives the solution for V(z) as V(x) = fo(r). Plugging this solution into Equation we can get
f
fi
06
dcp
-00 0f + b,
1
00 On.2
Tr
0
0x22
dr
Cg
by
8x
dx
Ox
) + O
Oca
fo
= 0.
Ix
We will work with this equation for equating the higher orders.
Order 00 In this order, we have that
05 0r
0 Ör
Ocg afo
en
0.
Ör
Ör
This is known as the transport equation (Bouchet & Reygner 2016). It shows how co changes along the gradient of fo- Next, we express the equation in terms of Co:
Ocn
0fo dr
0.
(15)
This implics that along the gradient of fo, oo is constant. Since the manifold M consists the min- imizers of fo, for any point in K, the value of (x( is the same the value at the corresponding minimizer y on M following the gradient flow of fo. Formally, we have
Co (x) = Co (+00)))
where (t) follows (t)ddt (0)( = x. Therefore, we see that to solve for do we need to know the value of on M. We find that the next order equation will help us solve for & on M.
(t) with (0) given the initial condition
23

---

Published as a conference paper at ICLR 2026
Order 0¹ In this order, if directly find all terms in Equation (4 that are of order 0¹, will find that includes higher order terms, c.g., C(z, 0). However, since we only care about the solution on M,, evaluate the equation M and interestingly find that it does not include such higher order terms, as erucially the factor 0fo/Or becomes 0) at M. Specifically, for x E M, we have that
f 0x
10 Oco
8oco
Tr
+
+ Tr
0.
0x
dx
Ox
Replacing co with čo
we have that
Co
000
0f
Tr
= 0.
2x
dx
dx
Our goal here would be to solve for 20 on M, and apparently it would be helpful to convert the equation the local coordinates and establish PDE for the manifold chart coordinate
Local coordinates We convert the above order f equation about ?o to the local coordinates = = (u,r) and get that for 0, i.e., points on M,
[J] div= [J]c]] 8eg
1
1-10f
0fi
0z
Oz
Ōz
(16)
[J] (( div, (IJ
1
0ān
8220 0=2
0h
Tr
||
G-1
Oz
Oz
Oz
where G = JTJ and the divergence of matrix is understood as the divergence of the column vectors. Note that cannot simply conclude from the above equation that èg is constant, by say, the strong maximum principle, since the gradients of ču include not only the manifold chart coordinate but also the normal coordinate Therefore, we have to further derive PDE about and any gradients čn w.r.t. r should be replaced by known functions. Fortunately those gradients can be solved by the equation we obtain order 6".
First, let us derive from Equation (66) a PDE about u!:
Lemma B.S. From Equation (16). have that for O.
Amēo(U) - dep 0u
0fi
1
J
0čo
/ 100 0.
8o Ör
9
Tr
0, (17)
du
Vigl
dr
Ör Ör
where AM is the Laplace-Belle Beltrami operator on M.
Proof. Let the index i, when showing at à be derivatives w.r.t. the i or j-th coordinate of and let p, be the derivatives w.r.t. r respectively. Any index variable that is not explicitly defined is understood to be summed over. From Equation (16). by carefully expanding the divergence, the inner product term becomes
(div (IJ(G-+), Vio)
-igla, [9- Occo + [o "I 11 Vg vlgl "9 Opdp.l, 0,00 + 0p 110
[
For the trace term, we have
Tr [ r= vlrl ( 0g + Villop.po.
Now look Equation (7). From the definition of haplace-Beltran operator, have
AmCo(u)
0 (vlg)
[ 1.. 0,āb)
Vlg
0: 0.vlo Igl
[ " 0,00 + a (g- i5 0,00 - -1 Oz
[
Since G-1 evaluated at r 0 is
0 7
the term
-(
06p 0ズ
G
0fi Equation (16) matches
8z
1
0f
06g お
0f 5t
in Equation
Now compare the terms of Equation 17 and
Equation 16 the only remaining term is
[ i.k Opdp, 1,= 0,ã0,
24

---

Published as a conference paper at ICLR 2026
which we will prove We will show that Ep Opdp.k
r=0
Since the columns of M are orthonormal, have for any p. Ej (Vi.p) = 1. Taking derivative for
both sides to Uj. we have for any p.j. Nep O. We also have by definition that for any p,
| -ii
Using the above two results, we have for any j.
Opdps = E O (Wim) = ENooO,Ni,p
p
From Equation 17 see that it contains gradients of čg w.t.t which we will solve by the order
8 equation.
Lemma B.6. From Equation
15 we have that the manifold M.
Icg
8eo h(w). Du
(u. 0))
(u, 0) = 0, and
Tr
(u. 0)
Or
where hu)) does not contain the unknown function Co-
Proof. Since we care about the evaluation of the equation on M,, we start by changing the coordi- nates to the local coordinates = (u, r) from Equation (15) to get that
Peo 0
0z = 0.
0z
Next, compute the gradient w.r.t.
( @veo [G-
T
8 Sfo 0z2
82 fo 0.2
0ep +
10
0āg
G
+
G
= 0,
(18)
Oz
Oz
Oz
Oz
dz
where is the Kronecker product. When we evaluate this equation
the factor 0 Or
becomes O, - (u.
g
0
I
and
8 052
(u,
0
0 fo (u, 0)
Then have
83 So
Deo
uu, 0)
(u, 0) 0.
0
Since 2 u, 0) is full-rank, we have that
86u (u, 0) 0.
3r
Next, we compute gradient again for Equation have the i, j-th element the matrix
18 and evaluate at r = 0. Ignoring 0fo/O= /0 which
fo
182 fo
822
8 dz,0zuZ2z
G-1 Gão
G
G
0z2
dz2
0z
dz2
Oz
i.
is
(19)
8 fo OG 1 0eg + Ozk dz Oz,Oz,
de,
OG 82 fu 0,
+
0z,0zz Øzj Ozp
where 06./3r is O. The first two terms have nice structure when evaluated at r = 0, as
00
G 02 fo 0=2
0
8 0² fa Jyer Ar
82:
0
0=2
0
8 8'
and
022
022
8 ta &a Or
a 0a
8r
Dr
Dr
0
We then multiply Equation (19 by matrix
82
-1
from the left, and get
0
i Brd
0 fo
+
8 &
8 G
+ remaining terms = 0.
By
Since 0ō./Or is O, the element the remaining terms all have one and only one factor of 0ē,/ /du for some Taking the trace of the above equation, and have proved the second statement.
25

---

Published as a conference paper at ICLR 2026
Now we plug in Lemma B.6 Lemma B.5 and obtain a PDE about co(- 0) whose second order derivatives are the Laplace-Beo Beltrami operator, and the zero-th order term, i.c., the term that includes the function value Co 0). is O. Therefore,
conclude by strong maximum principle (Gilbarg et al. 1977. Theorem 3.5) that Co(: 0) is a constant. According to the equation at order 0", we obtain that do off-manifold is the same constant.
B.5 PROOF FOR SECTION 5
We will first prove Theorem 5.1 which follows similar proof technique as Theorem 4.1 and then turn the harder case of Theorem 5.2
Theo- rem 1.1 with = o --=a_ In this case, fo(x) = | - (M) fi = 0 and all other terms are asymptotically small compared 22-a According the proof of Theorem 4.1 the determinant of the Hessian of fo in the normal direction the same for all u, therefore, we recover the uniform distribution on the manifold.
Proof of Theorem 5.7 The proof follows the same Theorem 4.1 except that now
The only thing remains to verify is to ensure
SRIVTALO exp(-o 0.
lim (c)(((()
To (x)dd
lim a a=0
SEr
Since we have lim S& ( + 1, we only need to consider within K. For the numerator, we can do similarly Lemma B.4 to obtain
0a
2
SKITUte exp(-o fo & Vol(K)
402-a + t
(2mo2)d/2
exp
where 2 a 0 and B 0. There exists dg. such that for all ou, the o(o term is upper bounded by e² 2/822-a Then we have the numerator upper bounded by
1
Vol(K)
(2TO2)a/2
exp
802-a
For the denominator, it is lower bounded by
JraCeep (2mo2)aa12
| {1x)12
+ o (patis) dx
exp
202-a
Sratere (2/102y1/2
0a
802-a + (o(tt)) dx.
?
exp
There exists d1 such that for all a 01, the 0(0 term is lower bounded by e² 22/1602-o Then the denominator is lower bounded by
2
Vol(TM(e/2))
(2m02)a/2
exp
1602-a
Therefore, the ratio is upper bounded by
Vol(K) (ol(TM((c/2)
2
exp
1602-a
which goes to zero O.
Next, for Theorem 5.2
we use results in Appendix B.4 to find an approximate stationary distribution
of the SDEs considered in Section and then use results in Appendix B.I prove the main theorem.
Proof of Theorem 5.2
The SDE we consider can be also written as
dX, = 0 0²s(X,,0) 0)
02-a + v 2aW,
26

---

Published as a conference paper at ICLR 2026
Therefore, want to apply Theorem under our assumption Theorem 5.2 we can write
B.3 with 8 0²-a and by o's(Xe,0)) We assert that
Ø||= 12 + (
bo(x)
Ox
meaning that fo 2 (M and fi = O. We will discuss the proof of this later. If we have the above, by Theorem B.3 the stationary distribution in (M(() is given by
|| - PM( +01))
To (3) x exp
22-a
where the error in the prefactor equivalent the error in the exponent. The remaining proof follows the same as Theorem 5.1
remains prove the assertion about by. A sufficient condition is that
0| PM 0(1).
sup (C
V Po (x)
(20)
d²
Fx
Because if Equation
20
holds, we have uniformly for any a Tm((),
be (x()) 0|| P. (2 12)
2 (5( 0| P.M(2))
Ox
0| (x)
o² d's(5, 0) - Voggoo + o (z)
dr
0| P M (x:)11///2
$ l 0) - V log (x)
0 (v)
8x
o(02+0) + 000²² o(02-9),
where the last inequality holds because a > max(-330) . In the theorem, we assumed L (TM(+)) norm, which is the same SuPzeTM((c) since s(x,,) and V log Po (x) continuous.
Therefore, it remains to prove Equation 20)). We will prove for the case of VE, and the case of VP holds with similar argument. The gradient of the distance function can be written as:
o| PM Or
(1-(OPM(()) Or
(xr PM(x)) -PM(()))
where the last equality holds because x PM(()) is orthogonal to the manifold and the image of a (an) is in the tangent space of the manifold (Leobacher & Steinicke 2021). Then note that
Oz
Vpo(z) SM u. o2 T (u) SM Na; u, 02 Pasta (u)du
f(u) du
V Vlggoo(
(x)
For the denominator, follow the same as in the proof of Theorem
B.2 to obtain that
Pe (x) = exp (- || -PM(2)1 (2ma²)
(n d)/2
Pdata c (PM(x))
(1+0(g)),
202
((
since Equation 2 holds and Pasta is uniformly bounded away from zero. We could do the same for the numerator, however, the ((a) error enough here. Intuitively, the numerator would be
- (2mra (n- d3/2
Paata ( ( (P((m)()))
(PM(x)
+o(1o)).
exp
20²
o²
M (PM((z)).
27

---

Published as a conference paper at ICLR 2026
Apparently, the error term is not enough to prove Equation 20
Therefore, we turn to stronger Laplace's 's method result that has an error term of (o²), i.., the h())
term in Corollary B. could be improved to O()) instead of O(v0). However, such result should have the cost of requiring the functiin (as the notation use Corollary B.1 be C+ and to be C², stronger condition Formally, we have that
o Q² log Po + (a PM(x))
Na; o² P)aatal (u) (T(u) PM)())
SM u, 0² PPPaats
and we want to prove its L (TM(+)) norm is O(1). For any x E TM(() and {ol ||l| 1]. we have that
T. (o2Vlogpale + (x PMM()))
JM Ju; [paata(u)"" ($(u) (m)) du
SM u, 0² Paaaaa
JMNNa;; D (psata()) (v (8( (M)) + 1) du
1.
Sm N(x;; 0² Pudaaauuudu
The last step where we add is a simple trick because the Laplace's method we will use does not allow the prefactor to be 0 at the minimizer. Next, we multiply the numerator and de- nominator by exp [-- [
/ // ((PM())))))
so that their limit does not diminishing to For the numerator, we apply Majerski (2055 Theorem 2.4) with their
2n
(2
1/02, t u, f(u) = || - (u)) a 2 (f(u) is O since ((u) is C*), Bs can be selected the same as in the proof of Theorem B.2, g(u) Pdata(u)) (1) (0(u) PM)()) + 1) (g(u) is C2 since The upper boundedness of the constants can be easily verified by compactness and can show that they uniform for and u. Crucially, (xr)) is uniformly lower bounded. The lower boundedness of Amin can be reasoned in the same way in the proof of Theorem
Puata(u) C²), and the minimizer is (P
(P (z))) = Pdata($
B.2 Therefore, have
1 + O(o²)
T. (o log (x) + (x PM(()))
O(o²) 1= 0
Since the bound is uniformly for x and ||0|| 1, we have that
sup oo'Vlogpata + (a PM(())] =ETM(e)
sup sup T (o'Vlogp. + PPM(x)) 0(03), [|]]]
which proves Equation (20).
B.6 PROOF FOR SECTION
Proof of Theorem 6.7 The proof follows the same Theorem 5.2
except that now we have fi
when applying Theorem B.3 and Theorem B.I
C EXPERIMENTAL DETAILS AND FURTHER EXPERIMENTS
C.1 NUMERICAL SIMULATIONS ON ELLIPSE
Loss function. In our experiments, we train the score network to predict
8(z,0) := o² (x,
Weaker condition such as 11.1 is also possible, see Majerski 2015 Theorem 2.4).
28

---

Published as a conference paper at ICLR 2026
instead of s(z, o) directly. This formulation is more stable across noise levels, since the leading term the score expansion is order 1/0², making $(x, o) an O(1) target. With this choice, the training objective becomes
o² s(2, o.) =-*(w) 2
Eu-paau E ((
0²
Eur-pasta (2-N(((u)
[|
{())13
The score function s is parameterized by a neural network consisting of four transformer blocks, each with hidden dimension 128.
Data and noise. Training data is generated from a von Mises distribution with parameter & = 1. The injected Gaussian noise variance o² is sampled from range E [0.01,50] 50].
Optimization. We use A with weight decay x 10 and global gradient clipping at norm
1.0. The initial learning rate is 3 10-3, decayed scshne-s over 10 steps down to 1% of its initial value, after which training continues with a constant learning rate of 4 x 10 4 The batch size is set 1024.
Sampling. For sampling, we run Langevin dynamics
dx; $(I,, miin) dt + v 20t min dWe. with min 0.01. This process has the same stationary distribution as
dx. = s(Et, mmin) ) dl + ddW,
For the TS Langevin dynamics, the diffusion coefficient is v 20 instead of 22 min' We employ the Euler-Maruyama scheme with step size of 0.1, running 10,000 steps with 10,000 runs.
C.2 IMAGE GENERATION WITH DIFFUSION MODELS
Algorithm details. We use a pre-trained Stable Diffusion 1.5 model with a DDPM sampler correcor (PC) scheme. The pre-trained network provides denoiser c(2, t, y), and the corresponding classifier guidance (CFG) score at time is
(x v) v, log Px(x) + w (vz Po Pe (x: v) V log
uncomditional
conditional increment
te(x,t,0) + w(e(z, t, (z,,,,))]
where y is the conditioning input (prompt embedding). w the guidance scale, a, vI ar, and āg is as Ho et al. (2020). Our tempered-score framework applies to this PC sampler by modifying only the unconditional component while leaving the guided increment unchanged:
1 of e(x, t, + we((x, t, y) - e(x, 1, (.0)))
(x y)
which is consistent with Equation (8) Let {t,) denote the discrete reverse-time schedule. After each DDPM predictor update at level 4, perform Noom. corrector steps of Langevin dynamics with the tempered score:
Fb+1 = + 6, 5t, (Ik.V) + 20, Ek.
E ~ N0,,
where the step size 8, follows Song al. 202 Algorithm 5). After the entire reverse process, apply an additional Recor. deterministic projection steps using the unconditional score (no guidance, no noise) to further project onto the data manifold:
dx,
log Pto (x+)d7.
We use the same number of projection steps for both the original PC baseline and our TS to ensure fair comparison.
29

---

Published as a conference paper at ICLR 2026
0.8
Pasts
0.8
0.6
0.6
Density
Density
0.4
0.4
0.2
0.2
0.0
0.0
3r
3n
2
Angle (rad)
Angle (rad)
(a) Distribution generated by Diffusion Model
(b) Distribution generated by TS-1
Figure 4: Comparison of distributions generated with VE diffusion model versus our TS Langevin dynamics Equation with a =
Hyperparameter setting. We adopt the default configuration of Stable Diffusion 1.5 httpps: / //huggingface..c cttpeeeooo///////a/aeeeeee stable diffusion v1- 201-5/stable di wwfffsiiiiooo (ffffussiovi-5))
Un- less otherwise noted, all results in Section 7.2 use guidance scale w = 7.5 and 30 inference steps. For the best-results reported in Table I perform grid search over the number of corrector steps in (5, 10, 15, 20, 30} and a E {0.1, 0.5, 1.0, 1.5). The original PC baseline is tuned over the same set of numbers of corrector step for fairness. For CLIP evaluations, generate 512 images per setting and downscale each to 256 x 256 before computing the scores.
C.3 CONTROLLED EXPERIMENT WITH GROUND TRUTH SCORES
To empirically validate the rate separation results in Theorems 4. 5.T we designed a controlled experiment using synthetic data where the manifold and ground truth scores known analytically.
We consider the unit circle manifold M = {ar E IR? | ||ж||| = 1} with a Von Mises distribution Pdata (0) exp(x cos(0- 0o)). where we used к 4 and 0n This setup allows to compute the analytic ground truth score s" (xx r.. We then inject a deterministic error field e(x) into the true score:
8(z,0) s" s" o) e(x),
with e(x) -v
The magnitude of this error term e(x) is 0(1) with respect to a.
We compare the performance of the standard reverse diffusion process against our proposed TS Langevin dynamics using this corrupted score s. shown Figure 4 the standard reverse diffu- sion process using & produces samples that deviate significantly from the ground truth Pdata con- firming that (1) score errors are sufficient to corrupt distributional recovery, while the TS Langevin dynamics with a robustly recovers the uniform distribution on the circle.
C.4 SENSITIVITY ANALYSIS OF HYPERPARAMETER @
To evaluate the sensitivity of the hyperparameter a, we performed an ablation study using the Stable Diffusion 1.5 model, under the same setting as in Section 7.2 of our paper. We tested
E (0, 0.1, 0.5, 1.0, 1.5) across three prompt categories, with the number of corrector steps fixed 10 and 20. Note that a = 0 corresponds to the standard predictor-correcto baseline.
As shown in Tables 3 and our method yields consistent improvements over the baseline (a 0) once a is sufficiently large (a 2 0.5), demonstrating that the performance gains are robust and not limited to narrow hyperparameter setting. The performance is particularly stable for E [1.0, 1.5], which aligns well with our theoretical framework (Theorems 5.1 and convergence to the uniform distribution for any 2. While utilized simplicity, these results suggest that slightly more aggressive tempering (a 1.5) can provide further gains in diversity and quality.
and 5.2) that guarantees in Table 2 for
30

---

Published as a conference paper at ICLR 2026
ax 0
0.1
=
0.5
a == 1.0
a: = 1.5
Prompt
P-sim(
I-sim, P-sim I-sim P-sim I-sim P-sim I-sim P-sim I-sim
Architecture
27.13
81.81 27.12 81.73 27.14 81.67 27.27 81.57 27.32 81.52
Furniture
29.30
81.24
29.32 81.37 29.33 81.06 29.58 80.95 30.16 80.76
Car
26.30
87.57 26.30 87.58 26.31 87.44 26.37 87.42 26.50 87.34
Table 3: Ablation of for 10 corrector steps.
a
Q = 0.1 a = 0.5
a a= 1.0 a 1.5
Prompt
P-sim/
I-sim| P-sim I-sim P-sim I-sim P-sim I-sim P-sim I-sim
Architecture
26.87
81,60 26.85 81.56 26.97 81.49 27.06 80.97 27.10 81.13
Furniture
28.98
81.72
28.99 81.65 29.07 81.40 29.52 81.15 30.20 81.39
Car
26.26
88.06 26.26 88.09 26.25 87.95 26.28 88.07 26.62 87.70
Table 4: Ablation a 20 corrector steps.
D CONVERGENCE OF TS LANGEVIN
In this section, deduce the mixing time analysis, i.c. the convergence analysis for a stochastic process, of the TS Langevin to the estimation of the Poincaré constant. The goal is to show that TS Langevin is not necessarily slower- and in fact be significantly faster-thar than the standard Langevin dynamics in terms of mixing time. To carry out such an analysis, we assume that the score network is gradient field, i.e. s(;oa) V log po for some parameterized density function. WLOG, we assume pe is normalized as the normalizing factor does not affect the velocity field 8.
D.1 CONVERGENCE ANALYSIS OF LANGEVIN DYNAMICS USING FUNCTIONAL INEQUALITY
To analyze the convergence of Langevin dynamics, it is customary to use functional inequality satisfied by the invariant measure Poe of the Langevin dynamics (Here, ps denotes the density of the process at time I, and Pre. its stationary distribution. This notation differs from pa. and the distinction should be clear from context). In this response, we focus on the Poincaré inequality (PI): We say Pee satisfies PI(CPI) for all f (poo) (Sobolev space weighted by Pso).
1 JU- [dp=)) dp S Opt
dpcc. |dpo
where we call Cp1 0 is the Poincaré constant.
Consider the overdamped Langevin dynamics with potential Un : Rd - IR:
dx((t) (X))(( (2)
and let pt La)(X Under mild assumptions, Poc x exp(-U,) is the unique invariance measure of the above dynamics. p & exp(-U.) satisfies PI(Crr), then
(pr. Poo) S e Critt 2
where x2 denotes the diveerege,, In particular, to ensure x" (pt: Poo) S 17 for some target accu- racy 17 0, it suffices to take i = 0 c. log )). Thus, the larger the Poincaré constant, the faster the convergence.
D.2 ANALYZING THE EFFECT OF DRIFT SCALING TO THE POINCARÉ CONSTANT.
Under the assumptions of our paper, the comparison between the mixing of standard Langevin and TS Langevin therefore reduces comparing their Poincaré constants. We illustrate how drift scaling affects the Poincaré constant in the simple case where the data manifold is the unit circle:
M f Rdd |||||| = 1}.
31

---

Published as a conference paper at ICLR 2026
In this case, the squared distance function can be computed in closed form:
1 d(z) 2 dist (x, M)
1 [[12] -))..
11
2
Following section 5 of our paper, we assume the score error is O for some - 0. Recall that we assume the learned score is gradient field, i.e. s(. = V log Po- Let us further suppose that the problem dimension learned score (()))) has the following form
2, i.e. a R22 and the density function po (corresponding the
llog Pe
02 d(x)) d(x) + o )(z), where ((x) (1=11 1)?
where 21 denotes the first coordinate of x. Clearly, this function satisfies all requirement in our paper. Crucially, such a construction ensures that the score error is O(r²).
Standard Langevin dynamics. We restate the standard Langevin dynamics for the ease of refer- ence:
(x Po
po (X())) + 22dW((). (t).
Without temperature scaling, the error function ((x) introduces two separated modes (-1, 0) and (+1,0). For such a multimodal measure, classical ciple results imply that the Poincaré constant can scale as
Kramers law or the large deviation prin-
CHP O(exp(-o"))
Consequently, the mixing time of the original Langevin dynamics can become exponentially large as
+ O.
TS Langevin.
We restate the standard Langevin dynamics for the ease of reference:
dx = log (X (t) + (2) = V log pă (X(t)) + /2)
Under mild conditions, the unique equilibrium measure is po" assumptions and a > -3, that Poincaré constant, denoted as Cpi. is uniformly bounded away from zero, independent of a for sufficiently small a. Here we summarize the main steps:
show that, under our standing
Recall the Holley-Stroock perturbation principle (Holley & Stroock 1987): Let U and Ú be two potential functions defined on Rºd Suppose that the corresponding Gibbs measures Pso x exp(-U) and Poo x cxp(-() satisfy Poincaré inequality with constants Cpi and Čpi respectively. One has
ČPI
where the oscillation between U and Ü defined as
osc(U Ü, U) == sup (Ü -U) inf (Ū -U). =ER
Rd
Since 2 > -8, a Dolley-Stroode perturbation argument implies that the PI constant of pç is comparable (up to fixed factor) to that of the measure μa x exp(-d(z)/o²> 2==0) for small ar. We denote the Poincaré constant this ideal potential as Cri..
A short proof for the above statement: Pick
Ü = log Pă and U = d(z)/o2-a.
One can bound osc(Ü, U) using Theorem 3.1 of our submission. Apply the above principle to yield
CTS (xp(-O(aat))))e 2 exp(-1))Cdat
for sufficiently small σ.
We note that the distance function d(x)) is locally Polyak Lojasiewicz, and hence one can expect the recent results (Gong al.) (024) on the ceaperrature-idepeedeee Poincaré constant for locally log-PL measure can be applied. The only requirement in (Gong et al. (204) that is not satisfied by pa that it is not 2 0.
32

---

Published as a conference paper at ICLR 2026
We therefore introduce smoothed potential
Va(x) = | 1 + 2 2
||
+ +
and apply Holley-Stroock again compare the Pl constant of Pd with that of jr x Choosing c = 0 we can verify that Vo satisfies the assumptions of the log-PL result (Gong et al. 2024), which implies that the corresponding Poincaré constant (de- noted as Oppioomn) is independent of
A short proof bound C'pirt with Opmuoth, Pick
(xx) d(z)/o²~n and U(x) Va(z)/o²~a
To bound osc(U, U) notice that
[d(x)) - Va(=:)| |||=||| - | | + v | Sc= :0=0
Apply the perturbation principle to yield
Cyllett 2 exp(-O(o))CP1 Camooth exx((11)Cppomm
for sufficiently small a.
Combining these comparisons shows that the Poincaré constant pg". i.e. CPP. differs from Cff and Cppooth only by a constant factor.
In this point, we discuss on proving Oppooth is independent of a. First, we note that directly apply the result in (Gong et 2024) the potential V. already yields that the Poincaré constant Opmnoth is of order [(c)): It is easy to verify the assumptions in (Gong et al. 2024). ). i.e. local PL, non-saddle point, growth condition beyond compact set, and the boundedness of AV i.c. the absolution value of the Laplacian of V. within a compact set. We can hence directly use Theorem 2 (Gong et al. 2024). However, the quantity |&Ve is of order ! in this vanilla analysis and hence с we would yield that the Poincaré constant Cpr is order 2((c). It turns out that by exploiting the particular structure of V.. we further improve this result: We note that | Vel does need to hold in the neighborhood of the local maximum set and their analysis still goes through. We hence pick this neighborhood as hall centered around the local maximum
0 with radius 1. One can see that outside of this neighborhood but within compact set, V is bounded by o-independent constant. Then OpT Owmooth could be proved to be ((1). We highlight that even the vanilla (ec) bound already establishes the exponential difference between CPS (lower bounded by a polynomial in σ) and CLP (upper bounded by exponential of (l/poly(ar), Of course, the 22(1) one leads to even bigger separation.
Putting these estimates together, we that, at least in this unit-circle example, TS Langevin mixes strictly faster than the original Langevin dynamics in the small-o regime. This illustrates that temperature-scalle Langevin not necessarily slower-and can fact be significantly faster-than the standard Langevin dynamics in terms of mixing time.
D.3 A REFINED ANALYSIS FOR Cpmnooth
Directly applying the result in GGong et al. 2024). we have that C'mooth (22 for sufficiently small a. In this subsection, we show that this be improved to Cssmonth 1(1) with small modification to the analysis of the Lyapumov function in (Gong et al. 2024)..
Proposition D.I. Menz & Schlichting 2014 Theorem 3.8) Consider the Langevin dynamics
ddx(t) (VV(X + 2edW
Define the associated infinitesimal generator L
L = -V - +EA
(21)
A function W Rd + o0) is Lyapunov function for L if there exists U S Rd b 0. 0, such that
Ve Rd
-1 -oW(2) blu(x).
(22)
33

---

Published as a conference paper at ICLR 2026
Given the existence of such Lyapunov function W. if one further has that the truncated Gibbs measure P., satisfies PI with constant Pl,, > 0, the Gibbs measure Pe satisfies PI with constant
Pc Z b+ Pe,
PGU-
(23)
In the context of this section, = 0²-a and V = Ve. In (Gong et al. 2024), the Lyapunov function
chosen be W exp(
and eq.
(22 can simplified
2c
CW cW
AV
VV
- S-o+blu:
(24)
2e
4
To establish the above inequality, Gong al. (2024) partition the whole domain IR into multiple
disjoint parts: (1) U, (2) neighhorhood of the global minimum but outside of U, (3) neighhorhoods of local maximum, (4) beyond a compact set that contains critical points, and (5) the rest. We discuss treatment of each subdomain.
On (I), we follow the choice of U in GGog et al. 2024 so the local Poincaré inequality there directly holds.
On (2), i.e. in the neighborhood of the global minimum (note that under the assumptions of (Gong et al.) (024), all local minima are global minima), but outside of the neighborhood U, we follow the argument as (Gong al. 2024).
On (4), Beyond compact set that contains all the local minima and maximum, Ve above fulfilles the requirements of V in (Gong et al. 2024) and hence the argument directly carries over.
verify that
On (3), i.e. in neighborhood of the local maximum, since the Laplacian is already negative, one can directly obtain eq. (24). Note that we will pick this neighborhood to be the ball centered at 0 with radius 0.1 for Ve, denoted by ((0,0.1).
On (5), i.e. within the said compact set, but outside of the neighborhoods of the global minimum and local maximum, (Gong et al. 2024) requires the Laplacian to be bounded. We note that analysis in (Gong et al.] 2024) is a bit loose and they require the boundedness to hold on the whole compact set. However, there is no need to assume the boundedness of the Laplacian on [B, 0.1)
eq. (24) already established in (3).
Based on the above discussion, we notice that the global bound on the Laplacian of V. is only required within a compact set, but outside of (0, 0.1), which is hence a constant independent of We hence obtain the 2211 bound on the Poincaré constant.
34


Reviews:
[{"review_id": "aj2r1XZYlQ", "reviewer": "Reviewer_G9vr", "summary": "While score-based models such as diffusion models are typically thought as approximating the data distribution, this paper hypothesizes that---assuming the popular manifold hypothesis---their success arises from implicitly learning the manifold instead (\"geometry learning\").\nThe authors provide theoretical results that show that (1) recovering the data distribution (as the noise level $\\sigma \\to 0$) is *difficult* as it requires a strict $o(1)$ error on the learned scores, while (2) mere *concentration* on the data support (i.e., the manifold) can be achieved with a much larger score error of $o(\\sigma^{-2})$. However, in the latter case, it is shown that the learned distribution supported on the manifold can be arbitrary. To this end, the paper proposes a simple modification to Langevin-based samplers that draws from the *uniform distribution* on the manifold, which still only requires $o(\\sigma^{-2})$ score errors. Similarly, the authors show that in Bayesian inverse problems, sampling from the posterior induced by a prior that is uniform on the manifold also tolerates $o(\\sigma^{-2})$ score error, while $o(1)$ error must be assumed when picking the (Gaussian smoothed) data distribution as the prior.\nThe author's theoretical results are empirically validated using low-dimensional synthetic data, as well as pre-trained large-scale diffusion models.", "questions": "+ How sensible are the results in Section 7 to the choice of $\\alpha$? \n+ A possible relaxation of the $L^{\\infty}$ error assumption is mentioned in the limitations. Can you elaborate on the importance of this relaxation? How could the presented results be related to e.g. the denoising score matching loss (Fisher divergence) of trained models?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ **Motivation and Relevance.** The work is clearly motivated: Understanding learning dynamics in score-based models is highly relevant for contemporary generative modeling and Bayesian inference, but also for potential downstream tasks such as using the learned density for out-of-distribution detection etc.\n+ **Presentation.** The theoretical results are presented well. The Taylor expansion in Theorem 3.1 gives a good intuition for the results, while being less technical.\n+ **Novel, important theoretical results.** The rate separation results (Theorem 4.1) are novel and important for this line of research. Under standard assumptions, the results give new insight into why learning geometry information is \"easier\" than distribution learning in score-based models.\n+ **Uniform-on-manifold sampling.** Tampered Score (TS) Langevin dynamics is a surprisingly straightforward adaptation that comes with relatively strong theoretical guarantees regarding convergence to the uniform distribution on the manifold. Specifically, the guarantees also hold in the practical case where $s(\\cdot, \\sigma)$ is a non-conservative vector field (as in most diffusion models that directly output the score).", "weaknesses": "+ **Empirical Validation.** \n\t+ The empirical results are limited in scope, and do not directly support the main rate separation results in Theorem 4.1. The paper would benefit from such experiments (e.g., synthetic data with known manifold and ground truth scores, controlled injection of score error, systematic analysis on how manifold concentration and distribution recovery behaves). \n\t+ The experiment in Section 7.1 would benefit from a quantitative evaluation of the qualitative results in Figure 2. Moreover, higher dimensional synthetic experiments (with known manifolds) would shed light onto more practical settings, as score errors can be very different in 2D when compared to high-dimensional problems.\n\t+ The experiment in Section 7.2 seems weak as it shows only slight quantitative improvements on merely three prompts. No error bars are provided, and it is unclear if the results are significant. While in Table 2, $\\alpha=1$ is fixed, in Table 1, TS was tuned on both the number of correction steps *and* $\\alpha$ , while PC was only tuned on the former, which makes the tuning budget unbalanced.\n\t\n+ **Discretization Error.** All theoretical analysis assume continuous time models, while discretization error is disregarded. However, in practice, discretization error has an empirically large influence on the performance. Extending the theory in this direction would make it more practically applicable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JFQ4NrP4Gg", "reviewer": "Reviewer_LahT", "summary": "In this work, the authors studies a rate separation when score-based methods learn manifold data, namely that recovering the data manifold is easier than recovering the exactly underlying distribution on the manifold, where the difficulty is measured by the error tolerance for score approximation. The authors thus argues for a paradigm shift from distribution learning to geometry learning - more specifically, targeting uniform distributions on manifolds - and introduces the Tampered Score (TS) dynamics to generate distributions on manifolds that are close to uniform.", "questions": "See the \"Weaknesses\" section above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors' main hypothesis - that recovering manifolds is easier than recovering the exact distributions on manifolds for Langevin-type dynamics with approximate score functions - is novel to my knowledge. If fully validated on score-based generative models, it can potentially shape our understanding of how such models fundamentally work.\n\n2. The authors derive precise theorems to quantitatively characterize the hypothesis, enabled by a rigorous framework based on differential geometry and impressive mathematical techniques (not fully checked).\n\n3. Although the idea of tempering the drift term in Langevin dynamics is not entirely new, the proposal to use it for learning uniform distributions on sub-manifolds is novel to the best of my knowledge, and also as a modification to the corrector step in predictor-corrector sampling.\n\n4. The presentation is clear overall.", "weaknesses": "1. The analysis is performed towards the stationary distributions of the various score function, rather than the distributions obtained from the denoising dynamics (i.e. reverse ODE / SDE) as in score-based generative models. Hence it is unclear to what extent the theoretical insights derived for the former idealized setting also hold for the latter (even in the continuous-time setup). In the latter case, for example, even small deviations in the score function can cause the final distribution to be supported beyond the manifold.\n\n2. In particular, I wonder whether it tends to take longer for the TS dynamics to converge to the stationary distribution than the (untempered) Langevin dynamics. Intuitively I suspect this is the case since the drift is reduced with $\\alpha > 0$. As the theoretical analysis is concerned only with the stationary distributions, the results are unable to address questions regarding the rate of convergence, which are nevertheless quite relevant for practice.\n\n3. In the paragraph after Theorem 5.2, the authors claim that the TS scheme helps with recovering \"the uniform distribution on the data manifold from samples of $p_{data}$\", but I don't see a justification of this by the theoretical results. In fact, all the main theoretical results require the regularity assumption (Assumption 2.2) that the noiseless distribution $p_{data}$ has a continuously differentiable density on the manifold, which does not hold in the case of empirical distributions of finite samples.\n\n4. In Remark 4.1 on Page 5, the authors claim that the assumption of a compact support of the limiting distribution is reasonable because \"many diffusion models apply clipping to generated samples\". I think this is a bit misleading - applying clipping as a post-processing step after sampling doesn't affect the properties of the stationary distributions of the score functions, about which the assumptions are stated.\n\nTypos:\n1. Line 202, \"difficult\" -> \"difficulty\"\n2. Line 336, \"satisfie\" -> \"satisfy\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "97luw9gmcx", "reviewer": "Reviewer_DVRd", "summary": "The paper studies diffusion models and argues that their success is due to learning the manifold supporting the data distribution rather than finely learning the data distribution itself. The authors support their claim by showing that, in the small noise limit, the dominating term in the noised log density is determined purely by the manifold and does not involve the data distribution. The authors go on to argue that from a methodological perspective, the aim should be to estimate the manifold rather than the distribution since this task can be done successfully even with larger errors in the score estimation; in this sense, geometric learning is easier than distribution learning. Additionally, the authors propose a sampling algorithm to sample from the uniform distribution on the manifold given a score function estimator. Finally, the authors present experimental results to support their claims.", "questions": "1) Can the authors discuss regularity conditions on $p_{\\text{data}}$? It seems that learning the uniform distribution on the manifold can get quite difficult if there are meaningful regions on $M$ where $p_{\\text{data}}$ is very small. But this doesn't seem to be reflected in any of the results. Is this due to the qualitative nature of Theorem 3.1? It would be very clarifying if the authors could discuss this point in some detail.\n\n2) Can the authors provide some high-level discussion on sample complexity? With finite data, is it the case that learning the uniform distribution on $M$ is only going to be feasible when $p_{\\text{data}}$ is bounded above and below by a constant on its support (i.e. there's enough mass everywhere on the manifold)?\n\n3) What are the difficulties in getting a quantitative version of Theorem 3.1 (say in Wasserstein distance or something else)? It would be nice to add a bit of discussion about this in the paper.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper makes a good case for switching paradigms to geometric learning (i.e. learning the uniform distribution on the manifold) as the score error requirements are much less stringent than that required for full distribution learning. It also seems natural to me from a generalization perspective that the manifold ought to be the target. It is also quite nice that a simple (one-line!) modification to the sampling dynamics can tolerate higher score errors and give uniformly distributed samples on the manifold. The paper is written very clearly and the authors provide good exposition. The experiments indeed show better diversity as one expects from targeting the manifold rather than the data generating distribution.", "weaknesses": "1) The result of Theorem 4.1 requires an $L^\\infty$ bound on the score estimation error. This is quite stringent and undesirable, especially since diffusion models are typically trained via score matching which is an $L^2$ loss. Furthermore, there is some empirical evidence that $L^\\infty$ assumptions are not typically satisfied in practice (e.g. see Section 3.1 of \"Fast Sampling of Diffusion Models with Exponential Integrator\", Zhang & Chen 2023). The authors do note this in their Limitations section, but it is a weakness nonetheless. \n\n2) The results in Theorem 3.1 are somewhat qualitative (i.e. consistency results and weak convergence), and it would be a stronger paper if quantitative rates in some distance (e.g. Wasserstein distance) were available. For example, if E(\\sigma) = o(1), is it meaningfully easier to estimate the uniform distribution on $M$ rather than the true $p_{\\text{data}}$? If it is not, then perhaps the justification for geometric learning becomes weaker. \n\n3) I'm confused about the justification for learning the uniform distribution from the Bayesian Inverse problem perspective. I agree that choosing it as the uniform yields a weaker score estimation condition for correct posterior sampling. But I'm not convinced this is by itself a good enough reason. There is strong information that is being lost by switching to the uniform distribution which might have resulted in a \"better\" posterior distribution but harder to sample from. It's not obviously clear why this is less desirable than a less informed prior with easier sampling requirements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)