Improving the Euclidean Diffusion Generation of Manifold Data by Mitigating Score Function Singularity
Authors: Zichen Liu
Wei Zhang
Tiejun Li
Venue: NeurIPS 2025 poster ()
Keywords: diffusion models
submanifold
score function
singularity
Primary area: deep_learning
TL;DR: We address the multiscale singularity in Euclidean diffusion models for manifold-constrained data and propose two novel methods to improve sampling accuracy on complex geometries.

Abstract: Euclidean diffusion models have achieved remarkable success in generative modeling across diverse domains, and they have been extended to manifold cases in recent advances. Instead of explicitly utilizing the structure of special manifolds as studied in previous works, in this paper we investigate direct sampling of the Euclidean diffusion models for general manifold-structured data. We reveal the multiscale singularity of the score function in the ambient space, which hinders the accuracy of diffusion-generated samples. We then present an elaborate theoretical analysis of the singularity structure of the score function by decomposing it along the tangential and normal directions of the manifold. To mitigate the singularity and improve the sampling accuracy, we propose two novel methods: (1) Niso-DM, which reduces the scale discrepancies in the score function by utilizing a non-isotropic noise, and (2) Tango-DM, which trains only the tangential component of the score function using a tangential-only loss function. Numerical experiments demonstrate that our methods achieve superior performance on distributions over various manifolds with complex geometries.

Improving the Euclidean Diffusion Generation of Manifold Data by Mitigating Score Function Singularity
Zichen Liu 1,
Wei Zhang +22 and Tiejun Li 61.34
CCenter for Data Science, Peking University ZZuee Institute Berlin
LMAM and School of Mathematical Sciences, Peking University 4 CCenter for Machine Learning Research, Peking University
Abstract
Euclidean diffusion models have achieved remarkable success in generative mod- eling across diverse domains, and they have been extended manifold cases recent advances. Instead of explicitly utilizing the structure of special manifolds as studied in previous works, this paper we investigate direct sampling the Euclidean diffusion models for general manifold-structured data. We reveal the multiscale singularity the score function in the ambient space, which hinders the accuracy of diffusionn-ennerated samples. We then present elaborate theoretical analysis the singularity structure of the score function by decomposing it along the tangential and normal directions of the manifold. To mitigate the singularity and improve the sampling accuracy, we propose two novel methods: (1) Niso-DM, which reduces the scale discrepancies in the score function by utilizing isotropic noise, and (2) Tango-DN MMM, which trains only the tangential component the score function using tangential-only loss function. Numerical experiments demonstrate that our methods achieve superior performance on distributions over various manifolds with complex geometries.
Introduction
have demonstrated remarkable success generative modeling in various domains, including image generation [12), audio synthesis (5), molecule generation (13), and other applications [22 48 These models operate through a forward process, which gradually perturbs data into noise, and reverse process, which reconstructs the data from noise.
Diffusion models I1 35 36 38
Beyond Euclidean space, many scientific fields involve data distributions constrained on Riemannian manifolds. Examples include geographical sciences (28) using spheres, protein structures [45] and robotic movements (34 with SE(3) and SO(3), lattice quantum chromodynamics (26 with SU(3), 3D computer graphies [14] with triangular meshes, and cell development (2]) with the Poincaré disk. To model such data, recent works g 17 26 49) have extended diffusion models to Riemannian manifolds, achieving notable progress by leveraging manifold geometry. Nonetheless, these methods face challenges due to the computational complexity of simulating diffusion and obtaining accurate geometric information like the heat kernel.
the other hand, a natural idea to directly apply diffusion models designed for Euclidean space to
data. However, this approach encounters a fundamental challenge due to the
Cooresponding author: @ pku.ecu.u (T. Li), wei.zhang @ @u-bbelinnde (W. Zhang)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

singularity of the score function. Prior studies 27, 29 have identified that under the manifold hypothesis, the norm of the score function explodes as the time in the reverse process approaches zero. Recent works 4 27 29 have theoretically analyzed this singularity under the Ornstein- Unaiinnnhhheeeke process, providing asymptotic bounds on the score function and introducing structural assumptions on the data distribution. In this paper, we further enhance these insights and propose methods for mitigating the singularity of the score function under the manifold hypothesis.
Our work. We consider a probability distribution on a known d-dimensional submanifold M embedded in R", where d < n. Given samples from this distribution, our objective is to generate new samples by directly applying Euclidean diffusion models. Within the framework of Variance Exploding Stochastic Differential Equations (VESDE) with such manifold setting, we theoretically demonstrate that the perturbed score function exhibits distinct scales along the tangential and normal components. To address this issue, we propose the following two methods improve diffusion models (DM):
1. Niso-DM: Perturb data with non-isotropic noise by introducing additional noise along the normal direction during the forward diffusion process. This reduces the scale discrepancy in the score function between the tangential and normal directions, making the relaxed score function easier to approximate.
2. Tango-DM: Train only the tangential component of the score function using tangential- only loss function. By bypassing the training of the normal component, this method overcomes the learning difficulties caused by the multiscale issue.
Our main contributions are follows:
We present an elaborate theoretical analysis of the multiscale singularity structure of the score function by separating it along the tangential and normal directions of the manifold in the embedded Euclidean space. Furthermore, we investigate the relationship between the full-space score function and the original Riemannian score function, which has not been considered before.
We propose two novel methods, Niso-DM and Tango-DM, making the relaxed score function easier to approximate. We also give theoretical analysis of Niso-DM. Empirically, methods demonstrate superior performance on distributions across various complex and non-trivial manifolds. This in sharp contrast with previous methods that can only handle manifolds with special analytical structure.
2 Backgrounds and preliminaries
2.1 Diffusion models
Diffusion models can be formulated using stochastic differential equations (SDEs) 38 Specifically, the data perturbation process modeled by the following forward SDE:
dX, f(X,, t)dt + g())
(1)
where and g are fixed drift and diffusion coefficients, respectively. By carefully selecting ff and g- the distribution of Xr, with the probability density function (pdf) Pr, can be well approximated by Gaussian distribution (0, af I), where of represents the variance of the Gaussian distribution. The corresponding reverse-time SDE, which transports pr back to the initial pdf po, is given by:
dX, = (f) g(t) Vv, log (X.)) dt + g())dW,,
(2)
where W, is standard Brownian motion in reverse time and pe is the time-dependent distribution driven by the stochastic process X,-
Inspired by this formulation, diffusion models employ neural network s0 t) to approximate the score function V, (o. Since (x(x) is analytically unknown, direct optimization of the quadratic loss:
Lquaa (0) E (ue
(3)
equad (t, 0) Ex, [ ()t) t) Vz, log (xx:))
(4)

---

is intractable. Instead, score matching techniques such as denoising score matching 43 are commonly used, which learn the score with the loss
(asm (t, 0) Ex,,, ( (Z,, t) Vz, loggp(zr((2))),
(5)
where (r (r) denotes the probability density of X; in ① conditioned on the initial state Xo = x. To utilize (5). the following two specific SDEs introduced in 38 are often considered, in which cases
the density (ma (r) has a closed-form.
1. Variance Preserving SDE (VPSDE):
1 dX, - - (tt) + Cotiut+BBBOB VB(daw.
(6)
where 3(() > 0.
2. Variance Exploding SDE (VESDE):
do? dw,,
dx, = de
(7)
where 0z is predefined noise scale, commonly chosen O ( / )/T )/TT for some d max > miiu > 0. In this case, (r(2y (x) follows a Gaussian distribution (2 [x, I).
In this study, we choose VESDE for its advantageous properties. Under VESDE, the mean of the perturbed distribution remains unchanged. At any time, the perturbed distribution (x) can be viewed as points the original manifold M with added noise. This simplifies theoretical analysis and avoids the complications introduced by the evolving manifold VPSDE. In fact, VPSDE involves an evolving manifold over time, defined as M₁ = { à fa 8(s)da x perturbation.
x E M] as the mean with noise
2.2 Riemannian manifolds and notation
In this paper, the manifold M is viewed as M = fx E R=(g(x) = embedded in IR,, : : R -> gn-djj is a known function, and we assume that Vg(x)= E B has full column rank for all M. The target distribution is formulated po( (z), where pa (x) is an unknown density and (z)) represents the volume form (32 of M. The unit normal vectors at a point & E M
are given by (x)) = VE(x) (VE(=)(VE(<z)
For x E M, the operator V-M is defined as VMM((z)= = P(x) V (z)( for any smooth function h, where P(r) is the projection matrix 32 onto
the tangent space of M and is given by
P(x) = -I V(VVVV(VV((/)V(()))
=I
(8)
3 Scale discrepancy of score functions
It is commonly understood that singularities arise when Euclidean diffusion models are directly applied to data with manifold structures. In this section, we examine the singularity diffusion models under VESDE, following the settings in Section 2.2 By introducing Gaussian noise across the entire space, the perturbed data become distributed throughout IR"", deviating from their original confinement to the d-dimensional submanifold M. The perturbed distribution is given by
Po (o()) - //x po (al=))o(( = (2ma²)= - 3 / po(z)e
2/2
20t do
(9)
where ă a variable in the embedded space IR",, Po (#(z) denotes the pdf W(r, 0²I) the isotropic perturbtion, and po(z) is only defined M. Hereafter, the subseript of denotes the noise scale instead of the time t.
When the noise scale is small, the perturbed distribution becomes tightly concentrated around its mean, resulting a steep gradient landscape for log Po (7). As a approaches zero, this steepness appears as sharp variations and introduces multiscale challenges. In Theorem 3.1 we provide rigorous
3

---

and refined results on this singularity compared to previous works. Eq. (TO) shows that the score function explodes entirely due to its normal direction, represented by the first term on the right hand side. After subtracting this term, the remaining component is of order O(1). )... Furthermore, for points on the manifold, Equations (TT) and (12) establish a novel connection between the Riemannian score function V log Po(x)) (1) and the perturbed score function in the ambient space. To the best of our knowledge, this connection has not been considered in previous works. The proof of Theorem 331 provided in Appendix A. and the assumption of uniform boundedness of the derivative is discussed in Appendix F3
Theorem 3.1. Ler P(x) E Rax" denote the projection matrix at x M.
Assume that
IM (=ol) ((x) +oo and M1 suPseM maxiskij'sn
0Pu (x)) +00. The following Dzy
two asymptotic expansions for po (ū) defined in 4 hold:
1. For It & M, assume that x² E M is the unique minimizer of mingeM || - Tl|, and that - O, have
xx - 3 011000 "MM Under these conditions,
Va Po = x² o² +0(1).
(10)
2. For &c E M, as a + O, we have
aP. (i) (z)
Va (T) V log
Σ
(11)
Bay
3,1==1
M lo
M log O()),
(12)
8P where Bzy
denotes the vector whose ith component is 0p
Next, based on Theorem 3.1 we analyze the scale discrepancy of the score function between its tangential and normal components. To achieve this, notice that the projection matrix P in (8) can be extended to the entire space IR", as long as & is well-defined and smooth on R" With this extended projection matrix, we further define the tangential and normal components of the score function for any 2 R" P(t) V log Po (ø) and P4 (i) log (ž), respectively, where denote (+) = I - P(ž). Using this extended definition, we then decompose the quadratic loss Eqund into two parts: ell respectively.
'quad and e quad- corresponding to the tangential and normal components,
Pquad -Bal1so(T - V (1]1
P [[l||(o)) PE - p* (+) log (1)
: ell quad + qquad-
(13)
P(š) = P(x*)+0(i-z))) and E, a (x - 2) = Using Eq. (T) and the facts that P() (a)). the two terms being approximated in (13 have scales of 0(1) and O(1/o), respectively, by the following two estimates:
E3 m P(E) log (i)] = Eziz ([(=')+0() x²)) (x" -
0² + +0(1)0 = =0(1), (14)
(x'- - + +0a)) ( (15)
E3 [P- (T) V log (T)) = EzjE (x*( ---f--1) -x)))
where Ezju denotes the expectation with respect Po (ū(z).
This multiscale singularity of the loss formulation poses significant challenges during training. In fact, the above analysis also applies to the loss (5). since it differs from 4 only by constant that is independent of 0. As a result, training with (5) inherently prioritizes fitting larger-scale features of the score, which are aligned with the normal component in our settings. This is because quadratic loss minimizes the average of squared errors, making disproportionnaely sensitive to errors that occur in larger-scale directions. In the context of this work, the normal component pulls samples

---

back onto the manifold, whereas the tangential component, which is more critical, captures the data distribution on the manifold. As result, the model fails to adequately capture finer details of the data distribution, ultimately reducing the accuracy of the generated distribution. In other words, the trained model primarily captures the manifold itself rather than the distribution on it, leading to a phenomenon known as manifold overfitting (24).
To address these limitations, specific methods are required during the model training phase to ensure training stability and accurately capture the intrinsic data distribution on the manifold. Based on this perspective, we propose the following two methods. The first method reduces the scale discrepancy between the tangential and normal components by modifying the structure of the noise. The second bypasses the learning of the normal component of the score function and employs a dedicated projection operator to project samples back onto the manifold.
4 Methods
In this section, propose two methods to address the singularity of the score function. The first method, referred to as Niso-DM, introduces additional noise along the normal direction to mitigate its dominance. The second method, called Tango-DM, focuses only the training of the tangential component of the score function when the noise scale og is small.
4.1 Niso-DM: Perturb data with non-isotropic noise
We propose a strategy to mitigate scale discrepancies by replacing the isotropic noise in the forward process of diffusion models with non-isotropic noise. The perturbed data is generated as H, + oef1 + oz (z)ee, where a ~ Po Po(x). (x). ~ (N,
a E (0,1), 1), and (z)) E ggx((ndd) is defined in Section 2.2 The term of (x))e2 represents an additional noise along the normal directions. The conditional probability density of the perturbed data is given by Po. (ā|x) (a, Mo, (x)), where Ex, (x) of 1 + of
(N
201 N (x))"))) The following theorem log Po, (t) and V log (ō), as + O.
establishes the relationship between Va log
Theorem 4.1. Let po (ū) denote the distribution under non-isotropic perturbation, defined
Do (x)= (2x) S. (dle) En (x)) be - (a (x) (z) (a-x) da (x)),
(16)
where [x)(x) 02I + 20 (x)) and a E (0, 1). By making the same assumptions as in PP 02y00y
Theorem 3.1 and further assuming that M2 SuPPEM max have the following results for Po (T):
(ar) < +00, we
1. For v & M, assume that x" E -M is the unique minimizer of min || - E||, and that
1x - 3 0l1o0 < min(1, n²(MM)+MM) }. Under these conditions, as o + O, we have
x²" - 22 1 02-2a (0(11-20))))
V (o
(17)
where the symbol is defined as a A min(a,b) b}.
2. For ă E M, as 0, we have
Z log (G) = V log + (2 203A1)
(18)
Eq. 17 shows that, by introducing additional noise along the normal direction, the scale of the normal component is reduced from O(1/a²) to ((1/o2a). Eq. (T8) states that this approach does not affect the characterization of the distribution on the manifold (i.e., the tangential component). As the noise level parameter o; approaches zero, the score function of the perturbed density Po. on the manifold converges to the Riemannian score function of Po- Therefore, by employing non-isotropic noise, we can still generate samples that adhere the original distribution of the dataset.
Noting that Va, log Po, (xz(a) = ---- Eo, (x) 11 - I)), the denoising score matching loss 5 can be written into the following form:
N (t, 0) = Br,,, |180 (r,, t) E,, (1 =
(19)
Under the settings Section 2.2 Eo, (x) has closed-form expression (See (49). Besides, we set ag log /uiso log de (i.c., Cniso of in practice, where Caiso is a fixed constant.
5

---

4.2 Tango-DM: Learn only the tangential component of the score function
Recall that the projection matrix P(x) in (8) is well-defined in the ambient space R". Accordingly, we define (z, t) (a 1) for RR, which represents the tangential component of the parametrized vector field.
As discussed in Section 3 (see 13 the loss function and (5) can be decomposed into two parts, Q quad and aund and the singularity issue is associated with the part (quad To address this, propose training only the tangential component of the score function using the loss &quad when the noise scale
o, is sufficiently small, thereby avoiding the singularity associated with quudd To compute ell introduce the following Tango loss
"quad
Crango (1, 0) Ex (2), - P(x+) Vx, log (22
(20)
The optimal score network s/g. (1r, I) that minimizes 20) satisfies a (2., 1) V(x) log log (4)
(See Appendix C.I the proof). This result ensures the validity the loss function in (20).
When the noise scale is smaller than a pre-selected Crango (i.e. < Crango). ). the tangential component
of the score function !!| (a., trained via the Tango loss lango- On the other hand, when o₁ 2 Crango- the singularity issue is less severe, allowing us to use the original denoising score matching loss (5)
to train the entire score function (1 t), which includes information along the normal direction. This enables the score function to push points toward the vicinity of the manifold. The overall loss calculation summarized in Algorithm| in the Appendix.
Generation
We propose two sampling methods to generate samples using the learned score function 8p. The first method, named Reverse SDE, is based on the standard generation process in R" with an additional projection step to ensure that the samples lie on man- ifolds. The second method, named Annealing SDE, adopts a two-stage approach, where the first stage simu- lates the standard reverse SDE, and the second stage performs annealed Langevin dynamics constrained to manifolds. The details of these meth- ods are provided in the sections below and are summarized in Algorithms and 3 in the Appendix.
Iso+res
0.10
0.05
0.10
Niso+res
0.10
0.05
0.00
0.2
1.0
Tangotres
0.05
0.00
0.05
0.0
0.2
0.4
0.8
1.0
5.1 Reverse SDE
1.4
1.2
1.0
0.8
0.6
0.4
0.2
Figure 1: The average eπтоr of the tangential component of the learned score function in the x plane, along the z-axis and f-axis, with aiin = 0.01. From top to bottom, the plots correspond to the vanilla algorithm (Iso-DM), our proposed Niso-DM and Tango-DM, all using the rescaling technique.
The first method follows the standard generation process, where samples are generated by simulating the reverse SDE (21) with decreasing from T to which is derived from (2) by setting
the drift term if to 0.
dX,, = - g(t) (X,, t)dt + g())WW,
(21)
However, the ending samples lie in the vicinity of the manifold rather than exactly on it. To address this, add final projection step ensure that the generated points lie the manifold.
In general, this method is not applicable to Tango-DM, which utilizes the Tango loss in (20). Since the normal component of the score function is not learned when is small, the score function (2

---

does not capture information in the normal direction. For Niso-DM, the final distribution of the reverse SDE is the same as the distribution of + Cniso ( where ~ Po and Cuiso represents noise along the normal direction. By employing projection onto the manifold in the subsequent step, the obtained samples can accurately approximate the target distribution.
5.2 Annealing SDE manifolds
The second method adopts a two-stage sampling procedure with a predefined threshold ō. In the first stage (Ø, σ), we simulate the reverse SDE (2T), which provides a coarse approximation of the target distribution. In the second stage (a, < σ). we refine the samples by simulating annealed Langevin dynamics constrained to the manifold. Specifically, for fixed t, we perform no steps of the following Langevin dynamics:
dx, = P(X.)s ()ds + V2dW...
(22)
After that, decrease t according to t + t At and restart the simulation of SDE (22) using the final samples from the previous iteration as the initial states. This process is repeated until the desired level of refinement is achieved. i.e., t = 0. This stage be viewed as the manifold version of the Corrector algorithms in [38]
To analysis the proposed sampling method, we assume that the neural network 8p has learned the optimal solution, i.e. ((r)ss(zz, t) V-M log Po, (x) for x E M. Under this assumption. the invariant distribution of the SDE (2) is paz (z))a According to (12) in Theorem 3.1 and
(18) in Theorem 4.1 regardless of whether non-isotropic noise is used, V. log (x) converges to V.M log (1) as t O. This demonstrates that, by employing annealed Langevin dynamics
constrained to the manifold, we can effectively sample from the target distribution po (o (x()) on the manifold.
6 Experiments
In this section, we evaluate our methods by studying several representative problems on typical manifolds. Specifically, we consider four datasets four different types of submanifolds: (1) hyperplane in 3D space, (2) mesh manifolds in 3D space, (3) the special orthogonal group of order 10, and (4) the level set of dihedral angles in the molecular system alanine dipeptide. Our code is available at:
1999/Niso' TangoDD
We perform experiments with the vanilla diffusion models, as well as proposed Niso-DM and Tango-DM, denoted as Iso, Niso, and Tango, respectively. Moreover, we consider using neural networks to approximate normalized score functions, defined as so(z, $0(2,t) t) is a neural network and wa is the scaling factor of the optimal score function. This rescaling technique improves numerical stability by allowing neural networks to approximate instead of one that grows explosively. Notably, this technique is equivalent to the e-parameterization introduced in 33 47 We denote this method with the suffix +res. After training, new samples are generated via two methods: Reverse SDE and Annealing SDE on manifolds. Although we use "vanilla" to refer to the Iso and Iso+res methods, these two methods also involve manifold information as we utilize the projection operator during the generation process.
åø (z, f)/ f)/w,, where
O(1) term
To evaluate the quality of the generated samples, we use several metrics to compare the generated distribution with the target distribution. The choice of metries balances the ability to capture on- manifold distribution differences with reasonable computational cost. We conduct using the same byperparameters, except for those specific to each algorithm. Overall, we recommend the Niso+re res training algorithm combined with the Reverse SDE generation method, as it performs best in most cases. In experiments on mesh data and the special orthogonal group, we also conduct Riemannian sliced score matching (RSSM, for simplicity) in 8 17] as a baseline. Further experimen- tal details are provided in Appendix [D] We also conduct ablation studies in Appendix to analyze the varianae tradeoff associated with the choice of hyperparameters.
experiments
6.1 The hyperplane in 3D space
We first consider a toy model: the hyperplane embedded three-dimensional Fuclidean space, specifically the set M {(z, 2) E 12222 2== 0}. The target distribution is mixture of Gaussian
7

---

distributions with nine modes located on the plane. In this example, the score function has closed-form solution that can be explicitly derived.
In Figure T we present the error of the tangential component of the learned score function. The results demonstrate that our proposed Niso- DM and Tango-DM significantly reduce the error of the tangential
component, as & approaches O. Table 5 in the Appendix reports the maximum mean discrepancy (MMD), where both Niso-DM and Tango-DM show significant performance improvements.
6.2 Mesh data
We consider the Stanford Bunny 40 and Spot the Cow 77. which contain meshes derived from 3D scanning ceramic figurines of rabbit and cow, respectively. To create the target densities, follow the approach in 66, [9] 32). which utilizes the cigenpairs of the Laplace- Belltram operator equal-proportion mixture of density functions
on meshes. The target distribution is chosen corresponding to the 0-th, 500-th, and 1000-th eigenpairs. The manifold M is a polyhedron composed of triangular faces, differing from the definition provided in Section 2.2 Although the manifold is not smooth along the shared edges between adjacent faces, which form zero-measure set, numerical performance ensures that this is acceptable setup.
Table reports the Jensen- (JS) divergence of the histograms on mesh faces between the generated samples and the original datasets. With the Rescaling technique, both Niso-DM and Tango-DM demonstrate significant improvements; without it, Tango- DMg still achieves improvements under the Annealing algorithm. Furthermore, suspect that the suboptimal performance of RSSM arises from the challenges in achieving a uniform distribution on manifolds with complex geometric structures, such the narrow neck in "Spot the Cow."
Table 1: Results for the mesh data: JS divergence of the samples generated by the Reverse SDE and Annealing SDE algorithms under different training methods. The symbol "-" indicates that the Reverse SDE sampling method is not applicable to Tango-DM. The results are reported the mean and standard deviation over five independent runs. The suffix +res indicates the rescaling technique.
Stanford Bunny
Spot the Cow
Reversal
Annealing
Reversal
Annealing
Iso
3.58e- 1e0e--11..600 1.36..3 3.586-1+11763
1.95e-141.60
3.41e-1=3200
3.65e-1=2.97e
Niso
4.12c- 100-1+2.666 1+2-630-3
3.38e-1=1.2705
1.68c-1=3.000- +3.0003
Tango
4.08e-1#3.11o4
3.51e-1#3.65e
Iso+res
3.2
3.94e- +
1.30e-1=200%e
3.48c-1
Niso+res
3.48c-148.77-4
3.86e-
1 1.80-3
3.28c-1 -1..1.1..--- 19c4
3.40e-
1
Tango+res
3.84e-1 1.5763
3.39e- 1.74663
RSSM
7.00c-
6.3 High-dimensions special orthogonal group
We evaluate the performance of our method in high-dimensional setting on the special orthogonal group S0(10), a di-eensiioonll submanifold embedded in 100 We construct synthetic dataset drawn from a multimodal distribution on SO(10), consisting of modes.
Our results demonstrate the effectiveness the proposed methods. In Table 2 we compare the sliced
1-Wasserstein (2 distances between the generated and the target distributions. These results highlight
the accuracy and reliability of our approach in high dimensional manifold settings. Moreover, our approach outperforms RSSM, while the vanilla algorithm does not.
6.4 Alanine dipeptide
We apply our method to alanine dipeptide, a model system frequently examined computational chemistry. The system's 's configuration is characterized by two dihedral angles, & and v (refer to Figure The manifold consists of the configurations of the system's 's 10 non-hydrogen atoms (in 30 with the angle fixed -70%, which is level set of the dihedral angle ø.
8

---

Table 3 exxhibits the results of the 2- --assersteinn distance between the test datasets and the generated samples, showing that our methods outperform the vanilla models under both the Reverse SDE and the Annealing SDE sampling algorithms. However, in this case, training the normalized score functions (+res) does not lead to better results. One possible explanation is that simultaneously rescales both the tangential and normal components, resulting in the loss of critical tangential information.
Table 2: Results for S0())): Sliced 1-
Wasserstein distance under different training methods. For more detailed explanations, refer to the caption of Table
Table 3: Results for dipeptide: 2- Wassersteii dis- tance under different training methods. For more detailed explanations, refer the caption of Ta- ble
Reversal
Annealing
Reversal
Annealing
Iso
1.76c-2=1.15-2 1.86c-2::9.65e-
Niso 5.58e-3= 1.84e3
1.16e-242.51e.
Iso
8.60c- 236
8.83e-2:t9.22-3
Tango
1.89e-2.2205e.1
Niso
8.31e-2
8.24e-2::5.96e
Tango
8.60e-245.71e.3
Isotres
9.49e-3 1.91e-3
1.17e-2.11.29
Niso+res
4.60e-3 8.24e
6.7.5-3
Isotres
8.59c-2#6.90e
1.21e-1#6.660
Tango+res
6.42e-3.11.97.3
Niso+res
1.3.5-21..56e..
9.47e-2..561e
Tango+res
9.62e-219.1e
RSSM
7.1.4-319.m64
7
Related work
Research on manifold related diffusion models can be broadly categorized into two directions: (1) diffusion models for distributions under the manifold hypothesis, where the underlying manifold unknown, and (2) diffusion models for distributions on a known manifold, which is the focus of this work. For the former, discuss the singularity and the scale discrepancy of score functions mentioned in the related work; for the latter, we explore several studies based on known manifolds.
Singularity of score functions. Several studies diffusion models under the manifold hypothesis 27 29] have highlighted the divergence of the score function as + 0. Beyond empirical observations, recent studies have provided mathematical analysis of diffusion models based on the VPSDE. For instance, Chen et al. [ present a mathematical analysis under the assumption that each data point lies hyperplane. In [29] is shown that the norm of the score function satisfies V px 2 1/VL. Furthermore, Lu et al. 27) rigorously prove that this singularity follows a 1/t scaling in a strong pointwise sense under the VPSDE, which similar to our results in (10). Further discussion can be found in 25
To mitigate this singularity issue, a number of works (e.g.
38
assume that the initial time is at
I = é > 0, instead of = 0. The study [9] introduces diffusion model on the product space of position and velocity, employing hybrid score matching to circumvent the singularity. Furthermore, in (37), the score function is parameterized by se (z, t) (z oz. where (z()) is neural network.
Scale discrepancy of the score function. The work (39 uses the singular value decomposition of the score matrix to identify the intrinsic dimension of the manifold, which essentially utilizes the discrepancy of the score function. References (00 42 44 46) investigate the fundamental principles of diffusion models under the manifold hypothesis, by studying the digenddecomposition of the Jacobian of the score functions. This analysis implicitly relates to the score discrepancy discussed in work. In particular, the decomposition of the denoiser mapping (Eq. (12)) in (20), which involves a geometry-adaptivs harmonic basis, also implies this discrepancy, aligning with (T0) in our paper. Papers ① and [10] study the Memorization and Generalization in Generative Diffusion under the manifold hypothesis through Hidden Manifold Models and Generalized Linear Models. Besides, [16] suggests that conservative diffusion is guaranteed yield the correct conclusions when analyzing local features of the data manifold.
While previous works have implicitly touched upon the scale differences in manifold settings, our work explicitly formalizes this phenomenon with rigorous mathematical formulations and introduces novel solutions to address it. Furthermore, we believe methods can enhance the accuracy of

---

diffusion models during the manifold consolidation phase (as described in 42 or the collapse transition times (as mentioned I0). ). by refining the computation of the tangential component of the score function.
Additional noise along the normal direction. Paper 15] proposes adding noise along the normal direction "inflate" manifolds under the setting of normalizing flows, which is similar our Niso method. Theorem and Proposition 7 in [15] analyze the relationship between the perturbed distribution and the original distribution on the manifold, which aligns with the conclusions in our
equations (12) and 18.
On the other hand, our method differs from [5] in several key aspects. In [1], the added noise has constant magnitude, requiring additional assumptions (e.g.. Q-normally reachable) to prevent interference between noise at different points. In contrast, our analysis (Theorem 3.1) permits the noise magnitude diminish toward zero, relying on weaker assumptions. Additionally, [15] applies noise inflation prior to training the normalizing flow, whereas method integrates noise addition directly into the diffusion process, accompanied by the noise dynamics of diffusion models. Finally, while the analysis in [15 focuses on the perturbed density function, our work studies the asymptotic behavior of the score function.
Diffusion models manifolds Riemannian Score-based Generative Models 8 extend SDE-based diffusion models to manifold settings by estimating the transition kernels using either the Riemannian logarithmic map the eigenpairs of the Laplacian-Beltree Diffusion Models [17] employ a variational diffusion framework for Riemannian manifolds and, sim- ilar our approach, consider submanifolds embedded in Euclidean space. Additionally, Trivialized Diffusion Models [49] adapt diffusion models from Euclidean spaces to Lie groups.
operator manifolds [26]. Riemannian
8 Limitations and future work
Compared to approaches under the manifold hypothesis, our method still requires knowledge of the manifold's definition. In fields like image and language processing, the manifold structure is often assumed but not explicitly characterized. Future work will focus on extending our approach to scenarios where the manifold is undefined or implicitly represented. Furthermore, Tango-DM DM is not suitable for the Reverse SDE Sampling algorithm and tends to be slower due to the annealing sampler. Designing Reverse SDE-style algorithms for Tango-DM remains an open direction for future research.
Conclusion
We address the multiscale singularity of the score function for manifold-structured data, which limits the accuracy of Euclidean diffusion models. By decomposing the score function into tangential and normal components, we identify the source of scale discrepancies and propose two methods: Niso- DM, which reduces discrepancies with non-isotropic noise, and Tango-DM, which trains only the tangential component. Both methods achieve superior performance on complex manifolds, improving the accuracy and robustness of fiffusion-based generative models.
Acknowledgments and Disclosure of Funding
Zichen Liu acknowledges support from the China Scholarship Council (Grant No. 202306010047). Wei Zhang acknowledges support from DFG's Eigene Stelle (Project No. 524086759). Tiejun Li acknowledges support from the National Key R&D Program of China (Grant No. 2021 0021YFA1000 YFA1003301) and the National Science Foundation of China (Grant No. 12288101).
References
[1] Beatrice Achilli, Luca Ambrogioni, Carlo Lucibello, Mare Mézard, and Enrico Ventura. Memorization and generalization in generative diffusion under the manifold hypothesis. Journal of Statistical Mechanics: Theory and Experiment, 2025()):073011 2025.
10

---

[2] Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and Radon Wasserstein barycenters of measures. Journal Mathematical Imaging and Vision, 51:22-45, 2015.
[3] Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. Transac- tions on Machine Learning Research, 2022.
[4] Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffuusion models on low-dimensional data. In International Conference on Machine Learning, pages 4672-4712 12. PMLR, 2023.
[5] Nanxin Chen, Yu Zhang. Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations. 2021.
[6] Ricky T. Q. Chen and Yaron Lipman. Flow matching general geometries. In International Conference Learning Representations, 2024.
[7] Keenan Crane, Ulrich Pinkall, and Peter Schröder. Robust fairing via conformal curvature flow. ACM Transactions on Graphics, 32(4):1-10, 2013.
[8] Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thortton, Yee Whye Tch, and Amaud Doucet. Riemannian score-based generative modelling. Advances in Neural Information Processing Systems, volume 35, pages 2406-2422, 2022.
[9] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically-damped Langevin diffusion. In International Conference on Learning Representations, 2022.
[10] Anand Jerry George, Rodrigo Veiga, and Nicolas Macris. Analysis of diffusion models for manifold data. arXiv preprint 0433399 2025.
[11] Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840-6851, 2020.
[12] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23 (47):1-33, 2022.
[13] Emiel Hoogchoom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning, pages 8867-8887. PMLR, 2022.
[14] Hugues Hoppe, Tony DeRose, Tom Duchamp. John McDonald, and Werner Stuetzle. Surface reconstruction from unorganized points. In Proceedings of the 19th annual conference computer graphics and interactive techniques, pages 71-78, 1992.
[15] Christian Horvat and Jean-Paseal Pfister. Density estimation on low-dimensional manifolds: an inflation deflation approach. Journal of Machine Learning Research, 24(61):1-37, 2023.
[16] Christian Horvat and Jean-Pascal Plister. On gauge freedom, conservativity and intrinsic dimensionality estimation in diffusion models. In International Conference on Learning Representations. 2024.
[17] Chin Wei Huang, Milad Aghajohari, Jocy Bose, Prakash Panangaden, and Aaron Courville. Riemannian diffusion models. In Advances in Neural Information Processing Systems, volume 35, pages 7761, 2022.
[18] Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statisties-Simulation and Computation, 1059-16191) 1989.
[19] Jachyeong Jo and Sung Ju Hwang. Generative modeling on manifolds through mixture of Riemannian diffusion processes. In International Conference on Machine Learning, volume 235, pages -23700. PMI.R, 2024.
[20] Zahra Kadkhodaie, Florentin Guth, Eero Simoncelli, and Stéphane Mallat, Generalization in diffusion models arises from geometry-adaptive harmonic representations. In International Conference on Learning Representations, 2024.
[21] Anna Klimovskaia, David Lopez-Paz. Paz, Léon Bottou, and Maximilian Nickel. Poincaré maps for analyzing complex hierarchies in single-cell data. Nature communications, 11(1):2966, 2020.
11

---

[22] Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen. Huajun Feng, Zhihai Xu. Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 479:47-59. 2022.
[23] Zichen Liu, Wei Zhang, Christof Schüttee and Tiejun Li. Riemannian denoising diffusion probabilistic models, arXiv preprint 2505::2505.04333, 2025.
[24] Gabriel Loaiza- Brendan Leigh Ross, Jesse C Cresswell, and Anthony L Caterini. Diagnosing and fixing manifold overfitting in deep generative models. Transactions on Machine Learning Research, 2022.
[25] Gabriel Loaiza-Ganem, Brendan Leigh Ross, Rasa Hosseinzadeh, Anthony Caterini. and Jesse C Cresswell. Deep generative models through the lens of the manifold hypothesis: survey and new connections. Transactions on Machine Learning Research, 2024.
[26] Aaron Lou, Minkai Xu, Adam Farris, and Stefano Ermon. Scaling Riemannian diffusion models. In Advances in Neural Information Processing Systems, volume 36, pages 80291-8800081 1 2023.
[27] Yubin Lu, Zhongjian Wang, and Guillaume Bal. Mathematical analysis of singularities the diffusson model under the submanifold assumption. arXiv preprint arXiv:2301.0788. 2023.
[28] Emile Mathieu and Maximilian Nickel. Riemannian continuous normalizing flows. In Advances in Neural Information Processing Systems, volume 33, pages 2503-2515, 2020.
[29] Jakiw Pidstrigach. Score-hased generative models detect manifolds. Advances in Neural Information Processing Systems, volume 35, pages 35852-35865, 2022.
[30] Boris T Polyak and Anatoli Juditsky. Accelcration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838-855 1992.
[31] William Press. Numerical recipes 3rd edition: The art of scientific computing. Cambridge university press, 2007.
[32] Noam Rozen, Aditya Grover, Maximilian Nickel, and Yaron Lipman. Moser flow: Divergence- based generative modeling on manifolds. In Advances in Neural Information Processing Systems, volume 34, pages 17669-17680, 2021.
[33] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
[34] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: SE(3)-cquivariant object representations for manipulation. In International Conference Robotics and Automation, pages 6394-6400. IEEE, 2022.
[35] Jascha using nonequilibrium thermodynamies. In International Conference on Machine Learning, volume 37, pages 2265. PMLR, 2015.
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning
[36] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, volume 32, pages 11895-11907 2019.
[37] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Advances in Neural Information Processing Systems, volume 33, pages 12438-12448, 2020.
[38] Yang Song, Jascha Sahl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic diffcrential equations. In International Conference on Learning Representations, 2021.
[39] Jan Pawel Stanczuk, Georgios Batzolis, Teo Deveney, and Bibiane Schönlieb. Diffusion models encode the intrinsic dimension of data manifolds. In International Conference on Machine Learning, 2024.
[40] Greg Turk and Mare Levoy. Zippered polygon meshes from range images. In Proceedings of the 21 Annual Conference on Computer Graphics and Interactive Techniques, pages 311-318, 1994.
[41] Richard S Varga. Geršgorin-type theorems for partitioned matrices. In Gersgorin and His Circles, pages 155-187. Springer, 2004.
[42] Enrico Ventura, Beatrice Achilli, Gianluigi Silvestri, Carlo Lucibello, and Luca Ambrogioni. Manifolds, random matrices and spectral gaps: The geometric phases of generative diffusion.. In International Conference on Learning Representations, 2025.
12

---

[43] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation. 23(7):1661-1674 2011.
[44] Peng Wang. Hujjie Zhang, Zekai Zhang, Siyi Chen, Yi Ma, and Qing Qu. Diffusion models learn low-dimensional distributions via subspace clustering. arXiv preprint 44X4:22409.04226, 2024.
[45] Joseph L Watson, David Juergens, Nathaniel R Bennell, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahem, Andrew Borst, Robert J Ragolte, Lukas F Milles, et al. De novo design of protein structure and function with rédiffusion. Nature, 620(7976):1089 1100, 2023.
[46] Li Kevin Wenliang and Ben Moran. Score-based generative model learn manifold-Ii like structures with constrained mixing. In NeurlPS 2022 Workshop on Score-Based Methods, 2022.
[47] Ling Yang, Zhilong Zhang, Yang Song. Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming- Hsuan Yang. Diffusion models: comprehensive survey methods and applications. ACM Computing Surveys, 56(4):1-33 2023.
[48] Linqi Zhou, Yilun
Jiajun Wu. 3d shape generation completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5826-5835, 2021.
[49] Yuchen Zhu, Tianrong Chen, Lingkai Kong, Evangelos Theodorou, and Molei Tao. Trivialized momen- tum facilitates diffusion generative modeling on lie groups. In International Conference on Learning Representations, 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
We provide the assumptions in the statement of the theorem and the proof of the theorem in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The code and the datasets are provided.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We provide open access to the data and the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.cc/pullc//uuiii wwwiicccguie///eeeeeeeeis CoodeSubeiiieeeec ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All the training and test details are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars are provided in the table.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information on the computer resources is provided in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The potential positive societal impacts can be found in the Appendix and there is no negative societal impact of the work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of all assets used in the paper are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http:///heriippsccc/coeeeeereeee for what should or should be described.
20

---

A Proof of Theorem 3.1
Preparation.
For simplicity of notation, we use | to denote the Euclidean norm in the following two sections. We assume that ă E R" is fixed. Let us recall the expression
Sux (o(r) (i (x),
Po
0.
(23)
Let x² E M be the unique minimizer of the minimization problem min MEM
(24)
In the following, we study the limit of the integral in 23 over M as a - 0 by change of variables
using the exponential map on M at x². To this end, let us first derive some asymptotic expressions related to the exponential map.
{uIVe(>*)*'> 0, v R" the tangent space of M at x². Given T.- M. let (t) E IR" be the geodesic curve on M starting from (0) = x at t = 0 such that ()) = v. We have Taylor's ss expansion
Denote by T-- .M TM
To 70(+==z* =0==xx=0 + ut (0)e (0) +tt),, as t + 0. 2
(25)
Since x² %o (0) is the minimizer of (44) know that 77x 2 (as function of t) attains its minimum at = Therefore, taking the derivative and setting = O, we obtain
v. -= 2) O, Ve T.
(26)
An expression for the second-order derivative % (0) in (5 is not available in general. However, by differentiating the identity (T - P(). f. 0 with respect to f, and setting t 0, we obtain the equation (i.e. the expression for the orthogonal component of (0))
OP, dzy
(1- =
(x 1030y'
(27)
3.j
where uP2 (x)" (x*) denotes the vector in ET whose i-th component is Be, (x*), for S n.
aP
The exponential map exp,- T--M - M is well-defined in the neighbourhood O of Tz.M, and it is related to the geodesic curves by exp2- (v) = 7a (Iv)), where = =//ul. Therefore, from 25 have
|u|2
exp_- (v)= x²' + 7o (0) O( as |o| - 0.
(28)
Next, let us derive an expression for expx (v) - =122 when |u|| is small. Using (26) and (8). we can obtain
x. (v) - = - + |u|
(0 (x' =)) + +(b))), as | 0.
(29)
Noticing that x= - F is orthogonal to the tangent space Tx. M (see 26) using 27 we can compute the second term on the right hand side of (29)
|n| (0) (x* - -)) = |u|2 ( P(x ()+(())) (x" - 7)
Σ
aPg
(x (ojuy - 2z) (30)
azy
2.3
Substituting the above expression into (29 get
exp,. (v) - 2 |a- 2 + of SSSSS + 0(10(3), as |ol 0,
(31)
where S E Ruxn is matrix whose entries are
1 Σ a.g aP
Syy = و5f + 2 dz, ony (x)) - - for S j.y S n.
(32)
From the Gershgorin circle theorem 41 all eigenvalues of are greater than max, En, 1( aly + Dey (xr)) - 2.).. Furthermore, this lower bound is positive, as || 01100 W.M Therefore, the matrix S is positive definite. With the above expansions, prove the two claims.
da,
21

---

Proof of the first claim. We consider the general case where 2 E R" may not belong to M. Since Po (o(2)) is the probability density of Gaussian distribution Max, I²I), we can derive
/
Po(x)== I- σ² Po (o(
V log Po (i)
Po (T)
(33)
x²
o² Po (3)2)dom((z) Po (T) J Po(x) Po(z)
-ー
x²
x²
On the one hand, since x" is the unique minimizer of (24 there exists > 0, such that |æ z/² xx* - -al² + for all M O', where O' == exp2 (O). Therefore, we have
1 po(x)= = 02
Po (z)
5(2T0?)-3
Un po(x)) x x'|dom(()) (a 2 20 )e
-----1
(34)
20
10?
=o(e 162
2n
On the other hand, using the fact that exp, is a diffeomorphism on O with det D exp... (v) = 1, applying the change of variables I exp, (v), we get
Io po(x)= x o² Po (3
(35)
-(2TO?) 1 po(expp-- (v)) expx (v) x²
(u)-a) a)
20
du
Using the expansion 28 we can write
po(exp_. (v)) (1)-2") = -poo=")==- ( - +((x)). (36)
Hence, using the expansions
31 and 36 we can derive the integral in
(35 as
x² 1 po(x)) σ² Po (x)
0² polxx)) v (VM popp"(")
v)c + ア "ol"oo("")) po())))))) +0)))
0
20
du
aP aP?
(2TT²²) - # 20
(37)
- Jo, po(a*) v _ + (V-M o')'' + 1
du
(2xT) 2 2 - 26
=0(1))a-Re 10 2/2
where the second equality follows from change of variables by oo' and O, == (o lo-v
0}, and the last equality follows from the fact that the integral So, o' e'e o solt del O(1).
o
Combining (34) and (37). we arrive
[N Po(x) o² -P (3 0(1)
(38)
Using similar derivation, for po we can obtain the expansion
Pa (i) = / (1)2))o(() (2TTa2)
-02- -
2nd
(po(*") +(o))
(39)
The first claim is obtained by combining 33
38
and
39
22

---

Proof of the second claim. case S I and 39 simplifies to
Now we consider the case where i E M. First of all, notice that in this
(T) = (2TT²) (po(t)+
(40)
We can derive
PPo (8)
M
Vol))P(2\VaP(((())d)0 - SN Pllo)/1-Paa)V
M Do
( + / polz) - P(())) x - 2 σ2 Po (i) (z)
(41)
where V.M denotes the gradient operator on M at a and we used the fact that x = P(x) V to obtain the third equality, and the last equality follows by using the integration by parts formula on M. For the first term in the last line of (41), similar to (39), we can obtain
(2Ta²) MMM Po())
(42)
For the second term in the last line of (1). in analogy to 33 we derive
Io Po(x)(( - P(x))
1
02 Po (d)
(43)
=(2mo?)- / po(eepp
((exp:(u))) expx(() (v)
5
(epppte al2
dv
o²
In this case, we have ē E M and x Therefore, the expansion (31 reduces eexp; (v) (1)
|u|2 + O(vl()) Applying (88 we then obtain
(T P(exp (x))) (v) - 5)
OP. (f) + 0(1o))
(())) (0) Σ
dxy
35'
(44)
OP, dzy
Σ = 2 j.j
+y +(u1))
where we have used (66 and (7 to derive the first and the second equality, respectively. We denote U E Raxd the matrix whose columns form orthonormal basis of M. It straightforward to verify that UTU = E Rdxd and P = UUT. We can compute (43 as
1o. (T(x) T P(x))
Pa (G(x
= =(mTO?) # /(po(()) + O(1u))
1
1
Σ
OP.
(T)yyty + 0(11)3))
Lu 20
du
2
day
3.
=(2m)- - =0 / -
1
Σ
OP
(z) + (
1'124000)
du'
dzy
j.j
=(2m) 28 30 d-n / [-
OP.j
(#)
Σ
U
Σ Uprwy)
(45)
Ww
2
dxy
3.3"
1=1
P=1
O(o))e
Wm 1m 2-00)
dw
=(2T)
2 dnn
Po(
Σ
OP. )s
Iw 2+0l0)
dw
dzy
W.
3.3'
1
OP,
=(2m) d-n
(T(
(2)
Day
23

---

where the third equality follows from a further change of variables with v' Uw and W. Uo O}. the fourth equality follows from [U /um2 and the fact that the integral
converges to an integral with respect to standard Gaussian density. Combining (41). 422 and (45)
and using the fact that the corresponding integral on M O' o(e =o²), we derive from
Vapo(T) Po (z)
oP,
V a Po)
V log
(46)
Day
aP1 Finally, define T(ž) L3.3'=1 Ja,y
(i) (2), and we have
P... Pos
a(P.P.s)
T(ě) =
Σ
Σ
P3f
dxy
oxy
38
(47)
OP. PjjP33' + P.i iPij
Σ
= +
Dzy
Ozy
358
This implies that ( 0. Therefore, multiplying P(ø) on both sides of (46) confirms 11)
The second claim is obtained.
B Proof of Theorem 4.1
Preparation.
We first study the properties of M, (z). Recall that for x E M,
(48)
By the Shermaa-Morrioon-Woobbbyy formula 31
Ex(x) σ²
1 2-20 Na))
(49)
1
P(x)
+
(I P(x)).
d2
22a
+ 22-20
Besides.
det E (x) = 22d 2022g 02-2a3n-d
(50)
Proof of the first claim. We first estimate the order of the following quadratic form:
(x 8) En 5)
(51)
- ( 2) P(x) - x) + 22a + 02 2a (x (1-
o2
We apply the same change of variables I = expx (v) as in the proof in Section A and æ
x² + O(w)(²). We obtain
(x- (x- (T==)()))) x)
=(x" 1 - 0((113)) (T-P(0)) E
OP(x*)
Ox un 0(10))) ) (s- 2+- +
(x+=++v+((u))) 0(1013)
=|x" - 2 +0(11))
(52)
24

---

and
(x - ((x)(( 2x)
- 10 - -+ + +((11)))
-(00
P(at)+ +- Σ
0p(x*)
0 P(x)))
Un + 0(1013)
+
Ozk
2
Owg
k
-
OP33
8 P(x")
=/u|2 +2
(x (x")0uy - 2,) + 2
Σ
33)a33 -By) + O(1o1))
drk
dr.dr
-o Sv + ((vu1))
(53)
The disappearance of the first-order terms in 52 and
53 is attributed to
PP(x")
-2)
(x"
Oik
Uk
O ((P)((x))
=(x" 2)²
Uk
(x" (x--ii -i)
dik
0P(x*) Uk
OP(x") Oerk
un vn)
-(x' P(=*)
(x' (x)
x)
+xx* 5)²²
Σ
P(o")(o" -5)
Ozk
-0,
(54)
and 5 E 1R is a matrix whose entries are defined by
Syy = d38 + Σ( P.. dx, 0zz
PP (x (x; 2.))
1
0 Pai(z")
(0) -x))(zf 21), (55)
dz 0r,0zz
K.J
for S 1.J
By the Gershgorin circle theorem
44 and the condition ||" i c0
min(1, n 2 +M,y) ). the matrix Š is also positive definite. From be written
53 and 55 Po (a (r) can
Po (o(xx) = (2t) 0 10 aa-aaa--- (1+ + [(1))
20 8 Bu-
(1v)))
(56)
Noting that (x -) (x))) 1xx- T) 2 Ioo J .P. there exists 8 >>>>>>>> such that (a T) (x) -1
(x) (x - T) ato 1x" - -
for all x E M O' where O' == exp,- (O). For any
E R. we have
nm M
|r x (20(())
<0 m e - -2pp- 20 / (2T)
7 det Vi +1 - x x=(po(()ddmmm)))
(57)
M
Se -2 ole 10a
Similarly, the order of -En (x) -t - 1) can be estimated as
(x)x) x))
P(z)(z -i)+ + 22a + 22-2x 20x (I- (Zx))(z - -))
02
1
Op
P(x") + Σ
(x (a')er + +0(1)3) (x-- - u uu
0((e3))
dik
(58)
+ 220
1
I I-P(a'")
Σ
0P
)u, + 0(1113) (x' (a'-z+va o(vl)))
+ 2a
dzk
k
1
1
1
(v + Hu + 0(1u13)) +
(x* - 5.- - Hu) 2a 0(101))
2r2 1 +
-2
2a
25

---

where we denote H E R with Hij = E1
PPa (2) z1). Therefore, 0zz
--
(2)
02a 1 82-2a x2-2aPo Po(
S Po(x)
x² 1
(Z) (2) (a x)
Po (o(
σ2a 1 -= 20
/ po(x)
1 0(wP)) po (((
(u+ Hu H 0(1o11))
H
2a
+
0²
22
2a
σ2a
1=1 olc ,,क
+ o2a
1
=
po(x*) + 0(1u))
02 (v - Ho 0(1211))
δ2 1 + 02-2a 2a
Hv + σ
0(Iu9))
E Zे o" But _ 0 + o(do)) du (2T) \ (o aqu-d ota 12p (1+ o(1)
+ - |-- 112 o(e a80)
= =/0 (po(z*) + +((o)) ( (1+ + H)u' 1 02-22 H + O(1)) e
-1-2a
d=)
(2T) - 7 aa--a adv |a -1 + e dd =11" oee - )नσ
=o(d-n)a - -a" ((011-20)00)
(59)
where the third equality follows from change of variables by e = or and O, = vo- E O}. and (()) + (a22-2)) (20(2-20)n Besides, using a similar derivation, we can obtain the expansion for Po.
(G) = (d-n)a stm la- (po(=") + O()))
(60)
The first claim is obtained by combining (59 and 60
Proof of the second claim. Now we consider the case where ă E M. In this case, E = x². By the change of variables x expa (v), the quadratic form becomes (a 2)7 E, (x) (z)= (x .)
-1
oz (|v|2 + (((ul)) + ح Olul Similar to (11 and 45) we aim to show that
/
(T)
V Po (alo)dd))))
M
(61)
=(2T)- # (d-nn)
1
1 Σ
V (0))())P)(()))000000-00))))))
J--=1
First, by integration by parts,
(T) - M V. (a
/ po(x) (2po(T()) V.M do
(62)
(x (x- x) = =/ po(x) (I - ((=)(
Σ
(a:
P.
-(E))-1
dra (x) - kt))
-2,)
2
i.j.k
po ((
where Σn refers to the (i, entry of the matrix Σ,, From 49
we have
1 1
1
(I (=)( x) 22a + 22-20 2a (I - P(x))( z) 02n 0(1o1) (63)
26

---

and
Σ
(1 P.)
(Ex) (z)(2 - -3y)
2
Dzk
1
(1i T. 0rk
Pip (z) - Tz) σ 01u))3)
(64)
2a
1.j.k
1
PP. (i)u,y + 0(1l))) 0(wo1))
1
P.k.
20
Ork
20
A.j.k
Next, we continue to calculate
62
using
63
and 664.
There exists > 0 and O' such that
2
(x (x=)))o(())~((
Vo Z
(65)
2a 20
02- 1
2a
22a
Therefore, the value of the integral (62 in 45). we have
M
O' is
ole
T
Using the same derivation as in
In ViM
Vapo(T)
/ po( + O(lol)
Σ iii
P.
OPss (i)eee + 20 20 0(10)) + -
1
2 0(1099)
20
Oak
(66)
Po (i) exp (t))du + oo( 1.
OP (Z) 0(0 toe 5 N
=(2T) n)a po (
Σ
P
Ozk
k.j."'
Therefore, 61 holds. Besides,
V Vis Pi
Σ ((P3)Pay)
35
Σ Papps + Σ PPppPapPps
(67)
k.j.
k.j.
Ppp
3.
implies Ej
Therefore, 61
becomes
(T) /
(a (2x) 3 o po()) (22-200)))))
(68)
Similarly, we have
/ (= = -2x)) (d-nna Po()) 0 ((((i))222)))))))))
(69)
Pa(#) po (
(2x) (d-nna (po(i) O(o (012))22))))
(70)
M
Using
68
69
and
70
the second claim is obtained via
V (T)
logpo
V.M logpo(() 0 11(2-22))))
(71)
(x)
C Algorithm details
Figure 2 illustrates the schematic diagrams of the vanilla method, along with our proposed Niso-DM and tango-DM.
27

---

soore function, v, ko
stangettial component of the score, P.X), Jog,, UK) nor mormall component of the scure, () P(x)) logas (x)
M
Iso DM Vanila mothods, muti scale singularity
Néc DM
Tango- OM Learn only the tangentill component
Reduce the the normal
Figure 2: Schematic diagrams of the vanilla method (left), our proposed Niso-DM (middle), and tango-DM (right).
C.1 Details of the Tango loss
We first prove the validity of the loss
20 Note that
Ez, (a,, P( (x.)
-Ez, [P(x))80(()) 1)1 2E,, (( P(x,) (x,)),).))) +
-E,, (P( 1)1 2 / (P(a.)so(E);) 1), V..Po
=R,, (P(2))))0) 1)1 - f ((())so(() 1), Vz, /
m Po, (2rllr)) dxy C
JR"
-E., PP(x)))))s t)11 - I (P(I))SO(TE, 1), Va, lo po, (2+(=)) Por (x)d))) + C
JRn M
-E., [c(, t)1 2E (P(z))s0(z1,t) t), Va, log Po. (xalT)) +C
-E,,e, (1s. t) ((x, log Po. (z) 12 C1,
(72)
where C anddd C1 are constants independent 0. Therefore, the minimizer of the loss 20) satisfies that (z, t) P( V log Po. (z). The overall loss calculation is summarized in Algorithm
Algorithm 1 The overall loss calculation with the Tango loss
Require: neural network sp, threshold Crango
1: Sample f (0, 1), x Po
2: if oe 2 Craago then
3: Calculate loss: Ar,,,,, [150 1) Vo, log (Ze 122
4: else
5: Calculate loss: A, P| P(x,) log Po. (z. [2)2
6: end if
C.2 Details of the sampling algorithm
Algorithms and 3 provide detailed descriptions of the Reverse SDE and the Annealing SDE on manifolds, respectively. The threshold & in Algorithm B set to Cniso and Otaugo for Niso-DM and Tango-DM, respectively.
D Experimental details
The details of each experiment are provided in the subsections below. The neural networks used are multilayer perceptrons (MLPs) with SiLU activations. Models are trained using PyTorch, utilizing the Adam optimizer with fixed learning rate, and gradients are clipped when their 2-norm exceeds predefined threshold. An exponential moving average of the model weights [30] is applied with
28

---

Algorithm 2 Reverse SDE Solver
Require: trained neural network 8a, total number of discrete SDE steps N, projection operator π
1: / +- T, Al T/N
2: Zt ~ (0, max
do 4: ~~ 10,1)
3: while
5: - 2 + g(t)²
(o t) g(t)
6: 1----L
7: end while
8: %Projection onto the manifold
9: 4- (x4)
10: return 21
Algorithm 3 Annealing SDE on manifolds
Require: trained neural network 8g, total number of discrete SDE steps N, threshold o, number of steps no and step size and for the Langevin dynamics on manifolds, projection operator π 1: t + T, At = T/N, 2t ~ (O, omax!
2: %Stage 1: Reverse SDE
3: while 2 ő do
4: 2 I)
5: St 4- Ut ((() () g(IVAl
6:
+-t-As
7: end while
8: %Stage 2: Annealing SDE manifolds
9: It + π (xx4)
10: while =000 do
11: for = to na do
12:
_N(0, I),
P(x)2
13: 2i + 2x ona ((x))s(((() + 20nazz
Vt 4 (x()
15: end for
t At
17: end while
return Z₁
decay rate of 0.999. In each run, the dataset divided into training and test with ratio 2. All experiments are conducted on either single Tesla V "100-PCIE-32GB GPU or NVIDIA A40 GPU with 48 GB of memory. The values of all parameters used in the experiments are listed in Table
When the rescaling technique not applied, the time reweighting coefficient X is defined as of. In contrast, when the rescaling technique is applied, X is defined aet,, where w; represents the scaling factor of the optimal score function. Specifically, w; corresponds to a in Iso-DM, Vol + 2i in Niso-DM, and max(or, ]rango} in Tango-DM.
D.1 Riemannian sliced score matching
Diffusion models constrained to manifold often rely complex geometric constructs, such as the heat kernel or logarithmic mapping. to compute transition probabilities. These dependencies limit the efficient computation of the denoising score matching loss. In contrast, Riemannian sliced
score matching (RSSM I7 offers broader applicability to general manifolds. This achieved by leveraging the implicit score matching loss [8), combined with Hutchinson trace estimation [18] The corresponding training loss expressed as:
ftssm(t, 0) = Ez, [0 t) + 21E 2. so,e)
(73)
29

---

Table 4: Parameters in our experiments. miins maxx and T are the parameters of the SDE. Cuiso and Crango are the parameters for Niso-DM and lango-DM, respectively. N, no, and Ond are the hyperparameters
the sampling algorithm. Napoch. B, Ir, and clip denote the total epochs, the batch size, the learning rate, and the gradient clipping threshold during training. N Nodde and Mayer represent the number of nodes per layer and the number of hidden layers in the neural networks, respectively. D indicates the dataset size.
Parameters
Hyperplane
Bunny
Spot
SO(10)
dipeptide
min
0.001
0.001
0.001
0.0005
0.0001
mmax T
3
1
Cniso
0.2
0.002
0.002
0.01
0.005
Otango
0.2
0.002
0.002
0.05
0.005
N
500
200
200
500
500
10
20
10
0.01
0.05
0.05
0.05
0.05
N Nuode
64
256
256
512
512
Mayer
3
3
N epoch
200
20000
20000
5000
6000
B
512
4096
4096
512
1024
Ir
0.0005
0,0005
0.0005
0.001
0.0005
clip
10
10
10
10
D
50000
60000
60000
50000
99999
Table 5: Results for the Hyperplane: MMD under different training methods. In this example. the Reverse SDE sampling method is applicable to Tango-DM due the complete decoupling of tangential and normal components. For more detailed explanations, refer to the caption of Table [
Reversal
Annealing
Iso
2.8le-445.66-5
7.97c-4=6396-
Niso
1.52e-44260-5 5.32c-4=3.0304
Tango
1.65e-41391es
5.82e-411.112
Isotres
2.79e-42941os
7.8.e-4=2.0004
Nisotres
1.45e-4126865
1.88e-4144499
Tangotres
1.19e-4#1.9655
2.20e-4=3.03e-
D.2
The hyperplane in 3D space
The target distribution is a mixture of Gaussian distributions with nine modes located on the plane.
Specifically, the means of the nine modes (-1,-1), (),,0), (-1, 1), (0, (0,-1), (0, 0), (0, 1), 11), (1, and (1, 1), and each Gaussian has a standard deviation of 0.3.
D.3
Mesh data
To create the datasets, we first obtain the clamped eigenfunctions of the Laplacian operator on a mesh that has been upsampled threefold for the original mesh. The target distribution is chosen as an proporton mixture of density functions corresponding to the O-th, 500-th, and 1000-th eigenpairs. For the distribution on the cow mesh, we additionally discarded the points on the horns and tail.
For & .,, the closest point (xx) on triangular mesh surface is determined as follows: First, projected onto the planes all triangular faces the mesh using the face normals and vertex positions. Barycentric coordinates are then computed to check whether the projected points lie inside the triangles. If a point falls outside a triangle, is further projected onto the nearest edge to ensure remains on the triangle's 's boundary. Subsequently, the Euclidean distance between the original query
30

---

point I' and all projected points is calculated, and the closest point (2)) is selected by identifying the minimum distance.
We assign the normal direction (x((x)) of π (x) as the normal direction for x (c é M). When (a) is located inside triangle of the mesh, the normal of the triangle is chosen the normal vector at T(x) (i.e. n(x) = (x))). However, when (I) lies on an edge (or vertex) of the mesh, it is simultaneously associated with two (or three) triangles. In this case, we select the normal vector of the triangle with the smallest index as the normal direction for π (z). Here, we leverage the property of the torch. argmin function.
D.4 High-dimensional special orthogonal group
The manifold (1() is defined as {Q E 1000010 QQ = 10; det(Q) = 1]. which represents (a connected component of) the zero-level set of the map & 1000 + 1R55 The components of correspond to the upper triangular portion of the matrix QQ /10- The dataset is constructed as a mixture of 5 wrapped normal distributions. Each wrapped normal distribution is the image (under the exponential map) of a normal distribution defined in the tangent space at a pre-selected center. For more details, refer [3 In addition, Figure 3 shows that compared with Iso-DM, the samples from Niso-DM have a better match with the ground truth.
Lr(S)
tr(S?)
Lr((
0.5
0.25
0:35
0.4
0.20
0.25
0.6
0.3
0,15
0.20
0.10
0.15
0.10
0.2
0.1
0.05
0.0
0.0
0.00
0.00
Figure 3: Results for SO())). Empirical densities of the statistics tr(S), tr(S²), tr(S*), and tr(s#) for Iso-DM (red), Niso-DM (purple) and the ground truth (green). Samples from Niso-DM (purple) better match the ground truth (green).
D.5 Alanine dipeptide
To generate the dataset, we follow 2 to obtain the target distribution on the manifold M {ar ((()) -70%) } We also ensure that the generated distribution is SE(3) -invarrint (i.e., invariant under rotations and translations) by incorporating alignment before feeding the data into the neural network.
E Ablation Study
Recall that the noise scale or is chosen as d: Juin (omaa/ )m/T )t/T and 0 < amin < 1. When t is sufficiently small, o, approaches omin. and the singularity of the score function primarily depends on amin- As a decreases, the singularity becomes pronounced.
CAL9
N:7
C:'5 5
For Iso-DM, there exists a bias-variance trade-off in the choice of mmin- A very small Jmin reduces bias but in- creases variance, resulting in a larger overall error. In this case, V, log Pamin (z) approximates V, log Po(z) well; however, severe multiscale discrepancies between its tan- gential and normal components hinder learning. Con- versely, a very large mmin increases bias and reduces vari- ance, yet the overall error still increases. In this scenario,
Figure 4: Illustration of the Alanine dipeptide system: The dihedral angles & and v are defined by atoms whose in- dices are 5, 7,9, 15 and 7, 9, 15. 17, re- spectively. This figure from 233
V_, log Po (x) fails to serve good approximation of V, log Po (r(),, although the singularity issue less pronounced. Figure 5a luuttattss this phenomenon, where the red line represents the error for M...
31

---

For Niso-DM, the singularity is governed by the additional noise scale Cuiso- rather than min- As result, our method exhibits greater robustness as min decreases. The blue line in Figure Sa demonstrates the stability of the error for Niso-DM under varying mmin-
Figures 5bland 5c show the impact of Cuiso in Niso-DM and Crango in Tango-DM on the error. The selection of these hyperparameters also involves a bias-variance trade-off.
-10-3
12
10
10
1644
56-4
562
0.001
0.005
0.55
0.53
(a) Error VS
(b) Error
(c) Error vs Ctango
Figure 5: Ablation Studies for SO(10): The solid line denotes the mean, while the shaded area indicates the standard deviation. (a) The error of the distribution generated by the Reverse SDE algorithm under the Iso-DM (red) and Niso-DM (blue) methods with varying min- (b) The impact of Cuiso the error with min 0.001. (c) The impact of Ctango the error with min 0.001.
In Tables G
present the numerical results of ablation studies mmin Cnisos and Crango for the (10) experiment. We report computational errors under different parameter settings to assess the sensitivity of our method hyperparameter choices. The results demonstrate that reductions in hyperparameter values may not substantially affect the performance of our method.
Table 6: The impact of mnin the under the Niso algorithm.
0 min
le-5
le-4
5e-4
le-3
lc-2
Iso
9.520-3-12563
8.97e-3=19463
9,49e-341.9103
182
6.70e-3423863
Niso
7.170-3423e3
6.1
4..24-348.444
5.62e-3t143e3
6.23e-3#1.88-3
Table The impact of Cniso on the error under the Niso algorithm.
Cniso Niso
lc-3
5e-3
lc-2
5e-2
9.52c-321.44-
5.47e-341.6043
1.626-3=14463
7.92c-3=2.7663
F
Discussion
Computational cost of Tango-DM
While Tango-DM tends to be slower due to the annealing sampler, the runtime is primarily determined by the number of steps of the inner Langevin dynamics. For example, in the toy experiment R2inR3, the sampling time for Reverse SDE is 1.68 seconds, while for Annealing SDE, the sampling time increases to 5.62 seconds with an inner step 5 and to 9.40 seconds with an inner step of 10. The first phase of Annealing SDE (see Algorithm 3) requires only 0.82 seconds. Overall, the computational cost of Annealing SDE is approximately 3-6 times higher than that of Reverse SDE. These measurements, conducted standard laptop, highlight the relative computational expense.
F.2 Learned manifold case
Unlike previous works on distributions on known manifold, our method avoids relying geometric information like geodesics or heat kernels, greatly reducing computational complexity. However, compared to approaches under the manifold hypothesis, our method still requires knowledge of the manifold's definition, including projection operators. fields like image and language processing, the manifold structure is often assumed but not explicitly known.
32

---

Table
The impact min on the error under the Tango algorithm.
min Niso
le-5
le-4
5e-4
le-3
le-2
2.12 1.05
1.97
6.05e-3428263
4.11e-24182e1
Table
The impact of Ctango on the enror under the Tango algorithm.
Stango
5e-3
le-2
5e-2
lc-1
Tango 6.95e-347.37644
5.64c-3+189-3
6.05e-3=28263
8.466-3426703
Next, we discuss how to extend our method to learned manifolds, using the AutoEncoder as an example. Specifically, first train Encoder de, : M + Rd and Decoder da, R M. which provide a parameterized representation of the manifold. The subspace spanned by Viea, (0o, (1)) corresponds the tangent space M point the manifold. Once the basis of the tangent space obtained, the score function can be further decomposed into its tangential and normal components, which enables the implementation of our proposed methods. Intuitively, the success of this approach hinges on the Autoencoder's ability to accurately capture the underlying manifold structure. In this work, leave this approach as direction for future research.
F.3 Assumptions in Theorem 3.1 and Theorem 4.1
The conclusions in Theorem 3.] and Theorem hold pointwise on the manifold, so in the proof, only require local boundedness. Specifically, the following assumption is sufficient:
For any x E M, there exists >> 0, such that supyEMS maxigiad'Sn
aPu Byy
(v)
< +00,
where M {arg min e. | x 1 j zl < 6).
To make the theorem statement more concise, we adopted stronger assumption:
S max Si.jd's
8P Zee
(a))
+00.
For compact manifolds, the uniform boundedness assumptions always hold, which implies that the local boundedness assumptions hold. Similarly, the boundedness assumption in Theorem 4. can also be relaxed to local boundedness.
G Impact statement
This paper contributes to the advancement of generative models for data with manifold structures, providing deeper understanding the singularity of the score function. Specifically, we identify the scale discrepancies between the tangential and normal components of the score function, which sheds light on key challenges modeling data on manifolds. We believe that our work bridges the gap between generative models specifically designed for manifolds and those aimed at handling data under manifold assumptions. While this study may have broader societal implications, none require particular emphasis at this stage.
33


Reviews:
[{"review_id": "m0miseQs9M", "reviewer": "Reviewer_42Gv", "summary": "The paper studies the behavior of Variance Exploding Diffusion models when the data distribution is supported on a $d-$dimensional submanifold embedded in $R^n$. Based on this theoretical analysis, two new diffusion models are proposed, Niso-DM and Tango-DM, which can be used to learn densities supported on known low-dimensional sub-manifolds. Empirical studies are conducted to support the theoretical results.", "questions": "1. In section 2.1, what is the $\\sigma_T^2$? You did not define it and the sentence in line 67 is confusing as $p_T$ approximates the true limiting distribution.\n\n2. I don't understand Figure 1. The tangential component is in this case the 3. coordinate of the learned score function. Close to the manifold, that is $t\\to 0$, this error decrease as seen by the funnel structure. However, the colorcode suggests that the average error increased closer to the manifold with the highest error for Iso+res. Would you mind elaborating?\n\n3. I noticed different metrics for evaluating the performance on difference datasets. 1-Wasserstein, sliced Wasserstein, MMD, Jensen-Shannon divergence. Can you provide some reasons for that? Can you use a single metric for all? Can you provide a counterexample of a manifold where the assumption do not hold?\n\n4. How strong are the assumptions of Theorem 3.1 and Theorem 4.1? In particular, the $M_1$ and $M_2$ boundedness?", "limitations": "The assumptions of Theorem 3.1 and Theorem 4.1 are not fully discussed, see Question 4 above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Using the variance exploding SDE, the peturbed distribution at time $t$ is essentially a convolution of $p_0$ with the noise distribution $p_{t}(\\tilde{x} | x)$. This observation is exploited in Theorem 3.1 to analyze the behavior of the true score close to the manifold and on the manifold when $t \\to 0$. Using the resulting decomposition of the score, the diffusion loss functions can be neatly decomposed as well. \n\nTheorem 4.1 provides a good theoretical analysis of Niso-DM showing that the score of $p_0$ on the manifold can be well approximated using the learned score under certain condition. \n\nThe proposed new algorithms are straightforward consequences of the theory and show good empirical results. Unfortunately, only the vanilla diffusion model is used as a benchmark. A comparison to other, similar methods is lacking. A practitioner who wants to decide which method to use for his/her manifold data is left behind.  \n\nIn terms of novelty, a very similar analysis was conducted in the context of using Normalizing Flows to learn densities on low-dimensional manifolds [1]. In particular, Theorem 4 and Proposition 7 in [1] confirm equation (10), (12) and (18). Nevertheless, the decomposition (13) and the derivation of the Theorems is novel to the best of my knowledge. However, a thorough comparison to [1] is needed in the Related Work Section in my opinion.\n\nFinally, I would like the authors to have a look at this paper [2]. They show that a conservative score function behaves very differently close to manifold data than a non-conservative score function. I wonder how your results will change when you use a conservative vector field as a score, that is, defining $s_{\\theta}$ as the gradient of some scalar function. I conjecture that the results will improve even further.\n\nGiven insufficient related work section, the lack of comparison to other methods and some mathematical flaws (see additional comments below), I lean towards rejection in the current version.\n\n[1] *Density estimation on low-dimensional manifolds: an inflation-deflation approach*, Christian Horvat, Jean-Pascal Pfister.\n[2] *On gauge freedom, conservativity and intrinsic dimensionality estimation in diffusion models*, Christian Horvat, Jean-Pascal Pfister.\n\n\nSome other interesting literature which might be interesting for you:\n+ *Diffusion Models Encode the Intrinsic Dimension of Data Manifolds*, Jan Pawel Stanczuk \n\n\n*Additional comments*:\n+ line 38, mention that $d<n$. \n+ mention citation [31] in Section 2.1\n+ Please properly explain the $\\nabla^{\\mathcal{M}}$ operator in the main text.\n+ Please use consistent notation for Expectation. Equation (4) is using $x \\sim p$, equation (5) is using $x$ and Theorem 3.1 is using $p$.\n+ Please explain all the relevant notation to understand Theorem 4.1. How is $\\wedge$ defined?\n+ Some References on volume forms on manifolds and equation (8) should be added in Section 2.2.\n+ Typo in line 284. Table 1 is not in the Appendix.", "was_revised": true, "final_justification": "The author resolved my 2 main concerns: first, the lack of discussion with [1] and other related methods, and second, the lack of comparison with another diffusion-based manifold method. I still think the method will benefit from using a proper score-function instead of an unconstrained one and encourage the authors to try it out.\n\nOverall, the theoretical contribution is solid outweighting the limited evaluation with other methods. For the method to be seen as a high-impact in the area of learning on manifold densities for known manifolds, a more thorough evaluation is needed including Normalizing Flow and other methods. Therefore, I recommend a weak acceptance.\n\n[1] *Density estimation on low-dimensional manifolds: an inflation-deflation approach*, Christian Horvat, Jean-Pascal Pfister.", "rebuttal": "We appreciate the reviewer’s valuable feedback and suggestions.\n\n## Strengths\n\nWe thank the reviewer for their positive feedback on our work. In Theorem 3.1, we exploit the observation to analyze the behavior of the true score both close to and on the manifold, allowing for a neat decomposition of the diffusion loss functions. The proposed algorithms, derived directly from the theoretical insights, demonstrate strong empirical performance.\n\n\n## Weaknesses\n### W1: Baseline\n\n\nFollowing the reviewer’s suggestion, we have conducted Riemannian diffusion models (RDM, [*]) on our experiments for comparison. This algorithm, based on sliced score matching and Hutchinson estimation, is applicable to general known manifolds. Therefore, we use RDM as a baseline and conduct experiments on Mesh data and SO(10). RDM failed in the Alanine dipeptide case, which we suspect may be due to the limitation of sliced score matching when applied to SE(3)-equivariant distributions.\n\nTable 1 presents the comparison between the vanilla algorithm (with a final projection step), our method (using Niso+res with Reverse SDE), and RDM (baseline). Our method consistently outperforms the baseline. Specifically, in the SO(10) case, our approach outperforms RDM, while the vanilla algorithm does not. For Mesh data, RDM does not perform well, which can be explained by the fact that RDM relies on defining a diffusion process on the manifold to achieve a uniform distribution. Due to the geometric structure of the manifold, such as the narrow neck of \"Spot the Cow\", achieving a uniform distribution via SDEs on manifolds proves challenging.\n\nTable 1: Computational errors for Mesh data, and SO(10). The results are reported as the mean and standard deviation over five independent runs. \n\n| Method   | Bunny           | Spot            | SO(10)          |\n|----------|------------------|-----------------|-----------------|\n| RDM      | 7.00e-1 (3.85e-3) | 7.22e-1 (3.32e-3) | 7.14e-3 (9.09e-4) |\n| Vanilla  | 3.52e-1 (2.17e-3) | 3.30e-1 (2.08e-3) | 9.49e-3 (1.91e-3) |\n| Ours     | **3.48e-1** (8.27e-4) | **3.28e-1** (6.19e-4) | **4.60e-3** (8.24e-4) |\n\nWe appreciate the reviewer’s suggestion and hope that these additional experiments will be taken into account when reevaluating our work. We will include these additional experiments in the revised version of the paper.\n\n\n[*] Huang, Chin-Wei, et al. \"Riemannian diffusion models.\" Advances in Neural Information Processing Systems 35 (2022): 2750-2761.\n\n### W2: Relation to Paper [1]\n\nWe thank the reviewer for pointing out this highly relevant paper and its similarities to our work. Indeed, the idea of adding noise along the normal direction is similar between the two studies. We will include the following discussion in a revised version.\n\n> Paper [1] proposes adding noise along the normal direction to \"inflate\" manifolds, which is similar to our niso method. Theorem 4 and Proposition 7 in [1] analyze the relationship between the perturbed distribution and the original distribution on the manifold, which aligns with conclusions implied by our equations (12) and (18).\n>\n> On the other hand, there are several key differences. In [1], the added noise has a constant magnitude, requiring additional assumptions (e.g., Q-normally reachable) to prevent interference between noise at different points. In contrast, our analysis (Theorem 3.1) allows the additional noise magnitude to approach zero, relying on weaker assumptions. Additionally, [1] applies noise inflation prior to training the normalizing flow, whereas our method integrates noise addition directly into the diffusion process, aligning with the noise dynamics of diffusion models. Finally, while the analysis in [1] focuses on the perturbed density function, our work studies the asymptotic behavior of the score function.\n\n### W3: Additional References\n\nWe thank the reviewer for pointing out these related works. We will include citations to these papers in the Related Work section.\n\n\n## Additional comments\n\nWe thank the reviewer for their careful reading of our paper. We will make the necessary revisions accordingly. The operation $\\wedge$ is defined as $a \\wedge b = \\min[a, b]$, and the definition of $\\nabla^{\\mathcal{M}}$ is as follows:  $\\nabla^{\\mathcal{M}}f(x) := P(x)\\nabla_{x}f(x)$ for any function $f$.\n\n## Questions\n\n### Q1: Refrasing\nThis section can indeed be written more clearly. In the revised version, we will rephrase the sentence as follows:\n\n> By carefully selecting $f$ and $g$, the distribution $p_{T}$ can be well approximated as a Gaussian distribution $\\mathcal{N}(0, \\sigma_T^2 I)$ at time $T$, where $\\sigma_T^2$ is the variance of $X_T$.\n\n\n### Q2: Explanation of Figure 1\n\nThe tangential component corresponds to the first two coordinates (rather than the third) of the learned score function, as the manifold constraint ensures that the third component is zero. The tangential component is the first two coordinates of the learned score function. From a theoretical perspective, due to the scale discrepancy we mentioned, accurate estimation of the tangential component becomes harder as $t \\to 0$, resulting in increased error. The experimental results are consistent with this theoretical analysis: the average error increased as $t \\to 0$. Our method (the bottom two plots) reduces this error. The magnitude of the error is not related to the funnel structure. The funnel structure reflects the effective range of the distribution. As $t \\to 0$, the distribution concentrates near $z = 0$, which leads to the funnel shape.\n\n\n### Q3: Utilized Metrics\n\nIn our setting, the data is strictly distributed on a submanifold of the space $\\mathbb{R}^n$. **Ideally, metrics for evaluation should compare distributions based on distances defined on the manifold itself. However, such computations are generally infeasible. As a practical alternative, we calculate distances between distributions in the entire space** (with the exception of JS divergence for mesh data, which effectively captures differences on the manifold). Due to the unique properties of each manifold and the distributions defined on them, no single metric can reliably evaluate performance across all datasets. For instance, in the case of $\\mathrm{SO}(10)$, which is embedded in a high-dimensional space, the 1-Wasserstein distance in $\\mathbb{R}^n$ is computationally expensive and does not effectively capture distribution differences on the manifold, making sliced Wasserstein a better choice. Besides, for $\\mathbb{R}^2$ embedded in $\\mathbb{R}^3$, we observe that the variance of the 1-Wasserstein distance is large, while the variance of MMD is more stable.\n\n### Q4: Assumptions in Theorems\nOur conclusions hold for points on the manifold, so in the proof, we only require local boundedness. Specifically, the following assumption is sufficient:\n\n$$\n\\forall x \\in \\mathcal{M},\\exists \\delta > 0, \\text{ s.t. } \\sup_{y \\in \\mathcal{M}^{\\delta}(x)} \\max_{1 \\leq i,j,j' \\leq n} \\left| \\frac{\\partial P_{ij}}{\\partial y_{j'}}(y) \\right| < +\\infty,\\text{ where }  \\mathcal{M}^{\\delta}(x)= [  \\arg\\min_{z \\in \\mathcal{M}} |\\tilde{x} - z|^2 :  |\\tilde{x} - x| < \\delta ].\n$$\nTo make the theorem statement more concise, we adopted a stronger assumption:\n$$\n\\sup_{x \\in \\mathcal{M}} \\max_{1 \\leq i,j,j' \\leq n} \\left| \\frac{\\partial P_{ij}}{\\partial x_{j'}}(x) \\right| < +\\infty.\n$$\n\nFor compact manifolds, the uniform boundedness assumptions always hold, and consequently, the local boundedness assumptions hold. We can always assume that the manifold is bounded, as the sample data we obtain always have finite values.\n\n## Limitations:\n\nWe will include a discussion on the assumptions of Theorem 3.1 and Theorem 4.1 in the paper, as mentioned in the reply to Question 4 above.\n\n\n## Final Words\n\nOverall, our work provides a mathematically explicit analysis of the singular structure of the score function along its tangential and normal components, even though the divergence of the score function has already been widely studied. Furthermore, we propose two methods to improve the performance of diffusion models by mitigating the scale discrepancy of the score function.\n\nWhile we acknowledge that some relevant papers were overlooked in the current version, we believe the novelty of our approach is still evident. **We will incorporate discussions of the related work in the revised version.** Once again, we sincerely thank the reviewer for their constructive feedback."}, {"review_id": "ZBeS6ySMYz", "reviewer": "Reviewer_3q97", "summary": "When trying to learn data that lies on a lower dimensional manifold with diffusion models that operate on Euclidean embeddings, the optimal score function exhibits a singularity at small noise scales $\\sigma \\rightarrow 0$. This work attempts to characterize this singularity and proposes two changes to the loss function to remove it from the training process. The authors first split the score function into a tangential and normal component w.r.t. the manifold and note that around $\\sigma \\rightarrow 0$ the loss contribution of the former grows as $O(1)$, while the latter as $O(\\sigma^{-1})$. Therefore the normal component dominates the optimization process (manifold overfitting) at small noise scales. The work proceeds by proposing two variants of the diffusion loss function to fix this: \n1. Niso-DM uses non-isotropic Gaussian noise for small $\\sigma$, where the noise in the normal direction is increased, giving direct control over the behaviour of the score function in the normal direction.  \n2. Tango-DM only learns the tangential components of the score function in the regime of small $\\sigma$. Sampling in this regime is achieved by annealing the reverse SDE restricted on the manifold, which circumvents the need to know the score in the normal direction.   \n\nThe authors show that all presented methods produce the correct on-manifold distributions (after projecting the samples to the manifold if required). Experimentally the authors validate their methods on a GMM on a 2-d plane embedded in 3-d, two synthetic mesh datasets, a synthetic dataset on SO(10) and alanine dipeptide with one restricted dihedral angle.", "questions": "Additionally to the questions in Strengths and Weaknesses:  \n- l.190 Why only \"almost surely\"?\n- Could you apply your method to learned manifolds (e.g. as represented by an AutoEncoder)?\n- Why does f(x, t) not appear in eq. (21, 22)?", "limitations": "There are limitations related to runtime (sampling speed of the annealed SDE, manifolds with expensive projections) which, depending on how relevant they are in practice, should potentially appear in appendix F. I would be interested in how much slower the annealed SDE sampling is in the authors' experience. Otherwise the limitations seem to be complete.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The work gives a detailed characterization of the score function behavior for the original diffusion loss as well as the two proposed variants. Both are well motivated\n- Experiments are extensive and oriented closely around benchmarks used for e.g. on-manifold ODE/SDE models.\n- Niso-DM is a simple change to the noise schedule, which requires little design overhead and knowledge about the manifold to implement.\n- Both training methods introduce little computational overhead, provided that projections to tangent/normal space are cheap  \n- Definitions and theoretical results are presented in a very comprehensible way, I especially enjoyed the introduction to diffusion models in Sec. 2.1\n\n**Weaknesses**\n- Projecting/calculating normal + tangential directions for noised samples in the embedding space can lead to problems if the manifold is not differentiable. Sec. 2.2 suggests that you assume a differentiable manifold, however this is contradictory to the experiments on mesh datasets. Can you explain how you compute normal/tangential components in regions of the embedding space that are projected to an edge or a corner of the mesh?\n- Tango-DM can only be used in conjunction with the annealed sampler in the general setting. This sampler seems to not only under-perform in the experiments but is also computationally more expensive. Could you provide a rough overview of how much slower it is in practice?\n- There are no baseline methods in the experiments, which incorporate any information about the manifold structure (e.g [1, 2]). Is there a reason you do not compare against them?\n\n**Minor Weaknesses**\n- For Niso-DM there exists a bias-variance trade-off between contracting closely to the manifold and avoiding the singularity. The authors analyze this trade-off in appendix E and show that it is still favorable over unmodified diffusion\n- A naive solution to the singularity would be to just add (normal direction) noise to the training data (moving it off the manifold), then projecting the samples of the diffusion model back onto the manifold. This approach seems very closely related to your final version of Niso-DM, where the off-manifold noise level is set to a constant in the initial steps. Are there significant differences that I missed? Do you have a motivation for introducing the more general formulation of Niso-DM including $\\alpha_t$ and have you investigated the effect of $\\alpha_t$ apart from using it as in l.180?\n\nIn conclusion, the work provides insightful theoretical results and of the proposed novel methods, Niso-DM consistently performs well. The information on practical considerations and experiments seems to be only partially complete, which is why I will only vote to borderline accept for now.\n\n[1] De Bortoli, Valentin, et al. \"Riemannian score-based generative modelling.\" Advances in neural information processing systems 35 (2022): 2406-2422.  \n[2] Huang, Chin-Wei, et al. \"Riemannian diffusion models.\" Advances in Neural Information Processing Systems 35 (2022): 2750-2761.", "was_revised": true, "final_justification": "The contributions of this work are twofold: It provides a theoretical characterization of the score function singularity when modelling on-manifold data with diffusion models and proposes two fixes.  \nWhile the authors acknowledge that this singularity has been known in the literature they claim to be the first to rigorously analyze its behavior in this way. While I am personally not very familiar with the related work on the topic, from the discussions of reviewers vZYH, TsMq and 42Gv it seems that the promised update to the related work section is sufficient to distinguish this work from previous ones on the topic.  \nOf the proposed methods only Niso-DM can consistently beat the best performance of the baseline method. Including the second method, Tango-DM, which performs worse due to the underperformant annealed sampler does not detract from the paper. There exist other methods, which incorporate the manifold structure that were not included as baselines in the paper. The authors have since provided these results in the discussion and will add them to the revised version of the paper. Technical questions of training on non-differentiable manifolds have been adequately explained in the discussion below.  \nI lean towards accepting the paper as it provides a rigorous theoretical analysis of the score singularity for manifold data and proposes a practical fix. My rating is not higher because, while the proposed fixes are more elegantly formulated than the naive solution of moving the data off-manifold before training, it is unclear whether they provide a consistent advantage in practice.", "rebuttal": "We appreciate the reviewer’s valuable questions and suggestions.\n\n## Strengths\nWe appreciate the reviewer for their positive evaluation of our work. Our study provides a detailed characterization of the score function behavior for both the original diffusion loss and the two proposed variants, each of which is well motivated and thoughtfully developed.\n\n## Weaknesses\n\n### W1: Smoothness of Manifolds\n\nIndeed, when describing our algorithm, we assume the manifold to be smooth, and part of the mesh dataset satisfies this assumption. The manifold $\\mathcal{M}$ corresponding to the mesh dataset is a polyhedron composed of triangles. On each face (triangle), the manifold is smooth, as it can be considered a part of a plane. **The non-smooth points are the shared edges between adjacent faces, which form a zero-measure set.** Therefore, this manifold is almost everywhere smooth, and our proposed algorithm is still applicable to this manifold without contradiction. In the implementation, for $x \\in \\mathcal{M}$, the normal direction of $x$ is the normal vector of the triangle (plane) that $x$ belongs to. The normal components are the projections of the score along this normal vector. For points lying on the shared edges (with zero probability of occurrence), we arbitrarily select the normal vector of one of the adjacent faces as the normal direction for that point. We will add more details about this example in the main text, as discussed above.\n\n### W2: Performance of Tango\n\nWe argue that Tango does not always underperform. In fact, our Tango algorithm still outperforms the vanilla diffusion model when using the annealed sampler. Moreover, in the dipeptide example, the optimal result was achieved with the Tango algorithm rather than Niso. At this stage, there is no definite conclusion regarding which method—Niso or Tango—is superior. \n\nWhile Tango tends to be slower due to the annealing sampler, **the runtime is primarily determined by the number of steps of the inner Langevin dynamics.** For example, in the toy experiment R2inR3, the sampling time for Reverse SDE is 1.68 seconds, while for Annealing SDE, the sampling time increases to 5.62 seconds with an inner step of 5 and to 9.40 seconds with an inner step of 10. The first phase of Annealing SDE (see Algorithm 3) requires only 0.82 seconds. Overall, **the computational cost of Annealing SDE is approximately 3–6 times higher than that of Reverse SDE.** These measurements, conducted on a standard laptop, highlight the relative computational expense. A detailed discussion on computational costs will be included in the appendix.\n\n\n### W3: Baseline\n\nFollowing the reviewer’s suggestion, we have conducted the algorithm in [1] on our experiments for comparison. Diffusion models constrained to the manifold often rely on complex geometric information, such as the heat kernel or logarithmic mapping, to compute transition probabilities, which limits the computation of the denoising score matching loss. In contrast, the algorithm from Riemannian diffusion models (RDM, [2]), based on sliced score matching (or implicit score matching in [1]) and Hutchinson estimation, is applicable to more general manifolds. Therefore, we use the RDM algorithm as a baseline and conduct experiments on Mesh data and SO(10). However, RDM failed in the Alanine dipeptide case, which we suspect may be due to the limitation of sliced score matching when applied to SE(3)-equivariant distributions.\n\nTable 1 presents the comparison between the vanilla algorithm (with a final projection step), our method (using Niso+res with Reverse SDE), and RDM (baseline). Our method consistently outperforms the baseline. Specifically, in the SO(10) case, our approach outperforms RDM, while the vanilla algorithm does not. For Mesh data, RDM does not perform well, which can be explained by the fact that RDM relies on defining a diffusion process on the manifold to achieve a uniform distribution. Due to the geometric structure of the manifold, such as the narrow neck of \"Spot the Cow\", achieving a uniform distribution via SDEs on manifolds proves challenging.\n\nTable 1: Computational errors for Mesh data and SO(10). The results are reported as the mean and standard deviation over five independent runs. \n\n| Method   | Bunny           | Spot            | SO(10)          |\n|----------|------------------|-----------------|-----------------|\n| RDM      | 7.00e-1 (3.85e-3) | 7.22e-1 (3.32e-3) | 7.14e-3 (9.09e-4) |\n| Vanilla  | 3.52e-1 (2.17e-3) | 3.30e-1 (2.08e-3) | 9.49e-3 (1.91e-3) |\n| Ours     | **3.48e-1** (8.27e-4) | **3.28e-1** (6.19e-4) | **4.60e-3** (8.24e-4) |\n\n\nWe appreciate the reviewer’s suggestion and hope that these additional experiments will be taken into account when reevaluating our work. We will include these additional experiments in the revised version of the paper.\n\n\n## Minor Weaknesses\n\n### W4: Bias-Variance Trade-off\n\nWe thank the reviewer for carefully reading our appendix. As one of the fundamental philosophyies in machine learning, the bias-variance tradeoff phenomenon is widely observed in parameter selection scenarios. In this ablation study, we demonstrate that our algorithm is still favorable over unmodified diffusion. We do not consider this a weakness but rather evidence further supporting the advantages of our algorithm.\n\n### W5: Non-isotropic Noise\nIn the implementation, we chose $\\sigma_t^{\\alpha_t} = c_{\\rm iso}$ as a constant, which aligns with the reviewer's understanding.  However, our approach is not entirely equivalent to pre-adding noise to the data before training, as our loss function (see Eq. (19)) employs non-isotropic score matching. The general formulation in the theorem was introduced to rigorously prove the scale differences from a mathematical perspective. While it is possible that selecting a more flexible $\\alpha_t$ might lead to better results, for simplicity in our numerical experiments, we only investigated the case where $\\alpha_t = \\log c_{\\rm iso} / \\log \\sigma_t$ in the paper.\n\n\n## Questions\n\n### Q1：'Almost surely'\n\nIn the appendix, we prove that $\\ell_{\\text{tango}}(t,\\theta) = \\mathbb{E}_{x, x_t} \\|s^{\\parallel}_{\\theta}(x_t,t)-P(x_t)\\nabla_{x_t}\\log p_{\\sigma_t}(x_t)\\|^2 + C_1$. \n\nThus, when $s$, $p$, and $P$ are continuous functions, $s^{\\parallel}_{\\theta}(x_t,t) = P(x_t)\\nabla_{x_t}\\log p_{\\sigma_t}(x_t)$ **always holds, not just \"almost surely.\" We will remove the phrase \"almost surely\" in the updated version.**\n\n\n### Q2: Learned Manifold Case\n\nIn principle, our method can be extended to learned manifolds. As suggested by the reviewer, we use the AutoEncoder as an example. At first, we train an Encoder $\\phi_{\\theta_1}: \\mathcal{M} \\to \\mathbb{R}^d$ and a Decoder $\\psi_{\\theta_2}: \\mathbb{R}^d \\to \\mathcal{M}$, which provide a parameterized representation of the manifold. The subspace spanned by $\\nabla\\psi_{\\theta_2}(\\phi_{\\theta_1}(x))$ corresponds to the tangent space $T\\mathcal{M}_x$ at point $x$ on the manifold. Once the basis of the tangent space is obtained, the score function can be further decomposed into its tangential and normal components, which enables the implementation of our proposed methods. Intuitively, the success of our method on learned manifolds hinges on the Autoencoder's ability to accurately capture the underlying manifold structure. However, in this work, we do not plan to implement this approach and instead leave it as a direction for future research.\n\n### Q3: f(x,t) Term\n\nIn this paper, we focus on the Variance Exploding SDE (VESDE), characterized by $f(x, t) = 0$. To ensure clarity for readers, we will explicitly highlight this in the explanations of equations (21) and (22).\n\n## Final Words\n\nOverall, our work provides a mathematically explicit analysis of the singular structure of the score function along its tangential and normal components, even though the divergence of the score function has already been widely studied. Furthermore, we propose two methods to improve the performance of diffusion models by mitigating the scale discrepancy of the score function. If one aims to improve accuracy by precisely computing the tangential component near $t=0$, we believe our approach captures the essence of the problem and has the potential to inspire future research.\n\nOnce again, we sincerely thank the reviewer for their constructive feedback."}, {"review_id": "edHFQiFfnG", "reviewer": "Reviewer_vZYH", "summary": "The paper identifies, for data on a low-D manifold, different magnitude of the score matching loss tangential and normal to the local manifold. The authors thus proposed two modifications to the score function for know manifolds. Experiments show that the modifications, usually when using the \"epsilon\" equivalent of the score matching loss, gives improvements on toy and real datasets.", "questions": "1. I'm curious about the sensitivity of hyperparameters mentioned before.\n2. Can you combine niso and tango with some weighting to get even better performance?", "limitations": "I think the limitation are not clearly described, but only touched on during writing. Please at least mention that the method requires knowing the manifold, which is not always the case in machine learning applications, e.g. on images.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strengths\n1. Many real-world data distributions fall under the manifold hypotheses. This paper addresses the score discrepancies within and normal to the manifold. This result is not entirely new, but as far as I know, the analyses the authors provide are first in the field. \n2. Using knowledge of the manifold, the authors proposed two intuitive methods to address the problem. These methods are clearly described with pros and cons discussed. \n3. The paper is written clearly with proper use of mathematical notations, and I find it easy to learn and absorb the non-trivial contributions and ideas. \n4. The authors tested the proposed remedy on a wide range of datasets and problems, with promising results.\n\n# Weaknesses:\n\n1. Using knowledge of the manifold is a big assumption, and is suitable only for low dimensional problems.  \n2. A lack of baseline methods that also use knowledge of the given manifold. Are the authors propose to use manifold information? Or to use manifold information in the particular ways the authors propose? If the former, then this is somewhat trivial; if the latter, then we need some other baselines. A easy baseline is to improve the iso with manifold information: project the samples to or towards the manifold with some parameters. This should give us a the performance of a naive diffusion + manifold method. \n3. A lack of Ablations and sensitivity analyses: for tango, how important is the choice of $\\sigma_t$ that we train only the tangent score? For niso, how sensitive is the performance to $\\alpha_t$?  \n4. The authors cited a few related work, but I think it's missing important conplementary / congruent prior work on showing the discrepancy of within- and out-of-manifold scores, though with less mathematical formalism and without using a given manifold function $\\xi$. \n    1. [*] uses a heuristic argument to show that the score perpenticular to noisy data of images is mostly Gaussian-like, whereas the score within the manifold as additional properties (no directly relevant for the current submission)\n    2. [**] Essentially uses the score descrepancy to identify the manifold.\n    3. [***] Describes denoising in terms of bases functions. In particular, Eqn (12) also embodies the scale discrepancy the authors of the current paper describe here.\nAll of these methods do not assume a given manifold, so it would be great to add these to the Related work, part 1 where you mention other methods that do not rely on a known manifold. \n\n[*] Wenliang, Li Kevin, and Ben Moran. \"Score-based generative models learn manifold-like structures with constrained mixing.\" NeurIPS 2022 Workshop on Score-Based Methods.\n\n[**] Stanczuk, Jan, et al. \"Your diffusion model secretly knows the dimension of the data manifold.\" arXiv preprint arXiv:2212.12611 (2022).\n\n[***] Kadkhodaie, Zahra, et al. \"Generalization in diffusion models arises from geometry-adaptive harmonic representations.\" The Twelfth International Conference on Learning Representations.", "was_revised": true, "final_justification": "The authors provided new / pointed to existing experimental results that resolved my concerns. The related work are more thoroughly acknolwedged. Limitations more explicitly discussed.", "rebuttal": "We sincerely thank the reviewer for the thoughtful comments and valuable recommendations.\n\n## Strengths\n\nWe thank the reviewer for their positive evaluation of our work. By addressing the score discrepancies within and normal to the manifold, our analyses provide a novel contribution to the field. Leveraging manifold knowledge, we proposed two intuitive methods to tackle this problem effectively.\n\n## Weaknesses \n\n### W1: Manifold Assumption\n\nOur current algorithm requires explicit knowledge of the manifold structure, which indeed limits its applicability to image or language data, where the manifold is often assumed but not explicitly known.\n\nOne possible approach to extending our algorithm is to train an Autoencoder to learn the manifold structure at first. Specifically, this involves learning an Encoder $\\phi_{\\theta_1}: \\mathcal{M} \\to \\mathbb{R}^d$ and a Decoder $\\psi_{\\theta_2}: \\mathbb{R}^d \\to \\mathcal{M}$, which serve as a parameterized representation of the manifold. The subspace spanned by $\\nabla\\psi_{\\theta_2}(\\phi_{\\theta_1}(x))$ corresponds to the tangent space at point $x$ on the manifold. After obtaining a basis of the tangent space, we can further decompose the score function into its tangential and normal components. In principle, our method can also work on manifolds learned via an Autoencoder. However, in this work, we do not plan to implement this approach and instead leave it for future research. \n\n\n### W2: Baseline\n\nYes, we aim to use manifold information in these particular ways. The reviewer's suggestion is excellent and precisely reflects our existing approach. The results of the baselines (Iso or Iso+res) shown in the paper are conducted exactly as suggested by the reviewer. Specifically, we use the Reverse SDE sampling algorithm, which includes a projection step as the final step (see lines 205-207 in the main text). In other words, the Iso and Iso+res methods in our paper have already been combined with manifold information. We will make this information more explicit in the revised version of the paper.\n\n\nBesides, we have conducted Riemannian diffusion models (RDM, [*]) on our experiments for comparison. This algorithm, based on sliced score matching and Hutchinson estimation, is applicable to general known manifolds. Therefore, we use RDM as a baseline and conduct experiments on Mesh data and SO(10). \n\nTable 1 presents the comparison between the vanilla algorithm (with a final projection step), our method (using Niso+res with Reverse SDE), and RDM (baseline). Our method consistently outperforms the baseline. Specifically, in the SO(10) case, our approach outperforms RDM, while the vanilla algorithm does not. For Mesh data, RDM does not perform well, which can be explained by the fact that RDM relies on defining a diffusion process on the manifold to achieve a uniform distribution. Due to the geometric structure of the manifold, such as the narrow neck of \"Spot the Cow\", achieving a uniform distribution via SDEs on manifolds proves challenging.\n\nTable 1: Computational errors for Mesh data and SO(10). The results are reported as the mean and standard deviation over five independent runs.\n\n| Method   | Bunny           | Spot            | SO(10)          |\n|----------|------------------|-----------------|-----------------|\n| RDM      | 7.00e-1 (3.85e-3) | 7.22e-1 (3.32e-3) | 7.14e-3 (9.09e-4) |\n| Vanilla  | 3.52e-1 (2.17e-3) | 3.30e-1 (2.08e-3) | 9.49e-3 (1.91e-3) |\n| Ours     | **3.48e-1** (8.27e-4) | **3.28e-1** (6.19e-4) | **4.60e-3** (8.24e-4) |\n\n\n[*] Huang, Chin-Wei, et al. \"Riemannian diffusion models.\" Advances in Neural Information Processing Systems 35 (2022): 2750-2761.\n\n\n### W3: Ablation Study\n\nFollowing the reviewer’s suggestion, we have added ablation studies to evaluate the sensitivity of our method to parameters. In our algorithm, $\\sigma_{\\rm min}$, $c_{\\rm niso}$, and $c_{\\rm tango}$ are relatively important hyperparameters. Under the $SO(10)$ experiment, we present the computational errors for different parameter choices. Some of the results below are already contained in Fig. 3 in Appendix E of the paper.\n\nTable 2: The impact of $\\sigma_{\\rm min}$ on the error under the Niso algorithm.\n| $\\sigma_{\\rm min}$ | 1e-5             | 1e-4             | 5e-4             | 1e-3             | 1e-2             |\n| ------------------ | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- |\n| Iso                | 9.52e-3(1.25e-3) | 8.97e-3(1.94e-3) | 9.49e-3(1.91e-3) | 7.30e-3(2.18e-3) | 6.70e-3(2.33e-3) |\n| Niso               | 7.17e-3(2.33e-3) | 6.18e-3(1.16e-3) | 4.60e-3(8.24e-4) | 5.62e-3(1.43e-3) | 6.23e-3(1.88e-3) |\n\n\nTable 3: The impact of $c_{\\rm niso}$ on the error under the Niso algorithm.\n| $c_{\\rm niso}$ | 1e-3             | 5e-3             | 1e-2             | 5e-2             |\n| -------------- | ---------------- | ---------------- | ---------------- | ---------------- |\n| Niso           | 9.52e-3(1.64e-3) | 5.47e-3(1.60e-3) | 5.62e-3(1.43e-3) | 7.92e-3(2.27e-3) |\n\n\nTable 4: The impact of $\\sigma_{\\rm min}$ on the error under the Tango algorithm.\n| $\\sigma_{\\rm min}$ | 1e-5             | 1e-4             | 5e-4             | 1e-3             | 1e-2             |\n| ------------------ | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- |\n| Niso               | 8.60e-3(1.05e-3) | 8.12e-3(2.33e-3) | 6.42e-3(1.97e-3) | 6.05e-3(2.82e-3) | 4.11e-2(1.82e-3) |\n\n\nTable 5: The impact of $c_{\\rm niso}$ on the error under the Tango algorithm.\n| $c_{\\rm tango}$ | 5e-3             | 1e-2             | 5e-2             | 1e-1             |\n| --------------- | ---------------- | ---------------- | ---------------- | ---------------- |\n| Tango           | 6.95e-3(7.37e-4) | 5.64e-3(1.89e-3) | 6.05e-3(2.82e-3) | 8.46e-3(2.67e-3) |\n\n\nThese results indicate that reducing the hyperparameters does not significantly impact the performance of our method. Moreover, with different $\\sigma_{\\rm min}$ (see Table 1), our algorithm consistently outperforms the vanilla method (iso+res). We will include all these numerical results in our paper.\n\n\n### W4: Citation\nWe appreciate the reviewer’s suggestions and will incorporate discussions of these three references into Part 1 of the Related Work section. Specifically, we will include the following points:\n\n> Wenliang et al. show that the score perpendicular to noisy data of images is mostly Gaussian-like, whereas the score within the manifold has additional properties. The work by Stanczuk et al. essentially uses the score discrepancy to identify the intrinsic dimension of the manifold. Kadkhodaie et al. describe denoising in terms of basis functions, which implies the scale discrepancy we discussed. In particular, the Jacobian eigendecomposition of the denoiser mapping (Eq. (12)) implicitly captures these scale differences, aligning with Eq. (10) in our paper. However, our work explicitly reveals this phenomenon in manifold settings using more formal mathematical formulations and provides solutions to address it.\n\n\n## Questions\n\n### Q1: Hyperparameters\n\nSee the Weaknesses section (W3).\n\n### Q2: Combination of NISO and Tango\n\nNISO and Tango handle the normal direction in fundamentally different ways: NISO seeks to relax the scale of the normal component, whereas Tango directly discards it. In the paper, NISO and Tango are introduced as two independent algorithms. It seems infeasible to effectively combine them as a new approach.\n\n\n## Limitations\nWe will address this limitation explicitly in the main text. As follows:\n\n> Unlike previous works on distributions on a known manifold, our method avoids relying on geometric information like geodesics or heat kernels, greatly reducing computational complexity. However, compared to approaches under the manifold hypothesis, our method still requires knowledge of the manifold’s definition, including projection operators. In fields like image and language processing, the manifold structure is often assumed but not explicitly known. Future work will focus on extending our approach to scenarios where the manifold is undefined or implicit.\n\n## Final Words\n\nOverall, our work provides a mathematically explicit analysis of the singular structure of the score function along its tangential and normal components, even though the divergence of the score function has already been widely studied. Furthermore, we propose two methods to improve the performance of diffusion models by mitigating the scale discrepancy of the score function.\n\nWhile we acknowledge that some relevant papers were overlooked in the current version, we believe the novelty of our approach is still evident. We will incorporate discussions of the related work in the revised version. Once again, we sincerely thank the reviewer for their constructive feedback."}, {"review_id": "EzZBIEvxdw", "reviewer": "Reviewer_TsMq", "summary": "The paper identifies a potential issue with generative diffusion with data supported on manifolds, the (well known)  fact that the orthogonal component of the score diverges as 1/sigma for t-> 0 while its tangential component approaches a constant. Based on this observation, the authors suggest a series of methods to stabilize both training and sampling essentially by separating the treatment of the two components. Their training method trains the orthogonal and tangential component separately and it can only be directly used when the m manifold structure is known. The increase in performance for toy models is small when compared with the standard and widely used rescaled approach, where the divergent term 1/sigma is factored out from the network.", "questions": "Can your method be used in real data where the manifold structure is not know? If not, is it possible to modify the approach so that it can be used for images or other kinds of low-dimensional data?", "limitations": "I would have apprecoiated to see a proper discussion of the limitations in the main text.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "This paper seems to imply that standard generative diffusion cannot deal well with manifold data. I think that this assertion is wrong as standard techniques can perform very well on data supported on manifolds and in fact they shine in this regime since alternative likelihood based methods are destabilized by the singularity of the loss function (see manifold overfitting [1]). The divergence of the score has already been studied extensively, see for example in [2,3,4,5,6]. The analysis in [3] is particularly interesting since it suggests that the ‘irrelevance’ of the tangential component of the score near zero is a feature, not a bug as in that phase (manifold consolidation) the role of the score is to project the samples on manifold since the intrinsic distribution of the manifold has been determined at earlier times (manifold coverage phase). This interpretation explains why,in general, diffusion models do not suffer from manifold overfitting. \n\nSo said, I do think that this topic definitely deserves further analysis and I do think that the method proposed by the authors can have value in specific scenarios where the correct estimation of the tangential components at times close to zero is vital. The experimental results are interesting and I can see the proposed methods to have some use in scientific applications.\n\nA major problem with the paper is that it ignores a very large body of existing literature on the subject. It is vital to properly engage with previous work and it seems that the authors missed some very relevant papers. Here is a selected list:\n\n[1] Stanczuk, Jan, et al. \"Your diffusion model secretly knows the dimension of the data manifold.\" arXiv preprint arXiv:2212.12611 (2022).\n[2] Ventura, Enrico, et al. \"Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusion.\" ICLR (2025)\n[3] Kadkhodaie, Zahra, et al. \"Generalization in diffusion models arises from geometry-adaptive harmonic representations.\" ICLR (2024)\n[4] Wang, Peng, et al. \"Diffusion models learn low-dimensional distributions via subspace clustering.\" arXiv preprint arXiv:2409.02426 (2024).\n[5] Achilli, Beatrice, et al. \"Memorization and Generalization in Generative Diffusion under the Manifold Hypothesis.\" arXiv preprint arXiv:2502.09578 (2025).\n[6] George, Anand Jerry, Rodrigo Veiga, and Nicolas Macris. \"Analysis of Diffusion Models for Manifold Data.\" arXiv preprint arXiv:2502.04339 (2025).\n\nI highly recommend the authors to incorporate this literature in their paper.\n\n[edit] I in creased my score due to the work of the authors on incorporating a discussion  of the relevant literature.", "was_revised": true, "final_justification": "I do think that the revised manuscript will be useful to the community as it formally characterize an important phenomenon and suggests an interesting mitigation technique \nThe authors should include the extended coverage of the literature in the revised manuscript.", "rebuttal": "We are grateful to the reviewer for the careful review and constructive comments. \n\n## Summary\n\nFirst, we would like to kindly point out that the reviewer’s summary of our work is somewhat unfair and inaccurate. The reviewer claims that \"the orthogonal component of the score diverges as $1/\\sigma$ for $t \\to 0$ while its tangential component approaches a constant\" is a \"well-known fact.\" This statement is unfair. The truly well-known fact is the divergence of the score and manifold overfitting phenomenon when the data is supported on manifolds, rather than this detailed statement about the behavior of its orthogonal and tangential components we provided. Although some earlier works such as **[1], [3] implicitly hinted** at the scale differences in spirit, the observation that \"the orthogonal component of the score diverges as $1/\\sigma$ for $t \\to 0$ while its tangential component is of order 1\" is **explicitly and mathematically formulated by us for the first time in the known manifold setup, accompanied by rigorous mathematical proofs**. Reviewers 42Gv and vZYH have also acknowledged that our result on the scale discrepancy is the first analysis in this field.\n\nRegarding the experimental evaluation, the reviewer stated that \"the increase in performance for toy models is *small*.\" We respectfully disagree with this assessment. We believe these experiments effectively demonstrate the validity of our proposed method and confirm the correctness of our theoretical analysis.\n\n## Strengths And Weaknesses\n\n\nWe have **never claimed that \"standard generative diffusion cannot deal well with manifold data\"**, which seems to be the viewpoint that the reviewer is attempting to refute. **Our actual argument is that mitigating the scale discrepancy of the score function can improve the performance of diffusion models, while it is true that diffusion models outperform likelihood-based methods.**\n\nThe statement that \"the 'irrelevance' of the tangential component of the score near zero is a feature, not a bug, as in that phase (manifold consolidation) the role of the score is to project the samples onto the manifold since the intrinsic distribution of the manifold has been determined at earlier times (manifold coverage phase)\" **can, at best, explain why diffusion models can work in manifold settings or outperform likelihood-based methods in such scenarios.** However, **the precision of the intrinsic/internal distribution on the manifold determined at earlier times (manifold coverage phase) may not be sufficient**. The tangential component of the score near zero time (manifold consolidation phase) **allows for a more refined characterization of the intrinsic distribution of the manifold** and consequently is not an \"irrelevant\" part. This is one of the motivations behind our proposed methods, i.e., Niso and Tango. We believe that this statement may not be sufficient to serve as evidence against the arguments in our paper.\n\nWe believe that the reviewer has fully understood our motivation, as evidenced by their statement: \"… have value in specific scenarios where the correct estimation of the tangential components at times close to zero is vital.\" However, we would like to argue that this is not limited to specific scenarios but rather represents a general setting.\n\nIn fact, if one truly aims to accurately compute the tangential component at times close to zero, we believe our methods effectively address this problem and lay the foundation for future studies. **As shown in Figure 1 in our paper, our method significantly reduces the error of the tangential component at times close to zero**, thereby improving overall accuracy.\n\n**We apologize for overlooking relevant literature and thank the reviewer for highlighting these important references**. We greatly appreciate their insightful feedback and thorough knowledge of the field. Specifically, we find reference [3] interesting because its Jacobian eigendecomposition of the denoiser mapping also implicitly reflects scale differences, which aligns with the statements in our paper.\n\nWe will incorporate the following discussion of these papers in the related work section in the revised version.\n\n\n> The work [1] by Stanczuk et al. essentially uses the score discrepancy to identify the intrinsic dimension of the manifold. Kadkhodaie et al. [3] describe denoising in terms of basis functions, which implies the scale discrepancy we discussed. In particular, the Jacobian eigendecomposition of the denoiser mapping (Eq. (12)) implicitly captures these scale differences, aligning with Eq. (10) in our paper. However, our work explicitly reveals this phenomenon in manifold settings using more formal mathematical formulations and provides solutions to address it. References [4] [5] [6] study the fundamental principles of learning manifold-structured data distributions through Gaussian Mixture Models, Hidden Manifold Models, and Generalized Linear Models. References [1] and [2] also investigate the intrinsic geometry of generative diffusion models by studying the eigendecomposition of the Jacobian of the score functions. Moreover, we believe that our proposed methods can further improve the accuracy of diffusion models during the manifold consolidation phase as described in [2].\n\n\n## Questions: Real Data Case\n\nOur current algorithm requires explicit knowledge of the manifold structure, which indeed limits its applicability to image or language data, where the manifold is often assumed but not explicitly known.\n\nOne possible approach to extending our algorithm is to train an Autoencoder to learn the manifold structure. Specifically, this involves learning an Encoder $\\phi_{\\theta_1}: \\mathcal{M} \\to \\mathbb{R}^d$ and a Decoder $\\psi_{\\theta_2}: \\mathbb{R}^d \\to \\mathcal{M}$, which serve as a parameterized representation of the manifold. The subspace spanned by $\\nabla\\psi_{\\theta_2}(\\phi_{\\theta_1}(x))$ corresponds to the tangent space at point $x$ on the manifold. After obtaining a basis of the tangent space, we can further decompose the score function into its tangential and normal components. In principle, our method can also work on manifolds learned via an Autoencoder. However, in this work, we do not plan to implement this approach and instead leave it for future research. \n\nWe will add a discussion on this limitation in the main text (see next section).\n\n## Limitations\n\nWe will address this limitation explicitly in the main text. As follows:\n\n> Unlike previous works on distributions on a known manifold, our method avoids relying on geometric information like geodesics or heat kernels, greatly reducing computational complexity. However, compared to approaches under the manifold hypothesis, our method still requires knowledge of the manifold’s definition, including projection operators. In fields like image and language processing, the manifold structure is often assumed but not explicitly known. Future work will focus on extending our approach to scenarios where the manifold is undefined or implicit.\n\n## Final Words\n\nOverall, our work provides a mathematically explicit analysis of the singular structure of the score function along its tangential and normal components, even though the divergence of the score function has already been widely studied. Furthermore, we propose two methods to improve the performance of diffusion models by mitigating the scale discrepancy of the score function. If one aims to improve accuracy by precisely computing the tangential component near $t=0$, we believe our approach captures the essence of the problem and has the potential to inspire future research.\n\nWhile we acknowledge that some relevant papers were overlooked in the current version, we believe the novelty of our approach is still evident. We will incorporate discussions of the related work in the revised version. Once again, we sincerely thank the reviewer for their constructive feedback."}]

Author rebuttal:

Meta-review: The paper studies the behavior of diffusion models under the realistic assumption of the manifold hypothesis, where the data distribution is supported on a lower dimensional manifold embedded in the ambient data space. The authors present theoretical analysis of the singularity that arises in the diffusion score function as it approaches the manifold in normal or tangential directions. Based on the analysis, two new diffusion models are proposed which try to resolve these singularities, and experiments are conducted to show improvements in the behaviour of these models.

The reviewers reached a consensus that the analysis in this paper is rigorous and useful. From the initial reviews, several issues were raised: failure to cite, discuss, and engage with the body of relevant literature; lack of baseline methods and ablations; unrealistic or impractical assumptions; and other more minor issues.

Most of the technical issues were addressed during the discussions, including with additional baselines and better discussion of limitations. The most common criticism was simply poor comparison to existing literature. The authors have given a longer write up of related work during the discussion which covered all the topics brought up by reviewers.

Hence, I am recommending to accept this submission. The authors must incorporate the changes brought up with reviewers into their final submission.
Decision: Accept (poster)