Title: Improving the Euclidean Diffusion Generation of Manifold Data by Mitigating Score Function Singularity
Venue: NeurIPS 2025
Accepted: yes
Keywords: diffusion models

submanifold

score function

singularity
Abstract: Euclidean diffusion models have achieved remarkable success in generative modeling across diverse domains, and they have been extended to manifold cases in recent advances. Instead of explicitly utilizing the structure of special manifolds as studied in previous works, in this paper we investigate direct sampling of the Euclidean diffusion models for general manifold-structured data. We reveal the multiscale singularity of the score function in the ambient space, which hinders the accuracy of diffusion-generated samples. We then present an elaborate theoretical analysis of the singularity structure of the score function by decomposing it along the tangential and normal directions of the manifold. To mitigate the singularity and improve the sampling accuracy, we propose two novel methods: (1) Niso-DM, which reduces the scale discrepancies in the score function by utilizing a non-isotropic noise, and (2) Tango-DM, which trains only the tangential component of the score function using a tangential-only loss function. Numerical experiments demonstrate that our methods achieve superior performance on distributions over various manifolds with complex geometries.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper identifies a potential issue with generative diffusion with data supported on manifolds, the (well known)  fact that the orthogonal component of the score diverges as 1/sigma for t-> 0 while its tangential component approaches a constant. Based on this observation, the authors suggest a series of methods to stabilize both training and sampling essentially by separating the treatment of the two components. Their training method trains the orthogonal and tangential component separately and it can only be directly used when the m manifold structure is known. The increase in performance for toy models is small when compared with the standard and widely used rescaled approach, where the divergent term 1/sigma is factored out from the network.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper seems to imply that standard generative diffusion cannot deal well with manifold data. I think that this assertion is wrong as standard techniques can perform very well on data supported on manifolds and in fact they shine in this regime since alternative likelihood based methods are destabilized by the singularity of the loss function (see manifold overfitting [1]). The divergence of the score has already been studied extensively, see for example in [2,3,4,5,6]. The analysis in [3] is particularly interesting since it suggests that the ‘irrelevance’ of the tangential component of the score near zero is a feature, not a bug as in that phase (manifold consolidation) the role of the score is to project the samples on manifold since the intrinsic distribution of the manifold has been determined at earlier times (manifold coverage phase). This interpretation explains why,in general, diffusion models do not suffer from manifold overfitting. \n\nSo said, I do think that this topic definitely deserves further analysis and I do think that the method proposed by the authors can have value in specific scenarios where the correct estimation of the tangential components at times close to zero is vital. The experimental results are interesting and I can see the proposed methods to have some use in scientific applications.\n\nA major problem with the paper is that it ignores a very large body of existing literature on the subject. It is vital to properly engage with previous work and it seems that the authors missed some very relevant papers. Here is a selected list:\n\n[1] Stanczuk, Jan, et al. \"Your diffusion model secretly knows the dimension of the data manifold.\" arXiv preprint arXiv:2212.12611 (2022).\n[2] Ventura, Enrico, et al. \"Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusion.\" ICLR (2025)\n[3] Kadkhodaie, Zahra, et al. \"Generalization in diffusion models arises from geometry-adaptive harmonic representations.\" ICLR (2024)\n[4] Wang, Peng, et al. \"Diffusion models learn low-dimensional distributions via subspace clustering.\" arXiv preprint arXiv:2409.02426 (2024).\n[5] Achilli, Beatrice, et al. \"Memorization and Generalization in Generative Diffusion under the Manifold Hypothesis.\" arXiv preprint arXiv:2502.09578 (2025).\n[6] George, Anand Jerry, Rodrigo Veiga, and Nicolas Macris. \"Analysis of Diffusion Models for Manifold Data.\" arXiv preprint arXiv:2502.04339 (2025).\n\nI highly recommend the authors to incorporate this literature in their paper.\n\n[edit] I in creased my score due to the work of the authors on incorporating a discussion  of the relevant literature.\n\nQUESTIONS:\nCan your method be used in real data where the manifold structure is not know? If not, is it possible to modify the approach so that it can be used for images or other kinds of low-dimensional data?\n\nLIMITATIONS:\nI would have apprecoiated to see a proper discussion of the limitations in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper identifies, for data on a low-D manifold, different magnitude of the score matching loss tangential and normal to the local manifold. The authors thus proposed two modifications to the score function for know manifolds. Experiments show that the modifications, usually when using the \"epsilon\" equivalent of the score matching loss, gives improvements on toy and real datasets.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. Many real-world data distributions fall under the manifold hypotheses. This paper addresses the score discrepancies within and normal to the manifold. This result is not entirely new, but as far as I know, the analyses the authors provide are first in the field. \n2. Using knowledge of the manifold, the authors proposed two intuitive methods to address the problem. These methods are clearly described with pros and cons discussed. \n3. The paper is written clearly with proper use of mathematical notations, and I find it easy to learn and absorb the non-trivial contributions and ideas. \n4. The authors tested the proposed remedy on a wide range of datasets and problems, with promising results.\n\n# Weaknesses:\n\n1. Using knowledge of the manifold is a big assumption, and is suitable only for low dimensional problems.  \n2. A lack of baseline methods that also use knowledge of the given manifold. Are the authors propose to use manifold information? Or to use manifold information in the particular ways the authors propose? If the former, then this is somewhat trivial; if the latter, then we need some other baselines. A easy baseline is to improve the iso with manifold information: project the samples to or towards the manifold with some parameters. This should give us a the performance of a naive diffusion + manifold method. \n3. A lack of Ablations and sensitivity analyses: for tango, how important is the choice of $\\sigma_t$ that we train only the tangent score? For niso, how sensitive is the performance to $\\alpha_t$?  \n4. The authors cited a few related work, but I think it's missing important conplementary / congruent prior work on showing the discrepancy of within- and out-of-manifold scores, though with less mathematical formalism and without using a given manifold function $\\xi$. \n    1. [*] uses a heuristic argument to show that the score perpenticular to noisy data of images is mostly Gaussian-like, whereas the score within the manifold as additional properties (no directly relevant for the current submission)\n    2. [**] Essentially uses the score descrepancy to identify the manifold.\n    3. [***] Describes denoising in terms of bases functions. In particular, Eqn (12) also embodies the scale discrepancy the authors of the current paper describe here.\nAll of these methods do not assume a given manifold, so it would be great to add these to the Related work, part 1 where you mention other methods that do not rely on a known manifold. \n\n[*] Wenliang, Li Kevin, and Ben Moran. \"Score-based generative models learn manifold-like structures with constrained mixing.\" NeurIPS 2022 Workshop on Score-Based Methods.\n\n[**] Stanczuk, Jan, et al. \"Your diffusion model secretly knows the dimension of the data manifold.\" arXiv preprint arXiv:2212.12611 (2022).\n\n[***] Kadkhodaie, Zahra, et al. \"Generalization in diffusion models arises from geometry-adaptive harmonic representations.\" The Twelfth International Conference on Learning Representations.\n\nQUESTIONS:\n1. I'm curious about the sensitivity of hyperparameters mentioned before.\n2. Can you combine niso and tango with some weighting to get even better performance?\n\nLIMITATIONS:\nI think the limitation are not clearly described, but only touched on during writing. Please at least mention that the method requires knowing the manifold, which is not always the case in machine learning applications, e.g. on images.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWhen trying to learn data that lies on a lower dimensional manifold with diffusion models that operate on Euclidean embeddings, the optimal score function exhibits a singularity at small noise scales $\\sigma \\rightarrow 0$. This work attempts to characterize this singularity and proposes two changes to the loss function to remove it from the training process. The authors first split the score function into a tangential and normal component w.r.t. the manifold and note that around $\\sigma \\rightarrow 0$ the loss contribution of the former grows as $O(1)$, while the latter as $O(\\sigma^{-1})$. Therefore the normal component dominates the optimization process (manifold overfitting) at small noise scales. The work proceeds by proposing two variants of the diffusion loss function to fix this: \n1. Niso-DM uses non-isotropic Gaussian noise for small $\\sigma$, where the noise in the normal direction is increased, giving direct control over the behaviour of the score function in the normal direction.  \n2. Tango-DM only learns the tangential components of the score function in the regime of small $\\sigma$. Sampling in this regime is achieved by annealing the reverse SDE restricted on the manifold, which circumvents the need to know the score in the normal direction.   \n\nThe authors show that all presented methods produce the correct on-manifold distributions (after projecting the samples to the manifold if required). Experimentally the authors validate their methods on a GMM on a 2-d plane embedded in 3-d, two synthetic mesh datasets, a synthetic dataset on SO(10) and alanine dipeptide with one restricted dihedral angle.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The work gives a detailed characterization of the score function behavior for the original diffusion loss as well as the two proposed variants. Both are well motivated\n- Experiments are extensive and oriented closely around benchmarks used for e.g. on-manifold ODE/SDE models.\n- Niso-DM is a simple change to the noise schedule, which requires little design overhead and knowledge about the manifold to implement.\n- Both training methods introduce little computational overhead, provided that projections to tangent/normal space are cheap  \n- Definitions and theoretical results are presented in a very comprehensible way, I especially enjoyed the introduction to diffusion models in Sec. 2.1\n\n**Weaknesses**\n- Projecting/calculating normal + tangential directions for noised samples in the embedding space can lead to problems if the manifold is not differentiable. Sec. 2.2 suggests that you assume a differentiable manifold, however this is contradictory to the experiments on mesh datasets. Can you explain how you compute normal/tangential components in regions of the embedding space that are projected to an edge or a corner of the mesh?\n- Tango-DM can only be used in conjunction with the annealed sampler in the general setting. This sampler seems to not only under-perform in the experiments but is also computationally more expensive. Could you provide a rough overview of how much slower it is in practice?\n- There are no baseline methods in the experiments, which incorporate any information about the manifold structure (e.g [1, 2]). Is there a reason you do not compare against them?\n\n**Minor Weaknesses**\n- For Niso-DM there exists a bias-variance trade-off between contracting closely to the manifold and avoiding the singularity. The authors analyze this trade-off in appendix E and show that it is still favorable over unmodified diffusion\n- A naive solution to the singularity would be to just add (normal direction) noise to the training data (moving it off the manifold), then projecting the samples of the diffusion model back onto the manifold. This approach seems very closely related to your final version of Niso-DM, where the off-manifold noise level is set to a constant in the initial steps. Are there significant differences that I missed? Do you have a motivation for introducing the more general formulation of Niso-DM including $\\alpha_t$ and have you investigated the effect of $\\alpha_t$ apart from using it as in l.180?\n\nIn conclusion, the work provides insightful theoretical results and of the proposed novel methods, Niso-DM consistently performs well. The information on practical considerations and experiments seems to be only partially complete, which is why I will only vote to borderline accept for now.\n\n[1] De Bortoli, Valentin, et al. \"Riemannian score-based generative modelling.\" Advances in neural information processing systems 35 (2022): 2406-2422.  \n[2] Huang, Chin-Wei, et al. \"Riemannian diffusion models.\" Advances in Neural Information Processing Systems 35 (2022): 2750-2761.\n\nQUESTIONS:\nAdditionally to the questions in Strengths and Weaknesses:  \n- l.190 Why only \"almost surely\"?\n- Could you apply your method to learned manifolds (e.g. as represented by an AutoEncoder)?\n- Why does f(x, t) not appear in eq. (21, 22)?\n\nLIMITATIONS:\nThere are limitations related to runtime (sampling speed of the annealed SDE, manifolds with expensive projections) which, depending on how relevant they are in practice, should potentially appear in appendix F. I would be interested in how much slower the annealed SDE sampling is in the authors' experience. Otherwise the limitations seem to be complete.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the behavior of Variance Exploding Diffusion models when the data distribution is supported on a $d-$dimensional submanifold embedded in $R^n$. Based on this theoretical analysis, two new diffusion models are proposed, Niso-DM and Tango-DM, which can be used to learn densities supported on known low-dimensional sub-manifolds. Empirical studies are conducted to support the theoretical results.\n\nSTRENGTHS AND WEAKNESSES:\nUsing the variance exploding SDE, the peturbed distribution at time $t$ is essentially a convolution of $p_0$ with the noise distribution $p_{t}(\\tilde{x} | x)$. This observation is exploited in Theorem 3.1 to analyze the behavior of the true score close to the manifold and on the manifold when $t \\to 0$. Using the resulting decomposition of the score, the diffusion loss functions can be neatly decomposed as well. \n\nTheorem 4.1 provides a good theoretical analysis of Niso-DM showing that the score of $p_0$ on the manifold can be well approximated using the learned score under certain condition. \n\nThe proposed new algorithms are straightforward consequences of the theory and show good empirical results. Unfortunately, only the vanilla diffusion model is used as a benchmark. A comparison to other, similar methods is lacking. A practitioner who wants to decide which method to use for his/her manifold data is left behind.  \n\nIn terms of novelty, a very similar analysis was conducted in the context of using Normalizing Flows to learn densities on low-dimensional manifolds [1]. In particular, Theorem 4 and Proposition 7 in [1] confirm equation (10), (12) and (18). Nevertheless, the decomposition (13) and the derivation of the Theorems is novel to the best of my knowledge. However, a thorough comparison to [1] is needed in the Related Work Section in my opinion.\n\nFinally, I would like the authors to have a look at this paper [2]. They show that a conservative score function behaves very differently close to manifold data than a non-conservative score function. I wonder how your results will change when you use a conservative vector field as a score, that is, defining $s_{\\theta}$ as the gradient of some scalar function. I conjecture that the results will improve even further.\n\nGiven insufficient related work section, the lack of comparison to other methods and some mathematical flaws (see additional comments below), I lean towards rejection in the current version.\n\n[1] *Density estimation on low-dimensional manifolds: an inflation-deflation approach*, Christian Horvat, Jean-Pascal Pfister.\n[2] *On gauge freedom, conservativity and intrinsic dimensionality estimation in diffusion models*, Christian Horvat, Jean-Pascal Pfister.\n\n\nSome other interesting literature which might be interesting for you:\n+ *Diffusion Models Encode the Intrinsic Dimension of Data Manifolds*, Jan Pawel Stanczuk \n\n\n*Additional comments*:\n+ line 38, mention that $d<n$. \n+ mention citation [31] in Section 2.1\n+ Please properly explain the $\\nabla^{\\mathcal{M}}$ operator in the main text.\n+ Please use consistent notation for Expectation. Equation (4) is using $x \\sim p$, equation (5) is using $x$ and Theorem 3.1 is using $p$.\n+ Please explain all the relevant notation to understand Theorem 4.1. How is $\\wedge$ defined?\n+ Some References on volume forms on manifolds and equation (8) should be added in Section 2.2.\n+ Typo in line 284. Table 1 is not in the Appendix.\n\nQUESTIONS:\n1. In section 2.1, what is the $\\sigma_T^2$? You did not define it and the sentence in line 67 is confusing as $p_T$ approximates the true limiting distribution.\n\n2. I don't understand Figure 1. The tangential component is in this case the 3. coordinate of the learned score function. Close to the manifold, that is $t\\to 0$, this error decrease as seen by the funnel structure. However, the colorcode suggests that the average error increased closer to the manifold with the highest error for Iso+res. Would you mind elaborating?\n\n3. I noticed different metrics for evaluating the performance on difference datasets. 1-Wasserstein, sliced Wasserstein, MMD, Jensen-Shannon divergence. Can you provide some reasons for that? Can you use a single metric for all? Can you provide a counterexample of a manifold where the assumption do not hold?\n\n4. How strong are the assumptions of Theorem 3.1 and Theorem 4.1? In particular, the $M_1$ and $M_2$ boundedness?\n\nLIMITATIONS:\nThe assumptions of Theorem 3.1 and Theorem 4.1 are not fully discussed, see Question 4 above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}