Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusion
Authors: Enrico Ventura
Beatrice Achilli
Gianluigi Silvestri
Carlo Lucibello
Luca Ambrogioni
Venue: ICLR 2025 Poster (2025)
Keywords: Generative diffusion models
differential geometry
spectral gaps
random matrices
generalization
manifold
Primary area: generative models
TL;DR: We study the temporal evolution of latent linear subspaces during generative diffusion by analyzing the evolution of spectral gaps in the Jacobian.

Abstract: In this paper, we investigate the latent geometry of generative diffusion models under the manifold hypothesis. For this purpose, we analyze the spectrum of eigenvalues (and singular values) of the Jacobian of the score function, whose discontinuities (gaps) reveal the presence and dimensionality of distinct sub-manifolds. Using a statistical physics approach, we derive the spectral distributions and formulas for the spectral gaps under several distributional assumptions, and we compare these theoretical predictions with the spectra estimated from trained networks. Our analysis reveals the existence of three distinct qualitative phases during the generative process: a trivial phase; a manifold coverage phase where the diffusion process fits the distribution internal to the manifold; a consolidation phase where the score becomes orthogonal to the manifold and all particles are projected on the support of the data. This `division of labor' between different timescales provides an elegant explanation of why generative diffusion models are not affected by the manifold overfitting phenomenon that plagues likelihood-based models, since the internal distribution and the manifold geometry are produced at different time points during generation.

Published as a conference paper at ICLR 2025
MANIFOLDS, RANDOM MATRICES AND SPECTRAL GAPS: THE GEOMETRIC PHASES OF GENERATIVE DIFFUSION
Enrico Ventura
Beatrice Achillia."
Gianluigi Silvestri2...
Carlo Lucibello !
Luca Ambrogion?
DDpaarmment of Computing Sciences, BIDSA, Bocconi University, Milan, MI 20100, Italy. Donders Institute for Brain, Cognition and Behaviour, Radboud University, 6500 HD Nijmegen, the Netherlands. *OnePlanet Research Center, imee-the Netherlands, Wageningen, the Netherlands.
ABSTRACT
this paper, we investigate the latent geometry of generative diffusion models under the manifold hypothesis. For this purpose, we analyze the spectrum of eigenvalues (and singular values) of the Jacobian of the score function, whose discontinuities (gaps) reveal the presence and dimensionality of distinct sub-manifolds. Using a statistical physics approach, we derive the spectral distributions and formulas for the spectral gaps under several distributional assumptions, and we compare these theoretical predictions with the spectra estimated from trained networks. Our analysis reveals the existence of three distinct qualitative phases during the generative process: a trivial phase; a manifold coverage phase where the diffusion process fits the distríbution internal to the manifold; consolidation phase where the score bocomes orthogonal to the manifold and all particles are projected on the support of the data. This 'division of labor' between different timescales provides an elegant explana tion of why generative diffusion models are not affected by the manifold overfitting phenomenon that plagues likelihood- based models, since the internal distribution and the manifold geometry are produced different time points during generation.
INTRODUCTION
Generative diffusion models have revolutionized the fields of computer vision and generative modeling, achieving stale-of-the-art performance image generation (Ho et al. 2020 Song
and Ermon 2019 Yang et al. 2021 and video generation (Ho et al. 2022 Singer et al. 2022
Blattmann et 2023 Tim al, 2024). Generative diffusion models synthesize images
through stochastic dynamical denoising process. Experimental and theoretical arguments suggest. that different. features such as frequency modes and class labels are generated at different times during the process. For example, it has been shown that scparation between isolated classes, as in the case of mixture of Gaussian models, happens at critical phase transition points of spontaneous symmetry breaking (speciation events) (Biroli et al. 2024). It is also well known that subspaces corresponding to different frequency modes emerge at different times of diffusion (Kingmna and Gao 2024). This idea has been recently refined by Kadkhodaie et al. 2024), who showed that dilluusion models give rise to a local decomposition of the image manifold into basis of geomet ry-adaptive harmonic basis functions. These decomposition phenomena cannot be directly explained in terms of critical phase transitions
"These authors contributed equally to this work

---

Published as a conference paper at ICLR 2025
Langant Spriee
(2")(xr
Ortoggazal Spose
Orthigonal Space
Later Sineticnalty
(a)
(b)
(c)
Figure 1: (a) Visualization of the gaps in the spectrum of the (negative) Jacobian of the score for data supported on a latent manifold. Blue line: idealized spectrum of distribution with uniform internal density; Orange line: spectrum of more realistic distribution. (b) Sketch of the local structure of data-manifold with tangent and orthogonal components of the score function. (c) Sketch of the geometric phases of generative diffusion and their trace measurable from the eigenspectrum.
as they are fundamentally linear processes. In this paper, we will provide a precise theoretical analysis of the separation of subspaces for data defined on low dimensional linear manifolds.
Our main contributions are: I) an in-depth theoretical random-matin analysis of the distribution of Jacobian spectra in diffusion models linear manifolds and II) detailed experimental analysis of Jacobian spectra extracted from trained networks on linear manifolds and image datasets. The analysis of these spectra is important as it provides detailed picture the latent geometry that guides the generatíve diffusion process. We show that the linear theory predicts several phenomena that we observed in trained networks. Based on our result, we divide the generative process into three qualitatively different phases: trivial phase, manifold coverage phase and manifold consolidation phase. Using these concepts, we provide concise explanation of why diffusion models can avoid the manifold overfitting pathology that characterizes likelihood-base generative models (Loaiza-Gance el al. 2022
2
THE MANIFOLD HYPOTHESTS
The manifold hypothesis states that the distribution on natural data, such as images and sound recordings, is (approximately) supported on a m m-ddimmmmess -i-dimnniional manifold M embedded
in a larger euclidean ambient space Rd (Peyré 2009 Fefferman et al., 2016). While data probability distribution supported proper density function, we loosely define such density as
m < d manilold M cannot. be expressed using
po(æ) (x)()) p(x)
(1)
where dy the Dirac function for the manifold and such that J (x(()) d SM * dæ. We call p(x)) the internal densily, that is the density restricted to the manifold. The density po(x)) is vero outside the manifold and diverges on the manifold. The hidden manifold model proposed by Goldt et al. (2020) is able to reproduce N data y that are embedded in latent n-ddimnnsional space,
(2)
where (xx) is a non-linear function of the d-dimensional vector æ, F E Rdxm is a reclangular matrix that. projects latent vectors 2 the manifold. In this work we are going to study generative diffusion while aligning with this modeling frame.
BACKGROUND ON GENERATIVE DIFFUSION MODELS
Here, will consider simple varianco-explodee forward process where the data æu po(
evolves according to the equation
dæ; dZ.
(3)
2

---

Published as a conference paper at ICLR 2025
where dZ, is standard Brownian motion. The formal solution of Eq. (3) can be given in 10gg
mgll
terms of the heat kernel (x() = Enoopo V2T
The target distribution po(
then recovered by reversing the diffusion process (Anderson 1982 We initialize this
reverse process from Lt, ~ N (O) at some large time iy, which evolves backward in time according to
dæ; V log (o + dZ.
(4)
The function s(x, t) = Va log (x) is the so-called score function. From a set of training
iid points {y',..., ........'""" ......y'' ~ Pn, we can train a neural approximation of s(x, 1) by learning a denoising autoencoder éx(x, (), which is trained to recover the standardized noise et from
the noisy state 24 = zo + Vte, (Hyvärinen and Dayan 2005; Vincent 2011 Ho et al. 2020). In order to avoid singularities in neural network output, the learned score is parametrized as so 5n, 1) (a(m..)
V
4 DYNAMIC LATENT MANIFOLDS AND SPECTRAL GAPS
Consider a generative diffusson model with (z( defined on a d-dimensional manifold M In the course of the diffusion process, we define a time-dependent
according to Eq. ①
locus of points
= 3,m(2", (t) 0, with (x= ,t) n.s.d.).
(5)
that we name stable latent set of the process. In Eq. 5 we have used == The negative semi-definiteness (n.s.d.) a stability condition on the Jacobian matrix (m, t) of the support score 5x(2",t), defined as the score function obtained from the uniform data distribution (x) = J. (M)) Due to the noise, the diffusing particles typically explore sholls of radius that concentrates on vt around each point of the latent stable set. For small perturbation around a point x" on the latent manifold at time t, the score function well approximated by its linearization:
s(p.)) J(x", ,t) p- D= (os PD) Ax"",,
(6)
where J(x",t) is the Jacobian of the score and the vy and My t) are respectively the j-th eigenvector and the associated eigenvalue -["(",", ,1). The spectrum eigenvalues provides detailed information concerning the local geometry of stable latent set. Perlurbations aligned with the tangent space M2 correspond to small eigenvalues, while orthogonal perturbations correspond to high eigenvalues, its fixed- fixed-points. Therefore, we can estimate the dimensionality of the manifold from the location of drop (i.c. sharp change) in the sorted spectrum of cigenvalues (Stanczuk) ct al.) 2022). This is visualized in Fig. [ pancls (a) and (b). This drop corresponds exactly to gap (i.e. separation) in the eigenvalues spectrum; in the following, we will refer to both as gaps
the score tends to push the stochastic dynamics towards
4.1 SUBSPACES AND INTERMEDIATE GAPS
Consider the situation where the internal density (x)( is not locally flat around a point x² E M. In this case, at a finite time & the actual score function does not vanish on the latent stable set M, as there is a gradient of the log-density along the tangent directions. This implics that the spectrum of tangent cigenvalues can have a scries of sub-gaps with separate different langent subspaces with different 'local variance.. In image generation Lasks, these subspaces are often associated with different frequency modes, noted in (Kingana and Gao. 2024b. Consequently, we can quantify the sensitivity to the internal density at time by studying the statistics and temporal evolution of intermediate gaps
GAP (2-AP(x",1 (x*,t) = Ax+1 - \s(x",))
(7)
where the indices & depend the dimensionality of the subspaces. Note however that under realistic data distributions it is unlikely to find sharp intermediate discontinuities since each subspace will have different eigenvalue, resulting in a smooth gradient.
3

---

Published as a conference paper at ICLR 2025
5 PHENOMENOLOGY OF GENERATIVE DIFFUSION ON MANIFOLDS
This section contains an intuitive picture that follows from our theoretical results on linear models, which will fully outline in the next section. The theory considers the case of linear manifold with Gaussian internal distribution. A lincar-manifodd data-model is made of a set of points
(8) where F and x" have been introduced Section 2 While only linear models are theorctically tractable, we conjecture that their phenomenology captures the main features of subspace separation in the langent space of curved manifolds (see Supp. D for further details). We validated the theory using networks trained on both linear data and highly non-linear data such as natural images (see Sections 7 and 8j. Based on the dynamics of the spectral gaps, we found that the generative dynamics ot according to Eq. 4 can be separated into three distinct phases. The phase separation does not correspond to singularities as there are cross-over events, not genuine phase transitions. During all our analysis we will exclusively work with eigenvalues since, in the linear manifold model, the Jacobian of the true score symmetric. The same phenomenology is nevertheless fully appreciable when using the singular values in our experiments with neural approximations of the score.
y" = Fz",
5.1 PHASE I: THE TRIVIAL PILASE
In the trivial phase, the diffusing particle moves according to the noise distribution without strong biases towards the manifold directions. In this dynamic regime, the stable latent Mr is a single point surrounded by an isotropic quadratic well of potential. The spectral gaps are not visible and all eigenvalues have approximately the same value due to the isotropy of the noise distribution. This trivial phase is analogous to the initial phases described in [Raya and Ambrogioni 2024 and (Biroli al. 2024).
5.2 PHASE II: MANIFOLD COVERAGE
The manifold coverage phase begins with the opening of the first of a series of spectral gaps corresponding to local subspaces. In this phase, different subspaces with different variances can therefore be identified by intermediate gaps in the spectra, Fig. D panel (c). When the intermediate gaps are opened, the diffusing particles spread across the manifold directions according to their relative variances. In other words, during this regime of generative diffusion, the process fits the distribution of the data internal to the manifold.
sketched in
We assume low-rank covariance Σ = FF for the data distribution. In terms of random matrix theory, the gap-forming phenomenology has two distinct processes: the emergence of intermediate gaps (i.e. steps in the dimensionality plot) between separated bulks the spectrum, and the opening of a final gap that allows infer the dimensionality of the full manifold. Our analysis gives us the time scale at which such intermediate gaps are maximally openod, i.c.
(9) where y- (oR+1) and 7+ (ox) are specific eigenvalues of Σ (see Fig. g associated with two hierarchically consecutive variances (see Supp. A.2 for an exhanstive analysis). In most cases, when of & of, the dependence on the two variances O( ok+1). This the timescale where the score is maximally sensitive to the relative variance of the two subspaces, which guides the particles toward the correct internal distribution.
max (k) = V (ak) (ax+1);
5.3 PHASE IIT: MANIFOLD CONSOLIDATION
Finally, the manifold consolidation phase is characterized by the asymptotic closure of the intermediate gaps and the sharpening of the total manifold gap, indicating the full dimensionality of M. In this final regime, the score assumes the form
Vx (x) ~
Ia
(10)
4

---

Published as a conference paper at ICLR 2025
where I= F(FTF)- is the projection matrix over the manifold. The component of the score orthogonal to the manifold diverges proportionally to t while the tangent components converge to constant and become therefore negligible in this regime. This results in the consolidation of the gap corresponding to the manifold dimensionality m and to the (relative) closure of the intermediate gaps. Therefore, in this final phase the dynamics of the model simply projects the particles into the manifold M, - M. In the generative modeling literature, this phenomenon is also known as manifold overfitting as the terms corresponding the internal distribution is negligible [Loaiza-Ganen] et al., 2022). In the next section we comment on how these three phases can explain why diffusion models are not affected by this phenomenon, namely why we claim that the manifold is consolidaled rather than overfilted.
5.4 THE GEOMETRIC PHASES AND MANIFOLD OVERFIITING
The probability density of data defined on a manifold is spiked object p)())) where the Dirac-delta (x) determines the manifold and p(x) determines its internal density, Likelihood-based generative models are defined by highly parameterized likelihood function f(:;; O();0), whose parameters are trained by minimizing the loss
(()) -B (f;;; 0)]
(11)
which maximizes the probability of the data given the model. This maximum likelihood loss minimized if f(x; 0) = Po(z). A trained likelihood-based model can only fit the true density by having it to diverge to infinity on the manifold. Such a divergence makes it impossible to correctly model the internal density p(z). More problenaticallyy the optimization problem becomes almost insensitive to the internal density p(æ). This phenomenon called manifold overfitting KLoaiza-Ganent et al., 2022). since the trained model fits the manifold while ignoring its internal density, resulting in poor generation.
Our analysis suggests that the temporal dynamics of generative diffusion models overcome this limitation because, for intermediate values of f, the score is still sensitive to the density internal to the manifold, which can be identified through the differences in the tangent singular values. During this manifold coverage phase, the score directs the dispersion of the particles according these differences, with higher singular values resulting in larger 'opposing force' from the score, which results in smaller displacements of the generated samples along these directions. For tending to zero, these differences are suppressed due to the divergence of the likelihood, which results in a score function that is orthogonal to the manifold and that insensitive to p(æ:). However, at this stage of generative diffusson the internal dispersion of the particles have already been affected by the previous coverage phase and therefore the manifold overfitting of the score does not negatively affect generation. Instead, the consolidation phase plays the important role of projecting the particles to the support of the data.
THEORETICAL ANALYSIS OF THE SPECTRAL GAPS IN LINEAR DIFFUSION MODELS
In this section, we provide our main theoretical results concerning the spectral distribution for random linear subspaces and the relative spectral gaps formulas. We start by reviewing diffusion with data supported on linear manifolds, where the exact score function can be computed.
6.1 LINEAR MANIFOLDS
Normally the distribution po( is unknown. It is however interesting to investigate tractable special case where the distribution is a multivariate Gaussian defined as in Eq. X where F Rdxm is an arbitrary projection matrix that implicitly define the structure of the latent manifold. and z [0, Im) the latent space vector. In this setting, the distribution can be explicitly written as po (x) (o.. Fz) Ax 0, FFT). Therefore, the density
5

---

Published as a conference paper at ICLR 2025
of the process at given time t is again Gaussian and can be computed from
pr(x) Ex
2 12x-
(12)
(2Tt)2
While linear manifolds are very simple when compared with real data, they still exhibit rich and non-trivial phenomenology that elucidate several universal phenomena of diffusion under the manifold hypothesis. In fact, these linear models capture the structure of tangent
spaces of smooth manifolds (see Supp. D
The score function of the linear model is solvable analytically since we only have to perform Gaussian integrals, from which we obtain quadratic form in x that we can rewrite as
lo x J/x const. 2t
(13)
where the constant does not depend on x and
-1 F
J. 4 Im
F
F Ta-
(14)
The score function is thus derived as Vx (x) J.x. It is then useful to analyze the spectrum of the matrix J,, since Jr is proportional fact, since the gradient of the score is orthogonal to the manifold sufficienlly close to it, the number of null eigenvalues of Jr will correspond to the manifold dimension and we should expect to see drop in the spectrum.
the Jacobian of the score function. In
In the following, we provide an outline of our theoretical results on the distribution of spectral gaps in the matrix J, under random linear manifolds. This choice reflects the fact that the distribution and support of the data are usually not known in advance, and therefore important to quantify the statistical variability induced by this uncertainty. We will consider two different distributions for the random projection matrices F: an isotropic Gaussian case, and multiple-variance one. To ensure tractability, we perform the analysis in the limit of large (visible) and m (latent) dimensions while keeping the ratio am = m/d constant.
6.2 THE ISOTROPIC CASE
If the elements of the projection matrix F' are sampled as F.j NO, o2/m2/////)/ /m), we are able to derive analytically the full expression of the distribution of the cigenvalues of Jr. It is given by simple transformation of the distribution of the eigenvalues of F F, which known to be the Marchenko-Pastuu distribution reported in Supp. A.I
In Fig. 7 we show the shape of the spectrum at different times. The bulk of the distribution, inherited from the density of the eigenvalues of F F, gradually shifts from left to right in the support. By measuring the cumulative function of the spectrum, one can isolate a drop in the effective dimensionality of the manifold, also plotted in Fig, 7 The step is present at any time in the process and it is implied by the gap between the left bound of the bulk and the spike in -1. The width of this gap evolves in time according to
o² +1 a' 1/2,2
A GAP iin
(1:0)
(15)
1+ 02 + am am 1/23/2
If name (a) the left bound eigenvalue of the bulk in the spectrum of FT F (see Fig, 6 one can recover a more general expression for the gap, being
(0) =A + (0)
(16)
Hence we can resolve the gap at scale A at the time
1-A
4n (0)
(17)
A

---

Published as a conference paper at ICLR 2025
6.3 INTERMEDIATE GAPS AND SUBSPACES WITH DIFFERENT VARIANCES
Another relevant case for our study is the one where we consider a manifold having multiple subspaces with different variances. Here we will focus on the instance of two distinct variances. This scenario is reproduced by considering a number f.m m of columms of F to have element. Gaussian distributed with zero mean and variance of/m, and the remaining (1 - -f) columns with elements from a Gaussian with variance o./m. The spectrum of Jt can be computed also in this case, as explained in Supp. [A.2] The density function of the cigenvalues shows a transient behavior of the spectrum in the form of an intermediate drop in the estimated dimensionality of the hidden data manifold. This bchavior reported in Fig. 2 Even though the expression of the spectral density doesn't have an explicit analytical form and has to be computed numerically, one can adopt a special assumption on the behavior of the density of the eigenvalues of F F to estimate the typical times at which the intermediate drop occurs. Generally speaking, the spectrum of F F can be composed of two separated bulks, observable in Fig, 8 This happens when o and of are significantly different. In analogy with the single variance scenario, we name 7+ the left bound of the bulk associated with higher eigenvalues, i.c. with the higher variance, and 7 the right bound of the bulk associated with smaller eigenvalues, i.e. smaller variance. Most commonlyy Y- (-2(0) and 7+ = 7+ (a1). In this case the gap-forming phenomenology provides for two distinct processes: the emergence of intermediate gaps (i.c. steps in the dimensionality plot) between separated bulks of the spectrum, the opening of a final gap that allows to infer the dimensionality of the full manifold. The width of the intermediate gap between two bulks can be obtained from Eq. 26 as
A GAP inter (1: 02) t (02) t t (+1)
(18)
By imposing A inter (t; 01, 02) = A one finds the following quadratic form
GAP
Al + (A 1)7- + (A + 1)7+]t+ A7- 0.
(19)
Considering A < 1 and 7+ < 7- < the opening time for the intermediate gap can be found by
tin (A) ~ (
(20)
that reference time at which the gap becomes visible. On the other hand, by assuming the closure time to be close to zero, it can be obtained as
taa (An( A74(02))
(21)
Furthermore, the time at which the gap is maximum in width, and so maximally visible, is located in between in and tim- This the most important time scale for the problem, it is obtained by imposing dA GAP 10t = 0 and it measures
1max = V7-((1))++
(22)
Indood, when of > o3 the total spectrum can be approximated by a mixture of two separated Prrchenko-Pasttur distributions, with variances of and ož, and paramcters am and y to be rescaled with respect to I and (1 f). This approximation becomes exact under a slight
modification of F which does not imply any loss of the quality of the description. Now the relevant quantities for the gap become
t t[fof(1
Fa - (1- f)03(1 + V (a-)am² )²]
GAP inter (t; (t:01.02) =
(23)
+ (1 - 1) + (I-f)am }2
|+Jop(1 V Fam )2
fof(1
1 2
2
(n(A) = A (1-
1
of
Lin (A) = S)
1+ V
02 (24)
foom
f)am
fmax= = - (1
1
1
V ffam
1+ V (1
f)am
σ1 01.02.
(25)
7

---

Published as a conference paper at ICLR 2025
Therry
Theory
Theory
Theory
mequancy
Prequsncy
Prequancy
Prooucncy
22
27
22
Eggnvokees
Eiensokes
Eigenvouess
Ergenvaliss
(a) 1 10
(b) = 0.1
(c) t = 0.01
(d) t 5e-4
14
Cumubatve Runction
Cumu Functin
Cum.lttiv Funcian
10
(1) t-
(g) 0.01
(h) t 5e-1
Figure 2: Spectrum of the eigenvalues of J and drop in the dimensionality of the data- manifold estimated from theory in the double-variance case, with Om 0.5, of 1, 0.01, f = 0.75. Numerical data are generated with d = 100 and collected over 100 rcalizations of the F matrix.
This same analysis can extended to the more general case where the spectral density is known to be formed by different detached bulks, associated with hicrarchically smaller variances the data. The evolution of the intermediate gaps in a double-variance model is reported in Fig. 2 notice that mmax (022) is consistent with Fig, P and and where the gap was found to be maximun in width. It is worth noting that subspaces with higher variances are the first ones be explored by diffusion, and be learned by the model. This point suggests that the model is sensitive to the parameters of the probability distribution on the manifold, as recently suggested by other works in the literature (see Section for further details).
variance diffusion
EXPERIMENTS WITH SYNTHETIC LINEAR DATASETS
We first measure the spectrum the singular values of the Jacobian of a score function trained through neural network a linear manifold data-model generated by two variances of, of as described in Section 6.3 Results are reported in Fig. 3 (left and central panels). The opening of the gaps is consistent with the theory for the exact score: intermediate gap associated with the subspace with higher variance first opens; subsoquently, the gap relative to the lower variance subspace, which here corresponds to the final gap, opens. We can infer the dimensions of the subspaces by subtracting the location of the dashed vertical lines from d. We underline the fact that higher-variance subspaces learned first by repeating the experiment after swapping the values of the variances. Eventually, the right panel in Fig. reports the same experiments where variances are uniformly generated in the interval [10 [10-2,1] 1]: it is evident that the d intermediate gaps is now continuous line, as is expected to be in more realistic natural data-sets. We will now compare the gaps computed analytically with ones obtained from real neural networks trained on the same linear data-model. The results such comparison are presented in Fig, 4 and they show a good agreement between the ordered distribution of the singular values obtained through empirical mothods, and the relative analytical countrrpart, computed through the replica method. The opening of the predicted intermediate gaps signal the right dimension of the linear subspaces as verified from the experiments. One can notice from the figure that the analytical profile shows the shape of sharp step between the zero value along the 2-axis
8

---

Published as a conference paper at ICLR 2025
0.3)
nommalii Sing vilum
FarmA End dropter Wein
mumak Skyu
2+655793
>>635333
undend sigudarvaassss
undare Singutar wolau5
oddeed Kalues
Ordered singular values obtained with the trained score model, for different variances on the subspaces. Data are generated according to the lincar-manifold model with d = 100 and 40. Left: of = 1, o2 0.01, f f 0.75; Center: of 0.01, 03 1, f 0.75; Right: a progressive number n of variances sampled uniformly between 10-2 and 1 each one assigned to fraction f 1/n of matrix columns. The neural network is trained as prescribed in Supp.
Figure 3:
B and spectra are measured according to Supp.
20 1.0.4
Sv 1-0.00 0.30
36 0-0.11
24
14
3.0
0.A
creane wwggush
-roened www.uinng........ DELLE
ortened dng 110.01110000 VOTUM
Figure 4: Comparison between spectra obtained with the trained score model and with the numerical analysis, for different variances on the subspaces. Data are generated according to the amner--maifolle model with d = 100 and m = 40, of = 1, oï = 0.01, f = 0.75; from left to right, the spectra are evaluated time t z 0.45, t sz 0.3, t x 0.11. The neural network is trained as preseribed in Supp. and spectra are measured according to Supp.
and the first appearing gap: this shape is related to the Dirac-delta spike that the spectrum of the eigenvalues presents at -1 (see Supp. A.2 for details about the spectrum); on the other hand, the numerical profile looks different in the same region, and this behavior is associated with the absence of the spike in the distribution of the singular values, that leaves room to a separated bulk from the other oncs. This evident discrepancy between theory and experiment is probably due to the final configuration of the trained neural network and leaves space for further investigatioos.
EXPERIMENTS WITH NATURAL IMAGE DATASETS
While our theoretical analysis is limited to linear random manifold models, several qualitative aspects of its phenomenology can be observed in networks trained on natural images. Fig. 5 shows the temporal evolution of the spectrum estimated numerically from the Jacobian of models trained on MNIST, C0 and CclebA. Details about the training process are provided in Supp. B and Supp. C In these experiments we can recognize the three geometric phases of diffuusion described above:
Trivial phase: at large times (i.e. from t= 200 to 100) the ordered spectrum of the singular values appears flat, suggesting the diffusive motion to be Brownian in the ambient space.
Manifold coverage phase: at intermediate times the spectrum shows a clear trace of multiple simultaneous opening gaps. The shape of the curves is, however, different from the controlled scenario showed in Section D due to two different reasons: similar latent variances associated to different latent dimensions imply smoothing of the curve, as displayed by Fig. (right panel); the local eigenspace of the data is complex and hard to model microscopically: for instance, the pixellated appearance of Cifar10 images might explain the scarce emergence
9

---

Published as a conference paper at ICLR 2025
(P difidionn
Cevett diftedin
53
Figure 5: Jacobian spectra of diffuusio models trained on MNIST, Cifar10 and CelebA. The neural network is trained as prescribed in Supp. B and spectra are measured according to Supp.
of the gaps, while the larger gap structure showed by CelebA might be due to correlations among the latent variances.
Manifold consolidation phase: at small times (below f 2) we finally see only the full manifold gap open and progressively sharpening, in the sense that many singular values become exactly zero. The spectra this phase are the only ones analyzed in Stanczuk et al. (2022) and used estimate the manifold dimensionality. For instance, (lelt panel), that at the end of the reverse process the network trained on MNIST shows a. latent dimension
from Fig.
100 that is coherent with Stanezuk al. 2022
We conclude that, although our analysis focuses on the local structure of the data manifold (or the stable latent set diffusion time), is effectively supported by experiments on real-world complex datasets.
9 RELATED WORK & DISCUSSION
The evolution of the fixed-points of the exact score was studied in (Raya and Ambrogioni 2024) for the analysis of the spontaneous symmetry phenomenon breaking and in (Birol and Mézard 2023) for the analysis of memorization and glassy phase transitions. The use of spectral gaps to quantify the dimensionality of the manifold was introduced in (Stanczuk el al. 2022). where the total manifold gap is analyzed. Several recent studies invesligales The local linear structure of trained diffusion models. For example, (Kadkhodaie studied the expansion of the Jacobian of trained models and described it. as an oplimal geometry-adaptive Harmonic representation. Similarly, (Chen et al. 2024b) characterized the linear expansion of the Jacobian of trained networks and characterized the resulting components in terms complement to this more applied line of rescarch, matrix analysis of the phenomenon in tractable models. The dynamic geometry of diffusson manifolds also investigated in (Chen et al. 2024a) using techniques inspired by research on latent generalors such as GANs and autoencoders.. Another recent work (Peng et al. 2024) uses parameterized low-rank score functions to show an equivalence between training diffusion model and subspace clustering. In our work, we prove results for the exact score for high-dimensional datasets. Finally, (Sakamoto et al. (0224) studies the dynamic geometry of tubular neighborhoods of the latent manifold and connect their singularities with spontancons symmetry breaking events. Generative diffusion models exhibit rich geometric structures that have the potential to explain their impressive generative capabilities. Our work introduces the use of random-matrix theory techniques for the analysis of their dynamic local geometry and paves the way for the use of advanced statistical physics techniques, which may in the future unveil more global, topological and non-linear aspects of the dynamic geometry of diffusion generative models.
al. 2024
their frequency content. Our work can be seen as a thcoretical provide comprchensive random-
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This publication benefited from European Union Next Generation EU funds, component M4. C2, investment 1.1. CUP J53D23001330011.
OnePlanet Research Center acknowledges the support of the Province of Gelderland. The authors are grateful Marc Mézard for useful suggestions and discussions.
REFERENCES
Anderson, B. D. (1982). Reverso-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326.
Biroli, G., Bonnaire, T., De Bortoli, V., and Mézard, M. (2024). Dynamical regimes of diffusion models. arXiv preprint 188112240..14/11
Biroli, G. and Mézard, (2023). Generative diffusion in very large dimensions. Journal of Statistical Mechanics: Theory and Experiment, (993(2)
Blattmann, A. et al. (2023). Align your latents: High-resolution vidco synthesis with latent diffusion models. In Conference on Computer Vision and Pattern Recognition.
Chen, J., Li, M., Pan, Z., Gao, X., and Tu, C. (2024a). Varying manifolds in diffusion: From time-varying geometries to visual saliency. aiv preprint arXiv:2406.1888 18588.
Chen, S. et al. (2024b). Exploring low-dimensional subspaces in diffusson models for controllable image editing. arXiv preprint wxXxx:2409.0237/
Fefferman, C., Mitter, S., and Narayanan, H. (2016). Testing the manifold hypothesis. Journal of the American Mathematical Society. 2944)9983-1049.
Goldt, S., Mézard, M., Krzakala, F., and Zdeborová, L. (2020). Modeling the influence of data structure on learning in neural networks: The hidden manifold model. Physical Review X, 10(4):041044.
Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffuusion probabilistic models. Advances in Neural Information Processing Systems.
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. (2022). Video diffusion models. arXiv preprint 2xiv:2204.03455
Hyvärinen, A. and Dayan, P. (2005). Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4).
Kadkhodaie, 7., Guth, F., Simoncelli, F. P., and Mallat, S. (2024). Generalization in diffusion models arises from geometry-adaptin harmonic representations. In International Conference on Learning Representations.
Kingma, D. and Gao, R. (2024). Understanding diffusion objectives as the elbo with simple data augmentation. Aduances in Newral Information Processing Systems, 36.
Loaiza-Gant Ganem, G., Ross, B. L. R., J., C. C., and Caterini, A. (2022). Diagnosing and fixing manifold overfitting in deep generative models. Transactions Machine Learning Research.
Peng, W., Hujjie, Z., Zekai, Z., Siyi, C., Yi, M., and Qing, Q (2024). Diffusion models learn now-dimensional distributions via subspace clustering. ar preprint arXiv:2409.022
Peyré, G. (2009). Manifold models for signals and images. Computer Vision and Image Understanding, 113(2):249 260.
Raya, G. and Ambrogioni, L. (2024). Spontaneous symmetry breaking in generative diffusion models. Advances in Newral Information Processing Systems, 36.
11

---

Published as a conference paper at ICLR 2025
Sakamoto, K., Sakamoto, R., Tanabe, M., Akagawa, M., Hayashi, Y., Yaguchi, M., Suzuki, M., and Matsuo, Y. (2024). The geometry of diffusion models: Tubular neighbourhoods and singularities. In Workshop (eometry-grounddee Representation Learning and Generative Modeling.
Salimans, T., Karpathy, A., Chen, X., and Kingma, D. P. (2017). Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXi preprint arxiv:1700 Xiv:1701.05517.
Singer, U. et al. (2022). Make-a-video: Text-to-video generation without text-video data. ar preprint urXiv:2209.1 01922099.479.. 14792.
Song. Y. and Ermon, S. (2019). Generative modcling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems.
Stanczuk, J., Batzolis, G., Deveney, T., and Schönlieb, C. (2022). Your diffusion model secretly knows the dimension of the data manifold. arXiv preprint 2rXiv2212.12611
Tim, B. et al. (2024). Video generation models as world simulators.
Vincent, P. (2011). A connection between score matching and denoising autoencoders. Newral Computation, 23671:1661111
Yang, S. et al. (2021). Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations.
12

---

Published as a conference paper at ICLR 2025
A ANALYTICAL DERIVATION OF THE SPECTRUM OF Jr
A.1 SINGLE VARIANCE SCENARIO
We want to compute the spectrum of the matrix in (14 Let us first consider the case in which F is a d x m matrix with Gaussian entries, and call y the eigenvalues of FFT.
The function that gives the cigenvalues of Jr function of y is
7j -1 t t +1+ 77
ry
(26)
Thus, knowing that the distribution of y is Marchenko-Pasturr, we can obtain the distribution of
Pc(r)
Om 2T r(1 + r)
V(+ - (r +1 am) 1) 1)0 - -1)
(27)
for E (t),)),+(tt) with (t)
11 Vam
One could ask whether the bulk of Jt separates from r -1 at discrete time. This separation corresponds to drop in the histogram of eigenvalues. According to Eq. (27). the
bulk always separated from the spike finite time t, because (1 + am 1/ for every t, and the width of the gap is given by A G = (t- +1
(1 + a. 1/2
GAPP(L) (1)
(28)
t am 1/3/2
With respect to the starting spectrum of F F, this condition reads
7+ A t + 7+
(29)
80 the time when we see the drop at a scale A is = 77 (1~ا)
++
0.25
(())
0.15
0.10
0.05
0.00
0
2
Y
Figure Spectrum of the eigenvalues FT F as obtained from random matrix theory with eigenvalue 7- indicated by red arrow. 7+ is provided by the Paachenno-Pastiin density function. Control parameters are chosen to be am = 0.5, の²
13

---

Published as a conference paper at ICLR 2025
Theory Numerics
Theory Numerics
Theory Numerics
12-
10
Frequncy
Frequeccy
-64 -0.4
-20
Eigenvalues
Eigenvalues
Figervalies
(a) = 10
(b) t-1
(c) 0.01
tegnitude ofe Etpetwalless
Megrittoe tudeetudeeoo
5.
02
04
2. 04 10 0.
1.0
88
02
066
03
3.0
Cumulative
Cumulative Function
Cumulative
(d) 10
(e)
(f) 0.01
Figure 7: Spectrum of the eigenvalues of Jr and drop in the dimensionality of the data- manifold estimated from theory in the single-variance case, with on 0.5, o² 1. Numerical data are generated with d = 100 and collected over 100 realizations of the F matrix.
14

---

Published as a conference paper at ICLR 2025
A.2 DOUBLE VARIANCE SCENARIO
We want to compute the spectrum of Jr when Fip o() for μ < fm/2 and Fip. (0, oz) for μ > (1 f)m/2, with f E [0, 1]. We use the replica method to compute the spectrum of A = FFF, then with a transform we obtain the spectrum of J.. In order to obtain the spectrum we need to compute the expectation of the resolvent of A in the + too limit, and to do this we will rely on the replica method
2 a
E [ga(=)] =
log
(30)
d
det (zla A)
2 lim E
7"
(31)
d 02
n n-0
with
7" = det (zIa - A) A)-R/2
(32)
d
dØ
- E.., Σ. Ei
(56., -,
(33)
22T
0=1i=1
and taking the expectation
E
7
døc
E.
(0:9
E
E.
E E,
0f Fsu)?
(34)
/2T
/II II
drf 2
$
E.
dej
2 E.
(2
evm E.
E
2m
(35)
where in the last step we have used the independence of the rows of F and applied Hubbard-Stratonovic transform.
We can separate the product over y and integrate over the distribution of F
du
(1)))
dØp
Σ E.,
(0))²
E
E
m.n
(36)
V 2T
22TT
fm--1
E.
of Prn
2zmm En
o8 FFFrN)
x
E
e 22mm
E
(37)
#=1
R=
- Ean (ni)
dog
E,, (07)
(38)
2T
a,5 22
Σ, Excim EE, 2
Σ; Σ
Emzim E, $211)))
X e pm
2m
(39)
dn
Ean (n1))
d
1/4 Σ 10999
(40)
2T
V2T
a
a.i
12, 2.) (
E
Ens (2 (, 69dt) (2) 124))
x 6
Tm
(41)
Introducing Job Σ C800
15

---

Published as a conference paper at ICLR 2025
daabdaab
dø
- tdas o 3, (0))²
E [2
(42)
2m
2
fm
dup a
E. (1)))+ of E., good
e
(43)
2m
(1- fmm
duy
E.. (1)))+ of
+2m Eas R
(44)
2T
daabdGab end9(a.4)
(45)
2m
with
((q.
Σ
(1 Gr(400)))02))
(46)
2n
0.
where
T1
dej
E.009++3 E.s 2
Gs((i)
log
(47)
n
2m
1
dry
E. (119)
2² Ens 7)
Ge(g,0)
log
20mm
(48)
2T
Using the replica symmetric ansatz gab dabl, Tab -aab
Gs(g) = log(2 + ()
(49)
2
GF(g,o) 1
o²q
log(1 (1-
(50)
Omm
Putting all together we have
Om ((=) = 4g log(z 4) S
(1
Om
log
- (1 f)
log
(51)
2
Omm
2
The integral can be evaluated by the saddle point method
(52)
am o. - (1 -)
am
-s
(53)
am 079 0m
024
We can find the Stieltjes transform
[[ga(2]] -2a, (4(2)
(54)
0z
.mmq" (2)
(55)
16

---

Published as a conference paper at ICLR 2025
where q² is found by solving the saddle point equation
zq² + q² am - 1-
20
20 m
+
of
2
2
a
+9 q
(z
foi -11 (1- f)) +
0.
(56)
0io2
o
The asymptotic distribution of eigenvalues can be obtained from the Stieltjes transform
1
p())
lim Im (ga(7 ic))
(57)
π c-0
1
lim Im (7 ie)
(58)
π e-
am lim Im[q"] c-+0+
(59)
π
1.4
1.2
1.0
p(v)
0.8
0.6
0.4
Y+
Figure 8: Spectrum of the eigenvalues of FT F obtained from random matrix theory with cigenvalucs 7- and 7+ indicated by red arrows. Control parameters are chosen to be Om 0.5, 0.5, of 1, 0.1.
Once the density of the eigenvalues is computed one can perform the same change of variables described in Supp. A.1 for the case of single variance, and obtain the density pe(r) for the eigenvalues of Jr.
Fig. 2 reports the evolution in time of the spectral density, as well as its cumulative function f = 0.75. The cumulative function has been used estimate the formation of the intermediate gaps to compare with the experiments for the estimation of the data-manifold dimension.
B NETWORK TRAINING AND MODEL A ARCHITECTURE DETAILS
Dataset
Image Size
Latent Dim. Channel Mult.
Param. Count
Batch size
Iterations
Cilarl
32
128
(I. 2, 2)
35.7M
128
500,000
MNIST
28
128
(1. 2,
24.5M
128
400,000
CelebA
64
64
(1, 2, 2, 4)
27.M
64
800,000
Table 1: Table displaying both model and training configurations for each dataset.
For the image datasets, we used the diffusion setting in (Ho et al. 2020 We use the variance scheduler with 3min 10-4 and Buin 10-2 T 1000 time steps, and score model
17

---

Published as a conference paper at ICLR 2025
CelebA singular values for different diffusion time
ColebA singular values for different diffusion time
6o
03
norralion
0-5
236
200
6693
10093
2030
6000
singular for different diffusion time
different diffusion
0.5
nemaleee
0.4
60
0.3
630
DeSeRed wwg.lnr......................... valuie
01teeed Ciguuer vnllie
Figure 9: Comparison between the standard SVD method employed by Stanczuk et al. (2022) (left) and the new orthogonalized method for the visualization of the intermediate gaps (right) for a neural network trained on the CelebA and MNIST data-set as described in Section B Spectra are measured according to Supp. C
backbone + (Salimans et al. 2017 Furthermore, for each of the datasets, we adjusted the partameters account for the different complexity (see [Table 1p. For each data-set, the number of used training data-points amounts to the full set ol data available.
For the linear models, we used Variance Exploding cont score model trained with 2M steps (batch size 128). The model had a Residual architecture with size 128 hidden channels in each layer, two residual blocks comprised by two linear layers with SiLu. In this case, the number of used training-date amounts to 2 10 synthetic examples generated according Scction 6.1
For all experiments we primarily utilized NVIDIA Tesla V100 GPUs with 32 GB of memory.
C EXPERIMENTAL METHODOLOGY: COMPUTING THE SINGULAR VALUES OF THE JACOBIAN OF THE SCORE FUNCTION
For computing the singular values, we used an improved version of the procedure from (Saanzuuk et al. 2022) that is reported in Algorithm ! As a difference from the methodology described in the litcrature, we perturb the position 2o where we compute the score function so along strictly orthogonal directions. Such orthogonal perturbations result from a Gaussian sampling of vectors that are subsequentlyy ort hogonaazzzz these perturbed images are not far from other examples present in the training-sel. Choosing orthogonal perturbations instead of random ones significantly reduces the anomalous slope appearing when plotting the ordered singular values that can hide some intermediate gaps (see Fig. 5 in Stanczuk let al. 2022 ). Fig. compares the standard SVD method from the literature with our technique by applying it model trained through from the CelebA and MNIST data-sct.
Morcover, for the lincar models and MNIST models we used a symmetrizod version which empirically found to be more stable, reported in algorithm
Once the SVs have been obtained through the algorithms, they will be normalized by the highest value in the set and subsequently ordered in decreasing fashion. In most cases,
18

---

Published as a conference paper at ICLR 2025
the very first singular value is removed from the set, due to divergent behaviour that is intrinsic to the neural network training.
Algorithm 1 Estimate singular values at 20
Require:
trained diffusion model (score), to sampling time. 1: Sample 2ig ~ po()) from the data set
2: d +- dim(xo)
3: S - empty matrix
4: for i= L,.... d do
5: Sample = (Ntt of perturbations
6: end for
7: (ato 1i + (Fto j=1
(
(
orthogonalized perturbations.
8: for i= L,.... d do
9: Append so ( to) as new column
10: end for
(s.) (1: (w:) (wx) SVD(S)
Algorithm 2 Estimate singular values at 2o with central difference
Require: trained diffusion model (score), to sampling time. 1: Sample Zg Pz(z) from the data set
2: d t- dim(xp)
3: S +- emply matrix
4: for i= 1,.... d do
5: Sample Fto (1)
N 1o, of right perturbations
6: Sample Eto (t) N( Fo, of
7: Fro ) + 2xp =to
(1) left perturbations
8: end for
9: (Tto Lo 1=1
+()
(1)
(Tto
+(0 (0) )
orthogonalized perturbations
10: for i= I,.... d do
11: Append sa(ETo to) 80 (F
column to S
12: end for
13: (st) (w:)d (wx) SVD(S)
D LINEAR MANIFOLD MODEL HYPOTHESIS
The manifold hypothesis is fundamental concept in machine learning. It states that d-dimensional data lie on a lower m-dimensional manifold embedded in the high-dimensional space, often called the ambient space (Fefferman et al. 2016). Whether it exists, such manifold supposed to present a local curvature in the ambient space, i.c. it is not representable d-dimensional hyperplane. The hidden manifold model proposed by Goldi et al. 2020) is able to reproduce data r that are embedded in a latent m-dimensional space, as
(60)
where (xx) is a non-linear function of the d-dimensional vector æ, F E Rudxm a rectangular matrix that projects latent vectors z" on the manifold. Nevertheless, in order to apply the replica method from statistical physics and compute the distribution of the eigenvalues of the Jacobian of the score-function,
constrain ourselves to the simpler case of Linear-manifole ie. $(x) = x (see Section 6.1 Indeed this might sound as a limitation to the reproducibility claim that, for ordinary diffusive diffusion models the variance exploding setup, the theory developed for the linear model still applies to non-linear manifold instances, due to the following reasons:
of our results to more realistic data-sets, e.g. natural images. However
19

---

Published as a conference paper at ICLR 2025
01
%
1.0
normalleed singuis values
sinquisr
0.6-
0.8-
0.4-
0.4
1:0 80000
[-0 59798
[-0 30596
[-0 19995
t-0 00001
1:0
ordered
ordered
Figure 10: Ordered singular value spectrum estimated experimentally from the analysis a data-set living on non-linear manifold built according to Supp. [ where parameters are chosen as in Fig. 3 The neural network is trained as prescribed in Supp. B and spectra are measured according Supp. C
1. The diffusive trajectories large where the trivial phase occurs, sampled by probability distribution that is smooth, due to the Gaussian kernel implied by the stochastic process in Eq. 4 As a consequence, the stable latent set defined in Eq. will be approximately linear.
2. The diffusive trajectories at small t, where the manifold coverage and manifold con- solidation phases occur, explore region contained into ball of radius proportional to vt, that is supposed to be smaller than then the inverse local curvature the manifold.
We now compare the distribution of the singular values of the Jacobian obtained from the linear manifold model and the non-lincar onc. Specifically, we will repeat the plot Fig. (left and central panels) with toy data-set generated as y FE" with E" z² /| As a consequence, the new data will now live on (d- 1)-dimensional ellipsoid inscribed in the original d- d-dimensional hyperplane. The results reported in Fig. I0 and they show evident discrepancy between the linear and non-linear case, as predicted by our argument. The only small difference consists of slight delay in time of the gap phenomenology that is present in the non-linear manifold data which impedes the final gap to fully open at the smallest observable time.
E ADDITIONAL EXPERIMENTS
We provide here additional results regarding the disposition of the ordered singular values of the Jacobian of the score function of models trained according to B and C Specifically, each plot represents the jacobian measured with respect to one single image in the training data-set. We conclude that all images present the same three-phase phenomenology. as general feature of the data-sets.
20

---

Published as a conference paper at ICLR 2025
11
31B
4G
1985S Singuaar wakess offerent
WMb singuar sakees efferrt
WNbi dngddar valkes afferet
20
31B
4G
50-
200
150
800
310
Figure 11: Spectrum of the ordered SVs of the Jacobian for a model trained on MNIST. Each panel is relative to different data-point in the full set.
35M
1506 31M
500
15.8
3/M
11:0
2550 2750
1506 2040
2040
Figure 12: Spectrum of the ordered SVs of the Jacobian for model trained on Cifar10. Each panel is rolative to different data-point in the full set.
21

---

Published as a conference paper at ICLR 2025
singular
singular dim
dngular
RIM
201a
KIM
2015
KIM
onderc
50
KIM
32400
201a
KIM
201
dinguaar
disuuar liferert
digudar Sfferent ciRuddo
250
30909
208
$1.00
00/
402
9:20
Figure 13: Spectrum of the ordered SVs of the Jacobian for model trained on CclebA Each panel is relative to different data-point in the full set.
22


Reviews:
[{"review_id": "4raC233v8n", "reviewer": "Reviewer_Q3Mt", "summary": "The paper studies the generative process of diffusion models, meaning how samples of the target distribution are generated starting from random initial values, by looking at spectral gaps in the Jacobian matrix of the score function. \n\nThe manifold hypothesis is adopted and, in order to conduct a mathematic analysis, the authors consider the simplified case of linear manifolds and Gaussian target measures. In this setting, the temporal evolution of the spectra of the Jacobian is computed either analytically or through a Replica computation, and the resulting spectral gaps are used to estimate the dimensionality of the data-manifold.\n\nThe main result of the paper is that, during the generative process, there is a series of spectral gaps opening, each corresponding to a different local subspace (this phase is denoted Manifold Coverage). Then, the more the process gets close to t=0, the more these gaps shrink, except from the principal one, which is the only one sharpening and remaining asymptotically.\n\nThe authors verify this theoretical prediction on synthetic datasets, by learning the score with a Neural Network and looking at the empirical distribution of the respective Jacobian.\n\nFinally, some experiments on image datasets are presented, to look at whether the phenomenology described for linear manifolds is valid, in some way, also for more realistic settings.", "questions": "- I find the definition of the \"support score\", and later of the \"stable latent manifolds\" in Eq. (5), very confusing. Why are the manifolds defined in this way? What is the role of this definition for the rest of the paper? It seems to me that from Eq. (6) onward, this support score is not mentioned anymore and we come back to the normal score.\n- Line 131: \"radius $\\sqrt{t}$\" should be \"radius proportional to $\\sqrt{t}$\"?\n- I don't know how to reconcile the phenomenology described in Section 5 with the results presented in the later sections. Indeed, from the plots shown in Fig. 3 and 4, it seems that at all times we are in the so-called \"Manifold Coverage\" phase, with the intermediate gaps open. Does this mean that the \"trivial phase\" is only at $t=\\infty$ and the \"consolidation phase\" is only at $t=0$? In general, I think the point of separation between these three phases is not well defined.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is fairly-well written.\n- The authors adopt an interesting new perspective on the study of generative diffusion processes.", "weaknesses": "- Section 7: The justification of the discrepancy in Fig. 6 between the analytical (replica) and the empirical results is weak, in my opinion. If the Neural Network is expressive enough to learn the exact score, shouldn't we expect the two curves to overlap? If one does the same experiments for the isotropic case for which an analytic solution is available, does one get the same discrepancy?\n- Section 8: The results for Cifar10 and CelebA do not show the appearance of any spectral gap. I didn't understand the justification at the end of Section 8. Also for MNIST, while a spectral gap seems to appear at small values of $t$, it seems to indicate a dimensionality of the manifold of around 700. How is this compatible to the fact that the dataset considers only 10 classes?\n- [minor]: In the Introduction, the Jacobian is mentioned, but it should be said it is the Jacobian of the score.\n\nThe paper presents many typos, here is a list of the ones I found (there could be many more):\n- Line 40: models should be modes\n- Line 66: idealize should be idealized\n- Line 113: is is\n- Line 126: Eq. (2) should be Eq. (1)\n- Line 229: the the\n- Line 357: spearated \n- Line 501: temputal\n- Line 510: suble\n\nI suggest a careful reading to correct any additional typo present.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O8pEgohykl", "reviewer": "Reviewer_fVvn", "summary": "The paper provides a random-matrix theory derived result computing the exact eigenspectra of the score of a diffusion modeling a distribution supported on a linear submanifold of a sample space. This distribution is a Gaussian supported on a hyperplane in Euclidean space. A three phase process on the eigenvalues is observed in the diffusion placed on the modeled density. This theoretical computation matches the observation of a number of previous works studying the eigenspectra of the score function of diffusion models.", "questions": "- The experimental detail of section 7 is very light and lacks much detail needed for recreation. For example the number of samples from the distribution from training is missing.\n- Section 7 leaves a poignant question unexplored - how does the number of samples from the data distribution used to train the score function affect the empirical results in estimating the dimension of the data manifold. The distribution of eigenvalues obtained in the results appear to not mach perfectly the theoretical results. Is this due to modeling issues, the amount of data used, or some other issues? Exploring a range of dataset sizes may shed some light on why the image experiments fail to show the definitive eigenspectra cliff needed to detect the intrinsic data dimensionality. Perhaps these datasets simply don't contain enough examples to learn the score well.\n- If the conclusion of section 8 is that for real datasets the proposed method of looking at the distribution of eignevalues of the score function fails to detect the intrinsic dimensionality of the data, can the authors comment on how this menthod might be of practical utility in training or understanding diffusion models?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The results are of some interest and validate observational results of the change in distribution of the eigen spectra of the score function of diffusion models through time.", "weaknesses": "- The setting studied is a rather limited setting, and given the linear nature of the submanifold the density is supported on, and the Gaussian nature of the density.\n- The experimental section of the paper is very light and leaves many questions unexplored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9LhiPxEgqr", "reviewer": "Reviewer_jgnp", "summary": "This paper provides a theoretical perspective on generative diffusion models under the manifold hypothesis, focusing on the spectrum of the score function's Jacobian. The authors propose that spectral gaps indicate the presence and dimensionality of distinct sub-manifolds. They derive analytical forms of the spectrum under the assumption that the manifold is a linear subspace, revealing three phases in the generative process: a trivial phase, a manifold coverage phase, and a consolidation phase. THey argue that this division explains why generative models avoid the manifold overfitting issues seen in likelihood-based models.", "questions": "Following the above weakness, I have the following questions. \n\n1. Can the authors either show some results for nonlinear manifold? If not, at least certain discussion is expected.\n2. Can the authors further discuss the results on CIFAR-10 and CelebA? These results are not sufficient to support the claim in the paper, nor the conjecture. \n3. This is not really a question. At a first glance I thought this is a theoretical paper, but after reading it carefully I feel it's more like a summary and conjecture based on empirical observation, as the theoretical part is only restricted to linear models.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The perspective on generative diffusion models, focusing on the spectrum of the score function's Jacobian, appears novel. \n\n2. The three-phase framework in generative diffusion models provides insight into model behavior across different generative stages.\n\n3. The approach is supported by mathematical derivations under a linear assumption and is further validated with illustrative examples on toy datasets like MNIST.", "weaknesses": "1. The linear assumption is overly restrictive and unlikely to hold in practical scenarios, limiting the generalizability of the theoretical results.\n2. The real data results on CIFAR-10 and CelebA (Figure 7) are difficult to interpret and lack sufficient clarity to be fully convincing.\n3. In Section 5, the conjecture that \"their phenomenology captures the main features of subspace separation in the tangent space of curved manifolds\" lacks a clear intuition or justification, making it unconvincing based on the current presentation.\n\nA minor issue:\nThere are some citation formatting issues. For example. the third reference in the first paragraph of the intro, should be Song et al, instead S et al. See also the B. et all in the same paragraph, and so on.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fPg1LmzZcG", "reviewer": "Reviewer_Cqtz", "summary": "This paper investigates the spectral properties of generative diffusion models within the framework of the manifold hypothesis. By analyzing the spectrum of the Jacobian of the score function, the authors identify and interpret three distinct phases in the generative process: trivial phase, manifold coverage phase, and manifold consolidation phase. These phases explain the internal distribution alignment and resilience against manifold overfitting, which is often seen in likelihood-based models. Through a mix of theoretical analysis using linear manifolds and random matrix theory, alongside high-dimensional experiments on image datasets, the paper demonstrates how these phases manifest and validate the theory.", "questions": "* In section \"The linear random-manifold score in large dimensions: A random matrix analysis\", is the approach aimed to study the behaviour of diffusion models when encountering random datasets?\n* In Equation (5), $M_t$ appears to represent the manifold where noised data reside. Is the idea that particles are found in shells of radius $\\sqrt{t}$ around the latent manifold consistent with this? If $M_t$ is not merely a union of spheres centered on data points (I don't think I believe it is the case), what precisely is its mathematical function? I understand you want to consider a specific space on which the noised data lies, in order to use the intepretations deriving from (6), but I don't think introducing $M_t$ is a good mathematical strategy", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Clear exposition, good fit in the existing literature\n* The analysis of the Jacobian spectrum and its phase-wise evolution provides a meaningful interpretation of the manifold structure and how diffusion models capture it\n* Using linear manifolds and random matrices enables tractable and insightful theoretical analysis supporting the findings.\n* Experiments on complex datasets, such as MNIST and CIFAR-10, align with theoretical predictions, showing the three observable phases.", "weaknesses": "* Introducing the mathematics for manifolds earlier could clarify the manifold overfitting problem and improve the logical flow.\n\nSee questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e2vpYEqjG9", "reviewer": "Reviewer_zg53", "summary": "This paper aims to understand the diffusion models from the geometry of the latent space. The authors did analysis in terms of the spectrum of the score functions' Jacobian, from which the spectral gaps indicates three learning phases of the models. Further, the authors reasoned that the fitting of the distribution and the manifold geometry are disentangled so to explain the indifference of diffusion models to the manifold overfitting.", "questions": "1. (044-045) These decomposition phenomena cannot be directly explained in terms of critical phase transitions as they are fundamentally linear processes. \n   - This reasoning is not clear to me. \n   - I assume authros wanted to explain the separation of subspaces in diffusion models using the phase transition idea, much like how B. et al. 2024a explained the separation of data classes. \n   - what's the main difference of this work from B. et al. 2024a?\n2. 091-095 meantioned that the divergence of data distribution implies the model density becomes larger during training. Could authros elaborate on this? Moreover, when claiming diffusion models having three phases are unaffected by manifold overfitting, could authors provide the analysis accordingly in comparison to the likelihood based models? \n3. 096-097: it's pointed out in this work, diffusion models are not affected by the manifold overfitting. Could authors provide some evidence on this, e.g., literatures? \n4.  140-146: it's not straightforward to see how perturbations aligned to the tangent space of the manifold correcpond to small eigenvalues, while the orthogonal ones correspond to high eigenvalues. Could authors provide more details on this?\n5. It's hard to parse Fig 1. Is this at a fixed time t?  I checked the cited paper Stanczuk et al. 2022, but it's still unclear to me. Why the x-axis is latent dimensionality? I do not directly see the link that perturbations more aligned to the tangent space of the manifold are located at the regim of smaller latent dimentionality, not the orthogonal ones have larger dimensionality. How do you find the one-to-one correspondence between the eigenvalues and the latent dimensionality? In Stanczuk et al. 2022, the authors pointed out that the vanishing singular values correspond to the tangent space, while the large ones correspond to the normal space. But it seems there is some nontrivial steps needed from there to here. \n7. 172: The phase separation do not correspond to singularities as there are cross-over events, not genuine phase transitions. \n   1. what are singularities in this context? \n   2. what are cross-over events?\n8. It seems $\\sigma_k$ is not defined. I assume it is the variance of the data distribution over manifolds with different dimensions. When would $\\sigma_{k+1}^2\\ll \\sigma_k^2$ happen? \n9.  What is asymptotic closure? Do you mean the end time of the phase II can be infinite and one considers an approximate of when this closure happens? \n10. Please also define $r$ in line 481.\n\nOther suggestions:\n- I think the template of the paper is different from the provided one.\n- Please correct and unify the bibliography.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "I think this paper is the first work using random matrix to analyze the jacobian in diffusion models, though the idea of analyzing high-dim jacobians using random matrix theory is not new, assuming I read enough literature on understanding diffusion models. Understanding diffusion models, especially from the latent space geometry, is increasingly researched and this paper provides a new way of approaching this analysis.", "weaknesses": "1. The paper is delivered in a rather vague way and there are multiple important points which should be clarified in terms of either equations, or citations on other literatures. When carefully reading the paper, I couldn't really parse the ideas at several important locations, though some of these unclear points are only explained later on in a more concrete way in Section 6. I have detailed a few of them in the questions section. \n2. The paper strives to better understand the generative diffusion models. While I appreciate the effort using RMT (random matrix theory), the analysis does not really offer deep insights to diffusion models.  In the practical diffusion models, the analysis in Figure 7 merely demonstrated there is a eigenspectrum transition in the Jacobian of the trained model on MINST, which does not have an apparent link to the three phases proposed in the paper. Similar observations have already been shown by _Diffusion Models Encode the Intrinsic Dimension of Data Manifolds_. It would be great if authors could point out the difference or novel insights compared to this work. \n3. The paper _Dynamical Regimes of Diffusion Models_ showed the three phases of the generative process in diffusion models using phase transition analysis where these key transitions are also analyzed via the distributional solution of the forward dynamics. These transitions are determined by not only the eigenvalues of the covariance matrix of the data, but also the entropy of the noised data distribution. It really interests me to see the difference between the analysis in this work and the one in _Dynamical Regimes of Diffusion Models_. So it's necessary to point out the new results from this paper compared to the previous analyses.\n4. I try to locate in the paper where these different phases explains why diffusion models are prone to manifold overfitting. Unfortunately, I could not find it. It's mentioned in section 2 that likelihood based models lead to increasing model density during training (091-092), thus, resulting in the manifold overfitting. Could authors explain how the three phases of diffusion models help them to be unaffected by the manifold overfitting from this perspective? For example, add a section or paragraph that explicitly connects their three-phase model to the issue of manifold overfitting, explaining step-by-step how each phase contributes to avoiding this problem", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)