A solvable model of learning generative diffusion: theory and insights
Authors: Hugo Cui
Cengiz Pehlevan
Yue M. Lu
Venue: NeurIPS 2025 poster ()
Keywords: high-dimensional asymptotics
statistical physics
diffusion model
Primary area: theory
TL;DR: We characterize the distribution of the generated samples for a solvable model of diffusion model in high dimensions, parametrized by a two-layer autoencoder.

Abstract: In this manuscript, we analyze a solvable model of flow or diffusion-based generative model. We consider the problem of learning a model parametrized by a two-layer auto-encoder, trained with online stochastic gradient descent, on a high-dimensional target density with an underlying low-dimensional manifold structure. We derive a tight asymptotic characterization of low-dimensional projections of the distribution of samples generated by the learned model, ascertaining in particular its dependence on the number of training samples. Building on this analysis, we discuss how mode collapse can arise, and lead to model collapse when the generative model is re-trained on generated synthetic data.

A solvable model of learning generative diffusion: theory and insights
Hugo Cui
Center of Mathematical Sciences and Applications, Harvard University
Cengiz Pehlevan
Center for Brain Science,
Kempner Institute for the Study of Natural and Artificial Intelligence, John A. Paulson School of Engineering and Applied Sciences, Harvard University
Yue M Lu
John A. Paulson School of Engineering and Applied Sciences, Center of Mathematical Sciences and Applications, Harvard University
Abstract
In this manuscript, analyze a solvable model of flow or diffusion-based gen- erative model. We consider the problem learning a model parametrized by a two-layer auto-encoder, trained with online stochastic gradient descent, on a high- dimensional target density with an underlying low-dimensional manifold structure. We derive tight asymptotic characterization of oow-dimensional projections of the distribution samples generated by the learned model, ascertaining in particu- lar its dependence we discuss how mode collapse can arise, and lead to model collapse when the generative model is re-trained on generated synthetic data.
the number of training samples. Building this analysis,
Introduction
Diffusion and flow-based generative models represent new paradigm in the sampling of high- dimensional probability densities. Such methods operate by recasting the sampling problem as
transport from a simple base distribution to the target density. The velocity field directing the transport
can further parameterized by a neural network, and learned from data [68] 70
34]. These ideas
have been successfully implemented in number of algorithmic frameworks 770 48
49
with
applications ranging from image generation [55] 58 63
to drug discovery 83
The surprising effectiveness of such models in learning probability densities in high dimensions hints at the presence of architectural biases built in the network parametrization, placing strong priors on the class of densities generated by the model. When aligned with the target density, these architectural biases can allow generative models to learn good approximation of the target from only small number of training samples [5). Naturally, when the biases are ill-suited to the task, they can also lead to poor solutions. Gaining solid theoretical understanding on how the neural network architecture shapes the generated density is hence central, yet still largely open, research question.
Arguably, the prominent technical obstruction lies in the need reach precise characterization of the density a given architecture learns to generate. A large fraction of theoretical studies of
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

43 analyze only the generative transport process, starting from the assumption that L accurate approximation of the velocity or score available. This which establishes that some target densities can provably be learned by neural networks, provided sufficient width and number of samples. Such sample complexity bounds are however little descriptive of the shape of the generated density, nor do they capture failure modes where the architecture is not expressive enough to yield good approximation. Closer to our work, authors of (21] conduct a tight analysis of a simple generative model, which is however limited to the highly stylized case of a binary Gaussian mixture target density with isotropic covariances. The present manuseript overcomes these barriers, and provides a sharp characterization of the generated distribution for models learning from a large class of non- non-trivially structured target densities.
generative models 17 14, 8, 40
gap has in part been filled by recent line of works 52 15 13
Main contributions
We consider generative models parametrized by a two-layer Denoising Auto-Encoder (DAE) with tied weights and trainable skip connection, trained with online Stochastic Gradient Descent (SGD), in the framework of stochastic interpolation 45. We consider target distributions given by (possibly infinite) Gaussian mixtures in high dimensions, with generic cluster covariances, and centroids spanning a low-dimensional manifold reflecting a pervasive intuition in machine learning [72 [84]. Overcoming previous limitations, we derive a fight asymptotic description of the generated distribution. More precisely,
We provide a tight characterization of the training dynamics in terms of a set of deterministic Ordinary Differential Equations (ODEs), bearing over finite set of low-dimensional summary statistics.
Building on these results, we similarly provide a tight low-dimensional characterization of the generative transport process, thereby reaching sharp description of low-dimensional pro- jections of the generated density, as a function of the number of samples and sampling time. The theoretical predictions further quantitatively capture experiments on simple real datasets.
We discuss and illustrate how a phenomenon akin to mode collapse to a loss of diversity in the generated density. Iterating and extending over our analysis to cases where the generated data is further re-used to train the generative model, we highlight how this mode collapse phenomenon can ultimately conduce to model collapse (66).
k2) can arise, and lead
The code employed in this manuscript is accessible on this online repository
Related works
Sampling accuracy and dynamics- A large body of works on diffusion-based and flow-based generative models has been devoted to the study of the generative process, assuming access to a
T,--accurate score estimate, or to the exact empirical score. (17 14
g 40
4
16
23 42
provide rigorous bounds on appropriate distances between the target and generated probability distributions. The sequential emergence of structure the transported density with sampling time has been investigated in [1] [10 665 64 [ [79 44 28 (5), evidencing the presence of rich critical phenomena. [30] explore the computational hardness of sampling for an array of graphical models.
This line of works, however, does not allow one to elucidate how such score estimates can be learned from data by practical architectures.
Sample complexity bounds- Complementing this set of results, a recent stream of work refined these bounds by further ascertaining the sample complexity of learning accurate score estimates 12 38 85 25 90 [33] 46, (29]. For data distributions close to the one considered in the present show how the score of Gaussian mixture densities can be learned algorithmically in efficient fashion. Similarly, for target densities with latent -ow-dimensional structure, [52] [5] [3]
work, 27 18
prove that parametetttee models are able to learn the latent structure, and thus break the curse of dimensionality. [82] provide full error analysis for models parametrized by deep ReLu networks. None of these bounds, however, allow for a precise elucidation of the geometry of the generated density. Furthermore, because such results primarily focus on settings where the target densities can be provably learned by the model with enough samples, they are not descriptive of unrealizable settings where the model is unable of perfect learning, and thus overlook possible failure modes and

---

biases. The present study on the other hand permits the exploration of the latter, and sheds light on mode and model collapse phenomena in the considered model.
Tight characterization of learning in AEs- In order to study such cases, sharper results are therefore warranted. In this direction, sizeable research effort has been devoted to analyzing the learning of AEs [6]. arguably the simplest instance of the class of denoiser neural networks used
in generative models. The learning dynamics of AEs under (S)GD was characterized in [54] in the
linear case, and [56] for non-linear models. [2] derive a tight asymptotic characterization of the learning of AEs for denoising task, reaching a precise description of the generated density when this network is used to parametrize generative model [21]. This characterization is however limited rather stylized target density, namely binary Gaussian mixture with isotropic clusters, and thus fails to describe real data distributions. The present manuscript overcomes this barrier, and considers realistically structured target densities.
Inductive bias in generative models- Even with moderately large training sets, generative models succeed in generating novel images, rather than reproducing memorized training samples
89
88
60
53 81 47 26 80). This surprising efficiency strongly hints at the presence of inductive
biases inherited from the network parametrization, that nudge the model towards good solutions. The inductive bias of U-net architectures [59] has been investigated in [35], who observe how such archi- tectures tend to learn adapted harmonic bases. In a similar spirit, the work of (37 evidences how the bias of such convolutional architectures towards learning equivariant and local scores helps to promote good solutions. Finally, (50) demonstrates that U-nets are closely related to message-passe passing algo- rithms on random hierarchical data, and thus particularly adapted to such structure. Complementing this line of works, the present manuscript offers insights on the bias of DAE architectures.
Problem formulation
We start by providing a succinet overview of the problem of sampling a target density p over Id using ideas from generative transport. For definiteness and ease of presentation, we consider in this manuscript the class of stochastic interpolant models [4], which shares substantial connections with other methods, including score-based diffusion models 77 and denoising methods (36 34
Sampling-
A sample X1 ~ p can be obtained from a Gaussian sample X, ~ N( Ta) by evolving the latter for t E [0, 1] with the Stochastic Differential Equation (SDE)
d.x, dt
3.
dt Be + ct
Be ai
f(t, X.)
( ax
6t a
X + 2GdWe
(1)
a
where W is a Wiener process. This statement holds for any choice of interpolants a, 3 E ([0, 1]),e ([O, IJ) provided (x)) 3(1) -01)=11 (1) = (()) = 0 and ai + 3 > 0, 2 0 at all times t E [0. 1] [.. In (I. the function f [0. x Rd + Rd is defined as = E[z, |a, - Baz = xr. with the conditional expectation bearing over æ1 ~ p, zg ~ (O, Intuitively, fl,, can be interpreted as a denoising function, which aims to recover the sample 21 from the interpolated version 04 + 3,011 in which it is corrupted by the noise xo Perhaps then unsurprisingly, the function f admits a natural characterization as the minimizer of the quadratic denoising objective
Rl
E af + B(Z1) z1ll? dt.
(2)
This formulation provides an opportune pathway to learn the function f governing the sampling (I) directly from data. The learning can be carried out following the usual machine learning rationale of (a) parametrizing ff by a denoiser neural network and (b) replacing the expectation in (2) by an empirical average over training set.
Note that the denoising function f is related to the score function of the density of 0n.0 + Bax by the simple linear relation af s(t, = ft,
3

---

[22] we consider the simplest instance of sensibly alter the phenomenology of the model. We discuss in Appendix [A] how the analysis can be extended to generically untied weights. In (3) p: [0. 1] I is an arbitrary time encoding scheme, and is embedded into the network preactivation through multiplication with set of trainable weights e E IR.. Note that this is equivalent to including the time encoding Pt as an extra dimension in the input æ, and acting upon it with an extended set of (d + 1) X weights with then corresponding to the first row thereof. Let us remark however that for the model (3). the inclusion of the time encoding scheme was not found to significantly alter the behaviour of the model in all probed settings.
Architecture- In the present manuseript, following denoising neural network, namely a two-layer DAE
w TTx vd
(t,
+ p.u)
(3)
vd
with activation function o, and trainable skip connection b R. For simplicity, we assume that the decoder and encoder are tied, namely parametrized by a unique set of weights w E Rdxr, This assumption allows for a more concise exposition of the technical results, and was not found to
Let us briefly situate the class of DAES with respect to models considered in related works. 15 52 consider deep networks with ReLU activations, but require sparsity constraints on the weights. Closer to our work, [3] similarly assume shallow DAEs, but place themselves in the limit of infinite width (r + co). On the opposite end of the spectrum, (2n) consider shallow DAEs with single hidden unit (r = 1), and sigmoidal activations a. The architecture (3) considered in the present manuscript, on the other hand, corresponds to a more flexible class of fintte-width networks with arbitrary activations. Let us note that while DAEs are amenable to theoretical characterization and consequently the main focus of the aforedescribed body of theoretical studies, they admittedly represent very simplified architectures when compared to e.g. U-Nets [59] used in practice. On the other hand, the model (3) retains the main architectural features of the latter, notably a bottleneck structure, skip connection, and a time encoding scheme, and thus constitutes valuable simplified model.
Training-
The neural network (3) can now be used to parametrize the denoising function s in the with online (single-pane pass) SGD, with learning rate
objective 2 We consider training the DAE B and weight decay A:
17 Er | Somaman, (1, (1,ara + Bex 11x de
bu+1 - b,= -
(4)
0p-1 Upp - Ee |لم - fom.way, (t, anz + 3127)11 -1 A
d ww.
(5)
Up+1 Up = -
v. Bb |-4 - fomwann (t, ax2 + 3,a1))1 7 Un-
(6)
The expectation E, over t bears over the uniform distribution over [0, 1], or any approximation
thereof by a chosen set of points g = {f1, ......). The updates (4) are iterated n times from a given initialization bo, Wo, Ug- Note that in online SGD, fresh pair of samples xf ~ p, p ~ ~N(0, Ia) is employed at each training step, and the number of training steps thus coincides with the number of samples. Finally, it will prove convenient to introduce the training time 2ym /a, quantity that remains finite in the asymptotic limit considered, which we detail below, We accordingly denote by br, Wr, Ur the values of the skip connection and weights at the end of training.
After training, the optimized DAE 3 proxy for the true denoising function f:
for.w-rr- can then be employed in the generative flow I as a
dx, = (3e- dt
du Be
Br a?
Jo (t, + (āt - 6t -x X. + +++V22gh 22 of
+ Ct
The SDE (7) can then be used for sampling. Because the learning is generically imperfect due to limited data and architectural bias, we generically have for.wr.-, * f. and thus the generated density (- -namely the law of X,- differs from the target density p, even as + 1. One of the primary objectives of this work is to give precise description of the discrepancy between p and p-(1) - beyond simply bounding probability distances therebetween-. through sharp asymptotic characterization of (t).

---

Target density- In this manuscript, we consider target densities given by (possibly infinite) Gaussian mixture supported on a latent low-dimensional manifold. Namely,
p N(ule),
In words, the centroids sa(c) E IR of the different clusters lie on a K- di-eennnnional manifold, equipped with a coordinate system с E R". The distribution of the clusters on the manifold is given by the relative weights π (c). Finally, each cluster can exhibit non-trivial covariance structure [c) E d For the analysis, we further assume that all the covariances {E(c) Je are jointly diagonalizable, and admit well-defined spectral densities Ve in the nigh-dimensional limit d + co, leaving the generic case to future work. The density (8) reflects and models the celebrated data manifold hypothesis which posits that real data distributions lie on low-dimensional manifolds.
722 84,
52 15 33 similarly consider linear subspaces embedded in high dimensions, which can be viewed as special instances of (8) in the limit of vanishing covariances E(c) = 0 for all c E R Note also that the K modal Gaussian mixture distributions considered in e.g. [21 86] correspond to
the special case of (8) where the cluster distribution is sum of K Dirac deltas. Finally, the
heavy-tailed distributions considered in 3 2 correspond to the special case u(c) = 0g for all c.
ffigh-dimensional limit- We aim at characterizing the generated density P. (1) in the asymptotic limit of large data dimension and commensurably large number of samples, namely d, n + co with w/a (1). This asymptotic limit captures the non-trivial regime where the number of samples is not small enough for the neural network to trivially overfit, and conversely not infinite - thus allowing the investigation of finite data effects. We further suppose that the number of hidden units r of the DAE and the intrinsic dimensionality of the data manifold K, alongside all other parameters, remain finite: r, K; X. 17 = Oa(1). Moreover, the diameter of the mixture (8) also supposed to remain finite, iee there exist D = Oa(1) such that [|a(c)]l25 D with probability 1 for e æ, Finally, we assume that the ambient dimension of the manifold is also finite, namely dim span (p(()) Oa(1).
2 Precise characterization of the generated density
We are now a position to detail our main technical findings, namely a sharp asymptotic character- ization of the generated density f, obtained from the sampling process 77) governed by the DAE fo..w.... (3) trained with online SGD (4). Because it is challenging to describe a high-dimensional rather aim at characterizing low-dimensional projections thereof. For- mally, let us fix a reference space & C Rd with finite dimensionality = (1), and let E Rd R be matrix whose columns form an orthonormal basis of & We aim at an sharp characterization of the law P- (t) of the projection E X, of a generated sample X, ~ p. (t). In words, the subspace corresponds an observation space of interest, chosen by the statistician. Natural choices consists in electing subspace where the target density exhibits non-trivial structure, with a view to probing how well it is reproduced the level of the generated density- for example, the space spanned by the directions of larger variance of the target density (s, or the space spanned by the latent manifold span( (c)
probability distribution,
The derivation of this characterization proceeds in two steps. First, we derive a sharp asymptotic characterization of the high-dimensional training dynamics induced by SGD (4) More precisely, deseribe the evolution of a set of low-dimensional summary statistics of the weights w over training time in terms of collection of limiting ODEs. These summary statisties encode all the geometric information on w necessary to reach, in a second step, a sharp asymptotic characterization of the generative SDE (T) in terms of low-dimensional processes. Finally, this characterization yields sharp description of (z) as a corollary.
2.1 Analysis of the learning
What weights are learned at the end of the training process (4), and how do they correlate with the main structural components of the target density, such as the centroids and directions of higher variance ? Answering this question is instrumental subsequently build an understanding of how, during the generative process, the weight matrix shapes the generated density and allows to reproduce
5

---

the overall structure of the target density. This is the goal of our first result. On a technical level, it shows that the evolution under the SGD dynamics ( of set of summary statisties of the weights w of the DAE (3) is asymptotically described by collection of deterministic ODEs. These summary statisties importantly quantify the correlation of the trained DAE weights with the key geometric features of the target density.
Result 2.1. (SGD dynamics) Let T 0 denote the training time, and w be the weight matrix obtained from the stochastic SGD dynamics (4) from an initialization wa- The summary statistics 2r R
Q7 R
R
G, RRTR M, I 12' defined as
2 W
Q-(c)= W., E(c)wr
w E
w. u(() vd
G. vd
M-(c)
(9)
d
asymptotically concentrate in probability to the solutions at time 0 = T of a system of coupled, finite-dimensional, deterministic ODEs
de - - Fo(Os) di Q0 = Fo(Os),
d -.
a do Ma Ma=FF((Od))
(10)
using the shorthand O₂ (Qa. Qo Go, Ma,by) bo) for more compact notation, although not all functions depend on the totality of the arguments in the collection Og. Similarly, the time encoding weights Ur converge to the solution of an ODE d/avva functions Fo.q.G.MM n. is expounded in Appendix A Finally, the vulue of the skip connection by after training from an initialization by is given by the compact closed-fon ffom expression
(Oa)) The expression for the update
[3e
(AB.18))) + boe + (AB1)))))2))))))
(11)
AE [32] R.
where we denoted A the average covariance eigenvalue A dT(()) J dvc(w)w.
The derivation of Result 2. which we detail in Appendix A follows the ideas of the seminal work of [61 62]. and is very close in spirit to the analysis of [56] for a related model, in the context of data reconstruction. It leverages the observation that in the nigh-dimensional limit, the SGD steps 4 avlf-average, and can be cap- tured by a set of deterministic differential equa- tions. These theoretical predictions accurately capture the dynamics observed in numerical ex- periments, illustrated in Fig. [ detail for illustration in Appendix A the particu- lar case of linear DAE, for which the limiting ODES (T)) take simple, compact forms.
1.00
0.75
0.50
0.5
Miu
0.00
-0.5
-050
Finally,
-075
0
4 10
Evolution of the summary statistics Mr, 2, and of the skip connection strength b, as function of the training time т, for o = tanh, r = 2, pt 00,ar 1 1-,,3, 1, G (1/2). The target density is a trimodal Gaussian mixture p == ( Ia) + 1/2N(u2, Ia) + +1/aN Ta)- Solid lines: numerical experiments in dimension d 1000. Dashed: theoretical characterization (T)) of Result
Figure 1:
The ODEs (0) capture the evolution of the low-dimensional summary statistics Q 2,Q,G. M,M, P,T, which subsume various geometric characteristics of the weight matrix w,, over the SGD dynamics (4). Most notably, the statistic M(c) measures the alignment between the weights with the centroids ((c) that constitute the manifold (8). and thus measures how well the manifold structure is identified and learned. In the simple example of
Gaussian mixture target density, (c) thus gives a quantitative sense of how well w identifies and correlates with the different centroids, thereby learning the key structure of the density. The statistic Q(c), on the other hand, gauges the alignment between the weights and the higher-variance directions of the density at point e of the latent manifold. In the simple case of a unimodal target density with a spiked covariance, Q(c) hence quantifies how well the spike is learned. Overall, the summary statistics appearing in Result thus capture the correlation between the DAE weights w and the key structural features of the target density (8) describing how well the latter are learned and encoded into the DAE parameters. The next section investigates how this encoded structure manifests during the generative transport, and allows turn the reproduction of the structural elements in the generated density.

---

2.2 Analysis of the transport
We are now in position to leverage the summary statistics Qr Or My, P.T characterized in Result -which capture key geometric statistics of the weights w, of the DAE after training- to analyze the generative SDE ( A crucial observation is that the SDE (T] is only non- non-linear in the r-dimensional subspace W- span(w', (wf YE=1 spanned by the columns of the weight matrix wy: and is on the other hand linear in the d-r dimensional orthogonal subspace Wi. The generative dynamics (I) of a sample X2 can accordingly be compactly described by the linear evolution of Y; = Iy, X, (where we denoted Tty the projection in W and the more complicated but fiite--iimenionnl non-linear evolution of Z wo xe/Va E W,. This statement is formalized in the following result: Result 2.2. (generative dynamics) Let X₁ be a stochastic process obeying the generative SDE (T) from an initialization Xo ~ (0, Ia)), and denote Y, = W. X2 and Z, w xe/Va. Further define the shorthands C, = B. - 6o + e/0e/ot, ; A b,T, + âe/o, - -4/02, which depend on the schedule functions a, B, e and on the skip connection strength after training b, Then Ze obeys the low-dimensional SDE
d 7 -77 + (7 + pro,) + 22,22.
(12)
dt
from an initial condition Zo ~ NO, 2+), with B a a dimensional Wiener process. On the other hand, Yg is independently Gaussian Gaussian-distributerr -dsttibbteed
2 dad;
{
-2 2hoh
Ye ~~NN N Owgye
1 1+2
ed
nhy
(13)
The SDE 12 and equation
13) fully describe the law of Xx in terms of low-dimensional quantities.
We have thus reached fully asymptotic characterization of the evolution of sample X, transported by the SDE ( Qualitatively, the density (r(t) of X, is shaped in the column-space W- of the weights w, by the action of the DAE network (second term in (12). ). which acts drift term, while its scale is controlled by the contraction/expansin term A7 (first term in (2)). ), in which the skip connection b, intervenes. In W P. (t) simply remains isotropic and Gaussian, with a time-van time-varying variance succinetly given by 13 We remind that, from Result 2.J the weights w, encode information on the key structural features of the target density (8) The corresponding drift term I; Q. o (Z, + p,u;) in (12) intuitively pushes the generated density along those important directions, ensuring that they can be approximately reproduced. This qualitative picture sheds light on the bias of the DAE-parametrizer generative model (3) The DAE weights identify and learn important features of the target density from the dataset, allowing the model to implement at sampling time non-trivial transport
(12) in the corresponding space W- to reproduce the main structural features of the target density. In the orthog- onal subspace W on the other hand, the DAE is only able to rather crudely approximate the target by an isotropic Gaussian distribution
(3). leveraging its skip connection b to adjust the variance thereof.
2.3 Projected density
While Result 2.2 fully characterizes the distribution of Xt, the description is set the space Wr, which changes with the training time т, rendering the result rather unwieldy. Ideally, we would like to transfer the characterization to a T- independent reference space E, in order to allow for an easier comparison of the generated density at different training times- a comparison that will be subsequently discussed in Section 3 and illustrated in Figs and The desired technical result is thus characterization of distribution of the projection E X, of the generated sample X, in the reference space& E. We state this characterization in the following result.
Corollary 2.3. (Projected generated density) The law of the projection E X. of sample X. in the space of interest E is given by
E X, d G.J 2 Z, NN OR,
-2 2
(IR (R-c,200.) (14)
1 +
c,ds
0
where the law of Ze is characterized in Resuli by the SDE and the summary statistics 2r, Gr are characterized in Resul12.1 2.1 Q+ denotes the Moore Penrose pseudo- pseudo-inverse of Q..
7

---

theory sepatmmets
12
HITUS Fube
3.6
0.4
Figure 2: (Left) Evolution of the projected density Hgg generated by a DAE (3) with
hidden
unit and R = tanh activation, trained on a bimodal Gaussian mixture, with r = 0.2, X = 1.5, ct = 8, = i, 9 {0.5). The generative SDE 7 was run up to = 0.9, and the subspace & is a plane containing the centroid of the target density. Different panels correspond to dif- ferent training times r. Blue contours: contour levels of the theoretical prediction of Corollary 2.3 for the density pr. Colormap: numerical experiments in large but finite dimension d = 1000. Green contours: contour levels of the target density p. (Right) Hellinger distance between the target and generated densities, projected in the space spanned by the centroid, as a function of the training time
P. = 0, 0z
Corollary 2.3 allows to transfer the characterization of Result 2.2, set in a training time dependent space Wr, into a fixed, -inndependent subspace E. (14) reveals that the generated density is given by the sum of a non-trivial term G_ 2 Z.. which captures the key learned structure of the target density, and simple Gaussian term, which intuitively tries to approximate the remaining structural features which were not learned and reproduced. Let us finally remind that the choice of & is made by the statistician. To give more concrete examples, in the following, when considering Gaussian mixture tar- gets, a natural choice for E is the space spanned by the cluster centroids. When dealing with unimodal targets with non-trivial covariance, we will choose the space spanned by the principal components.
3 Evolution of the generated density over training time
The theoretical characterizations of Results 2.1 2.2 and 2.3 afford a complete characterization of low-dimensional projections of the generated density Pr as function of the network architecture and training time 7. They thereby provide window elucidate how the DAE architecture shapes the generated density. Importantly, the present analysis also allows the study of the evolution of the generated density over training time T, in addition to its evolution over sampling time I, which has tra- ditionally been the focal point of past works, e.g. [0] 17]. In the next paragraphs, we discuss these questions the context of two examples, for Gaussian mixture density and real data distribution.
Example - Gaussian mixture We give as a first example the case of a Gaussian mixture target density P with 2 isotropic modes. We consider a generative model parametrized by DAE (3) with r = 1 hidden unit and = tanh activation. Fig.2] illustrates, for different training times т, the generated density Å- projected in two-dimensional space & containing the cluster centroids. A comparison between the theoretical predictions of Corollary 2.3 (blue contour levels) and numerical experiments in large but finite dimension d 1000 (orange colormap) reveals good agreement. Over training, the model successfully learns the data structure, and produces a generated density Pt exhibiting the correct bimodal structure in the relevant direction. Correspondingly, the estimated Hellinger distance between the generated and target densities in the subspace spanned by the cluster centroids monotonically decreases over training (Fig 2 right). This simple example, which echoes the finding of e.g. [21]. illustrates how the diffusion model (3) can successfully learn to reproduce simple target densities over training time. However, as mentioned above, the interest of deriving a precise characterization such as Corollary 2.3 primarily lies in the possibility to not only study settings where the model successfully learns the target density, but also settings where it fails and biased- - due to insufficient expressivity or data, or poor fit between the architecture and data structure. The following example instantiates such a case. We provide another example for a 3-modal Gaussian mixture, learned with width r = 4, ReLU-activated DAE in Appendix B
8

---

Y-L3 2-3355
T-125
7
7
7
7
7
7
Figure 3: (Left) Evolution of the density 7 generated by a DAE (3) with 2 hidden units and Ø = tanh activation, trained on Gaussian density with the MNIST sevens covariance, with L, Br = 9 = (1/2)- The generative SDE 7 was run up
" = 0.2, À = .784, et = P: = 0. az to = 0.98, and the subspace & is spanned by principal components of the target density. Different panels correspond to different training times т. Blue contours: contour levels of the theoretical prediction of Corollary 2.3 for the density Pa. Colormap: numerical experiments. Green contours:
contour levels of the target density p. (Right) Samples from the generated density Pr, from a common initialization Xo of the generative SDE
function of the training time т.
Example 2- MNIST In Fig. a DAE- AAA-parametrized generative model with r= hidden units and tanh activation is trained to generate a Gaussian distribution with covariance matching that of MNIST sevens 39 The generated probability År is represented in the principal two-dimensional eigenspace & of the MNIST sevens distribution. In similar fashion to the first example, the generated density progressively adjusts to the shape of the target density (green and purple contours) over training time, first stretching in one direction into bimodal density (0 ~ r ~ 1.5), with each mode being subsequently elongated (r z 1.8), approaching the variance of the target in the secondary direction. This sequential emergence of directions of variance in the generated density has very visual consequences at the level of the generated images. Fig. shows samples from the generated density År for varying training times т, transported from common base sample Xg. For 0 S T & 1.5, the generated image gains more resolution and becomes less noisy, as the first principal direction is learned. After 7 ze 1.8, idiosyneratic features such the horizontal bar of the seven- emerge, second direction is learned, signaling increased diversity.
While the structure of the generated density År progressively adapts to that of the target over training, retains a sizable discrepancy thereto notably, the variance of أ, is markedly smaller. This dramatic reduction in variance betrays a detrimental bias in the model, which in the next and last section. For completeness, we report in Fig. 10 in Appendix D an additional experiment on the FaSH [87 dataset of clothing item images. The experiment again reveals a good agreement between the theoretical predictions and simulations (conducted this time directly on the original dataset, rather than Gaussian approximation thereof), and similar phenomenology.
more extensively explore
4 Failure modes: mode(l) collapse
Mode collapse- The loss of variance in the modes of the generated density Pr when learning from the MNIST target distribution (Fig. is reminiscent of the mode collapse phenomenon most commonly observed in the context of generative adversarial networks [32] and score-based models 244, and analyzed for variational inference in [69]. Mode collapse entails a loss in diversity in the generated images. In the present case, mode collapse is in fact a consequence of the unfitness of the architecture (3) for the target density at hand. To see this, first observe that the skip connection 6, of the DAE model (3) contributes to increase the variance of the generated density Ôr. as it intervenes in the linear dilatation term A of the generative transport (12). see Result 2.2 In words, the stronger the strength b- of the skip connection, the more spread is the resulting generated density. However, note from (TT) that the skip connection b, becomes of the same order as the average eigenvalue A at large training times т. For real data distributions which tend to have a small number of large eigenvalues, and large tail of small eigenvalues, A is typically small, implying in turn a small by and small variance for the modes of p,. Mode collapse causes significant mismatch between the

---

2
Figure 4: (left) Target density o corresponding to a Gaussian density equipped with the covariance of the distribution of MNIST sevens (middle) generated density TT (right) second generation density (l obtained by training the generative model (3) on the synthetic distribution p Blue contours: contour levels of the theoretical prediction Corollary 2.3] Colormap: numerical experiments in large but finite dimension
1000. Green contours: contour levels of the target density p. At each successive generations, the same model specifications T = 2.8, = 2, tanh, 77 0.2, À -784, é Pt 0, oz I, S f, 9 (1/2) were employed, The generative SDEs 7 were run up to = 0.98 at reach generation. Finally, the subspace is spanned by principal components of the target density.
generated distribution P. and the target distribution p. A possible remedy would be to fix the skip connection strength b, instead of training it, ensuring it retains a sufficiently large value. We briefly present additional numerical experiments in Appendix
Model collapse- This mismatch can be further aggravated when biased synthetic data thus generated is re-used to train another generative model, with successive generations of generated densities P(1)) (.:))....... becoming increasingly biased. This rapid degradation is an instantiation of the model collapse phenomenon -described in [66] and analyzed in [91] for simple linear single-step denoising models- can be opportunely analyzed for non-linear diffusion-basse models within the present theoretical framework. To see this, observe that training the generation g ++1 model on data produced by the generation model corresponds to adopting the density P(a). generated by the latter as the target distribution. Seasonably, each generation P(n).5, further remains of the form (8), and Results 2. 2.2 2.3 thus apply. This observation is formalized in the following remark.
Remark 4.1. (Successive generated densities) At each generation, P(g),T, is again of the form (8).
with r, (c)) and
VOE E(c) = d=
2f e,ds
π 1 P(g).T
1 1+2
Iw
(15)
In 15 I. P(g).ta' A are given by Results 2.1 2.2 and 2.3 evaluated for target density
P(a- 1).rp We remind that Way,,, denotes the column- space of the trained weights of the generation
g model.
In words, Remark 4.1 ensures that one can apply Results 2. 2.2 and 2.3 to iteratively reach characterization of P(g+1),f0+) from that P(g),T, Fig illustrates the first two generations of synthetic distributions ((1), , when the initial target distribution is given by the same Gaussian approximation of MNIST sevens considered in Fig. The mode collapse phenomenon described in the previous subsection, and apparent generation re-training. As a result, the density P(2). generated at = exhibits smaller even variance, and is devoid of meaningful structure.
1, gets aggravated upon
Conclusions
In this manuscript, we provide precise and exhaustive theoretical characterization of the distribution of the samples generated by simple diffusion model. Considering model parametrized by shallow DAE, in the limit of high dimensions, we derive a tight characterization of low-dimensional projec- tions of the generated density. ascertaining in particular its evolution over training. Our results, which also describe tasks with realistically structured target densities, capture how the architectural bias can lead to mode collapse, and ultimately model collapse if the synthetic data is re-used for training.
10

---

Acknowledgements
The authors would like to thank Michael Albergo, Binxu Wang, Sabarish Sainathan, Lenka Zdeborová, and Emanuele Troiani for insightful discussions. HC acknowledges support from the Center of Mathematical Sciences and Applications (CMSA) of Harvard University. CP is supported by NSF Award DMS-2134157, NSF CAREER Award IIS-2239780, and Sloan Research Fellowship. YML. supported by the Harvard FAS Dean's 's Fund for Promising Scholarship, and by a Harvard College Professorship. This work has been made possible in part by gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
References
[1] Beatrice Achilli, Enrico Ventura, Gianluigi Silvestri, Bao Pham, Gabriel Raya, Dmitry Krotov, Carlo Lucibello, and Luca Ambrogioni. Losing dimensions: Geometric memorization in generative diffusion. arXiv preprint arXiv:2410.08727, 2024.
[2] Urte Adomaityte, Leonardo Defilippis, Bruno Loureiro, and Gabriele Sicuro. ligh--immnnsonal robust regression under heavy-tailed data: Asymptotics and universality. Journal of Statistical Mechanics: Theoor and Experiment, (0:2(11):111000)) 2024.
[3] Urte Adomaityte, Gabriele Sicuro, and Pierpaolo Vivo. Classification of heavy-tailed features in high dimensions: a superstatistical approach. Advances Neural Information Processing Systems, 36:43880-43893, 2023.
[4] Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Fijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797 2023.
[5] Michael S Albergo and Eric Vanden Eijnden. Building normalizing flows with stochastic interpolants. arXiv preprini arXivv2209. 2022.
[6] Luca Arabolli, Bruno Loureiro, Ludovic Stephan, Florent Krzakala, and Lenka Zdeborova. Asymptotics of sgd in sequence-single index models and single-layer attention networks. arXiv preprint arXiv:2506.00 2206.0261, 2025.
[7] Iskander Azangulov, George Deligiannidis, and Judith Rousseau. Convergence of diffusion models under the manifold hypothesis in high-dimensions. arXiv preprint arXiv:2409.18880 2024.
[8] Joe Benton, VD Bortoli, Arnau Doucet, and George Deligiannidis. Nearly d-linear convergence bounds for diffusion models via stochastic localization. OpenReview, 2024.
[9] Joe Benton, George Deligiannidis, and Arnaud Doucet, Error bounds for flow matching methods. arXiv preprint ar05v: /6860, 2023.
[10] Giulio Biroli, Tony Bonnaire, Valentin De Bortoli, and Mare Mézard. Dynamical regimes of diffusion models. Nature Communications, 15(1)99957, 2024.
[11] Giulio Biroli and Mare Mézard. Generative diffusion in very large dimensions. Journal of Statistical Mechanics: Theory and Experiment, 2023())093402, 2023.
[12] Adam Block, Youssef Mroueh, and Alexander Rakhlin. Generative modeling with denoising auto-encoders and langevin sampling. arXiv preprint arXiv:2002.00107, 2020.
[13] Nicholas M Boffi, Arthur Jacot, Stephen Tu, and Ingvar Ziemann. Shallow difffusion networks provably learn hidden low-dimensional structure. arXiv preprint arXiv:2410. 1275, 2024.
[14] Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning, pages 4735-4763. PMLR, 2023.
[15] Minshuo Chen, Kaixuan Huang. Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. International Conference Machine Learning, pages 4672-4712. PMLR, 2023.

---

[16] Sitan Chen, Sinho Chewi, Holden Lee, Yuanzhi Li, Jianfeng Lu, and Adil Salim. The probability flow ode is provably fast. Advances Neural Information Processing Systems, 36, 2024.
[17] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.112111 2022.
[18] Sitan Chen, Vasilis Kontonis, and Kulin Shah. Learning general gaussian mixtures with efficient score matching. arXiv preprini arXiv:2404. 18893, 2024.
[19] Hugo Cui. High-dimensional learning of narrow neural networks. arXiv preprint arXiv:2409.13904 2024.
[20] Hugo Cui, Freya Behrens, Florent Krzakala, and Lenka Zdeborová. A phase transition between positional and semantic learning solvable model of dot-product attention. Advances in Neural Information Processing Systems, 37:36342-36389, 2024.
[21] Hugo Cui, Florent Krzakala, Eric Vanden-Eijnded and Lenka Zdeborová. Analysis of learning a dow-based generative model from limited sample complexity. arXiv preprint arXiv:2310.03355. 2023.
[22] Hugo Cui and Lenka Zdeborová. High-dimensional asymptotics of denoising autoencoders. Advances in Neural Information Processing Systems, 36:1 1850-11890, 2023.
[23] Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. arXiv preprint arXiv:2208.05314, 2022.
[24] Jacob Deasy, Nikola Simidjievski, and Pietro Liò. Heavy-tailed denoising score matching. arXiv preprint arX09:21 2021.
[25] Zehao Dou, Subhodh Kotekal, Zhehao Xu, and Harrison Zhou. From optimal score matching to optimal sampling. arXiv preprint arXiv:2409.07032, 2024.
[26] Weiguo Gao and Ming Li. How do flow matching models memorize and generalize in sample data subspaces? arXiv preprint arXiv:2410.23594, 2024.
[27] Khashayar Gatmiry, Jonathan Kelner, and Holden Lee. Learning mixtures of gaussians using diffusion models. arXiv preprint arXiv:2404. 18869, 2024.
[28] Anand Jerry George, Rodrigo Veiga, and Nicolas Macris. Analysis of diffusion models for manifold data. arXiv preprint arXiv:2202.0443: 04339999 2025.
[29] Anand Jerry George, Rodrigo Veiga, and Nicolas Macris. Denoising score matching with random features: Insights on diffusion models from precise learning curves. arXiv preprint arXiv:2502.00336. 2025.
[30] Davide Ghio, Yatin Dandi, Florent Krzakala, and Lenka Zdeborová. Sampling with flows. diffusion, and autoregressive neural networks from spin-glass perspective. Proceedings of the National Academy of Sciences, 121(27):e211180022) 2024.
[31] Sebastian Goldt, Marc Mézard, Florent Krzakala, and Lenka Zdeborová. Modeling the influence of data structure on learning neural networks: The hidden manifold model. Physical Review X, 10(4):041044, 2020.
[32] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[33] Yinbin Han, Meisam Razaviyayn, and Renyuan Xu. Neural network-based score estimation in diffusion models: Optimization and generalization. arXiv preprint arXiv:2401.15604 2024.
[34] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851. 2020.
12

---

[35] Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptic harmonic representation. arXiv preprint arXiv:2310.02557, 2023.
[36] Zahra Kadkhodaie and Eero P Simoncelli. Solving linear inverse problems using the prior implicit in a denoiser. arXiv preprint arXiv:2007.1364 2020.
[37] Mason Kamb and Surya Ganguli. An analytic theory of creativity in convolutional diffusion models, arXiv preprint arXiv:2412.20292, 2024.
[38] Frederic Koehler, Alexander Heckett, and Andrej Risteski. Statistical efficiency of score matching: The view from isoperimetry. arXiv preprint arXiv:2210.00726, 2022.
[39] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11))2278-2)) 2324, 1998.
[40] Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems, 35:22870-22882. 2022.
[41] Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of score-based generative modeling for general data distributions. In International Conference on Algorithmic Learning Theory, pages PMLR, 2023.
[42] Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Towards faster -onyyyomit convergence for diffusion-based generative models. arXiv preprint arXiv:2306.0925// 2023.
[43] Gen Li and Yuling Yan. Adapting unknown low-dimensional structures score-based diffusion models. arXiv preprint arXiv:2405.14866 2024.
[44] Marvin Li and Sitan Chen. Critical windows: non-asymptotic theory for feature emergence in diffusion models. arXiv preprint arXiv:2403.016333 2024.
[45] Marvin Li, Aayush Karan, and Sitan Chen. Blink of an eye: a simple theory for feature localization generative models. arXiv preprint arXiv:2502.00921, 2025.
[46] Puheng Li, Zhong Li, Huishuai Zhang, and Jiang Bian. On the generalization propertics of diffusion models. Advances in Neural Information Processing Systems, 36:2097-2127, 2023.
[47] Xiang Li, Yixiang Dai, and Qing Qu. Understanding generalizability of diffusion models requires rethinking the hidden gaussian structure. arXiv preprint arXiv:2410.24060, 2024.
[48] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arX1v:2210.02747, 2022.
[49] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprini arXiv:2209.03003, 2022.
[50] Song Mei. U-nets as belief propagation: Efficient classification, denoising, and diffusion in generative hierarchical models. arXiv preprint arXiv:2404. 18444, 2024.
[51] Francesca Mignacco and Francesco Mori. A statistical physics framework for optimal learning. arXiv preprint arXiv:2507.07907 2025.
[52] Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. In International Conference on Machine Learning, pages 26517-26582. PMLR, 2023.
[53] Jakiw Pidstrigach. Score-based generative models detect manifolds. Advances in Neural Information Processing Systems, 35:35852-35865, 2022.
[54] Arnu Pretorius, Steve Kroon, and Herman Kamper. Learning dynamics of linear denoising autoencoders. In International Conference on Machine Learning, pages 4141-4150. PMLR, 2018.
13

---

[55] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.066/5,, 1(2):3 2022.
[56] Maria Refinetti and Sebastian Goldt. The dynamics of representation learning in shallow, non- linear autoencoders. In International Conference on Machine Learning, pages 18499- 18519. PMLR, 2022.
[57] Maria Refinetti, Sebastian Goldt, Florent Krzakala, and Lenka Zdeborová. Classifying high- dimensional gaussian mixtures: Where kernel methods fail and neural networks succeed. In International Conference on Machine Learning, pages 8936-88668888 8947. PMLR, 2021.
(58) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[59] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted interventionnMICCAA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. pages 234-241. Springer, 2015.
[60] Brendan Leigh Ross, Hamidreza Kamkari, Tongzi Wu, Rasa Hosseinzadeh, Zhaoyan Liu, George Stein, Jesse C Cresswel,, and Gabriel Loaiza-Ganem. A geometric framework for understanding memorization in generative models. arXiv preprint wrXiv:241..0011/ 2024.
[61] David Saad and Sara A Solla. Exact solution for on-line learning in multilayer neural networks. Physical Review Letters, 74(21):4337, 1995.
[62] David Saad and Sara A Solla. On-line learning in soft committee machines. Physical Review E, 52(4):4225, 1995.
[63] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic rext-to-image diffusion models with deep language understanding, Advances in neural information processing systems, 35:36479-36443 2022.
[64] Antonio Sclocchi, Alessandro Favero, Noam Itzhak Levi, and Matthieu Wyart. Probing the latent hierarchical structure of data via diffusion models. arXiv preprint arXiv:2410.133: 2024.
[65] Antonio Sclocchi, Alessandro Favero, and Matthieu Wyart. A phase transition in diffusion models reveals the hierarchical nature of data. arXiv preprint arXiv:2402. 2024.
[66] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data. Nature, 631(8022):755
759, 2024.
[67] Bernard W Silverman. Density estimation for statistics and data analysis. Routledge, 2018.
(68] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[69] Roman Soletskyi, Marylou Gabrié, and Bruno Loureiro. A theoretical perspective on mode collapse in variational inference. arXiv preprint arXiv:2410. 13300, 2024.
[70] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[71] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXivv2011.13456, 2020.
[72] Joshua B Tenenbaum, Vin de Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 290(5500):2119-2222 2000.
14

---

[73] Emanuele Troiani, Hugo Cui, Yatin Dandi, Florent Krzakala, and Lenka Zdeborová. Fun- damental limits of learning in sequence multi-index models and deep attention networks: High-dimensional asymptotics and sharp thresholds. arXiv preprint arXiv:2502000901, 2025.
[74] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[75] Rodrigo Veiga, Ludovic Stephan, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Phase diagram of stochastic gradient descent in high-dimensional two-layer neural networks. Advances in Neural Information Processing Systems, 35:23244- 23255, 2022.
[76] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-167 2011.
[77] Pascal Vincent, Hugo Larochelle. Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
[78) Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020.
[79] Binxu Wang and John J Vastola. Diffusion models generate images like painters: analytical theory of outline first, details later. arXiv preprini wxXiv23303.02-90 2023.
[80] Hanyu Wang. Yujin Han, and Difan Zou, On the discrepancy and connection between memo- rization and generation in diffusion models. ICML 2024 Workshop on Foundation Models in the Wild, 2024.
[81] Wenhao Wang, Yifan Sun, Zongxin Yang, Zhengdong Hu, Zhentao Tan, and Yi Yang. Repli- cation in visual diffusion models: A survey and outlook. arXiv preprint arXiv:2408.0000 00000, 2024.
[82] Yuqing Wang, Ye He, and Molei Tao. Evaluating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 37:19307-19352, 2024.
[83] Joseph L. Watson, David Juergens, Nathaniel R Bennett, Brian L. Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, (20(7976):1089-11
2023. [84] Kilian Q Weinberger and Lawrence K Saul. Unsupervised learning of image manifolds by semidefinite programming. International journal of computer vision, 70:77-90, 2006.
[85] Andre Wibisono, Yihong Wu, and Kaylee Yingxi Yang. Optimal score estimation via empirical bayes smoothing. arXiv preprint 20Xix:2402.07747, 2024.
[86] Yuchen Wu, Minshuo Chen, Zihao Li, Mengdi Wang, and Yuting Wei. Theoretical insights for diffusion guidance: A case study for gaussian mixture models. arXiv preprint arXiv:2403.01639, 2024.
[87] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 0784077777 2017.
[88] TaeHo Yoon, Joo Young Choi, Sehyun Kwon, and Ernest K Ryu. Diffusion probabilistic models generalize when they fail to memorize. In ICML. 2023 Workshop on Structured Probabilistic Inference { Generative Modeling, 2023.
15

---

[89] Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Peng Wang, Liyue Shen, and Qing Qu The emergence reproducibility and consistency in diffusion models. In Forty-first International Conference Machine Learning. 2023.
[90] Kaihong Zhang, Caitlyn H Yin, Feng Liang, and Jingbo Liu. Minimax optimality of score- based diffusion models: Beyond the density lower bound assumptions. arXiv preprint arXiv:2402.15602, 2024.
[91] Huminhao Zhu, Fangyikang Wang, Tianyu Ding, Qing Qu, and Zhihui Zhu. Analyzing and improving model collapse in rectified flow models, arXiv preprint urXiv:2412.08/75, 2024.
16

---

1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract accurately describes the topic, technical contributions, and points of discussion of the manuscript.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The manuscript considers a simplified solvable model, for the purpose of theoretical analysis. This fact is stressed in the title, and the simplifications clearly discussed the main text.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
17

---

The assumptions are stated in the main text, and the detailed derivation of the technical results is provided in the Appendices.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The implementation details are given the Appendix, and the code used
provided supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code is provided in the supplementary material, and will be released public online repository upon de-anonymization.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc
aublic/guidde//oddeuubbisss www.oeeeSuuuiissiooooooooooogg for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppplgguuuiiaeeeaec
Saaeee//CoSSSubisssS.S.. www.ssssiicgcyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All experimental details are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All experimental curves are for single simulation. Its good match with the theoretical prediction is consequence of the concentration of metries of interest in high dimensions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: The present manuscript is theoretical in nature, and all the experiments are very small in scale and simply run personal laptop.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httus://neuripps.c
Answer: [Yes]
Justification:
The authors reviewed the Code of Ethics, and the research conforms thereto.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: the present paper is theoretical and aims at advancing the field of machine learning theory. There are no societal impact we feel needs to be discussed particular.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The research poses no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper doess not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy pettps://neurr CcCC/oofeeeeecccs///2////// for what should or should be described.
23

---

A Derivation of Result 22.1
In this Appendix, we detail the derivation of the tight ODE description (10) of the SGD training dynamics (4. as provided Result 2.1 We sequentially examine the dynamics for the skip connection b and the weight matrix w.
A.1 SGD dynamics of the skip connection
We first derive closed-fone forsm expression for the evolution of the skip connection strength B over the SGD iterations. We recall that the latter read
bp+ b/
7 R2 d2
2(1 b33.). 2
(16)
keeping only leading order terms. Note that || (resp. 11=51//a)) asymptotically concentrate to A (resp. 1) in the limit d + co. Therefore, the increment db bp+1 by self-averages as
db = E, [1 - b3,) ) ba,
(17)
2m
The ODE (17) gives in closed-form the evolution of the skip connection strength over the SGD training 4 the considered high-dimensional limit. Note that interestingly. while the training
algorithm is inherently stochastic, the randomness averages out in the high-dimensional limit, leading the dynamics concentrate deterministic limiting ODE.
A.2
SGD dynamics of the weight matrix
A.2.1 SGD update
We now turn to deriving a similar tight asymptotic characterization for the evolution the weight matrix under the SGD dynamics. Let first write explicitly the SGD updates the derivative, and dropping the time index μ for readability, for < 7 I r, update of the weight matrix reads
(4. Developing d, the SGD
21
2n B, [(11 63.)at Vi
duiy
E, (o + v.p:)o(ws + USPt)) wis
(oot + vypi)l
20
R,
(arx) + (,00))
alws + vopa) Σ
W/ Wn of (w- + vypp))
Vd
8=1
20
E,
[aav BBa)) bb3,)
bord (wst + vypte]]
d Xwin
(18)
Vd
We introduced the shorthands
W T x
n _
1
2 =
w/ = a01 + 3c14.
(19)
Vd
dd
A.2.2 Expected increment
In the likeness of the settings studied by e.g. self-average. Let us accordingly evaluate the expectation E Eddw,,] over the running data sample x1/0 This can be compactly rewritten
62 56
we expect the dynamics asymptotically
E(dw;, =
(20)
with the expectation E, bearing over the manifold coordinate ~ π Conditional on the manifold
coordinate the expectation EE bears over the Gaussian random variable associated to the c-indexed cluster in (8), distributed as N (p((). E(e)). We remind the reader that the covariances (E(c)) c are assumed jointly diagonalizable. Without loss of generality, place ourselves in the basis in which
24

---

they are directly diagonal, and denote in the following by o; the i-th eigenvalue of E(c). Taking the expectation over (8). the expected increment then reads
2n
2n
E Eddw
4.0 =
Ea [olwn + Bypi)oluss + [opi)] wis +
vdl (1 -3,) EES [w + p.)
d
A3
2n
ba,
E [2. (ws + pypp
d
21
Wig Wy d
E x owo + espilo' (wt + v.pi)]
vd
6=1
Ci
21
Wiz Wh
E (afoww + uspilo' + v.ppil
d
8=1
21
2n
+
E. [x Ng (w), + MyPee)] bB:) (1(1
a [x. Ao (wt, + v.p)))
Vd
Vd
DOL.
D
27
27
2n Aw
+
3,(1
Bx X (w), + uype))
a4
3,3,EE [ o (wl, + uy
Vd
Vd
D1 11.te
D 20.4.0
(21)
The various coefficients Bt., Cas,,,,,eee C.t,,Dte can be computed leveraging the fact that the data components 1.0 are weakly correlated with the local fields w, 2 i.e. have @a(1/v) covariance. Using the expansions for weakly correlated Gaussian variables reported e.g. in [57] (Appendix B.
25

---

we reach Ang (,)
(22)
1 Biwn (T26())7) - -B (.M)
Bi = (>) +
(23)
vd 2igz
an
(120 BcM
(24)
vd Siy
176
1 -888
a
(T (7,50))))))) - (o (If wry (2:
was)
Vd
Pignng (ggg))
(I (7,6)) B.MgIoo (7, (D 24 win)
d76 ooW the
+ aelling
(1. (.MG
(25)
24/
Beei u.ie
pee (3,0(,00)) 01 + 1786 at
(26)
DoL.c
aiwa Vi @2Q2n00 P4i B²(0gn)²
1
the (7.76))) - Be (2)po (1))
(27)
D00.A.
1
W/y
The (7)
(28)
Vd Du ai 12n)²
1 1o (1:)+ + D10.E
1
B
the No - a
by
Vd 2n (Ox))
(29)
DILAE = (o Vd Q4 24 - 88 (Q
Wiy 0f
(he (( - -amn))-M(3hh Mo (7,
3
3
(30)
We introduced the summary statistics
M w TTTp (u(c)
Q w E(c)w
(31)
Vd
Q w w d
= =f a
T =(c) u(()
(32)
One also needs to introduce the further statistics
G w
=== (((),,
(33)
V
where we remind that the columns of E gdxR constitue orthonormal basis of the R- di-deennsional subspace & in which we aim to characterize the generated density. Finally, we also used the shorthands:
17, Ewy,os (o(w, u.Pi)o(ws + vaP.)l,
Ways we N (1)M Q(e
(34)
(7) = 1E,, (w + UyPr)], 15((1,0) 6) Bus was (w + u. (o8))
Wy N 1GM.G.9..
(35)
Why, we N (BeM ((700))
(36)
phe (T, 0,6) wu /o' (wy + (ypx)o((ss + vspe)wel,
wyy wa, we N MrM((ndee) (x.6,e)
A.e
(37)
The (v, 5) -Bur (o' (oe Uyps)o(ws + vopt)),
Why wa ~ (B)M ((TT))))
(38)
26

---

and
The 17, = BE P2o0 + tspe)uees
M 2y. Wo, We N (( (LAM(
Oir
0 ))
Be ((((
(39)
(6.0)
Lhe (22)8 (7,0)
Iusy (AA2) (ws + vsPi)l,
M N ~N (
( Ogz
BrQ D6
G.M1
(B.(09,s)
(40)
The
8, 0,, = Exy.woee (Ago' (wa + rspi)aeel
wg. We N (B.M
(10l))((a)))) (o((111)))
2xr
a a a(021180)) Did
))
(41)
The (7, = Eu 1229) (wa + uspi)l, 0
wo ~ N (B.013)
(ac(e).s)
OnT
a ar
(42)
(43)
Let us briefly summarize at this point the derivation so far. Starting from the SGD update on w, we showed that the expected increment can be written solely terms of a small number summary statistics M, Q, Q, S2 T, in addition to the weight components themselves. Next, we aim at closing these equations on the summary statistics, thus reaching concise low-dimensional description the SGD dynamics.
A.2.3 Update equation for the summary statistics
The training dynamics of the DAE weights w are thus governed by set of finite-dimensional summary statistics. To close the equations and reach a self-contained characterization, we now turn to deriving the induced dynamics of the summary statistics. To that end, following e.g. [3I] it proves convenient
to first introduce new set of summary statistics densities. For any R 4 R denoting joint sequence of eigenvalues {e(c)}- let us assume the existence of the densities m. IR x
F(R*, IR) R, g. xF R R) IR and 0 (R*)²² x F(R R) + R so that the summary statisties M Q, Q", T, G, can be decomposed as
M / dgm=((),
(44)
/ deq(e) Qc:
Q¹
(45)
2 degle),
(46)
/ dolr+(())
T
(47)
G / dgg(e),
(48)
/
p
dep)(().
(49)
The following subsections focus deriving the updates of the summary statistic densities m())g()) (()))))))))( inherited from the SGD dynamics of the weight matrix 21
A.2.4
Overlap m+)))
The expected increment for m (g) can also be decomposed as
E Eidm (e) = E,E, E 1Eddm (e)
(50)
27

---

with
- (dmm (e)x)
(7, (0)s + - 63:)44 (7)
21)
6=1
(1)((,)7) (1 b a.m * (0), - bajmm (1 3.M,II. (x))
2
(a] 0c)
2 (9gg))
88
(az + (e)
(Th - (1,7)) - Br Σ 2 p. (7,
144
a2 - bB:) m (e)
1 (o - 31 31110))0
0 24 (Q())
a bm (0m²((0) (g),
the (10p az az)m)
2 Le af ( 22,)
+ 1 63:)) A.c
X'o (7, 7)0rk(
Bi(1 - 63. b3:)) em (e) +
(The (7)7) (147)--Mrrkee
-8Q
2i (Q7x))
au,3:bl (7, 70ck) (e)
2 gam (
_tcc
Noor (1.77177)77) (1.7) M,B.B (7)))
Qyp af (Q
Xm)
(51)
A.2.5 Overlap g())
The expected SGD for g(e) can be derived along nearly identical lines. By the same token, for 1 < i < R, the decomposition
E(dg:(0) = E,E, Eg (2)
(52)
28

---

holds with
E dg
(0)]]= = (7,()((0)) + +1 (21--3,)) (1
2n
(1
bBq b(e))re - (0) (1 B,
+
(100())))
14
(af + ec) 2r8
(a + (c)
(1 (1:771))77 - - Be Σ 2roli (7,00011))
22/
a3 - bz,
The (1ow(11 - 3r (2 (7,7)
+
0/ 2ig (²(0g)))
bg (0),
(1 at em)
04 (0,)) )2
+ 8((1 63,) b1 Aio (1, (0))(0)
3.(1 bo)))
(1 ((a)))))1)) (1 -
+
(1.
Qy b3,
phc. (mmmm)))))))) -a (200)))0 -M (7,
Q (Q
>g:(e)
(53)
yielding the increment g(-) under the SGD dynamics.
A.2.6 Overlap q(-)
We now turn the summary statistic ((0) 31 First note that
B(d@]
1 Σ Eg
EE E, ded + Er,, (E ")T]
7
i=1
[E
E. (e)].
(54)
We have separated the linear term and the quadratic term. follows that the density statistic q(-) can be similarly decomposed as
E [[dq(e)) E,E, 1E E, B. 1 (e)).
(55)
29

---

In the following, we sequentially examine the linear and quadratic terms. The expected increment for the linear term da(1)() (-) can be read from 21 as
E ((((q
2m
1,, c)g(e)) + (1 b3:) (7)m
=
((1 - b8e) - ba ((0)) (160(1,)) - (x))
2499
(af + 88 (8 Qey
(1
Dig - (22
exy
+ The +000001 (1 (())))))
(az ee
(Te (10
3e Σ On ())
2
af - b (2)))
the foo ( Belxx))
2 2i - (Q
atb4(e)
The - [ +
2 - a2(2)))
81 633 (3))(())000)
(Ihe (1:7) -
-eBBbbT (7, m))
Qi - (,
Q - of(2r)))
Aq(e);
+ (7 6)
(56)
We now turn to the quadratic term (g)2)()) Keeping only leading order terms,
d
1
(2d(((
2 "
1000 (7, [(1 b1:) -b3,)b33 + b² Bayay
4np
- -v(e) Σ
phatt ooo! (1, c, Les (Be bBe)ee bba
+ v(e) (11 (1-b3p) 1 tht' (7, 6,, bay 1.ti 10 07, 0,)) [B 63.) - -baaar]
v(0)
Σ I'aada
, 6,,,, 2n. eo cnae]
vr(e) (11 by phit'e
bar t [1 Q.y 18.3re++aae
+ 11--6/11--00///////4000-----000- (1- t.
bbrr(1 b3))) (1,6016661,,,,0,7,,11 + 6 blara 1' (7, (())
aarr)
+ + ++ 8)
(57)
30

---

We introduced the integrals
T. (x,0) = Be we (ol + type)o(ws + vspt)
wy:ws N (v) @ Miro)).00.. (2:61)
(58)
1ood (1 (7, = Eur, (ws, + uxpi)o (wi + v.pi)o (ws + voPe).
Wy, Wy,We, WS N (Bt1 Beg Bea)) @ M 2 (vse,5)
(59)
plstaada
oo', (1,,, 0) = Bu sy + v. (w. + vePe Ab11
Wy.Wes AB N (Be1 B1g 1) Mcr...): (1(c,)
(2..2.)
(60)
(7, 0) = Bu sgloo, + vx (we + u. Ap
We 18 N (1 S,, 0) M M(nea):: v (()tt
(61)
(s.r,))
phitstt.. (7,6,,11) Bes, 10 (0l + uyPe (o + v.Pe)o (ws)ows) + v.p())
Wy ws We: ~ N (B11 Bes Bia 3e.) o M MC (7.0)))) (1.c,d))
(1))) da.tadiee)
(62)
pe o'
(7) = Eug. (0' (wy + u, (w + u.P)) ) (wa + 2spe)A!1
Wy,Wer No. No (Bo), Sr2 Br, 1) 3 M ((nna)))
(63)
(7:684))
wa (7,
+ u. Pe (w + u.P:) o (wa + (spt)
Wy. ws. Xi: ~ N (B Btz: 3r 0) 3 M MC (,)), v (2,ts,
(64)
o' (ulttahte"((,,,,, = Eu lo' (wy + u. )o (w. + 2
Wy.wes 2/. d ~ N (B,, Sr,, 1) 2 M (.)))))) P (4
(65)
(7:688))
( (7,,,0,1) = Eu,, wo ag.2 (wi, + up ) (w. + v.px)
Wy.Wes 18 11 N (1 (1e, Beg 0. 1) 2 M M(n p(400.).ba.daataa44
(66)
(1:6.844)
photats (7,e, - (0/ + uypt )o (w. v.p.)
o'
Wr, 28, 112 ~ N (Be., Br., 0, 3 M ())))))) (1..6.))
(67)
31

---

We further denoted
= 00:00 Q + Be
D
D.0)),r,b (7:,,5)
D
$ (2. (7:0,5)
Be Bn
Qaa
an 2,
(3).)
O12 2es
(2/6.0)
o7 on Ogs
01g Qca
Qa6
D...3..c
D'aitio
faitsie
dade
D4
Be, 2r
Mtadiio
Dig.ta.c
4 b9,b4. (1,66,1))
Br 2
Brg
Qi
2y.
daiiii (1566.)
an1 Qa:
Dtaitai
Dfa.ts.c
att Qs.
by an 2r.
Cita Qai
City Q6.
Qu
Be, Q
Br, Q
Br
Bi
(1,00,))
B 2c Br, O
Biz Qc
Qis
Q
Br
Q%
Q.
Whibe alaitio eey On 2no
Dhita Dtaitaso
an 27o at Qes
Br, Q
ta (1.6,6,6)
Br O
012 Les
Q58
Be, Q
Big Q
0
Q
D
au Q78
an Qre
(l1...0).t
Dlzit1ic on 2,8
Dlzitzic
Ctty Qes
04ty Qa
(68)
(35,,8,))
Ctta Qes
Q68
Qi.
an Zze
ai Que
Q
Q.
A.3
Update for
Finally, we can ascertain the asymptotic evolution the time encoding weights w. the considered limit, the update du again concentrates. As above, let us decompose the expected increment
E[de,] = E, E. [E dof;)
(69)
with
2n Eddote =- d
Σ
2 2o (7,0) - - -1- (3 (7, 7) bo,, 10, (7,7) + Ney
(2,
(70)
8
32

---

A.4 Continuous time limit
Equations (5..55 (3),553...00 560 and (7) provide the update equations for the summary statistic densities
ml)).g()) ( under SGD steps which take the form
d (m) Fm ML) Q.b).
2n
2n dg = Fo (())) M,), MQ
d
2n dq Fale, (()) M, M.Q,
(71)
where the update functions Fm.q.g denote the right hand sides of (1 (53. (56) and (57). and we have omitted the time step indices to ease the notations. From 44). these updates translate directly at the level of the summary statistics M, Q, 2, G into
d dM FM (M, Q,b,,,
/ Fl(e)
21j
Fa(.) / (gle)
JG = (M (,
2n
d
Fo(-) / m()
dQ
2,b,),
2T
Fo(-) / (()
dQ = Fc
2T
(72)
We remind that from the skip connection strength similarly obeys
d db (b(b),
(73)
21
where the update function F, corresponds to the right hand side of equation 17 Similarly, from 70). the time encoding weights obey
de = Q.b...
(74)
2n
where F, corresponds to the right-hand side of 70). Now remark that in the asymptotic limit d - oc. the coefficient 4/2n tends to zero. Introducing the time variable 2ur/a, so that dØ 2n/d, the
discrete processes and 73) are thus asymptotically described by the limiting ODEs
dM
d3 FMr Q, Q,bb)),
dQ
d0
dΩ
= F,
d0
db Fb(b).
d9
F. Q, 2, b,
(75)
dg
Finally, the ODE for b. governing the dynamics of the skip connection strength b over the SGD optimization dynamics, can be solved in closed-form as
[3]
(AR, (AT + boe (AE,))))
b(1) AE B + [.
(76)
where by designates the value of at initialization. This completes the derivation of Result 2.1
33

---

0.5
0.5
0.4
0.8
0.3
0.2
0.3
0.8
0.2
0.4
R2
R2
0.2
0.0
0.0
Figure 5: Evolution of the summary statistics (11 M (left), Q (middle) and skip connection strength b (right), characterizing the dynamics of the AE parameters (3) under SGD dynamics
(4). Parameters used, and the target density P was taken to
Lanh, 2, A 0, 0.2, 9 = (1/2) be a Gaussian mixture with three isotropic clusters (see also Fig. in the main text). The weight vectors initialized along the centroids of the target density, with norm 0.1, while the initial skip connection strength is by = O. Dashed lines: theoretical characterization of Result 2.1 Continuous lines: numerical experiments in d 1000, for single run.
2.-
1.0
0.01
R2
3.25
Figure 6: Evolution of the summary statistics (31) Q (left), b (middle) and skip connection strength M (right), characterizing the dynamics of the AE parameters (3) under SGD dynamics 4 Parameters tanh, 2, 0.784, 0.2, g {1/2} were used; weights were initialized with random independent Gaussian components and by = O. Dotted continuous lines numerical experiments for target density P given by the set of MNIST sevens. Continuous lines: numerical experiments for a unimodal Gaussian target distribution with covariance matching that of the set of MNIST sevens. Dashed lines: theoretical predictions of Result 2.1 for the latter Gaussian target density.
A.5 Numerical validation
We plot the theoretical predictions of Result 2.1 for the evolution of the summary statistics M, Q. Q, G. (31 under the SGD dynamics (4) in Fig.5) for a Gaussian mixture target density p with three isotropic modes, learnt by an AE with r hidden units and tanh activation, using learning rate 0.2 and weight decay
0. The centroids of the clusters were taken tee, 62 for two orthonormal vectors e1, e2, and the columns warm start as 0.1 e1,2. Finally, for simplicity, the expectation E, in (4) was chosen to bear over delta distribution around G = {1/2). instead of the full integral over [0, 1]. Including more points in the grid G was not found to significantly alter the qualitative aspect of the generated density. Fig reveals an overall good agreement between the theoretical predictions of Result (dashed lines) and numerical experiments (solid lines), obtained by simulating the model in large but finite dimension 1000.
weight matrix w were initialized with a
Fig. simmilayyy contrasts numerical experiments for target distribution corresponding to MNIST images sevens (dotted lines), a Gaussian target density with matching covariance (solid lines), and the theoretical predictions of Result 2.1 for the latter. All experimental details are specified in the caption. Although the agreement between the three curves is overall good, discrepancies appear, in particular due to the rather low dimensionality d 784.
34

---

A.6 Extensions
We briefly describe, for completeness, how the analysis can be generalized to characterize the learning of more complex DAE architectures. Namely, we discuss how the derivation can be adapted accommodate (a) untied weights and (b) time encodings.
Untying the weights- The analysis reported in the present appendix can be extended to untied DAE architectures of the form
v T x
(b = b xxxx+
v
(77)
Vd
vd
trained with online SGD
pu+1 2/ d2 (0 |af mo (araf BBatt) 11)
(78)
A U,+1 Uu V E | fommaann (arzt' + B,a - -"7uu upz
(79)
A
1/11 Up V F |0 Bannaann (orar e + Beat)ll - -"
(80)
Such an extension, however, comes at the price of more cumbersome expressions, as the summary statistics (3T) needs to be introduced for the two sets of weights u, v, in addition to statsssstasssics
of the form o [(e)o/d and u v/v/d. We refer the interested reader to Appendix B of (56) where such
derivation is detailed, in a closely related setting. Experimentally, in the probed settings, we did not
observe a significant effect of (un)tying the weights the qualitative phenomenology discussed in the main text.
Deeper architectures This manuscript focuses on the simpler instance of DAE architecture (3) namely a two-layer model with a skip connection. It is of interest to remark, however, that the technical ideas employed in its analysis can be extended to study more intricate models, at the cost of considerably heavier mathematical expression. For this reason, we present in this paragraph a model of multi-layer DAE which anticipate is amenable to similar analysis, and sketch the main ideas underlying the latter; the detailed analysis, however, is left for future work.
Consider the depth-21, model webeeu (x) hr, where the successive post-activations [he)eE[2) are defined by ho
t+1
(uft))The he
h411= = b6+1// he +
vd
06+1
Vd
(81)
In simple words, this multi-layer model simply results from the composition of multiple DAE layers 3 We denote 73 the width of layer f, so that ue, Ug Rdx We again consider the limit d - oc, while r3 = (1) for all & E [Z]. We anticipate that the analysis of the online SGD training dynamics trained with online SGD
111 =
1) JOyE, E,
1 - -S40 Me (mrag + Braf) Blatt)()
(82)
M+1
d²
|?2
"+1 uh, = V F,
x S40 heeu (aray) BBart)
nna" Up:
(83)
2
A
2411 o/2 =- v E, 1 of (arzt + B
d
Up
(84)
should once again be amenable to theoretical analysis, and shown converge terministic limiting ODEs, borrowing the same technical ideas as previously. We highlight
set of de-
some of the differences in the computations. Massaging the update equations 882 reveals that
they depend as previously in (18) on random variables of the form X = (a)) 2/Va, X
(uf) (oVT="/Va, over which the integrals appearing in the final expressions bear. The updates will also involve further summary statistics of the type Qun, (c) (1f) E(ene /a, Qua,e," (c) (af) TTejoo /d, Quu.f.f
(u')T=' ='/aa, 20. = (w') (M)T=%/Va =2/aa, Xo
Exejaa /4, and (at) /a, Qualt (af) w. /a, which will appear in the end
(c)
QueiE, Tf Quiile
35

---

result, and whose dynamics are tracked by a set of limiting ODEs. Importantly, we thus trade matrix r X r summary statistics for tensors of size L x x r. thus resulting more cumbersome equations. We leave for future work thorough theoretical analysis of this model, and the study of the effect of deeper architectures on the generated density.
Attention models Transformer architectures 74 are also often used to parametrize diffusion models. Consider d = L.d' dimensional data x, divided in I. patches 21,-.., E Rd and reshaped into a sequence (1q E IR Lx Consider the folowing attention-based denoiser:
fi (x) softmax (soq(z)ww" (s)(())))))))))))))))))))))))))) scq(x),
(85)
parametrized by w Rdxr trained on the denoising loss E3 fl (0,30 + 3,01) seq(z) (12) Let us further assume that all patches Zę for { E [L] are drawn from generalized Gaussian mixture of the form (8). with the different latent variables og allowed be correlated across rows thus reprising. and slightly extending, the data model in [19 In the asymptotic limit d' + co, L, r = O& (1), this simple model, close to that considered in 20 and is expected to once more amenable to analysis
using the same set of technical tools, in approach analogous in spirit to that of [6]. Let us mention that more complex (e.g. deep) architectures are also anticipated to be analyzable, leveraging similar ideas to e.g. [73]
We remark that, on a fundamental level, the anticipated analytical tractability of the above models stems from their pertaining to the class of sequence multi-index models introduced in [9]. This formalism allows for an unified analysis of this class of models. A study of online SGD on sequence multi-indes index models was reported in full generality in the recent work of [5T], to which we refer the curious reader. Finally, we note that while we assumed in this work finite-width networks r = Oa(1), the analysis can also in theory be extended to accommodate wider networks. We refer the interested reader (75 where such an extension is studied for two-layer feedforward networks, and leave this direction for future work.
A.7
T + Do limit
The summary statisties describing the weights of the DAE (3) at convergence captured by the
T - oo solutions the ODEs of Result 2.1 The value of the skip connection b at convergence
admits the simple closed-form expression
AE, [3]]
D00
(86)
AE, [ E, [a
as a function of the schedule fonetions Qee 8.. Interestingly, the target density only enters this expression through the effective variance A, which provides an intuitive notion of the spread of the data density, without allowing finer description of its structure - which captured by the network component of the DAE (3), rather than its skip connection. In contrast, the other summary statistics Qoo Qxx, Moo, Go at convergence obey a set four coupled equations, which generically admit no simple closed-form expression, aside from simple cases. For illustration, we detail in the following subsection such a simple setting (Gaussian mixture target, linear network) where the equations at convergence be solved in closed-form.
A.8
Example: linear model, Gaussian mixture
The previous analysis provides a tight characterization of the training dynamies of the non-linear DAE (3) under the SGD dynamics (4). For completeness and intuition, we conclude the present appendix by expounding a special simple case where the limiting ODEs of Result admit a compact,
closed-form expression, and consider the linear case (xx) Irr 1, = O, when learning target binary Gaussian mixture with isotropic clusters p = 1/2N Ia) + 1/2N(-p, Ia). We assume the squared norm || asymptotically concentrates and denote by its limiting value. Finally, we
4.c consider the limit of vanishing learning rate 1) + 0. In this limit, the quadractic term 9/2)- which is of
36

---

order ( can be neglected. The limiting ODEs
75 simplify in this case to
d
M E, 1 2( + + Q (a) 0 + p)) - 2 2 ((11-b3))))(0 1) agū + X| M
d0
d
di 2 -E, 22 (B) + 2 + -f)) 2 ((1 -81-b,8).. + - + 0º
(87)
The evolution of the skip connection b, on the other hand, remains unchanged from (7)). Let us now seek solution of (87) at convergence, in the limit 0 + co. It can be straightforwardly verified that,
E, [&, E2 [a]]
E. [0?+a]] ] P 2/2
M M= pQ.
(88)
E, p) af]
is a solution of 87) at convergence. Note that the identity M = pQ, together with the definitions w/d, implies that lies entirely in span(4). In other words, the weights w of
M w 4/va, the DAE recover perfectly the direction μ along which the data distribution p exhibits non-trivial structure.
Turning to the generative process, we aim at describing the generated density in the one-dimensional subspace & = span(M) in which the original distribution p exhibits non-trivial structure. The SDE (12) describing the evolution of the projection of a sample in this subspace is for the considered setting linear, and can be solved in closed form as
dddA,
Z, = Zoeo
Zo ~ ~N,, Q).
(89)
In the orthogonal subspace sрan(μ) the law of sample is still given by an isotropic Gaussian, as described by equation (13) in Result Therefore, the law of sample X, remains Gaussian at all times, with the variance in span(n) increasing over sampling time to adapt to and approximate- the structure of the target distribution p in this subspace. This simple linear case sheds light on the workings of the DAE-pan MAE-parametrized diffusion models. Over training. the weights identify and learn the relevant structural features of the target distribution. Subsequently, the learned weights drive the generative process to reproduce the target structure in the identified subspace, while approximating the density in the orthogonal space by isotropic Gaussian.
B Derivation of Result 2.2
In this section, derive the tight characterization of Result 2.2 process
2.2 for the learnt generative transport
B.1
Generative SDE
We remind the generative SDE, leveraged to generate samples from p(() starting from Xo ~ ((O,La)):
dX:
3t C: af fhr.w. (X,)
dxe Et
3 B.
of
x.
V2c,WW,
(90)
dt
Ctt
at
with W. Wiener process and 6, the diffusion schedule. Introducing the shorthands
di
Be
r  = B,- Be + E
(91)
at
b, + dt
Et
A
(92)
at
a?
the generative SDE can be written more compactly as
dX. = A, +r, Wr
w X + P:U+
+ /2,,WW.
(93)
dt
Vd
Importantly, note that the non-linear term (-)) acts on the projection of X, in the space W, spanned by the columns of the trained weights matrix w,. Furthermore, its image also resides in W,. Thus, the challenging non-linear part of the transport only acts in a small finite-dimensional space, and can be handled carefully in isolation. In contrast, the dynamics in the orthogonal space W,- is still non-trivially high-dimensional, but simply linear and thus easily to analyzed. This motivates one to examine in succession the variable Z, w,, Xs/Va and the projection Y, = 1I X, of X, in W.J.
37

---

B.2
Dynamics in W
Let us first ascertain the evolution of Zz. which tracks the evolution of a sample X, in the weight space W, It follows directly from (93) that Ze obeys the di-mennsional SDE
d Z = = + 1,Q,0 (Ze + ppee) + V2E
(94)
with B, r- --dimensional Wiener process, and Q, the summary statistic sharply characterized in Result 2.1 This recovers equation (12) of Result 2.2
B.3
Dynamics in W
In W the transport induced by the SDE 93 is simply linear:
dY;
ATY, + 2e
(95)
dt
with H2 here (d r) dimensional Wiener process. This SDE admits compact closed-form
solution
j dop
iepp
- ddhhj
Y = e0 Yo +
e
v de s.
(96)
By Itō isometry, Y, is Gaussian with law
2 sidss
f
N N
Oww
1 +
e
Ed
Thy
(97)
which recovers equation
13). This completes the derivation of Result 2.2
B.4 Discretized sampling
As final remark, let note that the derivation presented in the present Appendix can be carried out in completely unchanged fashion starting from any discretization of the generative SDE (I. Let la = 0, l,..., LT E (0, 1) and consider the Euler-Mayurama discretization of stochastic process
for k E [0.T 1]:
Xx+1 - Xx =(tx+1 - tx) (Box.
âre Bea + Etx at
Oie fo..w.. (Xy) +
diee at
Geu
)
X
at
2012 (tx+1 tK)
(98)
starting from Xto NO, Lt). In (98), Eu ~ NO0,, La) independently for each step k. Then the following version of Result 2.2 holds:
Result B.1. (Discrete dynamics) Consider a discretization ti....,tr tr E (0,1) and the discretized sampling process (Xx)KElL.T) (98) Denote Ys Tyy, Xx and Zy w x./Va, for process X
satisfying the generative process
98 from initialization X, N Ta). Then Za follows the
low-dimensional stochastic process
Zet1 - Zy = (t=+1 [T Z, + P eto (Zx + Pt, u+]] + 22a, (tx+1 - tx) t. (99)
from an initial condition Zo ~ No,, 2r), with Sa ~ N' I.) and E[GAGT] = -Saln. On the other
hand, Ys is independently Gaussian-distibuueed as
Yk ~ N (Ow/
T (140511-00 (140511-)44) (111-t)) πI
( (1(0))-1))42) +20 (Au 1) Uhe
wil
(100)
38

---

C Derivation of Corollary 2.3
Result 2.2 already provides tight asymptotic characterization of the law of sample X, in terms of its projection Ze (2) in the weights space W, (characterized by a r-dimensioonal ODE) and its Gaussian component Y, (13) in the orthogonal space W weakness of this characterization, however, lies in that it relies on a training-tine dependent space W,, with respect to which the characterization formulated. Intuitively, this space rotates and changes as the model further trained, making the result rather unwieldy. To palliate this shortcoming, one would rather select a fixed, r independent, reference subspace & of finite dimension R (1()), and transfer the characterization of Result 2.2 to this fixed subspace. Formally, this means ascertaining the law of projection of X; in &, from that of its projections Wr, W,- This made possible because the overlap between the spaces Wi and & is captured by the summary statistic G,, characterized in result 2.1 Formalizing this agenda is the object of the present Appendix.
Let us fix an orthonormal basis {ey {ey} of &, stacked vertically in the matrix E E RdxR, We remind that aim characterizing the law of E X,. To that end, for any 1 S j R, start from the decomposition
oj X, (IT,(Tw,,c)) (Tww, Xx) + e. T, Y,
(101)
where decomposed X, into its projections in W,, Wt Note that, from Result 2.2 the two terms of this decomposition are independent. In the following, we sequentially ascertain the distribution of each of the terms in the decomposition (101).
C.1 Law of (Ilw,e)) e,) (IIw,X))
To compute (,w, e2) ([w,, Xt), first aim to decompose e; X, in basis of W,. Let consider the eigendecomposition of the summary statistic 2, = w. w- (characterized in Result 2.3) as
e, U,SUU
(102)
This means that B, = )/a w. forms set of orthonormal vectors (or set of orthonor- mal vectors plus zero vectors if Q+ rank deficient), which we will use as basis. We denoted St the Moore-Penrose pseudo-inverse of S,. The components of the reference vectors E gdx (with columns (eyh) and X, in this basis are then given by
B,E 1 W [
(103)
B,X, 1
UTT U. w, = (5;)
(S,
(104)
Vd
where Ze is characterized in Result
2.2 Then, very simply, the decomposition of X, in the reference
basis E restricted to W, reads
(Iw, =G,0
=
(105)
C.2 Law of E Il,, Yt
In distribution, E I w. Y, inherits the Gaussianity of Y,, as established in Result 2.2 It has mean zero and covariance
2 dd E TT,
2 2fdsa
E.
ET (Taa- B, B.)E
2 2/de [UR - G. C.G.
(106)
39

---

C.3
Law of E X,
One is now position ascertain the law of E Xs. Putting the above results together, in distribution:
2/dos ORRE - - G.
E x
x 4 G. G 2z +
(107)
which recovers Corollary 2.3
D Additional experiments
D.1 Additional details on the numerical experiments
In this Appendix, we provide further specifications on the numerical experiments illustrated in Fig. 3 and Fig.14
Generative process- In all the figures, the sampling was carried out by discretizing the interval (0, 1) in N steps la = 1/N for k E [0, N]. and running the discretized SDE (98) in experiments, and the associated theoretical characterization of Results B.JulssB.1.. [. and 2.3 for the theoretical predictions, up to
stopping time < ł < 1. In Fig. 7 N 100, = 0.95; in Fig. 8 N 100, f 0.98 and in Figs. ?? and 4 N = 50, ty = 0.98. Note that all choices for N, by are up to the experimentalist, and captured by the theoretical characterizations. Generically, one needs to opt for f < 1 due to the DAE-parametrized SDE (7) being ill-defined at = 1, since a1 = O. This an artifact the neural network parametrization; the ground-truth SDE (I) is the hand well-defined even t 1.
Discretization of the manifold density x- In the generic case where π (()) (8)
not discrete, the ODE updates (TO) still involve an integral over dr (c), with spanning IR For instance, in the setting of Fig. at generation = 2. = r-- 2 and (c)) = , (() (c). The latter is however still characterized in terms of SDE I2 and not in closed form. As first step, thus generated 4000 samples from π, using the theoretical characterization of Result 2.2 and approximated the density using the scipy (78) implementation of Gaussian kernel density estimation (KDE), in order to access a smooth estimation of æ. The bandwidth was elected to be 1.5 times that determined using the Silverman method [67]. To perform the integral with measure dr we discretized и
10 10 grid, restricting the support to [-1.3, 1.5] x 2.5, where almost all of mass was found to lie. The relative weights of the 10 10 = 100 discretized points were then evaluated from the KDE estimation, and overall normalization was finally enforced to ensure the relative weights sum to used in evaluating the theoretical characterization of Result replacing the integrals over by finite sums over the 100 points of the discretization. All results have been observed to be rather robust with respect the choice of discretization, range, and bandwidth.
1. Finally, this discretization
Preprocessing the MNIST images- Finally, we detail the procedure used to evaluate the covariance of MNIST sevens used in Fig.3] The total MNIST training set was used, retaining only sevens. The data was vectorized (flattened), centered, and normalized by 300. The empirical covariance was finally evaluated over the entire dataset, and used to generate the Gaussian target density considered F
Evaluation of the Hellinger distance- To estimate the Hellinger distance between and (see Fig. 2 (right)) we first sample 5000 points from the trained diffusion model, project them in the considered subspace E, and approximate the density using the scipy implementation of Gaussian KDE. In the case Fig. 2 (right), we used = span(μ), and used 1000-points grid discretization of the interval 10, 10) for the purpose of the KDE. The Hellinger distance between p and the (KDE of) و is then numerically estimated as
H (TE) = /dz [=<(V(Te))a) - (T))
(108)
By the same token, the theoretical prediction of the Hellinger distance is obtained by sampling 5000 samples from the theoretical expression for á, as described Result 2.3 and repeating the same KDE procedure.
40

---

Figure 7: Evolution of the projected density Il generated by DAE 3 with r 4 hidden units and σ ReLU activation, trained on a trimodal Gaussian mixture, with - 0.2, À 1.5, f = t,g G {0.7}. from warm start. The generative SDE was up
0.1, Pt 0, 0z 1 to = 0.98, and the subspace & is spanned by the centroids of the target density. Different panels correspond to different training times Blue contours: contour levels of the theoretical prediction of Corollary 2.3 for the density Ộr. Colormap: numerical experiments large but finite dimension == 1000. Green contours: contour levels of the target density p. Over training time, the four branches the generated density rotate to align with the clusters of the target density, with two branches merging in the process.
Figure 8: Density 1, generated by DAE with 2 hidden units and tanh activation.
trained on a trimodal Gaussian mixture, with y = 0.2, À = 1.5, c, = 0.1, Pt = 0, = 1 f, 3r run up to = 0.98, and the subspace is spanned by the centroids the target density. Blue contours:
g = ( 2.8. The generative SDE
contour levels of the theoretical prediction of Corollary 2.3 for the density p Colormap: numerical experiments in large but finite dimension
1000. Green contours: contour levels of the target density
D.2 Additional examples
Trimodal Gaussian mixture For completeness, conclude this Appendix by illustrating the theoretical results 2.1 2.3 on an additional example, namely a trimodal Gaussian mixture density with
isotropic clusters.
We consider generative model parametrized by a DAE with r hidden units and ReLU activation. Fig. 7 illlustrates, for different training times the generated density Ô- projected in the space spanned by the cluster centroids of the target density. A comparison between the theoretical predictions (blue contour levels) and numerical experiments in large but finite dimension d = 1000 (orange colormap) reveals a good agreement. Interestingly, the modes of generated density Pr rotate over training time to align with the modes of the target density with two modes merging in the process. The resulting density P, at large training time exhibits a similar geometry to the target density without however perfectly reproducing it sign of the architectural bias due the limited expressivity of the model (3). which cannot perfectly generate the target distribution.
Perhaps unsurprisingly, this bias furthermore strongly depends on the architecture of the DAE. Fig. represents the density generated by DAE with r 2 hidden units and tanh activation, for the same target density p, with all parameters otherwise unchanged, revealing very different geometry compared the ReLU network.
particular, the model fails to generate a trimodal density, with four modes emerging instead. This instance of architectural bias can be easily rationalized. Observe indeed that from equation (12) of Result (2.2). for odd activations such as tanh, the transport process is equivariant with respect to the transformation X - -X. In other words, the generated
41

---

Figure 9: Evolution of the projected density πg generated by DAE (3) with r 2 hidden units and o = tanh activation, trained on a trimodal Gaussian mixture, with 7 = 0.5, À = 0.1, 6 = 0.0,, Pt cos(It), a 1 i, St t, 9 {0.2, 0.4, 0.6, 0.8). from random initialization. For two unit vectors é1, e2. the target density is the mixture = 1/22((3ee2 La) + 1//NN(3e1, La) 1/aN(-3e,, Ia). The generative SDE (7) was run up & 0.9, and the subspace & is spanned by é1. Different panels correspond to different training times т. Blue contours: contour levels of the theoretical prediction of Corollary 2.3 for the density p Colormap: numerical experiments in large but finite dimension d = 1000.
density år then necessarily exhibits a symmetry with respect to inversions around the origin -tttttthuu thus forbidding the existence of an odd number of modes. This provides a particularly simple yet telling example of how the choice of architecture can strongly constrain the geometry of the generated densities.
Class imbalance - The above example concerns a balanced Gaussian mixture, with all three clusters sharing equal relative probability 1/a. One may naturally wonder whether the model can also adapt to class imbalance. Fig. 9 is set for the same target density as Fig. 8 with the difference that clusters now have probabilities 1/2. 1/s, 1/s. While the generated density still presents a spurious mode (see discussion above), it correctly reproduces the clusters, and correctly gives higher mass to the most probable clusters.
Fashion MNIST - We finally provide another example on a real dataset, namely FashionMNIST
[87] This dataset corresponds 28 x 28 gray-scale pictures of clothing items; for simplicity, we only retain images of t-shirts (class 0) and dresses (class 3). We plot in Fig. 1h density produced by a model parametrized by a 2 DAE, alongside the target density. The blue curves on the other hand represent theoretical predictions for a target density corresponding to single Gaussian whose covariance given by the empirical covariance of the original dataset. Importantly, contrast to the MNIST experiment illustrated in Fig.3l in the main text, the numerical experiment were directly conducted on the original dataset, rather than Gaussian approximation thereof. Furthermore, two classes were kept, instead of single one in Fig.3 Fig. [0] shows that the theoretical prediction still reasonably captures the shape of the generated density, and notably once more exhibits a visible reduction in variance compared to the target density.
D.3
The effect of time encoding
We conclude this appendix by discussing the effect of including the time encoding Pt through its associated set of weights in the DAE model (3). For the binary target mixture described in the main text (see Fig. 2 we plot in Fig, the evolution of the summary statistics M, of Result 2. over training time, alongside that of the skip connection strength b and encoding weights v. The two plots correspond to a model with no time encoding (i.e. Pe 0) and model endowed with sinusoidal time encoding Pt = cos(Tt). As can be observed, the introduction of the time encoding has a very small effect, and the curves are left sensibly unchanged. The generated densities, illustrated Fig [2] are also strongly similar. A more quantitative viewpoint is displayed in Fig. 13 which shows how the introduction of time encoding yields a slightly lower, but overall very similar, Hellinger distance between target and generated densities at large training times. These observations temptingly suggest
42

---

Figure 10: (Evolution of the density i generated by DAE 33 with r r= 2 hidden units and tanh activation, trained on the original FashionMNIST 87 dataset, with r 0.2, c
0. a2 generative SDE 7 was run up to 0.98, and the subspace is spanned by principal components of the target density. Different panels correspond to different training times т. Blue contours:
f, Br g (2/2), namely the same parameters as Fig 3 The
contour levels the theoretical prediction of Corollary 2.3 for the density Pr. Colormap: numerical experiments the MNIST dataset. Green contours: contour levels of the target density
3.5 2.0 2,5
2.0 25
Figure 11: Evolution of the summary statistics M,, 2- and of the skip connection strength b, and time encoding weights UP as a function of the training time т, for a = tanh, r 1, a = t. 3g t, G (0.2, 0.4, 0.6, 0.8). The target density the same bimodal Gaussian mixture Fig. Solid
lines: numerical experiments in dimension d = 1000. Dashed: theoretical characterization
10
of Result 2.1 Right: no time encoding Pt 0. Left: with time encoding pt cos( The
introduction of a time encoding leaves the training dynamics sensibly unchanged.
T=0
T=0
T=2.8
Figure 12: Evolution of the projected density Ig generated by DAE 3 with
hidden
unit and σ = tanh activation, trained on a bimodal Gaussian mixture, with 1 0.2, À = 1.5, ez 0, = 1 t, Br Ł, g {0.2, 0.4, 0.6, 0.8). The generative SDE was run up to 0.9. and the subspace & is plane containing the centroid of the target density. Different panels correspond to different training times т. Blue contours: contour levels of the theoretical prediction of Corollary
2.3 for the density lgg Colormap: numerical experiments in large but finite dimension d = 1000.
Green contours: contour levels of the target density p. Top: no time encoding P: Bottom: with a time encoding p: cos(Tt).
43

---

with Une enooiing
111
no me ercoding
1.0
0.0
0.8
6.7-
0.6
0.0
05
100
155
2.
25
Figure 13: In the same setting as Fig. 12 Hellinger distance between the target and generated densities, projected in the space spanned by the centroid, function of the training time т. Red: model with a time encoding Pt = cos( Blue: without time encoding Pt = O.
Figure 14: (Evolution of the density Â generated by DAE (3) with r = 2 hidden units and tanh activation, trained on 5000 samples of a Gaussian distribution with covariance mathching that of MNIST sevens (see also Fig. 3 with r = 0.2, À .784, 6 Pt 0,9
{0, 1/1, 2/7, 3/7, 4/7, 5/7, 6/7). The generative SDE 7 was run up to & = 0.98, and the subspace spanned by principal components of the target density. (Ieft) ag 1 - t, Br = t (middle) ag cos(=t' /2), 3r sin(ne*/e /2) (right) a 1 t, 3r t and the skip connection strength is untrained and fixed at b = 0.75. Colormap: numerical experiments. Green contours: contour levels of the target density p.
that, in all probed settings, for this simple model, the inclusion of time encoding has an overall small effect on the qualitative behavior of the considered model.
D.4 Influence of the schedule and skip connection on the generated density
In the main text, we described how the limited expressivity of the considered DAE architecture could be conducive to a phenomenon akin to mode collapse. where the generated density displays largely reduced variance, when compared to the target density. This failure mode could in turn lead to model collapse, namely the rapid degradation of the successive generated densities when samples produced by the model are re-used for training. this subsection of Appendix [D] we briefly mention some parameters of the problem that qualitatively influence the shape of the generated density, and may influence the mode collapse phenomenon. Fig. [4 reprises the experiment on the Gaussian approximation of MNIST sevens of Fig. (reminded on the leftmost panel). In the middle panel, with all other parameters otherwise unchanged, the interpolation schedule is changed from the linear schedule a
t, B, to more intricate cosine schedule a, cos(x1' xt" /2), B, sin( mt' /2). This change of schedule is accompanied by a visible shift in the structure the generated density when all other parameters structure. In the rightmost panel, for the linear schedule, we illustrate an experiment where the skip connection is kept fixed at b- 0.75 and left untrained. Visibly, the generated density sees its variance increase. Qualitatively, a higher value preliminary experiments hint the role of many of the parameters of the problem in shaping the generated density. A precise and quantitative description of such effects warrants further theoretical and empirical studies, and we leave such questions for future work.
kept fixed. Notably, the latter displays less pronounced multimodal
translates into a density with larger spread. These
44


Reviews:
[{"review_id": "3NiAUSZt1R", "reviewer": "Reviewer_ecs1", "summary": "This paper analyzes a solvable flow or diffusion-based generative model, focusing on a two-layer auto-encoder trained with online stochastic gradient descent on high-dimensional data with a low-dimensional manifold structure. The authors provide a precise asymptotic characterization of low-dimensional projections of the generated sample distribution, highlighting its dependence on the number of training samples. The analysis also explains how mode collapse can occur and lead to model collapse when retraining on synthetic data.", "questions": "Does the analysis reported in this work provide any insights to general applications of diffusion and flow-based models?\nFor example, what kind of stratygies might help mitigating the problem of model collapse?", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The major strength of this work is a rigorous analysis of a solvable flow or diffusion-based generative model. The authors provide a precise asymptotic characterization of low-dimensional projections of the generated sample distribution. The results are interesting and help better understand the properties of diffuion models from a theoretical perspective. Since it is difficult to analyze the diffuion models, the results from\nthis work is interesting. \n\nThe main weakness is that it is not clear what insights the analysis can provided to practical implementations of diffucion and flow-based models.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their reading of our manuscript and pertinent questions, which we answer below.\n\n>The main weakness is that it is not clear what insights the analysis can provided to practical implementations of diffucion and flow-based models. Does the analysis reported in this work provide any insights to general applications of diffusion and flow-based models? For example, what kind of stratygies might help mitigating the problem of model collapse?\n\n Following a large majority of works in the theory of diffusion models, including for instance the related papers cited l.66, our paper primarily aims at understanding from a mathematical viewpoint how a diffusion model can learn the structure of the data distribution over training, and the effect of the model architecture and number of samples. While this field of research does not offer immediately transferable applications, we believe it constitutes a crucial endeavor to ensure a more principled usage and future development of such models. Our work advances this research endeavor, by being the first to our awareness to provide a full tight characterization of the generated density.\n\n The paper furthermore provides a number of insights that may offer some practical intuition, notably the role of the skip connection in shaping the final density, which we discuss in section 4. Following the reviewers' comments, we conducted further experiments that suggest that freezing the skip connection (rather than training it), and adopting alternative interpolation schedules (e.g. $\\alpha_t=\\cos(2t/\\pi),\\beta_t=\\sin(2t/\\pi)$), may help mitigating the collapse in the discussed tasks. We will include further discussion of these cases, along their theoretical analysis, in the revised version of the manuscript."}, {"review_id": "nG4c7e0adC", "reviewer": "Reviewer_Lq8Y", "summary": "This paper presents a fully tractable theoretical analysis of a simplified diffusion‐based generative model. Concretely, the authors study a two-layer denoising autoencoder (DAE) trained by online SGD on high-dimensional target distributions given by Gaussian mixtures supported on a low-dimensional manifold. The main contributions are:\n1. Training dynamics (Result 2.1): In the limit $d,n\\to\\infty$ with $n/d=\\Theta(1)$, the SGD evolution of a finite set of summary statistics of the weights is shown to follow a closed system of deterministic ODEs (Eqns (10), (11)).\n\n2. Generative sampling (Result 2.2): Using the trained DAE parameters, the sampling SDE is reduced to a low-dimensional nonlinear SDE in the learned subspace plus an analytic Gaussian in the orthogonal complement (Eqns (12), (13)).\n\n3. Projected density (Corollary 2.3): Combining these, the law of any fixed low-dimensional projection of the generated samples admits a fully explicit characterization (Eqn (14)).\n\n4. Insights into failure modes: The framework predicts and quantitatively explains mode collapse (loss of diversity) and model collapse (amplified collapse under retraining on synthetic data) for DAEs on realistic targets (Sections 4, Fig. 4).\n\n5. Empirical validation: Numerical simulations on synthetic Gaussian mixtures and MNIST-covariance targets confirm the tight match between theory and finite-ddd experiments (Figs. 2–3, 7–8).", "questions": "1. Can the methodology be adapted to multi-layer or even UNet architecture? What are some of the technical obstacles?\n\n2. How sensitive are the results if the target density exhibits heavy tails for non-diagonalizable covariances?", "limitations": "Unsure what practical insights we can gain from theoretical investigations like this. The authors should more clearly motivate why such an analysis is useful. Right now a characterization is given and then mode(l) collapse is given as an example demonstration of the usefulness of such characterization, but it feels lacking. More example applications/analyses should be given to justify the usefulness of such theoretical characterization. Moreover, what theoretical insights became available based on this particular characterization that were previously not available in prior studies of more unrealistic assumptions?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\nThe paper provides a rigorous comprehensive theoretical characterization of 2-layer DAE with skip connection, including both learning and generation dynamics for a finite-dimension network that is more realistic than prior works, assuming infinite dimension or trivial dimension. The theoretical results shed light on the diminishing variance over training leading to mode collapse and also model collapse iteratively training on generated data. The methodology employed here can be in principle applied to analyze more complicated architectures as well. \n\n**Weaknesses**\n1. While the theoretical results agree well with the experiments, both the architecture and the data distribution here still remain simplified for realistic cases. \n\n* As the authors pointed out themselves, the architecture analyzed here is oversimplified (2-layer DAE). While the analytical model is able to predict diminishing variance leading to mode(l) collapse, it does not provide us further insights into some of the more puzzling phenomena in large text-conditioned image-generative diffusion models, such as compositionality. \n\n* Moreover, the data distribution analyzed here is quite simple. The analysis relies on joint diagonalizability of cluster covariance and may not extend well to non-Gaussian, heavy-tailed or multimodal distributions beyond mixtures.\n\n2. The demonstrated agreement with experiment is shown for synthetic and toy examples. Even with the MNIST data, it is only tested on one of the digit classes, 7, which is also distributed based on Gaussian distribution. \n\n3. It is unclear what practical insights the theoretical characterization provides, besides the predicted mode(l) collapse, especially considering the toy and simplified setting of the derived analytical model. Here a few points might be interesting to theoretically explore further:\n\n* The authors have explained a bit about the functionality of the skip connection in preventing mode(l) collapse. A further characterization of the role of skip connection might be useful. \n\n* The authors have briefly explained that the time embedding has minimal impact on the theoretical results presented. In text-conditioned image generative models, conditional text embeddings are included in the same way as the positional embedding of time. It could be interesting to analyze/characterize the role of conditioning cues. \n\n4. How does theoretical insights gained from the analytical model inform us of practical architecture improvements or training strategies? For example, can we adjust the skip-connection schedules to mitigate mode collapse?\n\n5. In general in terms of presentation, while I appreciate that the authors try to provide intuition behind each equation and derivation, the notations become quickly out-of-hand and hard to keep track of beyond equation 14. I would suggest rather than presenting these as the main results of the paper, move them to the appendix and shifting the focus to the insights that can be gained from these derived results and how they are informative practically.", "was_revised": true, "final_justification": "While the current paper could benefit from a cleaner presentation, with fewer technical details and more insights and intuition behind the derived results in the main text, I do believe the technical contribution is solid hence recommend the paper for acceptance.", "rebuttal": "We thank the reviewer for their detailed reading of our manuscript and many constructive comments.\n\n>While the theoretical results agree well with the experiments, both the architecture and the data distribution here still remain simplified for realistic cases.\n\nWe acknowledge that these simplifications constitute an important limitation of current theoretical analyses of diffusion models. Reaching a theory capturing the full complexity of practical diffusion models is obviously an extremely ambitious and difficult goal, that will likely only be gradually reached through many successive papers by the research community, advancing the theoretical understanding step by step. We hence feel it makes sense to evaluate our paper not by how far its stands from modern practice, but rather by how it advances this line of theoretical research. In this light, our paper was the first to report a full characterization of the generated density, considerably refining many previous results.\n\n As we discuss in l.126-136, the architecture considered already prescinds a number of unrealistic assumptions (such as infinite width or sparse weights) considered in some prior works.\nWe further anticipate that the analysis can be extended to more complex architectures, including multilayer and attention models, provided all weights remain low-rank in the considered high-dimensional limit. While a careful exploration of these more realistic models would warrant separate studies, we will outline how they could be analyzed leveraging similar technical ideas in the revised manuscript.\n\nFinally, the data distribution considered is sizably more general than linear subspaces [12,14,48]  or finite Gaussian mixtures [18] often considered previously.\n\n>As the authors pointed out themselves, the architecture analyzed here is oversimplified (2-layer DAE). While the analytical model is able to predict diminishing variance leading to mode(l) collapse, it does not provide us further insights into some of the more puzzling phenomena in large text-conditioned image-generative diffusion models, such as compositionality. [...] Moreover, the data distribution analyzed here is quite simple. The analysis relies on joint diagonalizability of cluster covariance and may not extend well to non-Gaussian, heavy-tailed or multimodal distributions beyond mixtures.\n\nThe considered data distribution already constitutes an improvement compared to previous theoretical works that consider less realistic linear subspaces [12,14,48]  or finite Gaussian mixtures [18] distributions. This paper is furthermore the first theoretical work reporting analytical predictions displaying a tight quantitative agreement with simulations for realistic data distributions, such as the one illustrated in Fig. 3 (right). Prescinding data distribution assumptions necessary in theory works is a gradual process, and we also hope that the gap between theory works and real data distributions will be further bridged by future works by us or other researchers.  \n\nWe note that the joint diagonalizability of clusters is a common assumption in theoretical analyses, considered for instance in [18,24], or [a]. The considered data distribution in the present work, which encompass a manifold structure and generic (jointly-diagonalizable) covariance is more general. As we already mention in l.168, it also covers as a special case a large class of heavy-tailed distributions, as some heavy-tailed distributions can be expressed as an infinite superposition of Gaussians.  \n\nBeyond the jointly-diagonalizable case, we also expect to be able to analyze the setting where each cluster covariance has  eigenvectors in generic positions, mathematically modeled by independent Haar matrices. We will dicuss this possible extension in the revised manuscript. \n\n[a] Wu et al, Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models, 2024\n\n>The demonstrated agreement with experiment is shown for synthetic and toy examples. Even with the MNIST data, it is only tested on one of the digit classes, 7, which is also distributed based on Gaussian distribution.\n\nTo the best of our awareness, this paper is the first theoretical work reporting analytical predictions displaying a tight quantitative agreement with simulations for realistic data distributions. Note that the Gaussian approximation of MNIST 7s still yields realistic images very similar to the original distribution, see Fig. 3 (right). We will include further experiments in the revised manuscript, including on the original MNIST 7 images, mixtures of several MNIST digits, and other simple image datasets. \n\n\n>It is unclear what practical insights the theoretical characterization provides, besides the predicted mode(l) collapse, especially considering the toy and simplified setting of the derived analytical model. [...] How does theoretical insights gained from the analytical model inform us of practical architecture improvements or training strategies? For example, can we adjust the skip-connection schedules to mitigate mode collapse?\n\n\nWe acknowledge that similarly to many papers in the theory of diffusion models, e.g. those discussed around l.66, the aim of this paper is primarily mathematical in scope -- namely, deriving a tight analytical characterization of the generated density. On the other hand, the paper furthermore provides a number of insights on the role of the architectural components, notably on the role of the skip connection in shaping the final density, discussed in section 4. Following the reviewer's question, we furthermore conducted preliminary experiments that suggest that freezing the skip connection (rather than training it), and varying the interpolation schedule (e.g. $\\alpha_t=\\cos(2t/\\pi),\\beta_t=\\sin(2t/\\pi)$), may help mitigating the collapse in the examples discussed in the main text. We will include further discussion of these cases, along their theoretical analysis, in the revised version of the manuscript.\n\n\n\n>The authors have explained a bit about the functionality of the skip connection in preventing mode(l) collapse. A further characterization of the role of skip connection might be useful.\n\n\nFollowing the reviewers comment, we conducted supplementary experiments, freezing the skip connection to a fixed value and leaving it untrained, in order to more finely ascertain its role. These preliminary simulations suggest that depending on the value it is fixed at, freezing the skip connection can mitigate (for large skip connection values) or aggravate (for smaller values) the model collapse. Notably, fixing $b=0$ corresponds to a DAE architecture without skip connection, which we will add as an insightful baseline to  shed further light on the role of the skip connection.   \n\n>The authors have briefly explained that the time embedding has minimal impact on the theoretical results presented. In text-conditioned image generative models, conditional text embeddings are included in the same way as the positional embedding of time. It could be interesting to analyze/characterize the role of conditioning cues.\n\nWe fully agree with the reviewer that the study of conditional models constitutes an exciting direction. However, their study would warrant the development of new mathematical ideas and models that fall out of the scope of the present work.\n\n>In general in terms of presentation, while I appreciate that the authors try to provide intuition behind each equation and derivation, the notations become quickly out-of-hand and hard to keep track of beyond equation 14. I would suggest rather than presenting these as the main results of the paper, move them to the appendix and shifting the focus to the insights that can be gained from these derived results and how they are informative practically.\n\nWe thank the reviewer for this suggestion. In the revised version, we will move some of the most notation-heavy technical results to the appendix for better readability of the main text.\n\n>Unsure what practical insights we can gain from theoretical investigations like this. The authors should more clearly motivate why such an analysis is useful. Right now a characterization is given and then mode(l) collapse is given as an example demonstration of the usefulness of such characterization, but it feels lacking. More example applications/analyses should be given to justify the usefulness of such theoretical characterization.[...] Moreover, what theoretical insights became available based on this particular characterization that were previously not available in prior studies of more unrealistic assumptions?\n\nWhile prior studies report bounds on some probability metrics or distances (e.g. KL divergences) between the target and generated distributions, our work refines these results to a full characterization of the generated density, allowing in particular the elucidation of its structure. This allows in particular the study of mode collapse --- which cannot be fully described by the characterization of a simple metric."}, {"review_id": "dVZuKPmVTg", "reviewer": "Reviewer_pjCW", "summary": "This paper presents a solvable theoretical model for studying diffusion-based generative learning. It analyzes a two-layer denoising autoencoder trained via SGD on high-dimensional Gaussian mixtures with a low-dimensional manifold structure. The authors derive sharp asymptotic characterizations of the training dynamics and the generated density. The analysis captures both successful learning and failure modes, such as mode collapse and model collapse, offering insights into the inductive bias of DAE-based generative models.", "questions": "1. While the model studied is analytically tractable, it differs from practical diffusion architectures (See the weakness). Can the authors comment more explicitly on how the insights generalize to more realistic settings?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper provides a technically rigorous and well-executed analysis.\n2. The paper is generally well written and structured.\n3. The work advances theoretical understanding of how generative diffusion models learn from data. The treatment of architectural bias and re-training effects across generations appears novel and impactful.\n\nWeakness:\n1. While the assumptions are standard in theory papers, they may limit the direct applicability of the results to practical, large-scale models. First, in the training of foundation models, the number of samples is much larger than the data dimension. Second, the width of hidden layers is much larger than the input layer.\n2. Certain derivations, particularly in the appendix, are quite dense and may be challenging to follow without strong background knowledge. The paper would benefit from additional high-level explanations or intuitive summaries. In addition, the discussion of empirical results could be expanded slightly to aid readers less familiar with theory.", "was_revised": true, "final_justification": "1. The paper provides a technically rigorous and well-executed analysis.\n2. While the assumptions are standard in theory papers, they may limit the direct applicability of the results to practical, large-scale models.", "rebuttal": "We thank the reviewer for their reading of our work and insightful questions, which we answer below.\n\n>While the assumptions are standard in theory papers, they may limit the direct applicability of the results to practical large-scale models. First, in the training of foundation models, the number of samples is much larger than the data dimension. Second, the width of hidden layers is much larger than the input layer.\n\nWe acknowledge that these simplifications constitute an important limitation of current theoretical analyses of diffusion models. Reaching a theory capturing the full complexity of practical diffusion models is obviously an extremely ambitious and difficult goal, that will likely only be gradually reached through many successive papers by the research community, advancing the theoretical understanding step by step. We hence feel it makes sense to evaluate our paper not by how far its stands from modern practice, but rather by how it advances this line of theoretical research. In this light, our paper was the first to report a full characterization of the generated density, considerably refining many previous results.\n\nAs we discuss, the architecture considered already prescinds a number of unrealistic assumptions (such as infinite width or sparse weights) considered in a number of prior works. We strongly agree with the reviewer that pushing those analyses to cover more realistic models constitutes an important future research direction. \n\nAddressing more in detail the reviewer's concerns, we note that the large data regime can be accessed by simply taking the limit $\\tau\\to\\infty$ in our theoretical expressions. We will add a more detailed discussion of this limit in the revised version. The results can also be extended to large-width networks. Result 2.1 would need to be adapted similarly to [a]. Results 2.2 and 2.3 directly carry through to this case. We chose to focus on the small-width cases $r=O(1)$ so as to reach simpler characterizations, fully in terms of finite-dimensional quantities. We will discuss this extension in the revised version of the manuscript.\n\n\n[a] Veiga et al, Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks, 2022\n\n\n>Certain derivations, particularly in the appendix, are quite dense and may be challenging to follow without strong background knowledge. The paper would benefit from additional high-level explanations or intuitive summaries. In addition, the discussion of empirical results could be expanded slightly to aid readers less familiar with theory.\n\nWe will include more discussions to aid comprehension and build intuition, in between the technical segments, to enhance the readability of the Appendix.\n\n>While the model studied is analytically tractable, it differs from practical diffusion architectures (See the weakness). Can the authors comment more explicitly on how the insights generalize to more realistic settings?\n\nWe acknowledge the considered architecture is very simplified. On the other hand, it includes a number of the features of practical (e.g. U-Net) models, such as bottleneck layers and a skip connection. The insights on the respective role of these components, e.g. the discussion on the role of the skip connection on model collapse in section 4, will hopefully pave the way towards building insight on the role of these components in more complex models. Such a study would warrant detailed empirical work which fall out of the scope of the present manuscript -- whose main goal was to derive  a tight characterization of the generated density, advancing a line of prior theory works. \n\nWe finally anticipate that more complex architectures, including multilayer and attention models, can be analyzed using the same technical approach, provided all weights remain low-rank in the considered high-dimensional limit. While a careful exploration of these more realistic models would warrant separate studies, we will outline how they could be analyzed leveraging similar technical ideas in the revised manuscript."}, {"review_id": "TvRWwl9b6L", "reviewer": "Reviewer_WWBX", "summary": "This paper presents a theoretical analysis on the training dynamics for diffusion-based generative models. The authors examine a simple network architecture--two-layer denoising auto-encoder--trained with online stochastic gradient descent on high-dimensional data exhibiting a low-dimensional manifold structure. A key contribution is the derivation of a tight asymptotic characterization of the low-dimensional projections of samples generated by the learned model, detailing its dependence on the number of training samples. Building on this, the work discusses the emergence of mode collapse and its potential to lead to model collapse upon retraining with synthetic data.", "questions": "1. I am not convinced that the ODE of $\\mathcal{Q}_\\vartheta$, defined in Eq.10, is relied on $G_\\vartheta$. According to the definition of $\\mathcal{Q}_\\tau$ in Eq.9, it is totally determined by $w_\\tau$ and $d$. However, the definition of $G_\\tau$ depends on the matrix $E$, which is arbitrarily chosen and independent with $w_\\tau$. Could the authors further clarify the dependence of $\\mathcal{Q}_\\vartheta$ on the matrix $E$? Similar questions also exist for $Q_\\vartheta$ and $M_\\vartheta$.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "- Strengths\n1. Interesting Theoretical Results: The paper delivers insightful asymptotic results on the projection of the stochastic process. This can advance the understanding of learning dynamics in diffusion models.\n2. Strong Experimental Validation: The theoretical analysis is well-supported by experimental verification, demonstrating a strong coincidence between the theoretical predictions and empirical observations.\n3. Valuable Insights into Collapse Phenomena: The paper provides a very interesting and valuable explanation of the mechanisms underlying mode collapse and its progression to model collapse, which are crucial failure modes in generative models.\n\n- Weaknesses\n1. Limited Architectural Generality: While the theoretical insights are significant, their direct applicability is currently limited to a two-layer auto-encoder architecture. It would substantially enhance the paper's impact if these findings could be extended or shown to generalize to more complex network architectures commonly used in practice.", "was_revised": true, "final_justification": "The paper has made some progress of theoretical importance, lacking of architectural generality. After the disscussion during the rebuttal, I will not chage my rate.", "rebuttal": "We thank the reviewer for their reading of our work and their pertinent comments.\n\n> Limited Architectural Generality: While the theoretical insights are significant, their direct applicability is currently limited to a two-layer auto-encoder architecture. It would substantially enhance the paper's impact if these findings could be extended or shown to generalize to more complex network architectures commonly used in practice.\n\nWe acknowledge that these simplifications constitute an important limitation of current theoretical analyses of diffusion models. Reaching a theory capturing the full complexity of practical diffusion models is obviously an extremely ambitious and difficult goal, that will likely only be gradually reached through many successive papers by the research community, advancing the theoretical understanding step by step. We hence feel it makes sense to evaluate our paper not by how far its stands from modern practice, but rather by how it advances this line of theoretical research. In this light, our paper was the first to report a full characterization of the generated density, considerably refining many previous results.\n\nAs we discuss, the architecture considered already prescinds a number of unrealistic assumptions (such as infinite width or sparse weights) considered in a number of prior works. We agree with the reviewer that generalizing such results to more complex architectures is a crucial research direction. As a first step, we anticipate that the analysis can be extended to multilayer architectures, and architectures including self-attention layers, provided all the weights remain low-rank in the considered asymptotic limit, using the same technical approach. While a careful exploration of such architectures warrants separate analyses and are left for future work, we will outline how they can be analyzed in the revision of this manuscript.\n\nFinally, large-width networks should also be amenable to being studied building on the ideas of [a], as we discuss in our answer to Reviewer pjCW. Result 2.1 would need to be adapted similarly to [a]. Results 2.2 and 2.3 directly carry through to this case.\n\n[a] Veiga et al, Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks, 2022\n\n\n\n>I am not convinced that the ODE of $Q_\\theta$ should depend on $G_\\theta$\n\nThe reviewer is right that the ODE for $Q_\\theta$ does not depend on $G_\\theta$, for the reason they outline. We chose to write $ \\frac{d}{d\\theta} Q_\\theta =F_Q(O_\\theta)$, not making explicit which arguments $F_Q$ actually depends on, merely for notational compactness and convenience. We will clarify this point in the revised manuscript."}]

Author rebuttal:

Meta-review: This paper presents mostly theoretical contributions to the understanding of diffusion of flow-based generative models. It uses a highly simplified architecture, which, while not resembling the actual types of architectures used in practice, does remove some of the more unrealistic assumptions in other works (e.g. infinite width assumptions), and incorporates some realistic elements (e.g. skip connections). The results provide an asymptotic characterization of low-dimensional projections of the generated sample distribution.

The reviewers all entered a final score of “weak accept”, indicating that the paper has solid theoretical contributions, but is limited in scope and practicality. The major issues raised were: the architecture considered is too simplified and not realistic compared to how diffusion models are actually built in practice; the theoretical derivations lack explanation, intuition, and discussion; overall few/no practical insights are given as to how these theoretical results can affect how diffusion models are used in practice. However, the reviewers felt that the theoretical contributions were still valuable despite not having direct practical implications.

Hence, I am recommending to accept this submission, and encourage the authors to include more high level explanation and intuition to complement their dense mathematical push to make the paper more broadly accessible to the NeurIPS audience.
Decision: Accept (poster)