The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications
Authors: Binxu Wang
John Vastola
Venue: Accepted by TMLR ()

Abstract: Diffusion models have achieved remarkable results in multiple domains of generative modeling. By learning the gradient of smoothed data distributions, they can iteratively generate samples from complex distributions, e.g., of natural images. The learned score function enables their generalization capabilities, but how the learned score relates to the score of the underlying data manifold remains largely unclear. Here, we aim to elucidate this relationship by comparing the learned scores of neural-network-based models to the scores of two kinds of analytically tractable distributions: Gaussians and Gaussian mixtures. The simplicity of the Gaussian model makes it particularly attractive from a theoretical point of view, and we show that it admits a closed-form solution and predicts many qualitative aspects of sample generation dynamics. We claim that the learned neural score is dominated by its linear (Gaussian) approximation for moderate to high noise scales, and supply both theoretical and empirical arguments to support this claim. Moreover, the Gaussian approximation empirically works for a larger range of noise scales than naive theory suggests it should, and is preferentially learned by networks early in training. At smaller noise scales, we observe that learned scores are better described by a coarse-grained (Gaussian mixture) approximation of training data than by the score of the training distribution, a finding consistent with generalization. Our findings enable us to precisely predict the initial phase of trained models' sampling trajectories through their Gaussian approximations. We show that this allows one to leverage the Gaussian analytical solution to skip the first 15-30\% of sampling steps while maintaining high sample quality (with a near state-of-the-art FID score of 1.93 on CIFAR-10 unconditional generation). This forms the foundation of a novel hybrid sampling method, termed \textit{analytical teleportation}, which can seamlessly integrate with and accelerate existing samplers, including DPM-Solver-v3 and UniPC. Our findings strengthen the field's theoretical understanding of how diffusion models work and suggest ways to improve the design and training of diffusion models.

Published in Transactions on Machine Learning Research (12/2024)
The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications
Binxu Wang
Kempner Institute for the Study of Natural and Artificial Intelligence
Horvard University
Bosion, MA 02134, USA
John J. Vastola
john vastoladhen harvard. edu
Deportment of Neurobiology Horvard Medical School
Bosion, MA 02115, USA
Reviewed on OpenReview: https: https:// open:evvee......e...mm normf? id= IOukn.SHM2
Abstract
Diffusion models have achieved remarkable results in multiple domains of generative modeling. By learning the gradient of smoothed data distributions, they can iteratively generate samples from complex distributions, e.g., of natural images. The learned score function enables their generalization capabilitics, but how the lcarned score relates to the score of the underlying data manifold remains largely unclear. Here, we aim to elucidate this relationship by comparing the learned scores of neural-network-baseed models to the scores of two kinds of analytically tractable distributions: Gaussians and Gaussian mixtures. The simplicity of the Gaussian model makes particularly attractive from a thcoretical point of view, and we show that it admits closed-form solution and predicts many qualitative aspects of sample generation dynamics. We claim that the learned neural score is dominated by its linear (Gaussian) approxination for moderate to high noise scales, and supply both theoretical and empirical arguments to support this claim. Moreover, the Gaussian approximation empirically works for a larger range of noise scales than naive theory suggests it should, and preferentially learned by networks early in training. At smaller noise scales, we observe that learned scores are better described by a coarse grained (Gaussian mixture) approximation of training data than by the score of the training distribution, finding consistent with gencralization. Our findings enable us to preciscly predict the initial phase of trained models' sampling trajectories through their Gaussian approxinations. We show that this allows one to leverage the Gaussian analytical solution to skip the first 15-30% of sampling steps while maintaining high sample quality (with a near state-of-tho-art FID score 1.93 on CIFAR-10 unconditional generation). This forms the foundation of a novel hybrid sampling method, termed analytionl teleportation, which can soamlessly integrate with and accelerate existing samplers, including DPM-Solver-v3 and UniPC. Our findings strengthen the field's theoretical understanding of how diffusion models work and suggest ways to improve the design and training of diffusion models.
1 Introduction
Diffusion models (Sohl-Dickstein el al. 2015 Song & Ermon 2019 Ho et al. 2020 Song el. al. 2021 have
revolutionized the field of generative modeling by achieving remarkable performance across diverse domains, including image (Rombach et al. 2022), audio (Kong ct al. 2021 Chen al. 2021 Popov al. 2021 , and video (Harvyy et al.] [2022] Ho al. 2022 Blattmann et al. 2023 generation. Despite these successes, why diffusion models perform well as they do is poorly understood. Two major open questions follows.

---

Published in Transactions on Machine Learning Research (12/2024)
A. Gaussian structure dominates the score vector field
B. moise scale
Denoiser along Diffusion Sampling Trajectory
80.0
38.27
16.78
6.59
2:24
0.625
0.131
0.0037
distribution Data points Gaussian approximation data
Diffusion model (EDM)
Analytical teleportation
Anelytical solution to with Gauss- lan
Low noise Late diffusion
Score Neural Gouss an Datapeints
Sampling trajectory
Solution to PF-ODE delta mixture
High noise a Early diffusion
Neural
intial
Gaussaan Datapoints
Figure 1: Gaussian score well-approximates the learned neural score. A. Schematic illustrating our work's main claim. In the high noise regime, the neural score well-approximated by both the Gaussian/ling /iinear and the score of the training set; in the low noise regime, neural scores are better described by the Gaussian model. B. Visual demonstration of the effect. Denoiser outputs along PF-ODE trajectories were similar al high noise, regardless of the score model (neural score, Gaussian score, or delta mixture score); al small noise scales, neural denoiser outputs more closely resemble those of the Gaussian model.
First, given that samples are generated via dynamic process from noise, when do different sample features emerge, and what controls which features appear in the final sample? Second, given the empirical fact that diffusion models often generalize beyond their training data, what distribution do they learn instead?
Central to understanding these models is characterizing the score function, the dynamic vector field learned by a neural network during training. By modeling the gradient of the smoothed data distribution, the score function guides the iterative sample generation process. Curiously, it. has been observed deviate from its theoretically expected behavior: may not be the gradient of a mixture of data points, and it may not even be a gradient field al all (Wenliang & Moran 2023 Such deviations raise fundamental questions about the nature of what networks learn and how they generalize training data.
How does the lcarned score function compare to that of the underlying data manifold, especially given that neural networks only have access to a finite set of training examples in practice? Critically, a network that learns the exact score function the training set set-i.e. a mixture of delta functions centered on training data-can only reproduce training examples, and hence cannot genoralize. This observation suggests that neural networks optimize for balance between learning the score function of the training and capturing aspects of the underlying data manifold. However, the precise characteristics of the learned and how the development of this balance proceeds over the course of training, remain largely unexplored.
function,
One simple possibility is that diffusion models generalize in part by learning the score function associated with certain summary statistics of the traning set, like its mean and covariance matrix. If this were true, one would expect the score function to be that of Gaussian model, ie., linear in its state features. While is decidedly not true that learned neural scores are Gaussian, we find that this much closer being true than one might expect. To show this, first mathematically characterize the properties of the Gaussian model. Next, compare the Gaussian model's score function to real score functions, and find that the Gaussian model well-approxinates the behavior of neural network models for moderate to high noise levels (Fig. [ Finally, we show how this insight can be applied to accelerate sampling from diffusion models (Fig. (3).
1.1 Main contributions
Our main contributions are as follows:
We thoroughly analyze the Gaussian score model, including characterizing its sampling trajectories
by exactly solving the associated probability flow ODE (PF-ODE). (Sec. 3.1
2

---

Published in Transactions on Machine Learning Research (12/2024)
We show how the Gaussian model recapitulates nontrivial features of 'real' sample generation, including the low dimensionality of sampling trajectories, the time course of feature emergence, and the effect of perturbations during sampling. (Sec. 3.3
We theoretically (Sec. 4.1 and empirically (Sec. 4.2 4.3)
support our claim that, in the high-noise regime, the score field of real diffusion models is dominated by its Gaussian/linear approximation.
We find that learned neural scores align more closely with those of low-rank Gaussian mixtures than the (delta mixture) score of the training distribution. (Sec. 4.4
We characterize the learning dynamics of neural score functions, and in particular find that Gaus- sian/lina Maanrinearaaa structure preferentially learned early in training. (Sec.
We apply our findings by using the Gaussian model solution to accelerate sampling. (Sec. 6
2 Mathematical formulation
In this section, we review the mathematics of diffusion models and define the idcalized score models of interest. To streamline notation and match popular implementations of diffusion models, we use the "EDM" framework of Karras et al. 2022). This choice implics no loss of generality, as simple reparamctrization allows one to map to other formalisms (Song et al. 2021 Ho et al. 2020 See Appendix D.2 for more details on this point.
2.1 Basics of sasee modeling
Let (x) be data distribution in RDD The core idea of diffusson models is to corrupt this distribution with (usually Gaussian) noise, and to learn to undo this corruption; this way, one can sample from the generally complex distribution p(x) by first sampling from a Gaussian, and then iteratively removing noise. The noise-corrupted distribution is defined as
p(x;o)== = / dx' p(x|x', o) pddta(xx) = / dx' N(x;x', Pdota (x')
(1)
where a 2 0 is the noise scale. There are many ways to remove noise, but a popular method introduced by Song et al. (2021) is to utilize the PF-ODE, which in the Karras et al. (2022) formulation has the form
dx - s(x,o.) di
where & denotes time and o, denotes the noise scale at time (with òe its derivative The key ingredient of this process is the score function s(x, o) Vx (og;(x;;); i.e., the gradient of the noise-corrupted data distribution. To generate a sample, one samples xy ~ -N(x; oZ with a large or = dman, and then integrates Eq. 2 backward in time until tt.... = o min Rs 0. Since we are interested in understanding these dynamics in detail, we must carefully study the dynamic vector field o(x
There are various ways to learn a parameterized score approximator 6 6 o) YYnn al.) 2023), including
score matching (Hyvärinen) 2005) and sliced score matching (Song al. 2020b). but the current most popular
and performant approach is denoising score matching (Raphan & Simoncelli 2006 Vincent 2011 Song &
Ermon 2019.. The corresponding objective, which is minimized by the score function of the training set, is
Le Ey E0 De(y +, 2
where y denotes a sample from training data, and where the learned denoiser Dø(x, o) relates to the learned score function via
Do(x,o)-xx
(4)
a²
Sample generation dynamics are most usefully understood in terms of this denoiser, which supplies an estimate &o = DD((x, o) of the PF-ODE's endpoint given the current state and noise scale of The mathematical question we are most interested in becomes: how does &o evolve throughout sample generation?
Althoughh EDM conventionally chooses di 1, do assume this keep our results slightly more general. 2e will the 'denoiser output' D and 'endpoint, estimate" Xo interchangeahly in what follows.
3

---

Published in Transactions on Machine Learning Research (12/2024)
Gaussian sigma 0.5
Gaussian Mixture sigma 0.5 pdf
Detta Mixture sigma 0.5
pdf
Figure 2: Structure of Gaussian, Gaussian mixture, and delta mixture scores. Upper row: Examples of 2D probability density functions for Gaussian, Gaussian mixture model (GMM), and delta mixture distributions. Lower row: The respective score vector fields. For mixture models, the space is colored based on the weighting function (x) from Eq. with unique color (red, blue, or green) assigned to each Gaussian component.
Connection to alternative frameworks. In Eq. 2 the norm of x, changes significantly over time. Many alternative frameworks prefer a probability flow ODE or SDE where the norm of the state remains more stable, such as DDPM (Ho et al.] 2020), DDIM (Song et al. 2020a), and VP-SDE (Song et al. 2021). As noted by Karras et al. (2022), these formulations produce dynamics equivalent to Eq. 2 when X, is rescaled by a dime-dependent factor X + X2 axx; and time is reparametrized, allowing their solutions to be directly mapped to another. For simplicity, time-dependent scaling factors; however, with minor modifications, these results are applicable to models with stale scaling. Detailed results for alternative formulations are provided in Appendices
present most of our results (except some in Sec. 3.3 without the
D.3 and B.2
2.2 Idealized score models of interest
In this paper, we will try to understand the score functions learned by real diffusion models by comparing them to the score functions of certain idealized distributions. We consider three such idcalized score models: the Gaussian model, which assumes only the overall mean and covariance of the data are learned; the score of the delta mixture distribution, which is the Eract score the training dataset, assuming training data are precisely memorized and that no generalization occurs; and the Gaussian mixture model (GMM), which lies somewhere between the previous two models. Below, we write down some basic but important propertics of each of these idealized score models.
Gaussian model. Suppose the target distribution is Gaussian distribution with mean μ IR and covariance Σ E Px Since the effective dimensionality of data manifolds is often much lower than the dimensionality of the ambient space Turk & Pentland] 1991 Hinton & Salakhutdinov 2006; Camastra & Staiano 2016 (consider, c.g., the dimensionality of image manifolds versus the dimensionality of pixel space), we allow Σ to have rank r < D. At noise scale o, the score is that of N(,, + o²I), which reads
= =
Σ)
(5)
The optimal denoiser reads
21
Σ
D(x, 0) I Σ /1
(6)
II+
and can be interpreted as weriaiet combination of the distribution mean and the current state Early in sample generation, when o² is large, it dominates the signal variances, so the distribution mean is a
3
We slightly abuse notation write this expression more suggestiivly, since 2I E invertible and commutes with E.
4

---

Published in Transactions on Machine Learning Research (12/2024)
reasonable estimate. Later in sample generation, when o² is smaller, the current state is more informative about the outcome. Notice that different components of x contribute differently to D(x, a) depending on their corresponding signal variances; we will elaborate on this point in the next section (see Eq. 16p.
Gaussian mixture model. Suppose the larget distribution is a Gaussian mixture with K components. Denote the weight of mode by Ta and the correspouding mean and covariance by μe and E. At noise scale 0, the score function and optimal denoiser of this Gaussian mixture model (GMM) read
k
s(x, 0) E
w,
(o +.++) E1)- -x)
(7)
D(x, 0) > w6 (x, Ø)
K
O²I o²I E;
Σ
H: +
GII+ x
Note that both the score and denoiser are simply weighted sums of the score/denoiser of each mode. The weighting function wy that determines each mode's contribution is
Nx,((x; Me, o'I Ex)
wi(x 0)= = Σ3 (x; Mi, I Σg)
= softmax log m + log Nx; (4; + E,)
(8)
When the noise scale is small, the function w; (x, o) becomes one-hot, and the GMM score is locally identical
to the Gaussian score of the highlighted component (Fig. 2 Thus, the Gaussian mixture score can be interpreted as piecing together different Gaussian scores.
Delta mixture (exact) score model. Consider dataset {y: of N data points. The corresponding delta mixture distribution and its noise-corrupted version can be written
1 N p(x) = N
N
Σ y:)
p(x;0) = N
N(x o22)
(9)
The corresponding score function and optimal denoiser
N
N
s(x,0)
w (y: -x)
D(x, Σ
w. (x(xx yo
(10)
σ2
1=1
exp {
1xx 112)
w.(x, 0)
softmax
x (x112)
(11)
Σ K exp 1 |x 15
20
This model's optimal denoiser is a weighted combination of training examples. The noise scale a acts like a temperature parameter: in the large a limit, the score points towards the data mean; in the - limit, the score pushes with infinitely strong 'force' towards the training example closest to the current state Thus, generating samples using this score always (in the absence of numerical errors) reproduces training examples.
This score model is special, since it is the exact score of the finite training dataset (without augmentation), i.e., the score function minimizing the score matching objective (Eq. 3) Since we expect a sufficiently expressive score approximator to converge to this model in the absence of additional influences (e.g., regularization and early stopping), comparing learned scores to this model speaks to the question of generalization. If a score approximator learns something other than this, then the model has been implicitly regularized and generalizes beyond the training set to some extent.
3 Exact solution and interpretation of the Gaussian score model
In this section, we present the exact solution to the Gaussian model. The utility of this model is that it is simple enough that it can be solved exactly, and hence one can precisely quantify various aspects of sample generation dynamics. First, we briefly describe the solution method (Sce. 3.1 and then we discuss the qualitative insights we can obtain from (Sec. 3.3 In Sec. 5 we will show how this admittedly simple model relates to real diffusion models.
5

---

Published in Transactions on Machine Learning Research (12/2024)
3.1 Solution method: Exploiting decomposition of the covariance matrix
Since the covariance Σ is symmetric and positive semidefinite, it has compact singular value decomposition Σ = UAUT, where U [ui...., up] is D x semi-orthogonal matrix and A E R is diagonal with À := A&k > 0 for all k. The columns of U, Uk, are the principal component (PC) axes along which the Gaussian mode varies, and their span comprises the 'data manifold' of the Gaussian model.
This decomposition is useful since allows us write the score and denoiser of the Gaussian model in a more explicit form. Using the Woodbury identity (Woodbury 1950
(o (O'I+UAN -- 0² 0o L.((A'1 A- 22 U
1 --
1
1 I I-U diag
A
UT]
o²
d + o²
For convenience, define the diagonal matrix As = diagl du Aeto2
We can now write
s(x, 0) 0² (I U-- (n-
D(x,o)= UA U*-
(12)
Slightly more explicitly, the optimal denoiser can be written
A
D(x,0) u
lu (x 14]
(13)
k=1
3.2 Closed-form solution of PF-ODE for Gaussian model
Using the rewrilten score (Eq. 12 the PF-ODE (Eq.
2
takes a particularly simple form:
x [I - U²,
(14)
The above ODE is linear, and its dynamics along each principal axis un are independent. Solving it in the usual way (see Appendix D.l.
we find
o? (tt = cs (T) = u (XT 14)
X; =u +
Xi +
Σ (tc))))
us
(15)
OT
o7 +
x = -I- - UU
(XT-W) UXT))x---)
The solution has three components: (i) the distribution mean, (ii) off-manifold component, and (iii) an on-manifold component. The distribution mean term does not change throughout sample generation. The off-manifold component shrinks to zero as t - 0. The on-manifold component, which is determined by the manifold-projectedd difference between x and μ, evolves independently according to vt A) along each PC direction.
This solution allows characterize the evolution of the denoiser output given the initial state xx:
Σ E(t, (T) u
A
D(t) = o(x +
A)t,A) =
(16)
of (
Finally, the exact solution also allows us to explicitly write the mapping from x7 to the final state Xo:
d u (xT n1)
Xo
(17)
of + A
For the Gaussian model, the final sample determined by the combined influence of the covariance structure of data, and the overlap between the vector Xx z and the data manifold.

---

Published in Transactions on Machine Learning Research (12/2024)
solatio Analytical solation of Gausian diffusion tefeetin along of
A
Signal& soltessheeltee POW
(t. A)
2(t.
de 2),
E.
Ampitration perturbationa
9.
0.
coefigter
94
1030
VerwiD
LGD
67
-0
10
1.0
0.G
0.6
022
1.0
12.5
2.4
11:
1.0 0120 01.
0.2
1.0
0..1
11.
0.2
1100
1.0
0.3
0.5
0.4
0.0
Figure 3: Analytical solution to sample generation dynamics for Gaussian model. A. The noise and signal schedule of and of from ddpm-CIFAR-10. B. vt(t, X) governs the dynamics x₁ along each principal axis ug. C. E(t, A) governs the dynamics of the endpoint estimate {x(x)) along each PC, normalized by the standard deviation VX. D. Time derivative of E(L, V2, highlighting the 'critical period' when each feature develops the most rapidly, E. VA/(o) + )az),, which quantifies the amplification effect of a perturbation along PC un at time t' (Eq. 23
Solution with data scaling term. The popular VP-SDE (Song et al. 2021 is an alternative to the EDM formulation, and its forward process is characterized by the transition probability p(x,|xo) = -N(
Using the VP-SDE is equivalent to introducing a time-dependent scaling term a, in Eq. 2 Thus, we can obtain the solution by substituting x, + xt/ar and oz + 01/a,, The solution reads
0 + da
X; 0zfe +
XT +
(t, (T)UR
A(t, 1 o + da
(18)
OT
k=1
x7 =
U U)(x aTT4)
(T( = uf (xT - aTH)
(19)
D(t) + Σ (T)
A(L, = + + (
a+1
(20)
Rm
When 1, these solutions reduce to Eq.
15 and 16 expected.
3.3 Interpreting Gaussian generation dynamics
The closed-form solution of the Gaussian model pro- vides the exact relationship between the covariance matrix and sample generation dynamics. Since the dynamics are separable along each PC, we only need to study the functions and to describe how the variance of the data distribution along each PC in- fluences dynamics along that direction. Here, we highlight interesting consequences of the Gaussian model's exact solution related to four aspects of sample generation: 1) the determination of the final sample, 2) the geometry of trajectories, 3) the fcature emergence order, and 4) the effect of perturbations.
Orthogonal space; to data manifold
a. Actual trajetoryy x; travels towards the projected outcome
Data distribution Projected outcome & (x,)
trajetrra X
Out of manifoold directions shrin independently
0. Projected outcome (K.)
starts from center
Then migrates along high varia nce direction.
Then along lower varlance directions
Finally convenge the projected outcome (x,)
In this subsection, we describe these consequences in the context of the VP-SDE formulation, which includes the additional a, dala-scaling factor. The signal scale a; assumed to decrease with time, the noise scale assumed to increase with time, and we also assume of + of
Data manifold (subspace variance)
Figure Geometry of sample generation trajec- tories for the VP-SDE.
all times. We visualize ar, or along with the functions 5t. A), E(t, A) to provide intuition about. the Gaussian solution (Fig. 3)
7

---

Published in Transactions on Machine Learning Research (12/2024)
Mapping from initial noise Xy to final sample X,. The explicit mapping between the initial noise pattern and final sample, which is given by
Xo 4
(0, As) uu (xT aTT)
(21)
k=1
is reminiscent of linear filtering. The final location of xo along each axis uk is determined by the projection of the (mean subtracted) initial noise onto that axis, amplified by the standard deviation (0 A&) R Vdg- Thus, the subtle alignment between the initial noise pattern and the covariance of the data distribution determine what is generated. Note that even if the imitial overlap is dominated by particular features, due to the amplification effect of (0, Ax) the final sample may be dominated by other features that vary more.
Trajectory resembles 2D rotation.
The solution for x, also implies that
X R 0,X0 + V1 at XT - R=
Au - - -ar cs Ca(T)ux
i.e, that x; dynamics tend look like rotation or a spherical interpolation within the 2D plane formed by Xg and X7, up to on-manifold correction terms (sce Appendix D.4 for the derivation and more discussion). The correction terms tend to be small; assuming < D, and that the typical overlap between the initial noise and any given eigendirection is roughly 1/VD,
V2 2
2 r
X axxo at xT
1 1
(22)
D
2
Feature emergence order. The solution of the denoiser trajectory D(t) (Eq. 20 implies that the outputs of the denoiser remain the data manifold throughout sample generation. This explains why the outputs of well-trained models resemble noise-free images, rather than images contaminated with noise.
Initially, as all É(T,,) A) se O, the expected outcome D(T') starts close to the distribution mean For a face dataset, this might resemble the so-called 'generic' face (Langlois & Roggman 1990). The denoiser output moves along each PC direction according to the function (Fig. 3C). The sigmoidal shape of É(l, X) indicates that the feature associated with a given PC changes the most when or & a,vA, ie., when the noise variance matches the scaled signal variance. After that point, the feature stabilizes, suggesting a criticall period" of development for each feature (Fig, (D). Moreover, since the critical period happens when R 1 + X, features appear in order of descending variance. The highest-variance features appear earliest, and as generation proceeds more and more lower-variance features are unveiled.
This has interesting implications for image diffusion models. Natural images have more variance in low frequencies than high frequencies (Ruderman) 1994). Furthermore, for face images, 'semantic' features like gender, head orientation, skin color, and backgrounds have higher variance than subtle features such glasses, facial, and hair textures (Wang & Ponce 2021). Hence, facts about natural image statisties and sample generation dynamics together explain why features like scene layout specified first in the endpoint estimate-or or cquivalently why generation is usually, outline-frgg first, details later.
Effect of perturbations. Finally, we examined the effect of perturbations on feature commitment. Suppose at time i' E (O, the off-manifold directions are perturbed by 8x- and the on-manifold direction u, is perturbed by an amount dcs- The effect of the perturbation on the generated sample Xo
(0, ('tt,, Ab)
F oe + Noa
Axo
Sc Up
Σ
(23)
R=
Thanks to denoising, an off manifold perturbation has no effect on the sample, while on on-manifold perturbations have vorying effects depending liming. We visualized the amplification factor / oztt A of perturbation function of perturbation time t' and PC variance (Fig. E)-
8

---

Published in Transactions on Machine Learning Research (12/2024)
Perturbations along high variance axes (A 1) have amplified effect when injected at the start and less amplification when injected later. Conversely, perturbations along low variance axes (A < 1) have a diminished effect when injected early, and a less suppressed effect when injected later. This is consistent with the notion that features have a variance-dependent 'critical period' during which they are most easily affected by perturbations.
Even if random perturbation is injected, depending on the timing t', features with different variances will be most amplified. This time-dependent 'filtering' effect explains the classic finding (Ho et al. 2020) that early perturbations create variations of layout and semantic features, and late perturbations modulate high-frequency details.
Summary. The solution of the Gaussian model (Eq. 18] suggests geometric picture of sample generation, which we depict in Fig. the endpoint estimate Xy begins at the center of the distribution and travels along the data manifold, moving first along the high variance axes and then along the lower variance axes; concurrently, the state X₃ in the ambient space rotates towards the evolving endpoint estimate, Despite the Gaussian model's simplicity, various aspects of its behavior are qualitatively consistent with substantially more complex models like Stable Diffusion (Fig.
portrait aristocrat", PNDM,
trajectory projection
[% cosine- 3.542 angle=55.22 de)
Latent State
IRR noGg4 trajectory
66k)
groject with otmo
Difference c(obx.: x:)
Projected outcome
((,,04))
step
step's
step 10
step 15
step 20
step 25
Step 35
Step
projsctio
Figure Qualitative aspects of Stable Diffusion sample generation consistent with Gaussian theory. A. Trajectorics of states G(x²) (top row). scaled differences between nearby states G(k(x;-1 - x))) (middle row), and denoiser / projected outcome G())(x:)) (bottom row). G denotes Stable Diffusion's decoder, which converts latent states to images. B. Trajectory of x; projected onto the plane spanned by xf and No. Trajectories are effectively two-dimensional, and resemble a rotation from Xy to Xg. Note that both feature emergence order and the low dimensionality of trajectories are consistent with the Gaussian model.
4 The far-field Gaussian structure of real diffusion models
Most distributions interesting enough to train generative models on are not Gaussian, so how is the exact solution of the Gaussian model related to real diffusion models trained on complex datasets? Our central claim is that, at moderate to high noise scales, the score function of an arbitrary bounded point cloud is ncarly indistinguishable from that of Gaussian with matching mcan and covariance. We call such structure "far-field"; we borrow the term from physics, where it describes region far enough from the source of (e.g., electromagnetic) waves that the size and shape of the source can be neglected.
First, we provide thcoretical arguments to support this claim (Sec. 4.1 Second, we present two empirical analyses that support it. In Sec. 4.2 and 4.3 we study how well the Gaussian model approximates the score function, sampling trajectories, and samples generated by pre-trained diffusion models. In Sec. [.4 we go beyond the Gaussian model and systematically compare the learned neural score with the scores of GMMs with varying numbers of components and covariance matrix ranks. In the following section (Sec. 5p. we track the neural score field throughout training and examine when different. types of score structure emerge.

---

Published in Transactions on Machine Learning Research (12/2024)
4.1 Theoretical basis for far-field Gaussian score structure
Consider a point cloud [y:) C RRP When the noise level is high and/or the query point is far from the point cloud's support, we claim that the score of this distribution is nearly indistinguishable from that of a Gaussian with the same mean and covariance. Here, we provide a thcoretical argument for this claim.
Let l and Σ denote the mean and covariance of the point cloud. Recall from Sec. 2.2 that the score function of the (noise-coruuped) point cloud is
s(x: 0) Σ Wi (x)x)
(y: -
1 -=
Σ W/ (x(
(yr 11)
σ²
0²
(24)
wx(x) = exp
2
x y:
v:lla)
exp
20 | y.ll + +t - w)) - -1))
j=1 cxp
2x²
1 x y51
Σ j- exp
20² ||
d2 (x m)) - p))
Note that we have rearranged the score to be written in terms of distances to the point cloud mean ps. In the far-field regime, one expects this distance to account for most of the distance to each data point.
Next, note that (x - (x) ( (u-y;) is on the order of ovtrr and || (u---;l y: is on the order of trΣ (see Appendix D.8 for detailed derivation). Hence, when o >> vtr iie., when the noise scale is substantially larger than the radius of the point cloud along each direction the cross term will dominate. When this term dominates, to lcading order we have
1+ 10) (y: 11)
1+ (x 4) 1
IrE 0²
w/ (x)
+0
(25)
Σn 1 1) -+]
N
Substituting this result back into the point cloud score (Eq. 24 we find that
--x 1 11 s(x,0) = o² N2
Σ 1+
02 σ2 (y. - 10) (x u) (y, 1)
4-x
E(E x)
(26)
o² 04
Finally, we observe that the score function of the Gaussian model (see Sec.
2.2 is identical to leading order:
o² (I+ +1+ (4-xx x) ~
1 1- 0² σ
tr 04
(μ xx-x) x)
(27)
This proves the claim. As a parenthetical comment, we note that our expansion strategy is similar in spirit to the multipole expansion used in (for example) electrodynamics (Jackson 2012]; is approximate vector field by matching certain low-order statistics.
in both cases, the key idea
Is the Gaussian approximation really only accurate when > VtrE? In the next subscction, we empirically show that it works quile well even for somewhat lower noise scales and in particular when the noise scale is smaller than the top eigenvalue of the covariance matrix (Fig, 6 The surprisingly broad range of noise scales for which the Gaussian approximation works well is what led us to note its "unreasomable effectiveness!.
4.2 Learned score vectors are empirically well-approximated by the Gaussian model
Motivated in part by the theoretical argument from the previous subsection, empirically validated the claim that the score function of pre-trained diffusion models is well-described by the Gaussian model in the far-fied//high-noiss regime. To do this, we examined the score functions of three pre-trained models (of CIFAR-10, FFHQ64, and AFHQv2-64) from Karras et al. (022)). For each dataset, we computed the scores of several tractable approximations of the data:
Isotropic Gaussian score. (Iso) This score simplifies the Gaussian model by removing covariance
information, and has s(x, == (f4 x)
Gaussian score. (see Eq. The Gaussian model with the mean and covariance of the dataset.
10

---

Published in Transactions on Machine Learning Research (12/2024)
Exact point cloud score. (Eract delta, see Eq. 10 The score of the training dataset.
Per-class Gaussian mixture scorc. (GMM; see Eq. 7 For datasets with class labels, (e.g. CIFAR-10), computed the score of a GMM with one Gaussian mode corresponding each class, equipped with the mean and covariance of that class. See Appendix A.2 for more details.
For various noise scales o, we evaluated the neural and analytical scores at random points sampled from o2), and quantified their difference using the fraction of unezplained variance, defined via
fraction of unexplained variance |SEDMI (x, Sanaly (x, x)
(28)
SSDAS (x, 0
We characterized the average deviation as function of noise scale for each type of idealized score (Fig.
6
Approximation error between neural score and analytical score FFHQ
CIFAR10
AFHQV2
sqrt top eigenvalue sqrt mean eigenvalue
sqit top eigenvalue
sqit top siganvalue
mean eigenvalue
eiganvalue
Fraction of Squared Error
10a
Goussian Score
Fraction of Squared Error
10a
Gaussian
Fraction Squared Emro
10
Gaucdia Score
Isotropic Score
Isotropic Score
Isotropic Score
Score
Exact Point Cloud Score
10 class GMM Score
10-2
10
10-1
10-1-
10
10-4
10-4
10:
0
20
40
60
80
20
40
60
80
20
40
60
80
noise scale (time)
noise scale (time)
noise scale (time)
Approximation error between neural score and analytical score (zoom in)
CIFAR10
FFHQ
10a
AFHQV2
10a
sqrt mean eigenvalue
eigenvelue
10P
sqit top eigenvalue eigenvvllue
10L
-..:: top eigenvalue
mean eigenvalus
Caussian Score
Gaussian Score
Gaussian Score
Faction of Squared Entor
109
Isotropic Score
iraczon or Squareo erar
Isotropic Score
Fraction of Squared Enar
Isotropic
Finec Point Couud Score
10a
Score
109
Ekatt Point Cloud Score
10-1
10-1
10-2
10
3
10.5
10-4
10
2
10
10
10
noise scale a (time)
noise scale (time)
noise scale (time)
Figure 6: Gaussian score dominates the learned neural score at high noise scales. Fraction of squared error as a function of noise scale between neural score function and various analytical score approximations. Note the log scale on the y-axis. Upper panel, all noise levels. Lower panel, zoom in low-noise regime. For reference, vertical lines show the square roots of the 10 largest covariance eigenvalues.
Gaussian score predicts the learned score at high noise. For all three datascts, at most noisc levels (a > 1.08), the Gaussian score explains almost all variance (> 99%) of the neural score. As expected, it explains more variance than the isotropic model, which does not incorporate covariance information. Further, for CIFAR-10, at all levels, the 10-class Gaussian mixture score predicts the neural score slightly better than the Gaussian score, which shows that adding more modes indeed helps capture the details of the score field. Surprisingly, most. noise levels (a > 1.08) the 10-mode model improved the fraction of explained variance
11

---

Published in Transactions on Machine Learning Research (12/2024)
by less than 2.5 10-3, showing that even single Gaussian may be sufficient for explaining the learned neural score. To our surprise and against the intuition suggested by our earlier theoretical argument (see the red dashed line in Fig. 60. the Gaussian model well-approximates the neural score even when a < Vmmax.
Neural score deviates from 'exact delta' score at low noise. Although the exact point cloud score slightly outperforms the Gaussian model in the high-noise regime, it deviates substantially from the learned score in the low-noise regime (Fig. Gp. and in fact performs worse than the Gaussian and Gaussian mixture models for all three dalasets. This suggests that, especially in the low noise regime, the models learned something substantially different from the 'exact' score, and more similar to the Gaussian or Gaussian mixture scores. As previously argued, deviations from the 'exact' score model can be viewed as a signature of generalization (Kadkhodaie et al., 2023 Yi et al. 2023 In Sec. 4.4 we study the effect of adding more modes to Gaussian mixture approximation more comprehensively.
Deviation between the state trajectory x_t of EDM and analytical score
CIFAR10
0.25
FFHQ
AFHQV2
0.25
0.20
Gaussian-EDr Exact- EDM
0.20
Gaussian- Exact-EDM
Gaussian-EDM Exact-EDM
GMM-EDM
0.20
0.15
MSC
MSE
MSE
0.10
0.10
0.10
0.05
0.05
0.05
0.00
0.00
0.00
40 50 60 70
80
0
10 20
30
40
80
40 50 60 70 80 noise scale a (time)
noise scale d (time)
noise scale o (time)
Figure 7: Gaussian model solution predicts the early sampling trajectory of diffusion models. Deviation between the x, trajectory of the EDM neural score and the analytical score is plotted over time. Many initial conditions were used; the thick line denotes the ensemble average, and the shaded area denotes the 25%-75% quantile range.
4.3 Gaussian model empirically captures early sample generation dynamics
If the Gaussian model produces score vectors similar to those learned by neural networks, then their sampling trajectories may (at least initially) be similar too. On the other hand, it is also possible that initially small differences may accumulate over the course of sample generation, and hence that real sampling trajectories differ substantially from those predicted by the Gaussian mode To test this, we compared the solutions of PF-ODE based on the idcalized score models to the sampling trajectories of neural diffusion models with deterministic samplers. We used the exact solution for the Gaussian model (Eq. 15), Heun's 2nd-order method to integrate the neural scores, and off- sheftf RK4 integrator to integrate the mixture scores.
Gaussian model predicts carly diffusion trajectories. We found that the carly phase of reverse diffusion is weel-predicted by the Gaussian analytical solution (blue trace in Fig. 7 16 whose trajectory remained close to those of the trained diffusion models (MSE < 0.01) before diverging. For CIFAR-10, the point of divergence was around 9 sampling steps (a R 1.92), and for FFHQ and AFHQ it was around 17 steps (a R 4.37). This roughly corresponds to the scale where the Gaussian approximation of the score field breaks down, and the score needs to be approximated by that of a more complicated distribution (Fig.
The exact score model predicts early sample generation dynamics slightly better than the Gaussian model (green trace in Fig. 7 16 but diverges earlier (a 15) from the neural trajectory, and by much larger amount than the Gaussian model, consistent with our earlier score function observations (Fig. 6
"Theory suggests there is limit to how much score errors can produce sample generation errors. In particular, Girsanov's theorem implies that such errors cannot compound exponentially (?)-
12

---

Published in Transactions on Machine Learning Research (12/2024)
A.
EDM
Gauss Theory
C.
Distance between analytical and EDM samples
CIFAR10 (rescaled 224 pix)
EDM traj
0.124
MSE
LPIPS_vg9
0.8-
0.10
Gauss Theory solution
0.08
0.06-
B.
CIFAR10
FFHQ-64
AFHQV2-64
0.4
0.04-
EDM
EDM
0.3
Gauss
0.02
Gauss
0.2
GMM
0.00
Exact
Exact
Coussian
GHM
Dolia
Gaussian
GKM
Delta
Figure Gaussian model predicts carly denoiser trajectorics and low-frequency features of samples. A. The denoiser output D(x,,)) I) along a sampling trajectory of the EDM model and the Gaussian solution with the same initial condition xy. B. Samples generated by the EDM model, Gaussian solution, and the 'exact' delta mixture scores from the same initial condition. C. Image samples from the Gaussian and GMM models are closer to actual diffusion samples than samples from the delta score model.
Gaussian model predicts low-frequency aspects of diffusion samples. We visualized the samples generated by the neural and the idealized score models (Fig. g A,B), and found that the Gaussian model's samples closely replicated several characteristics of those produced by trained models, including their global color palette, background, the spatial layout of objects, and face shading, among other features.
This obscrvation is consistent with carlier theoretical results given well-known facts about natural image statistics. These characteristics represent relatively bow-spatial-frequeneyy information, which contains far more variance than results (Fig. BC), and empirically shown by Ho et al. (2020), high-variance image features are determined in the early phase of sampling, during which the neural trajectory is accurately described by the Gaussian model. Consequently, the Gaussian model can predict the 'layout' of the final image; given that it does not fully describe learned neural scores, especially in the low-noise regime, it is unsurprising that it is less successful at predicting high-frequency details like edges and textures.
information (Ruderman 1994). As predicted by our Gaussian model
Gaussian model predicts diffusion samples more accurately than exact delta score. Interestingly, the samples generated by the exact delta score visually deviate even more from the EDM samples than those generated by the Gaussian model (Fig. g B). We quantified this observation using the pixelwise mean squared error (MSE) and Perceptual Similarity (I.PIPS) metries, with further details provided in Appendix A.3 Across all metrics, samples from pre-trained diffusion models more closely resembled boussian-generater samples than samples produced using the exact delta score (Fig. g C). This discrepancy was significant, albeit less pronounced, when using the perceptual similarity metric, except in the case of the AlexNel-based LPIPS distance on the AFHQ dataset, where the difference was not significant (Fig. 19
These findings indicate, perhaps contrary to expectation, that the Gaussian model in many ways provides better quanlilative approximation of the behavior of real diffussio models than the score of the training set. This is true in various senses, and appears to be true whether one uses pixel space or perceptual metries.
4.4 Beyond Gaussian: Low-rank Gaussian mixture scores as a model of learned neural scores
Our previous analyses demonstrated that learned score functions closcly match the Gaussian model at high noise levels, but not at low noise levels. To study the behavior of the learned score ion at lower noise or, equivalently- closer to the data manifold, we need to go beyond the single Gaussian. As a next step, we studicd the score of a Gaussian mixture model whose covariance was allowed to be low-rank. We asked 1) to what extent does adding Gaussian components help explain the learned neural score? and 2) what is the minimum rank required for good score approximation?
13

---

Published in Transactions on Machine Learning Research (12/2024)
Score Residual of EDM vs GMM score CIFARIO uncond EDM pretrained
10a.
sigma-0.00
0.00
sigma-0.05
sigma-0.1
sigma-0.5
sigma-1.0
Score EV Residual
60-2
10"
10-4
10
10
30
4x
10-4
10
Rank Mode
2x
10a
10a
10a
10a
10a
10
10a
10
10a
10
10a
10
10a
10-
10a
16
Score EV Resiqual
sigma=1.5
sigma-2.0
sigma=5.0
6x-00-
sigma=10.0
sigma=20.0
64
32
10-
4 10-
x0
96
2x 10
10
128
10
10
10
20
10
256
384
512
10
10
768
10a
10a
10a
109
10a
10a
10a
10a
10
10
10a
10"
10a
10
10a
1024
Num of Modes
Num of Modee
Num of Modes
Num Modes
Num of Modes
2048
3072
Score Residual of EDM vs GMM score CIFAR10 uncond EDM pretrained
10a
sigma-0.0 0.01
sigma-0.05
10a
sigma-0.1
sigma-0.5
sigma-1.0
Score EV Resicual
6
10
10
60
10-2
10
30-1
10
10
xx10
10:1
2 2x-0
20
10
10'
16
10'
10'
10
10
10'
10*
10
10'
107
10
10'
10'
sigma=1.5
sigma=2.0
sigma=5.0
sigma=10.0
sigma=20.0
Num of Modes
Score EV Residual
60-1
10-2-
40
x0
3x
10-1
10
10
10
20
50
10
10-4
10-
100
10a
100
10a
10a
10a
10a
10a
10a
10a
10a
10a
10a
10
10a
100
200
Bank of Made
Rank of Mode
Rank of Mode
Rank of Made
Rank of Mode
Figure Deviation of learned neural score from Gaussian mixture model with varying com- ponents and ranks. Upper: Residual explained variance (EV) plotted a function of the number of Gaussian modes on the x-axis, with each colored line representing different rank. Each panel compares the score at a certain noise scale a. Lower: An alternative view of the same data plot, with the rank of the covariance matrix on the x-axis. See Fig. 23 for analogous MNIST results.
We systematically compared the structure of neural scores to the scores of GMMs fil to the same training data (Fig. 9h. We varied both the number of modes and the ranks of the associated covariance matrices; for the dotails of the fitting procedure, see Appendix A..5 In brief, we performed k-means clustering on the training set and utilized the empirical mean and covariance of each cluster to define Gaussian components. The fraction of samples within each cluster was used to define the weights of components. To obtain low-rank covariance matrices, we computed the eigendeconposition of the empirical covariances and retained the top r directions.
Neural score is best explained by Gaussian mixture with moderate number of modes. First, note that both the Gaussian model and 'exact" score models are special cases of the GMM; the former has K = 1 and the latter has K N. This leads us to expect that making K larger than 1 helps approximate the learned neural score, but only up to point.
Indeed, we found that at most noise scales (o > 0.05), increasing the number of Gaussian modes reduces the deviation between the neural score and the analytical score. Further, measured by the increase explained variance, the benefit of adding Gaussian modes increases as we lower the noise level: it rises from 10-5 10-2 2 (Fig. 10) But this trend does not hold indefinitely. At smaller noise scales, augmenting the Gaussian modes beyond certain number- for instance, 200-500 for MNIST and 100 for CIFAR-10 in000-nnnsssssss the gap
14

---

Published in Transactions on Machine Learning Research (12/2024)
between the neural and analytical scores, as illustrated by the U-shaped curves in the upper panel of Fig. This divergence highlights that the neural network learned a course-grained opprosimation of the score field.
9
A. Contribution of a of Mode beyond Gaussian B. Minimal Rank (increase residual 1.1) C. CIFAR10 uncond EDM
Score Residual of EDM vs GMM 1e-01)
CIFAR10 uncond EDM
CIFAR1O uncond EDM pretrained sigma-0011
mode
mode
11113 53 563 293 234 13.7 70 4T 27 23 38 15
10 2.
100
102
200
100
fraction Exp Var
Minimal Rank
Num of Madex
12 KM
11.
66 60
10
2333770 534 214 115 411
10a-
23
110
040
202 242
25
10-4
101
2
10-1
100
10
102
2
10-1
100
10'
10a
2
2003
sigma
sigma
Rank Mode
Figure 10: Impact of number of modes and covariance rank on neural score approximation. A. Contribution of multiple mixture components beyond one Gaussian (full rank). Improvement of the fraction of Exp. Var. is plotted on the y-axis; the vertical line means certain mixture models perform even worse than the single Gaussian. B. Minimal rank required for each noise level. Each line shows the minimum rank for different numbers of GMM components. Minimal rank is the smallest rank value that increases the residual Exp. Var. within 10% beyond the full rank model. C. Tradeoff between Gaussian rank and mode number. Residual Exp. Var. plotted as a function of mode mumber and rank, with the red dashed line denoting models with same explained variance.
Required covariance rank increases with lower noise level. We found that, in the high-noise regime, a low-rank Gaussian is sufficient to predict the score: for example, when a = 10, the explained variance is identical for Gaussian mixtures with full rank and rank 100 (Fig. lower panel; note that the curve has an 'elbow'). As the noise scale decreases, the minimum required rank gradually increases: the elbow of the explained variance curve moves rightward (Fig. 10 B). This phenomenon can be understood through the Gaussian score equation (Eq. 12). In the diagonal matrix Å,, entries for which d < 0² are rendered negligible, as X 0, i.e., those dimensions become effectively 'invisible' at that noise scale. As the noise scale is reduced, more principal dimensions of the covariance matrix are unveiled and become 'visible' to the score function, so the covariance matrix's rank effectively increases.
As
Trade-off between mode number and local rank. Interestingly, at small noise scales, one can trade off between the number of components and the rank of each component. For the CIFAR-10 dataset, at a = 0.1, a GMM with 200 components with rank 384 is as effective at explaining the neural score as a GMM with 10 components with rank 1024 (Fig. [OP, see also horizontal lines in Fig. 9. This suggests an interesting geometric picture of the score. Though globally the data distribution and its score appear to be high rank, locally the data distribution and the score are effectively low rank; this picture is reminiscent of a nonlinear manifold comprised of many glued-together linear manifolds.
Visual comparison of Gaussian mixture score versus neural score in low noise regime. The deviation between Gaussian mixture scores and neural scores in the low noise regime raises the question about the spatial nature of this difference. To explore this, we visualized various neural and idcalized score fields close to the image manifold. We selected three data samples to establish a 2D plane, and then evaluated and projected each score vector field onto this plane. Various qualitative comparisons of this sort (involving both inter- and intra-category planes, and mix of training and test examples) are illustrated in Figures 11
20 and
At a moderate noise level (e.g., o = 2.0), the vector field created by the delta point cloud or a large mumber of Gaussian mixtures could roughly rocapitulate the score field structure of the neural score (Fig. [1 upper panel). Intriguingly, at lower noise levels (a ~ 0.11, the neural score revealed a complex geometric structure, with the score vector field nearly vanishing inside the lines and triangles (simplexes) interpolating the data
15

---

Published in Transactions on Machine Learning Research (12/2024)
score vector field among samples with same class TEST set, [3658 4066 8719] labels [2 2]
sigma 1agma=2.00 gmm mode
edm NN
gmm delta
gmm mode
gmm
mode
gmm mode
gmm_5. 5_mode
gmm
gaussian
isotropic
76
score vector field among samples with same class TEST set, [3658 4066 8719) labels [2 2]
sigma=0. gmm mode
gmm
gmm.
gmm mode
gmm gmm
gaussian
isotropic
10
Figure 11: Visual comparison of the score vector field so(x, of pre-trained diffusion models and the Gaussian mixture model. All panels visualize score functions on the same 2D domain, namely the plane spanned by three test set examples not seen during training. Red dots mark the 2D coordinates of these three samples. The x-y axes correspond an orthonormal coordinate system on the plane, with the units denoting L2 distance Arrows visualize the projection of score vectors onto this plane, and the heatmap visualizes the L2 norm of the projected vector on the plane. The first panel shows the neural score vector after training: the other panels show idealized score models of decreasing complexity (number of Gaussian modes). See Fig. 20 and 22 for additional similar visualizations.
the plane. In this case, all three examples of the MNIST digit '2.
21
points, depicted in the lower panel of Fig. 11 This contrasts with the Gaussian mixture and delta mixture scores, which aligned with theoretical expectations of piece-wise linear vector fields demarcated by linear or quadratic hypersurfaces.
This phenomenon reminiscent. continuous attractors the dynamical systems literature (Samsonovich McNaughton 1997 Khona Ficte 2022), and suggests that the probability flow ODE could converge along these simplices where the vector field vanishes, which may be mechanism that supports generalization.
16

---

Published in Transactions on Machine Learning Research (12/2024)
Residual explained variance of EDM score by Gaussian and GMM scores (CIFAR10 origin EDM 5k epochs early phase)
10
sigma=0.00 0.01
10
sigma-0.05
10a
sigma-0,1
10"
digma-O.5
10
sigma-1.0
score residual
x 10
6 6
4x10-1 10-1
10
3x 10
10-1
ax 10.2
20
3x10-1
10
10
2000
4000
2000
4000
0
2000
4000
2000
4000
2000
4000
epproximation
100
sigma=1.5
10a
sigma=2.0
10a.
sigma=5.0
sigma=10.0
10a
sigma=20.0
meun, botropic gausiian
Score resicualEV
10-1
10
gmin2 m_mdde
10"1
1
gmm 5
gmm. 1C mode
10
gmm, __oddee
10-2
10
10
10-1
gmm_50, 50_mode
gmmn 100, mode
200 300dmode
2000 4000
2000
4000
2000
4000
2000
4000
2000
4000
gemm delta
apoch
spoch
opoch
spach
spach
Figure 12: Learning dynamics of score neural network s0(x, 0) with idcalized scores (Gaussian, Gaussian mixture, and Exact) reference. Each panel shows different noise scale a and plots residual explained variance (1 - EV) as function of training epoch. Each line shows the residual EV of the neural score with respect to one idcalized score model. Consistent with Fig. G at higher noise scales, all score approximators deviate negligibly from the neural score and from each other. At lower noise scales (a S 1.5), the deviation between the neural score and the Gaussian score first decreases, and then increases slightly. This indicates that the neural score approaches the score of the Gaussian model before it learns additional structure. Learning dynamics for other datasets and training settings are shown in Fig. 25 and 26
5 Learning of far-field Gaussian score structure
In the preceding sections (Sec. 4.2
our analyses focused on the structure of the score of trained diffusion models, but did not touch on the question of how that structure emerges during training. Given that overparameterized function approximators like neural networks are thought to learn 'simpler' structure first, stands to reason that Gaussian maassian/linearr /linear score structure may be learned relatively early. Is this true?
4.3
4.4
To test this idea, trained neural network score approximators different datascts using the training procedure described by Karras et al. (2022). Throughout training, we sampled query points X from the noised distribution, and compared the neural score to the three idealized score models from Sec. 2.2
As we have observed, at higher noise scales (a 2 10), all score approximators are similar to the neural score after convergence. During training, the neural score function steadily converges to these approximators as well (Fig. 12 Fig. p5p. Intriguingly, at lower noise scales, the network displays non-monotonic learning dynamics for simpler scores. Specifically, the neural score initially aligns with simpler score models (the Gaussian model and GMM with few modes) before starting to deviate from them. This pattern was consistently observed across various training sets (refer to Fig. 25 for CIFAR-10, AFHQ, FFHQ). On the other hand, the neural score approached more complex score approximators (the delta mixture) monotonically. This effect suggests an initial tendency to fit the score of simpler distributions.
This tendency manifested slightly differently on the MNIST dataset (Fig. 26 approaches different GMM score approximators in order of increasing complexity. Namely, the deviation between the neural score and simpler Gaussian scores decreased and reached a floor, before moving on to more complex models, such as 2-mode and 5-mode Gaussian mixtures. Although the non-monotonic effect was less pronounced, it supports the idea that the network initially maximizes the explainable variance for simpler distributions before progressing to more complex Gaussian mixtures.
Here, the score net
We visually demonstrate this phenomenon by plotting the evolution of the neural score field during training on the MNIST dataset (Fig. [3). At lower noise scale (a 1.0), the neural score field starts by resembling a Gaussian-like (single basin) vector field, then splits into multiple basins, so that it resembles the score of a multi-modal distribution.
17

---

Published in Transactions on Machine Learning Research (12/2024)
score vector field among samples with same class in TEST set, [3658 4066 8719] labels [2 2]
sigma= samma-1.00.0 edm NN ep2500
edm NN ep500
edm NN ep1000
edm NN ep5000
edm NN ep24999
15
gaussian
gmm, 10 mode
gmm, 100 mode
gmm_
55005500000 gmm_500
mode
gmm delta
15
20
15
10
15
10
15
Figure 13: Visual comparison of neural score vector field (x o) with idcalized scores throughout training. Layout and score domain are the same as in Fig. 11 The upper panels depict the neural score vector field at different training epochs; the lower row panels depict the scores of idealized models of increasing complexity. See Fig. 27 28 29
for similar plots on the plane spanned by training examples.
Our finding is consistent with the general observation that the learning dynamics of overparameterizee neural networks exhibit a 'spectral bias' (Bordelon et al.) 2020 Canatar et al. 2021), and tend to capture low-frequency aspects of input-output mappings before high-frequency ones (Rahaman et al. 2019 Xu el al. 2022). In this setting, the relevant mapping the one defined by the score s(x, σ) or denoiser D σ). For lower noise scales o, the Gaussian score is smoother and ewer-frrquency in x space than the score of GMMs, so is perhaps unsurprising that Gaussian structure learned preferentially by the network early in training.
6 Application: Accelerating sampling via teleportation
Above, we have shown that the Gaussian model's exact solution provides a surprisingly good approximation to the early sampling trajectory. We can exploit this fact to accelerate diffusion by 'analytical teleportation' (Fig. 14 A). By this, we mean replacing a certain number initial PF-ODE integration steps with a single evaluation of the Gaussian analytical solution some intermediate time (or equivalently noise scale kkksp). nontrivially reduce the number of neural function ovaluations (NFEs) required to generate
In this way; sample. In principle, this speedup can be combined with any deterministic stochastic sampler.
Experiment 1. First, showcase its effectivencss using the optimized socond-order Heun sampler from Karras et al. 2022), which yields near state-of-the-art image quality and efficiency. See Alg. and Appendix
A.7 B.2 for method details and additional experiments with DDIM (Song et al. 2020a.
We evaluated our proposed hybrid sampler on unconditional diffusion models trained on CIFAR-10, FFHQ-64, and AFHQv2-64. We sampled 50,000 images using both the Heun sampler and our hybrid sampler with various numbers of skipped steps (or cquivalently, different times t'), and evaluated the Frechet Inception Distance (FID) in each case. We found that we can consistently save 15-30% of NFES at the cost of less than 3% increase in the FID score (Fig. 14 B,C), even when competing with the optimized sampling method from EDM (Karras et al. 2022).
For CIFAR-10 and AFHQ, skipping sleps can even reduce the FID score by 1%, resulting in a highly competitive FID score of 1.93 for CIFAR-10 10 unconditional generation. (For the full set of results, see Sec.
B.8 Fig. 30 and Tab. ( Comparing Fig. 30 Fig. 16 we observe that the number of skippable steps roughly corresponds the noise scale which the neural score deviates substantially from its Gaussian approximation.
18

---

Published in Transactions on Machine Learning Research (12/2024)
Algorithm 1: Hybrid sampling using Heun's method and Gaussian model prediction
Require: Data: mean and covariance of dataset μ, Σ and its eigendecomposition Σ UAU
Require: Original Houn's 's sampler parameters: mins mazi
Require: Parameter: Skip time/noist scale = Sskip Input XxT
Compute Xy as the Gaussian solution at t' using p and Σ::
Gript thx
Xy Ookiy
UUT) 11) Ex
maatth the tns (xT - pt);
V
Begin numerical integration using Houn's method:
Use initial time skip and initial state xes Integrate until mmi to obtain the final sample Xn;
Return X
A.
Analytical teleporation
B. Heun 40 steps skip
Application: Accelerate sampling via analytical teleporation
Hybrid Sampler
Gaussian analytical solution
skip K steps with Gaussian + Heun
C.16.
FFHQ64 EDM-hybrid FID
skip 4
skip 8
skip 12 skip 16 skip 20
14
Deterministic sampler diffusion (e.g. Heun's)
10.
X init noise
8
6.
4-
2
40
50
60
70
80
skip to time point
Neural Function Eval # (NFE)
20
10 skipping steps
15
0
2
60 skip to noise level
20 30
Figure 14: Leveraging closed-form solution to accelerate sample generation. A. Schematic description of our 'analytical teleportation' method. B. Sampled image as funetion of number of skipped steps; our hybrid method combines Heun's method with Gaussian model prediction. C. Image quality (FID score) of the hybrid method a function of NFE and number of skipped steps (see Appendix A.8
Table 1: Teleportation improves sampling speed while maintaining FID scores across datasets. Noise in the table refers the noise scale that analytical teleportation skips to, i.c., Oskip in Alg. See
Fig. 30 and Tab. for more complete set of results.
FFHQ-64
AFHQ-64
CIFAR-10
NFE,
Noise
FIDJ
NFEJ
Noise
FID4
NFEL
Noise
FIDL
EDM baseline
79
80.0
2.464
79
80.0
2.043
35
80.0
1.958
67
32.7
2.561
59
16.8
2.026
25
12.9
1.934
Teleportation
55
11.7
2.841
51
8.0
2.359
21
5.3
2.123
To further demonstrate the generality of our method, we evaluated our hybrid sampler with several popular deterministic samplers: jon__oover++, saon_solver_vv heun, uni uni_pc_bhi, uni_pc_ bh2 Lu el al. 2022); Zheng et al. (2023); : Zhao et al. (024)) on the same pre-trained EDM models. We systematically varied the skip noise scale aakip and the number of sampling steps nette-
Experiment 2.
We found results consistent with the first experiment across all samplers (Fig. 15 Gaussian teleportation
improve sample generation time (i.e,, reduce the number of required neural function evaluations) without.
19

---

Published in Transactions on Machine Learning Research (12/2024)
A.
B.
FID Heatmap by Steps and Skip Value Datasat: athqu2-66 2-64 Solver: DPM-salvers
Dataset: dffrr0 Solver: 3PN-solver-V3
Dataset: mhq-64 Solver: DPM-solvers
00:15
30
10:07
54
15.43
18.23
17.13
15.05
14.00 15:45
12.14 22.35 12.02
11.07
11.07
20
18.57
13:04
14.05
13:59
15.31
13.27
15:15
13.02
25.05
13:01
Spma Stpp Value
Soma SUp VAlue
11:35
33:39
8:37
4.70
2:16
2.41
7.34
700 300 50
540
163
10.75
8:00
4.35
2.34
2:30
2.37
2.07
2.05
12.96
5.32
3
3.6
2.99
2.34
2.30
2:23
2.20
2.15
2117
16
28
18:44
6.6
5:40
432
5:78
5.31
5:15
5:05
3.00
2.51
20
7.32
4.53
375
7.57
7.16 2.04
1.48
1.40
17.57
10.24
5.56
4.59
3:0
7:40
1.77 2:70 2.67 2.05
420
2.33
240
227
225
2.00
2.07
430
14:12
9.04
4.31
311
2.50 2.25 2.02 1.35
1.0
1.57
2.57
00.0 430
28.17
13.08
5.50
4.21
353
300
2.75
2.43
2.3
2.54
2.52
17:47
8:35
3.22
2.85
2.16
2.08
2.07
17.00
10.0
4.15
5122
2.11
2.41
2:10
240
2.03
2.02
2.03
22.31
13.11
3.56
403
3
340
1010 2:51
235
2.48
6
10
12
1.5
20
25
30
35
40
10
12
35
20
23
30
15
40
Samping Steps
Samplngg Steps
Sampling Steps
Figure 15: Image quality as a function of skip noise scale and sampling steps number with DPM-Solver. for A. CIFARIO. B. AFHQv2. C. FFHQ dataset. FID scores lower than or within 1% increase of the original sampler (without teleportation) are colored in magenta. For full evaluations, see Fig.
3 33
reducing sample quality. Further, given a fixed budget of sampling steps, using Gaussian teleportation can improve sample quality (i.e., reduce FID score).
Our method compares favorably to the dpm_ solver_ _ approach to speeding up sampling proposed by Zheng
et al. (2023), which exploits model-specific statistics. They achieved FIDs of 12.21 (5 NFE) and 2.51 (10 NFE) on unconditional 10. Using our teleportation technique, we achieved FIDs of 9.63 (5 NFE, Sskip = 20.0), 2.45 (10 NFE, skip = 40.0), and 2.22 (12 NFF, akiip = 20.0). Similar results hold for the AFHQv2 dataset (see full results in Fig. 31 Our intuition is that substituting the easy-to-approximate (i.e., mostly linear) part of the score field with its Gaussian approximation allows the sampler to spend more steps on the taadee-to-approxiiate, nonlinear score field close to the data manifold. If this intuition is correct, it suggests that by spending the neural function evaluation budget more wiscly, we can improve sample quality.
For the FFHQQ64 dataset, we observed similar results to Experiment 1: analytical teleportation slightly increased FID given a fixed number of sampling steps (sec full results in Fig. 33 For example using con__sover++, 40 NFE with no skipping yields an FID of 2.46, while skipping to noise scale sakip = 20.0 yields FIDs of 2.65 (40 NFF) and 2.77 (25 NFE).. Though these changes FID are tiny, they show that for the face dataset, the neural score model may deviate from the Gaussian approximation in an interesting way in the high noise regime.
7 Related work
Diffusion models and Gaussian mixtures. There has been increasing intorest in characterizing diffiusion models associated with tractable target distributions, and specifically in Gaussian and Gaussian mixture models. Shah et al.] (2023) and Gatmiry et al.) (2024) focused on learning to generate samples from Gaussian mixtures using diffusion models, and SSahh et al. (2023) found a connection between gradient-based score matching and the expectation-mexxiizate
(EM) algorithm and derived convergence guarantees.
Concurrently, Pierret & Galerne (202)) derived the exact solution to the reverse SDE and PF-ODE for a Gaussian model, which allowed them to compare Wasserstein errors for any sampling scheme. But to our knowledge, no existing work has provided the in-depth comparisons between idcalized score models and neural scores that we present here.
Consistency of noise-to-image mapping. Recently, many rescarchers have noticed that the mapping from initial noise to images is highly consistent. across independentll-traiing across models trained on non-overlapping splits of the same dataset (Zhang et al. 2023 Kadkhodaie et al.) 2023). This phenomenon can be partially explained by two of our findings: (1) the noise-to-image mapping is largely determined by the Gaussian structure of the data (Eq. [6]. and (2) Gaussian structure seems to be preferentially learned by neural networks (Sec. 5 If different splits of the dataset have almost identical
diffusion models, and even
20

---

Published in Transactions on Machine Learning Research (12/2024)
Gaussian approximations, one expects different networks to learn similar linear score structure, and hence possibly similar noise-toiimage mappings.
Non-isotropic Gaussian initial states. Though most diffusion models sample initial states from an isotropic Gaussian, Lee et al. (021] explored sampling from non-isotropic Gaussian whose mean and covariance depend on the training set. Although our work does not involve any modification to the score matching objective, we also find that one can save computation by choosing a different initial stale, i.e., the state X (a, af + If predicted by the Gaussian model. In a sense, during the intial phase of sample generation, diffusion models convert their isotropic/ /white initial states to preconditioned non-istropic states.
Score field smoothness and generalization. Several recent papers have also observed that trained diffusion models learn score fields that are smoother than the scores of their training distributions, which in principle is helpful for generalization (Kadkhodaie et al. 2023 Scarvelis et al.) 2023). These findings are consistent with our observation that diffusion models learn smooth score functions that more closely resemble the scores of Gaussian mixture models with a moderate number of modes.
8 Discussion
In summnrry even for real diffusion models trained on natural images, in the high-noise regime the neural score is well-apppoximated by Gaussian model; in the low-noise regime, Gaussian mixture models approximate neural scores better than the score of the training distribution. We mathematically characterized the sampling trajectories the Gaussian model, and found that it recapitulates various aspects of real sampling trajectories. Finally, we leveraged these insights accelerate the initial phase of sampling from diffusion models. Below, we mention additional implications of our results for the training and design of diffusion models:
Noise schedule. As the early time evolution of the sample distribution well- rell-predicted by the Gaussian model, we in principle do not need to sample high noise levels to train denoiser D(x, or. Instead, sampling can directly begin (or 'warm start') from the non-isotropic Gaussian distribution NN oj + o(I).
Model design. Since it is empirically true that neural scores dominated by Gaussian Gaussian/linear structure at high noise levels, we can directly build this structure into the neural network to assist learning, For example, we can add a linear by-pass pathway based on the covariance of training distribution, and let the neural network only learn the nonlinear residual not accounted for by the Gaussian model term.
Training distribution. As the neural networks first need to learn the data covariance structure by score matching, we may be able to assist learning by reshaping the training distribution. One hypothesis is that if we pre-condition the larget distribution by whitening its spectrum, then the neural score may converge faster. If true, this may explain the higher efficiency of latent diffusion models (Rombach et al. (022):: with KT. regularization, the autoencoder not only compresses the state space, but also "whitens" the image distribution by morphing it to be closer to a Gaussian distribution.
9 Limitations and future work
Higher-resolution and conditional diffusion models. Our experiments focused lower-resolution image generative models. Generalizing results to higher-resolution models, or popular text-to-image conditional diffusion models (Rombach et al. 2022). involves overcoming difficulties related to covariance estimation. For high-dimensional models, estimating covariances is substantially harder given a limited number of training examples; for conditional models, especially for text-to-image models, there may not even exist training data corresponding to a specific prompt, making direct covariance estimation impossible.
Structure of neural scores close to image manifold. We focused on characterizing the linear structure that dominates neural scores in the high noise regime, but did not claim precisely understand neural scores at smaller noise scales, i.e. when the sample is closer to the data manifold. Even though the scores
21

---

Published in Transactions on Machine Learning Research (12/2024)
of Gaussian and Gaussian mixture models with moderate number of components better explain neural scores than the delta mixture model, both are far from perfect (Fig. [1], and the precise structure of the neural score may be better described by the score of some nonlinear manifold. We leave the elucidation of such structure, which we expect to be closely related to the generalization capabilities of diffusion models, to future work.
References
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution vidco synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition (CVPR), pp. 22563 22575, June 2023.
Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In Hal Daumé 111 and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1024 1034. PMLR, 18 Jul 2020. URL https: sttts::/pprceeedingg http://pppaccooo........eeese/es 11tpa////1111//ooooooooo02022a html
mlr wtwplppreees//vvveee
Francesco Camastra and Antonino Staiano. Intrinsic dimension estimation: Advances and open problems. Information Sciences, 328:26-41, 2016. ISSN 0020-0255. doi: URL
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature Communications, 12(1): 2914, May 2021. ISSN 2041-1723, doi: 0053//44467-0212200 URL https://doi. ort.oi...rr/1....1 1038/ 841467-021-231033
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2021. URL http::/pppeeeoeeeeeeeeee//rrtaaa
Khashayar Gatmiry, Jonathan Kelner, and Holden Lee. Learning mixtures of gaussians using diffusion models. arXiv preprint arXiv:2404. 18869, 2024.
David J Griffiths. Introduction to electrodynamics, 2005.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Dietrich Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL httpss:/opperrvvii ntt/foruenfffft Je
G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, (5786))) 507, 2006. doi: 10.1126/ 10.1126/science. http://ccccee.266411 URL https://www www.::/wwienc..e... org/doi/abs/10 1126/science 1127647]
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and Cur-
A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. ran Associates, Inc., 2022. URL http:::/prooceedngss.n www.ieuuiipp.. ctt.ppsccccpppp....p http://gs..sacaa..../ppppppee//p ftt.ee//ppapee/22///1eeee htt.ee/eeeeecaaacaaeacaeeaaaeeee
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching, J. Mach. Learn. Res., 2:695-709, dec 2005. ISSN 1532-4435.
John David Jackson. Classical electradynamics. John Wiley & Sons, 2012.
22

---

Published in Transactions on Machine Learning Research (12/2024)
Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from goomelry-adaplive harmonic representation. arXiv preprinl or Xiv:23100 0.0257,, 2023.
Tero Karras, Mika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basse generative models. arXiv preprint ar Xiv:22200. 000644, 2022.
Mikail Khona and lla R Fiete. Attractor and integrator networks in the brain. Nature Reviews Neuroscience, 23(12):744-766, 2022.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021. URL https
//openreview. http://wwwww.eeetteeeeeeeeeeeeee -osddd=aaxxK882255]
Judith H. Langlois and Lori A. Roggman. Attractive faces are only average. Psychological Science, 1(2): URL
115-121, 1990. doi: (..111/j.1467-99200.000
org/ 1wt../1111.11111111. 9280
tb90.9600079.x
Sang-gil Lec, Heesoung Kim, Chachun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie Yan Liu. Priorgrad: Improving conditional denoising dilfusion models with dala-dependent adaptive prior. arXiv preprint arXiv:2106.06406 2021.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXi::220..09778, 2022.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint xxxi:2211.01095 2022.
Emile Pierret and Bruno Galerne. Diffusion models for gaussian distributions: Exact solutions and wasserstein errors. Xr preprint arXiv:2405.14250 2024.
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-tts: A diffussio probabilistic model for text-to-speech. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8599 8608. PMLR, 18 24 Jul 2021. URL https::/prooceeding mlr.
wtt.:/sss....p..se/eeeeeeeeeeeee ppress/vis. ww33///ppv2221..m html
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pp. 5301 5310. PMLR, 2019.
Martin Raphan and EcrO Simoncelli. Learning to be bayesian without supervision. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, vol- ume 19. MIT Press, 2006. URL https::// htttp:://proccedingg. http://pwwwdddngggere.e.o.oo///e pdf
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pottern Recognition, pp. 10684-106000 10695, 2022.
Daniel I. Ruderman. The statistics of natural images. Network: compulation in neural systems, 5(4):517, 1994.
Alexei Samsonovich and Bruce McNaughton. Path integration and cognitive mapping in a continuous attractor neural network model. Journal of Neuroscience, 17(15):5900 5920, 1997.
Christopher Scarvelis, Haitz Sáez de Ocáriz Borde, and Justin Solomon. Closcd-form diffusion models. arXiv preprinl Xiv:22200. 12395, 2023.
Kulin Shah, Sitan Chen, and Adam Klivans. Learning mixtures of gaussians using the ddpm objective. A Advances in Newral Information Processing Systems, 36:19636-19644 2023.
23

---

Published in Transactions on Machine Learning Research (12/2024)
Jascha
Eric Weiss, Niru Maheswaranathan, and Surya. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256-2262222 2265, Lille, France, 07-09 Jul 2015. PMLR. URL ittps://proceednngs.nt mlr. htt.piiinnnnsi..rrrsss/sseeeeeee ssbhldiicksteinl...t
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusson implicit models. arv preprial arXiv:2010.0220 0.02002,222 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In
H. Wallach, H. Larochelle, A. Beygelzimer, F. d' A'Aché-Buu, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volune 32. Curran Associates, Inc., 2019. URL https: .
7/
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: scalable approach to density and score estimation. In Ryan P. Adams and Vibhav Gogate (eds.), Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pp. 574 584. PMLR, 22-25 Jul 2020b. URL https://proceeeding: mlr ppress/v15//ooree
html
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kunar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021. URL
Matthew Turk and Alex Pentland. Figenfaces for Recognition. Journal of Cognitive Neuroscience, 3(1):71 86, 01 1991. ISSN 0898-929X. doi: 10. 11116//jon.1991...1777 URL httpss//dbl..org//001//////// 19w.:o............ 1.71
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23 (7):1661 1674, 2011. doi: 10.1162/NECO 00142.
Binxu Wang and Carlos R Ponce. A geometric analysis of deep generative image models and its applications. In International Conference on Learning Representations, 2021. URL htttps://openrevies http::///pppeeeeeeeen///rrrrr Id=CHTQQR2DDXXG
Li Kevin Wenliang and Ben Moran. Score-based generative models learn manifold-like structures with constrained mixing. arXiv preprint arXiv:231 1. 09952, 2023.
Max A Woodbury. Inverting modified matrices. Department of Statistics, Princcton University, 1950.
Zhi- Zhi-Qin John Xu, Yaoyu Zhang, and Tao Luo. Overview frequency principle/spectral bias in deep learning. arXiv preprint 2011:2201.07395, 2022.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Hiung Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Comput. Surv., 56(4), nov 2023. ISSN 0360-0300. doi: 10.1145/3626235. URL https: http:////oooooo/../oo/o 1145/3626235
Mingyang Yi, Jiacheng Sun, and Zhenguo Li. On the generalization of diffusion model. ar Xio preprinl arXin:2305.14711 2023.
Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Liyue Shen, and Qing Qu. The emergence of reproducibility and consistency diffusion models. arXio preprint xrX1:22310.0 05264, 2023.
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipc: A unified predictor-correctorr correctoo framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-solver-v3: Improved diffusion ode solver with empirical model statistics. Advances in Neural Information Processing Systems, 36:55502-55542, 2023.
24

---

Published in Transactions on Machine Learning Research (12/2024)
A Detailed Methods
A.1 Image datasets and Pre-trained Models
Table 2: Specifications of the image datasets and diffusion models
Dataset name
Num Samples
Resolution
Pro-trained Model Spec
CIFAR10
50000
32
edm- httt:-ifffal--223---mmoooot
FFHQ64
70000
64
odm-fffh--64x64-mmeeee
AFHQv2-64
15803
A.2 Idealized Score Approximations
In the paper, we compared several analytical approximations of the score. We listed their formula below. Isotropic score, only depends on the mcan of data, isotropically pointing towards p.
x σ²
(29)
This is equivalent to approximating the whole training dataset with its mean.
Gaussian score, for Gaussian distribution of N(x; Σ), its score is
Vx 1x E)
(30)
-0²1 E) x)
(31)
1
(1--AA (( x)
- UA
(32)
0²
p-x 1
U x) 0² σ²
(33)
d
A diagl
d +o2
(34)
it depends on the mean and covariance of the data. We can added a non-isotropic correction term to the isotropic score.
Gaussian mixture score. For general Gaussian mixture with h component q(x) = Σt T,N(x;; E.), it's score function at noise scale is the following.
Vx log (Σ TxN((i o'I+E.))
Σ -0021 E.) - (x mi) TxN 2 E,) Σj Tx: M,j O2I E,))
(35)
Σ - -x Mi) (x));(x,0)
Where the covariance matrix for each Gaussian component can be cigen-decomposed and inverted efficiently. It's weighted average of the score of each Gaussian mode, with softmax-like weighting function w;(x,
25

---

Published in Transactions on Machine Learning Research (12/2024)
Exact point cloud score. For set of data point {yr), the score of x at noise scale is
xx log
N ((x
(36)
x + Σ Wi wi
(37)
exp (- 2/ l|y: - x(13)
X +
20 lly; - -x122) y
(38)
Σj exp
(x
exp
20
1. yy: - x x1
softmax
2o Ily: x1(
(39)
Ej exp
yy xx
2
A.3 Measuring Image Similarity
LPIPS am image distance metric trained to mimic human perceptual judgment of image similarity. We used LPIPS models with all three pretrained backbones: AlexNet, VGG, SqueezeNet. Note that, the images we were measuring has different resolutions, pixels for CIFAR10 and 64 pixels for FFHQ and AFHQ. Consistent with the FID measurement, for all datasets, we resized the image to 224-pixel resolution before sending them to the LPIPS model. We found that, without resizing, the mismatch of image size (e.g. 32 pixel for CIFAR) to the convnet can bias the image distance by the boundary artifact, and obscure the effect.
A.4 Sampling Diffusion trajectory with Gaussian Mixture scores
For the Gaussian score, we are able compute the whole sampling trajectory analytically. But for a Gaussian mixture with more than one mode, we need to use numerical integration. We evaluated the numerical score with the Gaussian mixture model defined as above and integrated Eq 2 with off-the-shelf Runge-Kutta 4 integrator (solve_ivg _ivp from scipy) from sigma 80.0 to 0.0. We also chose (t)) t. To compare the trajectory with the one sampled with the Heun method, we evaluated the trajectory at the same discrete time steps as the Karras et. al. (2022) paper. The ith noise level is the following,
0 = G1/0 +
o (llla)))
(40)
Thstep
we chose the same hyper paramcter Mmin mmaz: and Nstep as the original EDM paper.
A.5
Fitting Gaussian Mixture Model and Low rank GMM
Given large and high-dimensional image datasets, used fast and heuristic method for fitting the Gaussian Mixture model.
We performed mini-batch k-means clustering on the training data (sklearn. cluster Wwn.MMiiiiaian BatncikethMMM)) with batch size 2048 and fixed random seed 0 to cluster the dataset to cluster. For each cluster i, the number of samples belonging this class is N₁. We compute the empirical mean }a and covariance matrix E, of all samples belonging this cluster. Then we define the Gaussian mixture model
N No N(x E.)
Σ
(41)
N
This fitting procedure is roughly equivalent to one step of Expectation Maximization (EM) iteration, which is not optimal, but suffice our purpose.
For the low-rank Gaussian mixture models used in Sec. 1.4 we performed PCA of the covariance matrices and kept the top PC to define the low-rank covariance.
26

---

Published in Transactions on Machine Learning Research (12/2024)
A.6 Training diffusion models
We used the training configuration F from Table 2 in Karras el al. 2022 We note that non-leaky augmenlation was used in this training configuration.
Spocifically, we used the train.py py and train Seaii__emm_pp py function from the code base NVlabs/edm and https:///itthu: htt.://uwb..cmamuuaaaiiiiiiaa/aa -hae www/miniiiiiii/ii mini edm/
For mini-EDM training runs, hyperparameters are
MNIST, used batch size 128, Adam optimizer with a learning rate of 2E-4, we densely sampled checkpoints for the first 25000 steps. The channel multiplier is set "1 2 4", with the base channel count. 16, no attention, and there is layer per block.
CIFAR10, used batch size 128, Adam optimizer with a learning rate 2E-4, checkpoints for the first 50000 steps. The channel multiplier is set to 2 2", with the base channel count. 96, attention resolution is 16, and there are layers per block.
For EDM training runs, hyperparameters are
CIFARIO, used batch size 256, Adam optimizer with learning rate of 0.001, checkpoints for the first 5000K images (around 20000 steps). EDMPrecond class network was used, the channel multiplier was set to "2 2", with the base channel count 128. The augmentation probability was 0.12.
AFHQ and FFHQ, used batch size 256, Adam optimizer with learning rate of 2E-4, checkpoints for the first 5000K images (around 20000 steps). EDMPrecond class network was used, the channel multiplier was set to "1 2 2 2", with the base channel count 128. augmentation probability was 0.12.
Dataset
MNIST
CIFAR
CIFAR
AFHQ
FFHQ
Batch
128
128
256
256
256
Optim.
Adam
Adam
Adam
Adam
Adam
LR
2E-1
2E-1
0.001
2E-4
2E-1
Steps
25000
50000
20000
20000
20000
Chan Mult.
1 1 4
1
2
1 1222 2
1 2 2
Base Chan.
16
96
128
128
128
Attn Res.
None
16
16
16
16
Lyr per B
2
4
4
4
Prob
None SongUNet
None
0.12
0.12
0.12
Net. Class
SongUNet mini-EDM
SongUNet EDM
SongUNet EDM
SongUNet
Code Base
mini-EDM
EDM
Table Hyperparameters for diffusion model (mini-EDM and EDM) training runs Abbreviations:
Batch Batch Size, Optim. Optimizer, LR Learning Rate, Steps Steps/Images, Chan Mult.
Channel Multiplier, Base Chan Base Channel Count, Attn Res Attention Resolution, Lyr per B Layers per Block, Augm Prob Augmentation Probability, Net. Class Network Class
A.7 Hybrid sampling method
As we stated in the paper, the Hybrid sampling scheme can be combined with any deterministic sampler (e.g. DDIM (Snn et al.] (2020a), PNDM Liu et al. (2022) In our two main benchmark experiments, we combined it with the Heun sampler, and a few recent samplers in DPMSolver-v3 benchmark.
Experiment 1 In the first experiment, we used the following strategy for choosing dekips inspired by the bascline Heun method. The Heun method samples a sequence of natep noise levels [oo- oi 02,00, ...0........) where dg Omar and nnnep mmin- For each initial condition xy: we use the analytical solution to evaluate Xy or integrate the probability flow ODE to time I', where we chose l' the i-th noise level a; (Eq. 40). Then
27

---

Published in Transactions on Machine Learning Research (12/2024)
will skip the first step in the Heun method and start at initial state Xy. In this manner, when skip to the ith noise level, we will save 2 neural function evaluations.
Experiment 2 In the second experiment, we systematically varied both the skipping noise level skip and the number of sampling steps Pstep for a bunch of recent deterministic diffusion samplers: dpm_solvees solver+t, camn_sover___ v3, heun, uni mni_pp_bh1, uni_pc_ bh2. We used the implementation these samplers in the code base of DPMSolve-v's v3, used in their benchmark experiments https://githh:: tree/main/codeiaa codea//s//cdabbeeeeeeeed pre-trained on CIFAR10, AFHQ, FFHQ datasets.
wtwptthmmm-----eeeeeee We used the same models as in Experiment. 1, namely the EDM models
For each initial condition Xy, we use the analytical solution to evaluate X at = Oskip, and then run these diffusion samplers with setting maa = skip and Nstep steps. We systematically vary nstep and skip a grid.
A.8 FID score computation and baseline
We used the same code for FID score computation as in Karras et al. (2022). For each sampler, we sampled the same initial noise state 2r with random seeds 0-49999. We computed the FID score based on 50,000 samples.
For our bascline, we picked the same model configurations as reported in Tab. 2, specifically, Variance preserving (VP) config F, the unconditional model for CIFAR10, FFHQ 64 and AFHQv2 64. The default. sampling steps are 18 steps (NFE- 35) for CIEAR1O, 79 steps (NFE 79) for FFHQ and AFHQ.
28

---

Published in Transactions on Machine Learning Research (12/2024)
B Extended Results
B.1 Detailed validation results
Here we presented the full results for all datascts: approximating neural scores with analytical scores (Fig
deviations between sampling trajectory (Fig 7 and denoiser trajectory (Fig) 16 guided by neural score and analytical score.
Deviation between the denoiser trajectory D(x_t, t,t) of EDM and analytical score CIFAR10
FFHQ
AFHQv2
0.25
Gaussian- EDM Exact-EDM
Gaussian-EDM
Exact-ED EDM
0.25
Exact-EDM
0.20
GMM-EDM
0.20
0.20
0.15
MSE
MSE
MSF
0.15
0.10
0.10
0.10
0.05
0.05
0.00
0.00
0.00
10
20
30
40
50
70
80
10
20
30
40
50
60 70
80
0
10 20
30
40 50
60
70
80
noise scale (time)
noise scale a (time)
noise scale (time)
Figure 16: Deviation between the denoiser D(x,, t) of EDM neural network and the analytical score. The thick line denotes the mean over initial conditions; the shaded area denotes 25%, 75% quantile range over the initial conditions.
B.2 Additional validation experiments with DDIM
Teleportation results For the MNIST and CIFAR-10 models, we can casily skip 40% of the initial steps with the Gaussian solution without much perceptible change in the final sample (Fig 17 Quantitatively, for CIFAR10 model, we found skipping up to 40% of the initial steps can even slightly decrease the Frechet Inception Distance score (FID), and hence improve the quality of generated samples (Fig 17F.. For models of
higher resolution datasets like CelebA-HQ, we need to be more careful; skipping more than 20% of the initial steps will induce some perceptible distortions in the generated images (Fig 17 bottom), which suggests that the Gaussian approximation is less effective for larger images. The reason may have to do with a low-quality covariance matrix estimate, which could arise from the number of training images being small compared to the effective dimensionality of the image manifold.
29

---

Published in Transactions on Machine Learning Research (12/2024)
A.
Validation of Normative Theory
E.
Application: Hybrid sampling method
DDIM Trajectory
Gaussian theory prediction
Full DOIM
Theory DDIM hybrid
Full
51 steps
skipping
Gaussian
steps
25 stpps
30 steps
steps
Theory
CIFAR1O
Generated Image from DDIM vs Analytical models
DDIM
Gaussian
GMM
CelebA
Exact
Image Quality of Hybrid Sampler with different skip steps (CIFAR10)
DDIM vs Gaussian trajectory x, prediciin
D.
DDIM vs Analytical trajectory X, prediction
DDIM vs Analytical image prediction
F. FID
0.16
180
0.14-
DDIM
0.14
160-
MYEE deviation
0.12-
0.10
Mean Squared Enror
0.15-
Fxact DOII
0.12
Gaussian
GMM
140-
Exact
120-
0.08-
0.10-
100
0.06
0.06
80-
0.04-
0.05
0.04
60
0.02
0.00
0.00
0
Time (DDIM)
20
step (DDIM)
50
Gaussian
Score Model
GMM
Exact
Staps
#of skip step
Anaytical Solitio
Figure 17: Comparing analytical solutions of sampling trajectory with DDIM diffusion model for CIFAR-10. A. (x;) of a DDIM trajectory and the Gaussian solution with the same initial condition xy. B. Samples generated by DDIM and the analytical theories from the same initial condition. C. Mean squared error between the X₃ trajectory of DDIM and Gaussian solution. D. Comparing the state trajectory and final sample of three normative models (Gaussian, GMM, exact) with DDIM. E. Hybrid sampling method combines Gaussian theory prediction with DDIM. Image quality of the hybrid method (FID score) as function of different numbers of skipped steps (soc Appendix A.8
30

---

Published in Transactions on Machine Learning Research (12/2024)
MNIST modelling
A. All digits unconditional
Single Gaussian
CNM
modes
Deta GNM modes
DDIM uncond
Single Gaussian
B.
Delta GMM
conditional (each digit)
DDIM conditional
Guidance (cfg)
C.
0.08
Deviation unimodal
Caussian mixture
Unconditional model
DeltaGMM solution
D.
0.07
0.00
DeltaGMM solution DDIM
Unconditional Devietion unimodel.
Gaussian in
0.0
0.5
2,0
5,0
Unimodal to Geussian Hicture
0
0
Unmoddl to Exect
DalaaGNM
0.04
X
1
1
1
0.03
0.03
5 3
5
0.02
0.02
2
2
2
2
2
2
0.01
0.01
0.00
0.00
4
O
O
0
3
3
3
3
3 4
3
10
20
30
40
Time stops (DDIM)
steps (DDIM)
4
4
4
4
4
E.
Conditional
Conditional model (digit 1)
5 5
Deviation unimodal
Devistion of DeMtaGMM DDIM with different guidence
5
5
5,
5,
5
5
DettaGMM solution
0.035
0.06
Bagt
Digi00
0.030
X 1
6
6 7
6
6
6
6
New Sq ared
0.05
0.04
Dight
Dight
Dight3
Noan Squared
(2022
0.020
Exect CDIM w5.
RUM wo.5
7
7
7
7
0.02
Dights
8
8
8
8
8
8
0.01
0.005
Digits
0.000
9
9
9
9
9
9
10
30 Time step (DOIM)
20
40
50
step
Figure 18: Comparing diffusion dynamics guided by Gaussian and Gaussian mixture score with the ones learned by a neural network (MNIST). A. Unconditional model. In each row, it shows the image generated from single-mode Gaussian, 10-mode Gaussian mixture, Delta GMM defined on the whole MNIST dataset, and the unconditional diffusion model.
B. Conditional model of the 10 classes. In each row, from left to right, it shows the image generated via single Gaussian, delta GMM defined on all training data of that class, and the conditional diffussio model with different classifier free guidance strength 0.0 5.0.
C. Unconditional model, Deviation between trajectory predicted by unimodal Gaussian theory and 10-mode GMM and exact delta GMM. It shows that Gaussian mixture the same effect on the diffusion trajectory in the first phase as a matched unimodal Gaussian.
D. Unconditional model, Deviation between the trajectory predicted by different GMM theories and that sampled by DDIM. shows that the trajectory sampled by DDIM is actually closer that predicted by unimodal 10 mode GMM, than the exact delta gmm model. This means the model didn't learn the exact. score, but some Searse-graines approximation to it.
E. Conditional model, Deviation between unimodal Gaussian theory and exact delta GMM for different digits class. It shows some digits are better approximated by Gaussian than others. F. Conditional model, Deviation between the trajectory predicted by exact delta GMM and that sampled by DDIM with different guidance scales. It shows that larger guidance scale push the sampling result from the exact training distribution further.
31

---

Published in Transactions on Machine Learning Research (12/2024)
B.3 Extended results for Generated Image Similarity Comparison
Image distance between analytical score samples to EDM samples CIFAR10 (rescaled to 224x224)
M5E
LPIPS_alex
LPIPS_vgg
0.12
0.5
0.5
0.10
0.5
0.08
0.6
0.3
0.06
0.5
0.3
0.4
0.04
0.2
0.3
0.02
0.1
0.2
0.00
Goussian GMM
Delta
Gaussian GMM
Delta
Gaussian
GMM
Delta
Goussian
GMM
Delto
Image distance between analytical score samples EDM samples AFHQV2 (rescaled 224x224)
MSE
LPIPS_alex
LPIPS_Vgg
LPIPS_squeeze
0.12
0.60
0.70
0.50
0.10
0.65
0.45
0.08
0.50
0.60
0.40
0.55
0.45
0.35
0.06
0.50
0.30
0.04
0.45
0.35
0.25
0.02
0.30
0.20
Gaussian
Delta
Gaussian
Delta
Gaussian
Delta
Gaussian
Delta
Image distance between analytical score samples to EDM samples MSE
| FFHQ (rescaled to 224x224)
LPIPS_alex
LPIPS_vgg
LPIPS_ SiPS_squeeze
0.55-
0.45
0.10
0.50
0.6
0.40
0.45
0.00
0.35
0.10
0.5
0.06
0.35
0.30
0.30
0.25
0.04
0.25
0.20
0.3
0.02
0,20
0.15
0.15
0.10
Gaussian
Geussian
Delta
Gaussian
Gaussian
Delta
Comparing image samples generated from pre-trained EDM model versus an- alytical score models. Distance between the samples from the EDM and analytical score model ((Xo.EDM (xT); Xo,analyttcnt((x) (XT)) from the same initial noises XT were plotted. Fach panel showed one type of image metric d, MSE or LPIPS with different backbones AlexNet, VGG, SqueezeNet. Gaussian model consistently predicts EDM samples better than the exact delta score model, except for AlexNet based LPIPS in AFHQv2.
Figure 19:
32

---

Published in Transactions on Machine Learning Research (12/2024)
B.4 Extended visualizations of score vector field
field among samples
7785] labels
sigma=0.10
omm
cmmm
amm
-5
12
13
23
25
gmm 10 mode
omm mode
omm mode
goursian
mean Isotropic
1h
score vector field among 3 random samples TEST set, [5937 7460 7785] labels (5 7 2]
16
56
sigma=0.50 gmi, 100 mode
edin NN
gimm della
gmim 50 mode
grmm, 20, mode
field among random samples TEST (5937 7460 7785] labels 7
sigma=2.00
gimi
gmim_
grim,
gmmm,
gmim 10, mode
grnm mode
gmm 2, mode
gaussian
mean isotropic
15
10
45
Figure 20: Visualizing the score vector field sø(x, o) of pretrained EDM network with the analytical scores (Gaussian, Gaussian mixture, and Exact point cloud) as reference. Same plotting scheme as Fig. 11 evaluated on the plane spanned by three test set samples different classes, (2 7).
33

---

Published in Transactions on Machine Learning Research (12/2024)
score vector field among 3 random samples in TRAIN set, (17960 10172 762] labels [1 4
sigma-a m0.000..1 -0.10 gmm 100. mode
edm NN
gmm delta
gmm 50, mode
gmm 20, made
gmi 10, mode
gmm, mode
gmm, mode
geuasian
mean isotropic
10
score vector among random samples TRAIN set, (17960 10172 762] labels [1 6]
sigma=0.50 gimm mode
edin
gmm delta
gmm 50, mode
gmm 20, made
10
15-
gmm. mode
gmm
gmm
gaussian
isotropic
20
score vector field among 3 random samples in TRAIN set, (17960 10172 762] labels [1
=2.ma=2.00 cinm 100 mode
edm NN
gmm delta
gmm, 30 mode
gmm 20 made
gmm. 10 mode
gmm 5 mode
gmm. mode
geussian
mean isotropic
10
10
15
10
Figure 21: Visualizing the score vector field (x(x,a) of pre-trained EDM network with the analytical scores (Gaussian, Gaussian mixture, and Exact point cloud) as reference. Same plotting scheme as Fig, II evaluated on the plane spanned by three training set samples of different classes, (1 4 6).
34

---

Published in Transactions on Machine Learning Research (12/2024)
Score vector field among samples with same class TRAIN [57814 27510 32870] labels (5 5]
sigma-0.3
dalta
100
20
12
gmm
gmm
gaussian
Isotropic
20
score vector field among samples with same class TRAIN set, [57814 27510 32870] labels [5
sigma=0.50
edm NN
gmm delta
amm 100 made
gmm 50 made
cmm, 20, mode
10, mode
mode
mode
gaussian
mean isotropic
score vector field among samples with same class in TRAIN set, [57814 27510 32870] labels [5 5]
sigma-2u0
edm MN
gmm delta
gmmm 1000 mode
gmm. 50 mode
gmm, 20, mode
15
11
Figure 22: Visualizing the score vector field sg 5x(x,o) o) of pre-trained EDM network with the analytical scores (Gaussian, Gaussian mixture, and Exact point cloud) as reference. Same plotting scheme Fig. [1 evaluated on the plane spanned by three training set samples of the same classes (5).
35

---

Published in Transactions on Machine Learning Research (12/2024)
B.5
Extended results of comparing neural score with GMM with varying number and rank
Score Residual of EDM vs GMM score MNIST mini EDM
10".
sigma=0.01
10f
sigma=0.05
10"
sigma=0.1
sigma=0.5
sigma=1.0
Score EV Residual
1071
10-1
10-4
10
3x
10
10-1
10
10a
10a
10a
10a
10a
10a
10
10a
10
10
Rank of Mode
sigma-1.5
sigma-2.0
sigma-5.0
sigma-10.0
sigma-20.0
B
Score EV Resicual
2x
10
60
3 1x-0
32
10
10-1
10
10-5
x 10
128
10-1
10-
2 10
256
4 10
100
6x
10a
10
10a
10
16a
10a
10a
10
10
10
768
Num of Modes
Num of Modes
Num of Modes
Num of Modes
Num of Modes
Score Residual of EDM vs GMM score
MNIST mini EDM
sigma-0.01
0igma-0.05
sigma-0.1
sigma -0.5
sigma-1.0
10-1
Score EV Residual
110
10-4
10
10
10
10'
10a
10
10
10
10
10
10
10
10'
10'
10
10'
10'
Num Modes
sigma=1.5
sigma=2.0
sigma=5.0
sigma=10.0
sigma=20.0
Score EV Residual
2x100
10-3
6x 10
10
4x 10
2x10
10-2
3x 10
10
50
10"4.
10
100
10
10-5
10-3
10:
102
10a
10
10a
10a
10a
10a
10
10a
102
10a
10
102
10
500
1000
Figure 23: Deviation of the Learned Neural Score from the Score of a Gaussian Mixture Model
with Varying Numbers of Components and Ranks, Fitted on Training Data. Same plotting scheme as Fig. but for MNIST.
Upper: Residual explained variance plotted a function of the number of Gaussian modes on the x-axis, with each colored line representing different rank. Each panel compares the score certain noise scale o. Lower: The same data plotted alternatively, with the number of ranks on the x-axis.
36

---

Published in Transactions on Machine Learning Research (12/2024)
Contribution of # of Mode beyond Gaussian B. Minimal Rank (increase residual EV < 1.1) MNIST_miniEDA
SWIIST_minEEMM
10a
n. clcluter
10-2
fraction of Exp Var
100
Minimal Rank
1000
10-4
102
10-6
cluster
10-9
100
200
10
1000
-0
10-1
10"
10a
10-2
10-1
10" sigma
10a
10a
sigma
Figure 24: Comparing Gaussian Mixture Score Model with Gaussian Model for Explaining Neural Score. Same plotting format as Fig. I0 but for MNIST. A. Contribution of multiple mixture components beyond one Gaussian. B. Minimal rank required for cach noise level.
37

---

Published in Transactions on Machine Learning Research (12/2024)
B.6
Extended results of neural score structure during diffusion training
Residual explained variance of EDM score by Gaussian and GMM scores (CIFAR10 origin EDM 5k epochs early phase)
10"
0.0
10
s0gma=0.05
10a-
sigma=0.1
10a
sigma=0.5
10"
sigma=1.0
rasdual EV
30
15-1
10"
3xx 13-1
10
1x
2
30-3
10
10"
2000
4000
200C
2000
2000
2000
Score approximation
10"
11gma=1.5
10
sigma=2.0
10"
sigma=5.0
sigma=10.0
10a
sigma=20.0
mean, notrop
szore reddusi
10
10
goussian
mode
20-1
10-1
10"2-
10
gmm 11, mode
10-2
gmm, 20, mode
10-1
10
10
50 mode
gmm mode
gmm, 200, made
2000
4000
2000
4000
2000
4000
2000
4000
2000
4000
detta
apoch
epoch
epach
opoch
spoch
Residual explained variance of EDM score by Gausslan GMM scores (FFHQ origin EDM epochs early phase)
10
sigma=
0.01
10"
s.gma=0.05
10a
sigma=0.1
10a
sigma=0.5
10":
s.gma=1.0
10-4
score resiual
8 80
xx
10
10
4x 10
10
3
10
13-3
10-3
10m
10
2000
4000
200C
4000
2000
4000
2000
4000
2000
4000
10a
sigma
10a
sigma
sigma=5.0
sigma 10000-10.0
109
sigma=20.0
Score approximation botrppic gausslan
score residual
10
10
20
10
10-
gmm 2 mode
107
gmm__ 5 m_med
10
-0
10
10a
10-s_
grmm 20, mode
gmmm deita
2000
4000
2000
4000
2000
4000
2000
4300
2000
4000
spoch
epuch
egrcch
spoch
spoch
Residual explained variance of EDM score by Gaussian and GMM scores (AFHQ origin EDM 5k epochs early phase)
10
sigma=U.00
10a
sigma-0.05
10a
sigma=0.1
10a
sigma-0.5
10a
s1gma=1.0
residual EV
13-3
60-3
4
3 3
-0
2
1
10
2000
4000
2000
4000
2000
4000
2000
4000
2000
4000
109
s.gma-1.5
100
2ig
10a
sigma=5.0
sigm 110.0
sigma-20.0
Score approximation
scone midial EV
10
1011
mean. botropic gaussian
20
10
10-2
gmm 2 mode gomn
10
10-2
10-2
10
10
10n
gmm 20 mode
gomn mude
4000
2000
4000
2000
4000
2000
4300
2000
4000
epoch
egoch
egoch
epoch
epoch
Figure 25: Deviation between the score sp(x,,t) t) of EDM neural network and the analytical scores
during diffusion training process. Similar to Fig 12 but for CIFAR10, AFHQv2 64, FFHQ 64 datasets.
38

---

Published in Transactions on Machine Learning Research (12/2024)
Residual explained variance of EDM score by Gaussian and GMM scores (MNIST EDM 25k epochs early phase)
sigma=0.05
sigma=0.1
sipma=0.5
sigme=1.0
EV
60
10-7
10
10
2x
10
10
Score approx mation
10a
sigma=1.5
led 10"
sigma=2.0 lel 10"
5.g
1st
sigma=10.0
Led 10"
sigme=20.0 1u4
mean isotragc
score resioual
10-
10-
10-1
1011
1072.
mode grnm
gom mude
10
10:
gmm. Sl, mode
10
10"
10"
10
youm
100, mode
gmmm 5000 mode
opoch
opoch
lef
esach
1-f
epoch
1ed
opoch
umm dilta
Residual explained variance of EDM score by Gaussian and GMM scores (CIFAR10 EDM 50k epochs early phase)
10a
sigma=0.01
10".
sigma=0.05
10a
sigma=0.1
10a
sigmam0.5
10"
sigma=1.0
10-
4x10-1
10
2x10
10"1_
10-0
10"-
sigma=1.5
Aed 10"
sigma=2.0
1el 10"
5.0 1xt
sigrm=10..0 Iwl 10"
sigma=20.0
Score mation
score EV
10
1011
gaussian
10-1
10
10-2
mude gmmm__mm grmm 10, mode
10-2
10
10-2
10
gmm, 20, mode
2
50
gmm, 100
gmmm della
opoch
opoch
oooch
cpach
opoch
Figure 26: Deviation between the score (x,, t) of EDM neural network and the analytical scores
during diffusion training process. Similar to Fig 12 but for MNIST and CIFAR datasets, using an alternative code base for EDM training (mini-EDM)..
39

---

Published in Transactions on Machine Learning Research (12/2024)
B.7 Extended visualizations of neural score field during diffusion training
40

---

Published in Transactions on Machine Learning Research (12/2024)
score vector field among random samples TEST set, [5937 7460 7785] labels 7 2]
sigma=0.10
ep500
ep1000
ep2500
ep5000
ep24999
20
gaussian
gmm mode
mode
gmm 500 mode
delta
-5
15
25
10
10
score vector field among 3 random samples in TEST set, (5937 7460 7785] labels [5 7 2]
sigma-d 0.50
edm NN ep500
edm NN ep1000
edm NN ep2500
edm NN ep5000
edm NN ep24999
10
15-
20-
gaussian
gmm, mode
gmm gmm_100 100,
_00_mode mode
gmm__00 500, mode
gmm della
10.
20-
vector field among random samples in TEST [5937 7460 7785] labels 7 2] sigma=2.00
NN
ep2500
1.5
gaussian
mode
100 mode
500 mode
delta
15
10
20
10
Figure 27: Visualizing the score vector field s,(x,, t) neural network during diffusion training process with the analytical scores as reference. Same plotting format as Fig. Score functions are evaluated on the planc spanned by three samples with different classes.
41

---

Published in Transactions on Machine Learning Research (12/2024)
score vector field among 3 random samples in TRAIN set, (17960 10172 762] labels [1 4 6]
sigma=0.10 10 Edm NN ep2500
edm NN ep500
edm NN ep1000
edm NN ep5000
edm NN ep24999
mode
100 mode
gmm 500 mode
gmm delta
10
vector field among 3 random samples TRAIN set, [17960 10172 762] labels 6]
sigma=00 10yma=0,50 edm NN ep2500
edm NN ep500
edm NN ep1000
edm NN ep5000
edm NN ep24999
gaussian
gmm, 10,
gmm mode
gmm
gmm
10
15
10
score vector field among 3 random samples TRAIN set, [17960 10172 762] labels 6]
sigma=2.00 edm NN ep2500
edm NN ep500
edm NN ep1000
edm NN ep5000
edm ep24999
5
gaussian
gmm mode
gmm mode
gmm mode
gmm
15
20
10
15
10 15
10
35
10 35
10
15
Figure 28: Visualizing the score vector field sx(x,, t) of neural network during diffusion training process with the analytical scores as reference. Same plotting format Fig. 13 Score functions are evaluated on the plane spanned by three training samples with different classes
42

---

Published in Transactions on Machine Learning Research (12/2024)
score vector field among samples with same class in TRAIN set, [57814 27510 32870] labels [5 5]
sigma=0.10
edm NN ep500
edm ep1000
edm ep2500
edm ep5000
edm NN ep24999
gaussian
10 mode
100 mode
500 mode
delta
12
10
15
15
15
26
15
20
score vector field among samples with same class in TRAIN set, (57814 27510 32870] labels [5 5] sigma=0.50
edm NN ep500
edm ep1000
edm ep2500
edm ep5000
edm NN ep24999
15-
gaussian
10 mode
100 mode
500 mode
della
10
10
10
15
15
25
15
score vector field among 3 samples with same class TRAIN set, [57814 27510 32870] labels 5]
sigma=2.00
edm eg500
edm NN ep1000
edm NN ep2500
edm NN ep5000
edm ep24999
gaussian
mode
100 mode
500_mmdd
delta
5-
10
20
10
10
10 15
10
Figure 29: Visualizing the score vector field sø(x,, t) of neural network during diffusion training process with the analytical scores as reference. Same plotting format as Fig. evaluated on the plane spanned by three training samples with same classes (5).
13 Score functions are
43

---

Published in Transactions on Machine Learning Research (12/2024)
B.8 Extended results of FID scores for Gaussian analytical teleportation
FID score of hybrid sampler with analytical teleportation
CIFAR1O EDM-hybrid FID
FFHQ64 EDM-hybrid FID
AFHQV264 EDM-hybrid FID
16
14-
80
12-
10-
10-
F10
40-
6-
20-
4-
01
15
20
25
30
35
40
60
80
40
60
80
Neural Function Eval INFE)
Neural Function Eval (NFE)
Neural Function Eval (NFE)
20
10
20
10
skipping steps
skipping steps
skipping steps
1 2 5 10
20 40 60 80
2
5
10
20
30
40
60
80
2
5
10
20
30
40
60
80
skip noise level
skip noise
skip noise
Figure 30: Image quality a function of skipping steps for hybrid sampling approach. Note the main x-axes are the number of Neural Function Evaluation (NFE); the sccondary x-axes are the number of skipping steps from the Heun sampler; the tertiary-axes are the time or noise level askip at which we evaluate the Gaussian solution. See Tab for numbers.
The effects of applying the Gaussian analytical teleportation on the FID scores are presented below (Fig 30 Tab 5 6
For CIFAR-10 model, we found teleportation not only reduces the number of required neural function calls, but also lowers the FID score. Admittedly, the FID score effect is quite small the CIFAR-10 model we tried: it goes from 1.958 (no leleportation) to 1.933 (5 steps replaced by teleportation), which is around a 1% decrease. Visually, the difference in sample images is mostly negligible; there are some changes, but they are plausible changes in image details. Hence, teleportation saves 29% of NFE and improves the sample quality, without changing the model or the sampler. Further, skipping 6 steps will save 34% NFE but increase the FID score by less than 2%.
For AFHQv2-64, the teleportation can save 25-30% of NFE (10-12 steps skipped) or reduce the FID by 2% (6 steps skipped). For FFHQ64, the teleportation saves 10-15% of NFE (4-6 steps skipped) with around 3% increase in FID.
The full table of FID scores as function of the number of skipped steps is shown below.
The fact that replacing NN evaluations with the analytical solution can even improve the already low FID score (albeit very slightly in the tests we just ran) is intriguing, and it is not immediately obvious why this is true. One possibility that, early in reverse diffusion, the neural net might be a worse or noisier version of the Gaussian score.
44

---

Published in Transactions on Machine Learning Research (12/2024)
Table 4: FFHQ64 FID with analytical teleportation
Nskip
NFE
time/noise scale
FID
0
79
80.0
2.464
60.1
2.489
44.6
2.523
6
67
32.7
2.561
8
63
23.6
2.617
10
59
16.8
2.681
12
55
11.7
2.841
14
51
8.0
3.243
16
5.4
4.402
20
2.2
15.451
Table 5: AFHQV264 FID with analytical teleportation
Nskip
NFE
time scale
FID
0
79
80.0
2.043
2
75
60.1
2.029
4
71
44.6
2.016
6
67
32.7
2.003
23.6
2.005
10
16.8
2.026
12
55
11.7
2.102
14
51
8.0
2.359
16
47
5.4
3.206
20
39
2.2
14.442
Table 6: CIFAR10 FID with analytical teleportation
Nskip
NFE
time/noise scale
FTD
35
80.0
1.958
1
33
57.6
1.955
2
31
40.8
1.949
3
29
28.4
1.940
4
27
19.4
1.932
5
25
12.9
1.934
6
23
8.4
1.963
7
5.3
2.123
8
19
3.3
3.213
10
15
1.1
23.947
12
11
0.3
109.178
45

---

Published in Transactions on Machine Learning Research (12/2024)
B.9 Extended results of FID scores for Gaussian analytical teleportation: Experiment 2
FID Hesemap by Steps and Skip Value Dataset:
302
FiD Heahmao by Steps and S4ip Vilue Datase:: cfor10 Solven: dom solvara
FID Hewtmap by Steps and Skip Value Dacasot dpm solvar
25
30.07
20.75
26.27
26.12
24.42
$4,27
21.20
-3.333
27.02
26:34
15.54
27
75
480
4.12
14:27
13.99
9:35
2:25
6:26
5:59
5:29
4.35
15:21
65
9:91
1:39
12
2.22
232
33:39
2:75
8:37
22. SpoWe
20
300
1.9
78
35.20
43.76
12:13
5.00
3.70
2:65
2.00
2.01
14:34
2:22
3:71
2:44
2:31
2.34
2:04
2.25
6:51
5.74
127
3.52
3.20
3:12
2.07
2.00
405
24.27
3:41
5:07
2.23
2.06
16-31
9:20
2:45
2:14
2:04
2:86
2:45
2.15
7.00
0.07
10
16
35
6
17
15
30
Steps
Strpp
Steps
FID Hoacmap by Stops and Skip Value
FID Heabmap by Steos and Skip Value
30.00 62
31.04
20.38
27:54
21:03
20:44
25.33
25.70
33.73
33.02
30:40
20:31
27.40
25.72
20.00
9.29
3.74
5.04
23.14
21.03
5.82
5.22
20
1000
2.59
2.37
2.04
2.05
100
13.17
6.00
5.89
2:74
2:22
2:00
250
15
1:37
2.72
2:35
2.05
2:66
2.55
250
33:02
6:32
3:50
2:75
2:20
2.07
2:25
2.04
450
403
100
313
7:50
15
7.06
24:13
11.06
2.41
7.54
20
20
25
12
15
Stepe
Figure 31: Image quality as function of skipping noise scale askip and sampling step number for hybrid sampling approach (CIFAR10). y-axis of heatmap (Skip Value) denotes the askip (edm convention, min = 0.002, Jmaz = 80); x-axis denotes sampling steps. Each panel features hybrid sampler with one deterministic sampler: Heun, DPM solver+ DPM solver vs, UniPC-bh1, UniPC-bh2.
46

---

Published in Transactions on Machine Learning Research (12/2024)
Skip
Dataset: afhqv264 Salver:
Dataset: afhqu264 Solven dpm_ salvere-
16
54.98 54.39 54.10
54.49
54.36
26.25 25
34.61 13.74 12.65 12.30 12.02 11
17:03
15.33
14.30
13:46
12.74
5.3
Skip Velue
10.
8:32
5.01
200
2.92
20,0
7:82
4.51
420
3.68
2.38
9.04
4.31
7.75
4.01 2.74
2.33
17.86
10.55
4.18
6
10
12
15
70
75
30
195
5
10
12
15
70
75
40
-00
Stages
Stegps
Skip
Heatmap
Dataset:
Salven unn_pp__uu
afhqv264 Salver:
-53.08
05
54.61
54.26
33.66
53
46
35.87
54.41
40
Skip Wabie
120
30
7:95
5:12
30
2000
2.06
1.00
20,0
7.31
4.61
403
2.13 2.01 1.90
70
8:38
:2.90
8.22
4.61
3.84
2.77
2.27 2.07 2.08
2.01
00
2.00
17.33
10:10
4.04
3,42
2.64
2.07
2.02
2.01
3.00
2.00
8
10
17
15
70
75
30
89
5
8
8
10
12
11
70
75
Steps
Stops
Figure 32: Image quality as a function skipping noise scale aakip and sampling step number for hybrid sampling approach (AFHQv2). Same plotting format as Fig.
47

---

Published in Transactions on Machine Learning Research (12/2024)
FID Heatmap by Steps and Skip Value Dataset: finq64 Solver: houn
FID Heatmap Steps and Skip Value Daraset: thqg4 Solver: dim sal.erel
40
85.23
58
31
51.74
48.10 47 96 47.29 46.02 46.42 46.29 46.24 46.18
45.41
47.31
47.69
46.56 46.11 45.99 46.00 46.03 46.04 45.05 46.06
250
15.42
33.25
20.33
16.39 14:83
14.01 13.36 13.15 13.07 13.00 12.99
18.56
16.67
15.84
14
13.95
13.51
13.27 13.15 13.09 13.05 15.01
103.99
56
18.58
10.11
7.64
6.25 5.29 4.93
4.83 4.75
14.99
10.83
8.80
7:35
6.42
5.65
5.20
5.01
4,90
5kip Vabre
10.0
7:03
5:30
10.0
16.29
5:40
4.52
3.78
20.0
20.9
1600 40.3
40.0
1000
3.02
10
12
15
20
25
30
6
10
12
15
20
30
Stops
Stops
Dataset: rthq64 Solver: ani__pp_hll
Dataset: ffng64
bh2
40
45.79
45.98
46.08
46.10 46.20 46.09 46.08
30
30
Skip Weler
10.
3.70
10.
Ship
250
5.29
3.98
3:24
2.65
200
26.0
9.19
5.65
4.35
3:47
4033
5:57
3.13
2.75
403
17.33
9.96
3.92
3.21
-15
02.0
19.97
10.97
5.51
3.80
3.22 2.77
2.52
45
2.42
2.41
00.0
21,42
12.90
5.64
3.80
3.30
2.70
2.52
2.42
2.41
8
10
12
15 Stops
20
25
6
8
10
12
15
20
25
30
40
Stops
Figure 33:
Image quality as a function of skipping noise scale Oskip and sampling step number for hybrid sampling approach (FFHQ). Same plotting format as Fig. 31
48

---

Published in Transactions on Machine Learning Research (12/2024)
C Notation correspondence
Diffusion models usually have forward processes whose conditional probabilities are
xxx:(x0)= NAxxo B,I)
(42)
for all t [0, In the limit of small time steps, the transition probability distribution can be captured by the SDE
* = - + Den(t)
(43)
where 77(t) vector independent Gaussian white noise terms.
Papers discussing these models may use slightly different notation. In the table below, we briefly indicate how various choices of notation correspond to one another. To make comparing discrete and continuous models easier, assume the time step size is =t=1.
Table 7: Comparison of notation for diffusion model parameters. t: EDM formulation does not explicitly specify a forward SDE in the paper, so we left out these notations.
9,
C2
D2
DDPM
DOMM
Soau
oc/or-1
STABLE DOV.
ROMBACCE
2022
SDB
4 So piaida
scajde
A(N)22
B(e)
NAT
XDM WITH BOALING
KORRAS ST
at))
(1))e (X)
NAT
NAT
OuRS
a(O)
2(1)
the popular huggingface diffusers library implementation diffusion models, the function cumphrdd corresponds our of.
49

---

Published in Transactions on Machine Learning Research (12/2024)
D Detailed Derivations
D.1 Detailed derivation of the solution of the Gaussian Diffusion
In this section, we derive the analytic solution X for the probability flow ODE in EDM formulation (Eq with Gaussian score. We will assume that the score function corresponds to a Gaussian distribution whose mean lr and covariance matrix Σ. We will also assume, given that images data are usually thought of as residing on a low-dimensional manifold within pixel space, that the rank r of the covariance matrix may compact
be less than the dimensionality D of state space. Let Σ = UAU" be the cigendecomposition SVD of the covariance matrix, where U is D x semi-orthogona matrix whose columns normalized (i.e. UTU- = I),, and A is the r x diagonal cigenvalue matrix. Denote the kth column of U by and the kth diagonal element of A by λk-
The probability flow ODE used for the EDM model reads,
dx o I + + x x)dd
(44)
1 -1 UĂo )H x)do
dx = -
(45)
σ
d
Ao = diagl
(46)
d + a²
We can see it's linear ODE with respect x - n., with disentangled dynamics along each eigenvector us- Choosing dynamic variable to be the projection cocfficient each axes (x u (x μd) Then its dynamics could be written as
dc (o) Au o (o)do
(47)
Integrating this, we get the following, with integral constant K.
dex(o) = dlog V d + o²
(48)
cx(o)
(o)) dlog V d + x
(49)
cx(o) Vdu + O²K
(50)
Using the initial condition cx( up (XT 14), we have
cx(o)
A + 0
=: ()..0)
(51)
(T)
d + o7
Thus, we arrive at the solution Eq 15
N, 11 = Σ (k(uu
(52)
X, = p + Σ
(Ae,, o)ca
(53)
X, + Σ
A + o²
u (XT
p))
(54)
A of
When the U is rank deficient, will recover the off-manifold term as in Eq. For the denoising outcome, we have Eq. 3
15 by setting d 0.
D(x,)= o² V log (x) 0)
(55)
= = (1 UĂ )² x)
(56)
μ + UÂ TT (x-
(57)
50

---

Published in Transactions on Machine Learning Research (12/2024)
Bring in the solution of x above, we have
0) = = E
d Vds + (2) + of)
1 (4)
(58)
If we define
A O) V(A + ) + o
(59)
and use the definition of e (T), then we get the trajectory for denoiser image as Eq 16
D(x,: σ) = + Σ (()))
(60)
D.2 Deriving solution to PF-ODE with data scaling term
If we have data scaling term at, we can define the scaled state X; a,x,, and the effective noise scale the PF-ODE governing the dynamics of this state will be
o, 0,0c. Equivalent to Eq 2
dx, = a,dx, + da
(61)
= [-ajdro Vx log p(x, σe) + djx,ddt
(62)
[ of Vx log oz)]
(63)
at
Without solving this ODE directly, directly translate the solutions of Eq. 15 16 by subsituting
X xz/O; and o1/an. Using this simple rule, get the solutions for general probability flow ODE with data scaling.
2 02/a?
X = + Σ
us u
(1)
(64)
V
d + 07/
dxa + of
=0 + Σ
u (x) aTp)
(65)
dRa + of
D(x,: = +
Σ k
ard
nnu (xT arp)
(66)
(att + (f) +07)
If we set =1, 1, these solution recovers the form in Eq. solution of VP-SDE or DDIM.
15
16 If we set af + of = 1, it will recover the
D.3 Detailed derivation of the solution of the Gaussian Diffusion with VP-ODE
In the following section, we present an ab initio derivation of the solution to VP-SDE. We consider forward processes defined by the stochastic differential equation (SDE)
x - g(t))(t)
(67)
where ((t) controls the decay of signal, gt) a time-dependent noise amplitude, 17()) a vector of independent Gaussian white noise terms, and time runs from 0 to T. Its reverse process is
x - g(1)) g())(())
(68)
where s(x, t) := Vx lo,, t) is the score function, and where we use the standard convention that time runs backwardd, i.e. from = to 0. The variance- sirianccepreeevingg SDE, which enforces the constraint 3(1) (1). The marginal probabilities of this process are
p(x,|xo) [ Nxxxx
on = = S a(r)a
of -2 8(e))a
(69)
51

---

Published in Transactions on Machine Learning Research (12/2024)
where ag and de represent the signal and noise scale, satistying oj + of =. Normally, goes from 0 T, signal scale oz monotonically decreases from 1 - 0 and increases from 0 - 1. The equivalent probability flow ODE with the same marginal probabilities Song et al. (2021) is
1 x = -(((xx-- 2 (()
(70)
where time again runs backward from / = 10
For Gaussian score, the probability flow ODE that reverses a VP-SDE forward process Song al. 2021 is
-3()xx-- 1 (0) - (asfu - x)
(71)
Using the eigen decomposition of Σ described in the main text,
1 1 (I AI--UA,"")R (a, 2 ( /े
x = -
(72)
where Å, is defined to be the time-dependent diagonal matrix
af Ax A + o
A. diag
(73)
Consider the dynamics of the quantity aup Using the relationship between 3 and oz, have
d (x, a114 x2 -pcu
(74)
dt
x; Brasu
(75)
1
1 _
= BBlaast -x) -000()
(I UA, (am of
(76)
1 1 _ (I- UA -8/1 (x aap) of
(77)
If we assume that the forward process is a variance- variance-preservingg preeervieseeeg SDE, then Br = )t)))), which implies af =1 1 o.. Using this, we obtain
d d
(I UA,U7) - 1 (x asp)
(78)
dl
0
1 oे [ (x
3r 1)1
( 0,
(x- aep)
(79)
Define the variable y: == -x-- We have just shown that its dynamics are fairly 'nice', in the sense that the above equation is well-behaved separable linear ODE. As we are about to show, it is exactly solvable.
Write ye in terms of the orthonormal columns of U and component that lies entirely in the orthogonal space U-.
y y 1100+>ath cc(() y (
(80)
The dynamics of the coefficient cx()) attached to the eigenvector uy
1 of d
Es (t) =
(u = 31
2 - 1)
(uffr)
(81)
di
00
of
af
Br
ai
1
-
2
ct(1)
(82)
0,
of d o
Bra7
d
1
cx(1)
(83)
of
x o
52

---

Published in Transactions on Machine Learning Research (12/2024)
Using the constraint that of + oî 1, this becomes
cx(t) = B1af(1 - 2x)
(x()).
(84)
a² oo²
For the orthogonal space component y (t), it will stay in the orthogonal space U_, and more specifically the 1D space spanned by the initial y so, when going backward in time, its dynamics is simply a downscaling of (T).
(
1- - of
B of y' (1)
(²(1) Be of
(1)
(85)
Combining these two results and solving the ODEs in the usual way, we have the trajectory solution
d(l)y (T) +
cx(()ux
(86)
km
d(() exp (fr dr
3.02
(87)
oZ
(t) (T) exp (f, dr
Bra 2x)
(88)
a+ + o²
The initial conditions are
cx(T) = u YT
(89)
y (T) = YT Σ (T) (T))
(90)
To solve the ODEs, it is helpful to particular reparameterization of time. In particular, consider reparameterization in terms of ag using the relationship -B:ordt = doy. The integral we must do is
3 3, AR) Sr dr aZ + +
dr 3, (1-11- -14+)
(91)
1 1 a (}x - 1) pa:
T
ar (> - 1) 1 a - 1)
da,
(92)
2 log(1 + +z2 1)) a8
(93)
1 (A+ - 1)a
log 2
(94)
I+ (Ax - 1)o
Note that taking A = 0 gives us the solution to dynamics in the directions orthogonal to the manifold. We have
+ (Ax 1)a?
(+ = (T)
(95)
1 (AR - 1)ag
a3
d(1) =
(96)
o7
The time derivatives of these coefficients are
- (AA - 1)a B.
cx c& (T)
(97)
V (1+ (As 1) (1 (Av - 1)a
a7 B
d(t) ((1 [1 of
(98)
53

---

Published in Transactions on Machine Learning Research (12/2024)
Finally, we write out the explicit solution for the trajectory x,
x = =0 + + (T) + (t)uu
(99)
We can see that there are three terms: 1) ax/4,, an increasing term that scales up to the mean j of the distribution; 2) d(t)y (T), a decaying term downscaling the residual part the initial noise vector, which is orthogonal the data manifold; and 3) the es (f)ug sum, each term of which has independent dynamics.
We also now have the analytical solution for the projected outcome:
xx(xx) - 1r = UÂ (xt - aaH4)
(100)
asdR
(t(t) 2
Uk
ai A1 oz
k=
(T)
u
V(a + (a) + of7)
54

---

Published in Transactions on Machine Learning Research (12/2024)
D.4 Derivation of rotational dynamics
In this section, we derive various results quantilying how reverse diffusion trajectories are rolation-like. In particular, under certain assumptions, we will show that the dynamics of the state X₁ looks like a rotation within a 2D plane spanned by Xy (the reverse diffusion endpoint) and Xy (the initial noise). We will derive the formula by assuming that the training set consists a single Gaussian mode, but will explain why this assumption may not be strictly necessary.
D.4.1 Derivation of rotation formula and correction terms
Assume that reverse diffusion begins at time T with aT R O, and ends at time t = with ap = 1. Using our exact solution for x, (Eq. 15), at some intermediate time t we have that
a at
1+ (AA 1)a
X = app +
(T) +
ck (T)u:.
(101)
1+ (Ax 1)
It is also true, by substituting t 0 and t T, that
# Xu
Ck uu)
1 + (A+ 1)a9
(102)
(T) = XT O.T44 Σ Ck (T)(u
Using these two equations, we can rewrite Eq. 101
1 a²
1 (Ax 1)a2
Xx =
ory.
XT
aTH
Cx
(T)Tuu
+
1 + (Ax= - 1)07
Ca (T)us.
+ (Ax - -1)
1 at
0T
1 +
of
XT
1 (Ax 1)
az
Cx (T()))
(103)
a
1 at
at
QT
X0
XT
+
Rs
where the remainder term R2 is equal to
1 (Ax 1)a2
a
1 1- a²
A
R The expression simplifies somewhat if we take ar R 0. Doing so, we obtain the equation seen in the main text:
a, αT
Ca (T)
I+ (A& 1)
1 - a
1+ (Ax
1)
k=
X R 0xX0 + V1 1 ot XT + Σ
of (voz
d - ar aI -0 ca (T)uk
Let's examine the correction terms more closely. Define the function
Mai; = Voz do - ar V az
(104)
= -11+ (A - 1)a -VvV - V1-
Note that we can rewrite J as
(Vo + da - v.VT - -1)) (Vo7 + da + aav
Mai;A)= =
Vot + foaY + FavAto
(105)
=-2 oo + 1o tavvtor
55

---

Published in Transactions on Machine Learning Research (12/2024)
From this, it is immediately clear that the correction term is not completely arbitrary. First, J is always negative. Second, its time course is bowl-shaped: it begins close to zero (since R 0), becomes more negative, then ends close to zero (since do 2 0). It achieves its most negative value roughly when or and ag V are comparable, i.c. when
~ V VA+1
(106)
in which case
J(a; A) = -2 22 v2 2 A+1 2 -2 2(1 1-
V2 2 = -0.586
(107)
Notice that [J] < 1, regardless of X.
Although we derived this formula by assuming a single Gaussian mode, its form does not actually depend on any properties of the mode. This suggests that, as long as the score function landscape looks locally Gaussian, the formula may still be applicable. For example, suppose the learned image distribution is a Gaussian mixture. Even though the mean and the covariance of the nearest mode the one which we expect to dominate the score function may regularly change throughout reverse diffusion, even in a discontinuous way, the rotation equation should stay the same.
D.4.2 Low-rank image distribution sufficient for small correction terms
Suppose that the rank r of the covariance matrix Σ much less than D, the dimensionality of state space. The error in the rotation formula is
0xXo V1
of Xx = Σ Man Ax (T))
(108)
Recall that (T) is the coefficient of the original noise seed xy ~ I), along the direction ug. Assuming D is large, the norm of the noise seed is approximately 1. Since there is a priori no relationship between xy and uk, we expect that X7 uy ~ I/VD. (Suppose we express Xy in terms of a set of D orthonormal basis vectors. Given that its norm 1, and that it has no special relationship with any basis vector, the overlap between Xy and each vector must be about 1/VD.) The error becomes
Xx
a:xo -;
V1
af XT
R 1
Man Av) 2 D
D
1
2 2
D
= 4
1(1111111
2
2
D
r
where we have used the bound from the previous subsection. Since r < D, this error is small.
It is worth noting, however, that the r < D assumption not necessary for the rotation formula correction terms to be nogligible. Another case in which this is true is when the image distribution is isotropic, i.c. As- A for all k. Then the error is
x 0:50 - V of xr xr| ~
(1-
v2 2
A A+ID X+1
r
2
This is somewhat smaller than the typical scale of xt, since
|x 112 remains roughly between 1 and X.
D.4.3 Can the rotation formula be used to predict the trajectory endpoint?
Naively, since any two vectors are mathematically sufficient to define plane, the rotation plane should be completely determined from the first two steps-or if not the first two steps, one might naively expect the first several steps to be sufficient. In particular, since
X 0: / aixt R.,
(109)
where R, is the correction term we derived carlier, we can approximate X9 as
of
Xg
(110)
at
56

---

Published in Transactions on Machine Learning Research (12/2024)
The problem is that the correction term along each feature direction is 'large' until that feature has been 'committed to'. Concretely, the correction term along direction us is proportional to
Jay; Ab) -2 Vo? + Ao2 + tovā+o,
VIV
(111)
at
This function has salurating behavior, and remains high (in the sense of being Axx until around the time when / v1-až, az. But consistent with our other results (sce Figure 3 and the associated formulas), this is roughly around the time of 'feature commitment', or more specifically when the sigmoid-shaped coefficients of & begin to transition to their final value. So the rotation formula only becomes useful for determining the endpoint after enough time has passed that one is sufficiently close to the endpoint.
Using multiple X2 does not help reduce the error in applying the rotation formula, since the error equation above is monotonic time. In other words, averaging over multiple recent X₁ is striclly worse than just using the rotation formula and the most recent (i.e. greatest number of reverse diffusion time steps) X
As a final note: since we have the form of the correction terms, why not use that as additional information? We could do this, but this only works for the Gaussian case, where already have access to the full solution! And knowing these terms at all times is also roughly equivalent to knowing the full trajectory. So in summary, viewing reverse diffusion trajectories as 2D 'rotations' is a useful geometric picture, but it is less quantitatively useful than the full analytical solution to the Gaussian model, e.g. for accelerating sampling.
D.4.4 Rotation-like dynamics beyond the Gaussian model
In the more general non-Gaussian sctting, there an altornative line of evidence that reverse diffusion dynamics are rotation-like.
Derivation from EDM formulation with state scaling We can express the PF-ODE in EDM framework (Eq using the ideal denoiser (Eq
dx; dt
-,,0,((x00)) σ.)
(112)
δi D(xx: 0i) - xi)
(113)
d (D(x) xx)
(114)
di
When an arbitrary time-dependent state scaling function a introduced, we can substitute x, + x1/O1, 0, - 0z/0s, which yields the equation for the PF-ODE with the scaled states.
d
In D Ot
(115)
dl
Ctt
Ctt
Ot
When we assume the denoiser target
D dt)
D is fixed, then this equation can be solved directly,
as
ae
d Xt
= d
In
at
Xt
D
(116)
dt
04
dt
at
14
X
D exp(
d
dln
Xta
D)
(117)
0tt
Okto
Cto
Xta
exp(In
In
)
D)
(118)
0tt
Ok1o
Cito
Xto
D)
(119)
0400to
aito
57

---

Published in Transactions on Machine Learning Research (12/2024)
where to is reference time where start solving this. This solution shows invariant quantity the trajectory
X; a,D
X1p Otg D =; Const
(120)
0t
X a, + o2 Const
(121)
In the specific case of VP-SDE, the scaling term satisfics of + of = 1, then
= a, V of
(122)
This means the solution of x; can be interpreted as a spherical interpolation between the hypothetical initial state Const and denoiser state D. We used the word rotation a very loose sense: this trajectory will be an are on spherical surface when D and const have the same norms and orthogonal to each other. These criteria are not always true actual diffusion trajectories.
Derivation from VP-ODE formulation Similarly, we can also re-express the VP-ODE as rotation. Using the formula of the projected outcome
xx(x + of Vx p(x,()
(123)
at
we can write the probability flow ODE (Eq. 70 as
1 9 (t) logglo ()= x -B(t)x
g9((() [a,80(x) -x] x]
(124)
2 of
Notice that
all) = exp (-f" 3(F))d))
aa(1) = 31))a(()) =8()(a))()a())
di
(125)
which allows us to write
d
Xt
x;
àix;
x; +
3ix,
(126)
dt
at
2
Equivalently,
Xx
(t)
2x(xx))
Br of
&o(x:)
Xt
(127)
of
0+
From this, we can see that the quantity xz/an, ie. the state scaled by the signal scale, is isotropically attracted towards the moving larget &x(x,) at rate determined by 9((()) of
Suppose that the endpoint estimates Xy change slowly compared to the state x;; we will show that this gives us rotation-like dynamics.
First, note that can evaluate the integral
Sr
3r di of
1" 3x dt 1
(128)
oz
using a change of variables = a₁ with da 3a dt. The integral becomes
da
log
(129)
o9 a(1 - -²
OT
58

---

Published in Transactions on Machine Learning Research (12/2024)
Using this integral, can find that the solution to Eq.
127
under the assumption that Xy remains constant is
20
v
log
a:
= log
OT
80
a1
v at
(130)
X₁
02
XT
ai
of
Interestingly, this indicates that there is a conserved quantity
X QxXo V1 az
const.
(131)
along the reverse diffuusio trajectory, under this approximation. Since or R 0, so const., L.e. the value of the constant roughly matches the initial noise seed xy. Given any &o the solution to the ODE at time can be written as
X2 04 V1 v of const. 0,&0 + V1 V1-o2
of xT
(132)
In words: through rotation, x, interpolates between &y and const. This solution paints the picture that the state is constantly rotating towards the estimated outcome & with Eq. 132 describing that hypothetical trajectory's shape. But the target Xy moving, the actual trajectory will be similar to Eq. 132 only short time scales, and not on longer time scales. This idea is visualized by the circular dashed curves in Fig.
Beyond the constant & approximation, there will be some correction terms to the above rotation formula, whose precise form depends on the (generally not Gaussian) score function.
59

---

Published in Transactions on Machine Learning Research (12/2024)
D.5 Score function of general Gaussian mixture models
Let
g(x) = Σ (x; E.)
(133)
be a Gaussian mixture distribution, where the n, are mixture weights, b the i-th mean, and Σ, the i-th covariance matrix. The score function for this distribution is
Ei ty V XxNVNNN Mi; E;)
Vx log g(x)
g(x)
TNN(x; Mi, E.)
Ex 'x uii)
((x)
(134)
Txi Mi, Ei)
Vx log, NoggN(x) Mis E.)
g(x)
wx(x) Vx log Log.Nolg/s(ss/iii M4: E.) E.)
where we have defined the mixing weights
wx(x) = TNV(x: Hi, E,)
(135)
((x)
Thus, the score of the Gaussian mixture weighted mixture of the score fields of each of the individual Gaussians.
In the context of diffusion, are interested in the lime-dependent score function. Given Gaussian mixture initial condition, the end result of the VP-SDE forward process will also be a Gaussian mixture:
(x(x)= Σ m.N(x E.)
(136)
The corresponding time-dependent score is
s(x, t) = Vx log (x)
Σ - + Ex)-1 m1)) T:N of + E,)
(x(x)
(137)
Σ (o, + E2) -x (x)xx(xxxt
Note that have formula for (afI + E,) -1 (derived using the Woodbury matrix inversion identity; see 5 in terms of the (compact) SVD of Σ. We can use it to write
1 Σ (I (xAx, - (x,
s(x, 1)
(138)
0,
where
b A oo
Σ U.AUU
A. = :daddiag
(139)
This representation of the score function is numerically convenient, since (once the SVDs of each covariance matrix have been obtained), it can be evaluated using a relatively small number of matrix multiplications which are cheaper than the covariance matrix inversions that a naive implementation of the Gaussian mixture score function would require.
We used this formula (Eq 138 and an off the-shelf ODE solver to simulate the reverse diffusion trajectory of 10-mode Gaussian mixture score model (Fig. 17
60

---

Published in Transactions on Machine Learning Research (12/2024)
D.6 Score function of Gaussian mixture with identical and isotropic covariance
Assume each Gaussian mode has covariance Σ; I and that every mixture weight is the same (i.e. Wं = Ty, Vi,j). Then the score for this kind of specific Gaussian mixture is
E, T. VxN O'I)
V log (x)
Σ, M4, 0² Σ, (x 14) exp
20" |Ix mall2)
(140)
|x 14:12) Σ, exp -
Σ we (x) x)
where the weight (x) is a softmax of the negative squared distance all the means, with o² functioning temperature parameter:
wx(x) Softmar(1
2g xx Mul 1111
1
exp
2 x
(141)
E:
exp
20
|x
Since the weights (x(x) sum to 1, also write the score function in the suggestive form
Vx log g(x) C (x)xx))
(142)
o2
This has form analogous the score of single Gaussian mode but instead of being 'attracted' towards single mean Jr, is attracted towards weighted combination of all of the means, with modes closer to the state being more highly weighted.
D.7 Score function of exact (delta mixture) score model
A particularly interesting special case of the Gaussian mixture model is the delta mixture model used in the main text, whose components are vanishing- width Gaussians centered on the training images. In particular, consider a dataset {y.) with = 1,...,N, N, so that the starting distribution is
pxx)
1 N
Σ 8(x - yz)
(143)
At time i, the marginal distribution will be Gaussian mixture
Σ N (xxey,,, px(x.) = N
(144)
Then using the Eq. 140 above we have
8 sxx,, () V (x.)
wi(xx) xx)
0
X1
(x)X))
(145)
o
X4
Softmar(
ly: xxll)))x
o
207
The endpoint estimate of the distribution
Xt of lxx)
8o(x:)=
Σ So Softmar(f
20? Ily: xxl1?))x;
(146)
Σ wx(x)))s
61

---

Published in Transactions on Machine Learning Research (12/2024)
Thus, the endpoint estimate is weighted average of training data, with the softmax of negative squared distance as weights and of as temperature parameter.
62

---

Set up Assume we have set of data points { = I...N, their mean and covariance are defined as Published in Transactions on Machine Learning Research (12/2024)
D.8 Arguments of the approximation of score field of Gaussian and point cloud
In this section, we will argue from theoretical perspective, the score field of bounded point cloud is equivalent to a Gaussian with matching mean and covariance, when the noise level is high and when the query point is far from a bounded point cloud. We are going to use a technique inspired by multi-pole expansion in electrodynamics Griffiths (2005).
" = N Σxi
(147)
Σ N Evy!
mp
(148)
these are the first moment and the second central moment of the distribution. Let us assume this point cloud is bounded by a sphere of radius r around μ,, i.. ||y: - All S r. Since Σ; |ly - All = Ntrr, this bounds the covariance spectrum trΣ < r². The Gaussian distribution with matching first two moments is M(,, Σ). We are going to show that when the noise level a and ||x اهll are both much larger than the standard deviation of the distribution, the score at noise level a of the original dataset (point cloud) is equivalent to the score of this Gaussian distribution.
Score expansion for Gaussian distribution Consider the Gaussian distribution, noise level 0, the distribution is N(,, E of.).. Then its score be written
log p(: 0) + (m (x)
(149)
= -1 -
(150)
in which the
oo = diag
(151)
d + 02
diag
(152)
0²
0² + Av)
X2
2
= diag
+
(153)
0²
04
0+ + Ax)
13 1
02
A
A2
08
08
1
(154)
A
A
(155)
where the residual term A is the 2nd order term of X/o2, A ~ (2/02)2.
Using this expansion, the score field can be expressed as
log (ogp(x; = 0² 0
[T UAUT
+ +UAUT (--x) ( x)
(156)
(1
x)
1
E(p x) + 1
UAUT (p- -x)
(157)
0
x)
J
1
2(p x) UAUUT (n-
σ
x)
5 - UA x)
1
(158)
0 x)
1
E(p x) ot
1 xx)
1タ g8
1
(1 x) +..
(159)
From this result, we can express the score field of a Gaussian distribution by a series of terms: the first-order term is the isotropic attraction to the mean, the second-order term is the anisotropic term conditioned by the
63

---

Published in Transactions on Machine Learning Research (12/2024)
covariance matrix; the higher-order terms increasingly anisotropic with higher power of the covariance matrix, i.e. larger condition number. Note that this series is only convergent when o² > Amaz. Note also that, this series is always linear with respect to x, and it doesn't contain any quadratic term of x.
Score expansion for delta point distribution Consider the data points, at noise level o, the noised distribution is Gaussian mixture with N components, oxx Nx: I²I). As we have derived above, the score of a Gaussian mixture with identical variance at each mode can be expressed as
V log (x:
= 0
Σ
(x)xx -x)
(160)
Wi wi(x))s (x(x)y;= x)
(161)
σ²
(1
x) + a²
1
Σ
wx(x)(y: 1))
(162)
Where the weighting function is
exp
20 1 || yil13)
(x(x)
(163)
Ej exp
2 ||x
y;
Here we rewrite all the distances using the distributional mean as a reference
||x y=ll2
||x μr + μ - ||x 212 + 11 112 - 11) yi)
(164)
(165)
Multiply the same term exp 1 ||x - ul1) at numerator and denominator, we get
exp (-
1 (1l ys11 + 2(x - 14) (1-y))))
we wx(x)
2
(166)
Σn exp
2 (11 131 + 2(x - 14) (M - -y,)))
Order of terms
Note the identities
N
Σ( (u y,)
(167)
N
|p y112 = Ntr2
(168)
E Ey-(y.)) 111e -y11 tr
(169)
So [1--yyllž is on the order of trA.
The query points are sampled from the noised distribution, which convolves the data points cloud with an isotropic Gaussian, i.e. x ~ 1/N N(y, This is approximately M(u,o²1), so when a is large, ||x 12 on the order do?.
When we assume the query point is sampled from Gaussian with width o, we can estimate the norm of cross term.
2 -o |p y.ll Ex - w) y.)ll = /
(170)
Thus, the cross term (x- (μ- ys) is the order of a avtrA.
64

---

Published in Transactions on Machine Learning Research (12/2024)
When the noise scale > VtrA, the cross-term (x u) yz) dominates I| yy11 1 In this case, when we expand the weighting function wy by orders of 3. the cross term is one order higher. Thus, on the 1st order of = we get
1+ (x-x) p))
(x(x)
1+ (x -) 1)
1 (x- (y.(y)-1)
N + ,2 (x -1) Σ) (y, -1)
N [ गz (x my - n))
Using this approximation, the GMM score expansion has the same first-two terms as the Gaussian expansion
V (o;g((x;))=
1
(n
0² Σ w. (x) 14)
1
0²
(1 x) +
Σ
(x ,y) - ())=-))((x; 1))
σ N
1
x) +
E(x pt)
0f
65


Reviews:
[{"review_id": "iB3cWboYVz", "reviewer": "Reviewer_Rrvc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nThe experiments are quite comprehensive and address quite a large number of facets of this phenomenon. Beyond the initial finding that the score is approximately linear for much of the reverse process, they consider other analytical approximations (e.g. the \"exact\" score of the empirical distribution, Gaussian mixture approximation), they study ways in which failure of the Gaussian approximation can be corrected for with a suitable mixture model approximation (by modulating # components, rank of components, etc.), and they study the training dynamics of denoising networks and provide useful visualizations of the evolution of the score estimate over the course of training.\n\n### Weaknesses\nIt's unclear how surprising the Gaussian approximation finding is. The \"unreasonable effectiveness\" aspect of their claim largely stems from the finding that the approximation is good even at noise scales where the distribution is not statistically close to Gaussian, because there are eigendirections of the covariance along which the distribution has not yet mixed. But this would happen even if the data were a product of an arbitrary $(n-1)$-dimensional isotropic distribution and a high variance 1D Gaussian: even if the one bad eigendirection takes much longer to mix, the $(n-1)$-dimensional part mixes much earlier and an overwhelming fraction of the variance is already explained by that part. The authors do consider the mean eigenvalue in their experiments, but at the scales considered in this paper, this is considered to be close to the end of the reverse process, e.g. even under the zoomed-in plots in Figure 6.\n\nAdditionally, the finding that the Gaussian model predicts low-frequency features of the samples is perhaps not that surprising as we expect that matching the mean and covariance of any natural image distribution with a Gaussian will result in images that look very roughly like true images. The fact that the trajectories of the true reverse process and Gaussian reverse process is arguably also not that surprising: by Girsanov's theorem we know that score error does not compound exponentially.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper puts forth the claim that in diffusion models, a large portion of the reverse process can be replaced by its Gaussian approximation, whereby the score function at a given noise scale is approximated by the (linear) score function of the Gaussian with mean and covariance matching the noisy data. The paper's contributions towards this claim are the following empirical findings on real image datasets:\n1) For most of the reverse process, the Gaussian score explains a large fraction of the posterior variance, and the reverse process with Gaussian score stays close to the true reverse process. Moreover, the resulting samples retain low-frequency features of the true generated images\n2) Near the end of the reverse process, the true score is somewhat better approximated by a mixture of low-dimensional Gaussians\n3) Neural networks trained on the score matching objective initially seem to converge towards the Gaussian score, but at lower noise scales, at some point they start to deviate. This suggests that in early phases of training, the network is learning low-frequency aspects of the distribution.\n4) They use finding 1) to \"shortcut\" the reverse process and reduce the number of NFEs needed to sample compared to baselines like Karras et al. '22", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I would be interested to see how the score error achieved by the Gaussian score looks at the noise scale corresponding to sqrt(*median* eigenvalue)\n- The accelerated sampling results are somewhat far from current state of the art as the baseline of EDM is a bit dated, but this is maybe not a concern because the teleportation trick might be a useful orthogonal improvement on top of state of the art methods? For instance, instead of speeding up inference for diffusions, can this be used to speed up training for consistency models?\n- It would be helpful to include some images generated by using Gaussian score for most of the reverse process and then the best possible Gaussian mixture near the end of the reverse process, just to compare against Figure 8.", "broader_impact_concerns": "There are some broader impact aspects that are inherent to any generative modeling paper, but I'm not so concerned about this here as they are actually working towards more transparent interpretations of the dynamics of the reverse process.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0IJcruJnVx", "reviewer": "Reviewer_n5h4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths :The authors apply their findings on the algorithm to speed up the sampling process and evaluate it on several datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work provides a theoretical analysis of generative modeling by considering the associated ODE. They discuss both the high noise regime and the low noise regime for the approximation of real diffusion models. They propose methods to accelerate sampling process.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "If possible, include more baselines for the sampling speed comparison with prior work.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hv9DxCDugk", "reviewer": "Reviewer_ii8C", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is clearly written and of interest to the TMLR audience.\n* The proof that the score function of the (noise-corrupted) point cloud is dominated by the score function of the Gaussian model is clear and easy to follow.\n* Figure 1 and 6 does a great job of showing the theoretical results that the score field of real diffusion models is dominated by its Gaussian approximation. Figure 14 does a great job of showing the effect or teleportation and how skipping K steps by using the Gaussian approximation has little impact on the final sample.\n\nWeaknesses:\n* Several term definitions could be included in Sec. 2 so that the introduction is clear to readers without an understanding of diffusion. For example, unless I missed them, I do not think that $\\dot{\\sigma}$, $t$, and $\\mathbf{y}$ are defined.\n* The biggest weakness of the current work is the lack of standard deviations and comparison to other works in Tab. 1. It is unclear how good the improved sampling speed and maintained performance are without a comparison to current state-of-the-art methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper evaluates diffusion models with a score consisting of tractable distributions (Gaussian and Gaussian mixtures). They analyze the Gaussian score model, including characterizing the properties of its sampling trajectories by exactly solving the associated probability flow ordinary differential equation (PF-ODE). They theoretically and empirically support their claim that, in the high-noise regime, the score field of real diffusion models is dominated by its Gaussian approximation by proving that the score function of the (noise-corrupted) point cloud is dominated by the score function of the Gaussian model. They use these findings to speed up image sampling by 15-30% by skipping the initial phase (teleportation).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Consider rearranging terms in Eq. 1 so that the marginal likelihood is clearer\n\n\\begin{align}\np(\\mathbf{x} | \\sigma) &= \\int p(\\mathbf{x} | \\mathbf{x}’, \\sigma) p(\\mathbf{x}’) d\\mathbf{x} \\newline\n&= \\int \\mathcal{N}(\\mathbf{x} | \\mathbf{x}’, \\sigma^2 \\mathbf{I}) p(\\mathbf{x}’) d\\mathbf{x}.\n\\end{align}", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, the proposed analysis of the learned score model based on the analytical Gaussian score seems to be reasonable and supported by empirical validation on various datasets. Also, most concerns raised by the reviewers are sufficiently addressed in the revision with better description and additional experiments. Therefore, I would recommend the paper to be accepted.

More explanations, better descriptions and corrections, and additional experimental results in the replies need to be included in the final version.
Decision: Accept (poster)