Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
Authors: Emile Pierret
Bruno Galerne
Venue: ICML 2025 poster (2025)
Keywords: Convergence of diffusion models
Gaussian processes
Wasserstein distance
stochastic differential equations
discretization schemes
Primary area: theory->probabilistic_methods
TL;DR: A theoretical study of the convergence of diffusion models for Gaussian distributions.

Abstract: Diffusion or score-based models recently showed high performance in image generation.
They rely on a forward and a backward stochastic differential equations (SDE). The sampling of a data distribution is achieved by numerically solving the backward SDE or its associated flow ODE.
Studying the convergence of these models necessitates to control four different types of error: the initialization error, the truncation error, the discretization error and the score approximation.
In this paper, we theoretically study the behavior of diffusion models and their numerical implementation when the data distribution is Gaussian.
Our first contribution is to derive the analytical solutions of the backward SDE and the probability flow ODE and to prove that these solutions and their discretizations are all Gaussian processes.
Our second contribution is to compute the exact Wasserstein errors between the target and the numerically sampled distributions for any numerical scheme.
This allows us to monitor convergence directly in the data space, while experimental works limit their empirical analysis to Inception features.
An implementation of our code is available online.

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
Emile Pierret
Bruno Galerne 12
Abstract
SDE progressively transforms the initial data distribution by adding more and more noise time progresses. Then, the reversal of this process, called backward SDE, allows us to approximately sample the data distribution starting from Gaussian white noise. Moreover, the SDE is associated with an Ordinary Differential Equation (ODE) called the probability flow (Song et al., 2021b). This flow preserves the same marginal distributions as the backward SDE and provides another way sample the score-based generative model.
Diffusion or score-based models recently showed high performance in image generation. They rely a forward and a backward stochastic differ- ential equations (SDE). The sampling of data distribution is achieved by numerically solving the backward SDE or its associated flow ODE. Studying the convergence of these models neces- sitates to control four different types of error: the initialization error, the truncation error, the dis- cretization error and the score approximation. In this paper, we theoretically study the behavior of diffusion models and their numerical implementa- tion when the data distribution is Gaussian. Our first contribution is to derive the analytical solu- tions of the backward SDE and the probability flow ODE and to prove that these solutions and their discretizations are all Gaussian processes. Our second contribution is to compute the exact Wasserstein errors between the target and the nu- merically sampled distributions for any numerical scheme. This allows us to monitor convergence di- rectly in the data space, while experimental works limit their empirical analysis to Inception features. An implementation of our code is available on- line.
An important issue about diffusion models is the theoretical guarantees of convergence of the model: How close to the data distribution is the generated distribution? There are four main sources of errors study to derive theoretical guarantees for diffusion models: (a) the initialization error is induced when approximating the marginal distribution at the end of the forward process by a standard Gaussian distribution. (b) The discretization error comes from the resolution of the SDE or the ODE by a numerical method. (c) The truncation error occurs because the backward time integration stopped at a small time > 0 to avoid numer- ical instabilities due ill-defined score function near the origin. (d) The score approximation error accounts for the mismatch between the ideal score function and the one given by the network trained using denoising score-matching.
Despite these numerous sources of errors, a lot of numerical and theoretical research has been conducted to assess the generative capacity of diffusion models. Several articles (Franzese et al., 2023; Karras et al., 2022) provide strong experimental studies for the choices of sampling parameters. On the theoretical side, several works derive upper bounds on the I-Wasserstein or TV distance between the data and the model distributions by making assumptions on the L error between the ideal and learned score functions and on the compacity of the support of the data (Chen al., 2023b; Lee al., 2024; De Bortoli et al., 2021: Chen al., 2023c; Lee et al., 2022; Benton al., 2024), eventually under an additional manifold assumption (De Bortoli, 2022; Wen- liang & Moran, 2022; Chen et al., 2023a). Yet, on one hand, to the best of our knowledge, the derived theoretical bounds mostly rely on worst case scenario and are not tight enough to explain the practical efficiency of diffusion models. On the other hand, numerical considerations mostly rely on In- ception feature distributions through the FID metric (Heusel
1. Introduction
Over the last five years, diffusion models have proven to be a highly efficient and reliable framework for generative modeling (Song& & Ermon, 2019; Ho et al., 2020; Song et al., a0b];;b; Dhariwal & Nichol, 2021; Karras et al., discrete process, Denoising
2022). First introduced Diffusion Probabilistic Models (DDPM) (Ho et al., 2020) can be studied as a reversal of continuous Stochastic Dif- ferential Equation (SDF) (Song et al., 2021b). A forward
Universitéé d' d'Oréé,,s, Université de Tours, CNRS, IDP, UMR 7013, Orléans, France 'Institut universitaire de France (IUF). Cor- respondence to: Emile Pierret <emile <emile.pierretee univ-orleans.frz. Bruno Galeme <brumo, Caamoo.gllrmeeee @ uuiv-orlcanss.fe
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
al., 2017).
the corresponding processes, yielding ablation study for comparing the influence of each error type on several sam- pling schemes. In Section 5, we study numerically special case of Gaussian distribution for texture synthesis in order to evaluate the influence of the score approximation error occurring with a standard network architecture. Finally, we address discussion and limitations of our framework in Section
Ideally, given data distribution of interest, one would like to have an adapted estimation of the discrepancy between the data and the diffusion model samples, thus enabling adaptive hyperparameter selection for the sampling proce- dure. As first step towards reaching this goal, the present work we study diffusion models applied Gaussian data distributions. While this setting has priori no practical in- terest, since simulating Gaussian variates does not require diffusion model, it provides a large parametric family of dis- tributions for which the errors involved in diffusion model sampling can be completely understood.
2. Preliminaries: Score-Based Models through Diffusion SDEs
This preliminary section follows the seminal work of Song et al. (Song et al., 2021b) and introduces specific notation to differentiate the exact backward process and the genera- tive backward process obtained when starting from a white noise. Given a target distribution Pdata over It the forward diffusion process is the following variance preserving SDE
When restricting the data distribution to be Gaussian, the re- sulting score function a simple linear operator. Exploiting this specificity allows us make the following contribu- tions under the assumption that the data distribution is Gaussian:
We give the exact solutions for both the backward SDE and the probability flow ODE.
dx, = -Bxx,dt + 28,dwr, 0 sttss Zo ~ Pdata (1)
where (we) is a d-dimensional Brownian motion and 3 positive weight function. The distribution Pdata. is noised progressively and the function 3 is the variance of the added noise by time unit. We denote by the density of (x,) for > O since Pdata can be supported on a lower-dimensional manifold (De Bortoli, 2022). The SDE is designed so that pr is close to the Gaussian standard distribution that we denote No in whole paper. Under some assumptions on the distribution Pdata (Pardoux, 1986), the backward process (xT-U)sisT verifies the backward SDE
We fully describe the Gaussian processes that occur when using classical sampling discretization schemes.
We derive exact 2- Wassersttii errors for the corre- sponding sample distributions and are able to assess the influence of each error type on these errors, as illustrated by Figure 1.
Our theoretical study allows for an analytical evaluation of any numerical sampler, either stochastic or deterministic. In particular, it confirms the strength of best practice scheme such as Heun's method for the ODE flow (Karras et al., 2022). Our source code is available online and can be applied to any Gaussian data distribution of interest and gives insight to calibrate parameters of a diffusion sampling algorithm, e.g. by straightforwardly generalizing our study to higher order linear numerical schemes.
dy. = Br + 2V logpr- + 223r-
0 0SI<T, Yo ~ pr-
(2)
The objective is now solve this reverse equation in order to sample yr Pdata- However, the distribution pr is in general not known, and image' generation is achieved by sampling
While our theoretical analysis relies on an exactly known score function, we conduct additional experiments assess the impact of the score approximation error. Surprisingly, in the context of texture synthesis, we show that with score neural network trained for modeling a specific Gaussian microtexture, a stochastic Marler-aruuymaa sampler is more faithful to the data distribution than Heun's 's method, thus highlighting the importance of the score approximation ervor in practical situations.
dū. = Br + 2V 2V logpr ( + /28T-
0 0 Do ~ No
(3)
Note that approximating pr by No for the initialization Yo implies that the solution of the SDE of Equation (3) does not exactly follow the target distribution pdata at final time. An alternative way to approximately sample Paata is to use that every diffusion process associated with deterministic process whose trajectories share the same marginal probability densities (pt) the SDE (Song et al., 2021b). The deterministic process associated with
Plan of the paper: First, recall in Section 2 the con- tinuous framework for SDE-based diffusion models. Sec- tion 3 presents our main theoretical results detailing the ex- act backward SDE and probability flow ODE solutions when supposing the data distribution to be Gaussian. Section 4 gives explicit Wasserstein error formulas when sampling
AAthhough we may refer data as images, our analysis fully general and applies to any vector-valued diffusion model.
2

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
10a
OGE
SGE
Edior
Heun
DDIN RS4
10-1
10
W. WVG.P)
W-C, .Po
10-2
10"2
10-3
SDE
-36- BM
-36
DDFM
10
0
0.2
0.4
0.6
0.8
T=1
10
10-2
Time
Truncation time
(a) Initialization error along the integration time
(b) Truncation error for different truncation time &
Figure I: Wasserstein errors for the diffusion models associated with the CIFAR-10 Gaussian. Left: Evolution of the Wasserstein distance between Pt and the distributions associated with the continuous SDE, the continuous flow ODE and four discrete sampling schemes with standard No initialization, either stochastic (Fuler-Maruyann Probabilistic Models (DDPM)) deterministic (Fuler, Heun and Runge-Kutta Kutta 4 (RK4)). While the continuous SDE is less sensitive than the continuous ODE (as proved by Proposition 4), initialization error impacts all discrete schemes with comparable order magnitude. Heun and RK4 methods have the lowest error and are very close to the theoretical ODE, except for the last step (which is not represented because unstable) that is usually discarded when using time truncation. Right: Wasserstein errors due to time truncation for various truncation times Using time truncation increases the error for all the methods except Heun's scheme and RK4 due instability near the origin. Interestingly, for the standard practice truncation time 10-2 numerical schemes have a comparable error close their continuous counterparts.
(EM), Exponential Integrator (EI) and Diffusion Denoising
Equation (2) is
full generality obtained by applying the variation of con- stants (see the proof in Appendix B.1). This resolution also provides an ODE verified by the covariance matrix of æı, that we denote Σ: Cov(
dx; = -B, + V logpt (xt]]dt,
0 < T, wo
Pdata
(4)
This ODE can be solved in reverse-time to sample 2g from Fr Pr- Given (21)oSiST solution of Equation (4),
Proposition 1 (Solution of the forward SDE). The strong solution of Equation (1) can be written as:
(toss) is solution of
at = Bi ne- 0S1ST,
dy, = - y + Vlggr- dl, 05 < (5)
where B, = S0 Byds and n. c So c, /28,du is Gaussian process independent of xo whose covariance Consequently. the covariance matrix
Again, in practice the ODE which is considered to achieve image generation
matrix is
dy. Br + V (O.)).
E, of x3 is
0 5 < T, T, ~ No.
(6)
Σ, = -- c-B.)
(8)
where Pr is replaced by No. As consequence of this ap- proximation, the property of conservation of the marginals (pe)sEeT does not occur. We denote by (#t)SssT re- spectively ("t) the marginals of (WeosssT and (Ur)osisr and Pt -r- Pi -r- the marginals of (Ur-t)osesr and (Dr-)) such that P. and Pt are approximations of pt.
where Σ the covariance matrix of Øo ~ Pdsta- Futher- more, Σ, invertible for t > and verifies the matrix- valued ODE
dE,
OKEST.
(9)
For a general data distribution Pdata solving the backward SDE is infeasible, the main reason being that the expression of the score function to integrate is unknown. To circumvent this obstacle, we now suppose that the data distribution is Gaussian.
3. Exact SDE and ODE Solutions
Our approach relies on deriving explicit solutions to the various SDE and ODE. We begin with the forward SDE in
3

---

Diffusson models for Gaussian distributions: Exact solutions and Wasserstein errors
Assumption 1 (Gaussian assumption). Pästa is a centered Gaussian distribution N((0, Σ)
Remark Here we must highlight subtle issue: Whatever
the initial distribution of yo is, the ODE solution consists in applying the OT map between pT and PT at time t. If Yo follows pr. then y ~ Pt at each time 0 S t S T. But since in practice one cannot truly sample pr and uses Yu ~ No instead, the resulting flow is not an OT flow (even though it involves an OT mapping) and the distribution of yr differs from Pdata.
Note that E may be non-invertible and thus Pdata supported on a strict subspace of Rd a special case of manifold hypoth- esis. Consequently, the matrix Σ, is in general only invert- ible for t > O. Under Gaussian assumption, (xx) is Gaus- sian process with marginal distributions Pt = (0, 2,) and consequently the score is the linear function
For the sake of completeness, we extend Proposition 2 and Proposition 3 to the non centered case Pdata = (Me E) in Appendix C.
V (x = - Σ 1 x. 0 T.
(10)
Note that the linearity of the diffusion score characterizes Gaussian distributions as detailed by Proposition 5 in Ap- pendix A.
Links with related work. Some parts of the previous propositions have been stated in previous work.
The cornerstone of our work is that under Gaussian assump- tion we can derive exact solution of the backward SDE, without supposing that the initial condition is Gaussian. Proposition 2 (Solution of the backward SDE under Gaus- sian assumption). For Pdata = (0, E), the strong solution to the SDE of Equation (2)
Equation (7) of Proposition I given without proof in (Gao & Zhu, 2024), the variance ODE, that we generalize here to the full covariance matrix (Equation (9)), is given in (Song et al., 2021b), (Särkkä & Solin, 2019, Equation 6.20)), and the score expression under Gaussian assumption is reported in several recent references (Albergo et al., 2023; Zach et al., 2024; 2023; Shah et al., 2023).
dy.- = - + 2 y ( + 123r-rdus
(11)
with yo following any initial distribution can be wrillen
To the best of our knowledge Proposition 2 is new and is the cornerstone for our analytical and numerical study.
ye = =e-(Br-Br-)) Er Yu + E. 0 ST (12)
where& &, is a Gaussian process with covariance matrix
The relation between OT and probability flow ODE has been discussed in (Lavenant & Santambrogio, 2022; Khrulkov et al., 2023). (Lavenant& & Santambrogio, 2022) show that, in general, the flow ODE solution is not an OT between Pdata and No at infinite time T + +o0, thus contradicting a conjecture of (Khrulkov et al., 2023). Yet, they briefly discuss the Gaussian case as special case for which the conjecture is valid. Indeed, (Khrulkov et al., 2023) derive the solution of the flow ODE under Gaussian assumption at infinite time horizon (Appendix B (Khrulkov et al., 2023)). More recently, an expression of the solution of the flow ODE relying on the eigendecooposition of the covariance matrix of the data in Gaussian case is given in (Wang& & Vastola, 2023) assuming Yo No. None of these works discuss the mismatch between the OT map and the initialization of yo- Our Proposition 3 highlights that the generated process is not an OT flow due to the initialization error.
Cov(Ei) = ET-r- - - BB ET
(13)
Finally. if (Co((y)) and E commute,
Cov(y,) -t + B ET² [Cov(yo) - Erl.
(14)
While not as straightforward as the forward case, the proof also relies on applying the variation of constants and given in Appendix B.2. Note that if Yo is correctly initialized at PT. yT Pt at each time 0- < S T. As shown by the following proposition (proved in Appendix B.3), the flow ODE also has an explicit solution under Gaussian assumption which is related to optimal transport (OT).
Proposition 3 (Solution of the ODE probability flow under Gaussian assumption). For Paata (0, E), the solution to the reverse-time probability flow ODE of Equation (5) dy. = [3T- + Br- V log logpr dt, 0st<2 T
4. Exact Wasserstein Errors
(15)
The specificity of the Gaussian case allows us to study pre- cisely the diffferent types of error with the expression of the explicit solution of the backward SDE. In what follows, we designate by Wasserstein distance the 2- Wasssersttiin distance which is known in closed forms when applied to Gaussian distributions (Dowson & Landau, 1982). For two centered Gaussians (00 Σn) and (O, Σ2) such that the matrices Σ, and Σg are simultaneously diagonalizable with
for any yo! is
1/2 yi - -
0 0SIST,
(16)
which is the application of the OT map between pr and pr-t to the initial condition yo- Consequently, if Cov( and Σ commute,
Cov(y,) = ET : 0 OStST. T.
(17)
4

---

Diffusson models for Gaussian distributions: Exact solutions and Wasserstein errors
respective eigenvalues (Ar,))siga y (Ai2)
sample from the SDE of Equation (3), The Discrete De- noising Diffusion Probabilistic Model (DDPM) (Ho et al., 2020) can be interpreted as discretization of the backward SDE as detailed in (Song et al., 2021b, Appendix E). Eu- ler method, Heun's scheme and Runge- 4 (RK4) are Runge-Kutta methods with respective orders 1 1,2,4. Heun's 's scheme is recommended in (Karras et al., 2022) to model the ODE of Equation (6). Under the Gaussian assumption, the discretized processes obtained using these six schemes remain Gaussian processes, and the eigenvalues of their covariance matrix can be computed numerically and recur- sively in order to evaluate the Wasserstein distance. Indeed, under the Gaussian assumption, the score is a linear operator and the discrete schemes lead to linear operations described in Table in Appendix E. Then, a Gaussian initialization for yo provides a sequence of centered Gaussian processes (yk (yA")) ")x and if yo follows pr or No. the covariance matrix Cov(yk (") admits the same eigenvectors as Σ and we can use Equation (18) to compute Wasserstein distances. Let us illustrate the computation of the eigenvalues with the EM scheme. Denoting (As,)) the eigenvalues of E, and (A "isi<d the eigenvalues of the covariance ma- trix of the Fuler-Maruyama discretization of the SDE at the kth step, S k I N -V-1 1, the relation verified by these eigenvalues is
(1, = Σ (VAc, - 21,2) 1sisd
(18)
as used in (Ferradans et al., 2013). In the literature, the qual- ity of the diffusion models measured with FID (Heusel et al., 2017) which is the W2-error between Gaussians fitted to the Inception features (Szegedy et al., 2016) of two dis- crete datasets. Here we use the Wy-errors directly in data space, which is more informative and allows us to provide theoretical W2-errors. To illustrate our theoretical results, we consider the CIFAR-IO Gaussian distribution, that is, the Gaussian distribution such that Σ is the empirical covari- ance of the CIFAR-10 10 dataset. As shown in Appendix D, images produced by this model are not interesting due lack of structure, but the corresponding covariance has the advantage of reflecting the complexity of real data.
The initialization error. As discussed in Sections 2 and 3, the marginals of both generative processes ÿ and ÿ fol- lowing respectively Equation (6) and Equation (3) slightly differs from Pt due to their common white noise initial condition. This implies an error that we call the initializa- tion error. The distance between (Pe)oSesT. (Pe)osisT and (pt)osIST can be explicitly studied in the Gaussian case with the following proposition (proved in Appendix B.4).
A.EM A 1,+++1 -(1 + 4,3r 12 (1 21T--1 i,k
2 n.2 ))²4
Proposition 4 (Marginals of the generative processes un- der Gaussian assumption). Under Gaussian assumption, (Uo)sssTT and (Ut)oSesT are Gaussian processes. At each time I, Pr is the Gaussian distribution (0, Exe with En = Ex + e 22 B.) B.) E 'ET - I) and Pi is the Gaussian distribution N(0, Ex) with Σ, = > E. For all 0 S 1 S T, the three covariance matrices Σ,, Eg and En share the same range. Furthermore, for all 0 S t S T,
+ 22,,8r two s
d,0 < k < N 2 (20)
{ >A.T if if ~ pT detailed computations for EM and formulas for other schemes are presented in Appendix F. For each scheme, we recursively compute the eigenvalues at each time dis- cretization and present the observed Wasserstein distance in Figure 1.(a). The schemes are compared at fixed Number of score Function Evaluation (NFE). We can observe that RK4 and Heun's methods provide the lower Wasserstein distance, followed by EM, EI and the Euler scheme. Note that the discrete schemes does not preserve the range of the covariance matrix, contrary to the continuous formulas. This explains the fact that the Wasserstein distance increases at the final step.
with initialization A AM
1
if yf ~ ~o
More
W2 S (P)
(19)
which shows that at each time 0 S t S T (and in particular for t = 0 which corresponds to the desired outputs of the sampler), the SDE sampler is a better sampler than the ODE sampler when the exact score is known.
In practice the initialization error for the SDE and ODE samplers may vary by several orders of magnitude, as shown for the CIFAR- 10 example in Figure I.(a) (solid lines) which illustrates Equation (19).
The truncation error. As discussed in (Song et al., 2021 it is preferable to study the backward process on [s, T] instead of [o, 2] because the score a priori not de- fined for = O, which occurs in our case if Σ is not invert- ible. This approximation is called the truncation error. As consequence, even without initialization error, the backward process leads to Pe and not Po- Under Gaussian assumption, it is possible to explicit this error with the expression given in Proposition 3 and as done in Figure 1.(b) for both con- tinuous and numerical solutions. For the standard practice
The discretization error. The implementation of the SDE and the ODE implies choosing a discrete numerical scheme. We propose to study three different schemes for the SDE and the ODE, presented in Table 3 in Appendix E. The clas- sical Euler- Maruyama (EM) is used in (Song et al., 2021b) and the exponential integrator (EI) in (De Bortoli, 2022) to
5

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
NFF = 500
NFE= = 1000
except for strong decrease for À 1. Besides, as proved in the proof of Proposition 4 (see Appendix B.4), the error for the SDE is always lower than the error for the ODE and we can observe how tight is Equation (19). Unfortunately, once discretized, the stochastic schemes are not as good as the continuous solution. We can note that all schemes exhibit a strong decrease for À = 1, except EM and EI. The other peaks depend on the choice of the parameterization of 3. The EI scheme is the most stable across the range of eigenvalues, but in the end, it is generally more costly than the others in terms of Wasserstein error. DDPM seems to be the best choice of stochastic scheme for low eigen- values but fails for higher ones, which is largely penalized in our Gaussian CIFAR-10 example (see Figure 1). With- out truncation time, Heun's method and RK4 fail for low eigenvalues because Σ is not stably invertible. However, as seen in Figure 2.(b), with a truncation time 10-3 they are very close to the continuous ODE solution. This shows that for any Gaussian distribution, these methods introduce almost no additional discretization error. Due to its sim- plicity, Heun's 's method is the preferred choice in practice. Our code allows for the evaluation of any covariance matrix and the computation of Figure and Table (provided the eigenvalues can be computed, see the supplementary).
Pr
N6
pr
No
EM
E E = 10 3
0.32
0.32
0.16
0.16
0.40
0.40
0.27
0.27
EI
E 8 10 3 e e 10 3
0.30
0.30
0.16
0.16
0.41
0.41
0.29
0.29
0.47
0.47
0.23
0.23
DDPM
0.53
0.53
0.32
0.32
Euler
E ===0 E 10
0.20
0.26
0.10
0.17
0.27
0.32
0.21
0.25
& = 0
Hcun
e 10
0.16
0.18
0.17
0.19
8 0
RK4
g 10
0.15
0.17
0.17
0.19
Table Ablation study Wasserstein errors for the CIFAR- 10 Gaussian. This table is extracted from Table 5 in Appendix G. For a given discretization scheme, the table presents the Wasser- stein distance associated with the truncation error for different values of e. The columns pr and No show the influence of the initialization error. The columns provide the Wasserstein errors for each scheme for a given number of score function evaluation NFE.
truncation time 10-3 (Song et al., 2021b; Karras al., 2022), all numerical schemes have an error close to the cor- responding continuous solution. Using lower e value is only relevant for the continuous SDE solution.
5. Numerical Study of the Score Approximation
Ablation study. We propose in Table an ablation study to monitor the magnitude of each error and their accumu- lation for various sampling schemes for the CIFAR- 10 ex- ample. In accordance with Proposition 4, the initialization error influences the ODE schemes, while the SDE schemes are not affected. Schemes having sufficient number of steps are not sensitive to the truncation error for & < 10-3, except Heun's scheme and RK4, which are unstable near to origin. The discretization error is the more important ap- proximation but becomes very low for a sufficient number of steps for stochastic schemes. In a realistic setting, where Pr is unknown, and with truncation time e, the lower Wasserstein error is provided by RK4 and Heun's 's method with 500 NFE, with the classical choice E = 10 As in (Karras et al., 2022), our conclusions lead to the choice of Heun's ' scheme as the go-to method.
So far our theoretical and numerical study has been con- ducted under the hypothesis that the score function is known, thus discarding the evaluation of the score approximation. In practice, for general data distribution, the score function is parameterized by a neural network trained using denoising score-mas mote-mat This learned score function not perfect and while theoretical studies assume the network be close to the theoretical one (with uniform or adaptative bounds, see the discussion in (De Bortoli, 2022)), such an hypothesis is hard to check in practice, especially in our non compact setting. Thus, we propose in this section to train a diffusion models on a Gaussian distribution and evaluate numerically the impact of the score approximation.
The Gaussian ADSN distribution for microtextures. So far our running example was the CIFAR- 10 Gaussian but we will now turn to another example that produces visually interesting images, namely Gaussian microtextures. We consider the asymptotic discrete spot noise (ADSN) dis- tribution (Galerne et al., 2011) associated with an RGB texture u E 1x3MMN which is defined as the stationary Gaussian distribution whose covariance equals the autocor- relation of u. More precisely, this distribution is sampled using convolution with a white Gaussian noise (Galerne et al., 2011): Denoting m E B the channelwise mean of u and to VMN (ue me), S c S 3, its asso-
Influence of cigenvalues. The above observations and conclusions are observed on the CIFAR-10 Gaussian. How- ever, in general, they depend on the eigenvalues of the co- variance matrix Σ. Indeed, as seen in Equation (18), the Wasserstein distance is separable and each eigenvalue con- tributes to increase it. In Figure 2, we evaluate the contribu- tion of each eigenvalue by plotting À + VV - v scheme| for each scheme. Figure 2.(a) demonstrates that for con- tinuous equations, the error increases with the eigenvalues,
6

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
10a1
10a4
10
10-3
10'5
1015
I moame -15
Iv Vymmmme -V5
10
10
10
10-41
10-41
SDE 008
--.
EM
DDPM
SDE OD0
EM
EI
DDPN
Heun
Fuler
Heun
10-41
10-13
10-1
10
10
10
10-1
10
10a
(a) Initialization error at final time
(b) Truncation error at final time for 10
Figure Eigenvalue contribution the Wasserstein error. magnitude the Wasserstein error influenced by the eigenvalues the covariance of the Gaussian distribution. Left: Contribution to the Wasserstein emorr for the continuous equations and the discretization schemes with standard initialization No. Right: Same plot when using truncation time prove that the continuous SDE is always better than the continuous ODE (Proposition 4), it is not the same for the discrete schemes. With a truncation time 10-3 (see (b)), RK4 and Heun's 's method are nearly good as the continuous ODE solution for high eigenvalues, which shows they are well-adapted to any Gaussian distribution.
10 10-3 All schemes use 1000 NFE. While we
ciated texton, for No of size M X N the channel- wise convolution æ m + t E Xx follows Aw). This distribution is the Gaussian (mm, E). To deal with zero mean Gaussian, adding the mean m is con- sidered as a post-processing to visualize samples and we study (O, Σ). The matrix Σ well-known convolu- tion matrix (Ferradans al., 2013), its eigenvectors and associated cigenvalues can be computed in the Fourier do- main, as done in Appendix I.2. Σ admits the eigenvalues ACADDN = [t (c) + [ty (E) ft (E)) M x N and 0 with multiplicity 2MN and we can conduct the same analysis as before (see Appendix H). To evaluate if set of N samples sampled images is close the ADSN distribution Pdata- we evaluate problem- sppecffic empirical Wasser- stein distance: Supposing that the Namplee drawn from Gaussian distribution pmm (0 I') such that T admits the same eigenvectors as E, we compute
Original TM
AUSN
W H mp Pdata) )
Σ
S Cemp
ADSN
SEMON
po
+ A
E.mp.
+ 25 S.emp.
(21)
where (X5.emp ) are estimators of the eigenval- ues of T given in Appendix I.3.
Figure Texture samples generated with the learned score. First row: original image u. Second row: three samples of ADSN(u) Third and fourth row: Samples generated with the leamed score with EM and Heun's is discretization schemes. While both schemes use same learned score function. the generation with Heun's scheme can produce --disiiibiiitnoonnn samples, as seen with the third sample.
Learning the score function. We train the network using the code? associated with the paper (Song et al., 2021b).
"ode available https::/ htttps://github.... com/:/ngoccc//o//c//// song/
score_sde sctcch

---

Diffussin models for Gaussian distributions: Exact solutions and Wasserstein errors
Exact score distribution
Learned score distribution
P
W2(p-Pdaca) 4
2 (p'TF: Pdatt) &
FI TM
.Paata 4
Wath
(Pg
Pasta
FID(P' amp .Pasta)
EM
5.16
5.1630+77-5 3.7323+2FF4
0.0891=8E-4
15.6
1.02
Heun
3.73
0.0447+6F-4
56.7
19.4
Table 2: Numerical evaluation of the score approximation for a Gaussian microtexture model. For two schemes, the EM discretization of the backward SDE and Hcun's 's method associated with the flow ODE, the table shows the Wasserstein distance and FID for thcoretical and learned distributions. The thcoretical Wa value is computed with explicit formulas, as done in Table 6. The FID and empirical 22 w.r.t the theoretical distribution are computed on 25 samplings of 50K images while only one sampling of 50K images drawn for the parametric distributions (to limit computation time).
We choose the architecture of DDPM, which is a U-Net described in (Ho et al., 2020), with the parameters proposed for the dataset CelebaHQ256 to deal with the 256x 256 ADSN model associated with the top-left image of Figure 3. We use the training procedure corresponding to DDPM cont. in (Song et al., 2021b). /3 is linear from 0.05 to 10 with T 1. We train over 1.3M iterations, and we generate at each iteration new batch of ADSN samples. We implement the stochastic EM and deterministic Heun schemes replacing the exact score by its learned version with N = 1000 steps and a truncation time & = 10-3. We name EM and pg pg-un, the corresponding distributions and present samples in Figure 3. Both distributions accumulate the four error types.
6. Discussion and Limitations
The main limitation of our work is that our theoretical and numerical results are limited to Gaussian distributions. Re- sorting to diffusion models for sampling Gaussian distribu- tions is not necessary in practice, rather we use Gaussian distributions as a test case family to provide insight on dif- fusion models.
A natural extension of this work to compute error types for more complex distributions (e.g multimodal) such as Gaus- sian mixtures models (GMM). However, generalizing our results for these more complex distributions one faces three main difficulties. First, to the best of our knowledge, we are unable to derive exact solutions to the backward SDE or the flow ODE under GMM assumption, even though the score has a known analytical expression (Zach et al., 2024; 2023; Shah et al., 2023). Another key feature of this study is to evaluate exactly the Wasserstein error by using Equation (18), strongly relying on the Gaussian assump- tion. A closed-form of the Wasserstein distance between two GMMs is not known, leading to alternative distance definitions for such models (Delon & Desolneux, 2020). Furthermore, there is no guarantee that the processes ob- tained by discretizing the backward SDE follow a known distribution such as a GMM. Hence, to compare the distri- butions generated in practice with exact solutions of time continuous equations under GMM assumption, as we do for the Gaussian case, one should solve open theoretical problems.
Evaluation of the score approximation. It is not possi- ble to compute theoretically the Wasserstein distance be- tween Pdata = ADSN(u) and JM Heem due to the non- linearity of the learned score. To compute an empirical Wasserstein error between it, we use Equation (21). Let us clarify that this approximation underestimates the real Wasserstein distance since it wrongly assumes that the dis- tributions P.M, ppeu are Gaussian with a covariance matrix diagonalizable in the same basis than the covariance ma- trix Σ of ADSN(u). We complete this dedicated empirical measure with the standard FID. These metrics are reported in Table 2 where for theoretical distributions that are fast to sample we add the standard deviations computed 25 different 50k-samplings. For this Gaussian distribution, the score approximation is by far the most impactful source of error, which is in accordance with previous works (Chen et al., 2023c; De Bortoli et al., 2021). We observe that the stochastic EM sampling is more resilient to score approxi- mation than the deterministic Heun's 's scheme, resulting in -uttof-distributtin samples (Figure 3). We may explain this behavior by recalling the results of Proposition 4 that shows that SDE solutions are less sensitive to initialization errors than ODE, Indeed, adding noise at each iteration tends to mitigate the accumulated errors, and score approximation may be considered as some initialization error occurring at each step.
7. Conclusion
By restricting the analysis of diffusion models the specific case of Gaussian distributions, we were able to derive ex- act solutions for both the backward SDE and its associated probability flow ODE, We demonstrated that regarding the initialization error, the SDE sampler is more resilient than the ODE sampler for Gaussian distributions. Additionally, we characterized the discrete Gaussian processes arising when discretizing these equations. This allowed us pro- vide exact Wasserstein errors for the initialization error, the discretization error, and the truncation error as well as any
8

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
of their combinations. This theoretical analysis led con- clude that Heun's scheme is the best numerical solution, in accordance with empirical previous work (Karras al., 2022).
Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., and Zhang, A. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Repre- sentations, 2023c. Uttpst:s: /pwp/eveew...nnnn et/ pinerummii-zaee 2ll.orummnn..-ayyyygyyg22222200.
To conclude our work we conducted an empirical analysis with a learned score function using standard architecture which showed that the score approximation error may be the most important one practice and that this error has more impact the deterministic Heun's scheme than the stochastic Muler-Maruyane scheme. This suggests that as- sessing the quality of learned score functions is an important research direction for future work.
De Bortoli, V. Convergence of denoising diffusion models under the manifold hypothesis. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL ht s: ss//openrevieww net forum gR.
De Bortoli, V., Thornton, J., Heng, J., and Doucet, A. Diffu- sion schrödinger bridge with applications to score-based generative modeling. In Advances in Neural Information Processing Systems, volume 34, pp. 17695-17709, Cur- ran Associates, Inc., 2021, URL https: /ppppe/ppppaaaaaa i ps . cc htt.cc//ppppee.. /20p::pappee22111//p12 http:////hhhh///00000 adelcc201767cf8832 httpp////hhhhhh////5//e51
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Delon, J. and Desolneux, A. A wasserstein-type distance in the space of gaussian mixture models. In SIAM Journal on Imaging Sciences, pp. 936-970, 2020.
Acknowledgemmnts:: The authors acknowledge the support of the project MISTIC (ANR-19-CE400-00) and the CaSeiModOT Centre de Calcul Scientifique de la Région Centre Val de Loire for providing computer facilities.
Dhariwal, P. and Nichol, A. Diffusion models beat GANs
on image synthesis. In Advances in Neural Information Processing Systems, volume 34, pp. $780-8794. Curran Associates, Inc., 2021. URL http:: /ttpsttpp://pppr....../ http//appprrssee/ .nip
References
Albergo, M. S., Boffi, N. M., and Vanden-Eijnden, E. Stochastic interpolants: A unifying framework for flows and diffusions, 2023. URL https:tt//// httt:p:://axxivorg//// abs /303. 08797.
d8 dBa77602681af5cfaa htt.bBai....ccc---aaatatttt htmm.
Dowson, D. and Landau, B. The fréchet distance between multivariate normal distributions. Journal of Multivariate Analysis, 12(3):400455, 1982. ISSN 0047-259X. doi:
Benton, J., Bortoli, V. D., Doucet, A., and Deligiannidis, G. Nearly $d5-linear convergence bounds for diffusion models via stochastic localization. In The Twelfth Inter- national Conference on Learning Representations, 2024. URL https::tp///o.....tt. // //opnreveiie net/ forum? idd= J5njV3BBuD.
http://do.ong///////////////2229 URL e //atticcle/p1/004//25xx22222222/
Ferradans, S., Xia, G. Peyré, G., and Aujol, J.-F. Static and dynamic texture mixing using optimal transport. In Scale Space and Variational Methods in Computer Vision, 2013.
Chen, M., Huang, K., Zhao, T., and Wang, M. Score approx- imation, estimation and distribution recovery of diffusion models on low-dimensional data. In Krause, A., Brun- skill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, (eds.), Proceedings of the 40th International Confer- ence on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 4672-4712, PMLR, 23-29 Jul 2023a. URL httpst http::///ccccceeeeeeeeeee ml r. e.press/veel wttesss/v222/2 chen23e html.
Franzese, G., Rossi, S., Yang, L., Finamore, A., Rossi, D.,
Filippone, M., and Michiardi, P. How much is enough? study on diffusion times in score- score-based generative models. Entropy, 25(4), 2023. ISSN 1099-4300. doi: 10.3390/e2 5040633, URL https: : //////www wwww mdpi www.coommm///// 1099--099--- 300 0000/55/2 12/22//4// 4/333.
Chen, S., Chewi, S., Lee, H., Li, Y., Lu, J., and Salim, A. The probability flow ODE is provably fast. In Thirty- seventh Conference on Neural Information Processing Systems, 2023b. URL https:/ openreview. net / /formm? d= 6Mw.5Add.........
Galeme, B., Gousseau, Y., and Morel, J.-M. Random Phase Textures: Theory and Synthesis. IEEE Transactions on Image Processing, 20(1):257-267, 2011. doi: 10.1109/ TIP.2010.2058822.

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
Gao, X. and Zhu, L. Convergence analysis for general probability flow odes of diffusion models in wasserstein distances, ArXiv, abs/2401.17958, 2024. URL https: //api. waapp.ssmaaticcccholloooooooomee wtt//oog//uuuoo Corppoirpp:iScc. 7334658.
Pardoux, E. Grossissement d' d'une filtration retournement
du temps d' d'une diffusion. In Azéma, J. and Yor, M. (eds.), Séminaire de Probabilités XX 1984/85, pp. 48-55, Berlin, Heidelberg, 1986. Springer Berlin Heidelberg. ISBN 978-3-540-39866-88
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to a local Nash equilibrium. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vish- wanathan, S., and Garnett, R. (eds.), Advances in Newral Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL http::::ttt::ssseeet ccc naarrppp.ccc/ppeer//001////////p Teb0fefee63871360777400
Peyré, G. and Cuturi, M. Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-6); 355-607, 2019.
Särkkä, S. and Solin, A. Applied Stochastic Differen- tial Equations. Institute of Mathematical Statistics Textbooks. Cambridge University Press, 2019. ISBN 9781316510087. URL https::/t///oo http:///oooso///// google. r r/ t/books
Ho, J., Jain, A., and Abbeel, P. Denoising diffuusin proba- bilistic models. In Advances in Neural Information Pro- cessing Systems, volume 33, pp. 6840-6851. Curran Asso- ciates, Inc., 2020. URL wwt.:http:::t http::ppps////oceeeeeeeee nn urips. wrijgeeccc/apeer///000//at/tttee
Shah, K., Chen, S., and Klivans, A. Learning mixtures of gaussians using the DDPM objective. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: /ttp:///ppeeeieeeeeeeee net/ forum ?id=aig7sgdRff
html..
Song, J., Meng. C., and Ermon, Denoising diffusion implicit models. In International Conference on Learning Representations, 2021a. URL https: openrevi ew http://ouuuiiddsssiirrrrrrit
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. In Proc. NeurIPS, 2022.
Khrulkov, V., Ryzhakov, G., Chertkov, A., and Oseledets, Understanding DDPM latent codes through optimal transport. In The Eleventh International Conference on Learning Representations, 2023. URL https: //open review. net. www.moiiuriooor.... IrhAx1j41.
Song, Y. and Ermon, S. Generative modeling by estimat- ing gradients of the data distribution. In Wallach, H., Larochelle, H., Beygelzimer, A., Alché-Bue, F. d., Fox, E., and Garnett, R. (eds.), Advances in Neural Informa- tion Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: httpppss///rrrrceennggg neurip s cc httc://pper//111///////00ee 37 371a96dcd347c7799 http:/cwddddppPPeeeee--e Paper.pde
Lavenant, H. and Santambrogio, F. The flow map of the fokker-planck equation does not provide optimal trans- port. Applied Mathematics Letters, 133:108225, 2022. ISSN 0893-9659. doi: https://oo..ogg//00000//..amm. 22. 108225. URL.ttpss//ww.cciieeedddee
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b. URL https: /ttp:///peeeeeeee./ net forum? d=dxIII G12RRHS.
0180X.
Lee, H., Lu, J., and Tan, Y. Convergence of score-based generative modeling for general data distributions. In NeurlPS 2022 Workshop on Score-Based Methods, 2022.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), June 2016.
Lee, H., Lu, J., and Tan, Y. Convergence for based generative modeling with polynomial complexity. In Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN
Wang. B. and Vastola, J. J. The hidden linear structure in score-based models and its application, 2023.
Øksendal, B. Stochastic Differential Equations: An In- troduction with Applications. Universitext. Springer Berlin Heidelberg, 2010. ISBN 9783642143946 URL https: //opskk//ook..../ google. fr fb/books fr/bboksiidd00 ?id=E0z EAAAAQBAJ.
Wenliang, L. K. and Moran, B. Score-based generative model learn manifold-like structures with constrained mixing. In NeurIPS 2022 Workshop on Score-Based Methods, 2022. URL https: //opnrevieww...ccmom .et / Crorumiid=eSaae IrD...
10

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
Zach, M., Pock, T., Kobler, E., and Chambolle, A. Ex- plicit diffusion of gaussian mixture model based image priors. In Calatroni, L., Donatelli, M., Morigi. S., Prato, M.. and Santacesaria, M. (eds.), Scale Space and Vari- ational Methods in Computer Vision, pp. 3-15, Cham, 2023. Springer International Publishing. ISBN 978-3- 031-31975-4.
Zach, M., Kobler, E., Chambolle, A., and Pock, T. Product of gaussian mixture diffusion models. Journal of Mathe- matical Imaging and Vision, Mar 2024. ISSN 1573-7683.
doi: 10. 10.10077/1000111000000011110,,,0 URL https: //doi. /ttp://..g.....000///00010000000

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
A. Characterization of Gaussian distributions through diffusion models
The following proposition shows that our Gaussian assumption occurs and only the score function is linear.
Proposition 5. The three following propositions are equivalent:
(i) 20 (0, Σ) for some covariance E.
(ii) Vt 0, V log Pe (x) linear w.r.t æ.
(ii) IE 0, V. (a))) is linear w.r.t a.
In this case, for > O, V logp)(>))= = - Ei x, with E, defined in Proposition I.
Proof. (it) + (iti) is clear.
If (i), for 0. pe C2 exp 2-2 E z). Consequently, V Pe = =-,;" x and (i) = (it)
If (ii)), there exists A such that v log (a)) Az. Consequently, pi( C. Cxexp( 1/a= Ax)) and is Gaussian. This provides that zo e B. ni is Gaussian and (tii) + (i).
B. Proofs of Section 3
B.1. Proposition Solution of the forward SDE
We aim at solving:
dx; -B 22 ag Pdata-
(22)
By considering eBa,, where B, -T² 3,ds.
dz, = B. x,dt + B:
Be dx, = 3&C Br Br + BB
+ 223,dw,,) = 22 dw..
(23)
Consequently, for S
20 + 1 c Bdw,, Bo ED
(24)
and for 0 T,
B, Bi cB B: ( B. 23,dww cB n.
F, c BB.
Biz, 2,
(25)
By Itô's isometry (see e.g (Øksendal, 2010)).
Var (f"dB./2B,du
2B 2B 2B. 2Bo 2B.
(26)
which provides the covariance matrix of n.
Cov (m:) -2B (c2 1)I c1-c-281 I.
(27)
Because æp and n, are independent, E, = c -2B: E + (1 - (2Bi) I. And,
r = 2,3
2B, -2B, [ T)dt 223, Σ Ildi dt
(28)
12

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
B.2. Proposition 2: Solution of the backward SDE under Gaussian assumption
We aim at solving
-19, dy, = (r: - -,,, d2w,, OS1ST
(29)
Denoting C = So dds by considering 2, z; =
dz ct DT dy, + e dd fr
(30)
C ET c E d - -T- BT zt by derivative of the inverse matrix
(31)
-0 DT- (1y 9.) + v / -dw - 23r- Ca IT Er- - 1] 1 ,y + -
(32)
(using Equation (9))
(33)
[ ( - - 1 + 23r T dt + 28rr DT-,
(34)
= Br- - -2- - 1z + 23r 23 c. 223 - ,ww,
(35)
= 2Br Et ddwed
(36)
(37)
Consequently.
20 ((
2f c E T- dw. = -T yo + (/'' c C. ET-_dw dw..
(38)
And,
ΣT = c Dr Yo + Cc Er 1 ET- V dww
Vz c- ET
(39)
Finally,
( ET-
y/ = - - yo & with - ET- jo
(40)
By the multidimensional Itô's isometry.
Corl / / =2 S/" 200 -
(41)
Now, remark that for A, = 20, ΣT
d4, 23r-sA,ds + 20. 2
Σ --1
(42)
A,,ds 25
2C,
[-- -T1s] Er-_dd ds (using Equation (9))
(43)
20, Br - D- Σー² -
(44)
Cov (// c.
1/ dA, [Ael 20, D. -ET
/ /r. V -
(45)
Finally, Cov(E,) Σ/_r (Erl c- 2C. )-15 ΣT 20, 2 -
I-1
13

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
We have the final formula considering:
C2 [['
1 BT dds-dd [" Badx Badx= ! Badx Bada- J Bzdx Br Br-t
(46)
ST
that provides
Cov(y,) = Σr- +e 2[Br- By ET ( Cor(y)) Er-
(47)
In particular, if (ov( and E commute,
Cov(y,) = Σr- 2(B Br DT ETT (ET Cov(y)) 1)
(48)
B.3. Proposition 3: Solution of the ODE probability flow under Gaussian assumption
As done in (Khrulkov et al., 2023), the matrix x admits a derivative which is d | "1 diagonalisable. Let check that
1-2 because it is
y: ET 1/2 21/2 27
(49)
is solution of the ODE of Equation (5):
dy. -E 1/2 - -1
1/2 yo
(50)
2
1/1 [3--1E ET - BI E 1/2
- Un (using Equation (9))
(51)
13r-EET - 3r- -T- ET 1/22 1/2
T- Noct (by commutativity)
(52)
1 - ET= [ d
(53)
[8r-t+ + -r V. log pr ((y,) y,d..
(54)
Let us discuss the link between this solution and OT. The formula of OT map between two centered Gaussian distributions NO0, Σ and (0, Σ2) well known. In (Peyré Cuturi, 2019), the authors give the linear map (affine when distributions are not centered) T X AX with
A Σ
Σ
ΣT
(55)
When En and En commute, this expression simplifies to:
A = Σ 1/2 1/2221122 1/22
(56)
We showed that the solution (Equation (49)) of the backward probability flow in the finite interval [0, with 0 S $ T. corresponds to applying the initial point Yo the linear map
= Σ Σズー
(57)
-
that is, the OT map between yr NO Er) and pr-r = NO ΣT- Let us now derive the covariance matrix of the solution, which characterises a Gaussian distribution.
Cov(y,) = ETT 1//1/////2 Cov
(_ E. 1/22
(58)
In particular, if Cov(yn) and Σ commute,
Cov(y,) En DT Cov(yy))
(59)
14

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
B.4. Proof of Proposition 4
For St t E denoting (Ar,) the eigenvalues of En, the eigenvalues of Σ. (ov()r-)) are
( 1 -1) i= 1= l.....d
d1. = Xi,t te - (e(Br-Be)) X2. AiTT 2i.T
(60)
and the eigenvalues of Σ. (ov( are
Aig
1sisd.
(61)
A,T
Consequently, W2(Pt, Pe) the sum of the squares of all:
Vice Vi.e- Viee (1 1
1
1 - Xe.
(62)
No.T
25.T
Similarly, W2(Ps, Pt) is the sum of the squares of all:
Vie Vu.o-
vi
1-
(63)
Ai.T
Vis
11 V
-1
(64)
A
Let us now compare individually these differences.
2Br Bi) Ni. X,,
1.
1 -1
e
AiT
6- -((Br- Bi) dv.t
(65)
X
Ag.r
-2 B,) 2B, (): +1
(66)
2BT (A: 1) +1
(A: 1) + BB.
(67)
(4: - 1) + 2BT
(68)
Case 1: 0 < As < and >
In this case, dd. < and:
1 (1 - 1
0 < -
-2(BT
B.) Ai. Ao.
1.
(69)
Ni.T
Thus,
15

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
Viia - Vh.s = Vdss - Vdca
(70)
1
= -/1
1 + 1 N1 NiT
1 -1
(71)
Ao.T ----) -1)
1
1+
(72)
Vice
(73)
Vite
(74)
Case 2: A; and t 0.
In this case, for 1 < d, AA. AG, = O.
Case 3: A = 1. In this case, for 5 X1, 21,1 1. Case 4: As.
2( 11
dr, xx,
In this case, Ar 2 and
Ba) 2(B
Asit dx.T
I provides
-2(Br-Be)
2(
Bi) Aint
1
-1)
1
1.
(75)
A,
A1.T
Ar.
Finally,
Viue Av.1 = Ai.
V A1.
(76)
1
1 + E-2 B(Br-BB) Ao diT
No.T --1-11 )
1
(77)
V1+ -1)
Vv.e
(78)
Vv.e Vin
(79)
V 5r,0
(80)
This case study provides:
We(Pr.P) pi) S P.))
(81)
16

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
C. Solution of the backward equations for nonzero mean Gaussian distributions
In the main paper, we assume zero mean Gaussian for two reasons: first, to align with the machine learning framework, where data is typically normalized as a preprocessing; second, simplify the notation of the equations. Nonetheless, all the results can be extended the case of nonzero mean let us provide the solutions the backward equations to the case Pdata [N, Σ). In this case, (a = [e μa. Ee) (Albergo et al., 2023) and
Wx lo) E Br u), 0<LST.
(82)
C.I. Solution of the backward SDE
The backward equation is given by
dy: 3r-1 2 - -e-Br 'H]]dt + 123r-idws 0sts
(83)
By considering 2, = Vr c Br L,,
dz, = dy, - -1- BT 'pdll
(84)
-r - ---Br Br-Br-iuu wp]]dt + 1/3rr-ruu - - "pdt
-r-
(85)
Br--llz- - - + 2f
(86)
which is the backward equation for the zero mean distribution (00, Σ) (Equation (2)). Consequently, by Proposition 2,
(Br- Br 1) EIT-ET 20 Ec. Ost<T
(87)
with E. c 1o _T-- 2Br dw and
Vr = Br- μ + (Br
(Br- Br- ET (yo - 6 BB + + OSIST.
(88)
C.2. Solution of the probability flow ODE
The reverse probability flow ODE is given by
dy = Br 1 ΣT - - Br)+ dt, 0 0st<T
(89)
By considering = yr
Br ip,
dz, dy. 3I -16 Br pdlt
(90)
=3T 9) En (3), -Br [1)] dt -e- Br -udt
(91)
= -3r-1 2e Σ-- dt
(92)
which is the reverse probability-flow for the zero mean distribution (0, Σ) (Equation 5). Consequently, by Proposition 3,
zt = 1/
1/2
1/2
OstsT
(93)
and
y. =E= BBr "p + ET 1/2 1/2
(
e-Br Br μ), OStsT, 0
(94)
17

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
D. Gaussian CIFAR-10 samples
The Gaussian CIFAR-I0 produces unstructured images, A grid of samples is presented Figure 4. To sample from this Gaussian, the empirical covariance matrix of size extract square root matrix (see source code).
computed and then the SVD decomposition to
Figure 4: CIFAR-10 Gaussian samples. Samples are generated from the Gaussian distribution fitting the CIFAR-10 dataset.
18

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
E. Studied numerical schemes
The studied numerical schemes are presented Table 3.
G...M
No
SDE sehemes
Marıyama (EM)
9k PM
Vi.
-A.IM
Aifite.
U% 10 + /20, 32 No.
(95)
where
f(t, y)
BT 11 2/r 1.
a.A. No
Exponential integrator (EII
6 a... + 71.t (
82 /2Ta.ze. 21 No -,,
(96)
where 71,k
exp(Br Br Br let1) and "2.2 = = (exp(2Br i 2Br (4-1 1]
Denoising DifTusion
DDPM
5
3OMM D
DDPM
B x
mord) + Vouu 24 No
Prohabilistic Model (DDPM)
11- POPP
where 5DDPM
v Buler F4-1
No
ODE schemes
Explicit Buler
U2 + A. Pta.. Bulyy
(98)
where
[(1 v) = BT 4- 9r- -i
No
Nean
Heun's method
1 D2-1/2
Di
Hean
((12 Nemn + f(12+1 (411))))
(99)
where f(t. W1 = Br Br Eip :
9 G..
No
Titk. D& RK4,
Runge- Kutta 4 (RK4)
K2
711+11/2 D Ki)
K3
f1t 112- D
5: K2)
hxs Kg)
K4
flt D
Vh. gA. RK1 + (K2 + 2Kg + 2Ks + K4)
9
where (t. M) = 9r Br 1 Ee f441/2 t4
(100)
Table 3: Stochastic and deterministic discretization schemes. EM, and DDPM disctretize the backward SDE of Equation (3), Euler, Heun and RK4 schemes discrctize of the probability flow ODE of Equation (6) with a regular time schedule (tx Sikosse with stepsize A, T
N
19

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
F. Computation of the 2- Wasseerstni distances for numerical schemes
The 2- 2-Wasserstein errors can be computed by using Equation (18) recalled here:
(,, E2))² Σ (vAr1 2.,2) Igigd
(18)
for two centered Gaussians (0, Σ1) and (0, Σg) such that E1. En are simultaneously diagonalizable with respective eigenvalues (As,1) Isica (Ai,2) gissd We aim computing these errors between the Gaussian process folowing (pe (PelssiST and the processes induced by the discretization schemes. Table 3 shows that each discretization scheme leads to a discrete time Gaussian process whose covariance matrix is diagonalizable in the basis of Σ when intialize them with either No (usual sampling) or pr (no initialization error). Let detail this point for the Fuler-Maruyama (EM) scheme. Let denote (vt) sisd
basis of eigenvectors of E and its associated eigenvalues (Ni)s For 0 s T. E, 2B, Σ + (1 e 218 ) and
for all 1 S is d,
E = (c-2Bi X, + e1-e e--))
(101)
Consequently ΣS admits the same eigenvectors than Σ with associated eigenvalues (Ar.)igidd (e-2B:>;; + (1- 2B, Isisa Let study the covariance matrix of the EM process. Let denote
( (DARM)
1SISdOSKSN
the covariance matrix of the Gaussian process generated by the EM scheme at each step and (A eigenvalues. First,
A.EMM)
its
1SISd,0SKEN
{
if yr is initialized at No
EAFM
(102)
Er
if zrr is initialized at pr
And consequently,
if yr is initialized at No if yr is initialized at pr
A AME 1.U
1 1sisd d.
(103)
c- 2Br d + (1 cc-2Br
Then, by Table 3.
A.EM
(I+ (I 11-2ET-,)) y0 A.EM
+ 22 Zu. 20 No
(104)
and
M
(I+ A,
(I- (1-2=--10) M (I 4,8r to (I 2E-2)T-10)) + 24,33
(105)
= (I+ A,BT- 1-2E-11077 I + 2A, -1 because E and Σ EM commute.
(106)
Let Sisd.
AAEM M 0,
= (I
2
2 ))
A.EM X
24,
Vi
(107)
Consequently, (D:) is also basis of eigenvectors of E,A.M and
A.EM
1 108T-0 (1- Nr.T-10
2
A.EM + 22,,3T101
1sisd
(108)
(1AEM)
Thus, obtain the eigenvalues
at each time and plot each time
ISiSdOCkKN
Σ
(MAiT-m - AAA ISKSN
(109)
ISiSd
as done in Figure 1, These computations can be led for the different schemes, as presented in Table 4.
20

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
EM
AAPMM ,,+1
(1 11+
21,7-
PM
+ 12
ISigdosRSN
SDE schemes
1,4+1 (1+
A )
A 272,4 1
DDPM
1- 1
35.2 1/2
soom 15 4.0
Fuler
Add
A
Heun
X 1,
1+ Pr tk "ー
2000 1h+1 1 (-x,--1013 )
(1+ArPr AePT in (1 x- )) x
ODE scheenes
ig d,00
--1 Pr-t
C1
51.7-1k
C2
Pr-th11/2
x. 14/1/2
(1+ 4.))
RK4
Cs
C4
B1-1411
7,,7-1411
(1
(I+ (G1 4.2 20s
C4l
TEISADSKEN-
Table 4: Recursive form of the eigenvalues of the covariance matrix associated with the Gaussian process generated by the different schemes for a regular time schedule (44) with steps A,
N.
21

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
G. Ablation study of Gaussian CIFAR-10
The complete ablation study of Gaussian CIFAR-I0 presented Table
Continuous
NFE 50
NFE = 250
NFE = 500
NFE 1000
N6
pr
No
pr
No
pr
No.
pr
NN.
8
6.7E-04
4.78
4.78
0.65
0.66
0.32
0.16
EM
10
4.1E-03
4.2E-03
4.77
4.77
0.66
0.66
0.32
0.32
0.16
0.16
10
0.03
0.03
4.76
4.76
0.66
0.66
0.32
0.32
0.17
0.17
8 10
0.18
0.18
4.68
4.68
0.70
0.70
0.40
0.40
0.27
0.27
2.81
2.81
0.57
0.57
0.30
0.30
0.16
0.16
EI
& = 10
2.81
2.81
0.57
0.57
0.30
0.30
0.16
0.16
10
2.82
2.82
0.58
0.58
0.31
0.31
0.17
0.17
3 10
2.91
2,91
0.67
0.67
0.41
0.41
0.29
0.29
0
6.82
6,82
0.97
0.97
0.47
0.47
0.23
0.23
DDPM
& 10
6.82
6.82
0.97
0.47
0.24
0.23
= 10
6.81
6,81
0.98
0.98
0.47
0.47
0.24
0.24
3 10
6.74
6.74
1.00
1.00
0.53
0.53
0.32
0.32
8 0
0
0.07
1.72
1.78
0.38
0.44
0.20
0.26
0.10
0.17
Faler
10 10-5
4.1E-03
0.07
1.72
1.78
0.38
0.44
0.20
0.26
0.10
0.17
2 10
0.03
0.08
1.72
1.78
0.38
0.44
0.20
0,26
0.11
0.17
10
0.18
0.19
1.73
1.79
0.42
0.48
0.27
0.32
0.21
0.25
Heun
10
23.42
23.42
2.86
2.87
1.05
1.06
0.37
0.38
2
10
4.68
4.68
0.43
0.44
0.12
0.03
2 10
0.58
0.59
0.13
0.15
0.16
0.18
0.17
0.19
RK4
غ 10-5
67.32
67.32
5.49
5.49
2.11
2.11
0.77
0.77
10 10"4
14.35
14.35
0.93
0.93
0.29
0.29
0.07
0.10
3 10
2,60
2,60
0.09
0.12
0.15
0.17
0.17
0.19
Table Ablation study Wasserstein errors for CIFAR-10 Gaussian. given discretization scheme, the table presents the Wasserstein distance associated with truncation error for different values e. The columns pr and Mo show influence of initialization error. The continuous column corresponds to the continuous SDE or ODE linked with the scheme (identical values for EM, EI, DDPM and Fuler, Heun, RK4) and given number of score function evaluation NFE.
22

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
H. Theoretical Wasserstein distance for the ADSN model
As done the Gaussian CIFAR-IO, the Wasserstein errors can be computed for the ADSN model as shown Figure and Table
SDC
EM
EI
DDPM RK4
101
109
10a
10-1
W,6.pt .P)
Waf.p) (p).p)
10")
10-2
10
10-3
10-3__
10
10
10
10-1
5DE
-Xi
EM
-X6 EI
DDPM
10-
10-1
0.2
0.4
0.6
0.8
T-1
0
10
10
10
10
Initialization along the integration time.
(b) Truncation for different truncation time
for the diffusion models associated with the Gaussian microtextures. Left: Evolution of the Wasserstein distance between Pt and the distributions associated with the continuous SDE, the continuous flow ODE and four discrete sampling schemes with standard No initialization, either stochastic (Euler-Maruyaan Maruyama (EM), Exponential Integrator (EI) and Denoising Diffusion Probabilistic Model (DDPM) or deterministic (Euler, Heun and Runge-Kin Kutta 4 (RK4)). While the continuous SDE is less sensihle than the continuous ODE (as proved by Proposition 4), the initialization error impacts all discrete schemes. Heun's 's method has the lowest and very close the thcoretical ODE, except for the last step that is usually discarded when using time truncation. Right: Wasserstein errors due to truncation for various truncation times e. Heun's 's scheme not defined without truncation time to the zero eigenvalue. Interestingly, for the standard practice truncation time č 10-3 all numerical schemes have comparable eroo close to their continuous counterparts.
Figure 5:
Wasserstein
23

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
Contimuous
NFE
NPE
250
NFE: 500
NFE: 1000
Pr
No
Pr
No
pr
No
pr
No
pr
No
0
5,2E-06
53.37
$3.37
10.58
10.58
6.27
6.27
4.02
4.02
EM
E 10
0.36
0.36
53.35
53.35
10.57
10.57
6.26
6.26
4.02
4.02
10
1.15
1.15
53.21
53.21
10.53
10.53
6,25
6.25
4.03
4.03
&
3.84
3.84
51.92
51.92
10.55
10.55
6,80
6.80
5.16
5.16
0
30.91
30.91
8.85
8.85
5.71
5.71
3,84
3.84
8 10-3
30.92
30,92
8.85
8.85
5.72
5.72
3.84
3.84
8 10
31.01
31.01
8.92
8.92
5.78
5.78
3,90
3.90
10
31.94
31.94
9.74
9.74
6.76
6.76
5.24
5.24
8
53.66
53,66
10.58
10.58
6.27
6.27
4.02
4.02
DDPM
10
53.64
53.64
10.58
10.58
6.26
6.26
4,02
4.02
8 10-*
53,50
53.50
10.54
10.54
6.25
6.25
4.03
4.03
10
52.19
52.19
10.56
10.56
6.80
6.80
5.16
5.16
8
6,4E-03
5.69
5.70
2.50
2.51
1.87
1.87
Ealer
10
0.36
0,36
5.70
5.71
3.28
3.28
2.53
2.53
1.90
1.90
10-4
1.15
1.15
5.80
5.80
3.43
3.43
2.72
2.72
2.15
2.15
8 10
3.84
3.84
6.79
6.79
4.85
4.85
4,41
4.41
4.14
4.14
Heun
8 10
2.4E+03
2.4E+03
3.0E+02
3.0E+02
1.1E+02
1.1E+02
40.00
40.00
10
2.3E+02
2.3E+02
26.34
26.34
8.54
8.54
2.01
2.01
8 10
15.42
15.42
2.25
2.25
3,40
3.40
3.73
3.73
0
RK4
8 10-3
6.9E+03
6.9E+03
5.7E+02
5.7E+02
2.2E+02
2.2E+02
82.03
E 10-4
6,8E+02
6.8E+02
52.25
52.25
18.61
18.61
5.57
5.57
10
59.79
59.79
0.62
0.62
2.94
2.94
3.66
3.66
Table 6: Ablation study of Wasserstein errors for the Gaussian microtextures. For given discretization scheme, the table presents the Wasserstein distance associated with the truncation error for different values of The columns pr and No show the influence of the initialization error. The continuous column corresponds the continuous SDE ODE linked with the scheme (identical values for EM, El. DDPM and Euler, Heun, RK4). Note that the Heun scheme is not defined without truncation time due to the zero eigenvalue.
I. Study of the covariance matrix of the ADSN distribution
L.. Reminders the Discrete Fourier Transform (DFT)
For given image MxN define the DFT of 3 such that for c
M
2iTTI
2iT
Do.E
Σ
Ve. exp(
exp(
82 2-11
(110)
M
N
ZEMXN
where De.g is the value of e coordinate & of the k-th channel of ú. For R3MxN by defining the periodic convolution such that for 1 < e < x E AREMRMM M
uu v) = Σ
uc Ve.
(111)
NEMxA M
we have:
21 ④
(112)
24

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
where @ is the componentwise product. By denoting & the image which is reversing the arrangement of elements in vector
(113)
where i the component-wise conjugate of V.
L.2. Eigenvectors of the covariance matrix of the ADSN distribution
Let u E R and its associated texton E RXxMXN The distribution (DSN(u) is the Gaussian distribution of x = t x w such that:
X; x MNN 1sis3, No
(114)
Consequently, denoting Σ the covariance of ADSN for v E R Ev the Fourier transform of Ev and Σe, its ith channel for < i 5 3,
Ev. = E, Go0 + 0
(115)
This equation proves that the kernel of Σ contains the kernel of E R - t + 120 + +1z03 E RMN which has a dimension greater than 2MN. Consequently, is eigenvalue of E with multiplicity greater than 2MN. Furthermore, for & M x N, denoting 1. such that:
u (w) = 1
(116)
we have,
Eu 16 = ( [t(()) + (((())
(117)
Furthermore, (fL( + +E2(()) 2 For & E M X N, we denote 2. 3.5 such that for w E M x N:
the family (u':5)
{EMxN is orthogonal.
Thus.
the eigenvalues of Σ are
and 0 with multiplicity 2MN.
{EMxN
(w)
Iw=cta(w)
(118)
(w)
-w (w)
a (L((W) =0
un 3.5 (w) = =-1w=stahe
(119)
0 (w)
[w
We have
Euz 0.u
(120)
3u 3.5
(121)
Then, applying the orthonomalization of Gram-Schmidt each tuple (u u26 u3 ) obtain orthonormal basis in the Fourier domain (o 1 2. 3 ) of eigenvectors of E. More precisely, for E1. E x N, 1 j., J2 $ 3,
25

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
goriso
Σ
(122)
M 2
h= 61
(123)
which is applying the square root of Σ to the white Gaussian noise w. Furthermore, we can ensure that for ç * w E M N, 1 S 3, a2 (w) = 0 such that only the frequency & is active for V² Consequently, for w R3MxN
=I
Σ (E)
(124)
15153
In particular,
(01)) -i 0
-1001212=
Σ
(5) =1
(125)
15153
I.3. Computation of the empirical Wasserstein error in the ADSN covariance diagonalization basis
Let consider a Gaussian distribution /O I) such that there exists (Af, X5, AgDEEMXN such that for all E M N,
Tw = Xfrts 1 S j S 3.
(126)
Let w No R3MXN, (v' u2.5, 3. ) EEMNN an orthonormal basis in the Fourier domain such that:
w
Σ
④ 6165
v- +
--
w
2.6
2
w 0.55 53.5
(127)
ÊEMXN
(128)
A sample drawn from (), has the same distribution as Y given by
Σ
V
Σ
1 in
2
2.
Σ
VAg
i
3.5
(129)
ŞEMxN
ęEM*N
ŞEMxN
Note that the three channels of w are independent. Furthermore, for <sss
T
1 2 = 1
(130)
(131)
27
E
11
4
(132)
E
Σ
E
We
(5) (5)
o (E) (5) by Equation (124)
(133)
53
Σ
(
because the channels are independent
(134)
15e33
= MM
JoS T5)
because [.(()) = MN
(135)
18e53
MM by Equation (125).
(136)
26

---

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors
Finally,
E
Y
MN
(137)
Finally, for a given sampling (Y x) skk Namptes
following the distribution (o(plttiiioon,,,) an estimator of X5 is:
Nasuple
2
25 S5omp
1.
YK
(138)
samples
The empirical Wasserstein distance between the Gaussian distribution (00, r) and the ADSN model with texton is:
((VA
W (Nemp. (0, r), ADSN(u))
Σ
( V V...ADSN)
+ Ae A..em.. + A..
x.emp.
(139)
CEMXN
with XÇ. AGADDSN ((2 (t) + (2) for E N.
Furthermore, the computations can be vectorized by componentwise products in the Fourier domain.
27


Reviews:
[{"review_id": "kDiCzAUH6s", "reviewer": "Reviewer_cXeJ", "summary": "The article studies different types of errors when implementing diffusion models (DM) by constructing a Gaussian DM, for which the solution is known in closed form, and then measuring the errors between the exact and approximated solutions using the Wasserstein distance. They analyse the sensitivity of the error wrt different factors.\n\n## update after rebuttal\n\nI have a positive view of this work, which I maintain after the discussion period.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your thoughtful and positive feedback. We are pleased that the you find our work a meaningful contribution to the theoretical understanding of diffusion models (DMs) and recognize its relevance within the broader literature. Below, we address your specific questions and suggestions.\n\nAnswers to questions:\n\n>They claim that, for any starting distribution, the reverse SDE (assuming Gaussian p_data) is Gaussian \n\nIn Proposition 2, the expression of the strong solution in **Proposition 2** remains true without assuming that $\\boldsymbol{y}_0$ is Gaussian but $\\boldsymbol{y}_t$ does not necessarly has a Gaussian distribution even if we can write its covariance matrix.\nWe propose to clarify the assumption in Propositions 2 and 3 by writing explicitly \"Supposing that $p\\_\\text{data} = \\mathcal{N}(0,\\boldsymbol{\\Sigma})$\".\n\n\n> The paragraph on line 208 (left) could probably be a Remark?\n\nThank you for this suggestion, which we have added.\n\n> At some points, the authors seem to downplay their own contribution. Early in the paper they say that considering Gaussian DMs might not make much sense and then in the discussion they also say that \"the limitation of their work is that they only consider Gaussians\". Though a stylistic issue only, I do not agree with this: I think this work is important. Its contribution is limited, true, but all contributions are limited to their own specified area. If I am not mistaken, this work has value for the community and should be recognised as such.\n> One of the final observations of the paper (the score approximation being the most important error in practice) should be emphasised earlier in the paper to highlight the practical findings of the work.\n\nThank you for acknowledging the importance of our contributions. In case of acceptance, we will take into account your advice to better highlight our results. We do not want to downplay the importance of our work, but we want to ensure that readers understand that generalizing our results to more complex cases is very challenging.", "claims_and_evidence": "The contributions of the paper are both theoretical and experimental. They apply to, what is defined in the paper as, _the Gaussian assumption_. The former reveals that the ODE solution is better than the SDE solution, when it comes to sampling from p_data. The latter provides many insights regarding the effects of different approximations in the solution of the DM.\n\nThe paper does a good job within its declared scope. Understanding the theoretical performance of DMs is lacking in the literature, and this work points in this direction. The claims and evidence are well supported in that sense.", "methods_and_evaluation_criteria": "The experiments are not comprehensive, but that is because the setting laid out is a simple one. To me, this is well justified by the aims of the paper. The evaluation provided here makes complete sense for this work.", "theoretical_claims": "The theoretical claims are clear for the setting considered, this is well explained by the authors. How the results of the paper extend to more general settings (i.e., non-Gaussian DMs) is unknown and beyond the scope of this paper, but the theoretical claims of this article can be an interesting starting point.", "experimental_designs_or_analyses": "Experiments are appropriate for the claimed contribution", "relation_to_broader_scientific_literature": "Good account of other methods", "essential_references_not_discussed": "Not that I know", "other_strengths_and_weaknesses": "I find this work a good contribution to the literature. Perfectly within scope for ICML and at the expected conceptual/experimental depth.", "other_comments_or_suggestions": "NA", "questions_for_authors": "- In Proposition 2, the authors present one of the manuscript's main contributions. They claim that, for any starting distribution, the reverse SDE (assuming Gaussian p_data) is Gaussian. I am not sure I understand this. In the author's setting (i.e., Gaussian p_data, or \"the Gaussian assumption\"), the forward SDE constructs p_T in a way that it'll also be Gaussian (a sequence of linear transformations starting from a Gaussian). Therefore, when the authors say \"any starting dist\" do they refer to any \"Gaussian\" distribution? If not, how could the solution of the reverse SDE, say $p_t$, be Gaussian if starting from any, e.g., a multimodal, p_T?\n\n- The paragraph on line 208 (left) could probably be a Remark?\n\n- At some points, the authors seem to downplay their own contribution. Early in the paper they say that considering Gaussian DMs might not make much sense and then in the discussion they also say that \"the limitation of their work is that they only consider Gaussians\". Though a stylistic issue only, I do not agree with this: I think this work is important. Its contribution is limited, true, but all contributions are limited to their own specified area. If I am not mistaken, this work has value for the community and should be recognised as such. \n\n- One of the final observations of the paper (the score approximation being the most important error in practice) should be emphasised earlier in the paper to highlight the practical findings of the work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iZXOeFkr2B", "reviewer": "Reviewer_Rj1X", "summary": "The paper diffusion-based generative models assuming that the data distribution is a known Gaussian distribution. Since the data distribution is Gaussian the diffusion process score is linear and known in closed form, which allows the authors to:\n\n1. Solve in closed form the reverse  (noise → data) SDE and ODE since they are linear. \n2. compute the Wasserstein-2 metric between the model distribution and the data distribution since the model distribution is Gaussian, and the Wasserstein-2 metric between two Gaussians is available in closed-form. \n\nUsing these closed-form metrics and ODE/SDEs, the authors then proceed to study:\n\n1. the initialization error\n2. the error introduced by numerical ODE and SDE solvers\n3. the effect of step-size\n4. the error induced due to truncation, stopping the sampling process at $t = \\varepsilon > 0$. \n\nIn section 5, the authors then learn the score model (parameterized by a U-Net) for a Gaussian distribution to see the effect of the score function approximation error on sampling.\n\nTable 1 shows that:\n\n1. using RK4, a higher-order ODE solver, produces lower error\n2. using more NFE, a lower step-size for the solver, produces lower error", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your feedback. \n\n\n> The experiment in section 5 however does not highlight the impact of score function approximation error on the distribution metrics reported. The authors study the effect of sampling parameters on the distribution metrics, not the effect of higher or lower d.\n\nWe consider this part to be a study of the effect of score approximation error on the samplers because we have mapped the impact of each type of error in the use of the schemes. Then, we introduce this final error and observe that it causes all our errors to explode.\n\n\n\n> The model required for FID scores is not trained on data of the type generated by ADSN. Therefore, are the FID scores reliable?\n\nWe have chosen to use the standard metric. All the reported FID values vary consistently with other Wasserstein metrics. \nThe FID metric compares the statistics of image features, allowing it to discriminate between one distribution relative and another. Within the scope of our paper, without comparing our FID values to those from other tables in different contexts, this measure appears to be relevant.\n\n\n\n\n> On lines 424-427, the authors show claim that the experiment in section 5 demonstrated that the score approximation error “may be the most important one in practice”. However, this experiment does not demonstrate that since they do not show the effect of the score function approximation error on the distributional metrics.\n\nPrecisely, this section highlights a significant difference between theoretical analysis and practical implementation.\n\n\n> can the authors explain wha insight into diffusion models can be gained from the exact score, ODE and SDE in the Gaussian data framework that was not already known.\n\nWe must highlight an important subtlety regarding our work: we do not compare different schemes from an asymptotic perspective (with ($\\Delta_t \\to 0$), as has been extensively done in the literature and as you have rightly pointed out.  \n\nInstead, we focus on:  \n- The robustness of the schemes with respect to initialization and truncation errors, which is not commonly studied in the literature, as existing works typically assume a known initialization. On the contrary, these errors are central in our paper \n- The behavior of the schemes under a fixed NFE budget. Interestingly, we manage to obtain results comparable to those of asymptotic studies found in textbooks, even though this was not an a priori objective when considering the practical use of these models—particularly with a maximum budget of 1000 NFE.  \n- The study of high-dimensional schemes, particularly stochastic ones, which are rarely analyzed in textbooks. In particular, we consider very general covariance structures that may lead to distributions lacking a density.  \n- Exact Wasserstein errors rather than bounds, which may otherwise be excessively large.", "claims_and_evidence": "- On line 420-422, right panel: the theoretical analysis leads to the conclusion that Heun’s method is the “best numerical solution”:\n    - That finding is also contracted by numerical methods literature, which states that RK4 and other higher order solvers are better numerical solvers\n    - claim finding is also contradicted by the experiment in section 5, where EM leads to a lower value using the author’s proposed metric.\n    \n- on line 424-427, the authors show claim that the experiment in section 5 demonstrated that the score approximation error “may be the most important one in practice”:\n    - The experiment in section 5 however does not highlight the impact of score function approximation error on the distribution metrics reported. The authors study the effect of sampling parameters on the distribution metrics, not the effect of higher or lower score approximation error.", "methods_and_evaluation_criteria": "The paper proposes that analyzing diffusion-based generative models for Gaussian data allows for an evaluation of any numerical sampler. However, \n\n1. There is a well-studied framework for analyzing numerical solvers\n2. The authors do not show that any inference gained from the Gaussian setting transfers to other data distributions. Moreover, the experiment in section 5 disagrees with the findings in section 4.", "theoretical_claims": "The reverse ODE and SDE solutions are correct.", "experimental_designs_or_analyses": "In section 4, the experiment studies the multiple sources of error and sampling parameters as mentioned for a gaussian distribution, with covariance computed on Cifar10 samples. \n\nIn section 5, since the model distribution is not Gaussian anymore the authors make use of other metrics, such as FID and a problem-specific Wasserstein metric, proposed by the authors. \n\n1. These metrics assume the model distribution is Gaussian, however other metrics/divergences such as the kernel Stein discrepancy, maximum mean discrepancy, etc can be estimated without the Gaussian assumption. \n2. The model required for FID scores is not trained on data of the type generated by ADSN. Therefore, are the FID scores reliable?\n\nIn section 5, the finding regarding the stochastic EM sampler outperforming Heun’s method contradicts with the findings in table 1, where Heun’s leads to lower error, although the score function was known in closed-form. It is not clear what the added benefit of the experiment in section 5 was for the paper. \n\nOn lines 424-427, the authors show claim that the experiment in section 5 demonstrated that the score approximation error “may be the most important one in practice”. However, this experiment does not demonstrate that since they do not show the effect of the score function approximation error on the distributional metrics.", "relation_to_broader_scientific_literature": "- The findings in table 1 study are already well-studied in the numerical methods literature as well as the diffusion modeling literature:\n    - Using a higher-order ODE solver such as RK4 is more accurate than lower-order sampler\n    - using a lower step-size leads to lower error is a well-known fact", "essential_references_not_discussed": "Relevant references are discussed.", "other_strengths_and_weaknesses": "See claims and evidence, Methods And Evaluation Criteria and Experimental Designs Or Analyses", "other_comments_or_suggestions": "None", "questions_for_authors": "The main limitation I believe is the \"framework\" being proposed, since as the authors acknowledge studying Gaussian data distribution is already a well established practice and exact scores, velocities for mixture of Gaussian data have been computed in several prior works, it is not clear what insight can be gained from this work. \n- can the authors explain wha insight into diffusion models can be gained from the exact score, ODE and SDE in the Gaussian data framework that was not already known.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "swvKsypJzY", "reviewer": "Reviewer_CDeZ", "summary": "1. Under the assumption of a Gaussian data distribution, the authors derive exact solutions for both the backward stochastic differential equation (BSDE) and the probability flow ordinary differential equation (ODE) in the VP-SDE diffusion model.\n2. This paper systematically analyzes the convergence properties of diffusion models by identifying four primary sources of error, all of which are quantified using Wasserstein-2 distance ($W_2$):\n   - Initialization Error: This arises from approximating the final forward-distributed sample $p_T$ using a standard Gaussian distribution $\\mathcal{N}(0,I)$.\n   - Truncation Error: In practice, the backward integration is started at a small time $\\varepsilon>0$ instead of $t=0$ to avoid numerical instability, introducing additional bias.\n   - Discretization Error: Since the SDE or ODE is solved numerically using numerical schemes such as Euler, Heun, and RK4, numerical approximation errors accumulate.\n   - Score Approximation Error: In real-world applications, the score function is learned via a neural network, and training errors impact the final generation quality.\n3. The authors compute exact $W_2$-initialization errors of the SDE and ODE samplers respectively, which provides an inequality of $W_2$-distance.\n4. They investigate $W_2$​-discretization errors in various numerical schemes in both CIFAR-10 Gaussian data and ADSN distribution data settings, which are expressed by the eigenvalues of the covariance matrix at specific steps.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your feedback.\n\n## Answers to questions\n> I am curious about whether there truly have been no prior works exploring the explicit solutions of the VP-SDE and PF-ODE in the Gaussian data setting, as their solutions are not particularly complex and can essentially be found in any standard textbook on stochastic processes.\n\nIn the literature, the solution to SDEs is generally provided in 1D, which makes their generalization to diffusion models for image synthesis non-trivial.\n\n> For the eigenvalue contribution to the Wasserstein error, almost in all the numerical schemes, the function $|\\sqrt{\\lambda^{scheme}}-\\sqrt{\\lambda}|$ have a zero point near $\\lambda=1$. And this function manifests convexity in a small neighborhood of every zero point. I wonder if there are theoretical explanations.\n\nThank you for these observations.\nThe different schemes exhibit highly diverse behaviors, even when plotted in logscale as in Figure 2.\nOur code, available in the supplementary material, could be used for more precise plots around any point of interest for each numerical scheme.\n\n## Theoretical Claims and Evidence section\n\n>  I know theoretical analysis in general cases is almost impossible, while I hope this can be generalized to Gaussian mixture models. There are several reasons:\n> - If the data distribution is a mixture of Gaussian distributions, by the linearity and solvability of the VP-SDE, it is straightforward to prove that for any time $t>0$, $X_t\\sim p_t$ is also a mixture of Gaussian distributions.\n> - Write $GMM_d(K)$ the subset of $\\mathcal{P}(\\mathbb{R}^d)$ made of probability measures on $\\mathbb{R}^d$ which can be written as Gaussian mixtures with less than $K$ components (such mixtures are obviously also in $\\mathcal{P}p(\\mathbb{R}^d)$ for any $p>1$). Denote $GMM_d(\\infty)=\\cup{K\\geqslant 0} GMM_d(K)$. Then $GMM_d(\\infty)$ is dense in $\\mathcal{P}_p(\\mathbb{R}^d)$ for the metric $W_p$​, hence the Gaussian mixture model is sufficiently general to study the real-world data.\n> - $W_2$ distance between Gaussian mixtures is difficult to calculate, while analyzing $MW_2$ distance can be a feasible choice. Details can be found in [J. Delon et al. 2020].\n\nThe difficulty to extend our work to GMMs is discussed in Section 6. Discussion and limitations. The solution, derived in Proposition 1. is indeed true for very general data distributions but the solutions of the backward SDE and probability-flow ODE do not extend to the GMM case.\n\n\n> Commutativity of covariance matrices $\\Sigma_{t_1}$ and $\\Sigma_{t_2}$, where $X_{t}\\sim p_{t}=\\mathcal{N}(0,\\Sigma_{t})$, is frequently used in proofs of propositions, such as equation (52), (93). \nAdditionally, if $Cov(y_0)$ and $\\Sigma$ commutes, the proof of commutativity of $Cov(y_0)$ and $\\Sigma_t$ (for any $t>0$) needs to be supplemented. This property is used in equation (48), (59).\n\nWe do not understand the main point you want to emphasize: It appears clear that with $\\boldsymbol{\\Sigma}_{t} = e^{-2B_t} \\boldsymbol{\\Sigma} + (1-e^{-2B_t})\\boldsymbol{I}$, $\\boldsymbol{\\Sigma}\\_{t_1}$ and $\\boldsymbol{\\Sigma}\\_{t_2}$ commute. Similarly, if $Cov(\\boldsymbol{y}_0)$ and $\\boldsymbol{\\Sigma}$ commute.\n\n\n## Comments Or Suggestions\n\n>  In addition, $\\widehat{\\Sigma v_{i}}$ and $\\widehat{\\Sigma v}_{i}$ are difficult to be distinguished for readers who are new to them, which can cause difficulties in understanding this proof. Hence a clear explanation is needed. \n> Furthermore, this formula depends on a fact, where $\\widehat{u*\\overleftarrow{v}}=\\hat{u}\\odot \\bar{\\hat{v}}$ ($\\overleftarrow{v}$ refers to reversing the arrangement of elements in a vector $v$​). \n\nWe have added sentences to clarify these important points. Thank you for noticing them.\n\n> In equation (121)~(124), the coefficient should be $MN$, not $3MN$​. Hence the empirical Wasserstein distance in equation (125) is indeed wrong.\n> However, according to the **Theoretical Claims and Evidence** above, in the ADSN model experiments, the actual Wasserstein-2 distance should be $\\sqrt{3}$ times the value depicted in Figure 5. \n\nIndeed, thank you for your careful proofreading. \nLet us precise that this typo was not present in our code (no factor 3) so all reported results in Table 2 are valid, which explains the closeness between the theoretical and empirical Wasserstein distances. We hope that this clarification could convice you to raise your score.\n\n## Typos\n> $dt$ is missing in equation (23) \n> All instances of $\\xi \\in \\mathbb{R}^{M\\times N}$ and $\\omega \\in \\mathbb{R}^{M\\times N}$ should be replaced \n> Equation (101) should be replaced with [...]\n> $\\lambda_{1}^{\\xi}$ should be $\\lambda_{j}^{\\xi}$ in equation (116).\n> The notations $v^{j,\\xi}$ and $\\hat{v}^{j,\\xi}$ are used alternately after equation (107)\n\nThank you again for your careful proofreading. We have corrected all the mentioned typos and carefully rechecked the full appendix regarding componentwise products and vector indices.", "claims_and_evidence": "**Theoretical Claims and Evidence:**\n\n1. The authors provide thorough analysis both theoretically and numerically of the VP-SDE diffusion model, thanks to the assumption of Gaussian data. In particular, the $W_2$ error inequality between the SDE and ODE samplers (Proposition 4) is intriguing. However, I am not sure if these results can help to investigate in more general cases. After all, the Gaussian assumption is not sufficiently general and the real-world data can be much more complex. I know theoretical analysis in general cases is almost impossible, while I hope this can be generalized to Gaussian mixture models. There are several reasons:\n\n   - If the data distribution is a mixture of Gaussian distributions, by the linearity and solvability of the VP-SDE, it is straightforward to prove that for any time $t>0$, $X_t\\sim p_t$ is also a mixture of Gaussian distributions.\n\n   - Write $GMM_d(K)$ the subset of $\\mathcal{P}(\\mathbb{R}^d)$ made of probability measures on $\\mathbb{R}^d$ which can be written as Gaussian mixtures with less than $K$ components (such mixtures are obviously also in $\\mathcal{P}_p(\\mathbb{R}^d)$ for any $p>1$). Denote $GMM_d(\\infty)=\\cup_{K\\geqslant 0} GMM_d(K)$. Then $GMM_d(\\infty)$ is dense in $\\mathcal{P}_p(\\mathbb{R}^d)$ for the metric $W_p$​, hence the Gaussian mixture model is sufficiently general to study the real-world data. \n   - $W_2$ distance between Gaussian mixtures is difficult to calculate, while analyzing $MW_2$ distance can be a feasible choice. Details can be found in [J. Delon et al. 2020].\n\n   Additionally, the score function also has an explicit expression in Gaussian mixture model settings. See [Wang B et al. 2024]\n\n2. Commutativity of covariance matrices $\\Sigma_{t_1}$ and $\\Sigma_{t_2}$, where $X_{t}\\sim p_{t}=\\mathcal{N}(0,\\Sigma_{t})$, is frequently used in proofs of propositions, such as equation (52), (93). This commutative property can be proved directly by independence of the Gaussian noise, while the proof of this property still needs to be supplemented for the sake of proofs' completeness. \n\n   Additionally, if $Cov(y_0)$ and $\\Sigma$ commutes, the proof of commutativity of $Cov(y_0)$ and $\\Sigma_t$ (for any $t>0$) needs to be supplemented. This property is used in equation (48), (59).\n\n**Experimental Designs：**\n\nThe authors have conducted sufficiently detailed experiments, investigating the impact of various numerical schemes, different truncation times, and the number of score function evaluation (NFE) on the Wasserstein-2 error, on the CIFAR-10 Gaussian dataset and the ADSN model respectively. However, according to the **Theoretical Claims and Evidence** above, in the ADSN model experiments, the actual Wasserstein-2 distance should be $\\sqrt{3}$ times the value depicted in **Figure 5**. Nevertheless, this does not affect the experimental conclusions.\n\n**References:**\n\n- [J. Delon et al. 2020] J. Delon, and A. Desolneux. A Wasserstein-Type Distance in the Space of Gaussian Mixture Models, *SIAM journal on imaging sciences*, 13(2), pp. 936–970, 2020.\n\n- [B. Wang et al. 2024] B. Wang, and J. J. Vastola. The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications[J]. arXiv preprint arXiv:2412.09726, 2024.", "methods_and_evaluation_criteria": "See **Claims And Evidence**.", "theoretical_claims": "See **Claims And Evidence**.", "experimental_designs_or_analyses": "See **Claims And Evidence**.", "relation_to_broader_scientific_literature": "**Theoretical Analysis:**\n\n1. While prior research (e.g., [V. De Bortoli et al. 2021]; [Chen S et al. 2022]) derived upper bounds on 1-Wasserstein or total variation distances between the data and the model distributions under suitable assumptions, these were often worst-case bounds. In contrast, this paper provides exact analytical Wasserstein-2 errors under the Gaussian assumption.\n2. The connection between probability flow ODEs and optimal transport has been discussed in prior works ([Khrulkov et al. 2022]; [Lavenant et al. 2022]). However, this paper provides an explicit demonstration of how the probability flow ODE corresponds to an optimal transport map in the Gaussian case, strengthening this theoretical link.\n\n**Numerical Considerations:**\n\nThe analysis of different numerical discretization schemes for diffusion models builds upon prior empirical studies ([Karras et al. 2022]), which recommended Heun’s method based on experimental performance. The stochastic Euler-Maruyama (EM) method has been widely used in diffusion-based generative models ([Song et al. 2021]), but its theoretical properties have remained underexplored. This paper demonstrates that SDE-based samplers are more robust to initialization errors compared to ODE-based samplers under the  Gaussian data assumption, providing a clearer understanding of why EM can sometimes outperform deterministic ODE solvers in  practical applications.\n\n**References:**\n\n- [V. De Bortoli et al. 2021] V. De Bortoli, J. Thornton, J. Heng, and A. Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. In Advances in Neural Information Processing Systems, volume 34, pp. 1769517709. Curran Associates, Inc., 2021.\n- [Chen S et al. 2022] Chen S, Chewi S, Li J, et al. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions[J]. arXiv preprint arXiv:2209.11215, 2022.\n- [Khrulkov et al. 2022] Khrulkov V, Ryzhakov G, Chertkov A, et al. Understanding ddpm latent codes through optimal transport[J]. arXiv preprint arXiv:2202.07477, 2022.\n- [Lavenant et al. 2022] Lavenant H, Santambrogio F. The flow map of the fokker–planck equation does not provide optimal transport[J]. Applied Mathematics Letters, 2022, 133: 108225.\n- [Karras et al. 2022] Karras T, Aittala M, Aila T, et al. Elucidating the design space of diffusion-based generative models[J]. Advances in neural information processing systems, 2022, 35: 26565-26577.\n- [Song et al. 2021] Song Y, Sohl-Dickstein J, Kingma D P, et al. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "**Strengths:**\n\nThe authors provide relatively detailed proofs and comprehensive experimental results, which are convenient for understanding.\n\n**Weakness:**\n\nWhile the Gaussian setting allows for exact analytical results, real-world data distributions are highly non-Gaussian. Thus, while the theoretical insights are valuable, their direct applicability to practical generative modeling tasks remains uncertain.", "other_comments_or_suggestions": "1. In Appendix H, the authors study the covariance matrix of the ADSN distribution, and provide the computation of the empirical Wasserstein error in the ADSN covariance diagonalization basis. These proofs are generally not problematic, but there are many small typos and errors:\n\n- $dt$ is missing in equation (23) in the term $\\beta_t e^{B_t} x_t$. \n\n- All instances of $\\xi \\in \\mathbb{R}^{M\\times N}$ and $\\omega \\in \\mathbb{R}^{M\\times N}$ should be replaced with $\\xi\\in M\\times N$ and $\\omega\\in M\\times N$, respectively.\n- The notations $v^{j,\\xi}$ and $\\hat{v}^{j,\\xi}$ are used alternately after equation (107), which constitutes the abuse of notations that needs to be rectified and every notation needs to be thoroughly rechecked.\n- Equation (101) should be replaced with  \n$$ \\widehat{\\Sigma v}_{i}=\\hat{t_{i}}\\odot \\left( \\bar{\\hat{t_1}}\\odot \\hat{v_1}+\\bar{\\hat{t_2}}\\odot \\hat{v_2}+\\bar{\\hat{t_3}}\\odot \\hat{v_3} \\right) $$\notherwise the dimensionality of the vectors does not match. In addition, $\\widehat{\\Sigma v_{i}}$ and $\\widehat{\\Sigma v}_{i}$ are difficult to be distinguished for readers who are new to them, which can cause difficulties in understanding this proof. Hence a clear explanation is needed.  Furthermore, this formula depends on a fact, where $\\widehat{u*\\overleftarrow{v}}=\\hat{u}\\odot \\bar{\\hat{v}}$ ($\\overleftarrow{v}$ refers to reversing the arrangement of elements in a vector $v$​). A detailed explanation is needed in order to help readers understand the proof. \n- $\\lambda_{1}^{\\xi}$ should be $\\lambda_{j}^{\\xi}$ in equation (116).\n- In equation (121)~(124), the coefficient should be $MN$, not $3MN$​. Hence the empirical Wasserstein distance in equation (125) is indeed wrong.\n\n2. There are some word errors in the paper:\n\n- *konwn* should be *known* in Proposition 4.\n- *inependent* should be *independent* in equation (120).", "questions_for_authors": "1. I am curious about whether there truly have been no prior works exploring the explicit solutions of the VP-SDE and PF-ODE in the Gaussian data setting, as their solutions are not particularly complex and can essentially be found in any standard textbook on stochastic processes.\n2. For the eigenvalue contribution to the Wasserstein error, almost in all the numerical schemes, the function $|\\sqrt{\\lambda^{scheme}}-\\sqrt{\\lambda}|$ have a zero point near $\\lambda=1$. And this function manifests convexity in a small neighborhood of every zero point. I wonder if there are theoretical explanations.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qvwg2wbmSB", "reviewer": "Reviewer_fM2b", "summary": "This paper presents a mostly-theoretical study of four sources of error in diffusion sampling, in the special case of multivariate Gaussian distributions. First they derive exact solutions for the diffusion reverse SDE and ODE for a Gaussian, and then apply these to study the initialization, discretization, truncation, and score approximation error (the latter is actually studied empirically).\n\n**Update after rebuttal**\n\nThank you for the response, including the nonzero mean results, which I think are a nice addition to the paper. I maintain my initial positive rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your feedback.\n\n\n> L129 “sensible” = “sensitive”?\n\nThank you for noticing this typo, which we have corrected.\n\n> L148: Assumption 1: I am curious why you need to assume a centered (mean = 0) Gaussian? Could the results be extended to arbitrary mean? (Are you not doing this because it is trivial, or because it is hard?)\n\nWe assumed a zero mean Gaussian for two reasons: first, to align with the machine learning framework, where data is typically normalized as a preprocessing; second, to simplify the notation of the equations. Nonetheless, all the results can be trivially extended to the case of a nonzero mean. Essentially, the solutions are shifted, and the 2-Wasserstein error includes an additional bias norm component. For the sake of completeness, if $p_\\text{data} = \\mathcal{N}(\\boldsymbol{\\mu},\\boldsymbol{\\Sigma})$,\n\n\\begin{equation}\np_t = \\mathcal{N}(e^{-B_t}\\boldsymbol{\\mu},\\boldsymbol{\\Sigma}_t)\n\\end{equation}\n\nand the solution of Proposition 2 becomes\n\\begin{equation}\n\\boldsymbol{y}\\_t =  e^{-B_{T-t}}\\boldsymbol{\\mu}+ e^{-(B\\_T - B\\_{T-t}\n)}\\boldsymbol{\\Sigma}\\_{T-t}\\boldsymbol{\\Sigma}^{-1}\\_T(\\boldsymbol{y}\\_0-e^{-B_T}\\boldsymbol{\\mu}) +\\boldsymbol{\\xi}\\_t, \\quad 0 \\leq t \\leq T\n\\end{equation}\nwhere $\\boldsymbol{\\xi}_t$ is defined in Equation (13).\n\nSimilarly, the solution of Proposition 3 becomes\n$$\n\\boldsymbol{y}\\_t =  e^{-B_{T-t}}\\boldsymbol{\\mu}+\\boldsymbol{\\Sigma}^{-1/2}\\_T \\boldsymbol{\\Sigma}^{1/2}\\_{T-t}(\\boldsymbol{y}\\_0-e^{-B_T}\\boldsymbol{\\mu}), \\quad 0 \\leq t \\leq T.\n$$\n\nWe propose to discuss this nonzero mean case in a dedicated appendix to clarify that there is no difficulty to extend our result to this setting.\n\n\n\n> L435 \"unable to derive exact solutions to the backward SDE or the flow ODE under GMM assumption\" -- [Chidambaram 24] contains some ODE results for GMM that may be helpful/of interest (not exact solutions but bounds)\n\nThank you for this interesting reference that further strengthens the arguments of our sixth section *Discussion and limitations*, we propose to cite it.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No", "experimental_designs_or_analyses": "No", "relation_to_broader_scientific_literature": "For multivariate Gaussian data distributions, the solution of the forward SDE (Prop 1) and the score (eq 10) are of course well-known. \n\nThe exact solutions for backward SDE and probability flow ODE in the special case of multivariate Gaussians under VP schedule (Prop 2 and 3) have not been previously published as far I know. In related work, [Bradley Nakirran 24] (Appendix A.5) gives exact solutions to backward SDE and probability flow ODE for 1D Gaussians under VE schedule. Also, [Chidambaram 24] contains some similar ODE analysis for Gaussian mixtures. (However, I believe this paper may have appeared earlier on arXiv and also the works cover slightly different cases.) The Wasserstein bounds are also new as far as I know.\n\n[Bradley Nakirran 24] Classifier-Free Guidance is a Predictor-Corrector (https://arxiv.org/pdf/2408.09000)\n[Chidambaram 24] What does guidance do? A fine-grained analysis in a simple setting (https://arxiv.org/pdf/2409.13074)", "essential_references_not_discussed": "Please see Relation To Broader Scientific Literature for some related works.", "other_strengths_and_weaknesses": "Strengths: \nThe paper is clear and well-written. I appreciate the project of carefully identifying and analyzing various sources of error. I am also happy to see the reverse SDE and ODE solutions for multidimensional Gaussians clearly laid out.\n\nWeaknesses:\n--Single Gaussian data distribution is a very special case that may not tell us much real data distributions. At a minimum, Gaussian Mixtures (GM) at least capture the notion of multimodality, which is a key aspect of real data.\n-- Conclusions that e.g. SDE is better than ODE, Heun is best sampler, etc. are not very surprising.\n-- Score approximation is noted as being the most important source of error but theoretical analysis not available, which is a bit of a disappointment given the stated project of the paper (I understand that this is difficult).", "other_comments_or_suggestions": "L129 “sensible” = “sensitive”?\n\nL148: Assumption 1: I am curious why you need to assume a centered (mean = 0) Gaussian? Could the results be extended to arbitrary mean? (Are you not doing this because it is trivial, or because it is hard?)\n\nL435 \"unable to derive exact solutions to the backward SDE or the flow ODE under GMM assumption\" -- [Chidambaram 24] contains some ODE results for GMM that may be helpful/of interest (not exact solutions but bounds)", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Meta-Review
This paper offers a rigorous theoretical investigation into diffusion models under the assumption of Gaussian data distributions. The authors derive exact solutions for both the reverse SDE and probability flow ODE in the Gaussian case and use these to precisely characterize and quantify multiple types of error that arise in diffusion model implementations: initialization, discretization, truncation, and score approximation. By expressing these errors in terms of the Wasserstein-2 distance, the paper contributes a rare instance of fully analytical insight in a field often dominated by heuristic and empirical evaluations.
Strengths
* Novel Theoretical Contribution: The paper provides, for the first time to the best of reviewers' knowledge, exact Wasserstein-2 error analysis for various numerical solvers applied to diffusion models with Gaussian data. This includes closed-form solutions to the reverse SDE and ODE.
* Clarity and Structure: Reviewers appreciated the writing quality, clarity of the mathematical exposition, and the careful decomposition of different error types.
* Relevance: Despite its limited scope, the theoretical rigor and clarity make the paper a solid contribution to the understanding of diffusion models, particularly in how practical implementations can deviate from theoretical expectations.
Weaknesses
* Scope Limitation: All reviewers noted that the focus on Gaussian distributions, while enabling exact results, limits the direct applicability to real-world scenarios. Although the authors acknowledge this and motivate the Gaussian setting as a tractable and illustrative case, some reviewers (notably Rj1X) questioned the novelty and utility of the framework in light of existing numerical analysis literature.
* Score Approximation Analysis: While the authors claim that score approximation is a dominant source of error, the corresponding experiments were not seen as convincingly isolating or quantifying this effect. This weakened the empirical support for some of the claims.
* Inconsistent Findings: Reviewer Rj1X pointed out inconsistencies between theoretical conclusions (e.g., superiority of Heun’s method) and empirical outcomes (e.g., better results with EM sampler), highlighting a possible mismatch between the analytical setting and real-world approximations.
Reviewer Consensus
* Positive Reviews (3 out of 5): Reviewers cXeJ, CDeZ, and fM2b gave generally favorable reviews (ratings of 4, 2 then 3 after rebutal, and 3, respectively), commending the theoretical clarity and contribution, and emphasizing that the paper is within the expected scope of ICML theory tracks.
* Critical Review (Rj1X): One reviewer was significantly more critical (rating of 1), arguing that the insights were neither novel nor useful beyond well-known numerical analysis results and that the paper did not justify its relevance to non-Gaussian settings.
* Rebuttal Engagement: The authors responded in detail to all reviewers and made useful clarifications (e.g., extending results to non-zero mean Gaussians), though these were not sufficient to shift the most critical reviewer's stance.
 While the theoretical scope is narrow and broader generalization remains future work, the paper succeeds in delivering precise and novel insights within its chosen domain. It should be of interest to researchers in diffusion models and numerical analysis of generative models, particularly those seeking principled understanding of discretization behavior and sampling accuracy. Final recommendation: Accept.
Decision: Accept (poster)