An analysis of the noise schedule for score-based generative models
Authors: Stanislas Strasman
Antonio Ocello
Claire Boyer
Sylvain Le Corff
Vincent Lemaire
Venue: Accepted by TMLR ()

Abstract: Score-based generative models (SGMs) aim at estimating a target data distribution by learning score functions using only noise-perturbed samples from the target. Recent literature has focused extensively on assessing the error between the target and estimated distributions, gauging the generative quality through the Kullback-Leibler (KL) divergence and Wasserstein distances.  
Under mild assumptions on the data distribution, we establish an upper bound for the KL divergence between the target and the estimated distributions, explicitly depending on any time-dependent noise schedule. Under additional regularity assumptions, taking advantage of favorable underlying contraction mechanisms, we provide a tighter error bound in Wasserstein distance compared to state-of-the-art results. In addition to being tractable, this upper bound jointly incorporates properties of the target distribution and SGM hyperparameters that need to be tuned during training. Finally, we illustrate these bounds through numerical experiments using simulated and CIFAR-10 datasets, identifying an optimal range of noise schedules within a parametric family.

Published in Transactions Machine Learning Research (12/2024)
An analysis of the noise schedule for score-based gener- ative models
Stanislas Strasman', Antonio Ocello", Claire Boyert Sylvain Le Corfil, and Vincent Lemaire!
(LPSM, Sorbonne Université, UMR CNRS 8001, Paris, France.
"CMAP, École Polytechnique, Institut Polytechnique de Paris, France.
+LMO, Université Paris-Saclay, UMR CNRS 8628, Orsay, France.
IUF, Institut Universitaire de France.
Reviewed OpenReview:
B1YIPa0Fx1
Abstract
Score-based generative models (SGMs) aim at estimating target data distribution by learning score functions using only noise-perturbed samples from the target. Recent literature has focused extensively
assessing the error between the target and estimated distributions, gauging the generative quality through the Kullback- Leibler (KL) divergence and Wasserstein distances. Under mild assumptions on the data distribution, we establish an upper bound for the KL divergence between the target and the estimated distributions, explicitly depending on any time-dependent noise schedule. Under additional regularity assumptions, taking advantage of favorable underlying contraction mechanisms, we provide a lighter error bound in Wasserstein distance compared to state-of-the-art results. In addition to being tractable, this upper bound jointly incorporates propertics of the target distribution and SGM hyperparameters that need to be tuned during training. Finally, we illustrate these bounds through numerical experiments using simulated and CIFAR- datasets, identifying an optimal range noise schedules within parametric family.
1 Introduction
Recent years have seen impressive advances in machine learning and artificial intelligence, with one of the most notable breakthroughs being the success of diffusion models, introduced by Sohl-Dickstein et al. (2015). Diffusion models in generative modeling refer to class of algorithms that generate new samples given training samples of an unknown distribution Tdata This method is now recognized for its ability to produce high-quality images that appear genuine to human observers (see e.g., Ramesh et al., 2022, for text-to-image generation). Its range of applications expanding rapidly, yielding impressive outcomes areas such as computer vision (Li el al., 2022; Lugmayr el. al., 2022) natural language generation (Gong et al,, 2023), among others, see Yang et al. (2023) for comprehensive overview of the latest advances in this topic.

---

Published in Transactions Machine Learning Research (12/2024)
Score-based generative models (SGMs). Generative diffusion models aim at creating synthetic instances of target distribution when only genuine sample (e.g., dataset of real-life images) is accessible. It is crucial to note that the complexity of real data prohibits a thorough depiction of the distribution Adata through standard non-parametric density estimation strategies. Score-based Generative Models (SGMs) are probabilistic models designed to address this challenge using two main phases. The first phase, the noising phase (also referred to as the forward phase), involves progressively perturbing the empirical distribution by adding noise to the training data until its distribution approximately reaches an casy-to-sample distribution Too- The second phase involves learning to reverse this noising dynamics by sequentially removing the noise, which is referred to as the sampling phase (or backward phase). Reversing the dynamics during the backward phase would require in principle knowledge of the score function, i.c., the gradient of the logarithm of the density at each time step of the diffusion. To circumvent this issue, the score function is learned based on the evolution of the noised data samples and using a deep neural net work architecture. When applying these learned reverse dynamics to samples from Too, we obtain a generative distribution that approximates daata-
Related works. Significant attention has boen paid to understanding the sources of crrors that allect the quality of data generation associated with SGMs (Block et al., 2020; De Bortoli, 2022; Lee et al., 2022; 2023; Chen et al., 2023a;b). In particular, a key area of interest has been the derivation of upper bounds for distances or pseudo-distances between the training and generated sample distributions. Note that all the mathematical theory for diffusion models developed so far covers general time discretizations of time-homogeneout SGMs (see Song and Ermon, 2019, in the variance-preserving case), which means that the strength of the noise is prescribed to be constant during the forward phase. De Bortoli et al. (2021); Chen (2023) provided upper bounds in terms of total variation, by assuming smoothness properties of the score and its derivatives. On the other hand, the upper bounds total variation and Wasscrstein distances provided by Loe et al. (2023); Gao et al. (2023) also require smoothness assumptions on the data distribution, either involving non-explicit constants, or focusing iteration complexity sharpness. More recently, Conforti et al. (2023); Benton et al. (2024) established an upper bound in terms of Kullback Leibler (KL) divergence avoiding strong assumptions about the score regularity, and relying on mild conditions about the data distribution (e.g., assumed to be of finite Fisher information w.r.t. the Gaussian distribution). Regarding hime-inhomogeneuus SGMs, the central role of the noise schedule has alrcady been exhibited in numerical experiments, see for instance Chen (2023); Nichol and Dhariwal (2021); Guo et al. (2023). However, a rigorous theoretical analysis of it is still missing.
Contributions. In this paper, we conduct a thorough mathematical analysis of the role of the noise schedule in score-based generative models. We propose a unified framework for iime-inhomogeneous SGMs, to conduct joint theoretical analyses in KL and Wasserstein metrics, with state-of-the-art art set of assumptions, using exponential integration of the backward process. In our opinion, these upper-bounds provide numerical insights into proper SGM training.
We establish an upper bound on the Hullbackk-eibbles divergence between the data distri- bution and the law of the SGM. This bound holds under the mildest assumptions used in the SGM literature and explicitly depends on the noise schedule used to train the SGM. The proof follows the same steps as Conforti et al. (2023). However, it requires to establish Leibler upper bound for inhomogeneous forward diffusion which involves
2

---

(OU) process, solution to Published in Transactions Machine Learning Research (12/2024)
determining a non-asymptotic rate of convergence for the mixing time using Fokker-Planck equations and log-Sobolev inequality that depends on the noise schodule, and not only on the diffusion time horizon, see Lemma B.1. In addition, taking into account the backward contraction for the diffusion process (Proposition C.1) provides state-of-the-art results on mixing time convergence for SGM under the Ornstein- Uhlenbeck forward process, whether inhomogencous or not.
By making additional assumptions on the Lipschitz and strong log-concavity propertics of the score function, we establish a bound in terms of Wasserstein distance explicitly depending on the noise schedule. This extends the similar result for the KL in the Gaussian setting. These results are in the same line of work as Bruno et al. (2023); Gao et al. (2023), incorporating to the time inhomogeneous setting a refinement of the mixing time error based on an analysis of the modified score function.
We illustrate, through numerical experiments, the upper bounds obtained in practice in regard of the effective empirical KT. divergences and Wassertein metrics. These simula- tions highlights the relevancy of the upper bound, reflecting in practice the effect of the noise schedule the quality of the generative distribution. Additionally, the simulations conducted provide theoretically-inspee ducibility purposes, the code for the numerical experiments is available at https://githe: Com/Stanin wwwssssaaastteeaaaaa.. ScceeeSSehheeeeeeeeeeeeeeeeeeeee for Saee_Scoreebbb based Generative Se
guidelines for improving SGM training. For repro-
2 Mathematical framework for SGMs
Forward process. Denote as 3: [0, 7 Ry0 the noise schedule, assumed to be continuous and non decreasing. Although originally developed using a finite number of noising steps (Sohl- Dickstin al., 2015; Song and Ermon, 2019; Ho et al., 2020; Song al., 2021), most recent approaches consider time-continuous noise perturbations through the use of stochastic differential equations (SDFs) (Song al., 2021). Consider, therefore, forward process given by
dx,
B(t) X dt + /B(t)BB, Xo Tdata 20
(1)
We denote by Pe the density of X, at time t E (0, Note that, up to the time change + So (ss//2ds, this process corresponds to the standard
dx,= 1 x VddB Xo Taaaa
see, e.g., Karatzas and Shreve (2012, Chapter 3). Due to the linear nature of the drift with respect to (Xt),, it is well-known that exact simulation can be performed for this process (Section 1.1.2). The stationary distribution Too of the forward process is the Gaussian distribution with mean 0 and variance o² LL In the literature, when ((t) is constant equal to 2 (meaning that there is no time Piriancve-rrsrvving SDE (VPSDE, De Bortoli
change), this diffusion process is referred to as the et al., 2021; Conforti et al., 2023; Chen et al., 2023b), leading to the so-called Denoising Diffusion Probabilistic Models (DDPM, Ho al., 2020). Understanding the effects of the general diffusion model (1), in particular when reversing the dynamic, remains challenging problem, to which we devote the rest of our analysis.
3

---

Published in Transactions Machine Learning Research (12/2024)
Backward process.
The corresponding backward process is given by
dx = nlt, x,)t + VB(t)BB. with
{
B(1) = 3(T
A(t) ntt, 20
X B(t) log pr- (x.)
X ~ Toa Too
We consider the marginal time distribution of the forward process divided by the density of its stationary distribution, introducing
Ve E p=(x)= (x)))
22)
where Ppa2 denote the density function of oo Gaussian distribution with mean and variance La Thus, the backward process can be rewritten
ax. = n (1, de + V B(t)dB. Xo Teo:
(3)
+ V(1) lo (x.). The benefit of using the renormalization Dr in our analysis results in considering the backward equation as a perturbation of an OU process. This trick is crucial to highlight the central role of the relative Fisher information in the performance of the SGM. It has already been used by Conforti et al. (2023).
where 7(1,, x1)
B0 20
Score estimation. Simulating the backward process means knowing how to operate the score. However, the (modified) score function V log):(x)= = V po +w/o² cannot be evaluated directly, because depends on the unknown data distribution. To work around this problem, the score function V log Ps ncods to be estimated. In Hyvärinen and Dayan (2005), the authors proposed to estimate the score function associated with a distribution by minimizing the expected L LL-squared distance between the true score function and the proposed approximation. In the context of diffusion models, this is typically done with the use of deep neural network architecture [0, T] Rd + RRd parameterized by A E O, and trained to minimize:
so (T.X.) - V log pr
(x, (2.)
Lexplici (0)= =
(4)
with T ~ ((, T) independent of the forward process (X e)20. However, this estimation problem
still suffers from the fact that the regression target is not explicitly known. A tractable optimization problem sharing the same optima be defined though, through the marginalization over data of p- (see Vincent, 2011; Song et al., 2021):
Leccore (0) EE 11 (T,xx)) V logge (x (x=1x))1²)
(5)
where 7 is uniformly distributed on [0, 7], and independent of X, ~ data and X, Pr (-1X)). This loss function is appealing as it only requires to know the transition kernel of the forward process. In (1), this is a Gaussian kernel with explicit mean and variance.
Discretization. Once the score function is learned, remains that, in most cases, the backward dynamics no longer enjoys lincar drift, which makes its exact simulation challenging. To address
4

---

Published in Transactions Machine Learning Research (12/2024)
this issue, one solution is to discretize the continuous dynamics of the backward process. In this way, Song et al. (2021) propose maler-Maruyama (EM) discretization scheme in which both the drift and the diffusion coefficients discretized recursively (see (50)). The Euler Exponential Integrator (EI, see Durmus and Moulines, 2015), as already used in Conforti et al. (2023), only requires to discretize the part associated with the modified score function. Introduce (t x) = so(t, x) + x/0² and consider the regular time discretization to S t1 S..... S. T. Then, (X]))e is such that, for E [t&, tk+1);
axo ((t) (- 1 x + 50 (T- 12 xx. 20
dt 3(t)dB, X Tec
(6)
This scheme can be seen as refinement of the classical EM one as it handles the linear drift term by integrating it explicitly. In addition, (X)))e(te.....t) can be sampled exactly, see Appendix A. We consider therefore such a scheme in our further theoretical developments.
3 Non-asymptotic Kullback-Leibie bound
In this section, we provide thcoretical analysis of the effect of the noise schedule used when training SGM. Its impact is scrutinized through bound the KL divergence between the data distribution and the generative one.
Statement. The data distribution Taata is assumed to be absolutely continuous with respect to the Gaussian measure To. Define the relative Fisher information I (TTataa/TT)) by
/ V log
dddata do
I (Taata (To)
dTdaaa
and consider the following assumptions.
H1 The noise schedule 3 is continuous, positive, non decreasing and such that fo 3(t)dt = co.
H2 The data distribution such that I( daTatall Tx) < co.
H3 The NN parameter O and the schedule satisfy
B(t) || (s (T- - 30 (I 1, (r. ftaat at/ <00,
E
exp
with x)= V log (xr) and ýe defined in (2).
Assumption H1 is necessary to ensure that the forward process converges to the stationary distribution when the diffusion time tends to infinity. Assumption H2 is inherent to the data distribution, as involves only the L'-integrability of the score function. Such a kind of hypothesis has already been considered in the literature, see Conforti et al. (2023). We stress that, in this section, we do not require extra assumptions about the smoothness of the score function. Lastly, Assumption H3 is condition the approximation of the score by the neural network šø, weighted by the level of noise in play. We are now in position to provide an upper bound for the relative entropy between the distribution * (0.0) of samples obtained from (6), and the target data distribution Tenta-
5

---

Published in Transactions Machine Learning Research (12/2024)
Theorem 3.1. Assume that H1, H2 and H3 hold. Then,
KL [Taata] AN x(8.00)
- 13 + (0, (8),
where
(B) KL (Tdatal (ooaaaa/TToo)0 exp
0² ['" B(s)ds}
1
N-1
I- 8())dt,
(0, (3) = Σ
E
V Pr pr (Kr-ts) - 80 50(T (T- u, X xr
0=
(B) 2 I (Tdatal TxO.
with := (I... (+x 1x-1) small enough and to : 0.
The obtained bound is composed of three terms, all depending on the noise schedule, through either its integrated version over the diffusion time, or its final value at Lime The result was derived for the El discretization scheme, but it could be adapted to the Euler scheme up to minor technicalities. Remark also that using Pinsker's incquality, the obtained bound could be transferred in terms of total variation.
Dissecting the upper bound. The upper bound of Theorem 3.1 involves three different types of error that affect the training of an SGM. The term gKT. represents the mining time of the OU forward process, arising from the practical limitation of considering the forward process up to a finite time T. Indood, gKL is shrinked in EKL corresponds Assumption H2. The second term EKT. corresponds to the approzimation error, which stems from the use of a deep neural network to estimate the score function. Note that if we assume that the error of the score approximation is uniformly (in time) bounded by Ma (sce De Bortoli et al., 2021, Equation (8)), the term EKL admits crude bound Mg fo" 3(t)df, with the disadvantage of exploding when T +00. Otherwise, by considering Conforti et al. (2023, Assumption H3), one can make this bound finer and finite, by balancing the quality of the score approximation, the discretization grid and the final time T. Finally, EKLL is the discretization ervor of the EI discretization scheme. This last term vanishes as the discretization grid is refined (i.e., h + 0).
when T grows to infinity. Note that the multiplicative term the KL divergence between ddata and Too which is ensured to be finite by
Comparison with existing bounds. Under perfect score approximation,
and infinitely precise discrctization (i.c., when E (B) 0), we recover that the Variance Preserving SDE (VPSDE, De Bortoli et al., 2021; Conforti et al., 2023; Chen et. al., 2023b) converges exponentially fast to the target distribution. Beyond this idealized setting, the bound established in Theorem 3.1 recovers that of Conforti et al. (2023, Theorem 1) when choosing 3(1) 2, o² 1, T= 1, and using a discretization step size h < 1.
Refined analysis of the mixing time crror Still assuming "perfect score approximation" and infinitely precise discretization (i.e., (0, = ( = 0), one can assess the sharpuess of the term E EKL (ß) in the upper bound of Theorem 3.1. In particular, when restricting the data distribution to be Gaussian N(po, Ep), one can exploit the backward contraction assuming that
6

---

Published in Transactions Machine Learning Research (12/2024)
Amax(2o) o², where Amax (2o) denotes the largest eigenvalue of Co. In this specific case, we can obtain a refined version for EKL. (scc Proposition C.1), given by
2 KL (TTaatal) S KL (Taatall exp (- 02 / B(s)ds).
(7)
where (Qe) the Markov semi-group associated with the backward SDE. This idea is exploited in Section 4 to establish Wasscrstein bounds for more general data distributions than Gaussian, but requiring extra regularity of the score.
4 aon-asymptotic Wasserstein bound
In the literature, much allention is paid lo derive upper bounds with other metrics such as the W2 distance, which has the advantage to be a distance and to have sisier-tohaandee and implementable estimators. In Lee al. (2023), the authors obtaín control for the Wasserstein and total variation distances. However, those results rely on additional assumptions on Adata (which is assumed to have bounded support for instance in De Bortoli (2022))-
Regularity assumptions. We consider extra regularity assumptions about the modified marginal density Pt at any time of the diffusion.
H4 (i) For all t 2 0, there exists C, 0 such that for all x. y RR
(V log (v) Voog y) 2 C. C+Ilæ yll
(ii) For all t 2 0, there exists L 2 0 such that V log Pt is Li-Lipschitz continuous.
The strong log-concavity (i) (see, e.g., Saumard and Wellner, 2014) plays a crucial role in terms of contraction of the backward SDE. Classical distributions satisfying H4(1) include logistic densities restricted to compact set, or Gaussian laws with positive definite covariance matrix, see Saumard and Wellner (2014) for other examples. We observe, notably, that when the density of the data distribution is log-concave, this property propagates within the probability flow (Pr)osesT (see Proposition D.1). Similar conclusions can be drawn regarding the Lipschitz continuity of the score (Proposition D.2). This property formalized the Lemma 4.1 for Gaussian distributions.
Lemma 4.1. Assume that Taata is Gaussian distribution N Eo); such that Eo is invertible and Amax (Eo)= < o². Let m == exp(- Jo ((s)).. Then, the probability flow pe given by (1) initialized at Tdata is Ce-strongly log concave, with
mî (0² Amax (Eo)
C
m Xmxax (Eo) + 0² m??
In addition, the associated score V log is L, Lipschite continuous with
1
L, = min
0² (1- mz) Amin(2o)
0
This result, restricted to the Gaussian case, sets the focus on the importance of calibrating the parameter o² depending on the covariance structure of the data distribution, in order enhance strong log concavity of the probability flow through the diffusion.
7

---

Published in Transactions Machine Learning Research (12/2024)
Error bound. To establish 2- Wasserstein bound explicitly depending on the noise schedule, consider the following additional assumptions, respectively about uniform approximation of the score, and Lipschitz continuity in time of the renormalized score.
H5 There exists e 2 such that
sup NE(
|| (T - th, X²x) öy (T - tk, Xxx Se
H6 For regular discretization {4n, 0 < k S N of [0, I] of constant step size h, there exists M 2 such that
sup AE (O... N- 1J & S1Sta11
sup (T - t, x) - 3 (T - tke x)11,, S
We now have all the ingredients to present our thcoretical guarantee in terms of Wassorstein distance. Theorem 4.2. Assuming H4, H5 and H6 and that the time step h is small enough, it holds that
W2 Tdata: =(0.01) s = Ex (0.
(8)
(-1² T B(t)
with
( W2 (Tdatar Tool exp
G dt
c/as (0,3)
N-1
Σ
T,5(()a))
V2hB(T)
hB(T)
+ fra 2Tn B
(lat)
20
+ +TT((T) + MAT (1+2B),
B [E |1 + odd) for E [0, Le LT-t.
In Theorem 4.2, we exploit the contraction entailed by Assumption H4(i) of the backward diffusion processes on top of that of the forward phase. This idea leads to an improvement of all the existing bounds in Wasserstein metrics, by refining their mixing time term. The previous result can be established when the target distributioo has Lipschitz continuous score and strongly log-concave: by propagating these properties, the constants L, and C, can be characterized as a function of Lo and Co (see Propositions D.1 and D.2). The propagation of the log-concave property was also established in Saremi et al. (2023).
Corollary 4.3. Assume that V Pogaa is Lo-Lipschilz, that log Pasza is Co-strongly concuve such that Co 1/0². Under Assumption H5 and H6, with a time step small enough,
W/ Tdata; x) < W2 (Matta; Too) exp (-/" (030
3(1) (1 (1+C, at) avT o2h ETB(T)
with e LTB(TTT / and c2 3(TYT (Lo (1/(20²) + 2Lo) B(T) + M(1 + (1)+)
This provides an easy-to-hande handle upper bound in Wasserstein distance, encompassing the three types of error (e.g., mixing time, score approximation and discretization error), for Lipschitz scores and strongly-log concave distributions. We remark that also cxhibits an extra term in vh compared to the more general KL. bound obtained under milder assumptions. Note however that this term in line with what can be found in the literature for Wasserstein bounds for SDE approximation (see Alfonsi et al., 2015).
8

---

Published in Transactions Machine Learning Research (12/2024)
Discussion and comparison with other works. Theorem 4.2 requires more stringent assumptions on the regularity of the score function than Theorem 3.1. However, these assumptions are not specific to our setting. In particular, the strong concavity assumption has proven to be a key property for the fast convergence of sampling algorithms (see Dalalyan, 2017; Durmus and Moulines, 2017; Dwivedi et al., 2019). While this is a strong assumption to require on the data density, this can be mitigated.
In Benton et al. (2024), the authors propose quantitative bounds for the Kullback-Leiin divergence only assuming finite second moments of the data distribution and do not use any smoothness assumption. The authors use early stopping and stop the backward sampling at small time 8 >>> 0 to avoid the score explosion in the neighborhood of O. In another line of work, Chen et al. (2023a) used high-probability bound on the Hessian matrix of Pt to avoid additional assumptions (such as a bounded support) on the data distribution to oblain Kullback-Leiie upper bounds. These works offer promising perspective to obtain Wasserstein bounds under weaker assumptions.
The analysis of the modified score functions in the Gaussian case reveals that by properly adjusting the variance of the stationary distribution of the forward process and rescaling the larget distribution, we can attain the desired properties for the score function. This observation is specific to the Gaussian case one can easily derive values for Co and Lo with the eigenvalues of covariance matrix of the target distribution. However, we would like to strengthen the fact that similar preprocessing has been applied to more complex distributions in Section 5.
5 Evaluation of the theoretical upper bounds
The goal of this section is to numerically illustrate the validity of the theoretical bounds ob- tained in Theorem 3.1 and Theorem 4.2. More preciscly, we aim at unraveling the contribu- tions of each error term of the upper bounds. We consider simulation design where the target distribution is known, and the associated constants of inter- est (i.e., the strong log concavity parameter, the Lipschitz constant, W2 (Taata: ), ), I (oo) or KL (Taata I [Tc)) can be evaluated. The error bounds are assessed for dif- ferent choices of noise schedules of the form
(t) x (eat - 1)/(c=--11
(9)
34
with a IR ranging from 10 to 10 with unit step size. We set T = and adjust schedules that they all start at 8(0) = 0.1 and end at 8(1) = 20 (sce Figure 1). This choice has been made so that when a = 0 the schedule is linear and matches exactly the classical VPSDE implementation (Song and Ermon, 2019; Song et al., 2021).
24
Figure 1: Noise schedule Pa over time for a E (-10, -9. .., 10} with the linear schedule a = 0 shown as dashed line.
5.1 Gaussian setting
Target distributions. We consider the setting where the true distribution Tdata is Gaussian in dimension 50 with mean 1a and different choices of covariance structure:
1. (Isotropic, denoted by x
(1oo)) (iso)) 0..T..-
9

---

Published in Transactions Machine Learning Research (12/2024)
(beterosc)) ((heterose) Bddd a diagonal matrix such that 2. (Heteroscodasticc denoted by 4 ddata
(heterose) 1 for 1 S j S 5, and 2 (hecorros)) 0.01 otherwise.
3. (Correlated, denoted by Taata ( I(corr) Rddd is a full matrix whose diagonal entries are equal to one and the off-diagonal terms are "38 1/vlj 1' for 1 S j S d.
((corr)
SGM simulations. We simulate (B..0) from SGM using the forward process defined in (1) with TN t - (t) for the noise schedule. The score is learned via dense neural network with 3 hidden layers of width 256 over 150 epochs (see Figure 11) trained to optimize Lexplicit (4). This is feasible because the score is analytically derived when Edata is Gaussian (Lemma E.1). Numerical experiments have also been run with the commonly used conditional loss Lscore, without changing the nature of the conclusions, see Appendix F. For backward process simulation, we use an Euler-Maruyams scheme with 500 steps, as being the most encountered discretization in practice (with these discretization steps the difference with Exponential Integrator scheme is minimal as highlighted in the appendix). For each value of a, and each data distribution, we train the SGM using = 10000 training samples.
KL bound. In Figure 2 (top), we compare the empirical KL divergence between Adata and samples from "N to the upper bound from Theorem 3.1. We refer the reader to Appendix E.2.1 for implementation details. For Gaussian distributions, both the bound and KL divergence can be computed using closed-form expressions (sce Lemma E.2 and E.4). In all scenarios the noise schedule significantly impacts the value of (L(Taatal] 1|= and thereby the quality of the learned distribution. Moreover, in all three cases taking into acccount the contraction argument (7) is key to properly align the upper bound trend with the generation results. In all these experiments, the KT. upper bound indicates possible values for a improving over the classical linear noise schedule.
2- Wasserstein bound. In Figure 2 (bottom), we compare the empirical W2 distance between Tdata () to the upper bound from Theorem 4.2. For Gaussian distribut ions, both the
and samples from "x bound and the W2 distance can be computed using closed-form expressions (see Lemma 4.1, E.3, and E.5). For the isotropic case, the proposed W2 upper bound reflects the SGM performances, as already (heterosc)
highlighted by the KT. bound. However, non-isotropic cases, the raw distributions datta and (coer)
dataa do not directly satisfy Assumption 4 (i) when the variance of the stationary distribution is set to 1. Therefore, scaling the distributions in play becomes crucial for the theoretical W2 upper bound to hold. That is why we propose the following preprocessing: train an SGM with centered and standardized samples covariance D(stand) rescaled in turn by a factor 1/(2>max (F(saan)))))))))))) 1/2, This choice ensures that Amax (S(scaled)) < o² 1, for (sscaedd) the resulting covariance matrix,
and thus the strong log-concavity of PO = paata/pa>. We call (B..0) the resulting generative distribution, and the evaluated metrics is adjusted (sce (51)) to ensure fair numerical comparison. After this preprocessing, not only the W22 upper bound of Theorem 4.2 aligns with the empirical performances but the SGM performances can be also boosted (see degraded empirical performances on raw distributions in Appendix E.2.2). This highlights the importance of properly calibrating the training sample to the stationary distribution of the SGM. Note that data normalization does not only enforce the strong log-concavity of the modified score at time 0, but can lower the ratio Lo/Co- To see this, consider the heteroscedastic case, for which Amin (Shetcrossc) //max ( (heterowe)) = 100, whercas Amin (sacalld)) //aallddd))aaa /mmax (I ((scaled)) after scaling. This Gaussian sct-up reveals that data
10

---

Published in Transactions Machine Learning Research (12/2024)
10
-0.4
2.0
16-
-1.50
Uppar bounnd (ML.)
Killor f joadsssn]
-1.25
0.3
dieeggnn
90-
14-
11.00
88
Upper
86-
12
0.50
0.1
84
-0.25
0.00
10
0.00
10
-10
10
-10
10
Valuos of
Values
Values of
120
He fnte (NN)
Inta
0.44
50000
PolPara (MN)
1.55
Upper bound (WC2)
100
(NN)
VPSDE
(ftasii joxnst score)
0.42
222 ciaarrc
Fillf (onact scordt VPEDE
1225 8
80+
00.40
bound
30000
-1000
80-
3
0.38
20000
-0500
40
10 20-
0.2
0.38
5
10
10
10
Values of
Values of
Values of
(a) Isotropic setting
(b) Heteroscedastic setting
(c) Correlated setting
Comparison of the empirical KL divergence (mean + std over 30 runs) (top) and W2 (8.0) (orange) and the related distance (mean + std over 10 runs)(bottom) between ddata and #N upper bounds (blue) from Theorem 3.1 and Theorem 4.2 across parameter for noise schedule Ba, d= 50. In the KL case, the upper bounds in lighter blue are the theoretical upper bounds without taking into the contraction argument (7). We also show the metries for the linear VPSDE model (dashed line) and our model (dotted line) with exact score evaluation.
Figure 2:
renormalization improves the conditioning of the covariance matrix, and thereby the conditioning of SGM training. In particular, this is captured in the upper bound of Theorem 4.2 by limiting the growth of L, and inducing more balanced second term
We now consider varying dimension in (5, 10, 25, 50), and we compare the empirical W2 distance obtained by (i) 3o the classical VPSDE (Song et al., 2021), with a linear noise schedule (i.e., a = 0), (ii) Bens the SGM with cosine schedule (Nichol and Dhariwal, 2021), and (iii) 8.. the SGM with parametric schedule with a = a* approximately minimizing the upper bound from Theorem 3.1. In Figure 3, we observe that the SGMs run with Bar consistently outperforms those run with lincar schedule 3n slightly improving the data generation quality. It displays lower average W2 distances between Tdata and the generated sample distribution, but also reduces the standard deviation of the resulting W2 distances yielding more stable generation (sce Table 2). These performances are comparable to, and often surpass, those achieved with state-of-the-an schedules like the cosine schedule, particularly in higher dimensions.
5.2 More general target distributions
Beyond Gaussian distributions, numerical analysis in terms of KL divergence is not tractable standard estimators of the KL terms do not scale well with dimen-
11

---

Published in Transactions Machine Learning Research (12/2024)
A- Urear achocds
schoctle
Limarsched/le schodue R." (W2 bound)
0.25
Coder scheccde
(A2 bound)
0.35
R. (A2 bound]
W5 ddlanie
0.20
0.30
0.25
0.10
0.15
0.08
0.10
10
25
50
5
10
25
50
10
26
(a) Isotropic setting
(b) Rescaled heterosc. setting (c) Rescaled correlated setting
Figure 3: Comparison of the empirical W2 distance (mean value + std over 10 runs) between ddata and the generative distribution *(5.0) across varions dimensions. The distributions compared include SGMs with different noise schedules: 3a (blue solid), So (yellow dashed), and Boos (orange dotted).
sion. On the contrary, there exist compuuaationally-efcciee estimators of Wasser- (Flamary et al., 2021).
stein distances, as for instance the sliced W2 estimate
We use the latter assess the relevancy of Theo- rem 4.2 when the larget distribution corresponds a 50-dimensional Funnel distribution defined as: Tdata (x) = Pa (x)) II; (exp(2br,) (Iz)) with = 1 and = 0.5 (see Section E.2.3 for more details and additional experiments on a
Parem, sched. f. Cosine sched, frns
0.35
Upper bound (W2)
25-
Slicad-WY 12 distance
0030
20
Gaussian mixture model). As previously, the
-.25
samples are standardized and rescaled. Fig- ure 4, empirical results demonstrate that the minimum of the upper bound closely aligns with that of the empirical sliced 2- W-Wssersteinn dis- lance between the simulated and training data. Moreover, implementing SGM with the optimal parameter a yiclds consistent improvements of the data generation quality across different met- rics w.r.t. classical noise schedule competitors (linear or cosine). These experiments not only support the relevance of the theoretical upper bound beyond the assumptions required in Section 4, but also the validity of theoretically-inspii data preprocessing for improving SGM training with arbitrary target distributions.
15-
0.20
-10
10
20
Values of
Figure 4: Upper bound and sliced 2- Wasserstein distance on a Funnel dataset in dimension 50.
When dealing with high-dimensional real-world datasets, directly evaluating our theoretical up- per bounds (Theorems 3.1 and 4.2) becomes more challenging because relevant. quantities (dis- tances and constants) are either poorly estimated or unavailable. As first step toward real data, we evaluate the impact of the noise schedule on the sampling quality of models pre- trained using CIFAR-10 dataset. In Figure 5, we display the FID score with 50,000 generated samples using Euler-Maruyama discretization scheme for various noise schedules drawn from the parametric family in Equation (9). Additional implementation details are available in Ap- pendix F.3. Although the assumptions underpinning our results cannot be verified in this set- ting, the empirical performance trends mirror closely those observed in the simulated settings.
12

---

Published in Transactions Machine Learning Research (12/2024)
This consistency highlights that analyzing and optimizing noise schedules could be promising direction for improving (Gseb generation in more complex scenarios.
HID Score param. sched.
FID Score cosine sched. Bus.
FID Scones
6 Discussion
In this paper, we propose unified framework to analyze the impact of the noise schedule for time- inhomogeneous SGMs, providing upper bounds in KL and Wasserstein metrics. The KL upper bound follows the steps of recent. works using the mildest assumptions used the SGM literature. We also provide an improved upper bound in the Gaussian setting with numerical experiments highlighting the impact of the backward contraction of the forward noise process. Following Bruno et al. (2023); Gao et al. (2023), under additional assumptions on the Lipschitz and strong log-concavity properties of the score function, upper bounds for the Wasserstein distance. This bound highlights the role of the noise schedule and provides a detailed analysis based on the modified score function. Our results are supported by numerical experiments in simple settings to highlight the several terms the upper bounds and the role of the noise schedule. There are many perspectives to this work. Studying multi-diieensionll noise schedules is of particular interest. Indeed, they could be useful to understand how to deal with target distributions with complex covariance structures, and thereby alternative solution data normalization issues. Establishing upper bounds for Wasserstein distances under milder assumptions remains exciting open problem, which would shed light on the performances and limitations of score-based generative models. A specific perspective would be to adapt our result using carly-stopping to avoid the explosion of error terms the neighborhood of 0 and to provide other assumptions lo control the corresponding error close to
-200
2.0
Figure 5: FID Scores using 50,000 generated
samples for the parametric and cosine schedules (CIFAR-10 dataset).
establish
Acknowledgements
We would like to thank Gabriel Victorino Cardoso for his valuable insights and thoughtful help the numerical experiments involving real-world datasets.
Antonio Ocello was funded by the European Union BRMC-222-SYG-OCEAA--OLLO and opinions expressed are however those of the author only and do not necessarily reflect those the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
Views
References
Aurélien Alfonsi, Benjamin Jourdain, and Arturo Kohatsu-Higa. Optimal transport bounds between the lime-marginals of a multidinensional diffusion and its euler scheme. Electronic Journal of Probability, 2015.
Dominique Bakry, Ivan Gentil, Michel Ledoux, et al. Analysis and geometry of Markov diffusion operators, volume 103. Springer, 2014.
13

---

Published in Transactions Machine Learning Research (12/2024)
Paolo Baldi. Stochastic Calculus. Springer International Publishing AG, edition, 2017. ISBN 978-3319622255.
Joe Benton, Valentin De Bortoli, Arnaud Doucet, and George Deligiannidis. Nearly d-linear convergence bounds for diffusion models via stochastic localization. In The Twelfth International Conference Learning Representations, 2024.
Adam Block, Youssef Mrouch, and Alexander Rakhlin. Generative modeling with denoising auto- encoders and langevin sampling. arXiv preprint arXiv:2002.00107, 2020.
Stefano Bruno, Ying Zhang, Dong-Young Lim, Ömer Deniz Akyildiz, and Sotirios Sabanis. On diffusion-basees generative models and their error bounds: The log-concave case with full conver- gence estimates. arXiv preprint arXiv:2311.1358. 2023.
Djalil Chafai. Entropies, convexity, and functional inequalities. Kyoto Journal of Mathematics, 44 (2), 2004, ISSN 2156-2261. doi: 10.1215 ///jm//mm/// 1250283556.
Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning, pages 4735-4763. PMLR, 2023a.
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R. Zhang. Sampling as easy as learning the score: theory for diffusion models with minimal data assumptions, 2023b.
Ting Chen. On the importance of noise scheduling for diffusion models. arXiv preprint arXiv:2301.10972, 2023.
Jean-François Collet and Florent Malrieu. Logarithmic sobolev inequalities for inhomogeneous markov semigroups. European Series in Applied and Industrial Mathematics (ESAIM): Probability and Statistics, 12:492-500 2008. ISSN 1292-8100. doi: 00.1001/ps:200704..
Giovanni Conforti, Alain Durmus, and Marta Gentiloni Silveri. Score diffusion models without early stopping: finite fisher information is all you need, 2023.
Arnak S Dalalyan. Theoretical guarantees for approximate sampling from smooth and log- concaee densities. Journal of the Royal Statistical Society Series B: Statistical Methodology, 79(3):651-6%) 2017.
Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. Transactions on Machine Learning Research, 2022.
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 1417669--17709 2021.
Del Moral, M. Ledoux, and L. Miclo. On contraction properties of markov kernels. Probability Theory and Related Fields, 126(3):395 420, 2003. ISSN 0178-8051. doi: 10.1007/ 400-40-00300270-66
Alain Durmus and Éric Moulines. Quantitative bounds convergence for geometrically ergodic markov chain in the wasserstein distance with application to the metropolis adjusted langevin algorithm. Statistics and Computing, 25:5-19, 2015.
14

---

Published in Transactions Machine Learning Research (12/2024)
Alain Durmus and Eric Moulines. Nonasymptotic convergence analysis for the unadjusted langevin algorithm. The Amnals Applicd Probability, 27(3).1551-1587, 2017.
Raaz Dwivedi, Yuansi Chen, Martin Wainwright, and Bin Yu. Lonccoe sampling: Metropolis- hastings algorithms are fast. Journal of Machine Learning Research, 20(183):1 42, 2019.
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélic Boisbunon, Stanislas Chambon, Laetilia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal Iransport. Journal of Machine Learning Research, 22(78): 1 1-8, 2021.
G. Franzese, S. Rossi, L. Yang, A. Finamore, D. Rossi, M. Filippone, and P. Michiardi. How much enough? study on diffusion times score-based generative models. Entropy, 25:633, 2023. doi: 10.3390/e250446333
Xucfeng Gao, Hoang M. Nguyen, and Lingjiong Zhu. Wasserstein convergence guarantees for a general class of score-based generative models, 2023.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. In Proceedings of International Conference on Learning Representations, 2023.
Qiushan Guo, Sifei Liu, Yizhou Yu, and Ping Luo. Rethinking the noise schedule of diffusion- based generative models. visible on Open Review, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
Aapo Hyvärinen and Peter Dayan. Estimation non non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Ioannis Karatzas and Steven Shreve. Brownian motion and stochastic calculus, volume 113. Springer Science & Business Media, 2012.
Tero Karras, Milka Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generalive models. In Advances in Neural Information Processing Systems, volume 35, pages
2022.
Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems, 35:22870-22885 2022.
Holden Loe, Jianfeng Lu, and Yixin Tan. Convergence of scorc-based generative modeling for general data distributions. In International Conference Algorithmic Learning Theory, pages 946 985. PMLR, 2023.
Haoying Li, Yilan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image supertresolution with diffusion probabilistic models. Neurocomputing, 479:47-59, 2022.
15

---

Published in Transactions Machine Learning Research (12/2024)
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proccodings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 11461-11471, 2022.
Alexander Quiun Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8162-8171 PMLR, 18 24 Jul 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Cascy Chu, and Mark Chen. Hicrarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 12):3, 2022.
Saeed Saremi, Ji Won Park, and Francis Bach. Chain of log-concave markov chains. arXiv preprint arXiv:2805 2023.
Adrien Saumard and Jon A. Wellner. Log-concawity and strong log-concavity; A review. Statistics Surveys, 8(none):45 - 114, 2014. doi: 10.1214/14-SSI0
Jascha. Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proccedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings Lille, France, 07-09 Jul 2015. PMLR.
of Machine Learning Research, pages
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference Learning Representations (ICLR), 2021.
Michel Talagrand. Transportation cost for gaussian and other product measures. Geometric Functional Analysis GAFA, 6(3):587-600, 1996.
Achille Thin, Yazid Janati El Idrissi, Sylvain Le Corff, Charles Ollion, Eric Moulines, Arnaud Doucet, Alain Durmus, and Christian P Robert. NEO: Non equilibrium sampling on the orbits of a deterministic transform. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, oditors, Advances in Neural Information Processing Systems, 2021. URL https://proceedding. neurips. .
html.
Cédric Villani. Topics in optimal transportation, volume 58. American Mathematical Soc., 2021.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661 1674, 2011. doi: 10.1162/NECO 00142.
Qing Wang, Sanjcov R. Kulkarni, and Sergio Verdu. Divergence estimation for multidimensional densities via k-nearest-neighhorr distances. IEEE Transactions on Information Theory, 55()): 2392 2405, 2009. doi: 1111109/III.2009921111..
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming- Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
16

---

Published in Transactions Machine Learning Research (12/2024)
A Notations and assumptions.
Consider the following notations, used throughout the appendices. For all d 2 1, p Rd and definite positive matrices D E Rddd let Pp,E be the probability density function of Gaussian random variable with mean and variance E. We also use the notation fna P0.0 When the context is clear, we may indifferently use the measure and the associated density w.r.t. the reference measure. For all wrice-differentiable real-valued function f. let A be the Laplacian of f. For all matrix IRmxx, ||A|| is the Frobenius norm of A, i.e, ||A||Er = (2 Zj= | For all time-dependent real-valued functions h t + he f : + ft), we write he hr-t and f(t) = f(T t) for all t E [0.
Let mo be a probability density function with respect to the Lebesgue measure on Rd and a R - and g I- + R be two continuous and increasing functions. Consider the general forward process
dx = aat) dtt ,t + g(t)dB,, ~OM
(10)
and introduce De : + pr(x)/ where Pt the probability density function of Xx. The backward process associated with (10) is referred to as (Xt)etjo and given by
axx -x-= (5()) (-(0 P(!)) o²
X X.+ (() Pro (x.) dt o(t)dB, No~
(11)
with B a standard Brownian motion in Rd Moreover, consider
o= exp (-2 a(s)ds) L. (0) exp (2 (2// a(u)du) ds.
(12)
The approximate El discretization of (11) considered in this paper is, for tu St 12+1, 0 N-1
- = +
tx. X } dt + g(t)dBr.
Sampling from this backward SDE is possible recursively for h E Lz). For &e E NN....,N- - 1}, writing = T tk:
N 1), with (Ze)IgHeN
0 1++1 =
- a(s)ds to 4 + 80(T,, T0 b2 je
1 a(s)ds frr g (( a(v)dv dl
TK-1
1/2
-2 f T6ts a(s)ds Imh 2 a(e)do g g2)
+
Ze+1-
We denote by Qr E P(C([O, TI,R)) the path measure associated with the backward diffusion and by (Q:)oSeST its Markov semi-group. We also write X To and, for each time step 4 for 0 < k S N, X backward discretization denoted by Q. so that we have X ~ T5 N.H
to Qt. For each time step la for 0 S k S N, the kernel associated with the
In Appendix C, these notations are used for the specific case where a : t ((t)/(20²) and g g 1/2 and the associated backward discretization is given in (31).
17

---

Published in Transactions Machine Learning Research (12/2024)
B Proofs of Section 3
B.1 Proof of Theorem 3.1
We are interested in the relative entropy of the training data distribution Adata with respect to the generated data distribution =/6.0) Leveraging the time-reverse property we have:
T ( =KI. (pTQr|= xx)
KL Tdata
By the data processing inequality,
KL 1orer/2N0 & KL KL To (prerr|ITS
where Qr and @ denote the path measures of, respectively, the backward process and the SGM generation. Writing the backward time 7 = - and its discretized version Tą T lks with 0 =o to < <1 < ... < tN T, we have (by Lemma B.5) that
Tdatall 1| ||
I KL (pT)) + 2 2 3(1)
Btt 20
X, + V in (xx,
KL
(-
3(1) X B(t) (Tee xx. 20
dt.
From there, the KL divergence be split into the theoretical mixing time of the forward OU process and the approximation error for the score function made by the neural network, as follows:
| de.
1 (()11)) - KL (pT (TO) + J/
3(1) (3 (T) ST x..))
KL Tdata ||
2 o 3(L)
By using the regular discretization of the interval [0, 7), one can disentangle the last term as follows:
1 N N
KL Tdata
KL KL [p
+
Σ
|
B(JE
|| (T., - 50 (Th: )1)) di
k=0
$ Ei( + E2(0,)) Es(3),
where
E((3)
KL (pT(lPo)
(13)
(0) = Σ
N-1
dat1
B(t)E
8 Tky x.,) - 80 (Th. xx,) ) de,
(14)
R=0
N-1
|| (T,, x,, - ( X t )1( di.
EB(
Σ
B(()E
(15)
k=
18

---

Published in Transactions Machine Learning Research (12/2024)
Finishing the proof of Theorem 3.1 amounts to obtaining upper bounds for Ei(3), (0, and (3). This is done in Lemmas B.1, B.2 and B.3, so that (3 - (3)) (0, (,,3)) and (3) & (3)
Lemma B.1. For any noise schedule B,
1 E1(5) KL (pr [OO) S KL (Tdats (oo) exp (-oz 02 L'' (
Proof. The proof follows the same lines Franzese et al. (2023, Lemma 1). The Bokker-Plandee equation associated with (1) is
B(1) 20
div div (x))(1)) 8(() Apr(x) B(1) ( Vm(ot) 2
Sepe(a)
for t E 2 x E Rdd Combing this with the derivation under the integral theorem, we get
/ log
pe(x)
KT. (pellvo2))
Poz (2: Pe
(x))d
0t
Øt
3d
{
0
log Pr(x) dx +
(e(x))ape(() Pr(x)
dr
RRd
0
(o²(x)
Rd
P (1)) log Pc(x) dx +
Tr Po(o)dd nd
Rd at
(00(xx)
B(t)
((())) + Vp((x)) log Pr(x)
div
dr (x)
Rd
B(t)
(z(
div
(-Vlog pr(x) Vpr( log G (x)
dr
2 JBd
B(t)
(-Vlog p((x) Vp((z)) T Vlog (e(x) dr
JRd
6 (x)
B(t)
V Vog (2 dr
P((x) ( V (logp(
JRd
ipa (z)
B(t)
(x(
((())
v log
dz.
Jdd
(z)
Using the Stam-Gross logarithmic Sobolev inequality given in Proposition B.6, we get
i KL (pellp0z)
8(t) o² KL KL (pellv02)
Ht
Applying Grönwall's incquality, obtain
{
0² (" B(o)ds
(prll S KL (polls02)
which concludes the proof.
19

---

Published in Transactions Machine Learning Research (12/2024)
Lemma B.2. For all 0 and all B,
N
E2(0,3)
Σ E
||
Voggin (Xm) se (ts, (x,x)))''
|
B(t)dt,
k=1
where ()) is defined by (14).
Proof. By definition of E),
v log log -1 (x.s) 39 (T tke X tn )1(}
x=) dt
3)
B(t)L
(xxx) 50 (T (T- lkk x 11
plat B(1)dt
E
Σ
E
Volggu
Pee (Xa) 3
30 tx X
tx+1
3(t)dt,
k=
where the last equality comes from the fact that the forward and backward processos have same marginals since X ~ pr.
Lemma B.3. Assume that H1 holds. For all T, a > 0, 0 and all B,
Es(B) S (T) max (13(T)
402 :1 [TTdatalTTo))
where Ez(B)= is defoned by (15).
Proof. By Lemma B.7, with Y, = V logEr-r( ( t),,
3(t) dY,
Yjdt + / d. 20
By applying Itô's lemma to the function x + || we obtain
3(()
all
| ((1)) di + (B(1)VTT,.al
Fix 8 >> 0. From Baldi (2017, Theorem 7.3, p.193), we have that (55(s))TT2,dB,
is
1E(0,T-0
square integrable martingale
E
( V. 2.11² ds <0.
From the Sanchy-Schwara incquality, we get that
E
| 2.112]
E 11
S
E
|1Y.1
1/2
E
[ liee
1/2
20

---

Published in Transactions Machine Learning Research (12/2024)
Applying Lemma B.8 and B.9, we get that both B|||Y] and 2. [l] are bounded by a quantity depending - As the term Tr is uniformly bounded in [0, T 5] and by Fubini's theorem, ElS (s)) Zs ds] = S (s)BB[II Zell"]ds
Therefore, ( g()) tejo is
square integrable martingale. Therefore, we have
E J.
((s) 0²
| + [ B(s) Z, 1s
[||]]
[1
Y
and
B(x)
/' V Z.s
E
1
12.112]
E
Y,ds +
20
12
(8))
27
< 2B
20 Ydds
+ 2F
((s) Z.d
1
B(s)
2E ([llVBl0)
2E
V B(s)
Yeds
20 Ju
[Tae putt 3(s)
S 20
B(s)dsR
|| ds
2E
3(s) [z, ds
pla+i B(s)ds
< 2 max
.
E
|| cx+s |1 - E
||| [1
(16)
402
Without loss of generality, have that IN-1= - Then, the discretization error be bounded follows
N-1
|| Vlloggr (x.) Vlogār- (x)
B(DE
dt
k=
B(t)E ||Xe Yea 1c.12]
122 dt
N-1
1k+1
( 3(s)ds
1 } {E [[I [12]
2
B(t) max
E [IVY, 11) de
4
k=
I lx+s B(s)ds
[| 12 - EE 11,T / (utt B(t)dt
2
max
1
E
4
A=
fe+t 3())ds)
2
max
B(s)ds
(EE) ||Yeeum 113] - 11])
40²
(nlttt 3(s)ds)
da+1
Vlog log ūr (xex })1
22 max 05k5N
B(s)ds
IE
4
21

---

Published in Transactions Machine Learning Research (12/2024)
By HI, t ++ (t) is increasing, so that t + ((t) is decreasing. Therefore, using that since Xo pr. T 8 and X have the same distribution, yields,
V VlogEr- (x1) V Vlogpr-tz (Xxx)] 1'] dt
Σ
E(t)
R=0
((x++1 tx))(ttt))
(tx+1- to tw))((())) }
2
max
max
NS
40²
Vlooggrr-e-- (xex--)
x E
(tt)
1B((x) }}
2 max
max
N-
4
h0)
2h3(0) max
I(prQr-sl
47
h3(TT
(T) max
4
Finally, following the steps of the proof of Conforti al. (2023, Lemma 2), we can consider the limit when d goes to zero, under Assumption H2, concluding the proof.
B.2 Technical results
Lemma B.4. Assume that H1 and H2 hold. Let (X e)22 be weak solution to the forward process
(1). Then, the stationary distribution of (X 1)20 is Gaussian with mean 0 and variance o aa-
Proof. Consider the process
X. = exp (
[ 3(s)da) X..
20
Itt's formula yields
X exp
1 20
[ B(s)ds)
(Xo+ 1/" VB(s) exp (f B(w)/(20))du) dv.). (17)
First, we have that
lim exp ( I' B(s)ds) Xo = 0.
1 20
Secondly, we have that the second term in the r.h.s. of (17), by property of the Wiener integral, is Gaussian with mean O and variance If where
1 of =xp exp (- 02 [ s(s)as) / 3(s)olo s(0) ds o² 1 (c-cxp exx (-o2 f 3(s)ds) ))
1
By H1, lim, of o², which concludes the proof.
22

---

Published in Transactions Machine Learning Research (12/2024)
Lemma B.5. Let T and 61,b2: [0, x C((0,T),R*) + Rd be measurable functions such that for i (1,2),
ax!" = by (1, (xX])))E1070) dt + VB(T - -t)dB,
(18)
admits unique strong solution with X ~ Tb. Suppose that (be(t, (X! ) )e(0o is progres- sively measurable, with Markov group (P )=20 In addition, assume that
T 3[T
mefo.t) b2 (s, (8, (x((1)
)1''ds
E exp
2 Jo
8)
b1
8:
(x(()
wE u..)) we.
<00
(19)
Then,
(1)
p(1)
(2)
p(2) I KL (1) ((02)
KL
To
T
To
1 T 1
b1 (s, (x(") ue(o.)) ) -62 b2 (s, (xxx)
)1 dt. (20)
+ 2 J 3(T 1)
wE(o,a)
Proof. Consider the probability space (St, (Fi)osiST, P) and for i E (1, let w(t) be the distribution of (x (x(")e)).rT) kejo.T] on the Wiener space (C((0, 7]; (T) T]; Rd))) with x! mo
(6)
Define u(t,w)
u(t,w) = -t) b1 t, (x
) b2 (t, (tx(x!))
-b2
)efooa)
and define dQ/dP(w) = MT(W) where, for t E [0,
1 M((w) exp {-f- u) dB. 2 |u(ss
From (19), the Novikov's condition is satisfied (Karatzas and Shreve, 2012, Chapter 3.5.D), thus the process (Me)ssesT martingale. Applying Girsanov theorem, dB, = dB, + u(t, (x! >eejo,s) a Brownian motion under the measure Q Therefore,
dx(") = b1 (t. (xx!1)
dt + / -tddt = b2 (t, (x(1))
dt + V BBT -jdB
(E(D,,)
ue(o.t)
Using the uniqueness in law of (18), the law of (1) under P is the same the one of (2) under Q
with x22 solution (18) with = and x = To Denote by pi(2) the law x Therefore,
= (AAA= A) Q((() E EA (w)(())
which implics that
(2
-MT.
du(1)
23

---

Published in Transactions Machine Learning Research (12/2024)
Hence, we obtain that
d
KT. (100/1200)
KT.
To"
(1)
"6
(2)
E
log
d77(2)
(1)
(2)
[
u(8, w) Jd.. + 2 1 / [1((sw)))dss)
1
KL
+
KT. (1)
(2)
J. 1'
3(T -t) TF |
(x.") (1
b2(t, (x." (x("))) )0
To 0
+
which concludes the proof.
Lemma B.6. Let p be a probability density function on Rd For all 0² > 0,
p(x) KL (pllfpo)) = / p(z) log (x)
dx S 0²
2 | Vog p(z) 1
88o (x) p(x) dr.
Proof. Define for : x + p(z)//p>>(x). Since V lo) = -o Lla the Bakry-Emery criterion is salisfied with constant o² see Bakry et al. (2014); Villani (2021); Talagrand (1996). By the classical logarithmic Sobolev inequality,
J (o
s o² /
[+)o
[o=(x) Po (x)dr,
which concludes the proof.
Lemma B.7. Define Yg : V Vloggr- loggpr-((x) and Z, V2 ( where (x1122 is a weak
solution to (10). Then,
(t) dy, (5 G2 - -a()) Y,dl 2 (t) a(1) Fddl+7(1)2Zl)))
(21)
x² 20
Proof. The Fokker-Planck equation associated with the forward process (10) is
(x): = (t)ddiv (xp((=)) + g²(1) App(x),
(22)
2
for x E Rdd. First, we prove that Dr satisfies the following PDE
0 (og = d ((()) g2(1)
+ (Vlog)(()),)) ((()) g
|
(g(()
a(l)
20
σ²
20
(23)
(t)
(t) (p)
2 Pc(x)
Using that V (o) -0/002, we have
div(ep((x)) d p((x) + p((x) V logg((()
||-|||
(p))
+ ( Vog (ogg)
o²
(x) (dp(()) + Vpi(.) J ||||||
σ² pe(x))
24

---

Published in Transactions on Machine Learning Research (12/2024)
Then, since (x)=(x)= (po0 (1x=1//02 -d), d), we get
Apr(z) = p.(x)) ((o= + +2V)((z) T (z) + (x=
= [x) (p((z)
||011
2 [ x AA(())
0² -d)
0²
0²
Combining these results with (22), we obtain
aa(() (())
+ Vir(t) x (al1) - -2(()
0.p,(x) = d (x())
20
o2
|1111
(g²(1)
- a(1) + (2 2
+ (x(
(p)
o²
20
Hence, diving by Pe yields (23).
The previous computation, together with the fact that Ap./pe- = A log pt |/ log pe 1e yields that
the function (xx) :- log (x()) is solution to the following PDE
(x))(x)) = -d (tt) 5²(1) VoL(a) x ((()) (0(1)- ((1))
(24)
20
0²
||>||²
gg)(t) - -a(1) - g'(() 2 (Ao((T) + (+V[()))))
(25)
2n²
Following the lines of Conforti et al. (2023, Proposition 1), we get that, since a and g are continuous
and non-increasing, the map Pt. solution to (22), belongs to C²,2 T x RT). By (11), as Y₁ = (X,), we can apply Itô's formula and obtain, writing 7(t) a(t) - /0²
dr, = Vo (x.) Vede (x.) (7(0)) (x.)) + g²(()
AVol (x.) dt
2
+ + 0, (x,) dBc = [[ [[v(ard) ac (x.) + g'()) Adr (x.) Vol (x.) + +500% di (x.) x, di
+ō(1)22 (x.) dBe,
using that 2V 2V² = (=) Using (24), we get
ar [Tt))/h (x.)+ + 2
0² (a(1) g²(() X x (x.) X dt
202
+ +o(t)(²0, (x,) ddc, with 2/ = V=]((=) x. With the identity V (x Vo((x)) Vol(2) (2)))) (o)(( we have
dy: = [(F4) - a(1) Vol (x.) +
2
a(t) g((1) X dt + 2 2e (X.) dB.
20
( - a(1)
Yr + g² (a(1) (000
2
X di + g(1) Z,d.,
20
which concludes the proof.
25

---

Published in Transactions Machine Learning Research (12/2024)
Lemma B.8. Let Y, == V VlogTr-<(XX) with X satisfying (11). There erists a constant C > 0 such that
||ail*] SC (INI1)
[I + 8 E X (1)
(26)
with Ia) and of as in (12).
Proof. The transition density ge x) associated with the semi-group of the process (10) is given by
- | exp
(- 1-5a((S a(s)sss))
x) (2Ta?) 4/2 exp
20
Therefore, we have
pr-)(x) / pu(l) -
1
v Poggr-((()
yexp T+
(r))(x)) / po(())
07-1 qr (V,
This, together with the definition of f7, yields
V llogrr- (Xr-1)= oT2,B = Xoe
-o- a(u)du
- X XT-1 to 20-2X1-
Using Jensen's inequality for conditional expectation, there exists constant C > 0 (which may change from line to line) such that
V log pr- (xT-1) l'
0e ---''a(s)da a(s)ds x T- Xr-(
+ t -8
x x
T-
Xoe- 1 a(s)ds Xx-1
x R 1'
X T
+ o 8
Note that X, has the same law as exp(- So a(s)dss + o,N, with N NNO, This means that we have that
V loggr pr (xT-))l") < Cor 1 [I
e + [1==0"
E
Finally,
| E Vlogrr pr- (x.) (x.))
= =E | Vlogrr (XT=:)l²)
E
S o [|_V11.]
< c o [|N1]]] + - [=x00(])
which concludes the proof.
26

---

Published in Transactions Machine Learning Research (12/2024)
Lemma B.9. Let Z₁ := V log pT (( where (Xire20 is a weak solution to (11). There eaists a constant C such that
|2 SC = to (B 12 + dd
(27)
with Z ~ No(, Ta) and of as in (12).
Proof. Let 4e (y, be the transition density associated to the semi-group of the process (10). Write
2 log (x)
/ po(v) ye So "a(s)de
1 V
97 (y, x)dy
o7_
(fmnty) DE
-a"' a())ds
T
(p)-
2
qr (1, x)
(x)
o7-1
(1)) I po(u) ye So "als)de
1
x
o7
-1,,
QT
10-''aas) a(s)ds
1
(-1 ((r)
ye
Ør- x) dy)oyyd
2 T- ((()
((=)
2
T-
Ia + / 07-11
(ye -0 a(s)da
ye
- x
yy S a(s)da x3)
T
or gr- (n.=(u,x)polie )ppo())d)))
Therefore,
2 log (XT-1)
Vpr (XT-1)
(xoe -hee-o"" - "(e)dd - -X1-1)
E
X XT- +
2 -- +
Pr- (x-+))
X.c fo-1 a(s)ds xx-1) (Rue- fp--1
- fo-ra(s)dd a(s)dd _ Xx-1)
X t 2
+ oT E
ue So "(s)dd - -Xr- (xo))) E Xoe So- a((s)do -X X xx-1)
-t
E
+ (0 2
-oT2) Ld
(Roe - Tr a(s)ds - Xn-1) (Xne So a(s)ds _ xxx-1)
X
+ 0.-,B F.
27

---

Published in Transactions Machine Learning Research (12/2024)
There exists a constant C' (which may change from line line) such that
log Pr-t (xm-))1 ||!!
E
2
C
Xoe So¹ "a(oddo - -XT- X XI-1
Xoe - S0" "((s)do
"="'a(s)dd - -r-- XT-1
+
E
E
E
T-t
( dt
C
C
Xoe ---
a(s)ds xx-1) (Xoe- f a(s)ds Xr-1
E
TT-1
T-
As in the previous proof, we note that X, has the same law as e So a(x)ds x XutoiZ with Z ~ (( independent of Xu. Therefore, using Jensen's inoquality,
X "='""a()dde a())ds X XT- E
Xoe
a())ds X X T-1
E
E
X ue So a(s)dds -Xr- x
T- X uue --- 1--11((()) a(x)do -Xr-1 X XT-
< E
E
XT-
E
X T X oe So -or a(()ds X
< E
E
I o
1E
112
and
a(s)ds Xr- ) (Xoe-1i 1=
a(e)ds -XT-1)
T
E
oe
X X-
Fr
10- "a(s)do X1-1) Xoe T- a(()dds X
<
E
E
X 0e
X T-1
- a(s)ds -Xr-e
E
Xne
S S [1211]]
Hence, we can conclude that
E 1 =E E
72 log pr- (RT-:
s SC - +o- (B
+ +d²)
28

---

Published in Transactions Machine Learning Research (12/2024)
C Proofs of Section 4
C.1 Gaussian case: proof of Lemma 4.1
In the case where ddata is the Gaussian probability density with mean /o and variance Eo, we have
V Vlog = (m + oāTa) -1 (x - m4/1)) o²xx 2x,
with m, exp (- So ((s)ds/(2o)) and or = o² mî)- Let Σ, = mg + of be the covariance of the forward process X, and by = 3-'muu so that
Vlogg( = Ar + with
A, 4==
E -
2T2)
(28)
Note that, if we denote by N .... < Xo the eigenvalues of Co. which are positive as Mo is positive definite, we have that the eigenvalucs of A, are
1 0²
X m + oz
It is straightforward to that \o S S X4. Moreover, we always have that in this
v log - V Vlogp(())) (x - v) - x || - w11?.
IV (x)) - v (o S max {1>'l ||21 || vll
which entails that we can define
I := max {At x := -x
and apply Proposition C.2.
The condition X < 0, or equivalently o² (max)(o)) yields a contraction in 2- Wasserstein distance the backward process as well in the forward process from Proposition C.2. This shows that, in specific cases, with an appropriate calibration of the variance of the stationary law with respect to the initial law, we have contraction both in the forward and in the backward flows.
As consequence, note that
W2 (Taata, Qr) & W2 (pr, TTo) exp (-1 1 (T
02 (' 3(t)(1+2((o
Using Talagrand's Tr incquality for the Gaussian measure W2 (M, x00) s 2T and Lemma B.1 get
T ("
Wa (Taata; Qr)² < 2T (Rdata Too) exp
3(1((1 + 2Cho2)al)
Proposition C.1. Assume that Waata is Gaussian distribution N(Ho, 2o) such that Amax >max(22)) S o² where Xmax (Eo) denotes the largest eigenvalue of Eg. Then,
KL (Tdatal Too Qr) & KL (Tdatalll l exp (Toz) ("
T
3(s)ds)
29

---

Published in Transactions Machine Learning Research (12/2024)
Proof. In this Gaussian case, the backward process linear (see (28)) and the associated infinitesimal generator writes, for g C?,
Eng(z)= Vg(le) (- B(t) 2n² + ((0(Ax++b))
1 + 3(1)a(()),
where A, Ar-t and b. br-
Our objective is to monitor the evolution of the Kullback-Leibler divergence, (p Qr), for t E [0. We follow Del Moral et al. (2003, Section 6) (see also Collet and Malrieu, 2008). Let = 2r and Φi = Poa two densitics that satisfy the Fokker-Planck equation, involving the dual operator 2 of the infinitesimal generator T
diq: Lig,
(o(x) pr())
0::: Lio,,
Φ0(x) = par (x)
Let Jt = 4//00. By definition of KT((eeld.)) = af))
(x) we have
0,KL (gellde) / In (f((z)) (((())))) / i, (f((z)) (x())) [
By employing the Fokker- equation and the adjoint relation, which states that
J (())))))) [.f()))(())) we obtain
0,KL (4ell0)) / ' In (z) - ( (EMSC))aao))d
The infinitesimal generator I satisfies the change of variables formula (see Bakry et al., 2014) so that
1 I. =
2F F((.f).
where F, is the "carré du champ" operator associated with E, defined by ((( = -()))))) We then obtain
f (((x) ((()) dx - B(t) Vf((2))
f2(x) ge (x)dr - -EEf(()))r
O.KL (ge (gel)0))
f((x)
{(t) / [V((())
2 fe(x) (((ddd..
(29)
To obtain a control of the Kullback-Leible divergence we need a logarithmic Sobolev inequality for the distribution of density d, po=Qr. In this Gaussian case, if X N/ o²) then for all t E [) the law of X is a centered Gaussian with covariance matrix D, given by
E, = o² exp (f--Bta) [[3() o²
3(s) + 23. 23.A.ds) + / 8(s) exp) exp ((-=((a) o2 + (1 ds
B(u)
30

---

Published in Transactions Machine Learning Research (12/2024)
where we use the matrix exponential. As mentioned before, if Amax (2o) S o², the eigenvalues of A,, for s E T), are negative. We can easily deduce that Amax (x) S o². We recall the logarithmic Sobolev inequality for a normal distribution (see Chafai, 2004, Corollary 9)
1 KL(gel & 2 f((x)
Vf(()) (((x))TE (..V Xmmax(E)) (Ex)
V+(2)) f((z) (1(z)dz.
2
Plugging this into (29) we get
0,KL(q 5 _811 (L
Therefore, recalling that qo pr and do Po=²
KL (0)ll S (L(PTllV0=) exp (-J²
B(s)
B ds
We conclude using Lemma B.1.
C.2 Proof of Theorem 4.2
El scheme.
Using the fact that
[' ['o- (10)/)0)))) B(s)ds = 20² (1-o -fr p(v)/(20")ds le
the Exponential Integrator scheme that we consider consists in the following discretization, recursively given with respect to the index k;
X, = -f, ((o)/(200)ds X + 20" (1 1 - Sr p103/1203)/)) log Pr (X(x)
+ + (1-o Sig B(o)/o"eds Zx. (30)
where Z are i.i.d. Gaussian random vectors N (0, Ia). In particular, we have that
X = S 3(s)/(20))ds ds 8 + 20² (1-e S B(s)/(2?))da)
(T 1k, X
+ +o/ (1-0² Stg Bla)/oRde
3x. (31)
and X8 ~ ~ 2N, (2).
Note that
( =(3.0)) < W2 (Taata, TTQ W2 Too Qr: Tox 17)
W2 Taata:
(32)
where
W2 (Taata; Toc Qr) W2 Fo (arr
which corresponds to the discrepancy between the same process (3) with two different initializations. The first term of (32) is upper bounded by Proposition C.2.
31

---

Published in Transactions Machine Learning Research (12/2024)
Proposition C.2. Assume that W2 (Tdata xoo) < +00. The marginal distribution at the end of the forward phase satisfies
(-f""
3(1) o² dt)
W/ (pr,,T>) & W2 (Tdata; ] exp
(33)
Assume that H4(ii) holds. Then,
(-
/'
T B(t)
W2 (Tdata: To To Qr) S W/ (pr, To0) eexp
(1 20,0²) dt
0
o²
T {()
S W2 (Taata: Roo) Eexp
(2 -2
(34)
d²
Moreover, under Assumption H4(1), we have
W2 (Tdata: Toe Qr) S W2 (pr, (pr, 2 Pexp exp (-%
T 3(1)
(1 20,00) at)
S W2 (Taatar Too) 2eep (- 1-/%
T 3(1)
o² (2 + 2++C0o0) dt
(35)
Proof of Proposition C.2. Let x E Id (resp. y Rd) and denote by X= (resp. X²) the solution of (1), with initial condition X = x (resp. X = y). Applying the chain rule, we get
{ - || - v11 + %--
3(s)
X - X2 { ds.
202
Therefore, applying Grönwall's ' lemma, we obtain
x x2 17]
3(t) 02
E
sup [E
< exp
-
dt
||0 y11
From this, we can show contraction (33) Wasscrstein distance by taking the infimum over all couplings.
Now, let I E Rd (resp. y E Rd) and denote by X= (resp. Xv) the solution (3), with initial
condition X3 (resp. XØ v). Applying the chain rule and using Cauchy-Schwarl inequality, we get
87 - x:yl = =|,x v +2 2
3(s) 1x: Fyy|dds
202
+2 B(s) V Poggrr- -8 (I:) v Vlogār-o (x:)) (x: - - ds
< <|1 v/ 1
3(s)
(1 2L 0²)
1x: - xzy ds.
0
32

---

Published in Transactions Machine Learning Research (12/2024)
Therefore, applying Grönwall's 's lemma, we obtain
F] | S
exp (-% 3(t) o²
E sup [o
x
x
(1 (1-2L,07) 2L,00) dt) || y/1²
From this, we can show contraction (34) in 2 Wasserstein distance by taking the infimum over all couplings.
To establish (35) note that, under Assumption H4(i), have
x
| y12 +
/"
((s)
x xx [/]][an ds
J0 20²
J B(s) (Vlogrr pr -s (x:) Vlogrr- (xx)) (xx: -xx)x)
ds
11< y/ '," B(b) o² (1 20,0²) | - Xzy x ds.
Therefore, applying Grönwall's s lemma,, we obtain
((t) | S exp (-1-" 5000 o² (1 (l:20(0) de) ||0 111²
x X
E sup AE(O.T]
From this, can show contraction (35) in the 2 Wasserstein distance by laking the infimum over all couplings.
Note that similar assumption as Assumption H4(i) is used in De Bortoli et al. (2021, Proposition 10,11,12), in particular to bound the conditional moments X given X, for t 0. However, in this paper the authors also require additional assumptions, in particular that the score of daata has a linear growth.
Second term.
The second term of (36) can be handled as follows
Wz Qr, $ X X
To upper bound 18 Xf we aim at controlling ( Nomnlle by xX 1 to resort subsequently to a telescopic sum.
Proposition C.3. Assume that H4, H5 and H6 hold. Consider the regular discrelization {1a: 0 k N) of [0,T] of constant step size h such that for all tx with 0 < k S
20,
Mu++1
h 3(tx) (max,, Sostu (max
L.) La m
where m, := exp(- 10 B(s)ds///))))) (20))) m. exp(- So ((s)ds/(220)) Then,
1
X7
<ET3(T) + (TT (BB
L7
N-1
Seil
L, mr B(I)at)
2h
Mki1 1
+
mr
mT
Jte
20
+ 2T,,)
B())d) B,
0=
33

---

Published in Transactions Machine Learning Research (12/2024)
where M is defined in H6 and B [B]||xol|]+ + o2d)
Proof. Using (31) and the triangular inequality, we have
x
tR12
skl1
12
Mex+1 X
fet1
mh 3(1) (v log pT (xx²
50 (T Lu, X(c) dl
Min
Mia
L2
M X00
Mu+11 X
duti
mh 3(1) (V (VoggTT- pr (xx2) A Foggrr (X dl
Mix
m+
Mita
L/2
mi B(1) (V log (xx) V VoggTT= pr (xx²² dl
(36)
12
1k+1
mi
(() (V Wog (X() 5g (T- Lb: 44: X )) dl
mik
L2
Using the strong concavity and Lipschitz properties of the modified score function, we have that the first term of r.h.s. of (36) can be bounded as follows
Mi++1 00
Mua11 X
flt41 mi B(t) (V (Vlogrr-
dt ( V log Pr (xx)
+
mix
1/2
Mix
miz
mix+1
1xxc 81
pla-1
m
3(1)
V VlogāT- (xx0
V log Lo (x%) dl
-2
mi
m+t
+ m
munt1
cl++
mi
B(t)
[ x
T
log pr-
(x00
log Pog (xxx dt
Trez
<
x
X
||
M =2
+
plate
m Lt
B(t)dt
2
2
Me++1
C
m
a(t)
mix
mix
m
Using the Lipschitz property of the modified score and Proposition C.6, the second term of the r.h.s. of (36) can be controlled as follows
me B(t) ( log (xx²) - Vlog (xx(2)
dt
mix
L1
my 8(L)dt)
1::00 -
x00
-t mer
sup 11
tx
1/2
Lt te B(t)at) 2
hhB(T)) + exp
(-)
tx B(s)
(1+C,07) B.
mix
34

---

Published in Transactions Machine Learning Research (12/2024)
Using Assumption H5, we can control the third term of the r.h.s. of (36) as follows
my
B(t) log Lo (xx,) - 80 (T - tk x dt
Mix
112
1x42
me
3(t) V Loggr--1 (xx)) 3g (T - tx. xxx dt
max
122
M++1
my B(t) V loger (X².) log (xxx) dt
Mik
L2
1x+1
My 3(t)dt+ +
fett m
3(t) V log po (xx,)
V log pr (x%) dt
MiE
bkk
3(t)dt + hM (1+ ([xt
me B(t)dt. miz
mix
Note that X, has the same law m, o0 - mG)G, with G standard Gaussian random variable independent of Xy. We have that Xx ~ of Define (Xt)eeor satisfying (3) but initialized at
X. xm=mrXo
mr + va- m X0
with Xo -daaa- Employing Proposition C.5 and (42), obtain
1x S X2
xpc0
+ || X
+
[x% -
+BB
12
Therefore, combining the previous bounds, together with (36), we obtain
xx
<
1:00 16
- x%/
Mixm1
=2
+
L the B(()))) mi
2
2
Mez+1
dett
Ō, mr 3(()dL
1/2
Mii
mie
T, My
3(L)d)
22,1(TT) + nTT
1
+ 2T.
3(1))))
B
Mix
20
rett
mh 3()dd + hM 1+ ||xx - Xx
Nu+1
m 3()dl.
6
+ +BB
Mix.
L2
m
By the assumption on h and Proposition C.4,
m
2
let1
Ia m: B())dt)
2
Mix
C. mi 3(t)dt < 1,
0 1 +
2
m
Mi++1
35

---

Published in Transactions Machine Learning Research (12/2024)
and, using that vI- x S 1 x/2 for [0, 1], we conclude that
xx00
X
1++1
-
1x+1
L2
x00
12+1
16
Mig
x
1
1 min
La
me B(t)dt) mix
2 mi
dxts
C
me B(t)dt+
2 mi
t/.
max+1
mix
m
pl+t
The 3(t)dt
Mh
Mi++1
mr
T. mh 3(1)d)
( 223((T)) mr
20 + 2Tx
B(())t) B
mia
mt
mh
+
3(1)dl + hM (1 2B)
3(1)dL.
bu
mt
Mit
Define
5%
m Fkl1
1 +
mix
1R12
me Le mrsk
3(t)dt)
mix
Č my B(t)dt
mi
Mi
M
1x
miz
i Mh
pt&t
me
3(t)dt)
m tx+1
tu
1 miz
Lt The B(t)at)
mitx
dett
Cr m
1
B(t)dt
2 mzx
miz
Ma241
mia
m Mh
pl+s me
3(t)dt)
m
miz
By Proposition C.4, 8x < for any 0 A -1, which
which yields
N-1N-1
X2
<
II 6F
X6]
+ (ch3(I) M12 (1+
Σ
II
8
L2
k=0
N-1
tu+1
L mh
3()aL)
( 21 +
da+1
+ 2hx
3(())))
B II
N-1
20
k=
E=k
< ETB(T) + MhTB(T) (1 2B)
N-1
futl
L. M 8(J8))
( mT
tatl
20 + 2In B(()a)) B.
1
m+.
36

---

Published in Transactions Machine Learning Research (12/2024)
Final bound.
Finally, combining the results of Proposition C.2 and Proposition C.3, we conclude
that
( 3(1) Gro2) de
(- f-
30) (1
W2 Adatas )/i
W2 (Teata:: Toc) exp
o²
N-1
tk+1
L
me 3(t)dt)
( (T) + m&
(
1k1
+ +24,)
3(t)dt)
Mix
1/
20
k=0
+ (T) + MITB(T) (1 (1+2B)
C.3 Technical results for Wasserstein upper bound
Proposition C.4. Assume that H4 and H6 hold. Consider the regular discretization {ta, 0 S k S N of [0, T| of constant step size h. Assume that > 0 such that for all t with 0 < k $ N 1
20,
M
h < 3(tx) (maxi, So5te-1 La) Le
(37)
Mra
where m, == exp(- So B(s)ds/ /alls/22))) (202)), mi == exp(- So 3(s)ds/(2o)), Then, for all S k S
pl+t
La me B(t)dt
2
men (
Ct ma B(t)dt <
0< 1+
M
22
1z
max
2
Mu++ Le
max
In addition, if
2C. (maxu L3) L, mix
M
h
(38)
M + 3(tx)
then, for all 0 0 K N
1 m
plt+1
Le mh B(t)dtt mex
2
mix
1x41
The 3(t)dt
0 < 1+
O
2
M4411
22
th
Mix+s
mix
m
Tx41 my 3(1)dl < 1.
Mh
mix-1
2
mik
Proof. Denote E and E2 the following quantities
6 1 ilii
22
T ih 3(t)df
2 Titt..
det1
C mh ;(t)de,
(
(39)
m
mia
Miz+1
db
Mia
m4x
sx+1
L m 3())dt
Mute
let1
C₁ mr B(t)dt
62 1 +
2
1x
M
lx
mi
li+t1
ma B(t)dt.
+
Mh
(40)
mi++
m2
miz
37

---

Published in Transactions Machine Learning Research (12/2024)
First, we prove that Ei is positive. Completing the square, we obtain
c1 (1
mix
Mel
Li my
mir
1R11
me 3(t)dt
3(t)dt)
+
Lt
M1
Jec
me
Mext1
TikK
Mikk
ditt
Ci the B(t)dt
2 Maxt ex
mix
Max
fx
me B(t)dt L
+ 2
mix
Akll
(Lr Ci)
m B(t)dt.
Mix+1
mix
M
mix
The first term the r.h.s. of the previous equality square, therefore always positive. The second term is always positive as well, as L, 2 C. for any as the Lipschitz constant and the log-concavily coefficient of the score function respectively. Moreover, the previous is always strictly positive as
Mix
tk+1
L mh 3(t)dt > 0.
Mixas
Mi
Secondly, proving that the previous quantity is smaller than is equivalent to show that
i 2
Lt m
3(t)tt
Mia Meet1 11
futl C G. < 0.
1441
1188x
As ((t) is a decreasing function, we obtain the following bound
mia
datt
L. Mr
B(I)d))
Mrx
C₁ m 3())d
2
m
Myx
Mex+
St
m&x
mtn
L.3(tuh
Mea
Aut1
L mh 3(1)dt - 2
Mix
se+1
m ((t)dt
max
Mix
Mixx (1 1k
iiees
M 11
tx
m
te11
mik
Mix
L.B(tahh) LC-2C,
20,
mc 3(t)dt. 1i1
max SaStat1
M1 Jh
This means that, if we have
Mix
2.) B(tw)hL. - 20, <0
max (aStSto43
Mexi1
for to S t S tk+1, we have e1 1. Isolating h in the previous inequality, we obtain that it is equivalent to the condition (37).
Now we focus on 62. This quantity is clearly positive as the 2 E1. Moreover, following the same lines as to prove that é1 < 1, we have
Mit
1241
Min
Mh - 20, I.) b(ta)hL, + Mis Mix+s
m, B(t)dt. mme
62 15
max (aSoStee
Mox+s
Miai
38

---

Published in Transactions Machine Learning Research (12/2024)
This means that, if we have
Mis.
L.) B(tx)hLi + May Mh - -22; < 0
max (uStStot1
Mix
for to I I 1k+1, we have 62 < 1. Isolating h in the previous incquality, we obtain that it equivalent to the condition (38).
Proposition C.5. Assume that H2 holds. For all t 2 0,
x
mm [1xo1l]] + +1-11mmm)))) (immmodod'
1/2 1 [1o1l + +od)
sup 0S1ST
I sup 0S1ST
where my = exp(- Jo (ss)ds/20²).
Proof. Recall the following equality in law
x.= m, + (1-
with Xo Wdata and G ~ ~ (0,
Therefore, for any t E [0,
27 X
= =B -1.111]]
m | + 022(111 (1-m?) [|a|*]
mBB | 2 0² (1 - mn? d.
Proposition C.6. Assume that H2 holds. For all St ts+1,
sup S15t4441
12.0 - 800 L2 1
(=
1 (2BB(T) + MT
20a + 2T.,
B(O)a)) B,
(41)
tk
-xp0 xelle0 s ([1x011] + ofd)
1/2
T 3(s)
sup 0S1ST
exp
j0
20
ds
(42)
where m. exp(= fo (ss)ds/20²) and B = Xol1| + 222)
Proof. Note that X, has the same distribution m, Xo + a a -m²)G where G ~ N/(0 Ta) is
independent Xo. We have that X = G ~ N50, Define (X1)ee(0r7) satisfying (3) but initialized
X. = myY + V m
(43)
with Y Eaara independent of G (G being shared by X and xz0)
39

---

Published in Transactions Machine Learning Research (12/2024)
On the one hand, following the same proof as in Proposition C.2, we have that
|x, X 1/
s | - Kol1,, exp (-oo²
((}) 20 (1 20,02) (1+20,07) d8)
1.2
- ([IvPpe + o²d)
1/2 mvr,
where we have used (43) as well as the fact that
I1xo Gllr,2 ([1xIP]+ + [[cTT)) 1/2 B.
Therefore,
x - x.111 122
+ ([Ix)]]+0'd) 1/2
exp (- (-f-
T
3(s)
sup S
20
ds
corresponding to (42).
On the other hand, we have that
1x=- Xx0
s $1xx,
X in + +x - xX. - lt - -mm)
The process (x,0- - X.)120 is determined by the following ODE:
d -xx-xx)- (- B(t)
1x-0 -x,) + +28(1) (VlogēT- (xxx) -Vlog de- (Xx))) dt.
20
Then,
(8:00 -x,) (xx0-
-x,1
L1
11/0 3(s) :0-0 - X.) +)3(0) ( (Vlogrr (xx) V Vlog pr Vlogrr-
(x.)) ds
20
12-00 - -x,1
La (ml))
1
20 + 2T,,) 3(1)dl
sup to StSt-111
1 + 2L,
< BmT
3(t)dt.
20
Write (X.)eto,Tr the time reversal of 1. which clearly satisfies (1). Using the following equality in law
2) 1/2
XT- mr-
11 1x X T-1+ + (1-
m= th mT-t
oG,
40

---

Published in Transactions Machine Learning Research (12/2024)
with G ~ (d), we get
2
1/2
1 - -5xxxall L/2
= =1 - -xr-l/
1
mr-Le
+
MT--1
2) 1/2
MT-Tt
V2B,
mr-I
where have applied Proposition C.5 in the last inequality. Since
--
3(s)ds)
exp
mr-+
0² --t
T T-t
1 II-
exp
B(s)ds B(u)du
T-1
02
T-
hT((T)
σ²
which concludes the proof of (41).
D Discussion on the hypotheses
Proposition D.1. Assume that log data is Cu-strongly concave and that C, > 1/02. Then, the modified score function log Pe (z) is, for any t (0, Ci-strongly coneave, with
1
B(s)ds
m exp
20
C = m + r² (1 - mi)
a²
Moreover, we have that C. S C. 1/02 for any 120.
Proof. This result is also proved in Saremi et al. (2023). We provide an alternative proof here for completeness. For all 1 s T, Xx has the same law has m,Xo + +ov1- m Z where Xo Tdata and Z ~ No0, Ia) are independent. Therefore, writing
Tdata:
/ gd (2TO² (1 (1-mi) -mm))) -d/2)))42o0) d/2 exp
lly zumel 2
pr(v)=
po po(
(44)
2 (1 mt)
This implies that
d log pr(v) log (2:TO² (1 - (-m))) + llog
11y - somell
exp
Po (zo ddz
2
20 (1 mp²
=- log (2TT0² (1- (-m?)) + log
d
2
Rd
exp
2 (1 mm²
|1 u11
Po
u
du
d
[( 3(s)ds.
2
41

---

Published in Transactions Machine Learning Research (12/2024)
Since log Po is Cu-strongly concave, the function I po (u/me) C./CC//// C./m log- conccoecvve. More- over, have that the function y - expí-Ilvl| (120:llll////(1m-m//)) Applying Saumard and Wellner (Proposition 7.1 2014), since pt is convolution of the previous two functions up to terms independent in space, we have that log is (m,/C. + (1 - -m))) -1 -strongly concave. Note that if C. 2 1/0",
is (o (1-(1-1)))) - log-concave.
C.
mi + oCC. (1
mi)
This entails that log P. is Cysttrogly concave, with
C. m2/C. o2 (1 m²?
σ²
Finally, finding the maximum +++ Ct2 is equivalent to find the maximum of the following function on [0, 1]:
C.
1
1 2 2 C2C,(1 - 0²
We have that (0) C. 1/02, (1) 0 and for all E [0. 1/.
-2- (2) (z/C. + 02 (1=-)))) -))
which is negative since C. 2 1 S 1/02. Therefore, we get S C, S C. - 1/0².
Proposition D.2. If log Tdata is L, -moothh then for all S t < T, logg: Lipsschtt in the space variable with
L
= =in min 02 (1-
m
m
σ2
Moreover, L, > 1/02, we choose L, as follows:
1
L.
L, =in min (2 m
mi
Moreover, in this case, we have that La & Tr, for ony 2 0.
Proof. In the proof Proposition D.1, we proved that, if log Teata is C.-strongly concave, log Pr (mi/Ca CC. + o² (1 m²)) -strongly concave i.e.,
(= & m2/C. + + m?) la-
For Po == Taata: we have that Pt is given by (44). This means that Pt is the density of the sum of two independent random variables X1 + No of density respectively qo and mi, such that
(x(x) Pu
)o(()
P
m
91 (x) (2TO² (1 - mz?) d/2
1
exp
|
(x)
20 (1m=))]
42

---

Published in Transactions Machine Learning Research (12/2024)
for two functions do and Φ1 Therefore, as in the proof of Saumard and Wellner (Proposition 7.1 2014), get
(-llog))) Var Var(VVo((X)) X + X; = EVV +X, = x] - Var(VV x + X1 = x) + (I Xn + X1 = x]
Since V Po Pa is L.-Lisschitz and from the definition of Q
L. La m
2o &
Vol & 02 mz) la
m
Hence,
Lo
V (ogg
(x) x
min
la-
02 (1
m
mi
Therefore, since the difference between V log Pt and V log Pr is a lincar function, we can choose L as follows:
1
L. m
L, min 02 (1-mm)
Clearly we have that 0 S m S 1, therefore 1/m² z I and 1/ (1 (1-m;) 2 1. This means that, if L, 2 1/02,
La mi
min
2 g2
(1 - m
Thus, we can choose Lg to be
L = min { 02 (1-m))
m
2
σ²
Finally, since my we have that Lu L. 1/02. This function inereases up to the point where L, /mm = (o² m achieved for m (o² (o) + 1). At this point, we have that Ly = L.. After this point the Lipschitz constant decreases to 0, as - 0 for - co. This means that for any I, Le is bounded by L..
Proposition D.3. Assume that log daata is L.-smooth and C.- Ct-strongly concave. Consider the regular discretization (lx, 0 S S N} of [0, T) of constant step size h. By choosing h > 0 such that for all t with 0 < k < N 1,
S min {
12)
OC.-1 c²C. L1 + +1)L,B(T) ((T (02L. 1) L(,(i)
JC. -1
(45)
3(T)
then, for all < k S 1,
mii
22
plets
mt B(t)dt
2
Mt..
m 3(t)dt < 1.
+
=2
L
Mi++1
msx
Mu++1
de
43

---

Published in Transactions Machine Learning Research (12/2024)
In addition, if
h min 12g
O²C. -1
B(T)
O2M L. (02 11)
o²C,- -1
m² (1 mm
(o²C. - 11) L.
(46)
oCC. o2M (1- + ((T) (. 2 d²C, (o 1) (M + ((T)
lhen, for all N 1,
1
2
Le me B(t)dt))
mix
fEIL
my 3(t)dt
0
O
2 M (x11
Un
mrx
Mixs
Mix
m
mh 3(t)dt <1
Mh
m
Lv.
mia
Proof. Define 6 and as in (39)-(40). From Proposition C.4, we have that & E (0, 1), for = 1,2, we have (37)-
First, we prove that (45) implies (37). From Proposition D.2, we have that L, is bounded by L. find h small enough 2 1. This is equivalent to pla+s B(s)/2o²ds S log(2) and it is implied by
everywhere. Moreover, since May-1 exp flhtt (ss)/2o2ds) such that
log(2)20²
((T)
Now, we study the function t C./Lo Li- From the proof of the Proposition D.I, we have that
G== = m + 0² (1 m,) r2
which is decreasing function. Moreover, from the proof of the Proposition D.2, have that
1
L.
L, =
min min 02 (1 (1-m))) m o2
which an increasing function from 0 up to t², such that m = r.., and decrcasing for t 2 t". On the one hand, this means that for E [0, (*), the function + C,/L, decreasing, therefore reaching its minimum (o²C. - 1) (o² 1) in 0 which is positive quantity. On the other hand, for & t*, we have that
d²L.
Ct
(o²C. -1) mi 0² (1-m)) mm,)
L. m o (1 + mi) 0²
C.
m
> 1 OC. -1
(1-m?)
0² C.
CC -1 (1- m
o²C. -
22 C.
r²C, 1.2 +1
44

---

Published in Transactions Machine Learning Research (12/2024)
Therefore, combining the previous inequalities, we have that condition (45) implies (37).
Secondly, we prove (46) implies (38). Take h to satisfy
5 log )2)(2)20
B(T)
We now need to study the function & M+3())LLL On the one hand, this function is decreasing
d²C. for t [0, therefore reaching its minimum o ( in O, which is positive quantity. On the other hand, for & 2 t", have that
C M + B(T),LL
(o - 1) m
0² (1 - (1-m)) o² (1 m2) ((T)
m2 / 02 (1+ m2) 0²
C2
1OC.-1 σ² C. o2 m?) + mm.
m (1 m,)
Controlling from below the previous quantity, boils down control from below the function (1)) y(1 () for y E [mq., m We see that v in this interval can be bounded by minff (m,.)) ), (m Therefore, we gel
C.
L. oCC. - min M1 m + B(T)L, Lo + 1) I) 3(T)
2 (1 mZ))
M + B(T)LLL
r²C.
Therefore, combining the previous inequalities, we have that condition (46) implies (38).
Details on numerical experiments
This section divided into two parts. The first part dedicated providing detailed implementation choices for the numerical experiments presented in Section 5. The second part displays additional experiments and more details about the experiments of Section 5. All experiments were conducted on local computer CPU equipped with an Apple M3 processor (8GB of unified memory). This setup is sufficient to replicate the experiments of this paper.
E.1
Implementation choices
E.1.1 Exact score and metrics in the Gaussian case
Lemma E.1. Assume that the forword process defined in (1)
B(t) X dt + VB(t)dB,, X Tu.
ax
20²
is initialised with To the Gaussian probability density function with mean po and variance Eg. Then, the score function of (1) is:
VlogP: : Im + T²Iaa -1 mejto).
where Pe is the probability density function of X,, mn expl- 1or (ss)ds/(2o))) and o? o² (1-m)).
45

---

Published in Transactions Machine Learning Research (12/2024)
Proof. Note that X, has the same law as meXo + og where Z Lo) is independent of Xo- Therefore X, (mmpo Ed with E m Eo + ofTa which concludes the proof.
Lemma E.2. Let 11,412 u2 in Rd and En and Eg be two definile posilive malrices in Raxd, Then,
(L (1 2 log Σn
1
d Tr Tr En) + (N2 -mi) Enz (02 mn)
(47)
Lemma E.3. Let #1,42 in Rd and Dn and De be two definite positive matrices in Rdxd, Then,
(e( = llue mill + Tr Σl E2 - 2 (2//2>2211 2/2) 1/2
(48)
Lemma E.4. The relative Fisher information between Xo ~ N Noo, 2i)) and Xxo ~ NN(00 given by:
2d I (() oo (Ir(E)) + 11uoll²)
o² + Tr (E
Proof. The relative Fisher information between Xn and X is given by
= I So. = / Vlog (219.50(1))
(z)
)(
Pho, Fo (x)dx.
Write
v log
Xx (x)
x
-20 (x - Mo),
6 (x)
o²
that,
Lo (x)
Eo (x - nm)| ||
V log
262 (xr)
=
g2
(x - Ho) T (5= -Eo''' Eo'(x (I - Ho) ((o))))
Σn¹
02
||||22
1o (r- mo) + (o - no) T 2 (x 1o)
04
o2
First,
Xool
(Eo) + [|mol13)
01
Then,
E
T
(Xo -m0)
2
(Tr
E [XoX]]) mo ToT2o'n (m0)
2
46

---

Published in Transactions Machine Learning Research (12/2024)
Using that E [oo
20 + Hord yields
2
Xo 20 (Xo mm))
2
E
(Tr (2i (Zo + popo)) pn 2D uo)
0²
0²
2
(d + Tr (25 (ool) ) Ho 20 mo)
02
2d
Finally,
E
(Xo - 10) 2 - po)] = E [Tr ((Xo - -o0 -20² "X²(Xo mo))
E Tr (Eo (Xo (o) Mo) }
Tr (Fr [(oo 10) 140 T))
= Tr 1=x (m)
= Tr (Er
which concludes the proof.
Proposition E.5. Under the same assumptions as in Lemma E. the Euclidean norm of the score function admils the following upper bound for 1 S l2:
sup StSt0
[Vloopt, (c) V Vloggr S (12 - 41) max (llool) Ke; (2; (1 +1++8 ||=]|])
with
((ta] Amin 0²|
mi,
K1
[02 + mř, (Amin oo)) (a2 + m (Amin - =
and
9(1a) m&, 20 Me, M42 (Amin -o²) 02
K2
[02 + m (Amin o²)) (o² + m (Amin 02)]
where Amin the smallest eigenvalue of Eo-
Proof. Let t S t2
V log (x) - V Plgg (x)
-(m, 2 Eo + +o, 1a) '1 Ho) + (m Eo + -0,1a)) (x,m,to)
mo
Mig (mi, Eo + of Ia) m,, (mit En + oig 1) mo
(m, En + of -t (m² Fo + or Ia) (2.1a= 1 x
47

---

Published in Transactions Machine Learning Research (12/2024)
Writing M, = (m²Zio + oîla)
we have, for t S b2.
1
MM - -Ma2 s
i, Amin + toz mi Amin + o
(ml, - mL;) (Aein - 02)
2 + m², (Amin 0²)) (o² + mly (Amin - x
2 Sitzl
Amin - 0²
(12 11)
o²
(02 + mi, (Amin o2)) (o² + mi, (Amiu 02))
Moreover, for t S t2.
m4, mir Amin + 0/
m4z
||, Mr, me, 1
2 Amin + 0/2
(m,, mł, m,, m?, (Amiu 0²) + (m,, m4a)
(o² m (Amin d²)) (0² mi, (Muin -02)
mm, me, m4, m2, (Amin - -02) - 0²
(02 + mi, (Amin - -o2) (o² + m²g (Amiu - 02))
mi, A(t)) m mes (Amin - -0²) -22
(t2 - t1)
1 + mi, (Amin 02)) (02 + m (Amin 02)
62
Finally,
| log Pt, (x) - v log
S (t2 - +1) || R2 + (t2 (1) 1o ||=|||
S (t2 (1) max {llooll K2; x1 (1 ||=|])
E.1.2 Stochastic differential equation exact simulation
In certain cases, exact simulation of stochastic differential equations possible. In particular, due to the linear nature of the drift the forward process (1) can be simulated exactly. Indeed, the marginal distribution of (1) at time writes as
X = M,Xo + o,2,
with Z N La) independent of Xo, Xo ~ No. m, = expf S ((s)ds((20)) and of = o2 expl- S/ dss) Therefore, sampling from the forward process only necessitates access to samples from To and (OO
48

---

Published in Transactions Machine Learning Research (12/2024)
E.1.3 Noise schedules
Linear and parametric noise schedules. the form
In Section 5, we introduced parametric noise schedules
(t(t) x (o 1)/(c²T - 1),
with R ranging from -00 to 10 (see Figure 6). For all a, with a time horizon of T = 1, the initial and final values have been set match exactly the schedule prescribed by Song et al. (2021) (i.e. (0) 0.1 and (1) 20) when (linear schedule).
20.0
17.5
5.0
12.5
25
10.0
0.0
7.5
-255
5.0
-3.0
2.5
-75
0.0
Figure 6: Evolution of noise schedules 8a w.r.t. time, for different values of parameter between -10 to 10. The linear case a = 0 (Song and Ermon, 2019; Song et al., 2021) is dashed.
As shown in Section 1.1 m. and are the two quantities of interest in the calibration of the noising procedure of the forward proces. Their values for different choices of a are displayed in Figure
Cosine noise schedule. (2021) for which the forward process defined for E {1,..., T} as
We consider the cosine schedule introduced Nichol and Dhariwal
=== vI VI-600 aā,
with Xo Wdatas Z ~ N Ta) and with
f(t) f(1) = cos
f//'+S (7+8 2
ā; f(0)
To use this noise schedule in the SDE setting we notice that the forward process writes, for E [0,T],
X, m.Xo o,Z,,
49

---

Published in Transactions Machine Learning Research (12/2024)
28
13
20
53
23
23
04
20
0.8
m74
Figure 7: Evolution of m, and or over time, for different choices of a in the noise schedule 8a used see Section 5. The stationary distribution of the forward process o² set to 1. The range for a spans from -10 to 10, with the dashed line representing the linear schedule as proposed originally in the VPSDE models (Song et al., 2021).
with m. expl- So B(s)ds/(2o²)) of = o² m2) and Z N (0, La). Therefore, can simply identify Beos by solving
Boos (5) ds = log (āt)
202
which yields the following noising function:
T ( (x++1)
(oos(t) 0² (
(49)
Finally, to ensure fair comparison with the linear schedule and the parametric schedules defined in Section 5, we set in all our experiments 0.021122 so that Beos (0) = (a(0) = 0.1 for any a.
E.1.4 Discretization details of the diffusion SDE
In contrast to the forward process, described in Equation (1), which is simulated exactly, the backward process needs to be discretized. Recall that the backward process of (1) is given by:
dx.
B(1) X, 3(t) logpr (1, dt + v ((t))B, To Too 20
Consider time intervals 0 t st t.41 T, with >h Ve and T T=EN nh-
In our theoretical analysis, have considered the Exponential Integrator discretization, defined recursively for & E [t,, [++1] by
dxFi =3(1)
XFI logpT- (T- 12. TEI ) at + VB())BB. X Toc
20
50

---

Published in Transactions Machine Learning Research (12/2024)
200
Linear schedule
175
Cosine schedule
150
125
100
75
25
0.0
0,2
0.4
0.6
0.8
1.0
Figure Evolution of noising functions under the cosine schedule (orange, Bcos) compared the linear schedule (So, blue) over time with の² and 0.021122 122.. Additionally, since Bo increases unboundedly near T, clip its value to 200 for better visualization.
1.0
Linear schedule
1.0
Cosine schedule
06
0.8
0.2
0.2
Linear schedule
Cosine schedule
0.0 0.2
0.4 0.5
0.8
1.0
0.0 02
0A
0.6
0.
10
m74
Figure 9: Evolution of m. and oe for both the cosine schedule (orange) and the lincar schedule (blue) w.r.t. time, with 0.021122 and o² = 1. We clip the value of Boo by 200 for better visualization.
In the numerical experiments, we have given priority to the Euler-Maruyama discretization, which is widely used, and defined recursively for E [tx. tx+1] by
-(tx)) -02
X FEM + PT (XEM dt + / B(ta)dBr. FEM Too (50)
dx.M
51

---

Published in Transactions Machine Learning Research (12/2024)
To ensure transparency, the graphs presented in Figure 2 of Section 5.1 are reproduced in Figure 10 using Exponential Integrator discrctization scheme. As expected for fine discrotization steps (here 500 steps were used) the two schemes produce nearly identical results.
Uppar leard
0.6
94
17
92
2.0
Uppor bouard (601
9-
Upper (XL)
15
unEDE (Nl1
-0.3
80-
1.5
1.0
19
11-
82
10
10
0 Vilues
10
Values
120
50000
50
Wi Intac (V9)
Molitaa Al
2.0
Hy (fteafll (oxoot score)
(ftes joxaot scord)
-044
It.atsa (oxast scors)
Upper bound (W02
80
VPSDE (NN))
distanne
Upper bound
R0-
VPSDE (NN)
0.42
Upper bound
30000
40000
VP8DE (VN)
702 dastaoa
0040
80-
-033
20000
0.5
-02
0.36
10000
10
10
10
10
10
10
of
Vallues
(a) Isotropic setting
(b) Heteroscedastic setting
(c) Correlated setting
Figure 10: Comparison of the empirical KT. divergence (mean + std over 30 runs) (top) and W2 distance (mean + std over 10 runs) (bottom) between Tdata and f ((300) (orange) and the related upper bounds (blue) from Theorem 3.1 and Theorem 4.2 across parameter a for noise schedule Ba, d = 50 with Exponential Integrator discrotization scheme. We also show the metrics for the linear VPSDE model (dashed line) and our model (dotted line) with exact score evaluation.
E.1.5 Implementation of the score approximation in the Gaussian setting
Although the score function is explicit when Tdata is Gaussian (see Lemma F.1), we implement SGMs done in applications, i.e. we train a deep neural network to witness the effect of the noising function on the approximation error. We train a neural network architecture st E [0, T) Rd + Bd
using the actual score function as a target:
sp T, x.).
V loggr (x.)
Lexplicst (f) = E
s8 T, xx)
(m, + of4a) -1 -1 (X, + m. Ho)
where 6 - my and - at are defined Lemna E. and T u(0, T) independent of X. The neural network architecture chosen for this task is described Figure 11. The width of each dense layer mid_ featuee set to 256 throughout the experiments.
52

---

Published in Transactions Machine Learning Research (12/2024)
mid features
Linear layer
ReLU
ReLU
ReLU
Linear layer
[
output
Time embedding
mid mid_ features
Figure 11: Neural network architecture. The input layer is composed of vector in dimension d and the time t. Both are respectivcly embedded using linear transformation or a sinc/cosine transformation (Nichol and Dhariwal, 2021) of width mid features. Then, 3 dense layers of constant width mid_ features followed by ReLu activations and skip connections regarding the time embedding. The output layer is linear resulting a vector of dimension
E.2
Details on the experiments and additional results
E.2.1
Illustration of the KL bound in the Gaussian setting
Target distributions. We investigate the relevancy of the upper bound from Theorem 3.1 for different noise schedules in the Gaussian setting. We use as training sample 10a samples with distribution / (1a, Σ) for the dimension of the target distribution with different choices of covariance structure.
1. (Isotropic) Diso 0.5ld.
for d, 2. ((eteroscedassicc) hheterose E 1dd is diagonal matrix such that heterose and pheterose 0.01 otherwise.
2f
3. (Correlated) Eoor E R a full matrix whose diagonal entries are equal one and the off-diagonal terms are given by Ecom 1/vlj j'l for siit + j
< d.
The resulting data distributions are respectively denoted by daata x data
(iso)
(heterosc)
and T
(corr)
Upper bound evaluation. We leverage the Gaussian nature of the target distribution compute explicitly all the terms in the bound. On the one hand, the relative entropy in gKL, KL (Naaza )o)
53

---

Published in Transactions Machine Learning Research (12/2024)
is computed using the analytical formula for KL-divergence between two random Gaussian variable (Lemma E.2). On the other, the relative Fisher information in E I[Taaa oo)) is computed using Lemma (E.4). Moreover, the noise schedule function Ba and its primitive are analytically known, every occurrences of either of them are explicitly computed. Finally, it remains to estimate the expectation in (0, This is done via Monte Carlo estimation on 500 samples from the forward process (see Section E.1.2) for every step forward:
T-tx / Ba(t)dt. T-
PlggT--i (XT-)) (T LR, x 1 T-t
500
k=
Figures (top) of the main paper, we represent the following quantities the same graph, in dimension 50, for different values of a.
SGM data generation in dimension 50.
In blue the upper bound from Theorem 3.1. The dark blue color is used to refer to the upper bound with the contraction argument in equation (7) from Proposition C.1 while the ligther blue bound is the same bound without the contraction argument.
In orange (dotted line) the KL divergence between the target distribution Tdata and the empirical mean and covariance of the data generated using the true score function from Lemma E.1.
In orange (plain line) we represent KL datta I (0(0))0)) for a E (-10, -9.9-..... ...... 10). That is, the KL divergence between the larget distribution Tdata and the empirical mean and covariance of the data generated using the neural network architecture described in Figure 11 to approximate the score function.
In orange (dashed line) we represent (Tatta | (). That is, the KL divergence between the target data Tdata and the empirical mean and covariance of the data generated by the lincar schedule VPSDE presented in Song et al. (2021) with the neural network architecture described in Figure 11.
We generate 10 000 samples. The batch size is set to 64 and neural networks are optimized with Adam. All the KL divergences written above are computed using Lemma E.2. Due to the stochastic nature of our experiments, they are repeated 30 times so that the corresponding mean value and standard deviations of these results respectively depicted using plain and fill-in-between plots.
To disentangle the effect of each error term it is possible to plot the mixing time error (B), the approximation eтror (e, 33 and the discretization error (3) on a same graph for different values of a. However, for the schedule choice presented in Figure as B(T) is set to be 20 for every a values is pointless to display (3)) as it would not vary for different choices of schedule from 3.. The three error terms for Theorem 3.1, corresponding to the example in Figure (top) are provided below in Figure 12.
Optimal schedule versus classical choices. We investigate the gain from using the parametric schedule with a minimising the upper bound from Theorem 3.1 for d (5, 10, 25, 50} compared to using the lincar and cosine schedules (sce Appendix E.1.3) in the isotropic and correlated scttings
54

---

Published in Transactions Machine Learning Research (12/2024)
Moon lim ene 2
Moog aim enarte cortedbur Approomtion ever (116.
erer 13.00
Appraonada
12
Approimation
Enter Carbutoo
10
12
100
8.0
-10
7.5
480
Viluee
(a) Isotropic sctting
(b) Heteroscodastic sctting
(c) Correlated sctting
Figure 12: Error terms contribution from Theorem 3.1 displayed from the same examples as in Figure 2 (top).
(as mentioned in Section 5.1, up to rescaling the heteroscedastic setting boils down an isotropic setting).
To determine the optinal value a", upper bounds were initially calculated across various dimensions for range of a values from 19.0...9........................ ..... 10). This initial calculation aimed to identify preliminary minimum value. Subsequently, the search was refined around these preliminary values using finer step-sizes of 0.25 to more preciscly locate a*.
Results are given tabular form Table and in Figure 13. The parametric schedule optimized minimize the upper bound Ba- consistently surpasses the linear schedule, delivering significant improvements. This enhanced performance is shown by lower average Kullback-Leible divergence between data and the generated sample distribution, well reduction in the standard deviation of these divergences, which contributes to more stable generation. These results are competitive with or even exceed those oblained with state-of-the-art schedules such as the cosine schedule, particularly higher dimensions 25 and 50. However, one should note that this comparison may not be entirely fair, as the cosine schedule increases unboundedly near T, whercas we capped the parametric schodule B(T) 20 to align with the linear schedule described in Song et al. (2021).
E.2.2 Illustration of the Wasserstein bound in the Gaussian setting
Target distributions.
The target distributions Gaussian and are the same for the the (corr)
Kullback-Leibler bound: d data
(heterose) and T
Upper bound evaluation. We leverage the Gaussian nature of the target distribution to compute explicitly all the terms in the bound from Theroem 4.2. For the mixing time E/²s, the strong log- log-concavity constant C. is derived using Lemma 4.1 and (2 Too) is derived using Lemma E.3. For EW/, the analytical expressions for L, is given Lemma 4.1 and an upper bound to M derived in Proposition E.5. All non analytically solvable integrals estimated numerically using the trapezoidal rule, implemented with the built-in PyTorch function torch. trapezoid. To estimate e, we use Monto-Carlo simulations with 500 samples (in the same manner for the Bullback-Leiblee
55

---

Published in Transactions Machine Learning Research (12/2024)
Dimension
5
10
25
Upper bound min
1.75
1.00
1.50
2.00
Generation vahie in a²
0.001607 + 0.000462
0.005343 + 0.001155
0.026724 + 0.004046
0.095981 + 0.005485
VPSDE (linear sched.)
0.001935 + 0.000405
0.003594 + 0.001377
0.031748 1481
0.006158
0. 0.105992
0.019529
Isotropic
Cosine schedule
0.001390 0.
0.005097
0,001061
0.026900
0,001859
0.099917
0.001375
96 gain VPSDE)
%
+-1488
+15.50 %
+9.10 %
(ys Cosine)
-15.61
-4.83 %
+0066
+3.94 %
Upper bound min
2.25
1.75
1.75
2.25
Goneration vahue in a²
0.001861 + 0.000880
0.005871 + 0.001165
0.033156 + 0.003785
0.109649 + O. 0.008056
VPSDE (linear sched.)
0.002568 + 0.002708
0.006210 = 0.001816
0.038-134 = 0.010313
1.1776
0.016541
Correlated
Cosine schedulee
0.001197 0. 0,000332
0.005515
0,000775
0.040130
0,0034155
0,110515
0,001646
% gain VPSDE) (vs Cusine)
%
+5.46
+13.77 %
+18.63
-55.47 %
-6.46 %
+17.98 %
+0.78 %
Learning rate Epochs
To-4
1e-4
Te-3
1o-3
Purameters
20
30
75
150
Table 1: Comparison of the KL divergence between the target value and the generated value at at (the minimum value of the upper bound from Theorem 3.1) with the KT, divergence between the generated value by VPSDE with linear schedule and the target distribution. We display average KT divergences plus or minus standard deviations over 10 runs. The target distributions are chosen to be Gaussian with different covariance structures: isotropic (Taata) heteroscedastic (Tdata (heterose)) and
correlated (TT (corr)
0.12
0.150
Lineer
*
Cosine schedule
Gosine schedule
0.10
0 B. (KL bound)
0.125
f. IKL bound)
X divergence
KL divergencee
0.08
0.100
0.06
0.075
0.04
0.050
0.02
0.00
0.000
5
10
25
50
5 10
25
50
(a) Isotropic setting T data
(iso)
(b) Correlated setting Tatta
(corr)
Figure 13: Comparison of the empirical KI. divergence (mean value + std over 10 runs) between Tdata and the generative distribution â for different values of the dimension. The generative distributions considered are (18 (9)) obtained by the time-inhomogeneous SGM for Ba. (blue plain), (Bo..)
obtained by a standard linear VPSDE model (yellow dashed) and 7 (( obtained by using cosine schedule (orange dotted).
bound):
500
X 1, T-1x - 80 (T
ke (O..... N 1-1
sup
500
log Pr 12
T-
56

---

Published in Transactions Machine Learning Research (12/2024)
SGM data generation dimension 50. In Figures 14 (and Figures (bottom) of the main paper) represent the same graph, in dimension d = 50, for different values of
in blue the upper bound from Theorem 4.2.
in orange (dotted line) the W2 distance between the target distribution Adata and the empirical mean and covariance of the data generated using the true score function from Lemma E.1.
in orange (plain line) we represent (Tdtta N )) for a (-10, 8... 10}. That is, the W2 distance between the larget distribution Tdata and the empirical mean and covariance of the data generated using the neural network architecture described in Figure 11 to approximate the score function
in orange (dashed line) we reprensent Wa(Taata: =/0..)) That is, the W2 distance between the target data Tdata and the empirical mean and covariance of the data generated by the linear schedule VPSDE presented in Song al. (2021) with the neural network architecture described in Figure 11.
First results. We generate 10 000 samples. The batch size is set to 64 and neural networks optimized with Adam. All the W2 distances written above are computed using Lemma E.3. Due to the stochastic nature of our experiments, they are repeated ten times so that the corresponding mean value and standard deviations of these results are respectively depicted using plain and fill-in-ber between plots.
120
-066
1200
YS Ittaataa RI (NH)
WylTTers At (ckact soors)
Mifiteea fil (eccect score]
1.0
1000
Wg Ifaeea lexact
80-
VPSDE (NS)
-0.5
MP distance
VPSDE (NN)
700 diattence
800-
VPSDE (VN)
-0.4
10000
008
Uppar
60-
Upper
-0.3
5000
-044
400-
-0.2
22
100
Values
Values of
(a) Isotropic setting
(b) Heteroscedastic setting
(c) Correlated setting
Figure 14:
Comparison of the empirical 2- 2-sasserstein distance (mean value + std over 10 runs)
between Adata and ((%..)) (in orange) and the upper bound from Theorem 4.2 (in blue) w.r.t. the parameter a used in the definition of the noise schedule Ba, for d = 50. We also represent the Wassorstein distances obtained with the linear VPSDE model (dashed line) and the one obtained with the parametric model (dotted line) when the score is not approximated but exactly evaluated. (heterosc) and (c) The data distribution ddata is chosen Gaussian, corresponding to (a) (lso) (b) Tata (corr) π data
(iso) (heterosc) and (corr) are displayed in Figure Performances obtained from raw distributions for Taata: T
14. In the isotropic case (Figure 14 (a)) the curve for the upper bound (blue line) points a global minimum near the minimal values obtain by Wa(Taata: TN ((0a.0) (plain orange linc), which underlines
57

---

Published in Transactions Machine Learning Research (12/2024)
that the upper bound indeed informative in such case. However, the upper bounds obtained for the hetcroscodastic and correlated settings (Figure 14 (b,c)) are not in line with the generation results.
These observed discrepancies can be linked to the conditioning of the covariance matrices. In both heteroscedastic and correlated cases, the largest eigenvalue of the covariance matrices is not smaller than the variance stationary distribution of the forward process (set to o² = in those experiments) and Amax (I(cor)) R 15). This
violating the requirements of Lemma 4.1 (Amax (E(heteroae)) induces the default of strong log-concavity of the renormalized densities Pt. In this way, the Gaussian scenario highlights the critical influence of the covariance matrix conditioning on SGMs. Additionally, a smaller Amin (2) would increase Le and M, which in turn would increase the bound from Theorem 4.2.
Data preprocessing. As frequently done in practice, we expect better conditioning by rumning SGMs on a standardized distribution. In this way, note that Xo ~ Tdata we consider the centered standardized distribution sttand = (Xo- - μ4) with = diag(o1...... Ta) E Rdxdd a diagonal matrix with diagonal entries jj corresponding to the standard deviation of the j-th component of Xo and with [E [ E [Xo,a]] A last transformation shrinks the data into a rescaled version of Xstand defined as S = RD (Xo μ4) with k == 1/(2Amax (E(stani)) where Amax (E(stand)) is the largest eigenvalue of the covariance matrix of Staand- We then train SGMs to approximate the distribution of Secale- By doing so we ensure the applicability of Lemma 4.1 (with o² 1), as the largest cigenvalue of the covariance matrix of X is no larger than 0.5.
Adapted upper bound. We can finally adapt the upper bound from Theorem 4.2 to a rescaled setting by noting that
W2 (Taata: #) +) 1 max o3) W2 Tseaalss iT s.calll K (IS/Sd
(51)
where
seale is the distribution of scaled sample Xacale
(Ba00) scslle corresponds to the distribution of SGM trained Xscale
# is the distribution of the descaled generated samples, i.e., the distribution of D-1 k with ~ 7 (Ba.0) N,scaee
Therefore, we can evaluate the upper bound of Theorem 4.2 for scaled samples (r.h.s. of (51)), and transfer it up to constant to descaled generated samples (l.h.s. of (51)).
Results with scaled data preprocessing. The results are detailed in Figure 15 for the heteroscedastic (heterosc) and the correlated case (f) * (corr) : and are discussed extensively Section 5.1 of case (e) scaaee the main paper. Note that the minima of the evaluatod bounds now align closcly with the empirical metrics. However, the upper bound profile for the correlated case has been shifted up. This increase was anticipated due to the effect of rescaling by the largest eigenvalue of yeorr (sstadd) approximately 15, which reduces the magnitude of the values in (corr) This tends to increase the values of L, and
M through the effect on Amin ( (Sttad))) as explained above. Despite this effect, these experiments confirm the overall utility of the bound for selecting the appropriate noise schedule. The effect of
58

---

Published in Transactions Machine Learning Research (12/2024)
data rescaling on the Lispschitz continuity and log concavity of the true score function V log Ps are illustrated in Figure 16 the Heteroscodastic sctting.
50000
50-
Walffssss (NN)
0.44
Wallfaatt A (NN)
1.75
Upper bound (W2)
VPSDE (NN)
Vy(maua..)) (exact score)
0.42
Upper bound (W2)
40000
VPSDE (NN)
Wallagass fi (exact score)
-1.50
40-
792 distance
1.25
702 distance
0.40
30000
1.00
30
-..8
0.75
20000
0.50
0.36
20
10000
0.25
-10
5
0
10
-5
5
Values of
Values of
(d) Heleroscedastic rescaled setting
(e) Correlated rescaled setting
Figure 15: Comparison of the empirical 2-Wasserstein distance on rescaled datasets for (d) π seaale
(heterosc)
(e) a (cor)
Optimal schedule versus classical choices. We investigate the gain from using SGM with the schedule a² minimising the upper bound from Theorem 4.2 for d e (5, 10, 25, 50} compared to the linear and cosine schedules (see Appendix E.1.3). To determine the optimal value a", upper bounds were initially calculated across various dimensions for range of a values from {-10,- -9...., 10). This initial calculation aimed to identify a preliminary minimum value. Subsequently, the search was refined around these preliminary values using finer step-sizes of 0.25 to more precisoly locate a".
Results for the isotropic, heteroscedastic, and correlated cases are presented both tabular form in Table 2 and visually in Figure 3 within the main paper. These findings are discussed in Section 5.1 of the main paper.
E.2.3 Numerical experiments on more complex synthetic data
In the context of complex data distributions, the Kullback-Leibler bound (Theorem 3.1) appears to be of limited practical applicability. Specifically, cKT.(0,,3) implies that for each noise schedule tested, distinct score approximation (t, must be trained. This requirement renders the bound computationally intensive and therefore not realistically usable. Additionally, ( is independent of the schedule choice over (O, as it depends solely on its final value 3(T) which set constant our empirical setting (for all a, (T) 20). As a consequence, the last remaining error term to analyse the bound through the lens of noise schodules the mixing time (B). However, relying exclusively on (B) would suggest selecting a schedule + B(t) that maximises So dtt) As demonstrated in Section 5.1, this approach clearly fails to yield the schodule choices near the optimal solution.
Therefore, more reliable choice would be use the W2 bound of Theorem 4.2 for which most of the terms can be computed explicitly with reasonable computational cost in the Gaussian sctting.
59

---

Published in Transactions Machine Learning Research (12/2024)
Dimension
10
25
Upper bound min a"
4.5
4.25
3.75
4.25
Geveration value in d'
0,0:99241
0.012572
0.059274
0,008438
0.130829 :: 0.014245
0.233812 + 0.010584
VPSDE (linerr sched.)
0.036995
0.00-1683
0,063939 + 0.010876
0.141601
0,020417
0.256384
0.082709
Isotropic
Cosine schedule
0.030996
0.003254
0.0006499 + 0.007117
0.131234 + 0.004794
0.251950 + 0.005558
% gain (es VPSDE)
-6.07 %
+7.30 %
47.61 %
4-8.79 %
gain (vs Cosine)
%
+2226 %
+0.31
+7.20
Upper bound min a*
4.00
3,25
2.00
2.75
Generation value in o²
0.0965922 +: 0.003062
0.143224
0.004809
0.242493 +: 0.004769
0.372292 4: 0.004604
Heterose,
VPSDE (linear sched.)
0.0988899 1 0.003660
0.147478
1: 0.009638
0.249144 + 0.011394
0.385612
0.0098313
(with rescaling)
Cosine schesdule
0.0964377 0.002380
0.149701
0.002460
0,250520
0.004448
0.374868
0.003243
% gain (ys VPSDE)
+2.32 %
+2.89 %.
+22677 %
+3.46 %
%6 gain (vs Cosine)
-0.16 %
+0033 %
+33200 %
+00.69 %
Upper bound min a"
8,00
8:75
10,30
11.00
Generation value
0.009548
0.013873
0.107291
0.028451
0.261075 0.029633
0.676151 + 0.123277
Correlated
VPSDE (linear sched.)
0.072068
0.019861
0,1382 + 0:081119
0.302996 = 0.0:55539
0.897581
0.079860
(with resculing)
Cosine schedule
0.048276 + 0.008605
0.11 + 1284
0.391753 + 0 0.030112
O. 0.765524
0.022376
% gain (es VTSDE)
+7.65 %
+22.36 %
+13.81 %
+24.68 %
% gain (va Cosine)
-37.77 %
+4.96 %
+33.31 %
+11.67 %
1o-3
Learning rate
le4
le-4
(le-4 for Corr.) 75
(1e-4 for Corr.) 150
Parameters
Epochs
20
30
Table 2: Comparison of the W2 distance between the target value and the generated value a" (the minimum value of the upper bound from Theorem 4.2) with the W2 distance between the generated value by VPSDE and the larget distribution. We display averages plus or minus standard deviations over 10 runs. The target distributions are chosen to be Gaussian with different covariance structures: isotropic, heteroscedastic (with rescaling applied), and correlated (with rescaling applied).
10
Anisotropic distribution (original)
Anisotropic distribution (rescaled)
0.2
0.4
0.6 Diffusion time t
0.8
1.0
Figure 16: Comparison of the ratio song concavity / Lipschitz continuity for the true score function V log Pt in the Heteroscedastic setting before rescaling (Figure 14 (b)) and after rescaling (Figure 15 (d)) throughout diffusion time t E (0,T).
In particular, we leverage the Gaussian framework to estimate the constant terms and apply the rescaling defined in Appendix E.2. to ensure that C, is non negative for t (0, T). More preciscly,
60

---

Published in Transactions Machine Learning Research (12/2024)
Le and O₁ are given in Lemma 4.1 and are computed using the empirical covariance matrix associated with Tacale (and using when applicable the refinements in Propositions D.1 and D.2),
M is derived with Proposition E.5 with appropriate empirical estimators,
(T xoo) is computed using closed-form formulas for Gaussian distributions, involving empirical estimators of the mean and covariance of Tscale:
the term is deliberately omitted to avoid the prohibitively high computational costs associated with training distinct models for different noise schedules.
The experiments are run using the same neural network architecture as in the Gaussian illustraations of Appendices E.2.1 and E.2.2 (i.e., a dense neural network with 3 hidden layers of width 256). The network was trained over 200 epochs for {-10, -9,... ... 19, 20)- Contrary to the Gaussian case, conditional score matching Lecore (0) (5) is used, as being closer to what is done in practice (explicit scores are now out of reach). To assess the quality of the data generation three metrics are used:
(a) an estimator of the KL-divergence based on k-ncarest neighbors (Wang al., 2009) with [vd] neighbors,
(b) the sliced 2- Wasserstein distance (Flamary al., 2021) with 2000 projections,
(c) the negative log likelihood computed on 1000 samples defined as - 1000 E log detta dx;) with (zs)1sis100 1000 samples from the generated distribution and Taata the probability density function be estimated.
Funnel distribution. The first distribution considered is the Funnel distribution (Thin et al., 2021) in dimension 50, defined as
Tdata(x) [I Pexp (2bz, (Iz)
II
with and b b= 0.5. To ensure the applicability of Theorem 4.2 and Lemma 4.1 the samples standardized and rescaled according to the method described in Appendix F.2.2. The results, illustrated in Figures 4 and 17, show that the upper bounds effectively mirror the generation oulcomes across the three metrics considered. Moreover, the generation results for the parametric schedule at (the one that minimizes the upper bound) outperforms in all three metrics both the lincar and cosine schodules (see Table 3).
Gaussian mixture models. The second distribution considered is Gaussian mixture model with 25 modes in dimension 50, defined as
Taata (x) 25
Σ
(x)
(J,))(-22....2)
with E, denoting the probability density function of the Gaussian distribution with covariance matrix En diag (0.01, 0.01, 0.1, 0.1) and mean vector pya [j. O..., OoT. The results shown in Figure 18 and Table 3 confirm the relevance of the upper bound even for non-Gaussian datascts.
61

---

Published in Transactions Machine Learning Research (12/2024)
Param. schod, Pa
KNN approx, Kullback- Kulbback-Libller Liibler
Negalive log- ag--eellhood
Cosine schod, Brm
10
Cosine echedule
Negalive log-lkkllhood
Upper bound (W2)
25
Upper bound (W2)
99
20
-85
15-
15-
-10
0
10
20
-10
0
10
20
Values
Values
(b) KL divergence with k-nearest neighbors estimate
(c) Negative log- log-likelihood
Figure 17: Upper bound and empirical distances between the data distribution and the generated samples for different metrics on Funnel dataset. dimension 50.
22.5
-.10
22.5
schod, F
22.5
Cowne sched. Ros
Coaane scheel A.
(N02)
20.0-
Upper bound (NN))
20.0
4.0
Upper bound (W22)
20.0
Nogatio kop Beathooo
17.5
17.5
17.6
15.0-
15.0-
15.0-
12.54
0.04
12.54
125-
16.0
10.0
10.0
-10
20
-10
10
20
-10
20
a
Values
(a) Sliced-Wa distance
(b) k-nearsst neighbors estimator
(c) Negative log-likelihood
Figure 18: Upper bound and empirical distances between the data distribution and the generated samples for different metrics on mixture of 25 Gaussian variables datasct in dimension 50.
E.3 Numerical experiments on real-world datasets
To evaluate the impact of the noise schedule the performance of score-based generative models evaluate the parametric family Sa introduced in Equation (9) using CIFAR 10 dataset. We suggest to analyse the FID (Fréchet Inception Distance) score on 50 000 samples generated for different noise schedules (different values of in Ba, see Figure 1) on CIFAR 10.
We use pretrained models from Karras et al. (2022) with the recommended hyperparameters designed to replicate the experiments Song al. (2021) corresponding to our linear schedule (a 0) as shown in Figure 1. In particular, we let 7' 1, B(0) 0.1, 3(T) 20, 1000 discretization steps and sample over the diffusion [e, 1] with 10
The training process in Karras et al. (2022) is slightly different, though equivalent, to the original implementation. In particular, the networks are not trained to directly estimate V log pt (X,). .). Instead, denoiser function (g(X, a) is trained to isolate the noise from the signal for some noise level (sce Equations (2) and (3) in Karras et al. (2022)). With appropriate rescaling this denoiscr
62

---

Published in Transactions Machine Learning Research (12/2024)
Metric
Sliced Wasserstein
k-mn (Kullback Leibler)
NLL
Generation value in a²
0.218498 +: 0.049882
4.242455 :: 0.450224
82.25179 1: 3.1250
VPSDE (linear sched.)
0.240664
0.036378
6,0484403
0.726221
87,028933
33406112
Cosine schedule % gain (ys VPSDE)
0.221851
0.054309
4.927209 0.510968
63.73291
3.53262
Finnel distributiin
+99.21 %
20.88 %
+-5.49 %
% gain (vs Cosine)
+1.51 %
-1399 %
+1.77 %
Geueration value in VPSDE (linear sched.]
0.043388 + 0.005222
2.433759 = 0,180652
35.033176 + 1.97863
0.037763
0.004450
3. 3.003054
0. 0.126697
40.49867 3.13705
Cresie schedule
0.046816 +: 0.008402
2.541213
0.158563
34.76353 t: 2,20980
Gaussian mixture models
gain (va VPSDE)
+24.91 %
20.35 %
+ +33.49 %
% gain (vs. Cosine)
+7.32
+4,23
Learning
le-3
Io-3
le-3
Parsmeters
Epachts
200
200
200
Table 3: Comparison of the sliced-W2 distance, KL, divergence coupled with k-ncarest noighbors estimate and negative log-likell log-liiel between the target distribution and the SGM-generated one. For the latter, the SGM either trained with linear, cosine and Sa- schedules. We display averages plus or minus standard deviations over 10 runs. The target distributions are chosen are Funnel and Gaussian mixture models.
be used in the VP setting by letting
o7 m
Xx my
xx m
so X
t. t)
Do
m
where se is the score approximation defined in our paper, m. expl- ((s)ds/(2o)) and of = o1 (1)).. This formulation bridges the denoising approach with score-based methods in the VP framework.
Figure 19 displays the FID score for samples generated using the the backward process for different choices of 3a with E (-10, -9...., 10} and cosine schedule Peas- Although the assumptions of our results cannot be verified in such a setting, is interesting to note that the empirical performance follows the same dynamics as in the loy numerical experiments. This indicates that the analysis and optimization of noise schedules is interesting problem to be explored further for complex cases.
discretization
F Conditional training in the Gaussian setting
Section 5.1 of this paper is dedicated to the illustration of the theoretical upper bounds and their relevance in the Gaussian setting (i.e., when Taata is Gaussian). This choice has been motivated by the fact that, under this setting, all constants in the upper bounds from Thcorem 3.1 and Theorem 4.2 are either analytically available could be precisely estimated (see Appendices E.2.1 and E.2.2).
In particular, both upper bounds display error terms proportional to Lexplicn (4), which has motivated the use of explicit score matching during the training. To do so, we used a deep noural architecture (see Figure 11) trained to minimize Loxplicit (0) (4) using as a larget the true score function. This is possible because in the Gaussian setting, the true score function is analytically known (Lemma E. However, in most applications the score function is not available, because
63

---

Published in Transactions Machine Learning Research (12/2024)
Score param. sched. 8.
FID Score cosine sched. Beos
14
Scores
12
10
-10.0
-7.5
-5.0
-2.
0.0
2.5
5.0
7.5
30.0
Values a
Figure 19: FID scores on 50 000 generated samples using pre-trained models from Karras et al. (2022) with different noise schedules from the parametric family (9) and cosine schedule.
the data distribution is not known and has be learned. This the reason why, practice we rely on conditional score matching (i.e., the minimization of Lecore (5)). This approach
particularly relevant given the relationship between the explicit and conditional score functions:
Lexplicit (6) Lecare
E V logg-((X (x+) V (ogg--(Xx1X0)11
Consequently, all the theoretical upper bounds discussed Sections and can be adjusted by a constant (with respect 0) to account for discrepancies between the score function learned through Lucore Lexplicit-
The rest of this section demonstrates the numerical effects of employing conditional score matching instead of explicit score matching, following the numerical set-up of Appendices E.2.1 and E.2.2. In Figure 20, the Kullback-Leibler upper bound from Theorem 3.1 is depicted in varying shades of blue, while the empirical KL(Tdata|| (8).))) across parameters {-10, -9, -8, ..., 10} is shown in varying shades of orange.
Figure 20, three learning scenarios are presented: one using explicit score matching (which exactly matches the results Figure? 2 (top)), another with conditional score matching over 150 epochs, and third with 300 epochs. Both the generation results and the upper bounds show diminished performance the curves shifted upwards. Nonetheless, the overall curve shapes similar, and the optimal points remain closely aligned. Intcrestingly, both the upper bounds and the generation outcomes in the conditional scenarios demonstrate more pronounced peaks near the minimum values. This suggests that precise noise schedule selection may yield even better performance gain when SGMs are trained using conditional score matching.
Additionally, Figure 21 demonstrates that increasing the number of training ilerations when using conditional score matching provides results more and more similar to that obtained with explicit score matching. This effect is noticeable in both the KL divergence and the W2 distance.
64

---

Published in Transactions Machine Learning Research (12/2024)
1.4
12-
-12
Upper foundd D512
Mi/ra (NN) jonslist 190 spcers)
A./a R (NS) toonioiooaal 150 apochs)
0.8
fi joandiiond
sound (oxplot)
0.66
sound (conationsl spccnal
Upper bound (condioous epocha)
-02
75
Values
Isotropic setting
O dieeggnnc
Ki/ A (NN) josolist 192 socehs)
boundd
KR/ran Ai (NN) joanetionaal epocte)
jonnetionnl apoche)
(ouplot)
souns (conditiaul spcens)
Upper bouns gpondiious spocna)
-7.5
0.0
7.5
Vistowe
Heteroscedastic setting
255
Ka/ (l) (NN) josokit 191 sacehs)
KL/ A (NN) joanetionall epocte)
jounetional apoche)
Upper
10
(oxplot)
1.0
sound (conditaui spcens)
Upper bouns gponeiloue apocta)
-75
-50
-25
7.5
Villoe of
Correlated setting
Figure Comparison of the empirical KL divergence (mean value std over 10 runs) between Wdata and () (in orange) and the upper bound of Theorem 3.1 (in blue) w.r.t. the parameter N used in the definition of the noise schedule Ba, for = 50.
65

---

Published in Transactions Machine Learning Research (12/2024)
14
12-
Witees jsaoct
Upoer bounnd (KL)
6
Mt Al (NK) jessiet 1S0 azoets)
4./ee Ri (NN) foondsondl epoens)
000
joandioonl 1000
bound (oxpke))
406
Upper bound joordiaanal asooha)
Upper bound (pondiional 1000 spache)
-02
7.5
140
Upper bound (V02)
03
Ci d1 (NN) (esliitt 150 opocta)
Mi (98)
spocis)
(95) joanciiional 1002 spcens)
-00
Upper
Upper ascoho)
Upper bound spectro
60-
-00.0
10.0
Values ofa
Figure 21: Comparison of the empirical KL divergence (top) and the W2 distance (bottomn (mean (iso) value + std over 10 runs) between Taata data and (8a.0) (in orange) and the upper bound of Theorem 3.1 (top) and of Theorem 4.2 (bottom) (in blue) w.r.t. the parameter used in the definition of the noise schedule Ba, for d = 50.
66


Reviews:
[{"review_id": "TkGOrmOEM0", "reviewer": "Reviewer_cnqp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The contributions, assumptions, and analysis are clear.\n2. The paper is well-situated in the related work and extends it significantly.\n3. The numerical experiments provide some hints into the potential usefulness of the derived bounds.\n\nWeaknesses:\n1. The paper would benefit from expanding the discussion with the practical benefits gained from deriving these tighter theoretical bounds (such as some which have been hinted at in the experiments section).\n2. The upper bound is still very loose relative to the exact empirical computations, e.g. Figure 2. \n3. The paper suggests a potential improvement to the noise schedule using the upper bound but does not demonstrate whether this schedule has practical benefit in more real-world settings. Even in the synthetic setting, though, the improvement seem minimal at best.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper derives upper bounds for the KL divergence and 2-Wasserstein distance between the target and estimated distribution of a time-inhomogeneous score-based generative model which explicitly depend on the noise schedule. The numerical experiments show relationships between the choice of noise schedule and quality of the estimated model which align with those suggested by the bounds. The authors also use the upper bounds to propose an alternative noise schedule (based on what would approximately minimize the upper bound) which seems to outperform common noise schedules in synthetic experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "For major changes, please see the weaknesses above. More broadly, the paper could benefit from further discussion / analysis around the main theoretical results.\n\nOther minor notes:\n1. There's an unintentional paragraph break under the paragraph titled \"Comparison with existing bounds.\"\n2. Several statements in the introduction could be made more precise, including \"It is crucial to note that the complexity of real data prohibits a thorough depiction of the distribution $\\pi_\\text{data}$ through a conventional parametric model, and its estimation via traditional maximum likelihood methods\" and \"However, knowing the score amounts to knowing the distribution at time $t = 0$, i.e., knowing the distribution $\\pi_\\text{data}$ according to which we wish to simulate new examples.\" Namely, in the former it may be too strong to say \"prohibitive\" given the empirical success of many generative models that are directly trained via maximum likelihood; in the latter, \"knowing the distribution\" is a bit imprecise.", "broader_impact_concerns": "I see no ethical concerns with the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uMIVTqJge9", "reviewer": "Reviewer_goCE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The paper is clearly written and well-structured.\n- Sections 1 and 2 introduce well the framework and the notation used in the rest of the paper.\n- The Wasserstein bound presented in Theorem 4.2 is interesting and, as far as I know, a novel contribution to the field.\n\n**Weaknesses** \n- (a) The derivation of the Bound on the KL divergence presented in Section 3 follows step by step the one of Conforti et al. (2023). It looks to me that keeping the function $\\beta(t)$ generic does not require any additional technical step in the proof. Additionally, from the expression given in Theorem 3.1 it seems that $\\beta(t)$ does not play an important role in the bound. This makes the result a very marginal contribution in my opinion.\n- (b) The aim of section 5 is supposedly to \"numerically illustrate the validity of the bounds\" obtained in the previous sections. However, I find the results presented rather inconclusive: First, as can be seen from the left y-axis in Figs. 2 and 4, the upper bounds are always at least one order of magnitude larger than the numerically calculated values. The authors still claim that the bounds are \"clearly relevant\" and that they \"align with the empirical performance\" because taking the value of $a$, called $a^*$, that minimises the bound and calculating the error from the resulting noise schedule leads to better performance than typical noise schedules used in practice.\nHowever, I think it should be noted that in all cases analysed in the paper, $a^*$ is almost the same (between 1 and 5), regardless of the data set considered. Moreover, looking at the results in Tables 1 and 2, the performance of the $a=a^*$ schedule is similar to that of typical schedules within the standard error deviations, which means that the cases in which it seems to perform better could just be due to the limited statistics of the experiments (from Fig.3 it seems that only 10 experiments were performed for each point).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies score-based generative models theoretically, by presenting upper bounds on the error committed by these models in the estimation of the target measure. The emphasis is put on how the noise schedule chosen for the model affects these bounds. \nFirst, the authors analyse the error using the Kullback-Leibler (KL) divergence and generalise the recent results of Conforti et al. (2023) to find an upper bound for a generic noise schedule.\nThen, using the Wasserstein distance, a new upper bound is derived for score functions satisfying some additional regularity assumptions.\nFinally, the paper presents numerical simulations on some relevant probability distributions, for which the empirical upper bounds can be computed and compared to the actual values of the error.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The \"contraction\" argument in Proposition D.1, used to prove Theorem 4.2, is similar to the one used in \"Chain of Log-Concave Markov Chains\" by Saremi et al. (https://arxiv.org/abs/2305.19473) to prove Theorem 1. Therefore, in my opinion, the sentence after Theorem 4.2 that claims \"this feature has never been considered\" should be revised.\n- (See weakness b): Section 5 needs major revision. I would advise to focus more on the theoretical result of section 4 and less on how much the result itself is of practical interest, unless more convincing numerical results can be found.", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Twkjp3wIRf", "reviewer": "Reviewer_425e", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n1. Refined theoretical analysis of diffusion models based on time-inhomogeneous VPSDEs, providing upper bounds for both the KL divergence and Wasserstein distance between the target and generated distributions that reveal the role of the noise schedule.\n\n2. Empirical validation of the upper bounds through experiments on toy distributions, indicating potential benefits of optimized noise schedules and data preprocessing.\n\n### Weaknesses:\n\n1. **Limited discussion of novel techniques and related work:** The paper features a short discussion of the KL bounds and mentions that the $\\mathcal{W}_2$-bounds leverage the contraction of the backward diffusion. However, apart from that, the main part of the paper does provide many details on comparing their proof techniques and contributions to previous work. Further comments on related works:\n\t- \"Note that all the mathematical theory for diffusion models developed so far covers general time discretizations of time-homogeneous SGMs\": What about the work by De Bortoli (2022), which is also cited several times in the manuscript? It would be good to add a corresponding comparison.\n\t- \"an exact simulation can be performed for this process\": It should be noted that this is well-known.\n\t- \"The Euler Exponential Integrator (EI, see Durmus and Moulines, 2015), as already used in Conforti et al. (2023)\": Note that it has also been proposed or used in https://arxiv.org/abs/2204.13902 and De Bortoli (2022).\n\t- \"It has already been used by Conforti et al. (2023)\": To some extend, the trick has also been used in https://arxiv.org/abs/2302.13834.\n\t- \"In the literature, when $\\beta(t)$ is constant equal to 2 (meaning that there is no time change), this diffusion process is referred to as the Variance-Preserving SDE\": Song et al. (2021) already define the VPSDE for general, non-constant noise schedules. This is also noted later: \"the schedule is linear and matches exactly the classical VPSDE implementation\"\n\n2. **Limited practical relevance of the empirical results:** While the experiments validate the theoretical results, there is still quite a large gap between relevant, high-dimensional problems and state-of-the-art implementations of diffusion models. It is unclear if and how the findings of this work would translate to practically relevant settings. Further comments on the empirical results:\n\t- Is there any explanation for the quite large gap to the actual KL divergence/$\\mathcal{W}_2$-distance? Given this gap, one might ask why the tendency of the upper bound is still often reflected in the empirical results, which is crucial for claiming that better noise schedules are given by smaller upper bounds, e.g., in \"upper bound remains clearly relevant to assess the efficiency of the noise schedule used during training\".\n\t- \"values for $a$ improving over the classical linear noise schedule\": It should be emphasized earlier that the cosine schedule seems to be a classical, commonly used schedule for VPSDEs as well. Thus, it would also be interesting to see its performance in the plots of Figure 2.\n\t- \"For backward process simulation, we use an Euler-Maruyama scheme with 500 steps, as being the most encountered discretization in practice\": Did you also try the EI to follow the theoretical results more closely?\n\t- \"the refinement relying on contraction arguments specific to the Gaussian setting (see (7)) is indeed more informative in such a case\": could you show the corresponding bound in the plot to validate this statement?\n\t- \"Moreover, implementing SGM with the optimal parameter a yields consistent improvements of the data generation quality across different metrics w.r.t. to classical noise schedule competitors (linear or cosine)\": From Figure 4, it seems that the (untuned) cosine schedule would be approximately as good as the schedule with tuned $a$.\n\t- \"significant improvement in the data generation quality\": It seems to rather be a minor improvement. \n\t - The experiments show that data preprocessing helps and the lower ratio of $L_0/C_0$ is offered as a potential explanation. I think it would be good to provide numerical evidence and present a plot of performance vs. ratio (for suitably rescaled target distributions).\n\t- It would be helpful to remove the dashed line in Figure 2 (and similar figures), since it seems not to add additional information but can be confusing to the reader. Moreover, it might be valuable to depict the dotted line (exact score evaluation) in a separate plot and compare against the corresponding upper bound for exact score evaluation.\n\n3. **Limited scope:** The work seems to only cover noise schedules for VPSDEs but not for other SDEs which gained popularity, see, e.g., https://arxiv.org/abs/2206.00364.\t   \n \n\n**Additional minor issues:**\n\n1. H3 should probably hold in each time interval $[t_k, t_{k+1}]$? Also, without a specific bound, I would not call it a \"guarantor of a good approximation of the score by the neural network\".\n2. \"However, knowing the score amounts to knowing the distribution at time t = 0\": This seems not even sufficient, since the score cannot be obtained in a tractable fashion even if the density of the distribution at time t=0 was known.\n3. \"with $h$ [...] small enough\": I think it would be helpful to provide an estimate on how small $h$ needs to be. Similar for Thm 4.2.\n4. \"If the result was derived for the EI discretization scheme\": This sounds like the result has not been derived for the EI discretization scheme.\n5. \"and adjust schedules so that they all start at [...]\": It would be good to specify how the schedule is rescaled.\n6. \"By considering early stopping techniques (typical in the literature on SGM), we could adapt these bounds to more general data scenarios (e.g., not strongly log concave)\": How is this exactly meant?\n\n**Typos:**\n\n1. \"converge exponentially fast\": converges\n2. \"Isotropic, denoted by $\\pi^{(\\mathrm{corr})}\\_{\\mathrm{data}}$\": should be $\\pi^{(\\mathrm{iso})}\\_{\\mathrm{data}}$\n3. \"Wasserstein distances provided by (...\": Use `\\citet` instead of `\\citep`.\n4. \"on Gaussian mixture model\": on *a* Gaussian mixture model\n5. \"multi-dimensional noise schedule\": schedules\n6. \"particular interest what to deal with\": how to deal with\n7. \"restricted to a compact,\": restricted to a compact *set*?\n8. \"in top of\": on top of\n9. \"in function of\": as a function of\n10. \"for Wasserstein bound for\": bounds", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides refined theoretical upper bounds on the KL divergence and Wasserstein distance between samples from a diffusion model and the data distribution. The bounds describe the dependence of the errors on the mixing behavior of the forward process, the approximation error of the (normalized) score, and the discretization error of the numerical integrator. In particular, the bounds are explicit in the chosen (time-inhomogeneous) noise schedule as well as the properties of the target distribution, providing insights into their impact. A series of numerical experiments on toy distributions show that the theoretical bounds can potentially guide the selection of noise schedules and data preprocessing.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Comparison with existing bounds and related works. It would be good to add a discussion on what the novel proof techniques are, e.g., why the proofs (in particular for the KL divergence) do not follow from time-rescaling arguments, or more careful tracking of the involved quantities related to the noise schedule in previous proofs? Moreover, it would be good to also add sections like \"Dissecting the upper bound\" and \"Comparison with existing bounds\" for the $\\mathcal{W}_2$-result. \n\n2. The discussion in the last section should provide a more comprehensive summary and outlook.\n\n3. Can one add a more relevant general target distribution in Section 5.2, e.g., a simple image dataset such as CIFAR-10?\n\n4. Fixing the minor issues and comments mentioned above.", "broader_impact_concerns": "--", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After careful consideration of the reviews, the authors' responses, and the revised manuscript, my recommendation is for acceptance of the revised manuscript. 

The paper presents clear and rigorous theoretical contributions, particularly in deriving novel upper bounds for Wasserstein distance and exploring the role of noise schedules in score-based generative models. While the work's practical relevance remains limited, the authors have thoroughly addressed reviewer concerns, providing additional experiments and clarifications that enhance the manuscript. 

Importantly, all reviewers ultimately leaned toward acceptance, recognizing the paper's value to the theoretical understanding of score-based generative models.
Decision: Accept (poster)