Error Bounds for Flow Matching Methods
Authors: Joe Benton
George Deligiannidis
Arnaud Doucet
Venue: Accepted by TMLR ()

Abstract: Score-based generative models are a popular class of generative modelling techniques relying on stochastic differential equations (SDEs). From their inception, it was realized that it was also possible to perform generation using ordinary differential equations (ODEs) rather than SDEs. This led to the introduction of the probability flow ODE approach and denoising diffusion implicit models. Flow matching methods have recently further extended these ODE-based approaches and approximate a flow between two arbitrary probability distributions. Previous work derived bounds on the approximation error of diffusion models under the stochastic sampling regime, given assumptions on the $L^2$ loss. We present error bounds for the flow matching procedure using fully deterministic sampling, assuming an $L^2$ bound on the approximation error and a certain regularity condition on the data distributions.

Published in Transactions on Machine Learning Research (02/2024)
Error Bounds for Flow Matching Methods
Joe Benton
benton@stals..zz www.aalls.oo...co
Department of Statistics University of Oxford
George Deligiannidis
deligian@stats.or
Deportment Statistics University of Oxford
Arnaud Doucet
doucet@stots.or.
Department of Statistics Universily of Ouford
Reviewed on OpenReview: https: httpe:///penneevieu net/ forum? i id=ugOPyWFDhy
Abstract
Score-based generative models popular class of generative modelling techniques rely- ing stochastic differential equations (SDEs). From their inception, it was realized that it was also possible to perform generation using ordinary differential equations (ODEs) rather than SDEs. This led to the introduction of the probability flow ODE approach and denois- ing diffusion implicit. models. Flow matching methods have recently further extended these ODE-based approaches and approximate a flow between two arbitrary probability distribu- tions. Previous work derived bounds on the approximation error of diffusion models under the stochastic sampling regime, given assumptions on the L loss. We present error bounds for the flow matching procedure using fully deterministic sampling, assuming an I bound on the approximation error and certain regularity condition on the data distributions.
1 Introduction
Much recent progress in generative modelling has focused on learning a map from an casy-to-sample reference dist rib æn a target distribution л. Recent. works have remensttated how to learn such a map by defining a stochastic process transforming mo into कं and learning to approximate its marginal vector flow, a
procedure known flow matching (Lipman et al. 2023 Liu ct al. 2023 Albergo & Fanden-Eijnden 2023
2023 Heitz et al. 2023) Score-based generative models (Song et al. 2021b [Song & Ermon 2019 Ho et al.
Albergo et al.
2020 can be viewed as particular instance of flow matching where the interpolating paths
are defined via Gaussian transition densities. However, the more general flow matching setting allows one
to consider a broader class of interpolating paths, and leads to deterministic sampling schemes which are typically faster and require fewer steps (Song et al. 2021a Zhang & Chen 2023).
Given the striking empirical successes of these models generating high-quality audio and visual data (Dhariwal & Nichol 2021 Popov et al.) 2021 Ramesh et al. 2022 Saharia et al. 2022), there has been significant interest in understanding their theoretical properties. Several works have sought convergence
guarantecs for score-based generative models with stochastic sampling (Block et al. 2022 De Bortoli 2022 Lee el al.) (202)), with recent. work demonstrating that the approximation error of these methods decays polynomially in the relevant problem parameters under mild assumptions (Chen et al. 2023c Lee et al.
2023 Chen al. (022a)). Other works have explored bounds in the more general flow matching sctting (Albergo & Vanden- Eijnden 22023 Albergo et 2023). However these works either still require some stochasticity in the sampling procedure or do not hold for data distributions without full support.

---

Published in Transactions on Machine Learning Research (02/2024)
In this work we present the first bounds the error of the flow matching procedure that apply with fully deterministic sampling for data distributions without full support. Our results come in two parts: first, we control the error of the flow matching approximation under the 2- 2-aasserttein metric in terms of the L training error and the Lipschitz constant of the approximate velocity field; second, we show that, under a smoothness assumption explained in Section 3.2 the true velocity field is Lipschitz and we bound the associated Lipschitz constant. Combining the two results, we obtain bound on the approximation error of flow matching which depends polynomially on the L2 training error.
1.1 Related work
Probability flow ODEs were originally introduced by Song et al. (2021b) in the context of scorc-based generative modelling. They can be viewed as an instance of the normalizing flow framework Reeznde & Mohamed] 2015 Chen et al. 2018), but where the additional Gaussian diffusion structure allows for much faster training and sampling schemes, for example using denoising score matching (Vincent 2011
Flow methods:
compared to previous methods (Grathwohl et al. 2019 Rozen et al. 2021 Ben-Hamme et al. 2022).
Diffusion models are typically expensive to sample and several works have introduced simplified sampling training procedures. Song et al. (2021a) propose Denoising Diffusion Implicit Models (DDIM), method using non- Markovinn noising processes which allows for faster deterministic sampling, Alternatively, (Lipman
et al. 2023 Liu et al. 2023 Albergo & Vanden-Eijnden 2023) propose flow matching as technique for simplifying the training procedure and allowing for deterministic sampling, while also incorporating much wider class of possible noising processes. Allorgo et al. (023) provide an in depth study of flow matching methods, including a discussion of the relative benefits of different interpolating paths and stochastic versus deterministic sampling methods.
Error bounds: Bounds on the approximation error of diffusion models with stochastic sampling procedures have been extensively studied, Initial results typically relied either on restrictive assumptions on the data
distribution Lee et al. 2022 Yang Wibisono| 2022) or produced non-quantitative or exponential bounds
2022 De Bortoli et al. 2021 De Bortoli 2022 Block et al. 2022.. Recently, several works have derived polynomial convergence rates for diffusion models with stochastic sampling (Chen)
[Pidstrigach] 2022 Liu
et al. 2023c Lee et al. 2023
Chen et al. 2023a Li et al. 2023 Benton et al. 2023 Conforti et al. 2023).
By comparison, the deterministic sampling regime is less well explored. On the one hand, Albergo & Vanden- Eijnden )202)) give a bound on the 2- 2-Wasserstein distance between the endpoints of two flow ODEs which depends on the Lipschitz constants of the flows. However, their Lipschitz constant is uniform in time, whereas for most practical data distributions we expect the Lipschitz constant to explode as one approaches the data distribution; for example, this will happen for data supported on a submanifold. Additionally, Chen et al.] (2023d) derive polynomial bound on the error of the discretised exact reverse probability flow ODE, though their work does not treat. learned approximations to the flow. Li et al. 2023 also provide bounds for fully deterministic probability flow sampling, but also require control of the difference between the derivatives of the true and approximate scores.
On the other hand, most other works studying error bounds for flow methods require at least some stochas- ticity in the sampling scheme. Albergo et al. (2023) provide bounds on the Kullback-Leibler (KL) error of flow matching, but introduce a small amount of random noise into their sampling procedure in order to smooth the reverse process. Chen et al. (2023b) derive polynomial error bounds for probability flow ODE with a learned approxination to the score function. However, they must interleave predictor steps of the reverse ODE with corrector steps to smooth the reverse process, meaning that their sampling procedure is also non-deterministic.
In contrast, we provide bounds on the approximation error of a fully deterministic flow matching sampling scheme, and show how those bounds behave for typical data distributions (for example, those supported on manifold).
2

---

Published in Transactions on Machine Learning Research (02/2024)
2 Background on flow matching methods
We give brief overview of flow matching, as introduced by Lipman et al. (002))
: Liu ct al. (022) Albergo &
Vanden-Eijnden (2023) ;: Heitz et al. (2023). Given two probability distributions To, on RR, flow matching
is a method for learning a deterministic coupling between To and TT. It works by finding a dime-dependent vector field v Rd x [0, - IRd such that if 2 (E(o))) is a solution to the ODE
dzp dt
o(Z,00,
2 =x
(1)
for each x E Rd and if we define Z = (Z))ee by taking x ~ To and setting Z, = =F for all t E [0, 1J, then Z1 Ti. When Z solves (1) for given function v, we say that Z is a flow with velocity field v. If we have such velocity field, then (Zo. Z1) is a coupling of (TT, m1). If we can sample efficiently from To then generate approximate samples from the coupling by sampling Zo To and numerically integrating ( This setup can be seen as instance of the continuous normalizing flow framework (Chen et al. 2018
can
In order to find such a vector field v, flow matching starts by specifying a path (xx0, x1, t) between every two points Xy and X1 in Rd. We do this via an interpolant function I R2 x IR x [0.1] + Rd which satisfies I(xo, x1, 0) = Xo and [(x0; X1, 1) = X1. In this work, we will restrict ourselves to the case of spatially linear interpolants, where [(x0, x1, = 0x + 3ix1 for functions a, [0, 1] + R such that ap = 31 and 01 3 = 0, but more general choices of interpolating paths are possible.
We then define the stochastic. interpolant between To and Tı to be the stochastic process X = (Xt) formed by sampling Xa ~ no: X1 ~ Ti and Z ~ (0, La) independently and setting X, = (Xo, tx + + for each / E 10 where 7 : [0, 1] - (0) is function such that 70 the amount of Gaussian smoothing applied at time t. The motivation for including % Z is to smooth the interpolant marginals, as was originally explained by Albergo et al. (2023). Liu et al. (2023) and Albergo & Vanden- Eijnden (2023) omit Th leading to deterministic paths between a given Xo and X1, while Albergo et al. 2023) and Lipman et al. (2023) work in the more general case.
= 0 and which determines
The process X is constructed so that its marginals deform smoothly from to Th However, X is not a suitable choice of flow between and Ti since it is not causal - it requires knowledge of X1 to construct X,. So, we seek a causal flow with the same marginals. The key insight is to define the expected velocity of X by
(2) where X2 is the time derivative of Xx, and setting v. (x( = for x & supp(X.) for each t E [0, 1]. Then, the following result shows that v. (x, t) generates a deterministic flow Z between To and Ti with the same marginals as X (Liu et al. 2023).
X₂ =x] for all x E supp(X.), x- (x,t)=
Proposition 1. Suppose that X is path-wise continuously differentiable, the expected velocity field xx (x, I) ezists and is locally bounded, and there exists unique solution Z to 4 with velocity field x for each x E Rdd. Tf Z is the flow with velocity field xx (x,t) starting in no, then Law = Law(Z,) for all t E 1].
Sufficient conditions for X to be path-wise continuously diffcrentiable and for x xxx f) to exist and be locally bounded are that 3, and y are continuously differentiable and that To, have bounded support. We will assmme that these conditions hold from now on. We also assume that all our data distributions are absolutely continuous with respect to the Lebesgue measure.
We can learn an approximation to x by minimising the objective function
C(v) S/'' E [w] [[((,,)) - xx vx (Xx,t)]² dt,
over all E V where V is some class of functions, for some weighting function we : [0, 10 + o0). (Typically, we will take w. = 1 for all 1.) As written, this objective is intractable, but we can show that
((v) f E we ((Xx, - xx11"] dt constant,
(3)
3

---

Published in Transactions on Machine Learning Research (02/2024)
where the constant is independent of v (Limman et al. 2023 This last integral can be empirically estimated in an unbiased fashion given access to samples Xo no, X1 ~ лı and the functions 0tt, 8t, Tt. and wt- In practice, we often take V to be a class of functions parameterised by a neural network e, t) and minimise ((v)) over the parameters 9 using (3) and stochastic gradient descent. Our hope is that if our approximation "y is sufficiently close to vxx and is the flow with velocity field ri; then if we take Yo ~ mo the distribution of Y1 is approximately x1-
Most frequently, flow matching is used as a generative modelling procedure. In order to model distribution from which we have access to samples, we set π π and take тo to be some reference distribution from which it is casy to sample, such as a standard Gaussian. Then, we use the flow matching procodure to learn an approximate flow va(x, 1) between To and TT. We generate approximate samples from тı by sampling Zo ~ mo and integrating the flow equation [1] to find Z1, which should be approximately distributed according to TT.
3
Main results
We now present the main results of the paper. First, we show under three assumptions listed below that we can control the approximation error of the flow matching procedure under the 2-Wasserstein distance in terms of the quality of our approximation va. We obtain bounds that depend crucially on the spatial Lipschitz constant of the approximate flow. Second, we show how to control this Lipschitz constant for the true flow xx under a smoothness assumption on the data distributions, which is explained in Section (32) Combined with the first result, this will imply that for sufficiently regular To, Ti there is a choice V which contains the true flow such that, we optimise L(v) over all v E V, then we can bound the error of the flow matching procedure. Additionally, our bound will be polynomial in the L approximation error. The results of this section will be proved under the following three assumptions.
Assumption 1 (Bound on I,2 approximation error). The true and approzimate drifts vx (x, and vo t) satisfy So E [Ilvo(X,,,t) x' (Xe, t)l?]] dt <
Assumption (Existence and uniqueness of smooth flows). For each E Rd and s E [0. there cxist unique flows (Y%5)ee(s..)) and (Z% (Z starting in Y2,, x and 7 x with velocity fields vo(x, t) and x (x,1) 1) respectively. Moreover, Y.,, and Ze, are continuously disserentiable in x, 8 and L. Assumption 3 (Regularity of approximate velocity field). The opprozimate flow vo(x,t) is differentiable in both inpuls. Also, for each L E 10 there is a conslant L, such that vo(x, 1) is L, L--Lisschila in x.
Assumption is the natural assumption on the training error given we are learning with the L22 training loss in B3. Assumption 2 is required since to perform flow matching we need to be able to solve the ODE ① Without a smoothness assumption on ox(x, t), would be possible for the marginals Y, of the solution initialised in Yo ~ To quickly concentrate on subsets of Rd of arbitrarily small measure under the distribution of X,, Then, there would be choices of ug(x, t) which were very different from v (x, on these sets of small measure while being equal to x.X (x, everywhere else. For these vx(x, t) the L approximation error can be kept arbitrarily small while the error of the flow matching procedure is made large. We therefore
require some hnoot assumption on vo(x, L) we choose to make Assumption 3 since we can show that
it holds for the true velocity field, as we do in Section 3.2
3.1 Controlling the Wasserstein difference between flows
Our first main result is the following, which bounds the error of the flow matching procedure in 2- Wassersstni distance in terms of the L2 approximation error and the Lipschitz constant of the approximate flow.
Thcorem 1. Suppose that no, Ti are probubility distributions on Rd. that Y is the flow starting in no with velocity field va, and कn is the law of Y,. Then, under Assumptions
1 we have
Wa(f1, m1) S { { L, )t).
We see that the error depends linearly on the L approximation error and exponentially on the integral of the Lipschitz constant of the approximate flow LT, Ostensibly, the exponential dependence fo La dt
4

---

Published in Transactions on Machine Learning Research (02/2024)
is undesirable, since bo may have large spatial Lipschitz constant. However, we will show in Section 3.2 that the true velocity field is Li-Lipechitz for a choice of Li such that So Li dt depends logarithmically on the amount of Gaussian smoothing. Thus, may optimise (3) over class of functions V which are all Li-Lipschitz, knowing that this class contains the true velocity field, and that if vo lies in this class we get bound on the approximation error which is polynomial in the L approximation error, providing we make a suitable choice of 0, 3g and "/--
We remark that our Theorem | similar content to Proposition Albergo & Vanden- (2023)
The crucial difference that we work with time-varying Lipschitz constant, which required practice since in many cases L explodes as + or 1, happens for example if mo or т do not have full support.
A key ingredient in the proof of Theorem ! will be the Alekseev-Gröbner formula, which provides way to control the difference between the solutions to two different ODEs in terms of the difference between their drifts (Gröbnr 1960 Alekseev 1961).
Proposition 2 (Alekseev Gröbuer). Let μ((x, () Rd x [] + Rd be a function which is continuous in and continuously differentiable in x. Let X Fd x [0.7]2 Ed be continuous solution of the integral equation
x J plr, dr,
and let Y: : [0, 1] + IR be another continuously differentiable process. Then
0 Yr YT= 1-" (vaXZz (utr.
dy, dt dr.
We now give the proof of Theorem
Proof of Theorem Define Y and Z as in Assumption 2 As Y E C(R x 10,1) Zö E C C' x (0, and (x,,t) E CO. (Rá x [0, 1]) for any E Rd, the Meksoev-Gröbbee formula implics that
Yu, Zt, - 1/
-/6 (VaY??) (vo(Zs, vx (Z,, 8)) ds.
(4)
We can write Y x + So vo ( r) dr, and so
VXYx 0. I+ L( Vxva (VX VxVx
sir dr.
It follows that
<1 0t
0 op
V
1l
<
V
V () V S L. V
where llo denotes the operator norm with respect to the e metric on Rd. Therefore, by Grönwall's lemma
we have Vxxx I exp { L, dr} < K, where K := exp f So Le df}. Applied to d4 at t = 1, we get
|| - 2i11 - / V | op
-oo 5)) ds K // | (1*( vo vo(Ze. .)|12 ds.
Letting x ~ To and taking expectations, we deduce
27 11] < K2E [V/ (lo) - ()) ()),,))2000 dt
E
[|
vo(2,,
k2 f E. [I,o(Xe,1)) xx (X,, 1|| de.
Since W2(A1, #1) W2(Law LY., Law(Zr)) < || 21 1/2 the result follows from our assumption on the 12 error and the definition of K.
5

---

Published in Transactions on Machine Learning Research (02/2024)
The application of the Alekscev-Gröhnn formula in (4) is not symmetric in Y and Z. Applying the formula with the roles of Y and Z reversed would mean we require a bound on V which we could bound in terms of the Lipschitz constant of the true velocity field, leading to a more natural condition. However, we would then also be required to control the velocity approximation error [Y,, - (Y2,)ll2 along the paths of Y, which we have much less control over due to the nature of the training objective.
3.2 Smoothness of the velocity fields
As remarked in Section 3.1 the bound in Theorem is exponentially dependent on the Lipschitz constant of uo. In this section, we control the corresponding Lipschitz constant of x. We prefer this to controlling the Lipschitz constant of vy directly since this is determined by our choice of V, the class of functions over which we optimise (3).
In this section, we will relax the constraints from Section 2 that =1 = 0 and og = 3 =. We do this because allowing Tt > 0 for all & E [0, I] means that we have some Gaussian smoothing at every i, which will help ensure the resulting velocily fields are sufficiently regular. This will mean that instead of learning flow between No and πι will actually be learning flow between to and 11, the distributions of Xo + ro and 31 + M2 Z respectively. However, if we centre Xo and X1 so that E [Xo] = E [X,] = 0 and let R be such that | | < RG then W2( ma)² < (1 - =0 + dhð and similarly for ,i, so it follows that by taking 70: 71 sufficiently close to 0 and oz, 31 sufficiently close to 1 we can make No. t1 very close in 2- Wasserstein distance to To, ग respectively. Note that if no is easy to sample from then Ao is also easy to sample from (we may simulate samples from Ho by drawing Xo ~ To: Z ~ NO,, Ia) independently, setting Xo Xn + 7 and noting that X, ~ Tn), so we can run the flow matching procedure starting from no-
For arbitrary choices of To and ग the expected velocity field x.X (x, f) can be very badly behaved, and so we will require an additional regularity assumption on the process X. This notion of regularity is somewhat non- standard, but is the natural one emerging from our proofs and bears some similarity to quantities controlled in stochastic localization (sce discussion below).
Definition 1. For real number X A2 1, say Rd- -vauude random variable W is X-regular if, whenever we lake T E (0, oo) and & ~ NO0,0=Ta) independently of W and set W = W + s, for all x E TRd we have
|Cove|w=x(E) S Xr?
We say that distribution Rd is X-reguiar if the associated random variable is X-regular. Assumption 4 (Regularity of data distributions). For some À 2 1, Xo is X-regular for all t E [0, 1].
To understand what it means for random variable W to be A-regular, note that before conditioning on W we have || Cov(()) = 72. We can think of conditioning on W' = x as re-weighting the distribution of & proportionally to pw (x- E), where (-( is the density function of W. So, W is A-regular if this re-weighting causes the covariance of & to increase by at most a factor of X for any choice of T.
Informally, if r is much smaller than the scale over which log (-) varies then this re-weighting should have ( [[Cov [lop and we expect this to be less than 7² once is much greater than the typical magnitude of W. Thus X v 1 should suffice for sufficiently small or large T and the condition that W is X-regular controls how much conditioning on W can change the behavior of & as we transition between these two extremes.
negligible effect, while for large we can write ||Covgiw==(( lop =
If W is lon-concave or alternatively Gaussian on a linear subspace of IRd, then we show in Appendix A.1 that W X-regular with x 1. Additionally, we also show that a mixture of Gaussians क = Σ (x,N(;;,,, Ia), where the weights μ4; satisfy Σ A4 = 1, > 0, and ||x:|| < R for i 1,..., K, is A-regular with A = 1 + (/2)) This shows that. X-regularity can hold even for highly multimodal distributions. More generally, we show in Appendix A.2 that we always have |[Cov]w==<((oop/ < O(dr)) with high probability. Then, we can interpret Definition I as insisting that this incquality always holds, where the dependence on d is incorporated into the parameter X. (We see that in the worst case X may scale lincarly with d, but we expect in practice that À will be approximately constant in many cases of interest.)
6

---

Published in Transactions on Machine Learning Research (02/2024)
Controlling covariances of random variables given noisy observations of those variables is also a problem which arises in stochastic localization (Fldan 2013) 2020). and similar bounds to the ones use here have been established in this context. For example, Alaoui & Montanari (2022) show that E [Covqi"=x<W") S L in our notation. However, the bounds from stochastic localization only hold in expectation over x distributed according to the law of W whereas we require bounds that hold pointwise, or at. least, for x distributed according to the law of Y2,,, which much harder to control.
We now aim to bound the Lipschitz constant of the true velocity field under Assumption 4 The first step is to show that xx is differentiable and to get an explicit formula for its derivative. In the following sections, we abbreviate the expectation and covariance conditioned on X =x to Ex . and Covx(.) respectively.
Lemma 1. If X is the stochastic interpolant between To and T1, then vx (x, t) is differentiable with respect to x and
2t (X,x(Xx, 7tt
Vx
(x, 1)
7t
The proof of Lemma I is given in Appendix B.1 Using Lemma I we can derive the following theorem, which provides bound on the time integral of the Lipschitz constant of vp-
Theorem 2. If X is the stochastic interpolant between two distributions To and σi on Fd with bounded support, then under Assumption A for each t 10, there is constant L, such that v (x, t) is Li-Lipschit: in and
S. L dt S
(L( Hill dt [ra To
+ +2/22 +
(f. [[dall dt
f."
Bel at)
Ytt
7t
where supp To, supp nT S BO, R), the closed ball of radius R. around the origin.
Proof. Using Lemma 1 plus the fact that X, = āo + 3xX1 + 222 we can write
te La -oorxx(222)) %t
ai Covx 2)
Br
V Vxu (x,
Covx Z).
7t
7t
Since we can express X, = Xo + ()x1)) + 7 where ao + B,X1 is A-regular by Assumption and 7 ~ N(0 7 Ta), we have ||Covx((): (7:2 2lop I Xx? It follows that
la - Covx (Z, o)p S max (1, (1, 2) S A.
Also, using Cauchy Schwarz we have
||Covx llop S ||Covx(XXo)ll?? [[Covx 1/2R,
and similar result holds for X1 in place of Xo.
Putting this together, we get
Fiel
|àel
|Covx (Covx(Xo:2)1.) 1lp + | |Covx(Xi, lloo
(x, llop
<
1| - -Coxx(2,
1)pp
+
Yt
7
7
Hivl
Idkal [Bl
< X
11/2R
(5)
Yt
7
%
Finally, since x (x, f) is differentiable with uniformly bounded derivative, it follows that v (xx 1) is L:- Lipschitz with L; = supxer"
V/ (x, llop' Integrating from t = 0 to t= 1, the result follows.
We now provide some intuition for the bound in Theorem 2 The key term on the RHS is the first one, which we typically expect to be on the order of log(7max /mmi) 7mmn) where Ymax and Ymin are the maximum and minimum values taken by "r the interval [0, 1] respectively. This is suggested by the following lemma.
Lemma 2. Suppose that 70 71 Tmin and Tt increases smoothly from Yiin at 0 to "max before
decreasing back to 7min at t 1. Then fo (Hil//re) dt. = 2 log(Timax/7an //nin)-

---

Published in Transactions on Machine Learning Research (02/2024)
Proof. Note that write hvl/% [d(log nd/dtl, so So (H4l//m) di is simply the total variation of log " over the interval [0, 1]. By the assumptions on 7t we see this total variation is equal to 2 2ogg(7max///mn)) ]
The first term on the RHS in Theorem 2 also explains the need to relax the boundary conditions, since To = "/ would cause So (19)l/7)) at to diverge.
We can ensure the second terms in Theorem 2 are relatively small through a suitable choice of a,, S and To. Since a and fe are contimuuusly differentiable, Jåel and [år are bounded, so to control these terms we should pick % such that R (1/7) (1/7)) dt is small, while respecting the fact that we need 7/1 < 1 the boundaries. One sensible choice among many would be 7 = 2R where So (1))//%) dt llg((/v)) and So (1/1a) dt 2m. In this case, the bound of Theorem 2 implies So Li dt & (log +2 ((ogl so if < 1 the dominant term in our bound is Alog(1/6).
(6 + t) + 5 -- t) for some 6 < 1,
3.3 Bound the Wasserstein error of flow matching
Now, we demonstrate how to combine the results of the previous two sections to get a bound on the error of the flow matching procedure that applies in settings of practical interest. In Section 3.2 we saw that we typically have Jo L; de ~ (log(7max/7mmn) mmin). The combination of this logarithm and the exponential in Theorem I should give us bounds on the 2- Wasserstein error which are polynomial in E and Tmax Ymin-
The main idca is that in order to ensure that we may apply Theorem 1 we should optimise L(v) over a class of Tunctions V which all satisly Assumption 3 (Technically, we only need Assumption 3 to hold at the minimum of L(v), but since is not possible to know what this minimum will be ex ante, is easier practice enforce that Assumption 3 holds for all v E V.) Given distributions no, Ti satisfying Assumption i as well as specific choices of au, 3t and % define K, X(Hal//1) + X R[ ([áxl/au) /a) + (1811/B)) and let V be the set of functions IRd x [0, + Rd which are K,-Lipschitz in x for all t E [0.
Theorem 3. Suppose that Assumptions 4 hold and that n is concave [0, 1]. Then xx E V and for any Uo E V, if Y is flow starting in to with velocity field vo and ân is the law of Y2,
2A
W2(f1, S CAL
Tmax
Timin
where "imin = infreeo,.1 Ne 7max = supre/o.1] Yo and C' = exp {R 4 (léul/rt) dt + So (1361/7)) dt
Proof. First, note that x is KK-Lipschitz in x for all t E [0, 1] by the proof of Theorem
2 so x E V. Then, Thus, Jo K, dt S 2 2Xlog(7mmx /mmi))
since 7tt is concave, So (1t))/))) dt S 2 log((mmax/ (min) by Lemma 2
1/2 log C. Hence for any us E V we may apply Theorem to get
W2 #I) S eexp { K, de} s C
Ymax 2A
Tmin
where To, #n are replacing no, कi since we have relaxed the boundary conditions.
Theorem 3 shows that if we optimise L(v) over the class V, we can bound the resulting distance between the flow matching paths. We typically choose tt scale with R, so C should be thought of as a constant that is @(1) and depends only on the smoothness of the interpolating paths. For a given distribution, À is fixed and our bound becomes polynomial in and 7max/7Tmin-
Finally, our choice of Tmin controls how close the distributions No. Hn are to no, Tn respectively. We can combine this with our bound on W2(11, #1) to get the following result.
hold, that "I is concave on [0, 1], and that ao = 31 = and No = 7 = 7min- Then, for any vo E V, if a flow starling in To with velocity field vo and ân is the low of Yi,
Theorem 4. Suppose that Assumptions
2A
(1) & 1/
Ymax Vmin
+ Vd
8

---

Published in Transactions on Machine Learning Research (02/2024)
where Yuminy Tmax and C are as before.
Proof. We have (2 < (2(41, (+ II). The first term is controlled by Theorem 3 while for the second term we have 1 m s (1 -1-11- 101--p10 ) + dr noted previously, Using 31 and combining
these two results completes the proof.
Optimizing the expression in Theorem 4 over Timnn we see that for a given 1 error tolerance e, we should take "min ~ d 2))1142 We deduce the following corollary. Corollary
the setting of Theorem 4 if we take Tmin d=I1/uu+2) then the total Wasserstein error of the flow matching procedure of order W2(A1, m1) S @21(4A+2))1/(22411)
4 Application to probability flow ODEs
For generative modelling applications, कi is the target distribution and mo is typically chosen to be a simple reference distribution. A common practical choice for To is standard Gaussian distribution, and in this setting the flow matching framework reduces to the probability flow ODE (PF-ODE) framework for diffusion models (Song et al.) 2021b). (Teehnically, this corresponds to running the PF-ODE framework for infinite time. Finite time versions of the PF-ODE framework can be recovered by taking So to be positive but small.) Previously, this correspondence has been presented by taking % = for all t E [0, 1] and To = No Ia) in our notation from Section 2 (Liu et al. 2023). Instead, we choose to set a, = for all
[0, 1] and have T > 0, which recovers exactly the same framework, just with Z playing the role of the reference random variable rather than Xg. We do this because it allows us to apply the results of Section 3 directly.
Because we have this alternative representation, we can strengthen our results in the PF-ODE sctting. First, note that Assumption simplifies, so that we only need to assume that mi is X-regular. We thus replace |
Assumption | with Assumption 4 for the rest of this section.
Assumption 4' (Regularity data distribution, Gaussian case). For some 2 1, the distribution nı is X-regular.
We also get the following alternative form of Lemma 1 in this setting, which is proved in Appendix B.2 Lemma 3. If X is the stochastic interpolant in the PF-ODE setting above, then xx (x,t) is differentiable with respect to x and
= 4t. "1 Ia
(+
3e Cov
Vxx
Yt
B.
Using Lemma 3 can follow similar argument to the proof of Theorem to get additional control on the terms Li
Theorem 5. X is the slochastic interpolant in the PF-ODE selling above, then under Assumplion 4 for each t E 10, there is a constant Li such that x (x,t) is Li Lipschitt in x and
S Li dt (/o Hiel dt )
1/ min { 1del Be
l 7o
11/2
R
dt.
Yt
Proof. From Theorem we know that x (xx 1) is differentiable and Lipschitz with some constant Li. From in the proof of Theorem 2 for any Rd we have
hidl Yt
|
V (x, (1 S
A 1/22 R
(6)
7
since á; = 0 in this setting. Also, using Lemma
1) < Fir ||4 (ovx(Z)llap)) +
|3el
_ |i
13c
V (x,
(Covx(Z)ll
+
(7)
3r
Yt
3e
As L; supxerd Vxu op combining Gl. 77 and integrating from to t = 1 gives the result.

---

Published in Transactions on Machine Learning Research (02/2024)
To interpret Theorem 5 recall that the boundary conditions are operating under are "/o 31 1, 80 = 0 and "n < 1, so that to = (0, Ta) and 51 is plus a small amount of Gaussian noise at scale n The following corollary gives the implications of Theorem 5 in the standard variance-preserving (VP) and
variance-exploding (VE) PF-ODE settings of Song et al. (2021b).
Corollary 2. Suppose that we are in the selling of Theorem 5
that xx (x,1) is Lippchiil in x. Then,
(i) if y = cos(( - 5)1) and B. = =in(( - 5)() for ā < 1, corresponding to the VP ODE framework of Song (2021b), then So L di S A(1 + log(1/7));
(ti) if 3r for all E [0, [] and nt is decreasing, corresponding to the VE ODE framework of Song el al. 2021b), then So Li di S \log(1/n))
Proof. In both cases, we apply Theorem 5 to bound Jo Li As 7/ is decreasing, we have So hil/7 dt log(no/m), similarly to in the proof of Lemma 2 For (i), the second term on the RHS of Theorem 5 can be bounded above by A, while for (ii) it vanishes entirely.
In each case, can plug the resulting bound into Theorem to get version of Theorem for the PF-ODE setting with the given noising schedule. We define K, = a() + min (())).)))).. X1/2 R(A. /(1)) and let V be the set of functions v: I x [0, +P Rd which are Kj-Lipschitz in x for all t E [o. 1] as before.
Theorem 6. Suppose that Assumptions 3 and hold and we are in either (i) the VP ODE (ti) the VE ODE setting above. Then xx E V and for any do V, if Y is a flow starting in Āo with velocity field and 1 is the law of Y1, then
(i) in the VP ODE setting, we have W2 #1) s <(e/n)>;
(ti) in the VE ODE setting, we have W2 #1) S =(1/n)*
Proof. That x E V follows analogously to the proof for Theorem 3 The results then follow from combining
Theorem with the bounds in Corollary 2 as in the proof of Theorem 3
]
5 Conclusion
We have provided the first bounds on the error of the general flow matching procodure that apply in the case of completely deterministic sampling. Under the smoothness criterion Assumption f on the data distributions, we have derived bounds which for a given sufficiently smooth choice of at, 3r, Tt. and fixed data distribution are polynomial in the level of Gaussian smoothing 7max//Tmin and the L error of our velocity approximation.
However, our bounds still depend exponentially on the parameter À from Assumption 4 Tt is therefore a key question how X behaves for typical distributions or as we scale up the dimension. In particular, the informal argument we provide in Section | suggests that À may scale linearly with d. However, we expect that in many practical cases should be (1) even as d scales. Furthermore, even if this not the case, in the proof of Theorem [ we only require control of the norm of Vx t) when applied to VxYX, In cases such as these, the operator norm bound is typically loose unless VxYX, is highly correlated with the largest eigenvectors of Vxrg(x,t) t). We see no priori reason why these two should be highly correlated in practico, and so we expect in most practical applications to get behaviour which is much better than lincar in d. Quantifying this behaviour remains a question of interest.
Finally, the fact that we get weaker results in the fully deterministic setting than Albergo et al. 2023 do when adding a small amount of Gaussian noise in the reverse sampling procedure suggests that some level of Gaussian smoothing is helpful for sampling and leads to the suppression of the exploding divergences of paths.
10

---

Published in Transactions on Machine Learning Research (02/2024)
Acknowledgements
We thank Michael Albergo, Nicholas Boffi and Eric Vanden-Eijnden for their comments carly version of this paper. Joe Benton was supported by the EPSRC through the StatML CDT (EP (EP/S02315/1)) Arnaud Doucet acknowledges support from EPSRC grants EP/R034710/1 and EP/R018561/1.
References
Ahmed El Alaoui and Andrea Montanari. An CitoommaiinnThhooreee View of Stochastic Localization. JEEE Transactions on Information Theory, 68(11):7423- 7442)746622 2022.
Michael S Albergo and Eric Vanden-Eiinden. Building Normalizing Flows with Stochastic Interpolants. In International Conference on Learning Representations, 2023.
Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Lijnden. Stochastic Interpolants: A Unifying Frame- work for Flows and Diffusions. orXiv preprinl or XrXh::2303.08797, 2023.
Vladimir M Alekseev. An estimate for the perturbations of the solutions of ordinary differential equations. Vestn. Mosk. Univ. Ser. I. Math. Mekh, 2:28-30 36, 1961.
Heli Ben-Hamu, Samuel Cohen, Joey Bose, Brandon Amos, Aditya Grover, Maximilian Nickel, Ricky T Q Chen, and Yaron Lipman. Matching Normalizing Flows and Probability Paths on Manifolds. In Interna- tional Conference on Machine Learning, 2022.
Joe Benton, Valentin De Bortoli, Arnaud Doucet, and George Deligiannidis. Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localization. arXiv preprint arXiv:2308 036666, 2023.
Adam Block, Youssef Mroueh, and Alexander Rakhlin. Generative Modeling with Denoising Auto-Encoders and Langevin Sampling, arXiv preprint axXi:2200.000100 001007, 2022.
Herm Jan Brascamp and Elliott H Lieb. On extensions of the Brunn- Minkowski and Prékopa-Leindler thcorems, including incqualities for log concave functions, and with an application to the diffusion equation. Journal of Functional Analysis, 22(4):366-38) 389, 1976.
Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved Analysis of Score-based Generative Modeling: FsienFrie Bounds under Minimal Smoothness Assumptions. In International Conference on Machine Learning, 2023a.
Ricky T Q Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Differential Equations. In Advances in Newral Information Processing Systems, 2018.
Sitan Chen, Sinho Chewi, Holden Lee, Yuanzhi Li, Jianfeng Lu, and Adil Salim. The probability flow ODE is provably fast. arXiv preprint Xrx::22305.117 11798, 2023b.
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is easy learning the score: theory for diffusion models with minimal data assumptions. In International Conference Learning Representations, 2023c.
Sitan Chen, Giannis Daras, and Alexandros G Dimakis. Restoration DegatiiatnnDrraatttiinnn Beyond Linear Diffusions:
A -on--sspptoticc Analysis For DDIM-Type Samplers. In International Conference on Machine Learning, 2023d.
Giovanni Conforti, Alain Durmus, and Marta Gentiloni Silveri. Score diffuuson models without early stopping: finite Fisher information all you need. arXiv preprint arXiv:2308. 1 12240, 2023.
Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. Transactions on Machine Learning Research, 2022.
11

---

Published in Transactions on Machine Learning Research (02/2024)
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion Schrödinger Bridge with Applications to Score-Basee Based Generative Modeling. In Advances in Neural Information Processing Systems, 2021.
Prafulla Dhariwal and Alex Nichol. Diffusion Models Beat GANs on Image Synthesis. In Advances in Neural Information Processing Systems, 2021.
Ronen Eldan. Thin Shell Implies Spectral Gap Up to Polylog via a Stochastic Localization Scheme. Geometric and Functional Analysis, 23(2):532-569, 2013.
Ronen Eldan. Taming correlations through entropy-cfficient measure docompositions with applications mean-field approximation. Probabilily Theory and Related Fields, 176(3-4):737 755, 2020.
Will Grathwohl, Ricky T Q Chen, Jesse Bettencourt, 1lya Sutskever, and David Duvenaud. FFJORD: Frre-form Continuous Dynamics for Scalable Reverse Generative Models. In International Conference Learning Representations, 2019.
Wolfgang Gröbner. Die senschafien, 1960.
und ihre Anwendungen. Berlin: VEB Deutscher Verlag der Wis-
Eric Heitz, Laurent Belcour, and Thomas Chambon. Tterative a-(de) Blending: a Minimalist Deterministic Diffusion Model. In ACM SIGGRAPH Conference Prooceedings, 2023.
Jonathan Ho, Ajay Jain, and Picter Abboel. Denoising Diffusion Probabilistic Models. In Advonces Neural Information Processing Systems, 2020.
Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for score-based generative modeling with polynomial complexity. In Advances in Neural Information Processing Systems, 2022.
Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of score-based generative modeling for general data distributions. In International Conference on Algorithmic Learning Theory, 2023.
Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Towards Faster Asymssypto Convergence for Diffusion- Based Generative Models. arXiv preprint orXiv:2306.092. 2023.
Yaron Lipman, Ricky Q Chon, Heli Ben-Hamu, Maximilian Nickel, and Matt Lc. Flow matching for generative modeling. In International Conference on Learning Representations, 2023.
Xingchao Liu, Lemeng Wu, Mao Ye, and Qiang Liu. Let us Build Bridges: Understanding and Extending Diffusson Generative Models. arXio preprint arXiv:2208. 14699, 2022.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow Straight and Fast: Learning to Generate and Transfer
Data with Rectified Flow. In International Conference on Learning Representations, 2023.
Jakiw Pidstrigach. Score-Based Generative Models Detect Manifolds. In Advances in Neural Information Processing Systems, 2022.
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-TTS: A Diffusson Probabilistic Model for Text-to-Speech. In International Conference on Machine Learning, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:2204.06115,, 2022.
Danilo Jimenez Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows, In Interna-
tional Conference on Machine Learning, 2015.
Noam Rozen, Aditya Grover, Maximilian Nickel, and Yaron Lipman. Moser Flow: Divergenco- based Gener- ative Modeling Manifolds. In Advances in Neural Information Processing Systems, 2021.
12

---

Published in Transactions on Machine Learning Research (02/2024)
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, et al. Pho- torealistic Text-to-lmage Diffusion Models with Deep Language Understanding. In Advances in Neural Information Processing Systems, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In International Conference Leurning Represenlations, 2021a.
Yang Song and Stefano Ermon. Generative Modcling by Estimating Gradients of the Data Distribution. Advances in Newral Information. Processing Systems, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. International Conference on Learning Representations, 2021b.
Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Newral Computation, 23(7):1661 1674, 2011.
Kaylee Yingxi Yang and Andre Wibisono. Convergence in KL and Rényi Divergence of the Unadjusted Langevin Algorithm Using Estimated Score. In NewrIPS 2022 Workshop Score-Basee Methods, 2022.
Qinsheng Zhang and Yongxin Chen. Fast Sampling of Diffusion Models with Exponential Integrator. International Conference on Leorning Representations, 2023.
13

---

Published in Transactions on Machine Learning Research (02/2024)
A Exploration of Definition 1
A.1 Special cases of X-regularity
First, we show that all log-concave random variables are X-regular for À = 1. The key ingredient in the proof is the following result Brascamp & Lieb (9766).
Proposition 3 (Brascamp-Lic Lieb 1976). Suppose that W is an R valued random variable with density func- tion Pw (x) = -=)(x) where is strictly conver on R and twice differentiable. Assume that DPup 2 pla with μ > 0 and g E C (Rd). Then,
Varw(g(W)) S ((DP))) V((W)) < -E [[V /
Lemma Suppose that W an -vauuee log-coneave random variable. Then W is X-regular for À
Proof. Fix T and denote the density of W by pw (x) - for some convex function g. Take {~ (~N(00,²)a)) and set W W +. Then, the density of & conditional W is given by
(Elx') x e -11113/(223) (o pw - -E) = exp
Eell 2F
,(xx 13'-50
Let u E Rd be an arbitrary unit vector, and consider g(W) = u² W. Since (E) convex in E, and
|1E1 27 + pxxx -E) is strictly
D² + D2 D' 5) 2 r I
can apply Theorem to the random variable & conditional on W with 7 to get
Vargw: s F [I
Then,
(E)ll < sup
Vargw-(u' s -2
1=
Second, we show all random variables which are Gaussian on linear subspace of IRd are X-regular for À Lemma 5. Suppose that W on -alued random variable supported on some linear subspace S S Rd and that W restricted to S is Gaussian with positive definite covariance matriz. Then W is X-regular for X 1.
Proof. We decompose into two orthogonal components { and Eμi so that & { c + E,, and & is perpen- dicular to S while E₁ is parallel to S. Then, can write W (W + Enj E1. We denote W W + Ejj and note that W/ E S. From obsorving W' we can doduce the values of both W/ and {1. Therefore, (ove = Cove =E / (51),, where X|| denotes the projection of x onto S.
By restricting our attention to the subspace S, may therefore reduce to the case where W is Gaussian with full support. The result then follows from Lemma 4 since Gaussians with full support are log-concave. J
Third, we show that all random variables which are locally at least as smooth a Gaussian of covariance La and are bounded up to Gaussian tails of covariance L² are A-regular.
Lemma 6. Suppose that W is an R--vulued random variable which can be decomposed as W = UU+,, where U and are independent random variables such that ||U|| < R for some R > 0 and 1) ~ NO, oa for some + (R2/O?).
o > 0. Then W A-regular for
Proof. Fix r > 0, so that W W - + + + where 7~ ~ NN0,0201 and & NNO, and U, n, E are all independent. By the law of total variance, we have
Covelw [Coveiwrvu(]] WT + Cov(B [E] W W').
14

---

Published in Transactions on Machine Learning Research (02/2024)
The distribution of &l W. U is the same the distribution of { (17 + E): so it suffices to understand the latter. To this end, we define p 11 and w = oÂg - 781 It is straightforward to check that and are independent centered Gaussians with covariances (a² 7 and o0² + Id respectively. Then, we can write & = - o=+72 (r²p+ p+ + w), from which it follows that
Covelwr,o(E) = (,/ो)
Cov(T p
x (ス
la.
Therefore, E [Covelwr, (e) W = =(07 02,2 la. In addition, have
E W',
U1 W'
(मخनデ [T ( p:
[E W", (')
02472 ( )
x²
2
Covn w (n + E). Finally, have
||Cov,,eww (17 + E)Ilop ||Covoiw (W' -U)llop ||Covolw-(U]]on S R².
Putting this all logether, we see that
|[ove w: (4) S || [Covew''u(() ( + |[CovBB[| W",0] W')
0272
72
R2
02+72
02 +2
I X
for A x= 1+ as required.
We note in particular that Lemma i can be applied in the case where our target distribution is a bounded mixture of Gaussian components with covariance L² for some O. We obtain the following corollary. Corollary 3. Suppose that = Ei= o2), is a mirture of Gaussian components of covariance Ia for some > O, where the weighls μs sulisfy Σ Mi 1 and we have [|x < R for all i 1...., K. Then π is X-regular for == (R²/o²).
Proof. This follows immediately from the fact that can be represented in the form required to apply Lemma by taking U to have distribution Σ px0x.-
A.2 High-probability bounds
Lemma 7. Suppose that W is an Ralued random variable. For any 7 >> 0, if we take & &~ NN0, T1a) and |[Covg S 2dc
set W' = W + E, then we have
with probability at least 1 - 6de -2/2 for any 2 1.
Proof. First, have
[S < sup (argw:((u < sup Eew [(uTe) < E [[ISI] W'. [|u]
||
Next, we bound the quantity E [ W'] using Markov's incquality. If e² > 222 for some i between and d. Therefore,
||<l| > de? then we must have
E [1111211 de < E
[K
{ d
d
> E ? [g>ar
i=2
dd
[G192221)
d(dd 2p (5t > 277)
15

---

Published in Transactions on Machine Learning Research (02/2024)
A standard Chernoff bound gives P
( > e²T2) < 2e
and
x²
E
23
=
3/(22) )
dz
V 1
2T
2²
3 de
2
27
2
2//2
00
27
000 1
=2
dz
2
22T
c
2T C
22
+ 2T
(51 > CT)
22T
I 2T 2 -212
c +11
2xx
I 4 121
Hence
E
[11112 1813
S 2 22-
It follows by Markov's inequality that
EI'
|K 112
|| |113=00022
P E [|510 de
W
2 dc²72)
s
d
I 6de
-2112
Finally, can write
E [[[C| W'] <
W' + d
and (E)l S E [| W < 2d with probability at least 1 6de- as required.
B Derivation of formulae for gradients of velocity fields
B.1 Proof of Lemma 1
In order to prove Lomma we use the following intermodiate result. Lemma 8. If X is the stochastic interpolant between To and IT, then
Vx E X₂ = x]
Covx Z).
7t
Moreover, a similar expression holds for X, in place of Xo-
Proof. First, note that X,, X1 ~ N(ax + B,X,, 12 Ia), so
1
Vx 1o (x|xo, x:))
(x
04:0 8
Therefore,
Vx ( PX.IX (x|Xo. x x1) (Xo, x1) dxodxx)
Vx log(pxx =
px, px,(x) (x)
PXelXo (x Xo, Xi XXXx (Xo, x1 Vx Vx log(Px, Xo.X, (x/x0,xx) dxodx
Px, (x)
J 32d
1
IF.
[[x,
Xx BI Xx xx]
7
16

---

Published in Transactions on Machine Learning Research (02/2024)
We can then calculate
(x) y / Xo Px. (x|x0: xi 1xxx, (xo, xi) dxndxi)
v 8 X - x = Vx Vx
J
(xo x1 Vx (log (x|xo: x1)) dxodx1 px, (x) JRS JRd xo P.eeXooxx, ( x I
(Vxlogpx,xx)) (x) (px)(x))er [ xu Px.IXnxX (x|x0: x1 1)xxx.XX (xo. x1) dxodxx)
T Ex [Xo(X; ao 3rx,))]] +
1
- 1xx [(x, axXo 3xX Bxx
1
78
Covx (Xo, 2).
Lemma 1. If X is the stochastic interpolant between To and mı, then vx (x, t) is differentiable with respect
and
Vx (x, = "t la
Covx(Xxx))
7/ 7t
Proof. Since X, = a, + 8,X, + 7 and X, = 0o + +, + 7 we can write
it (x a:80 B2x1)
E [ X1 x, Xo = xo, X1 =xi] = -,,0 + Bix1 +
7t.
and therefore
E [ Taking gradients with respect to and applying Lemma
_x
it
(dr7 - ihar)
B B[xo Xx x] + (Br -inBB)
B -[[x Xx x].
7tt
7t
7t
111
(de )na)) Covx CooxxXo, 2)
(3r7 - (188)
Vx
(x
(x,
la
Covxx(XI 2)
7t
2
7/
21
tt
Covxl (x. -7:2,)) 1
la
+o +,X,,
7tt
7
la
Covx (X,,2) Z).
7t
7t
B.2 Proof of Lemma 3
Lemma 3. If X is the stochastic interpolant in the PF-ODE selting above, then x (x, 1) is differentiable with respect to x and
VxU (x, t) = 31 Ia-
21
1/4
3.
Covx(Z).
7t
Br
Proof. From Lemma have
V Vx (x,
21 la
1
Cov (Xi, 2).
7tt
7t
Then, since = 0 we have X, = -,x + 7 and X, = B.x1 + +-rZ, so we can write
Vxu (x,1)
20 Ia
1
Covx
(Bc +)
7t
Br
tt Ia
1tt
3
Covx(Z).
3r
noting that may discard the X, term since conditioning X,
17


Reviews:
[{"review_id": "u5SDrGMaAK", "reviewer": "Reviewer_28fW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- There is significant interest, e.g. among people in the target audience for TMLR, in developing rigorous performance guarantees for diffusion models and related generative modeling frameworks. The present submission provides fairly clean and interpretable bounds for a broad class of generative models based on \"flow-matching\"\n- The integrated Lipschitz constant is a natural quantity that has also arisen previously in discretization analyses for diffusion models, e.g. the ICML '23 paper of Chen-Daras-Dimakis on DDIMs, so it is nice that this paper initiates a principled study of this particular quantity and identifies some simple conditions under which it scales benignly.\n\n**Weaknesses**: \n- One somewhat unsatisfying aspect of the results is that it is unclear to what extent any of these bounds are tight. For instance, while the dependence on $d$ in the $W_2$ bound is polynomial, is the $\\epsilon^{\\Theta(1/\\lambda)}$ dependence at the end of Section 3 necessary or just an artifact of the proof? The Conclusion touches upon the question of characterizing $\\lambda$ for typical distributions and whether or not we should expect it to scale with the dimension, but there is the separate question of whether the exponential dependence on $1/\\lambda$ is \"real\" to begin with.\n- Relatedly, the paper only establishes regularity for fairly nice distributions like log-concave measures. One of the primary benefits of methods like sampling using the probability flow ODE is that one can hope to sample even from highly multi-modal distributions. How badly can regularity fail for such distributions? E.g. it would be nice to include bounds on $\\lambda$ when the distribution is a mixture of separated Gaussians.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers flow matching methods for sampling. Formally, to sample from a target distribution $\\pi_1$, one can try to learn a deterministic flow from a distribution $\\pi_0$, which is easy to sample from, to $\\pi_1$. This flow can be specified by giving a path (together with a time-dependent speed) between any pair of points in space and, optionally, a time-dependent rate of injecting Gaussian noise. That is, to sample from the marginal distribution of the flow at time $t$, one samples $x_0, x_1$ from $\\pi_0,\\pi_1$ and forms $I_t(x_0,x_1) + \\gamma_t \\cdot g$ for $g\\sim\\mathcal{N}(0,I)$. This work considers linear paths, for which it is known that such flows can also be realized by ODEs, of which one example is the probability flow ODE from diffusion generative modeling.\n\nThe paper begins by giving a bounds on how robust the distribution at the end of the flow is to perturbations to the velocity field of the ODE. Under the assumption that the average squared perturbation along the path is bounded by $\\epsilon^2$ and that the velocity fields are $L_t$-Lipschitz in space at time $t$, they show that the distribution changes in $W_2$ by at most $\\epsilon\\cdot \\mathrm{exp}(\\int^1_0 L_t \\, \\mathrm{d}t)$. This follows by a standard application of the Alekseev-Grobner formula.\n\nAt first blush, the exponential dependence on the integrated Lipschitzness might appear rather prohibitive; indeed, as the authors note, in the worst case the integrated Lipschitzness could be as large as the ambient dimension. The bulk of the paper is thus focused on showing that if the marginals of the process without Gaussian smoothing are sufficiently \"regular\", then the smoothed process' integrated Lipschitzness is actually not too large. For instance, if $\\gamma_t$ is concave and the flow is \"regular\" in the sense that the marginal covariances are not too affected by Gaussian tilts , then $\\int^1_0 L_t \\,\\mathrm{d}t$ can be bounded by $O(\\log(\\gamma_{\\max}/\\gamma_{\\min}))$, where $\\gamma_\\max = \\max_t \\gamma_t$ and $\\gamma_\\min = \\min_t \\gamma_t$. The authors show that regularity holds under log-concavity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I'm in favor of acceptance as I think it's a clean and self-contained result that should be of decent interest to TMLR's audiences. That said, implementing the following changes would strengthen the work in my view:\n\n- Some of the technical discussion could benefit from being converted into formal lemmas. For instance, the $\\log(\\gamma_\\max/\\gamma_\\min)$ dependence discussed at the end of P. 7 is invoked in the proof of Theorem 3 and should probably be moved into a standalone lemma statement.\n- There is some discussion about how Appendix A.2 suggests regularity can scale with the dimension, but my understanding is that all that is done in Appendix A.2 is to show a fairly loose upper bound in terms of $d$. It would be more satisfying if the authors could provide a simple example of a distribution for which there is a matching lower bound\n- The terminology \"regularity\" is a little overloaded, so it would be helpful for the authors to point out any prior works where the notion in Definition 1 is used. If not, perhaps they should note that this notion of regularity is non-standard and is the natural one that emerges from their proofs?\n\n**Minor typos:**\n- In Assumption 2, is Z^x_{s,s} supposed to start at x as well?\n- Middle of P. 4: \"smoothness assumption on v_{theta}(x,t) choose to make\"\n- Proposition 2: in the last equation in Proposition 2, should s in Y_s in the gradient be r?", "broader_impact_concerns": "There are no adverse ethical implications of this work. Of course, with generative modeling there are the usual concerns about privacy, amplification of biases, etc., but this is well out of the scope of the submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pmK5DFTVCU", "reviewer": "Reviewer_juW2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "My concern (and question to the authors) to help me assess the final decision about this paper is about the main result of the paper in Theorem 3. The bound depends on $\\gamma_{max}/\\gamma_{min}$ and $\\gamma_t\\in[0,\\infty)$. This ratio can be very large (even infinity if $\\gamma_{min}$ is 0). I am especially concerned with $\\gamma_{min}$ being zero or close to zero. The authors relax the problem to avoid $\\gamma_{min}$ being zero for $\\gamma_0$ or $\\gamma_1$ in Section 3.2 (second paragraph), but this does not ensure that $\\gamma_t$ can be very small or close to zero in the path from $t=0$ to $t=1$. Also, we need $\\gamma_0$ to be small to have a good approximation between $\\pi_0$ and $\\tilde{\\pi}_0$ (same of $\\gamma_1$ and $\\pi_1$), but at the same time, the better this approximation is the larger the bound in Theorem 3 is. It is unclear how this trade-off can be understood in the paper. In a way, we have an uncertainty principle at play here. If we make $\\tilde{\\pi}_1$ be close to $\\pi_1$ the bound on the error will be very large. We can only have a small bound if $\\tilde{\\pi}_1$ is not that close to $\\pi_1$, but in that case, are we drawing samples from $\\pi_1$? It will be good to understand this trade-off in the paper.\n\nSecond question, if we relax $\\gamma_0$ and $\\gamma_1$ away from zero and $\\alpha_0$ and $\\beta_1$ away from one, then $\\tilde{\\pi}_1$ would have full support because $\\pi_0$ has full support, which would void one of the main claims of the paper. Is this the case, and if not why?\n\nMinor comments:\n\nIn the second paragraph on Page 3, the authors say that the interpolation function is $I(x_0,x_1,t) = \\alpha_t x_0+\\beta_t x_1$ isn’t this linear interpolation too limited? Please comment.\n\nIn the second paragraph in Section 3.2 (page 6), the authors say that if it is simple to sample from $\\pi_0$ then it is easy to sample from $\\tilde{\\pi}_0$. This might be trivial, but it is not to me. Can you tell me why is this the case?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose an error bound for flow matching bounds for determinist matching when the data distribution does not have full support. Real data in high dimensions does not have full support, so being able to prove these results is important to advance the theory behind flow models. From this perspective, the paper presents a significant contribution.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The main changes are about a discussion on Theorem 3 limitations when we add the relaxations in Section 3.2", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YsrluTqkQR", "reviewer": "Reviewer_W9Ya", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n(1)\tThe paper provides a valuable extension of bounds for flow matching, targeting the important practical setting of deterministic sampling and distributions without full support. \n(2)\tThe paper is very well written, with good organization, and provides a clear exposition of the main results. \n(3)\tThe assumptions are discussed and, for the most part, they are justified and the motivation for adopting them is clearly explained.\nWeaknesses\n(1)\tDespite a good effort to make the assumptions reasonable and not particularly onerous, the paper doesn’t provide a clear example of how a practitioner might use the results. It would be helpful to understand, what steps a when presented with a dataset and making simple, reasonable assumptions about the approximation error and nature of the distribution of the data, to construct a bound. Perhaps the intention is to provide x\t\n(2)\tAs the authors specify in the conclusion, the bounds depend exponentially on a smoothness parameter and it is not obvious from the provided results how this parameter scales as the dimension increases or how it behaves for commonly encountered data distributions. \n\nOverall, this is a strong paper that provides a useful extension of existing bounds for flow matching. There are practical advantages to deterministic sampling cases and many distributions of interest do not have full support. \n\nWhile I think the paper is publishable in its current form, since it contributes to the theoretical understanding of flow matching in a valuable way, I think it could benefit from a little extra effort to make it clearer how a practitioner might make use of these results (or whether they even can). It would be good to be identify which assumptions are unverifiable and which components of the bounds are unquantifiable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents bounds on the error of flow matching for the case of fully deterministic sampling when the data distribution does not have full support. The first result in the paper establishes a bound on the error of the flow matching approximation under the 2-Wasserstein metric. The bound is expressed in terms of the L^2 training error and the Lipschitz constant of the approximate velocity field. The second main result first demonstrates that, provided a smoothness assumption is satisfied, the true velocity field is Lipschitz, and subsequently bounds the Lipschitz constant. The two results are then combined to construct a bound on the approximation error which depends polynomially on the L^2 training error.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The following changes are not critical, but additional discussion would strengthen the paper and make it more readily accessible and usable by practitioners of flow matching who would like to tailor results to their particular problem setting.\n\n(1) The key challenges in applying Theorem 3 appear to be identification of R, \\lambda, and \\epsilon. As a practitioner, if I were presented with a dataset that I would like to model, and I wanted to make reasonable assumptions about the target distribution generating the data, and I selected a particular neural network to learn the approximate flow, could I evaluate or bound these three quantities in any way? Is it possible to tie them back (even if it makes the bounds looser) to quantities that can be evaluated in a particular problem setting under reasonable assumptions about the distribution?\n\n(2) Theorem 4 leaves work for the reader to do to plug the results back into Theorem 1 to construct a result like Theorem 3 that is more direct and easier to understand. It would be helpful to provide corollaries, even if only in the appendix, to specify the results that arise in these common settings.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers raised a few technical points and questioned some sharpness and reasonableness of assumptions (or ability to verify assumptions).  The authors' rebuttal addressed these issues, sometimes resolving them and other times merely acknowledging them as difficult issues. Overall, none of the reviewers nor myself think that these issues are significant enough to recommend against publication of the paper, and we were happy to see progress made on an exciting topic.

For more details, below is an abridged version of reviewer W9Ya's final summary:

**Strengths**
1. It provides interpretable bounds for flow matching, extending existing results to address the fully deterministic setting, allowing for distributions that do not have full support and relaxing regularity assumptions. 
2. The results should be of interest to researchers exploring diffusion models - there is a strong desire for theoretical performance guarantees for these methods. While the results provided in this paper have aspects that are a little undesirable, they do constitute a useful stepping stone in the search for a more complete characterization. 
3. The paper provides some useful discussion and results concerning regularity conditions and establishes that (mixture of Gaussian) multi-modal distributions satisfy the regularity required for the main results.

**Weaknesses mentioned by more than one reviewer (mostly addressed)**
1. The bounds are not particularly tight. In particular, there is an exponential dependence and it is not clear if this is necessary or an artifact of the proof technique. [Response: The authors conjecture that the exponential dependences is probably inevitable if the results target a broad class of distributions, and give a convincing informal argument] 

2. The bounds are unquantifiable. For a given data distribution, one cannot establish whether the assumptions hold, nor evaluate or determine suitable constants. [Response: The authors have acknowledged that this is a limitation of the work. They point out that for some classes of distribution, the paper does derive some tractable bounds. Despite the limitation, the results still offer considerable value in terms of highlighting how the performance guarantees depend on aspects of the data distribution and the flow. The demonstration that a multi-modal distribution meets the required regularity condition is a valuable step in convincing a reviewer that the imposed assumptions are not unreasonably restrictive.]
Decision: Accept (poster)