Improving and generalizing flow-based generative models with minibatch optimal transport
Authors: Alexander Tong
Kilian FATRAS
Nikolay Malkin
Guillaume Huguet
Yanlei Zhang
Jarrid Rector-Brooks
Guy Wolf
Yoshua Bengio
Venue: Accepted by TMLR ()

Abstract: Continuous normalizing flows (CNFs) are an attractive generative modeling technique, but they have been held back by limitations in their simulation-based maximum likelihood training. We introduce the generalized conditional flow matching (CFM) technique, a family of simulation-free training objectives for CNFs. CFM features a stable regression objective like that used to train the stochastic flow in diffusion models but enjoys the efficient inference of deterministic flow models. In contrast to both diffusion models and prior CNF training algorithms, CFM does not require the source distribution to be Gaussian or require evaluation of its density. A variant of our objective is optimal transport CFM (OT-CFM), which creates simpler flows that are more stable to train and lead to faster inference, as evaluated in our experiments. Furthermore, we show that when the true OT plan is available, our OT-CFM method approximates dynamic OT. Training CNFs with CFM improves results on a variety of conditional and unconditional generation tasks, such as inferring single cell dynamics, unsupervised image translation, and Schrödinger bridge inference. The Python code is available at https://github.com/atong01/conditional-flow-matching.

Published in Transactions on Machine Learning Research (03/2024)
Improving and Generalizing Flow-Based Generative Models with Minibatch Optimal Transport
Alexander Tong"
alezander.tong@imid quebee
Mila Québec AT Institute, Université de Montréal
Kilian Fatras"
kilian.fatrrs Cimriiaa queber
Mila Québec AT Institute, McGill University
Nikolay Malkin"
nikolay. maliii @imila. quebec
Mila
Québec AT Institute, Université Montréal
Guillaume Huguet
guillaume quebee
Mila
Québec AI Institate, Université Montréal
Yanlei Zhang
Jonlei.zhang@mila. quubee.uueala.ueeeeeeeee quebec
Mila
Québec AT Institute, Université de Montréal
Jarrid Rector-Brooks
www.bboooooooooooooo j quebec
Mila
Québec AI Institute, Université de Montréul
Guy Wolf
guy. wwy.auolf@umoontraal.co
Mila Québec AT Institute, Université de Montréal
Canada CIEAR Al Chair
Yoshua Bengio
yoshua. bengio Avreila. @vreila.qucceer
Mila
Québec AI Institate, Université Montréal
CIFAR Senior Fellow
Reviewed OpenReview: https: https:///ppereeviau openreview. net/ ttt/formm? id= CD9Smc734W
Abstract
Continuous normalizing flows (CNFs) attractive generative modeling technique, but they have been held back by limitations in their simulation-based maximum likelihood training. We introduce the generalized conditional flow matching (CFM) technique, a family of simulation-free training objectives for CNFs. CFM features a stable regression objective like that used to train the stochastic flow in diffusion models but enjoys the efficient inference of deterministic flow models. In contrast to both diffusion models and prior CNF training algorithms, CFM does not require the source distribution be Gaussian or require evaluation of its density. A variant. of our objective optimal transport CFM (OT-CFM), which creates simpler flows that are more stable to train and lead to faster inference, as evaluated in our experiments. Furthermore, we show that when the true OT plan is available, our OT-CFM method approximates dynamic OT. Training CNFs with CFM improves results on a variety of conditional and unconditional generation tasks, such as inferring single coll dynamics, unsupervised image translation, and Schrödinger bridge inference. The Python code http:aa.........ooooooaooiiii..t available at https://github. com/ attogoo/ www.lolwwttcching
Equal contribution

---

Published in Transactions on Machine Learning Research (03/2024)
1 Introduction
Generative modeling considers the problem of approximating and sampling from a probability distribution. Normalizing flows, which have emerged competitive generative modeling method, construct invertible and elficiently differentiable mapping between a fixed (e.g., standard normal) distribution and the data distribution (Rezende & Mohamed, 2015). While original normalizing flow work specified this mapping as a static composition of invertible modules, continuous normalizing flows (CNFs) express the mapping by a neural ordinary different ial equation (ODE) (Chen et al. 2018). Unfortunately, CNFs have been held back by
difficulties in training and scaling to large datasets (Chen et al. 2018 Grathwohl et al., 2019 Onken et al., 2021).
Meanwhile, diffusion models, which are the current state of the art many generative modeling tasks (Dhariwal & Nichol, 2021: Austin et al.) 2021: Corso et al.] 2023 Watson et al., 2022b). approximate a stochastic differential equation (SDE) that transforms a simple density to the data distribution. Diffusion models owe their success in part to their simple regression training objective, which does not require simulating
the SDE during training. Recently, (Lipman et al.] 2023 Albergo & Vanden-Fiindden 2023 Liu 2022 showed that CNFs could also be trainod using a regression of the ODE's drift similar to training of diffusion models, an objective called flow matching (FM). FM was shown to produce high-quality samples and stabilize CNF training, but made the assumption of a Gaussian source distribution, which was later relaxed in generalizations of FM to more general manifolds (Chen & Lipman 2024), arbitrary sources (Pooladian et al. 2023 and couplings between source and target samples that are either part of the input data or are inferred using optimal transport. The first main contribution of the present paper is to propose a unifying conditional flow matching (CFM) framework for FM models with arbitrary transport maps, generalizing existing FM and diffusion modcling approaches (Table 1
A major drawback of both CNF (ODE) and diffusion (SDF) models compared other generative models (e.g., variational autoencoders (Kingma & Welling 2014), (discrete-time) normalizing flows, and generative adversarial networks [Goodfellow et al. (0144), is that integration of the ODE or SDE requires many passes through the nelwork to generate a high-quality sample, resulting in a long inference time. This drawbad has motivated work on enforcing an optimal transport (OT) property in neural ODEs (Tog et al. 2020 Finlay et al.] 2020 Onken et al.) 2021 Liu 2022; Liu et al. 2023b), yielding straighter flows that can be integrated accurately in fewer neural network evaluations. Such regularizations have not yet been studied for the full generality of models trained with FM-like objectives, and their properties with regard to solving the dynamic optimal transport problem were not empirically evaluated. Our second main contribution is variant of CFM called optimal transporl conditional flow malching (OT-CFM) that approcimales dynamic OT via CNFss We show that OT-CFM not only improves the efficiency of training and inference, but also Icads to more accurate OT flows than cxisting neural OT models based on ODEs (Tong et al. 2020 Finlay et al. 2020), SDEs (De Bortoli et al., 2021 Vargas et al. 2021), ) or input-convex neural networks (Makkuva et al. 2020 Furthermore, an entropic variant of OT-CFM can be used to efficiently train a CNF to match the probability flow of a Schrödinger bridge. We show that in the case where the true transport plan is sampleable, our methods approximate the dynamic OT maps and Schrödinger bridge probability flows for arbitrary source and target distributions with iimulation-free training.
In summary, our contributions are:
(1) We introduce a generalized formulation of the recent conditional flow matching framework $3. 1 , and prove its correctness encompassing many existing flow matching methods (Lipman et al. 2023 A Albergo & Vanden-Lijnden, 2023 Liu et al. 2022a (See Table Dp.
(2) We consider special case of CFM that draws source and target samples according to optimal transport plan, allowing us to solve the dynamic OT and Schrödinger bridge problems in a simulation-fres way, using only static OT maps between marginal distributions. We show that efficient minibatch approximations to the OT map still yield correct solutions to the generalive modeling problem while incurring a low detriment to the dynamic OT solution (3.2p..
(3) We evaluate CFM and OT-CFM in experiments on single-cell dynamics, image generation, unsupervisod image translation, and energy-based models. We show that the OT-CFM objective leads to more efficient training and decreases inference time while finding better approximate solutions to the dynamic OT
2

---

Published in Transactions on Machine Learning Research (03/2024)
Metching (lipmen
Conditional Matching
Matchinng
A
A
M
A
M
V
A
A
AA
A
Figure Left: Conditional flows from FM (Lipman al. 2023). I-CFM $3.2.2 and OT-CFM $3.2.3 Right: Learned flows (green) from moons (blue) to 8gaussians (black) using I-CFM (centre-right) and OT-CFM (far right).
and Schrödinger bridge problems. For high-dimensional image generation, also propose improved and reproducible training practices for flow-based models that siguificantly improve the performance of algorithms from past work (5).
(4) We release a Python package, torchcfm, that unifies new and existing algorithms for training flow-based generative models under a shared interface and provides implemennatiinns our main experiments. The Python code available at.
2 Background: Optimal transport and neural ODEs
Throughout the paper, we consider the setting of a pair of data distributions over Rd with (possibly unknown) densities (00)) and ((x1) (also denoted 9o. 91). Generative modeling considers the task of fitting mapping f from Rd to Rd that transforms do to T1, that is, zo is distributed with density do then f(xp) is distributed with density Q1- This includes both the typical case when go an easily sampled density, such as a Gaussian, and the case when yo and qi are empirical data distributions available as finite sets of samples.
2.1 ODEs and probability flows
A smooth time-varying vector field u: [0, 1] x Rdd + Rd defines an ordinary differential equation:
dx ux)) dt,
(1)
where we use the notation (z) interchangeably with u(t, x). Denote by (x) the solution of the ODE [ with initial condition Φo(x)) x; that is, (x(r) is the point x transported along the vector field u from time up to time
Given density Po over Rd, the integration map oe induces pushforward ps [dr] (po), which is the density of points po transported along from time 0 to time t. The time-varying density Pt: viewed as function p: [0. 1] x Rd + R, is characterized by the well-known continuity equation:
0p - (pru.)
(2)
Ōt
and the initial conditions Po. Under these conditions, is said to be probability flow ODE for p, and p is the (marginal) probability patt generated by u.
Approximating ODEs with neural networks. Suppose the probability path (c(x) and the vector field (x) generating it are known and p. (x) can be tractably sampled. If vo(-,-) 10,1] x Rd -> Rd is a time-dependent vector field parametrized as neural network with weights 0, va can be regressed to u via
TTo be precise, to ensure the uniqueness of integral curves (and thus of the corresponding flow), we assume the vector field at least locally Lipschitz in and Bochner integrable in t. 2The terminology due ++ Pt being path the infiiiteddimensional manifold of probahility distributions on Rd.
3

---

Published in Transactions on Machine Learning Research (03/2024)
the flow matching (FM) objective:
(FM == (er. -ur (
(3)
Lipman 2023 used a version of this objective with a stochastic regression target fit ODEs that map Gaussian density Yo to target Φ1- However, this objective becomes intractable for general source and target distributions. In $3 we develop generalizations that allow more flexible and efficient generative modeling.
The case of Gaussian marginals. Consider the special case of an ODE whose marginal densities are Gaussian: pı(z) = (( He, of). While the ODE that generates these marginal densitics is not unique, one of the simplest the one that satisfies
TD
140
oz = Me +
(4)
which is unique by the following theorem.
Theorem 2.1 (Theorem of Lipman et. al. 2023 4 has the form
The unique veclor field whose inlegration mop satisfies
uc(x) of (I ux) + +16
55)
where of and 14 denote the time derivative of σt and pe, respectively, and the vector field a with initial conditions Nuoo og) generates the Gaussian probobility path pe(x) = =N 14, o?).
2.2 Static and dynamic optimal transport
The (static) optimal transport problem seeks mapping from one measure to another that minimizes a displacement cost. The case of greatest intorest is the 2- 2-Wasserstein distance between distributions qo and di on Rd with respect to the Euclidean distance cost c(x, y) = || yll- The corresponding optimization problem
is
/ c(x, y)² dz
W (go- 41)2 inf nEll R
where II denotes the set of all joint probability measures Rd x Rd whose marginals are do and gi. For compactly supported distributions and for the ground cost c(x, y) = || - yll. the set of solutions of (6) is not empty Villani (2009). and W2 is a metric on the space of probability distributions on gd with finite second moment.
The dynamic form of the Wasserstein distance is defined by an optimization problem over vector fields u that transform one measure the other:
[ (((a))lu(()))1 dt dr.
W(o: 9 91)2 3
inf pe.ue
with 2 0 and subject to the boundary conditions
9o. and the continuity equation 2 The equivalence between the dynamic and static optimal transport formulations was first proven in Benamou & Brenier (2000) under the assumptions that go and 91 are compactly supported distributions with bounded density. We refer (Ambrosio Gigli 2013 Chapter2) for recent overview on optimal transport the relation between the two formulations.
Tong et al. 20200; Finlay al. (2020) showed that CNFs with L regularization approximate dynamic optimal transport. For general marginals, however, these models required integrating over and backpropagating through tens to hundreds of function evaluations, resulting in both numerical and efficiency issues. We aim to avoid these issues by directly regressing to the vector field in a simulation-free way.
Optimal transport also related to the Schrödinger bridge (SB) problem (Lconard 2014b). We show $3.2.4 that variant of the algorithm we propose recovers the probability flow of the solution to a SB problem with Brownian motion reference process.
4

---

Published in Transactions on Machine Learning Research (03/2024)
3 Conditional flow matching: ODEs from static couplings
3.1
Vector fields generating mixtures of probability paths
Suppose that the marginal probability path pa( is a mixture of probability paths (x(2(2 that vary with some conditioning variable 2, that is,
(x / P=(x12)g(2)
(8)
where (()) is some distribution over the conditioning variable. If the probability path pa (x(2()) is generated by the vector field (x(2))) from initial conditions Po(z/z) (sec $2.1 1]. then the vector field
Ur (a) (x)l))
Ut (x) == Ey(z)
(9)
pr (x
generates the probability path pe(a),, under some mild conditions:
Theorem 3.1. The marginal vector field g
generates the probubility path
8
from initial condilions po(x)..
All proofs appear in Appendix A This result extends [ipman et al. 2023 Theorem 1) to general conditioning variables and delincates some minor conditions on (()).
A regression objective for mixtures. We are interested in the case where conditional probability paths (x]22) and vector fields (x(2)) are known and have simple form, and wish to recover the vector field (x(),, defined by (9), that generates the probability path (r)).. Exact compulation via (9) is generally
intractable, as the denominator (r()) is defined by an integral (8) that may be difficult to evaluate. Instead, we develop an unbiased stochastic objective for regression of a learned vector field to uxxx), which generalizes the unconditional flow matching objective
Let v [0, 1] x _2 + Ed be time-dependent vector field parametrized as a neural network with weights 6. Define the conditional flow matching (CFM) objective:
(CF [e - (x]=))]]
(10)
The CFM objective describes how to regress against the marginal vector field (x(x)) given by (9) with access only to samples from the conditional probability path p=(2))) and conditional vector fields ue (x]2). This is formalized in the following theorem.
Theorem 3.2. If px()) > 0 for all E Id and L [0, 1], then, up to constant independent of 0, LCFM and LFM are equal, and hence
VoLFM = (eLCFM(O).
(11)
The CFM objective is useful when the marginal vector field u, (z) is intractable but the conditional vector field (x(2)) tractable. As long as we can efficiently sample from q(2) and pr (x]2) and calculate uz(x)2), we use this stochastic objective regress vo to the marginal vector field uz(x).
We discuss the variance arising from the stochastic regression target, and ways to reduce it, in Proposition B.2 Proposition B.3
3.2
Sources of conditional probability paths
In this section, we introduce several forms of CFM depending on the choices of q(=), pe(-1z), and (x(+2). All of the CFM variants and related objectives from prior work are sunmarized in Table
$3.2.1 We interpret the algorithm of Lipman et al. (023) (FM from a Gaussian) as a special case CFM. $3.2.2 We relax the Gaussian source requirement by lotting the condition & be a pair (#0, of an initial and a terminal point. In the basic form of CFM (I-CFM), we take the distribution (22) equal q(xo)q(z1), allowing generative modeling with arbitrary source distribution.
5

---

Published in Transactions on Machine Learning Research (03/2024)
Algorithm 1 Conditional Flow Matching
Input: Efficiently samplable q(2), Po(x)2), and computable (x(222) and initial network vg. while Training do
r {(2); t U(0, 1); p<(p(())))
(0F + [t)(t - -,,,))) (x|2))1
t Update(0, VoLCEM(0)
return 0g
Table 1: Probability path definitions for existing methods which fit in the generalized conditional flow matching framework (top) and our newly defined paths (bottom). We define two new probability path
objectives that can handle general source distributions and optimal transport flows.
Probabiity Path
e(=)
2, Good OT Marginal OT
General source
Var. Exploding Souz Sc Eraaon
2019
v(x))
Var. Preserving
8020
g(n)
V
oi...
Flow Matching Lapmon
2020
e(x))
Rectifed Fluw Lau 2022
(n + (1 Ouz
X
Var. Pres. Stoclmas
Interpolan Albergo & Vanden-Ejnckn
2023
gitololrg g(sto)p(11))
coe(j sinl (rr
0
X
Independent;
[1 rjmg
(Ours) Optinal Transpoet CFM
x1
(n + (1-f1g (10g
V
V
(Ouns) Schnödinger Bridge CFM
Pgg [1).
+ Gm
my (11
$3.2.3
We consider joint distributions q(z) ( 21) that are given by minibatch optimal transport mapps, causing the learned flow to be an (approximate) OT flow.
$3.2.4 we consider g(2) given by entropy-regularized OT map and show that the CFM objective with
this (() solves the Schrödinger bridge problem.
3.2.1
FM from the Gaussian
Lipman et al. (202) considered the problem of unconditional generative modeling given training dataset. Identifying the condition z with a single datapoint 31, and choosing smoothing constant o 0, one sets
(x(2]2) = N(x (x1, (lo t + 1)3),
(12)
(x(=]= 31 (1 o)x
(13)
- 1-(1- ot)
which a probability path from the standard normal distribution (po(=12) = 0, I)) to a Gaussian distribution centered at Iı with standard deviation (p)()))) = ((;;,,, If one sets g(2) g(x)))
be the uniform distribution over the training dataset, the objective introduced by Lipman al. 22023 is
equivalent to the CFM objective (10) for this conditional probability path.
We emphasize that although the conditional probability path (=(a])) is optimal transport path from [z]2)) to pi (I|z), the marginal path (r()) is not in general an OT path from the standard normal po(x) to the data distribution pı (x).
3.2.2 Basic form of CFM: Independent coupling
In the basic form of CFM (I-CFM), we identify with pair of random variables, source point x0p and target point 21, and set g(2) ((z))9(21) to be the independent coupling. We let the conditionals be Gaussian flows between zo and 21 with standard deviation o, defined by
p=(x]2) N/x tri (1 t) o²),
(14)
Zg.
(15)
We note that the formulation of Uz (z|22) follows from an application of Theorem 2.1 the conditional probability path with pt t. + f)zg and ar Furthermore, we note that p=(2)2) is efficiently

---

Published in Transactions on Machine Learning Research (03/2024)
samplable and u, is efficiently computable, thus gradient descent on LCFM is also efficient. For this choice of 2, pt((-l),, and ux(+)z), we know the marginal boundary probabilities approach go and respectively as a 0. This is made explicit in the following Proposition:
Proposition 3.3. The marginal corresponding to q(z) = ((zo)g(a1) and the (t(2)2), (x(=2) in 14 and
15 has boundary conditions p1 91 (a 0, 0, and po qu * Nx 0, 0, where * denotes the convolution operator.
In particular, as a + O, the marginal vector field uz(x) approaches one that transports the distribution ( to ((21) and can thus be seen as a generative model of ø1. Note that there is no requirement for ((2o) to be Gaussian. Conditioning on vn allows us to generalize flow matching to arbitrary source distributions with intractable densities. In the case of FM from a Gaussian, while each conditional flow is the dynamic optimal transport flow from M( o to MMI,, the marginal vector field ux(x) is not necessarily an OT flow.
Connection with related Rectified Flow and Stochastic interpolants methods. We note that I-CFM is closely related to the algorithms proposed by Albergo & Vanden- Vanden-Eijnden (2023) Liu 2022 In the case where the conditional probability path Ps is a Dirac (?.e., = 0), I-CFM is equivalent to (Liu 2022). Furthermore, if we consider the Gaussian mean 14 = ( = + sin( { =t)aa instead of the linear interpolation, I-CFM would be equivalent to the variance preserving stochastic interpolant in Albergo & 22023). which has also been further generalized.
Connection to FM from the Gaussian. There exists a set of conditional probability paths conditioned 2 and To _N 1) that have equivalent probability flow to the marginal p. of flow matching from the Gaussian S3.2.1 which is only conditioned on ø1. These paths are defined by
p((x]2) (x tz (1 -1-18 -t) (ot)² 20t(1 - t)).
(16)
Proposition B.1 states an equivalence between I-CFM with these paths and the objective from $3.2.1
3.2.3 Optimal transport CFM
In this section, we present our second main contribution. The formulation in the previous section can readily be generalized to distributions q(2) = (( 21) in which 2n and i are not independent, as long as q(z) has marginals q(zo) and q(21). Therefore, we propose set ((2) to be the 2- Wassersttni optimal transport map æ achieving the infimum in G namely,
((2) x < x1).
(17)
In this case, z is still a tuple of points, but instead of zo, ω1 being sampled independently from their marginal distributions, they are sampled jointly according to the optimal transport map т. We call this method optimal transport CFM (OT-CFM). If one uses the (x|22) defined by [4] and u (x(2) in (15p. OT-CFM is equivalent to dynamic optimal transport in the following sense.
Proposition 3.4. The results of Proposition 3.3 also hold for q(2) in [7]. Furthermore, assuming regularity properties of go, QI, and the optimal transport plan æ, as a² -> the marginal path pt and field ut minimize 0. i.e., un solves the dynamic optimol transport problem between qu and q1.
We consider two cases: (1) when the data set is small enough and we know the static optimal transport plan (e.g, single cell data). (2) when the data is too large (or continuous) (e.g, image data) and the static OT plan is computationally infcasible to determine exactly. In the first case map to unseen data similar to the task presented in Bunne et al. (2023). In the second case we show an approximation with minibatch OT improves over a random plan in terms of generative modelling performance and training time.
able to extend the transport
Minibatch OT approximation. For large datasets, the transport plan π can be difficult to compute and store due to OT's cubic time and quadratic memory complexity in the number of samples (Cuturi) 2013 Tong et al.) 2020). Therefore, we rely on a minibatch OT approximation similar to Fatras al. (201bb). Although minibatch OT incurs an error relative to the exact OT solution, it has been successfully used in
7

---

Published in Transactions on Machine Learning Research (03/2024)
many applications like domain adaptation or generative modeling (Damodaran et al. 2018 Genevay et al. 2018). Specifically, for each batch of data (1rg) (+) LB Ji=1, {यं (z(()) ) Ji=1) seen during training, we sample pairs points from the joint distribution Thatch given by the OT plan between the source and target points in the batch. (The OT batch size need not match the optimization batch size, but we keep them equal for simplicity.) Thus, solve a minibatch approximation of dynamic optimal transport. However, when the OT batch size equals the support size of (40, QI); recover exact OT and therefore, by Proposiiiion 3.4 learn the exact dynamic optimal transport. We show empirically that the batch size can be much smaller than the full dataset size and still give good performance, which aligns with prior studies {Fatras et al. 2020 2021a). Concurrently, a similar framework and thcoretical results appeared in Pooladian et al. (2023
3.2.4 Schrödinger bridge CFM
Recently, there has been significant effort in learning diffusion models with general source distributions, formulated as a Schrödinger bridge problem (De Bortoli et al. 2021 Vargas et al. 2021 Chen et al. 2022 or bridge matching Peluchetti (2023); Liu et al. (2022b]; Ye et al. (0022). Here we show that SB-CFM, an entropic variant of OT-CFM, can be used to train an ODE match the probability flow of Schrödinger bridge with a Brownian motion reference process.
Let Pref be the standard Wiener process scaled by with initial-time marginal Pror(Fo) ((0)). The Schrödinger bridge problem (Schrödinger 1932 seeks the process that is closest to Pref while having initial and terminal marginal distributions specified by the data distribution (xxo) and ((x1):
π
arg min
KL(T Prot).
(18)
=(3o)= (10)))=())))))))))0)))
We define the joint distribution
(==) = T2o² (x),
(19)
where π2 is the solution of the entropy-regullarized optimal transport problem Cuturi 2013 with cost. ||00- - =x1] and entropy regularization = 2o (see (33) for the background on entropic OT). We set the
conditional path distribution to be Brownian bridge with diffusion scale a between zo and æ1, with probability path and generating vector field
(x Nax lon (1 f) 1(1 1)a²)
(20)
- 2t
uy 2) = 2t(1 t) (x - (tx1 + (1 t)zo)) + (x1 - zo),
(21)
where ue is computed by ( as the vector field generating the probability path (=(2))) The marginal coupling 22 and (x(z)) define ut (2), which is approximated by the regression objective in Alg. The solution of the SB known to be the map which the solution of the entropically-reguuaiizee OT problem, motivating the next proposition.
Proposition 3.5. The marginal vector field ux(x) defined by probability path as the solution n" to the SB problem in 18
19 and 21 generates the same marginal
While we define SB-CFM with entropic regularization coefficient of 20 the flow still matches the marginals for any choice of e. Interestingly, we recover OT-CFM when -> 0 and I-CFM when co. A similar result was proven in concurrent work Pooladian et al. (022)
4 Related work
Simulation-free continuous-time modeling. Simulation-free training is common in stochastic flow
models where backpropagating through the simulation is numerically challenging and has high variance (Li
et al. 2020). While these diffusion models have recently achieved exceptional generative performance on many
tasks Sick--Dik et al.
2015 [Song & Ermon 2019 2020 Ho al. 2020; Song et al.) 2021b Dhariwal &
Nichol 2021] Watson al. 2022b), their simulation requires an inherently costly SDE simulation with many
follow-up works to improve inference efficiency (Lu et al. 2022 Salimans & Ho. 2022 Watson et al. 2022a
8

---

Published in Transactions on Machine Learning Research (03/2024)
Table 2: Comparison of neural optimal transport methods over four distribution pairs ( a over five seeds)
in terms of fit (2- Wasserstein), optimal transport performance (normalized path energy), and runtime. indicates a method that requires Gaussian source. Best in bold. CFM and RF models are trained on a single CPU core, other basolines trained with a GPU and two CPUs.
Datawel
8--Bgaussiass
moons Bgaussians
N-Hmoons
N-+scurve
Arg. urain liue
Alguritam Meteie +
W2
NPE
112
NPE
W2
NPE
W32
NPE
(x10* x)
OT-CFM
1.262s0.34x
0.018:0.114
a.923
0.053
0.239
0.087
0.264
0,027
1.129
I-CFM
1,28 160.364
0.202e0.002
1.97750,23
2. 2.738
0.338ea.0
0.841eaue
0.333
0.867=4.17
0.630
2-RF Lau
2120
1.136es0.44
0 0.069es.co7
2.31 149,42
19
0.278ea.res
0.076
0.395
0.112-n0s
0,862+0.00
SRF
Lat
A122
1,30
0, 0.065e..004
2.700
0.123
a.305
0.084
0.395=
0.964
FM
2125
1.062s0.00
A.17
0.316ea.077
0.778eau144
0.377=
0.772
0,708
Rog. CNP Finlay et al.
2020
1o.74
0.376
0.620 b.s
0.586 0.00
8.021 1..200
CNF
Chen
2015
10.55
0.151
0.387
2.937=1.073
0.645
10.548=x.100
18.8 110
ICNN
Maliiuaa of m 2020
1.771 10.31
0.747 Lo.ces
2.193
0.832
0.267 1o.00
0.753_
0.344_nos
1.912
Song et al. 2021a Bao et a.., 2022). These methods generally consider a simple Gaussian diffusion process, and do not consider gencralizing the source distribution. Other works consider general source distributions but this makes optimization and inference more challenging, needing multiple iterations or other tricks to
perform well (Wang et al. 2021 De Bortoli al. 2021 Vargas et al. 2021).
Prior work considcring simulation-free training of CNFs considers algorithms that are equivalent to CFM with Gaussian source distribution (Rozen el al. 2021 Ben- Hamu et. al. 2022 Lipman el al. 2023) or independent
samples from q0, 91 (Albergo & Vanden-Eijj Eijnden 2023 Albergo et al.) 2023 Neklyudov et al. 2023). Recent
work also studies Schrödinger bridges from unpaired samples (Shi et al. 2022) and regularization of flows using dynamic OT (Liu et al.] 2023b). We also note the work Pooladian et al. (2023). concurrent with the preprint version of this paper.
Other concurrent works explore various solutions to approximate Schrödinger
bridges (Somnath et al. 2023 Shi et al. 2023 Liu al.) 2023a).
Dynamic optimal transport. There are a varicty of methods that consider dynamic OT between continuous distributions with neural networks; however, these require constrained architectures (Leygonic et al.] 2019: Makkuva al. 2020; Bunne et al. 2022) or use a. regularized CNF, which is challenging optimize (Tong et al. 2020 Finlay et al.) 2020 Onken et al., 2021 Huguet et al. 2022a). With our work it is possible to achieve optimal transport flows without either of these constraints.
5 Experiments
In this section empirically evaluate the I-CFM, OT-CFM, and SB-CFM objectives, well as algorithms from prior work, with respect to both optimal transport and generative modeling criteria. All experiment details can be found in Appendix E
5.1 Low-dimensional data: Optimal transport and faster convergence
We ovaluate how well various models perform dynamic optimal transport and generative modeling in low dimensions. We train ODEs mapping between four pairs of two-dimensional datasets: between a standard Gaussian and 8gaussians, moons, and scurve and between moons and 8gaussians.
OT-CFM approximates dynamic OT. To measure how well model solves the OT problem we use normalized path energy (NPE), defined via the 2-Sassrsttcin distance as NPE(up) = PP(()))
W2 9:) 91)l (10. q1), where the path energy (PE) is PE(vş) = Ex(0)=((za) So ( att Table 2 summarizes our results showing that OT-CFM flows generalize better to the test set and are very close to the dynamic OT paths as measured by normalized path energy. We find transforming moons-8gaussianss lo be particularly challenging to learn for 1-CFM as compared to OT-CFM; the learned paths are depicted in Fig. (bottom). Although OT-CFM uses a minibatch OT map, we find that OT-CFM requires surprisingly small batches approximate the OT map well, suggesting some generalization advantages of the network optimization Fig. D.2
9

---

Published in Transactions on Machine Learning Research (03/2024)
Satver suker
3.5
-0-
1.-
Methood CT-CH
Mettod
63-
2.0
2.5
CTM
0.5
0.0
FM
CHN
1.5-
0.2
101
107
30
Figure 2: Left: OT-CFM trains faster, in terms of validation set error, than CFM and FM models. Right: With different ODE integrators, OT-CFM reduces the error for a fixed number of function evaluations during inference.
Dataset 1 Alg. +
SB-CFM
DSB
Table Schrödinger bridge flow comparison, showing average error over flow time ground truth averaged over 5 models for SB-CFM and dynamics from DSB
N -gaassiaans
0.454 + 0.164
1.440 + 0.720
moons +8gaussagasss
1.377 + 0.229
2.407 + 1.025
De Bortoli et al. 2021
N-ymoons
0.283 + 0.048
0.333 + 0.1
-
0.297 0.064
0.383 + 0.134
OT-CFM yiclds faster training. By conditioning minibatch optimal transport flows, OT-CFM is substantially easier to train, which we posit is due to the variance reduction of the conditional flow. In Fig. 2 (left), we evaluate the performance over time of OT-CFM against CFM and FM objectives. For the same number of steps OT-CFM has better performance on the validation set. In Table D.I we compare the training times for various Neural OT methods whose performance can be seen in Table optimization is significantly faster to train with equal superior performance.
Simulation-free
OT-CFM yields faster inference. We next evaluate the quality of samples during inference time. In Fig. 2 (right), we compare the quality of samples for different number of function evaluations (NFEs) across different flow matching objectives. In this experiment
sample from the source distribution test set and simulate the ODE over time for different solvers. We find that OT-CFM consistently requires fewer evaluations to achieve the same quality and achieves better quality with the same NFEs. This is consistent with previous work, which found OT paths lead to faster, higher quality inforence in regularized CNFs (Finlay et al. 2020 Onken et al. 2021) and flow matching vs. standard variance- preservine and variance-explodinng probability paths [Lipman et al. 2023).
SB-CFM reproduces Schrödinger bridge flows. There are a number of methods which theoretically converge a Schrödinger bridge between two datasets. Table 3 we compare SB-CFM and the diffusion Schrödinger bridge (DSB) method introduced in De Bortoli et al. (2021) on the quality of the learnt Schrödinger bridges based on the average 2- Wasserstein distance to ground truth Schrödinger bridge samples over 18 time steps. Furthermore, SB-CFM is also significantly faster than DSB Table D.1
5.2 Application to single-cell interpolation
As specific application, we consider the task of single-cell trajectory interpolation. In this task we use leave-one-out validation over the timepoints. From times data at times 00 1),
[t + 1, try to interpolate : Tong et al. (2020): Huguet et al. (2022a). Low error means we model individual colls well, which is useful in a number of downstream tasks such as gene regulatory network inference (Aliee et al.) 2021 Yeo et. al.] 2021). Following Huguet et. al. (022b),, we repurpose the CITE-seq and Multiome datasets from a recent NeurIPS competition for this task (Burkhardt et al. 2022). We also include the Embryoid body data from Moon et al. (2019); Tong et al. (2020). Table 4 shows the average earth mover's distance (1-Wasserstein) on left-out timepoints for three datasets. On all three datasets OT-CFM outperforms other methods and baselines average.
its distribution at time following the setup of Schiiebnnee et al. (0019)
10

---

Published in Transactions on Machine Learning Research (03/2024)
Table 4: Single-cell comparison over three datasets averaged over leaving out intermediate time- points measuring EMD to left out distribution following Tong et al. (2020). *Indicates values taken from aforementioned work.
Algorithm +. Dataset +
Cite
EB
Mulsi
TrajectoryNet (Tong et al.
2020
0.848 1
Reg. CNF Finlay
2020
0.825
DSB (De Bortali al.
0.140
0.862
0.023
1.079
0.117
I-CFM
0.963 0.111
0.872 + 0.087
1.085
0.009
SB-CFM
1.067 1: 0.107
1.221 1 0.380
1.129
= 0.363
OT-CFM
0.882 1 0.058
0.790 1 0.068
0.937
1 0.054
6.0
Model
Model
5.5
I-CFM
30
I-CFM
FM
FM
5.0
OT-CFM
OT-CFM
FID
20
4.5
4.0
10
3.5
100000
200000
300000 400000 500000 Train Step
10a
102
10
NFE
Figure 3: Left: Fréchet inception distance (FID) scores on CIFAR-10 for different numbers of training steps using a DOPRI5 adaptive solver. Right: FID scores on CIFAR-10 using Euler integration for various numbers of function evaluations (NFE) per sample after 400k training steps. In both cases, OT-CFM outperforms 1-CFM and FM models, showing the benefits of minibatch optimal transport.
5.3
High- digeediim data: Lower-cost training and inference
We perform experiment unconditional CIFAR-10 generation from Gaussian source to examine how OT-CFM performs in the high-dimensional image setting. We use similar sctup to that of Lipman et al. (2023), including the time-dependenl U-Net archilecture from Nichol & Dhariwal (2021) that is commonly used in diffusion models. We were not able to reproduce the results reported from Lipman et al. (2023) with the parameters specified in the paper 3 Therefore, we selected different training hyperparameters.
The main differences with Lipman el. al. (2023) are that we use a constant learning rale, set 2 10-1, instead of linearly decreasing one (from x 10 to 10-8). To prevent training instabilities and variance, clip the gradient norm to 1 and rely exponential moving average with decay of 0.9999. Regarding the architecture, used the same Lipman et al. (2023), but with smaller number of channels (128 instead of 256), leading to much faster training, as well 10% dropout. Furthermore, our batch size was 128 instead of 256, which leads to a reduced memory cost.
We train our OT-CFM, as well as I-CFM and the original FM, with this new training procedure and report the Fréchet inception distance (FID) Table 5] In Fig. (left), we show the FID over training time with the Dormand-Prinet Prince fifth-order adaptive solver (Hairer et al. 1993 DOPRI5) using a relative and absolute error threshold of 10-5 similarly to Lipman al. (2023). and Fig. 3 (right), we present the FID as function of the numbers of function evaluations (NFE) using Euler integration.
We find that:
"Specifically, find that the number generated samples for FID calculation, the value of the smoothing constant dmins any data augmentation used, the standard deviation of the distribution po. and the batch size used during evaluation (which affect the function evaluation count with adaptive integrators) not specified in Lipman al. 2023 's manuscript. addition, contradictory information is given about the number of training epochs.
11

---

Published in Transactions on Machine Learning Research (03/2024)
Table FID score and number of function eval- uations (NFE) for different ODE solvers: fixed-
NFE / sample +
100
1000
Adaptive
step Euler integration with 100 and 1000 steps and adaptive integration (DOPRI5). The adap- tive solver is significantly better than the Euler solver in fewer steps. First three results are from Lipman et al. (0233) and fourth from Albergo &
Algorithm +
FID
FID
FID
NFE
DDPM
7.48
274
OT-FM (reported)
6.35
142
VP-FM (reported)
8.06
183
S.I. (reported)
10.27
Handen-Eijnden (2023). The fifth line is our re-
OT-FM (reproduced)
13.742
12.491
11.527
139.83
produced results following Lipman et al. (2023)'s 's training procedure. We have run OT-FM, S.1.
LipmanngLLppmann
VP-FM (ours)
7.772
4.048
4.335
525.92
and VP-FM following and we have denoled them (ours). The two last rows report the results of our proposed methos I-CFM and OT-CFM.
training procedure
OT-FM (ours)
4.640
3.822
3.655
143.00
S.I. (ours)
4.488
4.132
4.009
146.12
I-CFM (ours)
4.461
3.643
3.659
146.42
OT-CFM (ours)
4.443
3.741
3.577
133.94
With improved hyperparameters, achieve significantly better FID with the FM training objective than the one reported by Lipman et al. (2023) at a lower cost.
For short compulation budget, OT-CFM out perforrs FM and (non-OT) I-CFM Tablle 5 left).
After a long training time, all methods achieve similar performance at high number of function evaluations
using fixed-step ODE intogration, but OT-CFM performs significantly better with a small number of lunction evaluations (i.e., allows more efficient. inference); indicating straighter, easily integrable flows
Table 5) right).
FM and 1-CFM are equivalently computationally efficient per iteration and OT-CFM comes with low (<1%) computational overhead during training.
Our new training procedures, available at https: o allow us to outperform the previous reported results from Lipman et al. (2023), while the results with our OT-CFM are state-of-the-an for simulation-fi neural ODE training algorithms.
5.4 OT-CFM for unsupervised translation
We show how CFM can be used learn mapping between two unpaired datasets in high-dimensional space using the CelebA dataset (Liu al., 2015 Sun al.) 2014), which consists of ~ 200k images of faces together with 40 binary attribute annotations. For each attribute, wish to learn invertible mapping between images with and without the attribute (e.g., 'not smilling" ++ 'smiling').
To reduce dimensionality, we first train a VAE on the images and encode them as 128-dimensional latent vectors. For each altribute, we learn a flow lo map between the embeddings of images without the altribute and those of images with the attribute. After the CNF learned, we push forward a held-out set negative vectors by the CNF and compare them to the held-out posítive vectors and vice versa. As metric of divergence, we use maximum mean discrepancy (MMD) with a broad Gaussian kernel (exp(-1|<r yll 128))). The results aggregated over all attributes are shown in Table 6 showing that OT-CFM discovers better mapping than other methods. Although MMD is lower for larger σ, found that the alignment is less natural when is large, and performance begins to degrade when o > 1. Fgg. E. shows several visualizations of the learned trajectories.
Finally, while here we work in a latent space, future work should consider learning flows directly in image space, where GAN-based approaches (Zhu al. 2017) continue dominate.
5.5 Additional experiments and extensions
We present numerous other extensions, applications, and evaluations of CFM in Appendix D) notably:
12

---

Published in Transactions on Machine Learning Research (03/2024)
Table 6: MMD (in units of 10-3) between target and transformed source samples of CelebA latent vectors, Mean and standard deviation over 40 attributes and both translation directions (- + +) for each attribute. 'Identity' refers to performing translation and treat- ing source samples as approximate samples from the target.
Algorithm |
a 0.1
0 = 0.3
0
Identity
9.17 5.68
9.17 + 5.68
9.17 + 5.68
I-CFM
4.09
3.44 + 2.03
1.59 = 0.83
OT-CFM
2.81 + 2.62
1.91 # 1.30
1.04 = 0.60
OT-CFM reduces variance in the regression target. To accompany the theoretical results in |C.1 we empirically study the variance of the stochastic regression objective in (OT-)CFM. The results suggest an explanation for the faster convergence models trained with OT-CFM.
in S DD.
Energy-based CFM. In SC.2 and SD.2 we show how CFM can be used fit samplers for unnormalized density functions, where exact samples from ((z)) or ((x)) are not available.
Extension to stochastic dynamics. from unpaired source and larget data.
Tong ct al. 2024 extends CFM to allow lcarning stochastic dynamics
6 Conclusion
We have introduced a novel class of simulation-free objectives for learning continuous-lime flows with a general source distribution. Our approach to training continuous normalizing flows and conditional flow models does not require integration over time during training. We have shown that lifting the static optimal transport problem to the dynamic setting leads to simulation-free solutions to the dynamic OT and SB problems, while also allowing more efficient training and inference of flow models by lowering variance of the objective and straightening flows. One limitation of CFM is that it requires closed-form conditional flows, which hinders its application to situations where we want to regularize the marginal vector field (x(r) based on prior information (Tong et al. 2020). In addition, the minibatch approximation to OT can incur error in high dimensions; subsequent work can consider the use of neural-network approximations to OT maps (Korotin al. 2023b a) conjunction with CFM. We expect future work to overcome these limitations and hope that ideas from conditional flow matching will improve high-dimensional generative models.
Contribution statement
A.T. initially conceived the idea. Y.Z., G.H., and N.M. led the development of the theory. High-dimensional experiments and open-source code were led by A.T. and K.F. Additional experiments were contributed by Y.Z., J.R.,, G.H., N.M., and A.T. All authors contributed to designing the experiments. N.M. and A.T. drove the writing of the paper, with contributions from all other authors. G.W. and Y.B. guided the project.
Acknowledgments
We would like to thank Stefano Massaroli for productive conversations as well as thank Xinyu Yuan, Marco Jiralerspong, Tara Akhound-Sadegh and Jocy Bose for their helpful comments and feedback the manuscript. We are also grateful to the anonymous reviewers for suggesting numerous improvements. This research enabled in part by compute resources provided by Mila (mila.quebec) and NVIDIA Corporation. The authors acknowledge funding from CIFAR, Genentech, Samsung, and 1BM. In addition, K.F. acknowledges funding from NSERC (RGPIN--001--66512)) and G.W. acknowledges funding from NSERC Discovery grant 03267 and NIH grant R01GM135929.
References
Michael S. Albergo and Eric Vanden- Eijnden. Building normalizing flows with stochastic interpolants.
International Conference on Learning Representations (ICLR), 2023.
13

---

Published in Transactions on Machine Learning Research (03/2024)
Michael S. Albergo, Nicholas M. Boffi, and Eric Vanden-Eijnder Stochastic interpolants: A unifying framework for flows and diffusions. orXiv preprint 2303.08797, 2023.
Hananeh Aliee, Fabian J. Theis, and Niki Kilbertus. Beyond predictions in neural ODEs: Identification and interventions. arXiv preprint 2106. 12430, 2021.
Luigi Ambrosio and Nicola Gigli. User's Guide to Optimal Transport, pp. 1 155. Springer Berlin Heidelberg, Berlin, Heidelberg, 2013.
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne den Berg. Structured denoising diffusion models in discrete state-spaces. Neural Information Processing Systems (NewrIPS), 2021.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang, Analytic-dpm: An analytic estimate of the optimal reverse variance in diffusion probabilistic models. International Conference on Learning Representations (ICLR), 2022.
Heli Ben-Hamu, Samuel Cohen, Joey Bose, Brandon Amos, Aditya Grover, Maximilian Nickel, Ricky T. Q- Chen, and Yaron Lipman. Matching normalizing flows and probability paths on manifolds. International Conference on Machine Learning (ICML), 2022.
Jean-David Benamou and Yann Brenier. A computational fluid mechanics solution to the Monge-Kantorovich mass transfer problem. Numerische Mathematik, 84(3):375 393, 2000.
Yann Brenier. Polar factorization and monotone rearrangement of voctor-valued functions. Communications on Pure and Applied Muthemalics, 44(4):375 417, 1991.
Charlotte Bunne, Laetitia Meng-Papaxantee transport modeling of population dynamics. Artificial Intelligence and Statistics (ATSTATS), 2022.
Andreas Krause, and Marco Cuturi. Proximal optimal
Charlotte Bunne, Stefan G. Stark, Gabriele Gut, Jacobo Sarabia del Castillo, Mitch Levesque, Kjong- Van Lehmann, Lucas Pelkmans, Andreas Krause, and Gunnar Rätsch. Learning single-cell perturbation responses using neural optimal transport. Nalure Methods, 20(11):1759 1767779997777---77 Nov 2023. ISSN 1548-7105. doi: 10.1038//411592-023--000 022-1999xx URL https://doi. 1tt.://00....0///444552000000100
Daniel Burkhardt, Jonathan Bloom, Robrecht Cannoodt, Malte D Luecken, Smita Krishnaswamy, Christopher Lance, Angela O Pisco, and Fabian Theis. Multimodal single-cell integration across time, individuals, and batches. In NeurIPS Compelitions, 2022.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud, Neural ordinary differential equations. Neural Information Processing Systems (NeurIPS), 2018.
Ricky T.Q. Chen and Yaron Lipman. Flow matching on general geometries. International Conference on Learning Representations (ICLR), 2024.
Tianrong Chen, Guan-Horng Liu, and Evangelos A. Theodorou. Likelihood training of Schrödinger bridge using borward-backward SDEs theory. International Conference on Learning Representations (ICLR), 2022.
Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi Jaakkola. DiffDock: Diffusion steps, twists, and turns for molecular docking. International Conference on Learning Representations (ICLR), 2023.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Newral Information Processing Systems (NIPS), 2013.
Bharath Bhushan Damodaran, Benjamin Kellenberger, Remi Flamary, Devis Tuia, and Nicolas Courty. DeepJDOT: Deep joint distribution optimal transport for unsupervised domain adaptation. Europeon Conference Compuler Vision (ECCV), 2018.
14

---

Published in Transactions on Machine Learning Research (03/2024)
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion Schrödinger bridge with applications to score-based generative modeling. Newral Information Processing Systems (NewrIPS), 2021.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. Neural Information Processing Systems (NeurIPS), 2021.
Kilian Fatras, Younes Zine, Rémi Flamary, Rémi Gribonval, and Nicolas Courty. Learning with minibatch Wasserstein: Asymptotic and gradient properties. Artificial Intelligence and Statistics (AISTATS), 2020.
Kilian Fatras, Thibault Sejourne, Rémi Flamary, and Nicolas Courty. Unbalanced minibatch optimal transport; applications to domain adaptation. International Conference on Machine Learning (ICMT.), 2021a.
Kilian Fatras, Younes Zine, Szymon Majewski, Rémi Flamary, Rémi Gribonval, and Nicolas Courty. Minibatch optimal transport distances; analysis and applications. arXiv preprint 2101 2021b.
Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam M. Oberman. How to train your neural ode: The world of jacobian and kinetic regularization. International Conference on Machine Learning (ICML), 2020.
Remi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar 7 Alaya, Aurelie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Leo Gautheron, Nathalie T Н Gayraud, Hicham Janati, Alain Rakotomamoniy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. POT: Python Optimal Transport. Journal of Machine Learning Research (JMLR), 22, 2021.
Aude Genevay, Gabriel Peyre, and Marco Cuturi. Learning generative models with Sinkhorn divergences. Artificial Intelligence and Statistics (AISTATS), 2018.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Neural Information Processing Systems (NIPS), 2014.
Will Grathwohl, Ricky T. Q- Chen, Jesse Bettencourt, Lya Sutskever, and David Duvenaud. Ffjord: Free- form continuous dynamics for scalable reversible generative models. Internalional Conference on Learving Representations (ICLR), 2019.
Ernst Hairer, Syvert P Nørsett, and Gerhard Wanner. Solving ordinary differential equations I. Nonstiff problems. Springer, 1993.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Neural Information Processing Systems (NeurIPS), 2020.
Matthew D. Hoffman and Andrew Gelman. The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research (JMLR), 15:1593-1622 2011.
Guillaune Huguet, D. S. Magruder, Alexander Tong, Oluwadamilola Fasina, Manik Kuchroo, Guy Wolf, and Smita. Krishnaswany. Manifold interpolating optimal-transport flows for trajectory inference. Neural Information Processing Systems (NeurIPS), 2022a.
Guillaume Huguet, Alexander Tong, María Ramos Zapatero, Guy Wolf, and Smita Krishnaswamy. Geodesic Sinkhorn: Optimal transport for high-dimensional datasets. arXiv preprint 2211.00800 2022b.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. International Conference on Learning Representations (ICLR), 2014.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Sclf-normalizing neural networks. Newral Information Processing Systems (NIPS), 2017.
Alexander Korotin, Lingxiao Li, Aude Genevay, Justin M Solomon, Alexander Filippov, and Evgeny Burnaev. Do neural optimal transport solvers work? a continnous wasserstein-2 benchmark. Neural Information Processing Systems (NeurIPS), 2021.
15

---

Published in Transactions on Machine Learning Research (03/2024)
Alexander Korotin, Daniil Selikhanorych, and Evgeny Burnaev. Kenrnel neural optimal transport. Interna- tional Conference on Learning Representalions (TCLR), 2023a.
Alexander Korotin, Daniil Selikhanovych, and Evgeny Burnaev. Neural optimal transport. International Conference on Learning Representations (TCIR), 2023b.
Erich L Lehmann and George Casella. Theory of point estimation. Springer Science & Business Media, 2006.
Christian Léonard. Some properties of path measures. In Catherine Donati-Martin, Antoine Lejay, and Alain Rouault (eds.), Séminaire de Probabilités XLVI, pp. 207 230. Springer, 2014a.
Christian Léonard. A survey of the Schrödinger problem and some of its connections with optimal transport. Discrete and Continuous Dynamical Systems, 34(4):1533 1574, 2014b.
Jacob Leygonie, Jennifer She, Amjad Almahairi, Sai Rajeswar, and Aaron Courville. Adversarial computation of optimal transport maps. arXic preprint 1906. 09691, 2019.
Xuechen Li, Ting-Kam Leonard Wong, Ricky T. Q. Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. Artificial Intelligence and Statistics (AISTATS), 2020.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. International Conference on Learning Representations (ICLR), 2023.
Guan-Horng Liu, Tianrong Chen, Oswin So, and Evangelos A Theodorou. Deep generalized Schrödinger
bridge. Newral Information Processing Systems (NeurIPS), 2022a.
Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A. Thoodorou, Weili Nie, and Anima Anandkumar. Pssb: Image-to-image schrödinger bridge. International Conference on Machine Learning (ICML), 2023a.
Qiang Liu. Rectified flow: marginal preserving approach to optimal transport. arXiv preprint 2209. 14577, 2022.
Xingchao Liu, Lemeng Wu, Mao Ye, and Qiang Liu. Let us build bridges: Understanding and extending diffusion generative models, 2022b.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. International Conference on Learning Representations (ICLR), 2023b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. International Conference Computer Vision (ICCV), 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Conference on Learning Representations (ICLR), 2019.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffiusio probabilistic model sampling in around 10 steps. Neural Information Processing Systems (NeurIPS), 2022.
Ashok Vardhan Makkuva, Amirhossein Taghvaci, Sewoong Oh, and Jason D. Lee. Optimal transport mapping via input convex neural networks. International Conference on Machine Learning (ICML), 2020.
Anton Mallasto, Augusto Gerolin, and Minh Ha Quang. Entropy-regnlarized 2- 2-Wasserstein distance between Gaussian measures. Information Geometry, 5, 07 2022.
Kevin R. Moon, David van Dijk, Zheng Wang, Scott Gigante, Daniel B. Burkhardt, William S. Chen, Kristina Yim, Antonia. van den Elzen, Matthew J. Hirn, Ronald R. Coifman, Natalía B. Ivanova, Guy Wolf, and Smita Krishnaswamy. Visualizing structure and transitions in high-dimensional biological data. Nature Biolechnology, 37(1))11482 1492, 2019.
Kirill Neklyudov, Daniel Severo, and Alireza Makhzani. Action matching: Learning stochastic dynamics from samples. International Conference Machine Learning (TCMT), 2023.
16

---

Published in Transactions on Machine Learning Research (03/2024)
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. International Conference on Machine Learning (TCML.) 2021.
Derek Onken, Samy Wu Fung, Xingjian Li, and Lars Ruthotto. OT-Flow: Fast and accurate continuous normalizing flows via optimal transport. Association for the Advancement of Artificial Intelligence (AAAT), 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorchh: An imperative style, high-performance deep learning library. Neural Information Processing Systems (NewrIPS), 2019.
Stefano Peluchetti. Non-denoising forward-time diffusions. arXiv preprint orXio:2312.1458, 2023.
Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-6):355 607, 2019.
Aram-Alexandre Pooladian, Heli Ben- Ben-Hamu, Carles Domingo-Enrich, Brandon Amos, Yaron Lipman, and Ricky T.Q. Chen. Multisample flow matching: Straightening flows with minibatch couplings. International Conference on Learning Representations (TCLR), 2023.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. International Conference Machine Learning (ICML), 2015.
Noam Rozen, Aditya Grover, Maximilian Nickel, and Yaron Lipman. Moser flow: Divergence-based generative modeling on manifolds. Newral Information Processing Systems (NewrIPS), 2021.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. International Conference on Learning Representations (ICLR), 2022.
Geoffrey Schiebinger, Jian Shu, Marcin Tabaka, Brian Cleary, Vidya Subramanian, Aryeh Solomon, Joshua Gould, Siyan Liu, Stacie Lin, Peter Berube, Lia Lee, Jenny Chen, Justin Brumbangh, Philippe Rigollet, Konrad Hochedlinger, Rudolf Jaenisch, Aviv Regev, and Eric S. Lander. Optimal-transport analysis of single-cell gene expression identifies developmental trajectories in reprogramming. Cell, 176(4):928 (((4)))988993.22,, 2019.
Erwin Schrödinger. Sur la théoric relativiste de l'électron et l'interprétation de la mécanique quantique. Annales de I l'Institu Henri Poincaré, 2(4):269-35) 310, 1932.
Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. Conditional simulation using diffusion Schrödinger bridges. Uncertainty in Artificial Intelligence (UAI), 2022.
Yuyang Shi, Valentin De Bortoli, Andrew Campbell, and Arnaud Doucet. Diffusion schrödinger bridge matching. Neural Information Processing Systems (NewrIPS), 2023.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning (ICML), 2015.
Vignesh Ram Somnath, Matteo Pariset, Ya-Ping Hsich, Maria Rodriguez Martinez, Andreas Krause, and Charlotte Bunne. Aligned diffusion schrödinger bridges. Uncertainty in Artificial Intelligence (UAI), 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. International Conference on Learning Representations (ICLR), 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Neural Information Processing Systems (NeurIPS), 2019.
Yang Song and Stefano Ermon. Improved techniques for training score- based generative models. Neural Information Processing Systems (NeurIPS), 2020.
17

---

Published in Transactions on Machine Learning Research (03/2024)
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference Learning Representations (ICLR), 2021b.
Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation by joint identificatioo-veriiiattot Neural Information Processing Systems (NIPS), 2014.
Alexander Tong, Jessie Huang, Guy Wolf, David van Dijk, and Smita Krishnaswamy, TrajectoryNet: A dynamic optimal transport network for modeling cellular dynamics. International Conference on Machine Learning (ICML), 2020.
Alexander Tong, Nikolay Malkin, Kilian Fatras, Lazar Atanackovic, Yanlei Zhang, Guillaume Huguet, Guy Wolf, and Yoshua Bengio. Simulation-free schrödinger bridges via score and flow matching, Artificial Intelligence and Slatistics (AISTATS), 2024.
Francisco Vargas, Pierre Thodoroff, Neil D. Lawrence, and Austen Lamacraft. Solving Schrödinger bridges via maximum likelihood. Entropy, 23),,, 2021.
Cédric Villani. Optimal transport, volume 338 of Grundlebren der mathematischen Wissenschaflen [Funda- mental Principles of Mathematical Sciences]. Springer- Opringer-Verlag, Berlin, 2009.
Gefei Wang, Yuling Jiao, Qian Xu, Yang Wang, and Can Yang. Deep generalive learning via Schrödinger bridge. International Conference on Machine Learning (ICML), 2021.
Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusson models by differentiating through sample quality. International Conference on Learning Representations (ICLR), 2022a.
Joseph L. Watson, David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen F. Fisenach, Woody Ahern, Andrew J. Borst, Robert J. Ragotte, Lukas F. Milles, Basile 1. M. Wicky, Nikita Hanikel, Samuel J. Pellock, Alexis Courbet, William Shefller, Jue Wang, Preetham Venkatesh, Isaac Sappington, Susana Vázquez Torres, Anna Lauko, Valentin De Bortoli, Emile Mathieu, Regina Barzilay, Tommi S. Jaakkola, Frank DiMaio, Minkyung Back, and David Baker. Broadly applicable and accurate protein design by integrating structure prediction networks and diffussin generative models. bioRrie preprint 2022. 12. 09..599..92.. 2022b.
Mao Yc, Lemeng Wu, and Qiang Liu. First hitting diffusion models for generating manifold, graph and categorical data. Neural Information Processing Systems (NeurIPS), 2022.
Grace Hui Ting Yeo, Sachit D. Saksena, and David K. Gifford. Generative modeling of single-coll time series with PRESCIENT enables prediction of cell trajectories with interventions. Nature Communicatiins, 12 (1):3222, 2021.
Qinsheng Zhang and Yongxin Chen. Path integral sampler: stochastic control approach for sampling. International Conference on Learning Representations (ICLR), 2022.
Jun- Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-inage translation using cycle-consistent adversarial networks. International Conference on Computer Vision (ICCV), 2017.
A Proofs of theorems
Theorem 3.1. The marginal vector field g
generales the probubilily path 6 from inilial conditions po(x).
Proof of Theorem 3.1 To verify this, we first chock that Pt and u, satisfy the continuity equation.
We start with the derivative w.r.t. time of 8
p()) dt (((ala)a())dz
18

---

Published in Transactions on Machine Learning Research (03/2024)
by Leibniz Rule,
= 5 dt
d (pr(xlz)a(2)) dz a
since uc(-12) generates pr(-)z),
= f-f div (u((al=)pp(()())(())) dz
exchanging the derivative and integral,
-di (fulalo)mmoot)))ee)
Using 9
div (wt (2))
satisfying the continuity equation (()( + div (#x())p((z)) = 0.
Theorem 3.2. If (x())) > 0 for all E TRd and & E [0, 1], then, up to constant independent of 0, LCrM and LyM are equal, and hence
VoLPm = VoLcrM(O).
(11)
Proof of Theorem 3.2 For this proof we need and the existence and exchange of many integrals. As in Lipman et al. (2023) we assume that 4, () that Uy ve, Vow are bounded.
2) are decreasing to zero at sufficient speed as |a||| - oo and
V V (0 ue(xx)]1== = VoBp.(e) (I/u0(,,=)]1= (va(4, ux + (1x (x))1²)
Vol (11v0(1,)))1 -2 (x,
Vo glx) (00l)))) (=]=))]]
V VoEg(s) (1)v0 x)| (vo(t, ue (x(===)) + ||ll (=|=)|]
- Vo (1t 2 (vo x), (x(x))))
By bilinearity of the 2-norm and since u, is independent of 0. Next,
E (tt,,)lloot, - / (1o((t=))ll
// (oa(==))((x(aa)))(())ddd (et)))m
Finally,
f (u(lal))p((1l)))((d pe(x))dx
Ep((s) (vo(t, ur(x)) = 00 (4, x), Pe(xr)
/ vol(, (u(ol))))naal)))()))))))))) dx
l] (vo u.) pe (x
(x))))
(g(=).pe =) (ve x), (x(2))
Where we first substitute g then change the order of intogration for the final equality. Since at all times the gradients of LFM and LCrM are equal, (OLFM(() VoLCFM
Proposition 3.3. The marginal Pt corresponding to q(z) ((z)))(21) and the (x(2))), (x(x2) 15 has boundary conditions Pı qı *N(x 0,02) and Po qu * Nx 0, 02),,, where denotes the convolution operator.
(14 and
19

---

Published in Transactions on Machine Learning Research (03/2024)
Proof of Proposition 3.5 ((z))g((11)
We start with
to show the result of the lemma. We note that g(z) q((20,
p((x) / (((zlz)g(z)dzz
/ N(x (tx1 + (1 (1--)aa 1)x0, o² )a((ro, x ei))d(2o.2)) x1)
/| ((x (1 1))o,, o²) )g(zz))))))))) (1))))(( )drodæ
evaluated at i == 0, respectively. Therefore, al = 0,
// N(x Iro, o²) (2)
po(
/ Nix (zo, o² (200))d
(0x)) (2x
This is also true for / t= 1.
Proposition 3.4. The results of Proposition 3.3 also hold for q(2) in (17. Furthermore, assuming regularity properlies of go. Q1, and the oplimal transport plan т, as o² - 0 the marginal path pt and field minimize 0 d.e. un solves the dynamic optimal transport problem between and q1
Proof of Proposition 3.4 We will assume certain regularity conditions on qu, U1 and प allow reduction known results. We leave to future work to determine which of these conditions are necessary and which are redundant with other conditions. However, because are concerned with approximation of u, (x()) with neural networks, which are typically smooth, results that relax the regularity assumptions may be vacuous in practice.
Preliminaries. We assume that go and qi are compactly supported and admit bounded densities with respect to the Lebesgue measure. Then the conditions for Brenier's theorem (Brenier 1991 are satisfied. By Brenier's theorem, the optimal joint и is unique and is supported on the graph (x)) of a Monge map T Fd + Rd and (z(): is equal to McCann's interpolation (Peyré & Cuturi 2019
Chapter
Ps = ((1- + #Po
(22)
In addition, know that T(x) can be parameterized the gradient of convex function, i.c., T(x) (z)
This characterization of T implies that the conditional probability paths, given by Φx(x) = ((7 x)
do not cross, i.e., (x(x) 2o. T(xo)) = (x( for all (t, x) It is known that the probability path ps = 10 #
and its associated vector field, given by u((oe(x)) = T(x) x, solve the optimal transport problem [enammou & Brenicr 2000 Proposition 1.1).
We assume that the induced marginals Pt have bounded densities with respect to Lebesgue measure and that T is almost everywhere continuous in x, which implies the same for dr. Injectivity of o, and nonerossing of paths implies (x)) is almost everywhere continuous in (1().
Formal statement of the result. Denote by pf (x), pf (æ/²) the densities in Proposition 3.3 and by p& (z). We will show that for plammoost every a and almost every t,
E (x|2)u((x1z)
uc (x) lim ux 7-
lim 0-50
(23)
po (x)
4Proof: If the paths from distinct ag and zy cross, so or(()) di (2/)) then (1 f)z (Vv(2z) = t)ag (2))))))
Taking dot product with zo - -agy (t 1> - 26 12 (Vo v. 26)- However, we have (t 1) - =/ 1 <0
and ((Vo(ao) Vw(sz)) mo =/) 2 by convexity v,, contradiction.
20

---

Published in Transactions on Machine Learning Research (03/2024)
Proof. It suffices to show the equality for in the support of pt, or in the image of the support of do under Φe. We identify in the expectation with a pair (zo, ((x)) due to (1) a (20, 21) having support on the graph of the Monge map. Noting that
Ue (x((2ro T(xo)) uo(x))) T(xo) 20 Vx,
we see that 23 equivalent
E of [2( (T( ((oo)) uo (20)
uo (006 lim
(24)
pi (x)
By the same argument as in the proof of Proposition 3.3 over To the conditional equality pe (zo, T(xo)) -(( 6gc(m0) (.((0,0))
we have that pi = Pt *N((0, by integration
Symmetry of the Gaussian implics that
(=) ((xo)) = PP(ool))))((
Therefore
Eg (w)( ((x))))u By(so) (d) ([ [o( [by change of variables x' = (x0)) pp(())) (x') (x), (x), (o;
(x
= (w) (+)))) (+))) [pt (x (x'))
Esz~N10 (pr( + uo) (0) (x Az))
= (pp())uo(o) (-)) * N(0,0²) (z).
The standard fact that NN -- & in distribution implies that f bounded, almost everywhere continuous, compactly supported function, then (f (x)x)) > f(x) pointwise for almost every x. By the hypotheses, Pt and pt (uo 1) have this property. It follows that, for every and almost all x,
lim E P9 ( (nro T(xo))uo(xs) lim ((p (no t,, (V,, (x)
p8 (x)
(pe* * *n*N(0,0²2)) (x)
(pr (no o dr ())
Pr(x)
= =oo (x)),
which proves 24
Proposition 3.5. The marginal vector field ue(x)) defined by probability path as the solution T² to the SB problem in 18
19 and
21
generates the same marginal
Proof Proposition 3.5 Using Theorem 2.4 of Léonard
2014a De Bortoli et al. (021 showed that the
initial and terminal marginals of n² are the solution to the static OT problem
(Zo, x1) arg min KL(TT" (Z""" x1 (2))
while the conditional path distributions "" I2o, 21) minimize
E Eng Pro (o (TT (- (2o, x1) |pret( 2o,,, æ1)).
The optimization problem for n" (zo, x1 equivalent the entropy-regularized optimal transport problem with optimum T2r2, observed by De Bortoli et al. (2021). (The key observation is that log (ref 21) e(xa.21)" + const., where c(x, y) = || - y/ and a = 2.) The divergences between conditional path distributions are optimized by Brownian bridges with diffusion scale a pinned at xy and 21, which are well-known have marginal probability path pt in 20 and, by 5)) are generated by the vector fields in 21
-0
21

---

Published in Transactions on Machine Learning Research (03/2024)
B Additional theoretical results
Proposition B.1. For any a E R.+ conditional flow matching with conditional probability paths given by 160 has an equivalent marginal probability flow pe(z) to Lipman et al. (2023) flow matching.
Proof. To prove the proposition, we use the fact that the Gaussian family can be generated by location-scale transformations (see, e.g., Lehmann & Casella 2006), i.e.; we can express any Gaussian Zo NNo, og) Zo No on where Z (O, f). Recall that the density pe(x) has the form pc(x) pe (=( to show the equivalence between the flow from FM and source conditional flow matching, that Pt (x] is the same for both methods, that is we show that CFM with variance (at)² + 2at(1 t) is equivalent to FM with variance (la - + 1)2 112 Since (r 21) is N(tx + (1 - 1)00); allal 21 + 2)) we can write the random variable Xo X1 as
have to show
X|Xo, X1 fx + (1 () (ot(at- - 21 22) 2))
where Z ~ I). Without conditioning on Xo, we have
X|X1 = tx + (1 t)Xo + (ot(at - 21 2))
By assumption Xu (N I), thus X|X1 is Gaussian, since linear transformation of Gaussian distributions is also Gaussian. To define its distribution, we only have define its expectation and variance. By linearity of expectation, we find ( = tæı, and by independence of Xo and Z we have
(ar(X = -1 (Xx)) + (al(at - 21 + Va)
= -) + (al(at - 21 + 2)) = (lo 1+ 1)²,
hence the flow from source conditional flow matching is the same as FM.
Proposition B.2. If is a Monge map, the objective variance of OT-CFM goes to zero as -+00 0, i.e.,
By(c) (((c)))((<(1)) ur (x)]11 0 0
for ue (x)2))
15
Proof. This follows from basic fact about the transport plan m. Specifically, that as o, DKL (K (I)2")]l(<<< oc for an t, for two distinct zi, 2.. This means that px(z))) pr(x) for any 1, therefore
(x)) = (xcss = =<(l22
Proposition B.3. The conditional vector field (x(lz) defined by 26 converges to marginal vector field uxxx) defined by goes to population size, i.c.,
[[/]) -1((x)11 +0
as m + |X)..
Proof. As |z| (2), by definition,
Σn u1(212 (x)22)))
We (x(2))
Σn p((2)=))g(=1))
Σ Exex (1(3)))))))
Ue (x(2))))
Ey(z)
(x(
= (x(x)
22

---

Published in Transactions on Machine Learning Research (03/2024)
C
Algorithm extensions
presented the general algorithm for conditional flow matching given q(2), pc(2)2), uz(z]z). In Table 1 presented number of settings of these leading to interesting probability paths. In practice, may wish to compule g(=) on the fly. Therefore in Alg. 2 Alg. 3 and Alg. 4 we give algorithms for the simplified conditional flow matching, and minibatch versions of OT conditional flow matching and Schrödinger bridge conditional flow matching. In general these consist of first sampling a batch of data from both the source and the target empirical distributions, then resampling pairs data either randomly (CFM) or according to some OT plan (OT-CFM and SB-CEM).
In Alg.
C.1 Variance reduction by averaging across batches
An interesting consequence of introducing optimal transport to conditional flow matching is that it greatly reduces variance of the regression target. Informally, as a -> 0, E.z,,=Iluu (x|]) (=))]) - for OT-CFM
and SB-CFM, which is not true of previous probability paths in (Tblle (See Proposition B3.2 for precise statement). As flow models get larger, more powerful, and more costly, reducing objective variance, and thereby faster training may lead significant cost savings [Watson al. 2022b). To this end we also explore reducing the variance of the objective by averaging over batch. This is not feasible in score matching where the flow condit ioned on multiple datapoints is complex. Our CFM framework naturally extends from pair of datapoints to batch of pairs. Instead of conditioning on single pair of datapoints we can condition on a batch of pairs. As the batch increases in size, we trade higher cost in computing the target for lower variance in the target as the batch size increases, the variance in the target goes to zero (see Proposition B.3 for a precise statement).
As formalized in Proposition B.3 we can reduce variance in the larget by averaging over multiple datapoints. Specifically, in this case we let = {2' == (z), km where of are i.dd. from q(2) and
(=(#) Dx (r(x )((=))))))))))
(25)
Σn (221)
(x(x|z) Σn u1(2 (2(2)))(())) pe (112))
(26)
It takes roughly m times as long to compute the conditional target u, (x(=)) but reduces the variance. As the evaluation and backpropagation through vo gets more difficult this tradeoff can be beneficial.
C.2 Modeling energy functions
If we have access to an energy function two (unnormalized) energy functions Rqu.11 Rd + R+ at the endpoints instead of ii.d. samples X, ~ (z,), then the objective must be slightly modified. We formulate the Energy Conditional Flow Matching Objective as
(o( [100(1,))) u. (1r x )1 (o(xn) (x1)
LECFM -BB.gg(eenssd((e)))ae.eelaa
(27)
We can use this object to train a flow which matches the energies without access samples. This is formalized in the following theorem.
Proposition C.I. Assuming that 93o,13 (1), (x),) pe(x)) for all x E X and t [0, 1] then the gradients of Lrm and LECFM with respect to 0 are equal up to some multiplicative constant c.
VoLFM cV
(28)
23

---

Published in Transactions on Machine Learning Research (03/2024)
Algorithm Simplified Conditional Flow Matching (I-CFM)
Input: Empirical samplable distributions qo: 91 bandwidth o, batchsize b, initial network vg- while Training do
/* Sample batches of size b i.i.d. from the datasets
~ qu 21 ~ (x11
~ 1/(0, 1)
Nr - tæj + (1 t)
x ~ 0²1)
(0) + |luo(t,)) =x))l
0 - Update(0, VoLCFM(())
return 0g
Algorithm Minibatch OT Conditional Flow Matching (OT-CFM)
Input: Empirical or samplable distributions qo: 91, bandwidth a, batch size b, initial network vo- while Training do
/ Sample batches of size i..d. from the datasels
3o ~ (o())):
x1 ~ (x11
и < (T( no)
(wo: x1) ~T
U(0,1)
Me tæ + (1 t)
a ~ N(p,,
(0) + [uu(t, -2)1=
+ Update(0, VoLCEM(8))
return tgg
Proof. Let 20 = fx Ro(x)dr, and 21 = Sx (1(x)de then g(zo) = Ro(xo)/20, similarly g(œ1) R1(x1)/211
then
Ro R1 (I1)
LECFM = E, (o
[1, (x/220 a )l
(29)
(o( (x1)
20 ((00))((1))
z0 Ee,
(t, we (x/20, x 1l3
(30)
(2)0)
=z0ž1
(2) un (x/2oo 2n 112 pe pr æ )dæoda de
(31)
20 (0)
(32)
where we use substitution for the first step and change the order of integration in the last step. With an application of Theorem 3.2 the gradients are equivalent up to a factor of 2g21 which does not depend on x.
Of course CECFM leaves the question sampling open for high-dimensional spaces. Sampling uniformly does not scale well to high dimensions, so for practical reasons we may want a different sampling stratogy.
We use this objective in ig. D.9 with uniform proposal distribution toy example of this type of training.
D Additional results
We start this section by the definition of the entropy regularized OT problem:
W(g,, 11)2 = inf /
clz, 2 xxxx, dy) AH(TT),
(33)
mxEll
x2
24

---

Published in Transactions on Machine Learning Research (03/2024)
Algorithm Minibatch Schrödinger Bridge Conditional Flow Matching
Empirical samplable distributions qo: 91 bandwidth a, batch size b, initial network vo- while Training do
Input:
/* Sample batches of size b i.i.d. from the datasets (x)): 21 ~ (x1) 2 < Sinkhorn(m) 1 mo, 20²)
(2o: x1) ~ T2o2
U(,,1)
Pe tæ + (1 t)
(u,, t(1 I)
ux(() + 2x[-]] (x (t1) + (1
1-
(1--tay ()xo)) + (x1 20)
From 21
(0) + (t) 2x(222)11
- Update(0, (OLCFM(0)
return 0g
Dataset gaussians
Dataset
Dataset
4.0
5
4.5
Metric Norma lized 2-Vasserstein
3.5
3.5
3.0
3.0
2.5
2.0
2.0
2-
1.5
1.5
1.0
Method
Optimal
25
Matric = Normali lized Path Energy
CNF
4-
3
15
salee
2-
10
1
10-1
10
10-3
10-1
10
10-2
10a1
10a
Squared L2 Reg
Squared L2 Reg
Squared L2 Reg
Figure D.1: Evaluation of regularization strength de over seeds in the range [0, 10-5 performs the bost in terms of minimizing path length and test crror. We call this model 'Regularized CNF'.
109]. A = 0.1
where À E + and H(x) f In (2, ydx dy)
Regularized CNF tuning Continuous normalizing flows with path length penalty optimize relaxed form of dynamic omtimal transport problem Tong et al. 2020: Finlay et al. 2020 Onken et al. 2021 Where dynamic optimal transport solves for the optimal vector field in terms of average path length where the marginals at time = 0 and = 1 are constrained to equal two input marginals yo and Qı- Instead of this pair of hard constraints, regularized CNE's instead set qo = N(x 0, 1) and optimize a loss the form
L(x(t)) = log p(g(((())) + de S x(t))l
(34)
25

---

Published in Transactions on Machine Learning Research (03/2024)
Table D.1: Mean training time till convergence in 10s seconds over 5 seeds, with the exception of DSB, trained over scod. CFM variants and DSB are trained on a single CPU with 5GB of memory where other baselines are given two CPUs and one GPU, CFM, with significantly fewer resources, still trains the fastest.
WV-8gaussians
moons >8gaussians
M->moons
N-+scurve
mean
OT-CFM
1.284 + 0.028
1,587 + 0.204
1.464 + 0.158
1.499 + 0.157
1.484 = 0.192
CFM
0.993 = 0.021
1.102 + 0.171
1.059 + 0.158
1.008 + 0.106
1.046 0.132
FM
0.839 = 0.096
1.076 = 0.126
1.127 = 0.123
1.014 0.170
SB-CFM
0.713 + 0.386
0.794 + 0.293
1.143 + 0.389
1.230 + 0.424
0.935 + 0.397
Reg. CNF
2.684 + 0.052
9.154 + 1.535
9.022 + 3.207
8.021 3.288
CNF
1.512 = 0.234
17.124 + 4.398
27.416 = 13.299
18.810 + 12.677
ICNN
3.712 + 0.091
3.046 + 0.496
2.558 + 0.390
2.200 + 0.034
2.912 0.626
DSB
5.418 = 5.682 +
5.428 + 5.560 +
5.522 =
where
de = =vo((xx(()) and logp(a(T)) is defined as di
logp(x = p(x(0)) + [."
T @logp(x(t)) dt dt p(=()) + f-= (tr(dun dug dt dz((t
-tr
(35)
Ot
where the second equality follows from the instantaneous change of variables theorem (Chen et al. 2018
Theorem 1). In practice it difficult to pick a de which both produces flows with short paths and allows the model to fit the data well. We analyze the effect of this parameter over three datasets in Fig. D.1 In this figure analyze the Normalized 2- Wasserstenn to the target distribution (which approaches 1 with good fit), and the Normalized Path Energy (NPE). We find tradeoff between short paths (Low NPE) and good fit (Low 2- Wasserstein). We choose A 0.1 as good tradcoff across datascts, which has paths that are not too much longer than optimal but also fits the data well.
Ablation results on batch size. Since we use Minibatch-OT for OT-CFM, when the minibatch size is equal to one, then OT-CFM is equivalent to CFM. This effect can be seen in Fig. D.2, where over four datasets, OT-CFM starts with equal path length and approximately equal 2- Wasserstein. Then the normalized path energy decreases surprisingly quickly plateauing after batchsize reaches ~64 While the minibatch size needed to approximate the true dynamic optimal transport paths will vary with dataset (for example in the moon- -oon--gaussian case we need a larger batch size) it is still somewhat surprising that such small batches are needed as this is less than 0.5% of the entire 10k point dataset per batch.
The effect of a on fit and path length. Next we consider o, the bandwidth parameter of the Gaussian conditional probability path. In Fig. D.3 we study the effect of a on the fit (top) and the path energy (bottom). With a > 1 mothods start underfit with high 2- Wasserstein error and either very long or very short paths. As for specific models, SB-CFM becomes unstable with too small due to the lack of convergence for the static Sinkhorn optimization with small regularization. FM and CFM follow similar trends where they fit fairly well with a < 1 but have paths that are significantly longer than optimal by 2-3x. OT-CFM maintains near optimal path energies and near optimal fit until > 1.
Schrödinger bridge fit over simulation time. In Fig. D.7 we compare the fit of Diffusion Schrödinger Bridge model with SB-CFM conditioned on time. The Diffusion Schrödinger Bridge seems to outperform SB-CFM early in the trajectory, however fails to fit the bridge after many integration steps.
D.1 Objective variance.
We consider the variance of the objective (x(2)) with respect to z. While for any x we have (z) = u<x)), we find a lower second moment spoeds up training. Specifically, we soek understand the effect of the second moment which we call the objective variance defined
OV E (xxp (l (x))))11
(36)
26

---

Published in Transactions on Machine Learning Research (03/2024)
Dataset gaussians
Dataset moons
Dataset scurve
Dataset moon agaussians
1.6
0.45
0.50
1.4
0.45
2.50-
W
v2lue
0.40
2.25
0.35-
2.00
0.25
1.75
0.20
Method
0.8
OT-CRM CPM
1.0
3.0
Optimal
0.20
0.0
2.0
Normalized Energy
0.6
value
155
0.10
0.4
0.4
0.2
0.5
0.00
0.0
1024
64
256
1024
1024 Betchsize
256
64
250 1024
Batchsize
Batchsize
Batchsize
Figure D.2: a t of mean path length prediction error seeds. Lower is better. Introducing OT to CFM batches straightens paths lowering cost towards the optimal W2 compared to standard random conditional flow matching network over all batch sizes.
Dataset moons
Dataset scurve
Dataset gaussians
10
Metric - Norma cormalized lized 2-Wasserstein
10P
Method
SB-CFM
10
OT-CFM
10a
Metric Normalizer lzed Path Energy
CFM
Optimal
109
10"4
10"
10
10a
Figure D.3: Evaluation of the effect of for conditional flow matching models. When < 1 OT-CFM outperforms the other flow matching methods. SB-CFM drops off performance when a is too small due to numerical issues in the discrete Sinkhorn solver.
27

---

Published in Transactions on Machine Learning Research (03/2024)
Dataset gaussians
Dataset moons
Dataset scurve
Dataset moon-8gaussians
Estinated Objective Variance
101-
10a
10
CFM
SB-CFM
OT-CFM
SB-CFM
OT-CFM
SB-CFM
OT-CFM
SB-CFM
Method
Method
Method
Method
Figure D.4: Estimated Objective Variance datasets. OT-CFM and SB-CFM have significantly lower objective variance than CFM and FM which have roughly equivalent objective variance.
36 for different methods with batch size 512, 0.1 across
4.5
Method
OT-CFM
4.0
CFM
2-Wasserstein
3.5
Batchsize
3.0
512
2.5
2.0
0
100
200
300
400
Time (s)
Figure D.5: Validation 2- Wasserstein distance against training time with variance reduction by aggregation either with aggregation (Batchsize 1) or aggregation over minibatch (Batchsize 512). Variance reduction leads to faster training, especially for CFM where the objective variance is naturally larger than OT-CFM which sees small performance gain.
Dataset scurve
moons
gaussians
maon-8gaussians
0.9
0.7
3.5
0.7-
0.5
0.8-
OT-CFM
FM
0.6
2.0
SB-CFM
0.4-
1.5
20
40
20
26
20
40
20
40
epoch
epach
epach
epoch
Figure D.6: Exlended results from fastest converging method.
F ig 2
(left) over two more datasets. OT-CFM is still consistently the
28

---

Published in Transactions on Machine Learning Research (03/2024)
gaussians
Dataset moon Egauzsians
moons
scurve
0.8-
2.5
0.5-
0.5
1.5
0.4
Mathod
CRM
1.0-
0.3-
0.3-
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.2
0.4
0.6
0.8
Time
Time
Time
Time
Figure D.7: 2- Wassersseen Error between trajectories and ground truth Schrödinger Bridge samples over simulation time.
4.5
Method
25
Method
OT-CFM
Objective Variance
20
OT-CFM
Wasserstein
4.0
CFM
Batchsize
CFM
3.5
FM
3.0
512
10
SB-CFM
2.5
5
2.0
0
4 6
8
10
100
200
300
400
500
batchsize as % of data
Step
Figure D.8: (left) Variance of the objective for varying batch size. OT-CFM has lower variance across
batch sizes. (right) Validation Wasserstnnn performance with batch averaging in SC.1 Reducing variance improves training elficiency.
on training speed for different objectives Table We estimate the variance on a small data with a known (x)). We examine this estimated objective variance and its effect on training convergence in Fig. D.8 showing that either OT-CFM variance reduced CFM with averaging
the batch results in lower variance of the objective. This in turn leads to faster training times as shown on the right. Averaging over batch of data leads to faster training particularly for methods with high objective variance (CFM) and less so for those with low (OT-CFM), which already trains quickly.
Variance in the conditional objective target (x(=)) varies across models. In Fig. D.4 we study the objective variance across CFM objective functions. Here we estimate the objective variance (36) as
Br (x. vo =1,
(37)
after training has converged. After training has converged ds should be very close to ux(x) so we use as an empirical estimator of (x(x) compute the variance. We find that across all datasets OT-CFM and SB-CFM
have least an order of magnitude lower variance than CFM and FM objectives. This correlates with faster training as measured by lower validation error in fewer steps for lower variance models seen in Fig. 2 (left).
We examine the objective variance OV by conditioning (x(2)) on batch of pairs of data points, E == { = (#f, of )mm, we can reduce the variance of the OV objective to 0 for all models as batchsize goes to population size. For the batchsize m range from 1 to the number the population, we uniformly sample m
pairs of points and compute the probability (x( and the objective un (=(xz) from 25 and 26
We also find that averaging over batches makes the network acheive lower validation error in fewer steps and in less walltime (Fg.. D.5
29

---

Published in Transactions on Machine Learning Research (03/2024)
Samples
Energy
((xx) = U-- 7)2
Figure D.9: Flows (green) from (a) moons to (b) 8- G-Gaussians unnormalized density function learned using CFM with RWIS.
D.2
Energy-based CFM
We show how CFM and OT-CFM can be adapted to the case where we do not have access to samples from the target distribution, but only unnormalized density (equivalently, energy function) of the target, R(x1) Fig. D.9.. We consider the 10-dimensional funnel dataset from Hoffman & Gelman (2011). We aim to learn flow from the 10-dimensional standard Gaussian to the energy function of the funnel. We consider two algorithms, each of which has certain advantages:
(1) Reweighted importance sampling (RWIS): We construct a weighted batch of target points æ1 by sampling 31 IN, and assigning a weight of R(z1)/N(z1; 0, normalized sum to over the batch. The FM and CFM objectives handle weighted samples in trivial way (by simply using the weights ((x1) in Table Ip, while OT-CFM treats the weights as target marginals in constructing the OT plan between To and 21. We expect RWIS to perform well when batches are large and the proposal and target distributions are sufficiently similar; otherwise, numerical explosion of the importance weights can hinder learning.
(2) MCMC: We use samples from a loug-run Metropolis-adjustee Langevin MCMC chain on the target density as approximate target samples. We expect this method perform well when the MCMC mixes well; otherwise, modes of the target density may be missed.
As an evaluation metric, we use the estimation bias of the log-partition function using a reweighted variational bound, following prior work that studied the problem using SDE modeling (Zhang & Chen 2022 The computation of this metric for CNIss is given in SE.6
The results are shown in Table D.2 When an adaptive ODE integrator is used, all algorithms achieve similar results (no pair of mean log-partition function estimates statistically distinguishable with p < 0.1 under a Welch's (-test) but OT-CFM is about twice as efficient as CFM and FM. However, with a fixed computation budget for ODE integration, OT-CFM performs significantly better.
E Experiment and implementation details
E.1 Physical experimental setup
All experiments were performed on shared heterogenous pigor cluster, This cluster is primarily composed of GPU nodes with RTX8000, A100, and V100 Nvidia GPUs. Since the network and nodes are shared, other users may cause high variance in the training times of models. However, we believe that the striking difference between the convergence times in Table D.I and combined with the CFM training setup with single CPU and the bascline models trained with two CPUS and GPU, paints clear picture as to how efficient CFM training is. Qualitatively, we feel that most CFMs converge quite bit more rapidly than these metrics would suggest, often converging to near optimal validation performance in minutes.
30

---

Published in Transactions on Machine Learning Research (03/2024)
Table D.2: Energy-based CFM results on the 10-dimensional funnel dataset: log-partition function estimation bias (mean and standard deviation over 10 runs) and time to generate 6000 samples from the trained ODE. With adaptive integration, OT-CFM requires fewer function evaluations. With fixed-interval solver, OT-CFM has lower discrctization error, leading to better estimate. PIS bascline is from Zhang & Chen
2022)
RWIS
MCMC
log& 2
time
log 2
time
adaptive Dormand- Princee (tolerance 0.01) integration
CFM
0.068 = 0.041
26.6 8.4s
0.029 + 0.037
34.6 + 6.0s
OT-CFM
0.076 + 0.098
13.3 + 1.7s
0.009 + 0.045
12.8 + 1.2s
FM
0.003 + 0.057
26.5 7.7s
0.027 + 0.031
30.9 5.8s
Euler (N = 10) integration
CFM
0.281 = 0.202
4.0 + 0.8s
0.336 + 0.030
3.7 = 0.7s
OT-CFM
-0.039 = 0.030
4.2 = 0.6s
0.146 0.107
4.1 0.8s
FM
0.176 0.044
4.1 + 0.7s
0.334 + 0.066
3.9 + 0.6s
Euler- MaruyaMaruyama (N 100) integration
PIS (SDE)
-0.018 + 0.020
E.2 2D, single-cell, and Schrödinger bridge experimental setup
For all experiments we use the same architecture implemented in PyTorch Paszke et al. 2019 We concatenate the flattened input x gd and the time as the d inputs to a network with three hidden layers of width 64 interspersed with SELU activations (Klambauer ct al. 2017) followed by lincar output layer of width d. This forms our vy for all experiments. For all 2D and single-cell experiments we train for 1000 epochs and implement early stopping on the validation loss which checks the loss on & validation set every 10 epochs and stops training if there is no improvement for 30 epochs. We also set a time limit of 100 minutes for each CFM model. This hit almost exclusively for SB-CFM models with small a which are unstable to train due to instabilities and non-convergence of the Sinkhorn (Cuturi 2013) transport plan optimization. We use the AdamW (Loshchilov & Hutter 2019) optimizer with weight decay 10-5 with batchsize 512 by default in 2D experiments and 128 in the single cell datasets. For OT-CFM and SB-CFM use exact linear programming EMD and Sinkhorn algorithms from the python optimal transport package (Flamary et al. 2021) For evaluation of trajectories unless otherwise noted we use the Runga-Kutta45 (rk4) ODE solver with 101 timesteps from 0 to 1.
E.3 Variance reduction by averaging
We tackle the exploration of the effects of reducing variance of the target ux(x22) from two directions. The first is for small example where we can compute the ground truth (x(x) quickly, and the second is in the setting of trained models where we can estimate ux(x) with v0(t, after Uo has converged.
We first consider the convergence of each flow matching objective (OT-CFM, CFM, FM, SB-CFM) to zero function of the batch size relative to the dataset size. This is done by first sampling t, then computing the true objective variance across many samples. This appears in Fig. D.8
We next consider the effect of averaging over a batch to reduce the variance of the objective in Fig. D.8 (right). Here Batchsize refers to the size of the batch we are averaging over. We aggregate this into a single target so that the model sees single dimensional target vector for one sampled x, This means that we can compare different aggregation sizes fairly.
31

---

Published in Transactions on Machine Learning Research (03/2024)
E.4 Schrödinger bridge evaluation setup
To evaluate how well Schrödinger Bridge models actually model a Schrödinger Bridge, we constrain ourselves to a small example with 1000 points. We note that the closed-form Schrödinger marginals are known for discrete densitics, for Gaussians (Mallasto et al.] 2022), and can be constructed for two approximate datasets Korotin et al.) 2021), which present other ways of evaluating Schrödinger bridge performance. For any time we can sample from the ground truth Schrödinger bridge density pt(x)) as
(ro, T262
X, N(c (x1 + (1 (0(1)))))))))))))) -))
We sample trajectories of length 20 from f = to t= 1 by integrating over time from / t= 0 to L == 1. At each of the 18 intermediate timepoints we compute the 2- 2-aassrsteiin distance between a sample of size 1000 from the trajectorics at that time and the ground truth X, as above at that time. We reported the average across the 18 intermediate timepoints in Table 3 and plot the 2-Wasserstein distance over time in Fig. D.7
SB-CFM Model We train SB-CFM with a 1 and batchsize=512 for each of the datasets. We save 1000 trajectories from a test set integrated with the tsit5 solver with atol=rtol=le-4.
Diffusson Schrödinger bridge model implementation details We use the implementation from De Bor- toli et al. (2021). Only the networks were changed for a fair comparison with CFM. The forward and backward networks are composed of an MLP with three hidden layers of size 64, with SELU activations in between layer. We used a time and a positional encoders composod of two layers of size 16 and 32 with LeakyReLU aclivations has inputs to the score network. The architectures are the same for the 2D examples and the single-cell examples (except for the input dimension). During training, code) to 0.001 and did 20 steps to discretize the Langevin dynamic. We trained for 10k iterations with 10k particles and batch size of 512, for 20 iteralive proportional fitting steps, and a learning rate set to 0.0001. For the interpolation task we used the tenth timepoint from the Langevin dynamic with the backward network trained to go from the distribution at time t to t + All trajectories are evaluated from the backward dynamic. We use o 1 and batchsize=512.
set the variance (y in the author's
E.5 Single-cell experimental setup
We strove to be consistent with the experimental setup of Tong et al. 2202 For the Embryoid body (EB) data, we use the same processed artifact which contains the first 100 principal components of the data. For our tests we truncate the first five dimensions, then whiten (subtract mean and divide by standard deviation) each dimension. For the Embryoid body dataset which consists of 5 timepoints collected over 30 days we train separate models leaving out times 2, in turn. We train CFM over the full time scale (0-4). During testing we push forward all points X,-1 to time f as a distribution to test against.
For the Cite and Multi datasets these are sourced from the Multimodal Single-cell Integration challenge at NeurlPS 2022, a NeurIPS challenge hosted on Kaggle where the task was multi-modal prediction (Burkhardi) let al.) 2022). In this competition they used this data to investigate the predictability of RNA from chromatin accessibility and protein expression from RNA. Here, we repurpose this data for the task of time series interpolation. Both of these datasets consist of four timepoints from CD34+ hematopoietic stem and progenitor cells (HSPCs) collected on days 2, 3, 4, and 7. For more information and the raw data see the competition site 3 We preprocess this data slightly to remove patient specific effects by focusing on a single donor (donor 13176), then we again compute the first five principal components and again whiten each dimension to further normalize the data.
E.6 Energy-based CFM
The 10-dimensional funnel dataset is defined by Xg ~ NN. X1....9 ~ NO, cxp(x))I) We attempted to mimic the SDE model architecture from Zhang & Chen (2022) for the flow model vo(t, x). The time step t is
httpp://www.aagleeecc//// htt.ggg......cccppppttaaa/oa/ppa mat.emmmmmmmmmiiiimmnmmanatett
32

---

Published in Transactions on Machine Learning Research (03/2024)
encoded with 128-dimensional Fourier features, then both and t are independently processed with two-layer MT.Ps. The two representations are concatenated and processed through another three-layer MLP to make the prediction. All MLPs use GELU activation and have 128 units per hidden layer. We trained all models with o = 0.05 and learning rate 10-2, the highest at which they were table, for 1500 batches of size 300, to be consistent with the settings from Zhang Chen (022)).
The importance weighted estimate of the log-partition function is defined
R(a(") Na 00,1) azo
log 2 =log,
20 20=20
(0) where =0 are independent samples from the source distribution and #1 xo pushed forward by the flow (note that the Jacobian can be computed by differentiating the ODE integrator). We used K 6000 samples.
(
For MCMC, to be consistent with Zhang & Chen 2022). we generated 15000 samples, each of which was seen 30 times in training. We used 1000 steps of Metropolis-udjusted Langevin sampling with linearly decaying from 0.1 to 0.
The flow network used to generate ig.. D.9 followed similar settings to those used in $5.1
E.7 Unsupervised translation
We trained a vanilla convolutional VAE, with about 7 million parameters in the encoder, on CelebA faces scaled to 128* x 128 resolution.
For the flow network (t z), we used MLP with four hidden layers of 512 units and Ieaky ReLU activations taking the 129-dimensional concatenation of æ and t as input. All models CFM and OT-CFM were trained for 5000 batches of size 256 and the Adam optimizer with learning rate 10-3 Integration was performed using the Dornand-Prince integrator with tolerance 10-3, For each attribute, 1000 positive and negative images each were used as a held-out test set.
Fig. E.1 shows some examples of the learned trajectorics.
E.8 Unconditional CIFAR-10 experiments
For the CIFAR-10 experiments we followed the setup as described in Lipman et al. (2023). All methods were trained with the same setup, only differeing in the choice of probability path. Since code has not been released for this work, there are few parameters which may differ. We summarize the setup here, where the exact parameter choices can be soen in the source code.
We used the Adam optimizer with 31 = 0.9, $3 0.999, e 10-8, and no weight decay. To reproduce Lipman et al. (023), used the UNet architecture from Dhariwal & Nichol (2021) with channels 256, depth = 2, channels multiple = 2,,,, heads = 4, heads channels = 64, attention resolution = 16, dropout = 0.0, batch size per gpu = 128, gpus = 2, epochs = 2000, maximum learning rate = 5 x 10-4, minimum learning rate = 0, with a learning schedule that increases linearly from the minimum to the maximum learning rate over the first 200 epochs, and decays linearly from back to the minimum after that. We used = 10-6, For sampling, we use Euler integration using the torchdyn package and DOPRI5 from the torchdiffeq package. Since the DOPRI5 solver is adaptive step size solver, it uses different number of steps for each integration. We use batch size of 500 for 100 total batches and average the number of function evaluations (NFE) over batches.
For our improved models we use the same parameters as above except we use channels = 128, dropout = 0.1, a single A100 GPU, stcps=400000, a = 0, constant lcarning rate of x 10-4, gradient clipping with norm = 1.0, and exponential moving average weights with decay 0.9999.
33

---

Published in Transactions on Machine Learning Research (03/2024)
Figure F.1: Image-to-image translation in the latent space of CelebA images: An OT-CNF trained to translate between latent encodings of images that are negative and positive for given attribute. The first column is reconstructed encoding Fy real negative image. The next ten columns are decodings of images along the flow trajectory with initial condition zg, with Ii shown in the right column. Top row: not smiling smiling, not male male, showing the preservation of image structure and other attributes. Bottom row: no mustache + mustache, not wearing necktie + wearing necktie, showing partial failure modes. Both features are well-predicted by the latent vector, but infrequent in the dataset and highly correlated with other attributes, such as 'male,, leading to unpredictable behaviour for out-of-distribution samples and modification of attributes different from the target.
34


Reviews:
[{"review_id": "qsbnJIHj4q", "reviewer": "Reviewer_HJwq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The contributions in this paper are original and the experiments are rather extensive. \n- I found the paper to be well written and everything is explained quite well also. \n\nWeaknesses.\n\nThe weaknesses/questions I see are rather on the practical side:\n- It seems to me that SB-CFM uses the Sinkhorn algorithm in order to sample from the coupling during training whereas DSB tries to learn the coupling. In Table 3 the authors outperform DSB on the 2d example and in Table 4 for the higher dimensional, SB-CFM underperforms wrt DSB. Can the authors explain why this is the case? Can the authors also explain when they expect SB-CFM to perform better than DSB and why? \n- it seems that one of the selling points of dynamic OT for generative modeling is that it allows straighter flows than the traditional diffusion models and hence, one can use fewer steps to discretize the ODE, yielding faster inference. In table 5 the authors perform 100 steps for simulation but this is rather large in comparison to what is currently done with denoising diffusion models. For example, with fewer steps (see [1] for example, although it is quite old and far from state of the art), 50 steps yield good enough performance already. It would have been great to include an experiment in which the authors compare OT-CFM with std Gaussian source with traditional diffusion models and compare the quality of the samples with the same number of steps. Even assuming that somehow OT-CFM allows faster sampling than diffusion models, why is it interesting (for generative modeling) to integrate the ODE if one has already access to the static coupling. \n- I believe that the paper does not accurately reflect the literature. I think that propositions 3.4 and 3.5 are well-known results but the authors do not give any references. Also, the difference with [2], which is very close to this work, is not clear. \n\n[1] Song, Jiaming, Chenlin Meng, and Stefano Ermon. \"Denoising diffusion implicit models.\"\n[2] Aram-Alexandre Pooladian, Heli Ben-Hamu, Carles Domingo-Enrich, Brandon Amos, Yaron Lipman, and Ricky T.Q. Chen. Multisample flow matching: Straightening flows with minibatch couplings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Flow matching models are recent generative models that allow faster and more stable training of continuous normalizing flows, as they do not require simulating the SDE/ODE and the backpropagation through the trajectories. \nThis paper presents a generalization of this framework. Indeed, it shows that it is possible to train flow matching models when only conditional probability paths $p_t(x|z)$, vector fields $u_t(x|z)$ and conditioning distribution $q(z)$ are given. The authors then consider the three relevant applications: \n- An independent coupling of the source distributions: in this case the conditional path is $p_t(x_t|x_0, x_1)$ and $q(z) = q_0(x_0) q_1(x_1)$. The authors propose one example of conditional probability paths and vector fields and their methods allows learning the unconditional vector field $u_t(x_t)$, which is intractable otherwise.\n- Optimal transport: here the condition distribution is the optimal coupling $\\pi$ between the two sources. The authors then notice that the solution of dynamic OT fits exactly their framework; the marginals are $p_t(x_t) = \\int p_t(x_t | x_0, x_1) \\pi(dx_0, dx_1)$ where $p_t( \\cdot | x_0, x_1)$ is known as well as the conditional vector field. The optimal coupling is not known in practice but the authors propose to use minibatch OT to draw samples from the coupling, during training. This is an interesting contribution. \n- Schrodinger bridge problem: Similarly, the authors also note that the marginals $p_t(x_t)$ of the solution to the SB problem are mixture of bridges over the solution to the entropy-regularized OT problem, and the bridges aswell as the vector fields are known. Sampling from the coupling is done using the Sinkhorn algorithm.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors should add the appropriate references next to Propositions 3.4 and 3.5 and discuss in more detail the difference with [2]. The authors should also mention that Theorem 1 is a straightforward extension of the first theorem in Lipman, 2023.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PcvjfV4F5t", "reviewer": "Reviewer_c8zh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nThe approach of presenting flow matching algorithms as conditional on a general variable effectively unifies multiple algorithms within the same framework. The work is polished and well written.\n\n## Weaknesses\nThe novelty of the stated contributions is unclear. I will now discuss the \"summary of the contributions\" with which the authors open the paper.\n1. In the \"Summary of contributions,\" the authors claim to introduce a \"novel class of objectives\" and prove their correctness. This class is defined by conditioning on an abstract variable $z$ rather than $x_0$ or $(x_0, x_1)$ as done in earlier works. However, it appears that all specific examples of this broad class of objectives suggested by the authors have been previously implemented in other articles. Moreover, the results and proofs of correctness in this section offer only marginal theoretical advances over the work cited in [1], essentially substituting $x_1$ with $z$.\n\n2a. The authors claim they address the dynamic OT or Schrödinger problem, having only access to the static solutions to these problems. They present a training objective for calculating the drift of a dynamic OT or Schrödinger map from a static solution. This involves selecting $q$ as the OT or OT-entropic regularized coupling. Nevertheless, given a static solution to the OT or Schrödinger problem, a closed form solution for the dynamic versions exists - by simply interpolating static couplings with geodesics/straight lines or bridges of the corresponding stochastic process. This is also quite cheap to simulate. For generative modeling, the static coupling suffices. When having access to a static OT/OT-regularized coupling, the rationale for training the drift using a neural network or the targeted real-life applications remain ambiguous.\n\n2b. The authors then propose that an approximation to the OT/OT-regularized maps can be achieved by calculating the OT/regularized OT solution only per-batch during the training. While this is an intriguing concept, it has already been introduced in the work referenced in [2]. The authors acknowledge this in the \"Related works\" section, noting that [2] introduces \"closely related\" objectives. However, it seems they are actually identical.\n\nAdditionally, the authors declare in their introduction that their first main contribution is the introduction of a unifying conditional flow matching framework. As previously noted in the Strengths section, this is a noteworthy contribution. However, its utility to the reader is lessened due to the authors' inadequate classification of existing literature within this framework.\nThe authors apply their framework to specific settings of $z$ and $q(z)$. \n- Section 3.2.1 effectively presents the objective from [1] as a particular instance of their own objective.\n- However, in 3.2.2, their description of the objectives in [3] and [4] as \"closely related\" is somewhat ambiguous. This leaves the reader uncertain as to whether [3] and [4] actually represent special cases within the framework they have introduced.\n- Sections 3.3.3 and 3.3.4 overlook the fact that the objective they propose has already been introduced in [2]. This acknowledgment is made only subsequently in the Related Works section.\n\n\n## Minor\n1. The $U(0, I)$ should be a $U(0, 1)$ in equation $(3)$.\n\n[1] Flow Matching for Generative Modeling, Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, Matthew Le\n\n[2] Multisample Flow Matching: Straightening Flows with Minibatch Couplings, Aram-Alexandre Pooladian, Heli Ben-Hamu, Carles Domingo-Enrich, Brandon Amos, Yaron Lipman, Ricky T. Q. Chen\n\n[3] Michael S. Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants\n\n[4] Qiang Liu. Rectified flow: A marginal preserving approach to optimal transport. arXiv preprint 2209.14577,\n2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work proposes a modification to the flow matching algorithm, leading to a coupling which is closer to an optimal transportation coupling.\n\nThe flow matching algorithm for arbitrary source and target distributions works by optimizing the loss\n$$\\mathbb{E}[\\|v_\\theta(t, x_t) - u_t(x_t | x_0, x_1)\\|^2],$$\nwhere $u_t$ is often chosen to be proportional to $x_1 - x_0$. The distribution over $(x_0, x_1)$ is often chosen to be an independent coupling of $q_0$ and $q_1$ for simplicity. As often done so, the objective is optimized by subsampling minibatches from $q(x_0)$ and $q(x_1)$ to approximate the above loss.\n\nThe authors propose to calculate an optimal transport or entropic regularized optimal transport coupling for each of the minibatches to intuitively get straighter lines.\n\nThe authors suggests an improvement to the flow matching algorithm, with the goal to learn a coupling that is closer to an optimal transportation coupling.\n\nThe training part of the flow matching algorithm, when applied to various source and target distributions, works by minimizing the loss\n$$\\mathbb{E}_{q(x_0, x_1)}[\\|v_\\theta(t, x_t) - u_t(x_t | x_0, x_1)\\|^2],$$\nThere are different choices for $u_t$, but a very common one is to set it equal to $x_1 - x_0$. \nThe distribution $q(x_0, x_1)$ is often chosen as independent coupling of $q_0$ and $q_1$. The optimization is implemented by sampling minibatches from $q(x_0)$ and $q(x_1)$.\n\nOne key contribution of the authors is their suggestion to compute either an optimal transport or entropically regularized optimal transport coupling for each minibatch. This method is proposed with the intuition that it would lead to more direct, linear paths.\n\nFurthermore, the authors set up a framework in which they can express multiple related flow matching algorithms, by conditioning on a general variable $z$ which will either be only $x_1$ or $(x_0, x_1)$.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors state two main contributions, the first being related to unifying existing literature, the other one related to solving the optimal transport problem. I discussed these claims in the Strength's and Weaknesses sections. The Weaknesses stated there need to be addressed. \n\nSpecifically, the paper needs to clarify the novelty of the introduced algorithms. Where similar algorithms exist or when theoretical results are minor generalizations of existing findings, such instances should be prominently noted where these algorithms or theoretical aspects are presented in the paper. Additionally, the framing of these contributions should be appropriately toned.", "broader_impact_concerns": "I do not perceive any specific ethical implications of this work that would require a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GCcwQo4IeO", "reviewer": "Reviewer_gmjr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- This is one of the early works to use static OT maps (or approximations more precisely) to improve bridge / flow matching procedures. Although the disintegration of KL into coupling and bridge is widely known, it has not been applied before in the context of generative modelling due to the difficulty in computing the coupling.\n- The authors show that a minibatch OT heuristic can benefit flow matching training by improving stability and lowering variance of the loss objective\n- The authors improve generative performance of flow matching for cifar10 benchmark to around fid 3.6. Although this does not compete with diffusion, diffusion bridge or IPF based SB [7] methods at around FID 1.7/ 1.7 / 3.0 respectively, it is an improvement over prior flow based approaches.\n- The torchCFM package is a nice addition (though baselines in the paper appear to be missing)\n- The energy based approach appears novel and interesting; extending the methods beyond sample only paradigms\n\n\n**Weaknesses**\n- Some claims are overstated and unsubstantiated\n  > OT-CFM is the first method to compute dynamic OT in a simulation-free way \n\n  > Our work is the first to enable simulation-free training of dynamic OT maps and Schrödinger bridge probability flows for arbitrary source and target distributions.\n\nThis work relies on minibatch OT, hence reliant on minibatch solvers. It is well known that minibatch solvers output a biased coupling for the true OT between marginal distributions; only coinciding for discrete measures on full batch OT solvers; hence it has not been proven that learning the vector field wrt minibatch couplings will return the SB for *arbitrary source and target distributions* only for discrete measures, even under the perfect training regime in terms of network and training time (not infinite batch size) - this is quite different to other approaches such as IPF / IMF [7,8,11,10] which should converge to (regularized) OT in the perfect training regime with finite batch size.\n\nThe proposed method may surpass [7,8,11,10] in terms of practical application for generative modelling where one does not care about OT, however for OT it is not clear that even practically the discrete solvers would be feasible for discrete measures with a large number of points nor if OT solvers for discrete measures approximate OT to any accuracy for continuous measures in high dimension. The sample complexity of the convergence of minibatch OT to full batch is exponentially poor wrt the dimension of the support. Minibatch OT suffers from $\\mathcal{O}(n^3)$ complexity for linear solvers in the non regularized case and $\\mathcal{O}(n^2)$ complexity in the regularized case with Sinkhorn, hence it is not practical to have high number of samples. It is far from clear how this method would be useful for computing OT beyond low dimensional toy examples other than as some sort of heuristic. Indeed I imagine in the worst case the regularization would need to be high in order to solve resulting in close to independent coupling or Sinkhorn would not converge and hence close to independent coupling.\n\nFurthermore, if one can perform Sinkhorn / minibatch OT and pretend this is the optimal coupling for the marginal measures, then one can simply interpolate between the measures using a Brownian bridge or linear interpolation to return the dynamic coupling. It is not clear why this method would be useful for approximating OT beyond Sinkhorn - which has been applied to point clouds > 10,000 points, more than the number of points considered in the examples of this paper - \"we constrain ourselves to a small example with 1000 points\". \n\nI do appreciate that OT may not be the goal of this paper, so this should be clarified and focus more on generative modelling. The generative modelling experiments do not reach the performance of many other recent methods. Indeed given glow matching from Gaussian to data is a particular case of diffusion models, I would expect better performance if the emphasis is that this improves flow / diffusion models for specifically generative modelling.\n\n>  CFM is able to learn conditional generative models from any samplable source distribution by conditioning on paired source and target samples, generalizing existing methods (Lipman et al., 2023; Albergo & Vanden-Eijnden, 2023; Liu, 2022; Pooladian et al., 2023)\n\nLiu et al 2022 [9], also use non Gaussian source and target measures in experiments, see Figure 1. Albergo & Vanden-Eijnden, 2023 also use non Gaussian source and target measures, see section 3.1. In addition, earlier work performs bridge matching for non-Gaussian source / targets [5,6,11].\n\n- Convergence guarantees \\\nThe results are presented as if there was a static coupling however given minibatch couplings, it is not clear if any convergence results hold - I imagine they do but as far as I am aware this has not been shown . I am unaware of any proof that suggests the learnt vector field will even converge or the coupling will converge to a fixed distribution, only that the potentials of minibatch OT converges to an invariant measure.\n\n- Lack of discussion to prior work \\\nThe method to perform bridge matching between coupling is well known for a number of years now. As far as I am aware the first notion of forward bridge matching is from Peluchetti Sept 2021[1]. This proves a diffusion process may be approximated between arbitrary measures by learning the drift of the forward process with h-transform from the diffusion bridge between points sampled from a coupling - hence mixture of diffusion bridge. This has been built upon by many works including Liu et al 2022 [2,3] and has a special case for zero noise - known as flow matching [4]. These works are for the diffusion bridge case which can be particularised to flow matching for zero noise. The results for these works are for arbitrary target / source measures though experimentally are only tried on Gaussian source measures.\n\nJust this year a number of published papers perform bridge matching on couplings which again result in SB if the coupling is optimal [5,6] between arbitrary measures. It is an open question of how to compute an optimal coupling in the general case for arbitrary measures (high dimensional, continuous support) and indeed the coupling is never optimal in prior work nor in the work under review here. Given Pooladian 2023 is cited and discussed I would expect these papers preceding it by a couple of months to also be discussed. \n \n- Concerns about baselines \\\n> We use the implementation from De Bortoli et al. (2021)\n\nI understand this code base does not use continuous time training but discrete; in addition has a different reference diffusion, integration time and hence different regularization for the coupling and does not coincide with Brownian bridge used as ground truth. Was this taken into account during the baselines? \n\nSimilarly it is stated 20 diffusion steps were used for De Bortoli et al. (2021) yet for SB-CFM \"tsit5 solver with atol=rtol=1e-4\" was used. Is this really comparable?\n\nIs the code available for the baselines?\n\n\n- [1] Peluchetti 2021 https://openreview.net/forum?id=oVfIKuhqfC&referrer=%5Bthe%20profile%20of%20Stefano%20Peluchetti%5D \n- [2,3] Liu et al 2022 https://arxiv.org/abs/2209.01170; https://arxiv.org/abs/2208.14699 \n- [4] Lipman et al 2022 https://arxiv.org/abs/2210.02747 \n- [5] Somnath et al 2023 https://arxiv.org/abs/2302.11419 \n- [6] Liu et al 2023 https://arxiv.org/abs/2302.05872 \n- [7] Chen et al 2022 Likelihood training of SB https://arxiv.org/abs/2110.11291 \n- [8] Liu 2022 https://arxiv.org/abs/2209.14577 \n- [9] Liu et al 2022 Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow \n- [10] Bortoli et al 2021, https://arxiv.org/abs/2106.01357 \n- [11] Shi et al 2023, https://arxiv.org/abs/2303.16852", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors propose flow / bridge matching on minibatch OT couplings for regularized and non regularized OT on minibatches\n- The authors extend this where the marginals are only known in energy form\n- The authors provide code package for their method (though not baselines)\n- The authors improve state of the art performance for flow based generative models on cifar10", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please temper the claims stated above. I understand this is one of the earlier works performing bridge / flow matching for (approximate) OT couplings. This has been found to be useful, though I imagine it has not yet been published due to such overreaching claims.\n- Provide justification of convergence for flow matching on (changing) minibatch couplings / samples from couplings.\n- Please correct lack of citation to prior works such as Peluchetti 2021\n- Given it is almost December 2023, I would expect work from August 2023 and earlier to be discussed and cited. Even to state such work is concurrent at the first time of attempted publication.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is well-written and presents nice contributions. My only concern is regarding the comparison with the work of [1].
Indeed, while [1] say that in practice they use independent coupling, they note that any coupling could be used. Therefore, in my opinion, the main generalization of the present paper lies in the use of more general bridges $p_t(x|z)$. 

In addition, the comparison with [2] appears to be ambiguous. 

As a result, I ask for the authors to clarify their claims in light of this observation and elaborate on their discussion concerning these two papers.

[1] Liu et al 2022 Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow 
[2] Michael S. Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants
Decision: Accept (poster)