Building Normalizing Flows with Stochastic Interpolants
Authors: Michael Samuel Albergo
Eric Vanden-Eijnden
Venue: ICLR 2023 poster (2023)
Keywords: normalizing flow
ODE
generative model

Abstract: A generative model based on a continuous-time normalizing flow between any pair of base and target probability densities is proposed. The velocity field of this flow is inferred from the probability current of a time-dependent density that interpolates between the base and the target in finite time. Unlike conventional normalizing flow inference methods based the maximum likelihood principle, which require costly backpropagation through ODE solvers, our interpolant approach leads to a simple quadratic loss for the velocity itself which is expressed in terms of expectations that are readily amenable to empirical estimation. The flow can be used to generate samples from either the base or target, and to estimate the likelihood at any time along the interpolant. In addition, the flow can be optimized to minimize the path length of the interpolant density, thereby paving the way for building optimal transport maps. In situations where the base is a Gaussian density, we also show that the velocity of our normalizing flow can also be used to construct a diffusion model to sample the target as well as estimate its score. However, our approach shows that we can bypass this diffusion completely and work at the level of the probability flow with greater simplicity, opening an avenue for methods based solely on ordinary differential equations as an alternative to those based on stochastic differential equations. Benchmarking on density estimation tasks illustrates that the learned flow can match and surpass conventional continuous flows at a fraction of the cost, and compares well with diffusions on image generation on CIFAR-10 and ImageNet $32 \times 32$. The method scales ab-initio ODE flows to previously unreachable image resolutions, demonstrated up to $128\times128$.

Published as a conference paper at ICLR 2023
BUILDING NORMALIZING FLOWS WITH STOCHASTIC INTERPOLANTS
Michael S. Albergo
Eric Vanden-Eijnden
Center for Cosmology and Particle Physics
Courant Institute of Mathematical Sciences
New York University
New York University New York, NY 10012, USA
New York, NY 10003, USA
albergo@nyu.een
Pre2@cimm.nyu.co
ABSTRACT
A generative model based on a continuous-time normalizing flow between any pair of base and target probability densities proposed. The velocity field of this flow is inferred from the probability current of time-dependent density that in- terpolates between the base and the target finite time. Unlike conventional nor- malizing flow inference methods based the maximum likelihood principle, which require costly backpropagation through ODE solvers, our interpolant approach leads to simple quadratic loss for the velocity itself which is expressed in terms expectations that are readily amenable to empirical estimation. The flow can be used to generate samples from either the base or target, and to estimate the like- lihood at any time along the interpolant. In addition, the flow can be optimized minimize the path length of the interpolant density, thereby paving the way for building optimal transport maps. In situations where the base is a Gaussian den- sity, we also show that the velocity of our normalizing flow can also be used construct a diffusion model to sample the target as well as estimate its score. How- ever, our approach shows that we can bypass this diffusion completely and work at the level of the probability flow with greater simplicity, opening avenue for methods based solely on ordinary differential equations as an alternative to those density estimation
based on stochastic differential equations. Benchmarking tasks illustrates that the learned flow can match and surpass conventional contin- uous flows at fraction of the cost, and compares well with diffusions image generation on CIFAR- 10 and ImageNet 32 x 32. The method scales ab-initio ODE flows to previously unreachable image resolutions, demonstrated up to 128 128.
INTRODUCTION
Contemporary generative models have primarily been designed around the construction of map between two probability distributions that transform samples from the first into samples from the second. While progress has been from various angles with tools such as implicit maps (Goodfellow cet al. 2014 Brock et al. 2019). and autoregressive maps (Menick & Kalchbrenner 2019 Razavi al.
2019 Lee et al.] 2022), we focus on the case where the map has a clear associated probability flow. Advances in this domain, namely from flow and diffusion models, have arisen through the introduction algorithms or inductive biases that make learning this map, and the Jacobian of the associated change of variables, more tractable. The challenge is to choose what structure to impose on the transport to best reach a complex target distribution from a simple one used as base, while maintaining computational efficiency.
In the continuous time perspective, this problem can be framed as the design of a time-dependent map. (x)x) with [0. 1], which functions as the fush-forward of the base distribution time t = 0 onto some time-dependent distribution that reaches the target at time t = 1. Assuming that these distributions have densities supported Ω C Rdd, say Po for the base and Pi for the target, this amounts to constructing X, Ω + Ω such that
if Po then X((x)) Ps. for some density As such that Pi= = Po and Pt=1 P1. (1)

---

Published as a conference paper at ICLR 2023
Figure 1: The density Pc(x) produced by the P1 stochastic interpolant based on (5) between standard Gaussian density and Gaus- sian mixture density with three modes. Also shows white are the flow lines of the map X₂ (1: our method produces.
space
One convenient way to represent this time-continuous map is to define it as the flow associated with the ordinary differential equation (ODE)
(x(x) = uy (x)())),
Xz-o(=)=a
where the dot denotes derivative with respect t and (xx) is the velocity field governing the trans-
port. This is equivalent to saying that the probability density function pr (z) defined as the push- forward of the base Po(2)) 1r) by the map X2 satisfies the continuity equation (see e.g. (Villani 2009 Santambrogio 2015) and Appendix A)
i.pL + V (VePt)= =0 with Pt=0 Po and Pt=1 Pl.
and the inference problem becomes to estimate a velocity field such that 3 holds.
Here propose solution to this problem based introducing time-differentiable interpolant
4 : + Q such that 4==0=0, x1) = zo and (x=1 = I1
(4)
A useful instance of such an interpolant that we will employ is Ie(00, = cos( T)) + sin()
(5)
though we stress the framework we propose applies to any (x z1) satisfying (4) under mild additional assumptions on Po. Pl. and 4, specified below. Given this interpolant, we then construct the stochastic process by sampling independently zo from po and from Pl. and passing them through I,:
21 (x 21),
2y Pos 21 P1 independent.
We refer the process 24 stochastic interpolant. Under this paradigm, make the following key observations as our main contributions this work:
The probability density (z(z) ) of 2t connecting the two densities, henceforth referred to as the interpolant density, satisfies (3) with a velocity (2) which the unique minimizer of a simple quadratic objective. This result is the content of Proposition below, and be leveraged to estimate vy (x) in a parametric class (e.g. using deep neural networks) to construct generative model through the solution of the probability flow equation ( which we call InterFllow
By specifying an interpolant density, the method therefore separates the tasks of minimizing the objective from discovering a path between the base and target densities. This is in contrast with conventional maximum likelihood (MLE) training of flows where one is forced to couple the choice path in the space of measures to maximizing the objective.
We show that the Wasserstein-2 (W2) distance between the target density #I and the density ý obtained by transporting Po using an approximate velocity ig in (2) is controlled by our objective function. We also show that the value of the objective ég during training can be used to check convergence of this learned velocity field towards the exact U/.
We show that approach sity and optimize the transport by additionally maximizing our objective over the interpolant 21) and/or adjustable parameters in the base density po-
be generalized shorten the path length of the interpolant den-
By choosing Po to be Gaussian density and using (5) as interpolant, we show that the score of the interpolant density, V log Pt. can be explicitly related the veloeity field vt- This allows us to draw connection between our approach and score- score-based diffusion models, providing theoretical groundwork for future exploration of this duality.
We demonstrate the feasibility of the method on toy and high dimensional tabular datasets, and show that the method matches or supersedes conventional ODE flows at lower cost, as it avoids the need to backpropagate through ODE solves. We demonstrate approach image gener- ation for CIFAR-10 10 and ImageNet 32x32 and show that it scales well to larger sizes, e.g. on the 128 x 128 Oxford flower dataset.
2

---

Published as a conference paper at ICLR 2023
Map
Finite Time Integration
Samulation- Training
Computable Likelibood
Optimizalle
Methods
Type
Transport
FF/ORD
D
ScorePlaws
S/D
1.1 RELATED WORKS
Schriddinger Bridge
InserFlow (Ouns)
D
Early works on exploiting transport maps for generative modeling go back at least to
Table I: Description of the qualities of continuous time transport methods defined by a stochastic or deterministic process.
Chen & Gopinath (2000). which focuses on normalizing dataset to infer its likelihood. This idea was brought closer to contemporary use cases through the work of Tabak
Ganden-Eijnden
(2010) and Tabak & Turner (2013), which devised to expressly map between densities using simple transport functions inferred through maximum likelihood estimation (MLE). These transformations were learned in sequence via greedy procedure. We detail below how this paradigm has evolved in the case where the map is represented by a neural network and optimized accordingly.
Discrete and continuous time flows. The first success of normalizing flows with neural network parametrizations follow the work of Tabak & Turner (2033) with a finite set of steps along the map. By imposing structure on the transformation so that remains
efficiently invertible diffeomor- phism, the models of Rezende & Mohamed (2015); Dinh et al. (2017); Huang et al. (2018); Durkan eet al.] (2019) can be optimized through maximum likelihood estimation at the cost of limiting the expressive power of the representation, as the Jacobian of the map must be kept simple to calcu- late the likelihood. Extending this to the continuous case allowed the Jacobian to be unstructured yet still estimable through trace estimation techniques (Chen et al. 2018 Grathwohl et al. 2019 Hutehinson] 1989). Yet, learning this map through MILE requires costly backpropagation through numerical integration. Regulating the path can reduce the number of solver calls (Finlay et al. 20200) Onken et al. 2021), though this does not alleviate the main structural challenge of the optimization. Our work uses a continuous map X, as well but allows for direct estimation of the underlying velocity. While recent work has also considered simulation- free training by fitting a velocity field, these works present scalability issues (Rozen et al. (02I)) and biased optimization (Ben- Hamu et al., 2022]. and are limited to manifolds. Moreover, (Rozen et al. (02T) relies on interpolating directly the probability measures, which can lead to unstable velocities.
Score-based flows. Adjacent research has made use of diffusion processes, commonly the Ornstein- Uhlenbeck (OU) process, to connect the target P1 to the base Po. In this case the transport is governed by stochastic differential equation (SDE) that is evolved for infinite time, and the challenge of learning a generative model can be framed fitting the reverse time evolution of the SDE from Gaussian noise back to P1 (Sohl-Dickstein et al. 20155 Ho et al. 2020: Song et al. 2021b). Doing so indirectly learns the velocity field by means of learning the score function V log (z), using the Fischer divergence instead of the MIL objective. While this approach has shown great promise to model high dimensional distributions (Rombach et al. 2022 Hoogeboom et al. 2022). particularly in the case of text-to-image generation (Ramesh et al. 2022 Saharia et al. (2022) there is an absence of theoretical motivation for the SDE-flow framework and the complexity it induces. Namely, the SDE must evolve for infinite time to connect the distributions, the parameterization of the time steps remains heuristic (Xiao et al. 2022), and the criticality of noise, as well as the score, is not absolutely apparent (Bansal et al. 2022 Lu et al. 2022). In particular, while the objective used in score-based diffusion models was shown to bound the Kullback-Leibler divergence (Song et al.) (001a)), actual calculation of the likelihood requires one to work with the ODE probability flow associated with the SDE. This motivates further research into effective, ODE-driven, approaches to learning the map. Our approach can be viewed as an alternative to score-based diffusion models in which the ODE velocity is Iearned through the interpolant I, rather than an OU process, leading to greater simplicity and flexibility (as we can connect any two densities exactly over finite time interval).
Briege-based methods. Heng et al. 2021) propose to learn Schroedinger bridges, which are a entropic regularized version of the optimal transportation plan connecting two densities in finite time, using the framework of score-based diffusion. Similarly,Peluchettii Peluchetti (2022) investigates the use of bridge processes, i.e. SDE whose position constrained both at the initial and final times, to perform exact density interpolation in finite time. A key difference between these approaches and ours is that they give diffusion based models, whereas our method builds a probability flow ODE directly using quadratic loss for its velocity, which is simpler and shown here to be scalable.
Interpolants. Co-incident works by Liu et al. (2022): ) Lipman et al. (202) derive an analogous op- timization to us, with focus straight interpolants, also contrasting with score-based methods. Liu et al. (2022) describe an iterative way of rectifying the interpolant path, which can be shown to
arrive at an optimal transport map when the procedure is repeated ad infinitum (Liu 2022). We also
3

---

Published as a conference paper at ICLR 2023
propose solution to the problem of optimal transport that involves optimizing our objective the stochastic interpolant.
1.2 NOTATIONS AND ASSUMPTIONS
We assume that the base and the target distribution are both absolutely continuous with respect to the Lebesgue measure on IRd with densities po and pl. respectively. We do not require these densities to
be positive everywhere on IR but we assume that (a) and @1 (r) are continuously differentiable in z. Regarding the interpolant 4 : IRd x IRd + IR' we assume that it is surjective for all E [0, 1] and satisfies 44.. We also assume that 4 (2o, x1) is continuously differentiable in (t, zo, T1), and that it is such that
E (20, < 00
A few additional technical assumptions Pa- P1 and 4, are listed in Appendix B Given any function fe 21) we denote
([ff(=o.=1)11 -fo JEE fe(zo, ) (2o)) (21) ddrddd dt
JR
(8)
its expectation over I, vp. and 1 drawn independently from the uniform density on [0, 11. po. and pl. respectively. We use V to denote the gradient operator.
2 STOCHASTIC INTERPOLANTS AND ASSOCIATED FLOWS
Our first main theoretical result can be phrased as follows:
Proposition 1. The stochastic interpolant It defined in 6 with Le 21) satisfying 44 has
probability density pe(x) that satisfies the continuity equation 3 with velocity ve(x) which is the unique minimizer over 0g (x) of the objective
G( E [[6 21 21))1 20,I (T
(9)
In addition the minimum value of this objective is given by
G(u) = E [2))
x -1)1)
(v((=)) ~00 -00
(10)
Proposition [ is proven in Appendix under Assumption B. ment of the proposition remains true if the expectation is performed using any probability density w(t) > 0, which may prove useful in practice. We now describe some primary facts result- ing from this proposition, itemized for clarity:
As this proof shows, the first state-
The objective G()) is given in terms of an expectation that is amenable empirical estimation given samples i, To. and Ty drawn from Po- and ((0, 1)). Below, we will exploit this property to propose a numerical scheme to perform the minimization of G(o).
While the minimizer of the objective G(ū) is not available analytically in general, notable excep- tion is when po and are Gaussian mixture densities and we use the trigonometric interpolant (5) or generalization thereof, as discussed in Appendix C
The minimal value in (10) achieved by the objective implies that a necessary (albeit not sufficient) condition for is
G(i) G(t) E[ (Zo x )) =0 0.
(11)
In our numerical experiments we will monitor this quantity. This minimal value also suggests to maximize G(u) min; G()) with respect to additional control parameters (e.g. the interpolant) to shorten the W₂ length of the path {p((z)
E [0, 1]]. In Appendix D we show that this procedure achieves optimal transport under minimal assumptions.
The last bound in (IO), which proven in in Lemma B.2 implics that the path length always
finite, even if it is not the shortest possible
4

---

Published as a conference paper at ICLR 2023
Let us now provide some intuitive derivation of the statements of Proposition Contimuity equation. By definition of the stochastic interpolant 2, we can express its density (z using the Dirac delta distribution as
p=(x)
8 (ar (e (2to) (11)) (drdddd......
(12)
Since It=0(20 21) = zy and I==1 (zg. 21) 21 by definition, we have Pt=0 = Po and Pt=1 P1.
which means that (z) satisfies the boundary conditions at = 0, in 3 Differentiating (12) time using the chain rule gives
in
Oepe(x)) - ((pc(x)=-
(cT( V8 (xr 1 (x000 (o( Madde - -e(x)) (13)
where we defined the probability current
/ xa (24, 21) (()))((()(() (11)ddd
je(z)
(14)
Therefore if we introduce the velocity (x() via
()(())/((a))
if px > 0,
(x)
(15)
0
else
see that can write (3) as the continuity equation in (3
Variational formulation. Using the expressions in (12 and for pe(ar) and je(x) shows that we can write the objective (9 as
G( = [ (10(())) (0)()) fe(x)) drdt
(16)
Since (a( and ja(r)) have the same support, the minimizer of this quadratic objective is unique for all (1, x) where pe(2)) > 0 and given by (15).
Minimum value of the objective. Let (xr be given by 15 and consider the alternative objective
H(6) = 1'' J (0((x) xx(x) p((x)dodt
(17)
L. / (10(())PP((z) - 20,(z) (x) + (xx()) dxdt
where we expanded the square and used the identity 2 (x)), = J. (z) to get the second equality. The objective G(t) given in (16) can be written in term of (0)) as
L. ga
(xx(z)) (x = (0) E (4. (
(18)
The equality in (10) follows by evaluating (18) (z)) = vz(x) using H (v) = 0.
Optimality gap. The argument above shows that we can use B[ x1)) I .1) = as alternative objective to G(t) since their first variations coincide. However, it should be stressed that this quadratic objective remains strictly positive at
general so offers no baseline measure
of convergence. To see why, complete the square in G( to write (88
H())= E (T.) 2 1 E I ? + (1, Z E [ + (L)(
(19)
where we used the shorthand notation I, = Ie Lz 21) and 0, = (. 1)). Evaluating this inequality ē e using H (vv) 0 we deduce
E [[ 2 B (lob((4)))
(20)
However we that this inequality not saturated, i.e. [[].((1)) E[0..1] in general (see Remark B.3). Hence
E (Tx((T)) 0 min E (()o)(To) 0
(21)
min G(D) E J. -E
(I))
1 2 0.
5

---

Published as a conference paper at ICLR 2023
Optimizing the transport. It is natural to ask whether stochastic interpolant construction can be amended or generalized to derive optimal maps. Here, we state positive answer to this question by showing that the maximizing the objective G(Ø) in (9) with respect to the interpolant yields solution to the optimal transport problem in the framework of |enamou & Brenier (2000). This is proven in Appendix [ where we also discuss how shorten the path length in density space by optimizing adjustable parameters in the base density po- Experiments are given in Appendix HH Since our primary aim here construct a map T Xt=1 that pushes forward po onto Pi, but not necessarily to identify the optimal one, we leave the full investigation of the consequences of these results for future work, but state the proposition explicitly here. In their seminal paper, [Benamou & Brenier (2000) showed that finding the optimal map requires solving the minimization problem
min L'' S (0((=))P (0.0)
(22)
subject to: 0.p: + V (DrDe) = O, ~== = po: pt= = P1-
The minimizing coupling (0t, o)) for gradient field 27 (2) = V (x) is unique and satisfies:
0.p: - (Voip:) = 0, Pi Pa- Pi- Pl. 0,0; + Vo, = 0.
(23)
In the interpolant flow picture, (a( is fixed by the choice of interpolant (z0, 21), and in general (c(x)) * (x)) Because the value of the objective in (22) is equal to the minimum of G( given in (TO), natural suggestion to optimize the transport is to maximize this minimum over the interpolant. Under some assumption on the Benamou
-ennnmou-Brenier Brenier density p (x) solution of (23), this procedure works. We show this through the use of interpolable densities discussed in Mikulincer
& Shenteld (2022) and defined in D.I
Proposition 2. Assume that (i) the optimal density function pi (z) minimizing 22 interpolable and (ii) 23 has a classical solution. Consider the max-min problem
max min G(ū)
(24)
1 6
where G(ö) is the objective in (9) and the maximum is taken over interpolants satisfying 4 Then maximizer of (24) exists, and any maximizer Iğ (zg, is such that the probability density function of x; I (To. 21). with 2g ~ Po and xı ~ P1 independent, is the optimal p (1)) the mimimizing velocity oz (z) = Voz (z), and the pair (pt (z), oz (z)) satisfies (23).
The proof of Proposition 2 is given in Appendix D along with further discussion. Proposition relies on Lemma D.3 that reformulates (22) in a way which shows this problem is equivalent the max-min problem in (4 for interpolable densities.
2
2.1 WASSERSTEIN BOUNDS
The following result shows that the objective in (17) controls the Wasserstein distance between the target density #i and the the density أJ obtained as the pushforward of the base density Po by the map Xi=1 associated with the velocity fy:
Proposition 3. Let pe(x) be the exact interpolant density defined in 12) and, given a velocity field (z).. let define pz( the solution of the initial value problem
Sepr + V (fr.pt) = 0, Pt= Po
(25)
Assume that 0z(z) is continuously differentiable in (t, and Lipschitz in x uniformily on (t, x)) E [0, 1] x dd with Lipschitz constant K. Then the square of the W2 distance between Pi and p is bounded by
(p pi) < <e 122k H())
(26)
where (0) is the objective function defined in 11
The proof of Proposition 3 is given in Appendix of W-2 distance
it leverages the following bound on the square
[/ [x==(x))
W (p1,
(x)1(x))² - (o Po(z)dzdt
(27)
where X, is the flow map solution of (2) with the exact (z) defined in [5] and X, is the flow map obtained by solving (2) with (x)) replaced by (z()).

---

Published as a conference paper at ICLR 2023
2.2 LINK WITH SCORE- GENERATIVE MODELS
The following result shows that if po is a Gaussian density, the velocity (x( can be related to the score of the density pe(x)::
Proposition 4. Assume that the base density Pa(x)) is standard Gaussian density N(0, Id) and suppose that the interpolant [1 X1) is given by 5 Then the score log pe(x) related to velocity vx())
2 tan (tn
te (0,
V log pc(x)
π
(28)
0101
t=
The proof of this proposition is given in Appendix F The first formula for t E [0, 1) is based on direct calculation using Gaussian integration by parts; the second formula at = is obtained by taking the limit of the first using 1t= (2) 0 from (B200) and H'Hôpital's rule. It shows that we can
principle resample Pt at any t [0, 1] using the stochastic differential equation artificial time т whose drift is the score (x) obtained by evaluating (28) the estimated dy (z)):
dz, (x ( )dr 2dW,,
(29)
Similarly, the score (xr) could in principle be used in score-based diffusion models, as explained in Appendix F We stress however that while our velocity is well-behaved for all times in [0. 1]. as
shown in B B.20), the drift and diffusion coefficient in the associated SDE are singular at = 0, 1.
3 PRACTICAL IMPLEMENTATION AND NUMERICAL EXPERIMENTS
The objective detailed in Section is amenable to efficient empirical estimation, see which utilize experimentally validate the method. Moreover, it is appealing consider neural network parameterization of the velocity field. In this case, the parameters of the model 0 can be op- timized through stochastic gradient descent (SGD) its variants, like Adam (Kingma & Ba 2015). Following the recent literature regarding density estimation, we benchmark the method on visualiz- able yet complicated densities that display multimodality, as well as higher dimensional tabular data initially provided in Paaaaaakaiiss et al. (207)) and tested in other works such as Grathwohl et al. (2019). The test cases demonstrate the ability to flow between empirical densities with no known analytic form. In all cases, numerical integration for sampling is done with the Dormand- Prince, explicit Runge-Kurte Kutge of order (4)5 (Dormand & Prince 1980). Sections 3.1 the choice of in- terpolant for cxperimentation to the technique of score based diffusions in Proposition
selected to be that of (5) it is the one used to draw connections Section H we use the interpolant (B.2) and optimize a; and b, investigate the possibility and impact of shortening the path length.
3.1
2D DENSITY ESTIMATION
An intuitive first test to benchmark the validity of the method is sampling a target density whose analytic form is known whose density be visualized for comparison. To this end, we follow the practice of choosing few complicated 2-dimensional toy datasets, namely those from (Grathwohl al. 2019). which were selected to differentiate the flexibility of continuous flows from discrete time flows, which cannot fully separate the modes. We consider anisotropic curved densities, mixture of separated Gaussians, and checkerboard density. The velocity field the interpolant
EH:055
Figure 2: Left: 2-D density estimation. Right: Learning flow map between densities when nei- ther are analytically known.

---

Published as a conference paper at ICLR 2023
CIFAR 10
ImaggeNet 32x332
Method
MINI- BOONE
NLL
FID
NLL
FID
MADE
3,08
20.95
15.59
-148.85
Glow
3.35
4.09
Real
18.71
DDPM
S 3.755
3.17
Glow
-0.17
-8.15
18.92
11:35
-155,07
DDPM
CPP -0.32
-10.35
16.93
10.58
-158,99
ScoreSDE
2.99
2.92
NSP
14.75
VDM
22.65
7.41
$3.72
HUORD
Soft Trancation
2,88
3.45
3,55
5.42
DT-Fllow
-0.30
-9.20
17.32
10.55
-154.20
ScureFllow
3.76
Ours
0.37
12.35
14.85
10:42
-156.22
Ours
348
Table Left: Negative log likelihoods (NLL) computed on test data unseen during training (lower is better). Values of MADE, Real NVP, and Glow quoted from the FFJORD paper. Values of OT- Flow, CPF, and NSP quoted from their respective publications. Right: NLL and FID scores on unconditional image generation tasks for recent advanced models that emit likelihood.
flow is parameterized by simple feed forward neural network with ReLU Nair Hinton (2010) activation functions. The network for each model has 3 layers, each of width equal to 256 hidden units. Optimizer is performed on G()) for 10k epochs. We plot kernel density estimate over 80k samples from both the flow and true distribution in Figure 2 The interpolant flow captures all the modes of the target density without artificial stretching or smearing, evincing a smooth map.
3.2 DATASET INTERPOLATION
As described in Section 2 the velocity field associated to the flow can be inferred from arbitrary densities Po: P1 this deviates from the score-based diffusion perspective, in which distribution must be taken to be Gaussian for the training paradigm to be tractable.
In Figure 2 we illustrate this capacity by learning the velocity field connecting the anisotropic swirls distribution to that of the checkerboard. The interpolant formulation allows us to draw samples from Pe at any time E [0. 1]. which we exploit to check that the velocity field is empirically correct at all times on the interval, rather than just at the end points. This aspect of interpolants is also noted in (Choi et al. 2022)) but for the purpose of density ratio estimation. The above observation highlights an intrinsic difference of the proposed method compared to MLE training of flows, where the map that the minimizer of G( is not empirically known. We stress that query access to Po or P1 is not needed use our interpolation procedure since it only uses samples from these densities.
3.3 TABULAR DATA FOR HIGHER DIMENSIONAL. TESTING
A set of tabular datasets introduced by (Papamakarios al.) 2017) has served consistent test bed for demonstrating flow-based sampling and its associated density estimation capabilities. We continue that practice here provide benchmark of the method models which provide an exact likelihood, separating and comparing to exemplary discrete and continuous flows: MADE (Germain et al. 2015). Real NVP (Dinh al. 2017). Convex Potential Flows (CPF) (Huang al.
2021). Neural Spline Flows (NSP) Durkan et al. (2019), Free-form continuous flows (FFJORD) (Grathwohl et al. 2019), and OT-Flow (Finlay al. 2020). Our primary point of comparison is to other continuous time models, so we sequester them in benchmarking.
We train the interpolant flow model on each target dataset listed Table choosing the reference distribution of the interpolant Po to be a Gaussian density with mean zero and variance Ja. where d the data dimension. The architectures and hyperparameters are given in Appendix | We highlight some of the main characteristics of the models here. In each case, sampling of the time was reweighted according to a beta distribution, with parameters at, provided in the same appendix.
Results from the tabular experiments are displayed in Table 2 in which the negative log-likelihood averaged over test set of held out data computed. We note that the interpolant flow achieves all ODE based models, except BSDS300. in which the FFJORD outperforms the interpolant by ~ 0.6%. We note upwards of 30% improvements compared to baselines. Note that these likelihoods achieved without direct optimization of it.
better or equivalent held out likelihoods
8

---

Published as a conference paper at ICLR 2023
3.4 UNCONDITIONAL IMAGE GENERATION
To compare with recent advances in continuous time generative models such DDPM (Ho (et al., 2020), Score SDE(Song et al., 2021b), and ScoreFlow (Song et al. [02Ia), we provide demonstration of the interpolant flow method on learning to unconditionally generate images trained from the CIFAR-10 10 (Krizhevsky et al.) 2009) and ImageNet x2x32 datasets (Deng et al.) 2009 Van Den Oord et 2016), which follows suit with ScoreFlow and Variational Diffu- sion Models (VDM) (Kingma et al. 2021). We train an interpolant flow built from the U- Net architecture from DDPM (Ho al., 2020) on a single NVIDIA A100 GPU, which was previously impossible under maximum likelihood training of continuous time flows. Experi- mental details can be found in Appendix ! Note that we used beta distribution reweight- ing of the time sampling as in the tabular experiments. Table 2 provides a comparison of the negative log likelihoods (NLL),, measured in bits per dim (BPD) and Frechet Inception Dis- tance (FID), of our method compared to past flows and state of the art diffusions. We focus our comparison against models which emit likelihood, as this is necessary to compare NLL.
We compare to other flows FFJORD
and Glow (Grathwohl et al. 2019 Kingma & Dhariwal 2018
as
well as to recent advances in score based diffusion in DDPM, DDPM++
VDM, Score SDE, ScoreFlow, and
Soft Truncation (Ho et al. 2020
Nichol & Dhariwal 1002; K Kingma
al. 2021 [Song al.
202 |
Kim
et al. 2022 We present results with-
out data augmentation. Our models emit likelihoods, measured in bits per dim, that are competitive with diffu- sions both datasets with a NLL of 2.99 and 3.45. Measures of FID are proximal those from diffusions, though slightly behind the best results. We note however that this is a first example of this type of model, and has not been optimized with the training tricks that appear in many of the recent works on diffusions, like exponential moving averages, truncations, learning rate warm-ups, and the like. To demonstrate efficiency on larger domains, we train on the Oxford flowers dataset (Nilsback & Zisserman 2006), which are images of resolution 128 x 128. We show example generated images in Figure
Figure 3: Left: InterFlow samples training on 128x 128 flowers dataset. Right: Samples from flow trained on ImageNet-32x32 (top) and CIFAR-10 (bottom).
4 DISCUSSION, CHALLENGES, AND FUTURE WORK
We introduced continuous time flow method that can be efficiently trained. The approach has number of intriguing and appealing characteristics. The training circumvents any backpropagation through ODE solves, and emits stable and interpretable quadratic objective function. This objective has an casily accessible diagnostic which can verify whether a proposed minimizer of the loss is valid minimizer, and controls the Wasserstein-2 distance between the model and the target.
One salient feature of the proposed method is that choosing an interpolant Lę (x æ1) decouples the optimization problem from that of also choosing a transport path. This separation also exploited by score- ccore-based diffusion models, but our approach offers better explicit control on both. In particular we can interpolate between any two densities in finite time and directly obtain the probability flow needed to calculate the likelihood. Moreover, we showed in Section 2 and Appendices [ and G that the interpolant can be optimized to achieve optimal transport, a feature which can reduce the cost of solving the ODE to draw samples. In future work, we will investigate more thoroughly realizations this procedure by learning the interpolant 4; in a wider class of functions, in addition to minimizing G(o).
The intrinsic connection to score-based diffusion presented in Proposition may be fruitful ground for understanding the benefits and tradeoffs of SDE ODE approaches to generative modeling. Exploring this relation is already underway (Lu et al. 2022, Boffi & Sanden-Eijnden, 2022). and can hopefully provide theoretical insight into designing more effective models.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank Gérard Ben Arous, Nick Boffi, Kyle Cranmer, Michael Lindsey, Jonathan Niles-Weed Esteban Tabak for helpful discussions about transport. MSA is supported by the National Science Foundation under the award PHY-2141336. MSA grateful for the hospitality of the Center for Computational Quantum Physics at the Flatiron Institute. The Flatiron Institute is a division of the Simons Foundation. EVE is supported by the National Science Foundation under awards DMR- 1420073, DMS-2012510, and DMS-2134216 by the Simons Collaboration on Wave Turbulence, Grant No. 617006, and by a Vannevar Bush Faculty Fellowship.
REFERENCES
Arpit Bansal, Eitan Borgnia, Mong-Min Chu, Jie S. Li, Hamid Kazemi, Furong Huang, Micah Gold- blum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise, 2022. URL httpts://////t /tttpps:/aaxii.... http//aivvvv/// org/ ab///ab///// 2208. 09392
Heli Ben- Ben-Hamu, Samuel Cohen, Joey Bose, Brandon Amos, Maximilian Nickel, Aditya Grover, Ricky T. Q. Chen, and Yaron Lipman. Matching normalizing flows and probability paths on mani- folds. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICMI. 2022, 17-23 July 2022, Bal- timore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 1749- 1763. PMLR, 2022. URL Utthttps:: Ittp: http:::://ppcceeciinge m.lr wwwppprssssllvv.mm w2////l2/////// wwwbe/n///eeeeeeeeeeeeeeeeeeeeee hamu22a. html
Jean-David Benamou and Yann Brenier. A computational fluid mechanics solution to the monge- kantorovich mass transfer problem. Numerische Mathematik, 84(3):375-383 0017775--33333 2000.
Probability Nicholas M. Boffi and Eric Vanden-Eijnden, Probabbaai flow solution of the bakker-planckkee equation, 2022.
0444 04642
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL
net/ http:/ooommmdiillx000010
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Resa-Bianchi, and
R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran As- sociates, Inc., 2018 URL httptttpss//ppppeeeeeiieeeeeeeee neur
neur ips. cct/paper//011//f11ee pdd
Scott Chen and Ramesh Gopinath. Gaussianization. In T. Leen, T. Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Systems, volume 13. MIT Press. 2000.
URL
http://eeeeeaaa.e..e.iaaccccaaaa pdf
Kristy Choi, Chenlin Meng, Yang Song, and Stefano Ermon. Density ratio estimation via infinites- imal classification. Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera (eds.), Inter- national Conference Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event, volume 151 of Pruceedings of Machine Learning Research, pp. 2552-2573, PMLR, 2022.
mlr www.s/pssse...........o /ttes//115.. http://..iiiaiaaoccooooo html
Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). Yoshua Bengio and Yann LeCun (eds.), 4th Inter- national Conference on Learning Representations, ICLR 20/6, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Pruceedings, 2016. URL http::tt//// / /tatt::///xxivii /ttp/vaaavv/////////////////.av. wwtpaiivvorrreeee wogg/bogga.......... ha/aa/bs1111111 07289
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 1ww.10///VVRR22009552200 5206848.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In International Conference on Learning Representations, 2017. URL holhhtts: httttp::///peeeevveee
10

---

Published as a conference paper at ICLR 2023
J.R. Dormand and P.J. Prince. A family of embedded runge-kutta formulae. Journal of Compu- tational and Applied Mathematics, 6(1):19-25 1980. ISSN 0377-0427. doi: https//doiioo org/ 00.00660771-00000880900000000000
URLhttps::. htttpp::/www.... www.wwwwwweeeerrettccomm
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https: http::s:s///ooreeeeeggeo neur http:/ssss..isssspppar/////////a wwtp//paaee/21////eeeee. 7ac71d433f2822 00011663322223340044434182888811
pdf
Chris Finlay, Joern-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman. How to train your neural ODE: the world of Jacobian and kinetic regularization. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, 3154-3164. PMLR, 13-18 Jul 2020. URL htttps:/pproeeedingg ml wwwpress/viii win.ay2a..aaaammmmmmmm html
Mathicu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In Francis Bach and David Blei (cds.), Proceedings of the 32nd Inter- national Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Re- search, pp. 881 889, Lille, France, 07-09 Jul 2015. PMLR. URL htttp::://ttcceeeeeeeee m1 prw.s/es//vee v3 http:///www/ggrmmmaam........
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- mation processing systems, pp. 2672-2680, 2014.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and David Duvenaud. Scalable reversible generative models with free-form continuous dynamics. In International Conference on Learning Representations, 2019. URL
net forum PidrruummeeiU
Jeremy Heng, Valentin De Bortoli, Arnaud Doucet, and James Thornoon. Simulating diffusion www./aggs.....oooo.. httpaa/////111111////../.. 07243
bridges with score matching, 2021. http::ttt///att
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851. Curran Asso- neurips. cc/ppaper/222//i1ee pdff
ciates, Inc., 2020. URL aabbbfeee8aaaaf00666lllaa.......
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffu- sion for molecule generation in 3D. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Confer- ence Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 8867-8887. PMLR, 17-23 Jul 2022. URL /ttt::ttt httpss:/ppooceedngg....x www.inngsm.l.......m htw.:///...........////..... ptess/v1s2//// hoogebbom222... html
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Con- ference on Machine Learning, volume 80 Proceedings of Machine Learning Research, pp. PMLR, 10-15 Jul 2018. URL htt.h:tts///// http:////ccceeinng..mmm ml htt.://s...prrr....empmmm wwwpppewsss....
Mmang18d.htm.
Chin-Wei Huang, Ricky T. Q. Chen, Christos Tsirigotis, and Aaron Courville. Convex potential flows: Universal probability distributions with optimal transport and convex optimization. In International Conference on Learning Representations, 2021. htthhttss://ppereeeeet nett/Eorum2id--tePPVV
M.F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics Simulation and Computation, 1853):1059- 1076, 1989. doi: 10.1080/0361001180000 12806. URL Ittps: :/tpps:://d...oo http//...org/0..100000
036 091 89088 2806

---

Published as a conference paper at ICLR 2023
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estima- tion. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 11201-11228. PMLR, 17-23 Jul 2022. URL https::/proceccing mlr wwt.m/.ppeee..e////ienteee 1ttm/ httl
Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representutions (ICLR), 2015.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. On density estimation with dif- fusion models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, 2021. URL /tpL/tttp////pmeeeiieew... net/ Forum? 11 Ld=2Ld
Durk Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolu- tions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Biande and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018 URL. https: htttps::/proceeeeee http:///ooreeeddiieeoo www.eeeddiiiinnnnnieeeeeee www.iieuripe. http://g....aaaaos.o....p/epep//
pdf
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced re- search). 2009. URL www.tRtt://////. http:://ww/ cs. wwr.nto.o.ouu.... http://www.....tooo..eeeeeee
wttrronnooeddd/krrr///// wwt.:///u.uiiiaarraaraaaaaaaaaaa .html
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image
generation using residual quantization. CoRR, abs/2203.0194/ 2022. doi: 10.48550/arXi.22203. 01941. URL http:tt://ooo.oog/////// http://.........../...0000000000
40544/850///// /a..5///a/iivvv/. 2203.0194)
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2022. URL httptt///////ii...ot org/ wrt./bs////aao....ooooo......../ aallab///22222 12210.02747
Qiang Liu. Rectified flow: A marginal preserving approach to optimal transport, 2022. URL
https: htts:///axxv..cro abs// 220.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow, 2022. URL ftttps://axxxi org. attpaab//.2..000000000
Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likeli- hood training for score-based diffusion ODEs by high order denoising score matching. In Ka- malika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 14429-14466 PMLR, 17-23 Jul 2022. URL https::/ppoceeeinng www.siinnssm.....pmmmmmm.m.....e m.r.press.ss...... v1 v262/ 1u22f. html
Jacob Menick and Nal Kalchbrenner. GENERATING HIGH FIDELITY IMAGES WITH SUB- SCALE PIXEL NETWORKS AND MULTIDIMENSIONAL UPSCALING. In International Conference on Learning Representations, 2019. httptttsss//ppeeeetteee nwtp//ne//ttteeeeeee fformm?
Id=Hy12TiC5Kmm
Dan Mikulincer and Yair Shenfeld. On the lipschitz properties of transportation along heat flows, 2022. URtttttps::
www.vorrg/ae. abs 2201 .2.01..1100 01382)
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference Machine Learning, ICML'10, pp. 807-814, Madison, WI, USA, 2010. Omnipress. ISBN 9781 1005589077
Kirill Neklyudov, Daniel Severo, and Alireza Makhzani. Action matching: A variational method for learning stochastic dynamics from samples, 2022. URL htttps://axxi..rrg// ortpa///rrga/b//22//11
06662
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic mod- els. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Confer- ence Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8162-80 8171. PMLR, 18-24 Jul 2021. URL httl:/ttt//pooceeeeeeeet ni 21a. html
mlr wtw..pprrsss/////
12

---

Published as a conference paper at ICLR 2023
Maria-Elena Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In IEEE Conference on Computer Vision and Pattern Recognition, volume 2, pp. 1457,4454 2006.
Derek Onken, Samy Wu Fung, Xingjian Li, and Lars Ruthotto. Ot-flow:
Fast and accurate tinuous normalizing flows via optimal transport. Proceedings of the AAAI Conference on Ar- fificial Intelligence, 35(10):9233-923 92322, May 2021. doi: 10.1609/aaai w35i1......5110.1.11.. URL
https://ojs.co aaai. org/ index.
AAAI/ aAAI/articless /AA//article/view//1111
George Papamakarios. Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Proceedings of the 31 International Conference on Neural Information Process- ing Systems, NIPS' 17, pp. 2335-2344 Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN
9781510860964.
Stefano Peluchetti. Non-denoising forward-time diffusions, 2022.
URL https://
openreview foo.mmiidooViiuuhrt.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. URL https://arxiv.ort http://xxxiv..r/abb/////
2224.06125
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL httpppps///cccoo.........o.o www.iiin.....eu..s.....e.......e wwtpaaps.ccppppppppppe cctpapper/2199//1le.... 5f8e2fal 1el2
pdf
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learn- ing, volume 37 of Proceedings of Machine Learning Research, pp. 1530-1538, Lille, France, 07-09 Jul 2015. PMLR. URL /tL/itttp///prcediingg wwtpengssmm.....m html
mww. press /v2/ vit.////e/v/r/eeeeeee1515
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Noam Rozen, Aditya Grover, Maximilian Nickel, and Yaron Lipman. Moser flow: Divergence- based generative modeling on manifolds. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:://ppenevviet net iot.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Sal- imans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image dif- fusion models with deep language understanding, 2022. URL https:///axxii wwwpa/brrrgaabb. abs/ 2205. 7705.11487
Filippo Santambrogio. Optimal transport for applied mathematicians, Birkäuser, NY, 55(58-63):994, 2015.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Pro- ceedings of the 32nd International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 2256-2265, Lille, France, 07-09 Jul 2015. PMLR. URL wt..L..ree///////0/0 wwc.hhiiiiksiiinn.i... html
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 1428. Curran Associates, Inc., 2021a. U neer.... http:////2....aaaaooooooooooaaan
13

---

Published as a conference paper at ICLR 2023
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021b. URL https:// net/
Corum?id=PxTI
2RRHS
E. Tabak and Cristina V. Turner. family of nonparametric density estimation algorithms. Com- munications on Pure and Applied Mathematics, 66(2): 164, 2013. doi: https//doi.orr/100 1002/cpa.21423.
URL htttps://emimeelizarry www.y..wee................ com/ doi/
cpa 211221255
Esteban G. Tabak and Eric Vanden-Fijndenn. Density estimation by dual ascent of the log-likelihood. Communications in Mathematical Sciences, 8(1):217 233, 2010. doi: cms/ 26693500. URL
https::/dbi.org/
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33rd International Conference on International Conference Machine Learning Volume 48, ICML. pp. 1747-1756. JML..org. 2016.
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference on Learning Representations, 2022. URL https://openrevalt net/ wttpeeerreeeeeeeee/oouum....eeee
A BACKGROUND ON TRANSPORT MAPS AND THE CONTINUITY EQUATION
The following result standard and can be found e.g. in (Villani 2009 Santambrogio 2015)
Proposition A.1. Let pe(x)) satisfy the continuity equation
Sept + (VaPt) 0.
(A.1)
Assume that ve(x) is C in both t and for 2 0 and globally Lipschitz in Then, given any I, l' 2 0, the solution of (A.I) satisfies
pc(x) pr Pu (Xe,e())) exp (- (-f" V (xs((Xeo())ddd
(A.2)
where Ns is the probability flow solution to
d (x)(1)) (x))),,(x)),
(1,)(0)=
(A.3)
dt
In addition, given any test test function Φ Ω R, we have
[ (()) = [ o ((Xx, (z)dx.
(A.4)
In words, Lemma A. states that evaluation of the PDF Pt at given point may be obtained by evolving the probability flow equation (2) backwards to some earlier time t' to find the point x' that evolves to x at time assuming that pe (x') ) is available. In particular, for L' = 0, we obtain
pc(x) = pol (o(Xx.(x)) exp (-f% V.-vs(Xe, (x((e.(())dd)
V-
(A.5)
and
I. ((=))(())e / ((No.//()) ((X)rcl)))
(A.6)
Proof. The assumed C and globally Lipschitz conditions on nn guarantee global existence (on 0) and uniqueness of the solution (2). Differentiating (x)) with respect to and using and A.D we deduce
de (((e,((())) d, (xp((Xe,(x)) d di Xx,(w) vn((X,,(()) 3.p((Xe,:(t)) (x),,((()) -vx(Xe.)))
d
(A.7)
14

---

Published as a conference paper at ICLR 2023
Integrating this equation in from t' to t gives
px(Xv,(1=) Pe (x) exp
(A.8)
Evaluating this expression at = (,r((()) (a) and using the group properties (i) X (x,,, (a)) =
and (ii) Xe, (Xe,, = (z) gives (A.2). Equation (A.4) can be derived by using (A.2)
to express (x() in the integral the left hand-side, changing integration variable
x
and noting that the factor exp (- V us (z))ss is precisely the Jacobian of this change of variable. The result is the integral at the right hand-side of (A.4.
B PROOF OF PROPOSITION
We will work under the following assumption:
Assumption B.1. The densities po(x) and (1( are continuously differentiable x; Le x1) is continuously differentiable in (t, T1) and satisfies (4) and (7): and all E [0, 1] we have
J/c Vee 21 (o(x))p (Ir )drodd dk < oc
(B.1)
[ See Sele(wo, 21
)e de (e po( ( (x1)dr dk < 0o
This structural assumption guarantees that the stochastic interpolant Ty has probability density and probability current. As shown in Appendix C it is satisfied e.g. if po and p1 are Gaussian mixture densities and
I [2 21) a, + b
(B.2)
where a and are O function of [0, 1] satisfying
à S 0, b. 2 0, ag 1, an 0. bo = 0, by
(B.3)
> 0 on / E [0, 1), b. >
0 1 E (0,
The interpolant in this class for the choice
a₂ c
b, - sin
(B.4)
Our proof of Proposition ! will rely on the following result that quantifies the probability density and the probability current of the stochastic interpolant x, defined in (6):
Lemma B.1. If Assumption B.l holds, then the stochastic interpolant x, defined in 6 has prob- ability density function (z(x) given by
px(x) (2x)-d
c -o- (-to-e--1(zxzz (((20.1))) po(x)) (x) )dzodz dk
(B.5)
Jud xE=xR=xxB xBd
and it satisfies the continuity equation (.p((z) + V j((x) = O,
p==1(x) = (1
(B.6)
withe the probability current je(x) given by
je(x) = -2 [ J 0.1e(xo, x Di)e ihl () (z0))) (x1))odd dk
(B.7)
In addition the action of pe(x) and ja(x) against any test function & Rd R can be expressed
[ (()))=(z)dr = J (T (1) ddnder
(B.8)
[ (x))i(Z)de== f J x (1)())))))(((()))))))))
(B.9)
15

---

Published as a conference paper at ICLR 2023
Note that (B.8 and (B.9) can be formally rewritten 12 and (4 using the Dirac delta distribution.
Proof. By definition of xy in (6). the characteristic function of this random variable
E cxp mx)]
cdk-Ielmama)) pa( (21)dradr
(B.10)
Under Assumption B.1 the Fourier inversion theorem implies that has density pr(a) given by B.5). Taking the time derivative of this density gives
(1p)(1) = (2xr)
ik (, xz ( th to-l--l--aaaaee (1)) (a(z)) (2) ddrodd dk
(B.11)
= (x)
with je (x: given by (B.7
Lemma B. shows that (z() satisfies the continuity equation (3) with the velocity field vx(x) defined in (T).. also show that the objective function G( in (9) well-defined, which implies the first
part Proposition
For the second part, we will need
Lemma B.2. If Assumption B. holds, then
S/' fee (e(x))P((x)daaa
(0(0 < o
(B.12)
Proof. For K oc, define
[1
if [xx(x) < K
(x)
10
(B.13)
clse
Then, using the pointwise identity v. (x())(((r) (x) well as B.8 and (.9)). we can write
[ J of (2)o(()))) 20.(x)) je(x)) drdt
20
[o (4)).4. [ee)
(B.14)
= (4)) (4))o())) - 0,1e1
- (1))01))
E E( (1))l (
where we use the shorthand I, [, 21) and Ü. O,To .11. Therefore
0 < E (T, [T.) < B o ( (T)))
(B.15)
Since limx-+o E oo (4.) = | and this quantity is finite by assumption we deduce that lim E o (I.)) exists and is bounded by E [110.1113].
Lemma B.2 implies that the objective H(0)
17 well-defined, and the second part of statement
of Proposition i follows from the argument given after (17). For the third part of the proposition we can then proceed as explained in main text, starting from the Poisson equation B.2I
Remark B.3. Let us show on simple example that the inequality TE [ [Te x1 [] S [( [ not saturated in general. Assume that pu(x) is Gaussian density of mean zero and variance one, and P1 (x) a Gaussian density of mean m E and variance one. this
case, if the trigonometric interpolant (5). (C.4) indicates that pr(z) is Gaussian density
with mean sin( ( Tt and variance one, and (C.6) simplifies to ve(x) = 1 m² cos? ( Tt),
(t so that
(c = 2 cos (2xrt
(B.16)
16

---

Published as a conference paper at ICLR 2023
At the same time
10.T( xi)) po(r))m (11))rdde1
)dvodo
Rx
{
-1
+
(21)dzodr1
Rxi
(B.17)
1 nT? (sin" (2 cos c (1 (1+))))
7"d (1 m² cos ( (t)
and BlI 20 )) |. ( mn ))] 1x²² (F0,
The interpolant density (x)) and the current je (x) given explicitly in Appendix C in the where and P1 are both Gaussian mixture densities and we use the linear interpolant (B.2)
Notice that we evaluate v== and (x)) more explicitly. For example, with the linear interpolant B. we have
)==0(z) (x) bopo(x) / (11((x))dr1,
(B.18)
ji=1 (2) = (x) + (1))
roPo(zo)dro-
3d
From 12). this implies
vx āgæ + bo / d1P1(z1)dz1,
ia (4) = b1æ à1 L zopo(zo)dao
(B.19)
For the trigonometric interpolant that uses B.4 these reduce to
(== 1 [ x1P1 (21 (d1)d1, (=) 1 Rdd
2opo(1o)ddo-
(B.20)
Finally, we note that the result of Proposition remains valid if we work with velocities that are gradient fields. We state this result
Proposition B.4. The statements of Proposition i hold if G(o) is minimized over velocities that are gradient fields, in which case the minimizer of the form (x(r) Vol(()) for some potential 6: Fd + R uniquely defined up constant.
Minimizing G(o) over gradient fields (x(x) (x) guarantees that the minimizer does contain any component 0e (z) such that V V- (0)( = O. Such component of the velocity has no effect on the evolution of (z(x) but affects the map Xt, the solution of (2). We stress however that: even if we minimize G(θ) over velocities that are not gradient fields, the minimizer v, (x)) produces a map X, via that satisfies the pushforward condition
Proof. Define the potential di Rdd R as the solution
Vo, = V -.
(B.21)
with pr( and je (x) given by (12 and (14.. respectively. This is a Poisson equation for 6.(x) which has unique (up to constant) solution by the Fredholm alternative since pe(xr) and je je(x) have the can therefore be written as B
same support and fge V j((x)dx= 0. In terms of (x()),
O.p: + Vp.)
(B.22)
The velocity (o)(2)) ) is also the unique minimizer of the objective 9 over gradient fields since if we set (x) (xx) and optimize G (V²) over à. the Fuler-Lagrange equation for the minimizer precisely the Poisson equation (B21). The lower bound on the objective evaluated at Vœr(x) still holds since the argument above involving (17) and (18) can made by replacing the identity
vy (x))p, = je(z) with Sgo Vor(z). V (z) (z)) S&a Vdr(x) ja je(w)dz, which is BB21 written in weak form.
17

---

Published as a conference paper at ICLR 2023
Interestingly, with ey (2)) defined in (15) and (x()) defined the solution to (B.21) have
f
/ (e
2
(B.23)
consistent with the fact that the gradient field (ér(x) is the velocity that minimizes Jau (x))) over all (z(z) such that dep, + V (6ept) with pe(z) given by (12). We stress however that, even if we work we gradient fields, the inequality (2T) is not saturated in general.
Remark B.5. we assume that ür (x) (v)()), can write the objective
16 as
G(VD) - S/¹ | (1V()())PP) - 2v²((z) ix(()) drdt
I, [ (1V (1N je(1)) drdt
I / (1V6,(.)))) (x dedl
(B.24)
! J (IV((())P)))))) + (z) drdt
+ 2 Rd (d=0(=))o(x)-
(z)) (o dr,
where we integrated by parts in to get the second equality, we used the continuity equation 13 to get the second, and integrated by parts in get the third using Pt=0 Po and pt=1 = P1. Since the objective in the last expression is an expectation with respect to pt. we can evaluate it
G(V = (IV6,(1)
(00)==(((((())))) + ( 22)))(44)) + 2Eg2r= - 2:
(B.25)
where E, and Eg denote expectations with respect to P1 and po, respectively. Therefore, we could use (B.25) alternative objective to obtain ve(x) = (x) by minimization. This objective is, up to sign and a factor 2, the KILBO objective introduced in Neklyudov et al. (2022). Notice that
using the original objective in (9) avoids the computation of derivatives and t, and allows one
to work with (x) directly.
C THE CASE OF GAUSSIAN MIXTURE DENSITIES
Here we consider the case where Po and # are both Gaussian mixture densities. We denote
(z/mm, C) - (det 11-1/2 exp (- [= ( 30-mm m))
(C.1)
=2T) -d e (x- dk:
Rd
the Gaussian probability density with mean vector Rd and positive-definite symmetric covari- ance matrix C C=C² C E Rd x Rdd We assume that
No
N
po(x) = Σ
p. Nx
(I(x) = Σ
p. N (cllm,,.c) c
(C.2)
where No. N2 N. p. 0 with I.oo pl 1, m Rd., C. (C.)T" E Rd Rd. positive- definite, and similarly for pt. my. and C.. We assume that the interpolant of the form B. and we denote
i) = amm + bem). mi
c = of RCC.
i= 1 No.
j= 1, N (C.3)
Note that if all the covariance matrices the same, C. = CJ C. with the trigonometric inter- polant in (5 we have Ciü C. which justifies this choice of interpolant.
We have:
18

---

Published as a conference paper at ICLR 2023
Proposition C.1. The interpolant density Pi ((()) obtained by connecting the probability densities in (C.2) using the linear interpolant B.2 is given by
Pa(
(z C.²)
(C.4)
and it satisfies the continuity equation (xp:(x) + [x( = 0 with the current
No
N₂
m )x(z) - -(z)=>E> p.p
15 + 1 (c (z 1amm-mm))) m,²1) N(alm C.)
(C.5)
This proposition implies that
Σ No N1 1 p.
(m + 1,, (c Marm )J)
De (z(2)
(C.6)
Σ N1 p.p ((r CP))
This velocity field is growing at most linearly in x, and when the mode of the Gaussian are well separated, in each mode it approximately reduces to
t (C (x- (m")))
(C.7)
Proof. Using the Fourier representation in C.1 and proceeding as in the proof of Lemma B. deduce that (x( is given by
No
ckl(aa mi) 2 Cftk dk.
(x = (2Tr)
P.P
(C.8)
Performing the integral over la gives
(C.4.
Taking the time derivative of this density gives
No
N
(ik mi + ( k)) st (x- - 2 dk
@epi(z) (2xx -
PiPj
i=1
1=1
JRd
No
N
{ Jen
ik (m) 110k5k) ( mit) -fr o dk
(C.9)
= -2xx) -d Σ
P.P.
0
1=1
- (e(x))
with
pPp (mi - (c f cf
je(x) (2T)
Rd
No N;
pip ( (() f ct
" dk
(2m) -
im
Na N.
p.p C / (o- fo ct ddk
2 (2x)
(C.10)
No N1
Pip mi - (2 N(xmmm", cP)
pip (in?)) + (c -m))))) NaalmiH. cl
19

---

Published as a conference paper at ICLR 2023
D OPTIMIZING TRANSPORT THROUGH STOCHASTIC INTERPOLANTS
Using the velocity (x(2) in 15 that minimizes the objective 9 in the ODE gives an exact transport map X;=1 from Po to Pl. However, this map not optimal in general, in the sense that it does not minimize
/ T(x) (z) E
(D.1)
over all T such that Tipo P1. It is easy to understand why: In their seminal paper Benamou & Brenier (2000) showed that finding the optimal map requires solving the minimization problem
min / J (0,(<r))) (e,p)
(D.2)
subject to: O.p. V (Ur = 0, Pt=0= po: pt= = P1-
As also shown in (Benamou & Brenier 2000), the velocity minimizing (D.2) is a gradient field, of (z) = Vor (x),, and the minimizing couple (pt. of) is unique and satisfies
dep + V (Votpi) = 0, Pi-o - Po: Pi=1 -
(D.3)
0,0 + Vo 2 0.
In contrast, in construction the interpolant density (z( is fixed by the choice of interpolant (e(1o, x1), and pc(x) * pł (x) in general. As a result, the value of fo Sou (e)(=))))(())dddd))
([]))( for the velocity Uz (z)) minimizing (9) is not the minimum in (D2))
Minimizing p over gradient fields reduces the value of the objective in D.2p. but it does not yield an optimal map
the gradient velocity Vol (x) with the potential (x)) solution to (B.2I)
only minimizes the objective in (D.2) all 0n (x) with (xx pe (x) fixed, explained af- ter (B.33.
It is natural to ask whether our stochastic interpolant construction be amended generalized to derive optimal maps. This question is discussed next, from two complementary perspectives: via optimization of the interpolant (= 1, and/or via optimization of the base density Po. assuming that we have some leeway to choose this density.
D.1 OPTIMAL. TRANSPORT WITH OPTIMAL INTERPOLANTS
Since (10) indicates that the minimum of G(t) is lower bounded by minus the value of the objective (D2),. one way to optimize the transport to maximize this minimum over the interpolant. Under some assumption on the Benamou- Baeamou-BBenier density P (z) solution of (D.3). this procedure works, as we show now, Let us begin with definition:
Definition D.1 (Interpolable density). We say that one-parameter family of probability densities pa(x) with E [0, 1] is interpolable (in short: pr(x) is interpolable) if there exists one-parameter family of invertible maps T. Rd - Rd with E [0, 1], continuously -iiiiouuss--dfffeenttbbb space, such that pt is the pushforward by T, of the Gaussian density with mean zero and covariance identity, iee. T0,N Id) = for all I E 1).
time and
Interpolable densities form a wide class, as discussed e.g. in Mikulincer & Shenfeld (2022) These densities also the ones that can be learned by score-based diffusion modeling discussed in Sec- tion 2.2. They are useful for our purpose hecause of the following result showing that any interpo- lable density can be represented
interpolant density:
Proposition D.I. Let pe(x:) be interpolable density in the sense of Definition D. Then
(2 TT, (xa) cos( Tt) )-)) + (in)
(D.4)
satisfies 4 and is such that the stochastic interpolant defined in 6 satisfies It
We stress that the interpolant in (D.4) is in general not the only one giving the interpolable density Pe (x:), and the actual value of the map T, plays no role in the results below.
20

---

Published as a conference paper at ICLR 2023
Proof. First notice that
1t=0 x1) ( zo,
1=1(20021) T (T; (xo)) 21
(D.5)
so that the boundary condition in
4 are satisfied. Then observe that, by definition of Ti,
To (xo) (0,
20 Po
(D.6)
T (21) Id),
21 P1
This implies that, if No ~ Po. 21
Pl, and they are independent,
To (z0) cos( Ti (21) sin (
(D.7)
a Gaussian random variable with mean zero and covariance Id cd Id ( ld, Le. it is sample from (N, Id). By definition of T,, this implies that (æo) It (x z1) = Ti (To (xo) cos( Tt + + (xi) sin rt))
(D.8)
(D.9)
a sample from Pe if xg ~ po. 3ı ~ Pl: and they are independent.
Proposition D.I implies that:
Proposition D.2. Assume that (i) the optimal density function P (I)) minimizing (.2 is interpolable and (ii) D.3) has classical solution. Consider the max-min problem
max min G(i)
(D.10)
where G( is the objective in (9) and the maximum is taken over interpolants satisfying 4 Then
a maximizer of (D.10) exists, and any maximizer 1ī (10, is such that the probability density (Ty, T1), with To Pu and Iı 01 independent, is the optimal p (z), the
function of at
mimimizing velocity is u (x) Voi (z), and the pair (pt (z), (x)) satisfies D33)
The proof of Proposition D.2 relies on the following simple reformulation of D.2
Lemma D.3. The Benamou Beramer minimization problem in D.2 is equivalent to the min-max problem
(' / (L10(()))P)))))) (x(x) -(())))))))))))))))))))))))))- drdt
Jgd max min AJ
=min max (-) J (2l0((2)) - (,(x)) (x())) dadt
(D.11)
P.
subject to: Depi + V Jc O, -i= = Po. p== = P1
In particular, under the conditions on Pa and PI such that (D.2) has minimizer, the optimizer
(pt uf.5.. is unique and satisfies (x)) Voj(x), (z) Vol (x)pf with (pi of) solution
to (D3
Proof. Since D.11 is convex in e and concave in (p.j), the min-max and the max-min are equiva- lent by von Neumann's 's minimax theorem. Considering the problem where we minimize over (x)) first, the minimizer must satisfy:
(x())x((z) = =e((x)
(D.12)
Since (x) 2 0 and (x( have the same support by the constraint in (D.1I). the solution to this equation unique on this support. Using (D.12) in (DII)), we can therefore rewrite the max-min problem as
/ ( max - (' Jor subject to: 0.p: + V (Prpt) = O, i Po: P P1 This problem is equivalent to the Benamou-Brentn minimization problem in (.2)).
(D.13)
21

---

Published as a conference paper at ICLR 2023
To write the Euler-Lagrang equations for the optimizers of the min- max problem (.11 let us introduce the extended objective
L. Sec (H(((())) 0z(2)) (x)) dxdt S(' S (s()) + v (=)( drdt
(D.14)
(61 (o))
where δe : IRd + R is a Lagrangian multiplier to be determined. The Euler-Lagrange equations can be obtained by taking the first variation of the objective (D.14) over d. ò. and ú, respectively. They read
0 dept + V J Pi=o Po. Pi= P1
0 θ:0ī 2/10
(D.15)
0 -nद + V
0 = pi - j
These equations imply that 2z (z) = Voi (z). [1) solution to (D.3), as stated in the lemma. Since the optimization problem in D.11 is convex in and concave (p.j), its optimizer is unique and solves these equations.
Voi (z)) with (pī oi)
Proof of Proposition D.2
We can reformulate the max-min problem (D10
max min /.' / (2(0(())))) - (x)) (x)) dædt
(D.16)
AJ
where the maximization is taken over probability density functions (x( and probability currents (x) in (B.5) and (B.7) with I replaced by It, i.c. formally given in terms of the Dirac delta distribution as
(x(x)= f J 8(x fe( (1)) (In) )dodde)
(D.17)
(x)x) S 0,0, 11) Fe 2n (o) daade
Since this pair automatically satisfies
S.pr + V J: 0, P=-0 Po:
Pt-1 Pl,
(D.18)
the max-min problem D.16 is similar to the one considered in Lemma D.3) except that the max-
imization taken over probability density functions and associated currents that can be written as in (D.17) By Proposition D.I this class is large enough to represent pt (x) since we have assumed that p (z) is an interpolable density, and the statement of the proposition follows.
Since our primary aim here is to construct map T X i that pushes forward pn onto Pr. but necessarily to identify the optimal one, can perform the maximization over (e x1) in restricted class (though of course the corresponding map is no longer optimal in that case). We
investigate this option numerical examples in Appendix H. using interpolants of the type (B.2) and maximizing over the functions a; and b,, subject to also discussion generalizations of the interpolant that can render the optimization of the transport easier to perform. We leave the full investigation of the consequences of Proposition D.2 for future work.
bj 1, a = by = 0. In Section G we
Remark D.4 (Optimal interpolant for Gaussian densities). Note that if po and P1 are both Gaussian densities with respective mean mg E Rd and m. E Rdd and the same covariance E Rd x Rd an interpolant of the type D.4
Te 21) = ( (x0 - me) +
mm 1)= + (1 (1-1)mo + lmi, (D.19)
and a calculation similar to the one presented in Appendix[[shon that the associated velocity field we (x(x)) is
vx(x)) = (m, - mp)
(D.20)
This is the velocity giving the optimal transport map Xx x. + (m1 mn)t.
22

---

Published as a conference paper at ICLR 2023
where approach similar to ours using the linear interpolant Ie = 1o(1--10000 -t) + æit is introduced, an alterative procedure is proposed to optimize the transport. Specifically, it is suggested rectify the map T X learned by repeating the procedure using the new interpolant 5i xo - t) + T(xo)t with in ~ Po- As shown in (Liu (022) iterating on this rectification step yields successive maps that are getting closer to optimality. The main drawback of this approach is that it requires each of these maps (including the first one) be learned exactly, i.e. we must have TEpo Pt to use the interpolant x/ above. If the maps are not exact, which is unavoidable in practice, the procedure introduces bias whose amplitude will grow with the iterations. leading to instability (as noted in Liu al. (2022): Liu 2022
Remark D.5 (Rectifying the map). In (Liu et al. 2022 Liu 2022),
D.2 OPTIMIZING THE BASE DENSITY
While our construction allows one to connect any pair of densities Pe and pt, the typical situation of interest is when P1 is complex target density and we wish to construct generative model for this density by transporting samples from a simple base density Pa. In this case it is natural adjust parameters in po optimize the transport via maximization of G(v) ming G(o) over these parameters-indeed changing Po also affects the stochastic interpolant xy defined in (6) and
hence both the interpolant density (z( and the velocity (e(x). For example we can take po to be
a Gaussian with mean m E Rd and covariance C' E R x RR and maximize G(v) = ming G(o) over m and C. This construction is tested in the numerical examples treated in Appendix We also discuss how to generalize it in Section G
Optimizing Pu only makes practical sense if we do so a restricted class, like that of Gaussian densities that we just discussed. Still, we may wonder whether optimizing Po over all densities would automatically give Po = / and i 0. If the interpolant is fixed, the answer no, in general. Indeed, even if we set Po = pl, the interpolant density will still evolve, i.e. P: o Po except at t = 0, I, in general. This indicates that optimizing the interpolant in concert with po is necessary if we want to optimize the transport.
E PROOF OF PROPOSITION 3
To use the bound in 27 let us consider the evolution of
Qi Q== f (X((x) - (
(E.1)
Using Xx(x) (x)))(z)) and (x(x)) (8((())) we deduce
Q. 2 (X((x) {((x)) (x)(X(())) (x((((()))
2 (X,(x) x.(x)) (x)(((())) (o)
(E.2)
+ 2 JR
(Xi(x) (((1)) (x)(
Now use
2(X, - 8,) (v,(X.) (x,)) $ [x, -x,1- + 1e - 62 (X..)
(E.3)
and
2(X,-X;
- -x.) -(,(())) - (x.)) 221|x, - X,P
(E.4)
obtain
② (1+ (1+1k)) + / (x.()x))) 6 (Xx(2)))
(E.5)
Therefore, by Gronwall's inequality and since Qo we deduce
Q S cet22 S/' _g
- (x) 1122 H()).
(E.6)
Since W3( < <Qi by (7 are done.
23

---

Published as a conference paper at ICLR 2023
Note that the proposition suggests regularize G()) using e.g.
/,"' / [ii(()) G(o) + AE (B (VV 2n m))l
Gx G(t)
(E.7)
Rdd
with some small > 0. In the numerical results presented in the paper no such regularization was included.
F PROOF OF PROPOSITION 4 AND LINK WITH SCORE-BASED DIFFUSION MODELS
Assume that the interpolant is of the type (B.2) so that (z 21) à,30 bez For t E (0,1) let us write expression (14) for the probability current as
{
je(z)
(ā,T0 + b(z1))(xx a,o0 + b, JRdd x
/
by (aro + +bar2) + (ās baa)) 5(x a+To b2r1))o
by
b,
b,
by
be as) xx((2) + (is
2uδ(I ao + (2)
= b
be /xxx"
(F.1)
d/2e If Po(an) (2m)-d/2 e t|2al we have the identity xxPo -V 220Po(00) Inserting this equality in the last integral in (F.I) and integrating by part using
Va 5(x a,30 + b,o1) =a V, a + b1
(F:2)
gives
by xx(())) ad aa( b,
by au (p((x)
(c(
(F.3)
This means that
bn x at (ar
dn az log (x
0y (x)
(F44
b
by
Solving this expression in log (x() and specializing given in (B.4) gives the first equation in (28) The second one can be obtained by taking the limit of this first equation using (x) = from B.20 and l'Hôpital's rule.
the trigonometric interpolant with a, b
Note that F shows that, when the interpolant of the type (.2) and Po(2)) = (2-x)-d/2g d/2 Imol? the continuity equation can also be written the diffusion equation
b
bc
(xp V (xpr(x)) as ( by
Api(1)
(F.5)
be
Since we assume that 0 and by 2 (see B.3 the diffusion coefficient in this equation is negative
b < 0
at à;
by a1
(F.6
This means that F.5 is well-posed backward time, i.e. it corresponds to backward diffusion from Pt=1 = PI to Pt=o Po -d22 =lall" Therefore, reversing this backward diffusion, similar what done score-based diffusion models, gives an SDE that transforms samples from Po to P1. Interestingly, these forward and backward diffusion processes arise on the finite time interval & E [0, 1]: notice however that both the drift and the diffusion coefficient are singular at t == 1. This is unlike the velocity vz(x) which is finite t 0, 1 and is given by (B.19).
G GENERALIZED INTERPOLANTS
Our construction can be easily generalized in various ways, e.g. by making the interpolant depend on additional latent variables to be averaged upon. This enlarges the class of interpolant density we can construct, which may prove useful to get simpler (or more optimal) velocity fields in the continuity equation B Let consider specific generalization of this type:
24

---

I (ao, 21) and 0.1 O. (zo, addition the Published as a conference paper at ICLR 2023
Factorized interpolants.
Suppose that decompose both pu and P1
k
po(x) Σ (x) p1(x)
Σ k=1
Pupl
(z),
(G.1)
where K E N, pú and of are normalized PDF for each h: == =b=1,... I,..., K, and ph > 0 with Σ
A= =
1. We can then introduce K interpolants 1 (2), with k = I,.... K, each satisfying and define the stochastic interpolant
27 = 21).
k P
20 Po σ1 ~ p
(G.2
This corresponds to splitting the samples from Po and into K (soft) clusters, and only interpo- lating between samples in cluster in Po and samples cluster k in ft. This clustering can either be done beforehand, based on some prior information we may have about po and Pt, or be learned (more on this point below).
It is easy to see that the PDF Pr. of is formally given by
(x = Σ Ph
8(x
1 (No, (1 (1)0 (x1dddd
(G.3)
k=1
and that this density satisfies the continuity equation (B.11) for the current
K
/
je(z) = Σ 22
(,1, 1 x (za) (z) ddzddr
(G.4)
Therefore this equation can be written as (3) with the velocity (x) which is the unique minimizer of generalization of the objective (9. We state this
Proposition G.1. The stochastic interpolant Ie defined in G with (xa, v1) satisfying 44 for
each k 1....,R K has a probability density (e(x)) that satisfies the continuity equation 3 with velocity (xx)) which is the unique minimizer over de ve(x)) of the objective
Gx(o)= - Pk S. Se (10((11))= -20,1 (((11))) (2: ddrddr
(G.5)
k=
where we used the shorthand notations It minimum value of this objective is given by
Gk(v) = Gk(u)=-
J [vx(2)) px)))ddt 000
(G.6)
and both these statements remain true if G(ū) minimized over velocities that are gradient fields, in which case the minimizer is the form ve(2) ())(x)) for some potential de .Rd R uniquely defined up to constant.
We omit the proof of this proposition since it is similar to the one of Proposition 1 The advantage of this construction is that it gives the option to make the transport more optimal by maximizing (u( over 1 and/or the partitioning used to define P pf. and Py- For example, we know that define Pu as a Gaussian mixture
the target density P1 has K clusters with relative mass Pk, with K modes, set the weight of mode to Pk- and maximize Gx(z) = ming Gx (6) over the mean my dd and and the covariance Cg E Rdd x Rd of each mode k 1...., K in the mixture density Po-
H EXPERIMENTS FOR OPTIMAL TRANSPORT, PARAMETERIZING It, PARAMETERIZING Po
As discussed in Section 2 the minimizer of the objective in equation 100 can be maximized with respect to the interpolant I, as a route toward optimal transport. We motivate this by choosing a parametric class for the interpolant and demonstrating that the max-min optimization in equation
25

---

Published as a conference paper at ICLR 2023
D.10) can give rise to velocity fields which easier to learn, resulting in better likelihood estima- tion. The 2D checkerboard example is an appealing test case because the transport is nontrivial. In this case, train the same flow in Section 3.1 with and without optimizing the interpolant. We choose a simple parametric class for the interpolant given by a Fourier series expansion
(zo, 21) = 0,30 + b
(H.1)
where parameters {a, 3) define learnable âr, b via
M
M Σ m==1
à = cos (t) +
cos
am (mmtt, M
Σ m=1
6, = sin (
+
B. sin (mmt).
(H.2)
M
This is but one spline, so that the endpoints for ût, b, can be properly constrained as they are above in the Fourier expansion. In Figure H.1 we plot the log likelihood, the learned interpolants â, by compared to their initializations, as well as the path length it evolves over training epochs. With the learned interpolants, the path length is reduced, and the resultant velocity field under the same number of training epochs endows model with better likelihood estimation, shown in the left plot of the figure. For the path optimality experiments, M = 7 Fourier coefficients were used to parameter- ize the interpolant. This suggests that minimizing the transport cost create models which are, practically, easier to learn.
possible parameterizations. Another, for example, could rational quadratic
log-likelihood
åc 6
[[v+(x)]
3.4
6.0
1.0
-3.5
MyMp Wwpl
5.0
nonparametric 4 parametric f,
0.4
4.0
0.2
-40
0.0
-41
3.0
5000
10000
0.0
0.5
1.0
5000
10000
Epoch
Epoch
Figure H.l: Comparison of characteristics and performance of the Iearned nonparametric in- terpolant on the checkerboard density estimation. Left: Comparison of the evolution of the log- likelihood. Middle: The initial versus learned interpolant terms à, b,. Right: The path length of the learned nonparametric interpolant.
In addition to parameterizing the interpolant, one can also parameterize the base density Po in some simple parametric class. We show that including this in the min-max optimization given in equation (D.00) can further reduce the transport cost and improve the final log likelihood. The results are given Figure H.2 We train an interpolant just as described above, but also allow the base density Po to be parameterized as a Gaussian with mean f and covariance Σ. The inclusion of the Iearnable base density results in a significantly reduced path length, thereby bringing the model closer to optimal transport.
I IMPLEMENTATION DETAILS FOR NUMERICAL EXPERIMENTS
Let { be samples from the base density Po. { n samples from the target density Pl, and {4*] K samples from the uniform density on (0, 1]. Then an empirical estimate of the objective function in 9 is given by
K
GN), KON
1
ΣΣΣ
(2 Or (Io, (26. (xbr 20,1, (x), Or (1e (1g.
(L.I)
k=1j=1
This calculation is parallelizable.
26

---

Published as a conference paper at ICLR 2023
log-iikelihoad
år, D
of Āo - -N(()
F: [[v+-0013]
-3.5
My
3.8
nonparametric Ps parametric
-3.8
2
-4.1
3.0
5000
-5.0 0.0
Epoch
Epoch
Figure H.2: Comparison of characteristics and performance of the learned nonparametric in- terpolant on the checkerboard density estimation, while also optimizing the base density Āo- The parametric ýu is given as bivariate Gaussian N (f, Σ). Left: Comparison the evolution the log- likelihood. Center left: The initial versus learned interpolant terms â, by. Center right: The learned covariance Σ in red compared to the original identity covariance matrix. Right: The path length of the learned vs nonparametric interpolant.
POWER
GAS
HEPMASS
MINIBOONE
BSDS300
Dimension
6
21
43
63
# Training point
1, 615, 917
852,174
315,123
29,556
1. 1,000, 000
Target batch size
800
800
800
800
300
Base batch size
150
150
150
150
200
Time batch size
10
10
20
10
20
Training Steps
105
10a
10
105
103
Learning Rate (LR)
0.003
0.003
0.003
0.003
0.002
LR decay (4k epochs)
0.8
0.8
0.8
0.8
0.8
Hidden layer width
512
512
512
512
1024
# Hidden layers Inner activation functions Beta a, B, time samples
4
4
4
5
ReLU
ReLU
ReLU
ReLU
ELU
(1.0,0.5)
(1.0,0.5)
(1.0,0.5)
(1.0.1.0)
(1.0,0.7)
Table 3: Hyperparameters and architecture for tabular datasets.
The architectural information and byperparameters of the models for the resulting likelihoods in Table [2 presented in Table 3 ReLU (Nair & Hinton, 2010) activations were used throughout, barring the BSDS300 dataset, where ELU (Clevert et al., 2016) was used. Table formatting based on (Durkan et al. 2019).
In addition, reweighting of the uniform sampling of time values in the empirical calculation of (L.1) was done using a Beta distribution under the heuristic that the flow should be well trained near the target. This is in line with the statements under Proposition | that any weight w(t) maintains the same minimizer.
The details for the image datasets are provided in Table 4 We built our models based off of the U-Net implementation provided by lucidrains public diffusion code, which we are grateful for We use the sinusoidal time embedding, but otherwise use the default implementation other than changing the U-Net dimension multipliers, which are provided in the table. Like in the tabular case, reweight the time sampling to be from a beta distribution. All models were implemented on a single A100 GPU.
https://github.. com womm///cidraiins/ddnoiiinnng
I.1 DETAILS ON COMPUTATIONAL EFFICIENCY AND DEMONSTRATION OF CONVERGENCE GUARANTEE
Below, we show that the results achieved in 3 are driven by model that can train significantly more efficiently than the maximum likelihood approach to ODEs. Following that, we provide an illustration the convergence requirements on the objective defined [TI.
27

---

Published as a conference paper at ICLR 2023
CIFAR-10
ImageNet 32x32
Flowers
Dimension
32x32
32x32
x28 128
# Training point
5 x 104
1,281,167
315,123
Batch Size
400
512
50
Training Steps
5 10
6x 105
1.5 105
Hidden dim
256
256
256
Learning Rate (LR)
0.0001
0.0002
0.0002
LR decay (Ik epochs)
0,985
0,985
0,985
U-Net dim mult
[1,2,2.2,2] (1.0,0.75)
[1,2,2,2]
[1,1,2,3,4]
Beta a, 8, time samples
(1.0,0.75)
(1.0.0.75)
Learned sinusoidal embedding # GPUs
Yes
Yes
Yes
1
Table 4: Hyperparameters and architecture for image datasets.
Triining time compatison
lig ple) comparison
We briefly describe the experimental setup for test- ing the computational efficiency of model as compared to the FFJORD maximum likelihood learning method. We use the tectures for the interpolant flow and the FFJORD) implementation, taking the architectures used in that paper. For the Gaussian case, this is a 3 layer neu- ral network with hidden widths of 64 units; for the 43 -i-dimensional MiniBooNE target, this is 3 layer neural network with hidden widths of 860 units.
network archi-
223
Dus Gaaden
Duns Mirioon mepRoo Minbsors
mypro
Figure L1: Left: Training speed for ours vs. MLE ODE flows, with 400x speedup on the MiniBooNE. Right: InterFlow shows more efficient likelihood ascent.
Figure I.1 shows a comparison of both the cost per training epoch and the convergence of the log likeli- hood across epochs. We take the architecture of the vector field as defined in the FFJORD paper for the 2-dimensional Gaussian and MiniBooNE, and use it to define the vector field for the interpolant flow. The left side of Figure I.I shows that the cost per iteration is constant for the interpolant flow, while grows for MLE based approaches as the ODE gets more complicated to solve. The speedup grows with dimension, 400x on MiniBooNE.
The right side of Figure [.I shows that, under equiva- lent conditions, the interpolant flow converge faster in number of training steps, in addition to being cheaper per step. The extent of this benefit is dependent both hyperparameters and the dataset, so a general state- ment about convergence speeds difficult to make. For the sake of this comparison, we averaged over 5 trials for each model and dataset, with variance shown shaded around the curves.
Convergence metric for POWER cata
stifled Ios Givl
G-Ciut+ ppotteaappaa
Ma/
As described in Section [ the minimum of G(D) is bounded by the square of the path taken by the map Xx(x). The shifted value of the objective G(O) in (1I can be tracked to ensure that the model velocity (x()) meets the requirement of the objective. It must be the case that (()) 0 if (x) is taken to be the minimizer of G(i), so we can look for this signature during the training of the
Epoch
Demonstration of the convergence diagnostic on POWER dataset. This is necessary but not suf- ficient for convergence. See Section 2 for definition of G.
Figure 1.2:
interpolant flow. Figure I.2 1. displays this phenomenon for interpolant flow trained on the POWER dataset. Here, the shifted loss converges to the target G(v) = 0 and remains there throughout train- ing. This suggests that the dynamics of the stochastic optimization of G(0) are dual to the squared path length of the map.
28

---

Published as a conference paper at ICLR 2023
Figure 1.3: Uncurated samples from Oxford Flowers 128x128 model.
Figure I.4: Uncurated samples from ImageNet 32x32 model.
Figure L.5: Uncurated samples from CIFAR-10 10 model.
29


Reviews:
[{"review_id": "zXT7v2I8nR", "reviewer": "Reviewer_xdhs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to train continuous normalizing flows by specifying a continuum of distributions using an interpolation function. The neural network model is trained to estimate the expected velocity of the interpolation function though MSE minimization. This training approach does not require numerical simulation of ODEs, and is therefore much more efficient compared to conventional maximum likelihood approaches. The method can be viewed as a generalization of score-based diffusion models to non-Gaussian noise. Experiments on toy datasets and tabular datasets confirm that the resulting continuous normalizing flows obtain competitive performance.", "summary_of_the_review": "Overall nice discussion of a clever idea for training continuous normalizing flows (generative neural ODEs) without ODE simulation. However, the method is not entirely novel since it is a special case of an existing paper. Experiments are not comprehensive and more comparisons are needed to understand the trade-offs with existing simulation-free approaches.", "strength_and_weaknesses": "## Strength\n\n1. The proposed formulation is very clear and easy to understand. The proof is easy to follow.\n2. The results of Wasserstein bounds offer valuable insights.\n\n## Weaknesses\n\nI have major concerns regarding novelty and the discussion of relevant work in this paper:\n\n1. The motivation of training continuous normalizing flows without requiring numerical ODE simulation is not new. The same challenge has been tackled in many previous works, such as [1][2][3]. In particular, the closest work to this paper is perhaps ScoreFlow [3], which leverages score-based diffusion models to design a simulation-free approach for maximum likelihood training of continuous normalizing flows. This paper needs to include a comprehensive discussion on what sets this work apart from existing ones. For example, is there anything that was impossible before now becomes tangible with the new method? Does the performance improve in comparison to other simulation-free approaches? Additional comparisons with prior methods, such as ScoreFlow in [3], need to be included in the experimental section (Figure 2, Table 1,2 and Figure 4).\n\n2. The proposed stochastic interpolant method is not entirely new. It can actually be considered as a special case of the result in [4], where SDEs are replaced by ODEs (by setting the diffusion coefficient to zero). Specifically, the interpolation function $I_t(x_0, x_1)$ in this work defines an ODE connecting $x_0$ and $x_1$, with the form $x'_t = \\partial_t I_t(x_0, x_1)$. By averaging this ODE over the joint distribution of $(x_0, x_1)$, we can obtain another ODE that smoothly converts $\\rho_0$ to $\\rho_1$. This ODE can be derived with the techniques in [4], given by $x'_t = E[\\partial_t I_t(x_0, x_1) | I_t(x_0, x_1)]$, which is equivalent to the velocity field $v_t( I_t(x_0, x_t))$, the target to be learned in this work.\n\n3. The proposed stochastic interpolant approach was also implied by the work [5] (see Appendix D). This paper needs to clarify the connection.\n\n## References\n\n[1] Rozen, Noam, et al. \"Moser flow: Divergence-based generative modeling on manifolds.\" Advances in Neural Information Processing Systems 34 (2021): 17669-17680.\n\n[2] Ben-Hamu, Heli, et al. \"Matching normalizing flows and probability paths on manifolds.\" arXiv preprint arXiv:2207.04711 (2022).\n\n[3] Song, Yang, et al. \"Maximum likelihood training of score-based diffusion models.\" Advances in Neural Information Processing Systems 34 (2021): 1415-1428.\n\n[4] Peluchetti, Stefano. \"Non-Denoising Forward-Time Diffusions.\" (2021).\n\n[5] Salimans, Tim, and Jonathan Ho. \"Progressive distillation for fast sampling of diffusion models.\" arXiv preprint arXiv:2202.00512 (2022).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "asrTPrl53T1", "reviewer": "Reviewer_tuy2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a continuous-time normalizing flow which does not need (i) to propagate gradients through ODE dynamics for training or (ii) to formulate the flow using a diffusion. The method is tested empirically on 2D data and standard tabular data benchmarks. ", "summary_of_the_review": "I'm torn between the compelling idea presented in the paper, and the lacking empirical evaluation. With more thorough experimental evaluation, this would be an excellent submission. As it stands, I would still lean towards acceptance because I think the idea deserves to be communicated. ", "strength_and_weaknesses": "**Strengths**\nThe fundamental idea proposed in the paper is excellent. Diffusion models showed how to design normalizing flows which bridge the data and base distribution by definition, and how to train them efficiently without taking gradients through ODE or SDE dynamics. This paper takes that idea one step further and shows that such a flow can be designed without resorting to a diffusion process, and that training amounts to minimizing a simple quadratic loss analogously to how denoising score-matching facilitates score-matching without access to the ground-truth score. Moreover, the paper relates the quadratic objective to the Wasserstein-2 distance between the target distribution and the distribution defined by transporting the base under the model, and the authors also demonstrate that the proposed flows can bridge between any two distributions from which we have samples, not just a target distribution and tractable base distribution such as a Gaussian. These are all strong contributions. \n\nI'd also like to highlight a key point made by the paper, namely 'By specifying an interpolant density, the method therefore separates the tasks of minimizing the objective from discovering a path between the base and target densities. This is in contrast with conventional maximum likelihood (MLE) training of flows where one is forced to couple the choice of path in the space of measures to maximizing the objective.' I think this is a significantly underappreciated point in the modern generative modeling literature, and explains many of the issues with vanilla flows and VAEs which have been to some extent addressed by diffusion models. I am glad the paper explicitly notes the importance of this decoupling. \n\n**Weaknesses**\nUnfortunately, the experimental evaluation is lacking. \n\nThe 2D experiments are useful as a sanity check, but are not convincing enough to warrant their own section. Moreover, the fact that the proposed method can formulate a flow between two arbitrary but is again only tested on 2D data is unfortunate. For example, a compelling experiment which immediately jumped to mind is image upsampling. Diffusion-based image upsamplers transport a Gaussian to the high resolution image conditioned on a naively upsampled image using e.g. bilinear interpolation. In the case of the proposed method, a flow could be formulated to bridge between the naively upsampled image and the ground-truth directly. An experiment like this would be a strong demonstration of the advantage of the interpolation approach. \n\nIn terms of the tabular data experiments, I appreciate that this benchmark of 5 datasets has been standard in the literature, but it was originally introduced in 2017 and is now quite outdated. I would question its usefulness beyond a sanity check for the method, similar to the toy experiments. For example, BSDS is a dataset of 8x8 black-and-white image patches whose train split contains 1 million examples. Diffusion models have been fit to datasets consisting of hundreds of millions or even billions of images at higher resolution. While it's not possible for all new methods to be tested at this scale, it's still necessary to compare on a reasonable-scale problem, ImageNet. Alternatively, design a new benchmark for tabular data. I appreciate this is a lot to ask, but at some point benchmarks become outdated and are no longer useful to track progress in modeling. \n\nFinally, and related to the previous points, the comparison to FFJORD is worthwhile as a baseline, but surely the primary mode of comparison should be flows trained in the diffusion model setting? These are much more competitive, and can even be trained by maximum likelihood without the need to backprop through ODE dynamics. This would provide a much stronger baseline, and a more convincing argument that the proposed method provides an advantage. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FRUMhgryDuc", "reviewer": "Reviewer_ogEs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an alternative learning objective for a continuous normalizing flow. Instead of standard maximum likelihood training, which requires backpropagating through the ODE solver, authors define a continuous sequence of intermediate densities between $p_0$ and $p_1$, and train the dynamics of the continuous flow to match this sequence. The intermediate densities are defined using a differentiable interpolation function. The dynamics of the flow are trained to minimize an objective function that only requires samples from the base density, samples from the target density, and samples of time $t$. Authors show that this objective has a unique minimum, which is achieved when the _true_ dynamics are recovered. In addition, authors derive a bound on the Wasserstein distance between the learned density and the true target density, as well as link the proposed model to score-based diffusion models. Numerical experiments are undertaken which demonstrate the effectiveness of the method on real tabular data.", "summary_of_the_review": "I strongly recommend accepting the paper. The proposed method draws inspiration from a set of methods in the field, yet is sufficiently novel, and is analyzed in-depth, both theoretically and empirically (albeit the empirical part could be even stronger, see suggestions above). Experimental results are strong, and the manuscript itself is excellent.", "strength_and_weaknesses": "Strengths:\n- An idea that echoes prior work (pre-defining a sequence of densities and fitting it), yet is sufficiently novel and required a non-trivial amount of mathematical reasoning.\n- Further theoretical study of the method, deriving a bound on the density fit, as well as showing a connection to the important class of generative models (diffusion).\n- Experimental results: clear improvement on competing continuous flows on tabular data. Sensible results on synthetic datasets.\n- High-quality, polished write-up with crisp notation.\n- An in-depth discussion and analysis of the computation efficiency of the method.\n\nWeaknesses/questions:\n- Motivation for the interpolation function chosen in the paper: why have authors not started with a simpler, e.g. linear, interpolation function? Are there any preliminary experiments with alternative interpolation functions to discuss, and especially whether the method is sensitive to this choice?\n- Diffusion-based models are known for their excellent sample quality. Given the shown connection between two classes of models, have authors tried fitting their model to higher-dimensional image datasets, and evaluating the samples?\n- Fundamentally, the method aims to pre-define a continuous sequence of densities, and recover the dynamics from this sequence. Have authors considered alternative ways of constructing this sequence? Does the section 2.2 suggest that we could use a diffusion process instead?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5Z4QjgQgpY", "reviewer": "Reviewer_KfAo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, we authors propose a new way to interpolate between two probability distributions. This is done by constructing a stochastic interpolant between two points $x_0$ and $x_T$ distributed w.r.t. $\\pi_0$ and $\\pi_T$ respectively. It turns out that the amortized dynamics obtained by integration w.r.t. to the product measure $\\pi_0 \\pi_T$ satisfies a continuity equation and can also be related to an Ordinary Differential Equation (ODE). The drift of this ODE can be learned in a manner akin to the Implicit Score Matching (ISM) in diffusion models. The authors elaborate on the links with diffusion models by showing that for a specific choice of interpolants the estimated velocity is related to the score of a diffusion model. They evaluate the performance of their approach on several toy and tabular datasets.", "summary_of_the_review": "To conclude, I think that the approach developed by the author is interesting. \n\nThe methodology and theory introduced in that paper are coherent and represent a nice extension of diffusion models.\nHowever, I think that more investigation (in particular numerical) is required to make the case that this approach can be used 1) in a large-scale setting 2) outside of the diffusion models setting.\n\nFor these reasons I think that the work is currently not mature enough.", "strength_and_weaknesses": "STRENGHTS:\n* I think this is a nice idea that showcases how the manipulation of (Fokker-Planck) PDEs can yield new algorithms. I thought the paper was well written with the main contribution clearly presented. It extends denoising diffusion models in a new direction by focusing on the issue of the speed in these models.\n* The theoretical results regarding the control in Wasserstein 2 are new. \n\nWEAKNESSES:\n* The experiments are quite weak. Considering that the presented approach extends diffusion models I would have expected the authors to compare their approach to diffusion models at some point. This impression is reinforced by the fact that \"The choice of interpolant for experimentation was universally selected to be that of (5)\" (where (5) highlights the link with diffusion models). It is not clear to me that the current approach brings any benefit compared to diffusion models. I think that the authors miss the opportunity to compare and evaluate different interpolants. Indeed, they introduce a whole new framework but end up only using one interpolant, which is underwhelming. \n\n* I find a bit misleading that bold numbers correspond to only the best methods among continuous normalizing flows (CNF) in Table 1, since we can find that other normalizing flows yield better results for each considered task. I have read that the authors \"primary point of comparison is to other continuous time models, so we sequester them in benchmarking\" but no explanation is given as to why CNFs perform worse than NFs for tabular data and what could be done to remedy this situation. Also, confidence intervals are missing in Table 1. \n\n* The authors do not discuss some related literature. In particular, one work that seems to be particularly relevant is [1] which considers a similar approach to the one of the author. Indeed in [1] the author considers a stochastic interpolant between $x_0$ and $x_T$ given by a bridge (in this paper a Stochastic Differential Equation (SDE)). Then the obtained diffusion is amortized with respect to some coupling between $\\pi_0$ and $\\pi_T$. The author of [1] shows that the obtained dynamics can also be described by an SDE whose drift can be learned. Finally, I think that the authors miss to discuss extension of denoising diffusion models which also consider finite-time dynamics like [2,3,4]. Is there any link between the proposed approach and control and optimal transport?\n\n* Regarding the result in Wasserstein 2. Am I correct assuming that the exponential bound should depend on the finite time $T$? The current bound is correct because the authors restrict themselves to the $[0,1]$ interval but any rescaling of the time would harm the bound. This exponential dependency comes from the use of a Gronwall lemma and should be discussed.\n\nOTHER COMMENTS/QUESTIONS:\n\n* What limits the authors to consider an amortization w.r.t. to the independent coupling in Equation (6)? One could imagine other couplings and I think that the obtained results would still be true. Am I correct assuming that? In that case, a link with diffusion models can be drawn in a more explicit fashion by considering the coupling given by $X_0 \\sim \\pi_0$ and $X_T = e^-T X_0 + (1- e^{-2T})^{1/2} Z$ with $Z$ a Gaussian random variable. For $T= +\\infty$ we recover the independent coupling but diffusion models are always run up to a finite time $T$.\n\n* The current work is limited to ODEs. From a generative modeling point of view it has been shown [3] that ODEs and deterministic methods in general yield worse results than their SDEs counterpart. Can the theory developped by the authors be extended to the SDE setting? I am guessing that in that case the authors would recover derivations similar to [1]. \n\n[1] Peluchetti -- Non-Denoising Forward-Time Diffusions\n\n[2] Vargas, Thodoroff, Lawrence, Lamacraft - Solving Schrödinger Bridges via Maximum Likelihood\n\n[3] De Bortoli, Thornton, Heng, Doucet - Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling\n\n[4] Chen, Liu, Theodorou - Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory\n\n[5] Karras, Aittala, Aila, Laine - Elucidating the Design Space of Diffusion-Based Generative Models", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster