Turning Normalizing Flows into Monge Maps with Geodesic Gaussian Preserving Flows
Authors: Guillaume Morel
Lucas Drumetz
Simon Benaïchouche
Nicolas Courty
François Rousseau
Venue: Accepted by TMLR ()

Abstract: Normalizing Flows (NF) are powerful likelihood-based generative models that are able to trade off between expressivity and tractability to model complex densities. A now well established research avenue leverages optimal transport (OT) and looks for Monge maps, i.e. models with minimal effort between the source and target distributions. This paper introduces a method based on Brenier's polar factorization theorem to transform any trained NF into a more OT-efficient version without changing the final density. We do so by learning a rearrangement of the source (Gaussian) distribution that minimizes the OT cost between the source and the final density. The Gaussian preserving transformation is implemented with the construction of high dimensional divergence free functions and the path leading to the estimated Monge map is further constrained to lie on a geodesic in the space of volume-preserving diffeomorphisms thanks to Euler's equations. The proposed method leads to smooth flows with reduced OT costs for several existing models without affecting the model performance.

Published in Transactions on Machine Learning Research (03/2023)
Turning Normalizing Flows into Monge Maps with Geodesic Gaussian Preserving Flows
Guillaume Morel
gulllaoneemor/@@@ttaaaaaiiu
IMT Atlantique, LaTIM, U1101, Brest, Prance
Lucas Drumctz
bucas. por...ddmametttkimmmllaaaaigpppp
IMT Atlantique, Lab-STICC, UMR CNRS 6285, Brest, Prance
Simon Benaichouche
simon. Sinnon.bmnncccooccceeeeelllllllu
IMT Atlantique, Lab-STICC, UMR CNRS 6285, Brest, France
Nicolas Courty
Université Bretagne sud, IRISA, UMR CNRS 6074, Vannes, France.
François Rousscau
IMT Atlantique, LaTIM, U1101, Brest, France
Reviewed on OpenReview: At sp // en ew n et eft um 7 d 20 1Cv9
Abstract
Normalizing Flows (NF) are powerful likelihood-based generative models that are able to trade off between expressivity and tractability to model complex densitics. A now well established research avenue leverages optimal transport (OT) and looks for Monge maps, ie. models with minimal effort between the source and target distributions. This paper introduces method based on Brenier's polar factorization theorem to transform any trained NF into more OT-efficient version without changing the final density. We do by learning rearrangement. of the source (Gaussian) distribution that minimizes the OT cost between the source and the final density. The Gaussian preserving transformation is implemented with the construction of high dimensional divergence free functions and the path lcading to the estimated Monge map is further constrained to lie on a geodesic in the space of colume-preserviee diffeomorphisms thanks Euler's equations. The proposed method leads to smooth flows with reduced OT costs for several existing models without affecting the model performance. The code is available here https: htt::/gittti wttpghtt.......ooloooeoeeee
1 Introduction
Modeling high dimensional data is central question in data science as they are ubiquitous in applications. Various tasks such as probabilistic inference, density ostimation sampling of new data require accurato probabilistic models that need to be defined efficiently. There exists a large variely of generalive models in the literature. Among other approaches, diffusion / score based models (Ho et al.) 2020 SShh--iickstein et al,
2015; Song & Ermon 2019), variational autoencoders (VAES) (Kingma & Welling 2014 Rezende al,
2014) and generative adversarial networks (GAN) (Goodfellow et al.) 2014) are frequent choices, each with their strengths and weaknesses.

---

Published in Transactions on Machine Learning Research (03/2023)
Normalizing flows. A fourth popular class of generative models is Normalizing flows (NF). NF models transform a known probability distribution (Gaussian in most cases) into a complex one allowing for efficient. sampling and density estimation. To do so they use a diffeomorphism f E² + R which maps target
probability distribution l to the known source distribution v = f# (Dinh et al. 2014 Rezende & Mohamed]
2015 In practice the flow satisfies the change of variables formula:
log (x) = log (f(x)) + log det Vf(x)]-
(1)
There are many possible parameterizations of f, usually relying on automatic differentiation to train their parameters via first order optimization algorithms. For density estimation applications, training is done by maximizing the likelihood of the observed data. The data are generally high dimensional and accessing (x.xx) for given x requires computing determinant of the Jacobian matrix of f. This operation has a complexity of O(d5) in general and thus requirement for the flow architecture to have a tractable determinant of the Jacobian while remaining expressive enough (Dinh et al. 2014 Kingma & Dhariwal 2018 Papamakarios et al. 2021
Rezende &
Mohamed 2015
Optimal transport. A diffeomorphism transforming any well-behaved distribution into another always exists in theory [Papamakarios et al.) 2021). However, there can be many ways to transform one probability generally not unique. This measure μ into another probability measure v, and therefore the function 2018 Grathwohl et al. 2018 has led to many proposed architectures in the literature (Kingma & Dhariwal
Huang et al., 2018 De Cao et al. 2020; Papamakarios et al. 2017). 'he question of choosing the 'best." transformation among all existing ones is therefore crucial, independently of how accurately μu models the target distribution. One way to make the architecture unique (under appropriate conditions on the two distributions) is to use optimal transport (Hamfeldt) 2019 Peyré et al.) 2017 Santambrogio 2015 Villani
2008), that is to choose the one giving the Wasserstein distance between 1a and v, with a squared L2 ground cost:
W2 = min / [f(x) x(3du((x), V == fg f
where f denotes the push forward operator of f through f. An optimal model in the sense of 2 minimizes the total mass displacement which can be a desirable property even if it is often a difficult task. In particular Brenier's theorem (Brenier 1991] states that the optimal function f is the gradient of a scalar convex function, which is widely used when solving (2)
One key property of OT mappings is that they should better preserve the structure of the distribution compared to non OT transformations. This makes them particularly appealing for machine learning applications, and may also help with generalization performance (Karkar et al. 2020
Optimal transport and NF models. Including OT in NF models has recently received much attention with various approaches to oblain a map g which satisfies the properly (2). Among all these methods, many use either directly Brenier's theorem (Brenier] 1991] or the dynamic OT formulation with the Benamou-Brent approach (Benamou & Brenier 2000). One important remark is that most of the approaches considered need dedicated architectures in order to satisfy the OT property. For example the transformation f is often written neural network modeling the gradient of (possibly convex) scalar function (Amos et al. 2016 Finlay et al.
2020a Huang et al. 2020: Onken et al.
2020 Zhang et al. 2018 This sometimes requires
2020) and or the addition of some penalization terms in the loss function (Onken et al.) 2020 Finlay et al. 2020b Yang & Karniadakis 2020). When considering the Benamou-Brenn
some particular training process (Finlay et al. 2020a
Huang et al. 2020 Onken et al.
formulation, the normalizing flow is interpreted as the discretization of a continuous ordinary differential equation (Chen et al., 20188) and the optimal transport problem is then solved dynamically (Finlay et al., 2020b Onken et al. 2020 Zhang et al. 2018
The methods mentioned above require to constrain the architecture and/or the training procodure which can be problematic in some cases and has already motivated other OT related approaches (Uscidda & Cuturi 2023). In this paper we present a method which achieve the optimal map of a given NF without constraining the architecture or the training procedure.
1.1 Main contributions
2

---

Published in Transactions on Machine Learning Research (03/2023)
Polar factorization. An overlooked implication of Brenier's theorem is the so-called polar factorization theorem, that states that the optimal transport map Val solving 2 can be factorized into the composition of two functions ViA- = s f, the function being some arbitrary smooth map from I to and s an associated measure preserving function of (Brenier 1991). The idea we exploit is the possibility, from given flow f (and its corresponding inverse = f-), to rearrange the distribution y using s to obtain a new map reducing the OT cost without changing the distribution given by the push-forward j = B# = (g o s "w.. The OT-improved map can then be obtained with the composition g gos- Interestingly this property is not popular as the previous one and to our knowledge is not used when dealing with OT and NF' models. Yet normalizing flows can take advantage of this formulation mostly because the distribution v is known and simple (here and in the following v is standard normal) which makes it possible to construct architectures preserving V.
Figure 1: A GP transformation applied on particles sampled from a two-dimensional Gaussian distribution. The mean and standard deviation stay the same; only the positions of the particles change.
Target Distribution
BNAF
BNAF+ GP projection
Exact discrete OT projection
color coded
projection
Figure 2: Eight Gaussians test case with colored distributions. A GP flow is trained on a pre-trained BNAF model De Cao al. 2020) reduce the OT cost.
Gaussian preserving flows. Our work differs from the state of the art as we do not propose new normalizing flow model. Instead, we propose to use Brenier's theorem to compute the Monge map for any pre-existing architecture. Indeed there exists a wide variety of architecture available in the literature (Kingma
& Dhariwal 2018 Grathwohl et al.] 2018 Huang et al.] 2018 De Cao et al.] 2020 Papamakarios et al. 2017) each with their pros and cons which sometimes depend specifically on the test case considered. Our idea. is to use Brenier's polar factorization theorem rearrange the points in the known distribution to obtain the optimal map associated with a given flow. We consider the most common case where the known distribution is a standard normal and call such rearranging maps Gaussian Preserving (GP) flows, see Figure An important point is that by construction our GP map will only change the OT cost of the model. The target density and therefore the training loss given by the model will stay the same. This allows us to take any pre- preetraieed model and compute the associated Monge map, thus improving the model in terms of OT displacement from the source to the target distribution, without changing the modeled density see Figure 2
Construction of divergence free functions in high dimensions. We show that divergence froe functions can be used to model GP flows. We therefore derive an elfective construction of divergence free functions in high dimensions and apply it in the latent space of some popular VAE models.
Euler's equations. Since several GP flow models can solve the same OT problem, we also look for a way to find the 'best' GP flow. This is somehow similar to the approaches from Finlay et al.] (2020b); Onken et al. (2200) where the trajectories of a contimuus normalizing flow are penalized to be straight lines. This is not strictly noeded to find the Monge map but can be interpreted as some geodesic over all the flows which solve the associated OT problem. In this work, we show that the geodesics associated with the OT problem are actually given by solutions to the Fuler equations, following celebrated result by Arnold (Arnold 1966).
3

---

Published in Transactions on Machine Learning Research (03/2023)
The penalization of Euler's equations in high dimensions and its practical implementation is therefore also considered, which is to the best of our knowledge an original contribution.
Data structure preservation with optimal transport. Finally we show one potential interest of GP flows by studying the preservation of the data structure experimentally. More specifically we focus on the preservation of disentanglement on the dSprites (Mathhey et al., 2017); MNTST (Lecun al. 1998) and Chairs (Mathieu al.) 2014) datasets in some variational auto-encoder (VAE) latent space, On this particular example we show that OT allows to improve the preservation of the structure of the latent data points which is otherwise destroyed when applying the NF model.
2 Polar factorization theorem
The main idea is to use the Brenier's polar factorization theorem to construct the Monge map with a rearrangement of the known probability distribution v.
Notation and measure preserving definition. In the following we will use the shortened notation (X, v) for probability space (X, p) where X' is set, B o-algebra of its subset and v probability distribution. We recall the definition of measure preserving function use in Brenier (1991): measure preserving mapping from a probability space (X',) into itself is a mapping s X' + X' such that for every measurable subset A of x. (A) is y-measurable and v(s (is-1(A) (A)) = =AA).
Theorem (Brenier's polar factorization (Bremier 1991)). Let (X,v) be a probability space, X bounded. Then for each g E L"(XXv, I2) satisfying the non degeneracy condition
v(g-' 0 for each Lebesgue negligible subset E of IR!,
there ezists unique conver function v/ : X + E and a unique measure preserving function s X + X' such that
g(s(x)) Vv(x),
and s(x) minimizes the cost Xx (g(((x))) - x (x).
Note that in his work Brenier mention the maximization of J x g g sxx)dx(xx which is equivalent to the minimization with repect to of the quadratic cost thanks to the measure preserving property of s. Our goal is to leverage the polar factorization theorem in order to solve the OT problem between v and μ= g# where g is given and v (0, Id), by looking for the rearrangement 8 via an optimization problem. To do so we need to construct a class of measure preserving maps.
Remark 1. Since in practice we consider u be a standard normal, the domain X is not bounded and therefore does not strictly satisfy the hypothesis of Theorem 7 We do not investigate this point further and simply quote remark from Brenier's 's work (Brenier 1997): 'we believe that the result is still true when X' is unbounded, provided that
and Sx (x)xxx < +00, where 1/g + 1/p=1'. The function ((x) -11 /2 the probability density of v, and the incquality is therefore satisfied.
3 Gaussian preserving flows
In order to apply Brenier's polar factorization theorem, it is therefore needed to construct class of measure preserving maps. Since consider the case where y is standard normal, call such maps Gaussian preserving (GP). All proofs of the propositions and lemmas are given in Appendix B
Consider two probability measures a and 3 with density ha and ha respectively. A mmp is measure preserving between a and |3 it satisfies the change of variable equality (same as 11) without the log)
(x) = h=(s(x)) (Vs())))) In our case, we want s to be Gaussian preserving therefore ha = hg = e |x| - and one gets
det Vs(x)) = e (|exxx)]2
(3)
It turns out that Lebesgue preserving functions (i.e. satisfying det VΦl 1) can be used to construct maps satisfying (3). In the following we will denote erf : RRd + Rd the distribution function of one dimensional Gaussian (that is erf(()) 3/h 1o e =ddt) applied component wise.
4

---

Published in Transactions on Machine Learning Research (03/2023)
Proposition 1. Let Ω = (-1,1)4. The map is smooth Gaussian preserving function (t.e. satisfying iJ and only is there exists Φ 52 > Q such that det VΦj = and
s(x) /2erf op erf
XER
From now on we will focus on the construction of volume and orientation preserving maps (i.e. satislying det VΦ 1) since functions satisfying det VΦ -1 can be constructed from them see Appendix B.1.2 Moreover, one has the following result regarding the regularity of GP flows.
Lemma 1. Assume the Monge map and the NF architecture are C diffeomorphisms. Then the corresponding GP flow is C² the associated function & is also C and either salisfies det Vo((x) everywhere or det VΦ(x) -1 everywhere.
3.1 Volume-orientation preserving maps
First we introduce the space SDi will working with from now Let Diff be the set of all dilfeomorphisms in Q then SD) == {vb e Diff( det(V 1, Vx E Ω}. where f = (-1,1)4. That is we need a transformation which satisfies two properties: 1) the function must be volume and orientation preserving, 2) the solution must stay in the domain (-1,1)d. Consider the following ODE:
d x(t, = X(t,xx xEΩ, O X(0,x) x.
(4)
We impose conditions on the velocity V:
V V = 0, Ω
(5)
V. n = 0, on an
(6)
where is the outward normal at the boundary of Ω. We define & to be the solution at the final time <xx) == x)), Property (5) implies that det VΦ = 1 (this can be checked with the formula d det VX =
di div(v) det VX), and property (6) ensures that & has values in Ω. Any function in SD) can be written as a solution to (4) for d (Shnirelman 1993), for = 2 some pathological cases can constructed
Shnirelman 1994).
Divergence free vector fields. First focus the vector fields satisfying 5 for arbitrary large dimensions. Property (6) can then be incorporated with very little additional work.
Proposition 2. Consider an arbitrury vector field v Rdd + Rdd. Then V -v= if and only if there exists smooth scalar functions vs. _dd I., with vi - such that
(x)x Σ Oz wit (x), i= I....,d
(7)
where (Uly Un).
To impose the boundary conditions 6 one can simply multiply each oy by - (x]3 - -1).
Lemma 2. Consider the coefficients
(x) ((x)))))(
(8)
where h₂(x,) = hz - (xh +), h. h are functions satisfying h2(00) = h2( 0 and 3 (x) Rd - R
are bounded functions satisfying J - Then the function defined in Proposition satisfies V and v 0 Nn.
5

---

Published in Transactions on Machine Learning Research (03/2023)
The function h (NI......h)) hu) can typically be parametrized neural networks with no bias but in experiments we have simply chosen to consider the case hx(z;) = (x² -1). One drawback of Lemma 2 is that it does not guarantee universal approximation of divergence free functions near the boundaries. Note that
when we are away from the boundaries however Proposition gives this universal approximation rcsult.
our experiments we observed that. we can at least. significantly reduce the OT cost. with the construction and we therefore let the investigations related to the boundary conditions for future work.
The incompressible property (5) and the boundary conditions (6) can be exactly implemented in the network in any dimension. Note however that in order to get all the incompressible vector fields 7 we need to construct at least d(d 1)/2 arbitrary scalar functions. See Appendix A for the practical construction of these divergence free functions in high dimensions.
4 Euler's geodesics
GP flows give way compute the Monge map for any trained NF architecture. Many transformations achieve this goal and the question of finding the best flow among all volume preserving transformations needs to be considered. In the following we will consider a regularization term to smooth the trajectories by minimizing the energy l'v². While it could be tempting to try to penalize directly J'v?, the energy will unfortunately have an opposite objective from the minimization of the OT cost. Indeed the global minimum for the energy is v² = 0 (that is the particles do not move) which obviously not the velocity field which minimizes the OT cost. In fact the global minimum of a loss composed with the two terms OT + ʃ v² may not minimizes the OT cost. On the contrary, as we will detail below, Euler's equations minimize the energy SDiff over all other solutions with the same initial and final states. By finding the correct final state and solving Euler equations it is therefore possible to obtain solution which both minimizes the OT cost and the energy.
4.1 Arnold's theorem
In 1966, Arnold AAnold 1966) showed that the flow described by Euler's equations coincides with the geodesic flow on the manifold of volume preserving diffeomorphisms. This theoretical result therefore gives the reason why regularizing our flows with Euler's equations is a desirable property. Mainly that Euler's oquations take the path with the lowest energy to reach the final configuration. Consider the Euler equations:
0.v+ + (v. V)v= Vp, EE [], x LE [0, T], X te [0, T] E An,
V.v Vv==0 = 0,
(9)
v.n= 0,
v(0,-) = vo:
where v:= vtt, x) is the velocity field, p:= p(t, x) the pressure and n n(x the outward normal at the boundary of Ω. Here the pressure ensures that 0,v + (v- V)v can be written as the gradient of some scalar function which is uniquely defined (up to a constant) thanks to the additional divergence free and boundary conditions on v. Additionally the pressure field can be interpreted as the Lagrange multiplier of the divergence free constraint for the associated variational formulation of Euler's equations. In particular, it may not be needed to compute p when solving numerically function X(t,-) :):
We introduce & the energy of smooth
10. J, dxxdt,
E(X)=
(10)
Now assume & E (D).. Arnold's problem's consists in finding the path X(1, )ueo,, in SDiII(A) joining the identity to & which minimizes E:
min 8(X), X(0,-) ld, - (SDu)i)
(11)
In other words 11 is the geodesic in (Diff( between Id and Φ.
Theorem Arnold (1966)). Assuming the existence of solution to Arnold's problem, X solution to if and only if v(1,)) O, x(, x1 salisfies Fuler's equations 9
6

---

Published in Transactions on Machine Learning Research (03/2023)
4.2 Penalization of Euler's equations in high dimensions
Numerical schemes developed to efficiently solve the Euler equations (Canuto et al., 2007 Quarteroni 2009
(mainly for fluid mechanics problems, i.e for dimensions up to 3) scale badly when the dimension increases. In this work, the solution to Eulcr's equations is interpreted as the geodesic to reach the solution of the OT problem and the dimension can be arbitrary large. Therefore we approach the equation (9) through a penalization procedure which can be carried out in any dimension. As explained in the previous section we notice that the second and third equations in (9) are satisfied by construction in the network.
Our remaining goal is to constrain the network to be smooth solution to d.v + (v. =>v=-Vi The left hand side can therefore be written as the gradient of scalar function and we note that a vector satisfies Wt.x E I2 satisfies Wr,x = Vp(t, x) if and only if its Jacobian is symmetric Vwr.x = (Vwr.)" In order to solve the first equation in (9. we propose to penalize the non-symmetric part of the Jacobian for the total derivative of v. Since a Jacobian-vector product can be efficiently evaluated in high dimensions (unlike the calculation of the full Jacobian which is computationally expensive), we do not calculate directly the Jacobian and use instead the following properly of symmetric matrices: M is symmetric if and only if y" "zz - z" My O, Vy, R.. The idea is to sample random vectors y, during the training and to penalize this term for the total derivative, that is to minimize:
R(x) = Ey.. 1-² (y (VWxxx) x" (VWex)y)) de
(12)
with y, z ~ No Id) and Wx, = 0.v + (v- Vjv. In practice, we do not compute the full time integral in 12) as it would be computationally too expensive but calculate the penalization only at our time steps discretization.
Approximation of the total derivative. To reduce the computational burden, we do not calculate exactly the total derivative Wx,x but use an approximation of its Lagrangian formulation instead. More precisely, consider the variable x(t from (4) that is the position of particle at time t with initial position We recall the equality (see Appendix [B3.1 Di v(t, ((,, (t, ((t, + (vt,,X(t, x)) ()v(t, X(t, x)) and therefore choose to approximate the right-hand side by using first order Taylor expansion of Dv/Dt:
D DI ((t,
v(t X(tt+1 x)) v(t", X(t",x)
(13)
Al
At = 10+1 - t", In practice, At is set to 2 where e is the machine precision. This approximation can be easily computed since requires only the evaluation of the velocity at two positions of a particle.
To summarize approach requires to penalize a Jacobian-vector product of the form y (Vw)z x (Vwy where is given by (13). Penalizing Jacobian-vector product has already been done in other contexts and prove to efficiently scale with the dimension Song et al. 22020).
5 Procedure
In order to solve the optimal transport problem, we can either use the forward NF function f or g f-1 Depending on this choice the loss function is then either so or Exco]|x - g sxx)]
In practice the GP flow is parametrized a standard residual network (ResNet) with Runge-Kutta 4 time discretization (Atkinson] 1989) (other discretization are possible) and is estimated by minimizing the parameters of the velocity field. When regularizing with Euler's equations, we replace the term Vw in (12 by
13) and calculate the Jacobian-vectod product with the function autohrau from pytorch. A parameter 0 is also added in front of the penalization term that is if we use the function f:
f(x) V2
min E2O0))
x - so f(x)12 + XR erf o
(14)
0
or if the function g is considered instead
x ||x -6-go go (x))) + XR o erf o
min Fxix)
(15)
0
/2
7

---

Published in Transactions on Machine Learning Research (03/2023)
where the vector 0 denotes the parameters of the velocity field f is the NF architecture, the GP flow and R corresponds to the term penalized with Euler's equations. The subscript 0 has been added to highlight are two distinct optimization
the dependence of and R to the parameters. We emphasize that [4 15 strategics:
If we do not want to invert the NF model (for example is computationally expensive) we can simply minimize (14) over the training data. The probability distribution p in (14) is then the unknown distribution from which the data are taken. This assumes however that the training data are correctly mapped to the standard distribution with f.
If the NF model is cheap to invert we can minimize 115 instead. To do so we optimize over samples from the standard normal distribution v. In this case, the probability distribution μ is the transformation of v by the inverse of the NF model μ -- g
If the NF model transforms perfectly the training data over the standard normal v, these two approaches are equivalent. If this is not the case, we notice that the second approach requires a cheap inverse of the NF model, but has the advantage of not using any training data to train the GP flow and may therefore better generalize.
6 Results
We apply GP flows on two popular NF models: BNAF, discrete NF (De Cao et al. 2020) for two-dimensional test cases and FFJORD, a continuous NF (Grathwohl et al.) 2018 for higher dimensional cases. Both of these models are solid references among NF and do not incorporate any OT knowledge in their architecture training procedure. The codes are taken from the official repositories]] The FFJORD model has an inverse function directly available in the code, which is not the case for the BNAF model. For this reason we consider only the FFJORD model when interpolating in the latent space of the dSprites and MNIST datasets because interpolations require the NF architecture to have an inverse function available. To compare our results we consider the CP-Flow architecture (Huang et al., 2020). The CP-flow network constrained be the composition multiple blocks which are gradient of scalar convex function and therefore converges by construction towards the optimal map when considering only block (provided the optimization reaches global minimum). This makes CPFlow good candidate for comparison.
6.1 Density estimation toy 2D data
In this section we perform density cstimation several 2d standard toy distributions (Grathwohl et al. 2018 Wehenkel & Louppe 2019). In particular we train our GP flow on pre-trained BNAF model. We provide two dimensional toy examples for the eight Gaussians, two moons and pinwheel test case on Figure 3 In the case of the pinwheel dataset we use Euler regularization see below for more details. To compute the exact discrete OT projection we use the POT library Flamary et al. (2021). The distribution colored to compare the transformation with the exact OT map. We observe on each experiment that adding GP flow makes the transformation closer to the Monge map.
Euler regularization. Let's briefly recall the goal when considering Euler's regularization. As discussed in Theorem 1 the measure preserving transformation s which minimizes the OT cost is unique. In this work we have constructed s to be the solution of an ODE with divergence free velocity and therefore even if s (that is the solution of the ODE at the final time) is unique, there are infinitely many trajectories which reach this final configuration. The role of Euler regularization is to obtain smooth trajectories for our ODE, a property which may be helpful during the training process to minimize the OT cost.
To highlight the value of using the Euler's penalization we focus on the pinwheel test case. If the pinwheel experiment from Figure 3 was done with Euler regularization, this is because we notice a bad convergence of the GP flow without it. Figure shows the reason why the training has trouble to converge: the trajectories are very irregular when no penalization is applied. This is an example of the practical utilization of Euler's
gtthab.comm//uggchen//ffffie /ffjord,, githhubbcom/nicccheeecaaaa//o// github.com /CW-/n/Waaaagggg Huang/CP-Floo FC--Flow
8

---

Published in Transactions on Machine Learning Research (03/2023)
Target Distribution
BNAF
projection
Target Distribution color codod
BNAF
BNAF GP
Exact discrete projection
Target Distribution color coded
BNAF
BNAF GP
Exact discrete OT projection
Figure 3: Color map for several 2d toy examples. Adding GP flow makes the transformation closer to the OT map.
Figure 4: Trajectories for the pinwheel test case from T 0 (leftmost image) to T = 1 (rightmost image).
Top: Gaussian motion without regularization. Bottom: Gaussian motion with Euler regularization. The latter gives much smoother trajectories.
Target distribution
GP flow without Euler
GP flow + Eulor
Exact discrete OT map
Figure 5: Comparison of GP flow with and without Euler for the Pinwheel test case. Euler regularization leads to better convergence result.

---

Published in Transactions on Machine Learning Research (03/2023)
No regularization
Euler regularization
Figure 6: Representation of the trajectories of the GP flow for the two moons test case with and without Euler penalization. Trajectories obtained with Euler regularization are more structured.
regularization and as shown in Figure it leads to a better global convergence. We give another ialustration of the trajectories for the two moons test case on Figure B We observe that the trajectories obtained with Euler regularization are more structured.
Note that Euler regularization is not always needed to reach the OT map and practice it seems to depend on the case considered. More details on Euler penalization are given in appendix [D]
6.2 An example of data structure preservation: improving disentanglement preservation with optimal transport
The goal here is to show that OT transformations better preserve the data structure than non OT ones. Preserving data structure can be very useful for example when trying to interpolate between dała points. Indeed, interpolating directly in the data space often leads to poor results mainly because may interpolate through low density regions. In this case it can be a good idea to map the data density into Gaussian, interpolate in the Gaussian space and then apply the reverse transformation. However, for the interpolation to be meaningful we expect the NF architecture to preserve as much of the data structure as possible. In order to study high dimensional data structure we therefore seek a data space which satisfy two properties:
The initial data space should be very structured.
The interpolations between data points should be casily comparable and in particular we should be able to classify them easily.
Disentangled representations satisfy these two properties and allow to encode the data in a latent space where change in one direction result in the change over one generative factor in the data. Recently the construction of variational auto-encoders (VAE) (Kingma & Welling 2014h with disentangled latent space has received much attention (Higgins et al.] 2016: Burgess et al. 2018 Chen et al. 2018 Kim & Mnih 2018). We therefore propose experimentally study disentanglement preservation of NF with and without OT. To do this we apply a NF architecture (FFJORD in case) to the VAE's latent distribution and consider the addition of a GP flow. The goal here is therefore to get as close as possible to the initial interpolations in order to show that even if the initial density is transformed to a Gaussian the data structure is preserved at least to some extent.
For the disentangled interpolation in the NF target (Gaussian) space we consider the same directions which are present in the VAE latent space and are aligned with the axes. This may be a little naive since these directions could be slightly modified depending on the source and target distributions, but it seems good enough approximation here. Our experiments are run with the 3-TCVAE architecture (Chen et al. 2018a and the latent space dimension is 10.
dSprites dataset. The dSprites dataset MMathey et al. 2017) is made of 64 x 64 images of 2D shapes procedurally generated from 5 ground truth independent latent factors. These factors are shape, scale, rotation, x and y positions of a sprite. Since the factors are known, we can compute a quantitative evaluation of disentanglement and we choose here to consider the metric from (Eastwood & Williams, 2018 the continuous factors (i.. all the factors except the shape) for the three criteria: disentanglement,
10

---

Published in Transactions on Machine Learning Research (03/2023)
Model
Disentenganlement t
Completeness
Informativness
Init. latent space
0.58
0.81
0.55
FFJORD
0.39
0.26
0.62
FFJORD+GP
0.58 0.01
0.66 + 0.01
0.62
PFJORD+GP+BULEN
0.58 = 0.01
0.66 + 0.01
0.58 +
Table 1: Quantitative evaluation from (Eastwood & Williams 2018) of disentanglement (higher is better), completeness (higher is better) and informativeness (lower is better) on the dSprites dataset. Adding GP flows make the scores closer to the initial ones. For all models considered here (except in the case of the vanilla FFJORD model) the values represent the mean and standard deviation taken over runs.
Model OT cost FFJORD
dSprites
MNIST
chairs
10.45
6.81
5.98
FFJORD+GP
5.69 + 0.03
3.11 0.01
2.34
FFJORD+GP+EULER
5.30 0.01
3.01
2.30
CPFlow (3 blocks)
6.27 0.06
6.01 2.12
2.46
CPFlow (1 block)
8.97 + 3.44
27.92 3.00
0.50
Loss FFJORD
-16.62
-0.45
5.48
FFJORD+GP
16.62 + 0.
-0.45 0.
5.48 + 0.
FFJORD
-16.62
-0.45
5.48 + 0.
CPFlow blocks)
19.007 + 0.06
-0.06 0.05
5.94 0.05
CPFlow (1 block)
166.99 + 0.08
0.95 = 0.11
6.45 0.01
Table 2: Losses and mean OT costs. GP flows reduce the OT cost without changing the loss. Adding Euler regularization allows to further reduced the OT cost on the dSprites dataset. For all models considered here (except the case of the vanilla FFJORD model) the values represent the mean and standard deviation taken over 3 runs.
completeness and informativeness. Table 1 shows that FFJORD destroys the latent structure and gives the worst disentanglement, completeness and informativeness scores. Adding GP flows allow to recover the same disentanglement score as the initial latent space and get values closer both for completeness and informativeness. On Table g the OT costs are compared, and as expected GP flows allow to reduce the OT cost without changing the loss. Interestingly GP flows with no additional regularization do not converge completely to the Monge map because particles get out of the domain at some point, making it impossible continue the training process (the training is stopped at = 60 epochs). We conjecture that this may be due to non-smooth trajectories of our GP flows and regularization is therefore needed. As shown on Table 2 adding Euler regularization fixed this issue and allows to further reduce the OT cost.
To illustrate the preservation of disentanglement, some interpolations are also presented in Figures and in Appendix F. The dimensions are sorted with respect their KT, divergence in the initial latent space and therefore only the first dimensions carry information: cach of the first 5 lines correspond to a generative factor while the last dimensions leave the image unchanged. This structure lost when mapping the latent space to Gaussian distribution with the FFJORD architecture. The addition of a GP flow fixed this issue and the interpolation better match the initial latent one.
MNIST dataset. We also consider the MNIST dataset. (Lecun el al. 1998). As opposed to the dSprites test case, GP flows do not seem to have trouble to converge here without Euler penalization and therefore we obtain comparable OT costs with and without Euler regularization see Table 2 Since the generative factors are known in this case we cannot make quantitative evaluation of disentanglement dSprites dataset. We focus instead on the interpolations presented in Figures 11 and 12 in Appendix E GP
we did for the
11

---

Published in Transactions on Machine Learning Research (03/2023)
flows better preserve the data structure of the initial latent space compare when applying only the FFJORD model. This can be seen in particular on the last two rows of each block which are not changing in the initial latent space. This structure is lost with the FFJORD model and recovered when training a GP flow.
Chairs dataset. Finally we look at the chairs dataset (Mathieu et al. 2014 Again, the GP method converges both with and without Fuler's penalization and greatly reduced the OT cost see Table 2 Interpolations are presented in Figures 13 and [4 in Appendix D By looking at the last rows we observe once again that GP flows better preserve the data structure of the initial latent space compare when applying only the FFJORD model.
Comparison with CPFlow. To check that GP flows get close to the Monge map we also compare the OT costs with the CPFlow ure Note that the number of parameters as well as the training procedure differ from the FFJORD+GP and CPFlow architectures so we only use CPFlow OT costs as a baseline to ensure that FFJORD+GP close to the Monge map. The losses are only given here as an indication to show that the probability distribution of both FFJORD and CP flow are close from each other and a comparison between their OT costs is therefore relevant. Finally note that we both test the CP flow architecture with 1 and 3 blocks. The advantage with the 1 block architecture is that CP flow should theoretically converges to the Monge map. However it seems harder to make the network converges in this case and we therefore also show some comparisons with 3 blocks which leads to much lower losses.
GP flows get OT values which are always lower than the CP flow ones for similar losses and we therefore conclude that GP flows are at least as close to the Monge map as CPFlow. The only exception is the chairs dataset where CP flow with one block has a lower OT cost. We note however that in this case the loss for CP flow is much bigger and therefore the comparison of OT costs with GP flow may not be relevant. We also notice that CPFlow may leads to high OT cost. when trained on the MNIST dataset. One possible explanation for this is that CPFlow is trained only on the data, Since the OT costs are evaluated with random samples drawn from the standard normal this may show some issuc with the generalization during the backward process. On the contrary GP flows are directly trained on random samples from this distribution and may therefore better generalize.
It is therefore possible to better preserve the data structure with GP flows by significantly reducing the OT cost. Note however that since we were not able to make the CP flow architecture converge with only 1 block in our experiments, it could be a good idea to try to assess more precisely how close GP flow is to the Monge Map. To this end there exists many other approaches which could be considered. One could for example use dedicated OT benchmarks Korroin et al. 2021 or try more robust approaches (Korotin et al. 2022 Makkuva et al. 2020
7 Discussion
This article describes a method to reduce the OT cost of any pre-trained NF model without changing the estimated target density. The proposed method has been tested up to d = 10 and does not require to constrain the architecture of the original model. The procedure relies on building Gaussian preserving flows to rearrange the source distribution to satisfy the OT property. The proposed approach is based on incompressible vector fields which allow to use a nice interpretation of Euler's equations as a geodesic in the group of volumc-preserving diffoomorphisms between the identity and the transformation minimizing the OT cost. This original contribution allows to add regularity condition to the estimated map addition to simply enforcing the OT property.
Perspectives. The numerical experiments presented here pave the way to new research perspectives. First compared to other OT approaches in the NF literature, GP flows is to the best of our knowledge the first one which does not constrain the NF architecture to obtain the Monge map. This could be great advantage when the NF architecture is already constrained for other reasons (for example to satisfy some symmetries or data-related properties). We believe that in this case GP flows may stand out as it could be difficult to further constrain the network to satisfy the OT property with standard approaches. The proposed approach could also be a starting point to investigate other type of (potentially non OT) costs, and more specifically non-quadratic OT costs, such the Ln norm which is much less considered in the literature due to the lack
12

---

Published in Transactions on Machine Learning Research (03/2023)
of theoretical foundations (the OT map is not the gradient of convex function anymore). In GP-based framework, such extension could be easily implemented since we do not rely explicitly on this property. Finally let's mention the case where we consider a map h between two unkown distribution D1 and D2 but none of them is Gaussian. To recover the Monge map associated with h a possibility might be to map D1 to a Gaussian, makes the rearrangement in the Gaussian space with respect to the quadratic cost of h (that, is between D1 and D2) and then map the points back to D1- In this case we could recover the OT map between Dj and D2 even if none of them is Gaussian.
The main limitation of the proposed method is probably related to the number of independent functions require to construct incompressible vector fields in high dimensions. Indeed, as explained in Proposition 2 one needs to construct at least d(d - 1)/2 scalar functions to get all the divergence free vector fields in dimension d and in practice our approach requires d - 1 vector valued functions in R.. It could therefore be desirable to scale up more efficiently with the dimension. One possible way to overcome this difficulty would be to give up on the exact implementation of divergence free functions. For example one could add & penalization term of the divergence in the loss with an unbiased estimator of the divergence (Song et al. 2020). Also, let mention that while Euler's regularization adds nice properties to the flows considered, it also increases the computational time required to train the model. Finally the total duration of the training can be impacted by the initial NF chosen since its evaluation is required to compute the OT cost. NF architectures with fast. forward (or backward) pass should therefore be preferred.
Limitations.
Acknowledgments
The rescarch leading to these results has received funding from IMT Atlantique, Cominlabs Labex (DY- NALEARN project) and ANR (AT4CHILD, ANR-18-CCHIA-0015-00,, project LEMONADE, ANR-21-CF48- 0005 and project OTTOPIA ANR-20-CCHAA-0030)
References
Brandon Amos, Lei Xu, and J. Zico Kolter. Input convex neural networks, 2016. URL https://arxiv.orr
abs/1609.
Vladimir Arnold. Sur géométrie différentielle des groupes de lie de dimension infinie et ses applications à l'hydrodynamique des fluides parfaits. Annales de l'Institut Fourier, 16(1):319-311 1966. doi: 10.5802/air 233. URL www. Saamndamorg/itte//Mee 1166___66_1_________11_11111__11
Kendall E. Atkinson. An Introduction to Numerical Analysis. John Wiley Sons, New York, second edition, 1989. ISBN 0471500232. URL http:p///wwww.ordddaaaarrr//aaas
Jcan-David Benamou and Yann Brenier. computational fluid mochanics solution to the monge-kantorovich mass transfer problem. Numerische Malhemalik, 84:375 393, 2000.
Yann Brenier. Polar factorization and monotone rearrangement of voctor-valued functions. Communications
on Pure and Ayplied Mathematics, 44(4):375 417, 1991 doi: https://00..oo///00000///oo.0000 http://rrrrrr..l..oo.oooo/o/a/aa
URL https: http:/:///mmmneeeeeeeeee wiley. wtt.://oc////cc/eeeeeeeaeooaaa/t 1ttp:/......0//...a..aaaa 311.0/40000000
Christopher P Burgess, Irina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaune Desjardins, and Alexander Lerchner. Understanding disentangling in beta-vae. arXiv preprint arXiv: 804999999 2018.
Claudio Canuto, Alfio Quarteroni, M. Yousulf Hussaini, and Thomas A. Zang. Spectrol Methods. Springer Berlin URT. httt:::///aai.orr//000000////// 3000--00
Heidelberg, 2007. doi: 1110079778-3--00-110
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenand. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018a.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018b.
Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. End-to-end differentiable physics for learning and control. Advances in newral information processing systems, 31, 2018.
13

---

Published in Transactions on Machine Learning Research (03/2023)
Nicola De Cao, Wilker Aziz, and Ivan Titov. Block neural autoregressive flow. In Uncertainty in artificial intelligence, pp.
PMTRR 2020.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint ar 14xx:1410.885 2014.
Yann Dubois, Alexaudros Kastanos, Dave Lines, and Bart Melman. Disentangling vae. htttp://github.com com [Yammbbus//issetttggggngggg march 2019.
Cian Fastwood and Christopher KT Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
Chris Finlay, Augusto Gerolin, Adam M Oberman, and Aram-Alexandre Pooladian. Learning normalizing flows from entropy-kantorovich potentials. arXic preprint Xrxvv:200 06000, 2020a.
Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman. How to train your neural ode: the world of jacobian and kinetic regularization. In International conference machine learning, pp. PMLR, 2020b.
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Taetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Téo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titoran Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1 8, 2021. URL
http://ww...rrr/apppee////////11 html
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Πla Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. In International Conference on Learning Representations, 2018.
Brittany Hamfeldt. Optimal transport. Youtube videos, 2019. URL http://://ww.yyottttl.cc///a/aat [00.://ww....sssssrrrrrrraaaaaer
Irina Higgins, Loie Matthey Arka Pal, Christopher Burgess, Xavier Glorot, Malthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. BLA, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and Lin (eds.), Advances in Neural Inf Information Processing Systems, volume 33, pp. 6850. Curran Associates, Inc., 2020. URL httts:://proceeddngg neurips
Chin- Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville, Neural autoregressive flows, International Conference on Machine Learning, pp. 2078 7788- PMLR, 2018.
Chin- Wei Huang, Ricky TQ Chen, Christos Tsirigotis, and Aaron Courville. Convex potential flows: Universal probability distributions with optimal transport and convex optimization. In International Conference on Learning Representations, 2020.
Skander Karkar, Ibrahim Ayed, Emmanuel de Bézenac, and Patrick Gallinari. A principle of least action for the training of neural networks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 101-117. Springer, 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649 2658. PMLR, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. stat, 1050:1, 2014.
14

---

Published in Transactions on Machine Learning Research (03/2023)
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural informalion processing systems, 31, 2018.
Alexander Koroti,, Lingxiao Li, Aude Genevay, Justin M Solomon, Alexander Filippov, and Evgeny Burnaev. Do neural optimal transport solvers work? continuous wasserstein-2 benchmark. Advances in Newral Information Processing Systems, 34:14593-14605, 2021.
Alexander Korotin, Daniil Selikhanovych, and Evgeny Burnaev. Neural optimal transport. arXie preprint 2rXiv:2201.1920 2022.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998. doi: 2.11099/5.76770
Ashok Makkuva, Amirhossein Taghvaci, Sewoong Oh, and Jason Lec. Optimal transport mapping via imput convex neural networks. In International Conference on Machine Learning, pp. 6672-667769 6681. PMLR, 2020.
Aubry Mathieu, Maturana Daniel, Efros Alexei, Russell Bryan, and Sivie Josef. Seeing 3d chairs: exemplar part-based 2d-3d alignment using a large dataset of cad models. In CVPR, 2014.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing 2017.
sprites dataset.
Derek Onken, Samy Wu Fung, Xingjian Li, and Lars Ruthotto. Ot-flow: Fast and accurate continuous normalizing flows via optimal transport. 2020. doi: 10.48550/
URL https:
//rxxtv.org/bb//2000000000 00104
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Advances in neurol information processing systems, 30, 2017.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1 64, 2021.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport. Center for Research in Economics and Statistics Working Papers, (2017-86), 2017.
Alfio Quarteroni. Numerical models for differential problems; 1st ed. MS&A modeling, simulation and applications. Springer, Milano, 2009. doi: 10.1007/977--884470-0777 URL https://cds. cern. ch/reco
rd/1639539
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference machine learning, pp.
PMLR, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286.
PMLR, 2014.
Filippo Santambrogio. Optimal transport for applicd mathematicians. Birkäuser, NY, 55(55-63):94, 2015.
A.I. Shnirclman. Attainable diffcomorphhsmms Geometric and functional analysis, 3:279-20 1993. URL
A.I. Shnirelman. Generalized fluid flows, their approzimation and applications. Geometric and functional analysis, 4(5))586-66) 620, 1994. URL httt:///euuml.oor///////11111
Jascha Sohl-Dickstein Eric Weiss, Niru Maheswaranathann, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256 2256-2265, Lille, France, 07-09 Jul 2015. PMLR. URL https: attps://procceddngs. mlr. mtt.prress/33//o... Fdickstein15. html
15

---

Published in Transactions on Machine Learning Research (03/2023)
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Aduances in Newral Information Processing Systems, 32, 2019.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score cstimation. In Uncertainty in Artificial Intelligence, pp. 574-584. PMLR, 2020.
Théo Uscidda and Marco Cuturi. The monge gap: regularizer learn all transport maps, 2023. URL https://arxiv. http:////aaxiw..og/bb/////////// wtg/abs//3//a///. 04953
C. Villani. Optimal Transport: Old and New. Grundlchren der mathematischen Wissenschaften, Springer Berlin Heidelberg, 2008. ISBN 9783540710509.
Antoine Wehenkel and Gilles Louppe. Unconstrained monotonic neural networks. Advances in neural information processing systems, 32, 2019.
Liu Yang and George Em Karniadakis. Potential flow generator with 12 optimal transport regularity for generative models. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Linfeng Zhang, Lei Wang, et al. Monge-ampere flow for generative modeling. arXie preprint Xrx 1809. 2018.
A Practical construction of incompressible vector fields in high dimensions
The goal here is to have a GPU-friendly construction of the incompressible vector fields given Proposition In the following we consider stationary divergence free functions but the time variable can be added with no additional work simply by considering functions in gd+1 instead of IRd (the gradients are still taken only the space variables though). All the proofs are given in Appendix B
D
Notations. Regarding the notations we will use the operator diag for two distinct cases: 1) When w is a vector diag(w) denotes the diagonal matrix obtained from the vector w. 2) When W is a matrix diag(W) denotes the vector oblained from the diagonal of W. The operation denotes the scalar product between two vectors. Finally we have adopted the convention that when a scalar multiplies a vector it multiplies each of component.
Practical construction. Let u Rdd Rd-n We construct divergence free function with the functions by defined as
u up-n
+
(g)g))
(16)
otherwise.
To construct this divergence free function we define the matrix (Vu(x))* E IRdxd and the vector 1 E Rd
O 2f Hi,jnn1
(Vu(x))ū
(17)
0,
otherwise.
{
1,
ifient1
1
0,
otherwise.
Lemma 3. Let n N, n S d - and consider the function u Rd Rd-",
Then the vector field
: Rdd + Rdd defined as
vx(x) (Vu)" 1" - [diag Vu)" 1"
(18)
is divergence free.
To construct the functions 18 we need 1) to compute the product between the Jacobian of a vector valued function and constant vector 2) sum the diagonal elements of the Jacobian matrix. Both of these operations
16

---

Published in Transactions on Machine Learning Research (03/2023)
can be done efficiently GPU. Note that in order to satisfy the boundary conditions v 0 one
modify the equation 18 in Lemma (recalling we are taking in practice h; = x 1) to obtain
v*(x) = (x2
(19)
2M - (V))"(x)(x² - - [00] -1) diag( (V((x))"))
where Mi u. uz, if ú n and Mi 0 otherwise.
It possible to recover all the incompressible functions from Proposition by adding the blocks v + v vd2
Proposition Let v be divergence free function IRd. Then there crists d functions vo vd-2
constructed as in Lemma such that
v(x) Σ (x).
The attentive reader would have noticed that with the vector functions u", 0, ...,-- we have a total of independent scalar functions while Proposition only requires the construction of d(d- 1)/2 scalar functions leaving d - additional functions which are not strictly needed to obtain the divergence free vectors. This is due to the vectorized constructions free functions on GPU. Having d 1 additional scalar functions in return is not a big issue since the general order remains (dd)).
(d +
- 19 which allow a fast evaluation of the divergence
Also note that the practical implementation of the equations 16 requires to find a pythonic way efficiently pad group of matrices with different dimensions. We have not yet find such way and therefore have simply chosen in applications to construct d - 1 vector valued functions u² Rdd O,..., d 2 and fill the appropriate dimensions in (16)- [1] with O. Again even if not optimal this is not big issue it multiplics the number of independent scalar functions by a factor 2, but the general order remains (dd))
In practice, we have written the functions u² as the output of a big function u Rd (Rdd-1xxd allowing to evaluate all the functions u" in single pass. The vector u is written as the composition of linear functions with some simple non-linearity
nxx) = Mn + b
x, = σ (M + bz-1), i n, Xg X,
(20)
where M, are rectangular matrices, b, a vector field and typically we have taken a = tanh. One big advantage of the formulation 220 is that the Jacobian of (and therefore all the functions u²) can be computed analytically
Vu(x) = Mn VXn, Vx; = diag(o' (o'g (M-IVVx-- + b(-1))
(21)
for = 1,..., n 1. The formulation 21 therefore allows fast evaluation of the term (18 in particular when summing the diagonal elements of the Jacobian. In our experiments we have noticed that the analytical formulation of the Jacobian 21 was faster than using lorch. aulograd.
B Technical material
B.1 Gaussian preserving flows
B.1.1 Proof of Proposition I
We recall that here erf Rd - Rd the distribution function of one dimensional Gaussian applied component wise.
17

---

Published in Transactions on Machine Learning Research (03/2023)
Proposition. A map is smooth Gaussian preserving function satisfying 3 Φ: (-1,1) - (-1,1) such that det VΦl = and
if and only if there erists
s(x) 22eff= op erf
x XER Rd.
(22)
Proof. We recall some basic properties about the distribution function of a one dimensional Gaussian and its inverse. One has for r E R
2
{ dd.
d
2
er((x)
VT 0
dt,
dr
er((x)
= V
(23)
orf = Va (()) 2
dx
Consider the function Φ: (-1, - (-1, 1)dt defined as
Φ(x) erf v2
V2eff
(24)
The goal here is to show that det VΦl = 1 then equation
22 will follows from 24 By definition
22 (x) (xx)
det VΦ(x)) = det
crf
V2
Applying the equalities
23
component wise and denoting x (21, one gets
det VΦ(x)
II
2
o2 (x))?
VII
det Vs(V2erf-14 (x))
V2
III
2
(erf (x))
2
since the determinant of the composition is the product of the determinants. That
III (ver det -s(v2erf=
det vœ(())
cof (x))
which can be written
det VΦ(x)] = (x)) (x)]))/22
det Vs -s(V2eerf((x))) (x))l
Finally using det (s(x) = (llx(x)))2- one obtains det V vΦ(x)] = 1.
Following the same kind of arguments one shows the other way around that is if s(x) = V2erf opocrf( x with det V then det Vs(x)) = el ((])=(x))) - 12/2 that is s is Gaussian preserving.
]
B.1.2 Orientation reversing functions
In the following Lemma we prove that very orientation reversing function satisfying det V =-1 can be written the composition of volume and orientation preserving function and the function o(;
(-11, 23, (3.,,,,
Lemma. Assume 2b Rdd Rd is a function satisfying det V = -1 and let h(x1,..., Itd)
(-I1, 23; vd). Then there exists a volume and orientation preserving function & such that ab d
Proof. We define & as Φ 2b h. This function is indeed volume and orientation preserving since it satisfies det VΦ det V det Vh 1. By noticing h o h Id one gets ab & h.
18

---

Published in Transactions on Machine Learning Research (03/2023)
B.1.3 Proof of Lemma
Lemma. Assume the Monge map m and the NF architecture are C' diffeomorphisma. Then the corre- spordiigg GP flow s is C₁ the associaled function & is also Ct and either salisfies det Vo((x) = everywhere or det VØ(x)= -1 everywhere.
Proof. The definition of :- oggg ensures that is indeed C Morcover since is invertible cither det Vg > everywhere or det Vg < 0 everywhere (if det Vg(æ) = 0 this would mean that is not invertible at this point) and the same argument applies to tog and to the Monge map m. Therefore the equality s = mog implies that either det Vs everywhere or det Vs everywhere. The equality & = erf & 2eeff /2
shows that Φ also C and that the sign of det V does not change.
B.2
Divergence free functions
B.2.1 Proof of Proposition 2
Proposition. Consider arbitrary vector field v R2d -> Rd.
Then V P.v- 0 if and only if there erists
smooth scalar functions v/ I2 - IR, with v/ -t such that
(x(x) = Σ Ox (x)) i= 1=,..dd
(25)
where V = (Ui,...,
Proof. 1) First prove that V v= 0. The divergence of can be written
V v Σ Or:
Σ Oa Vij
Using d/ = -b one has
v.= Σ
Oz, -
Σ
0z
(26)
For the term ΣEΣ, Oz, Vi on the left hand side one can sum over the index j first instead of the index that is
jet
ΣΣ Oz, = ΣΣ
0n, uz.
151
Injecting this equality in
one gets
V.= ΣΣ
Or O2,
ΣΣ 3
pn 0n,
159
2) Now prove that every vector field satisfying V 0 can be written under the form 25 from Stephen Montgomery-Smithh is available onlind for completeness we rewrite it here. The proof is made by induction with the following assumption.
The proof
Let N, k d. Given smooth vector field such that div == E Us 0, there erists scalar functions voJ Rdd + TR, 1 1 i,j S k with vif = - such that v; = E, 0,15..
Assumption 1.
19

---

Published in Transactions on Machine Learning Research (03/2023)
The Assumption 1 is trivial for A 0. Suppose it is true for k prove it for k: assume divæ 0 and let.
fi .., (I,, mn) = [
mm
us (5, 12, Ln, I2,,
(27)
Since Oifi = U2 one has
(11 + fi) + 0202 + ... Ok 11 O. there exists functions ز/ with vn = -w! such that
Thanks to Assumption
k- 1/3 =
+ f1 = Σ
Σ ==1
J5 for 2 2
(28)
==1
Now we define
()) /0/0
f2(31,-..
U& 1k 1,, )df
(29)
fo
fi (A F.))
then
0xf2 = -n.
(30)
and using 27
in 29
gets
0,52 = Up (x(1111.. 22 12 0,.., xz)
(31)
Oy Uk (2).......
2x... #k E,..., Ra) = -2k
Now we extend the functions thd.
i < k- by defining ws - and vx - for s k.
Then extending the equations 28
with & one has
* Σ
0 =
8xf2 = 01.
Σ 0
30 in the first equation. Moreover with the definition of for 2 25isk and where we used the equality the function wbth one has
2 = -0 = Uk:
j=1
thanks to 31 This proves Assumption for k.
B.2.2 Proof of Lemma
Lemma. Let Ω [-1, and consider the coefficients
= hz (2,))
where h₂(x,) = h/ - 1)A2 + 1), h. h are functions satisfying h,(0) = 1, = 0 and 3 (x) Rd + R
are bounded functions. Then the function defined in Proposition satisfies V v 0 and v on i.
Proof. Indeed since vig there is no 0z term which appear in the sum of 7 for the component 2/. The term hy (x:)) can therefore be factored that is vi 0 if x; = +1. Hence on M..
20

---

Published in Transactions on Machine Learning Research (03/2023)
B.2.3 Proof of Lemma
Lemma. Let n E N, n & d-2 and consider the function u R. -> RR-=". Then the vector field v R - R
defined as
v"(x) = (Vu)" 1 [diag( Vu)" 1"
(32)
is divergence free.
Proof. As explain in Appendix A the lormulation 32 is equivalent to consider v under the form with
ds defined as in From Proposition the function v is divergence free.
B.2.4 Proof of Proposition 3
Proposition. Let be a divergence free function Rd. constructed as in Lemma such that
Then there evists functions vo vd-2
d-2
v(x) = Σ (x).
(33)
7=0
Proof. Let v(x) be divergence free function and denote (155) its coefficients from We construct the associate vectors u" of the functions v procedurally: for h = 1, d - 1, we iteratively chose u, k-1 arbitrarily and define the other components of the vector = as
k-2
k-1 4j-k+1 = ui
- v.-n) + utjy -
1
(34)
note that the sum is well-defined because the components w are constructed iteratively from k 1 k d 1 and we have adopted the convention En=o O. We claim that with this construction recover the equality (33). Indeed from 34 one has
k-2
(uk - - j2
k+1
Using 16
one gets
2-2
4 = (w7))-2
n=0
Again using 16 one has (w/)" = 0 for n 2 k and therefore
d-2 = Σ (wh), j2
1=0
Since the coefficients dy and (1f)" are all antisymmetric this equality is also satisfied for j < l: + 1. We conclude with the decomposition ( of the divergence free functions.
B.3
Additional material
B.3.1
Total derivative expansion
In this subsection we recall the expansion of the Lagrangian derivative Dr (v(t,x)- V)v(t, X(t, x)). Assume X(t, x) follows the ODE
vtt, = Lv(t, +
4 x = x(t,x( XES, S 0stsT, # x(0,x)= =x,
21

---

Published in Transactions on Machine Learning Research (03/2023)
nb params (pre-trained model) + GP
Model
epochs nb layers
batch size
lr
nb docay euler
eight Gaussian
BNAF+GP
(15.4K) + 332
2000
20
1000
10-2
5
two moons
BNAF+GP
(15.4K) + 332
1000
15
1000
5 2 x 10-3
pinwheel
BN
(15.4K) + 332
800
15
1000
x 10
4
Table 3: Parameters used for the training of GP flows on the 2D toy examples.
We recall that the notation d/dt must be understood as deriving the first variable of v(t, X(t,x)) while D/Dt represents the derivative in time of the function f(1) == v(,, ((, x)). By deriving the first and second variable with respect to one has
D Dt
d (t,, x(t,x))) + d X(t,x) V)v(t, vtt, Ett,,X(t,xx
dt
dt
Using / x1, = v(t, ((t,x) one finally obtains
D
Dt ((t,(( dt x(t,((t,,)) + (v(ttx V)v(t,
C Experiment details
We run the experiments two separate GPUs: a NVIDIA Quadro RTX 8000 and NVIDIA TITAN X. Our loss is given by the negative log-likelihood 11. The FFJORD model has an inverse function directly available in the code, which is not the case for the BNAF model. Therefore, as explained in the Section 5
the GP flow is trained using f-1 applied on a standard mulli-dimennsonal normal distribution for FFJORD, whereas only the training data are used for BNAF.
Toy datascts. We give the parameters used for the 2D toy experiments in Table We consider a training set of 80K samples and testing set of 20K samples, 15 time steps with a Runge-Kutta 4 discretization and a GP flow with two intermediate layers of size 15. For the Euler penalization take the exact same parameters with x = 5 10-4 which is divided periodically by factor 2 the nunber of division is given in
table 3
dSprites, MNIST and chairs datascts. The VAE architecture used in the experiments taken from the github repository of Yann Dubois (Dubois et al.) 2019). The parameters used for the GP flows are given in Table and the architectures of CPFlow are given Table 3 For all test cases we consider GP flow with 15 time steps, three intermediate layers of 50 parameters for the dSprites datasct and four intermediate layers of 100 parameters for the MNIST and chairs dataset. For the Euler penalization we take an initial parameter X = 5 x 10-5 which is then divided 10 times periodically by factor 2 during the training.
D Additional details on 2d examples
We provide additional details some toy examples presented in Section 6.1 First consider the eight Gaussian test case, study the transformation of uniform mesh by the NF model and compare with the CPFlow architecture. As shown in Figure] 7 the mesh transformation when adding the GP flow is getting closer to the CP-Flow one, and the OT cost roughly the same. Another illustration the transformation of the source distribution given the bottom of Figure 7 We clearly see the addod value of GP flow, as the points' configuration gets much closer to an isotropic distribution indicating that we get closer to the Monge map. Note that adding the GP flow does not affect the estimated density the test loss.
22

---

Published in Transactions on Machine Learning Research (03/2023)
Model
# params (pre-trained model) + GP
epochs
batch size
Ir
dSprites dataset
FFJORD+GP
(17.8K) + 10.3K
60*
1024
10-4
FEJORD+GP-EULER
(17.8K) + 10.3K
1250
1024
10-4
MNIST dataset
FFJORD+GP
(36.9K) + 40.6K
1250 1250
1024
x 10-1
-EUURD-GP-EULER
(36.9K) + 40.6K
1024
5 10
Chairs dataset
FFJORD+GP
(36.9K) + 40.6K
1250
1024 1024
X 10 10-4
FFJORD+GP+EULER
(36.9K) + 40.6K
1250
X 10
Table 4: Parameters used for the training of GP flows. Each epoch is made of 2.4 x 10 samples randomly generated from a normal distribution. * The training is stopped carly due out-of-domain particles.
Model
nb neurons
nb layers
nb blocks
Total nb params
CPFlow (1 block)
128
10
89.7 7K
CPFlow (3 blocks)
64
5
3
36.6K
Table Architectures used for CPFlow for the dSprites, MNIST and chairs datasets.
CP-Flow
BNAF
BNAF + GP
Figure 7: Eight Gaussian lest case. Top: density estimation. Middle: deformation of a uniform mesh by the NF model. Bottom: Colored source distribution. From left to right: CP-Flow (OT=2.62, loss=2.86), BNAF (OT-2.88, 1os5=2.85), BNAF+GP (OT-2.60, 1oss=2.85).
23

---

Published in Transactions on Machine Learning Research (03/2023)
GP only
GP + Euler penalization
GP + Euler spectral
-2-
110
338
1.10
325
3.5
375
100
355
315
0.8
111
-5.79
-1.00
on
om on 20 on 125
1100
am 5th Om rer
-3.00
1m 03 AM
53h
Figure Two moons test case with the BNAF model (initial OT-1.35). Top: larget density after the application of the GP flow. Middle: representation of the initial incompressible velocity field vo of the GP flow in (-1, 1)d (where the color map represents the norm of the vector field)
Bottom: trajectories of the Gaussian particles under the action of the GP flow. From left to right: GP only (É = 0.28, OT=1.05), GP with Euler through the penalization procedure 12 (É = 0.09, OT=1.04), GP with a spectral method solving directly the Euler equations (É = 0.08, OT=1.06). For the last two mentioned the initial velocity fields and the trajectories are very similar and the energy & is lowered showing that the penalization ion procedure efficiently solve Euler's equations.
24

---

Published in Transactions on Machine Learning Research (03/2023)
Euler's penalization. We now turn our attention to the two moons dataset and evaluate how close penalization procedure is from the true solution of Fuler equations. To this end we compare our solution with more standard method where the initial condition is optimized through a numerical scheme (implemented in Pytorch to be differentiable) which directly solves the Euler equations, in the line of "differentiable physics' approaches de Avila Belbule-Peres et al. 2018). The baseline numerical scheme
spectral method, very efficient and popular numerical method (Canuto al.) 2007). We do not go into details on how to implement such methods since they are restricted in practice low dimensions and we refer to Canuto al. (2007); Quarteroni 2009) for complete presentation. To have more precise comparison we use 20 layers and learning rate of x 10-3, On Figure we compare three different cases: GP alone, GP with Fuler penalization and GP with Euler constrain through a spectral method. As indicated by the OT costs the final positions of particles are very similar for each model. We observe that the trajectories, the initial incompressible velocit ies and the energy & = # >> Jo fllv(t,,;)li dt are very similar both for the penalization-bsse procedure and the spectral method which show that we can correctly solve the Euler equations in 2D with our penalization approach (the additional benefit. being the generalization to high dimensions). Without the addition of the Euler constraint however, the initial velocity field and the trajectories look very different resulting a higher energy, and thus a transformation that does not correspond geodesic in SDiff
E Interpolation examples
The 2D cases be found for example here http:////ttaaaamm/////////////// _ata..p
25

---

Published in Transactions on Machine Learning Research (03/2023)
Initial latent space
FFJORD
FFJORD GP EULER
Initial intent space
FFJORD
FFJORD GP EULER
Initial latent space
FFJORD
FFJORD GP EULERR
Figure 9: Examples of interpolation for the dSprites dataset where each block correspond to the interpolation of a different data point along the 10 dimensions axis represented by the rows. The dimensions are sorted with respect to their KL, divergence in the VAE latent. space, so the higher rows carry more information while the last rows should leave the image unchanged.
26

---

Published in Transactions on Machine Learning Research (03/2023)
Initial iatent space
FFJORD
FFJORD GP EULER
Initial Intent space
FFJORD
FFJORD GP EULER
Initial latent space
FFJORD
FFJORD GP EULERR
Figure 10: Examples of interpolation for the dSprites dataset where each block correspond to the interpolation of a different data point along the 10 dimensions axis represented by the rows. The dimensions are sorted with respect to their KL, divergence in the VAE latent. space, so the higher rows carry more information while the last rows should leave the image unchanged.
27

---

Published in Transactions on Machine Learning Research (03/2023)
Initial latent space
FFJORD
FFJORD
GP EULER
9
9
9
9
0
4
4
4
4
4
/
/
/
1
/
/
/
/
/
4
9
7
6
6
4
4
4
Initial latent space
FFJORD
FFJORD
GP
EULER
/
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
3
2
2
2
3
3
3
2
2
2
2
2
2
3
2
2
2
2
3
3
3
77
7
3
3
2
2
8
8
2
2
2
2
2
7
7
3
3
3
2
2
6
2
2
2
2
2
2
2
2
22
2
2
2
2
6
2
2
2
2
3
3
3
2
2
2
2
2
2
3
3
3
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2 2
2
2
22
2
2
2
2 2
2
2
2
2
2
2
2
0
0
2
2
2
2
2
2
2
2
2
2
/
2
2
2
2
0
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
3
O
O
9
2
2
2
2
2
2
2
Initial latent space
FFJORD
FFJORD
GP
EULER
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
4
4
4
6
6
6
6
6
6
6
6
6
6
6
4
4
6
6
6
6
6
6
6
8
3
3
3
3
6
6
6
6
6
8
3
6
6
6
8
3
3
3
6
6
6
6
6
6
6
3
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
8
9
9
9
6
6
6
6
6
6
6
6
6
6
6
6
6
/
6
6
6
6
4
6
6
6
6
6
6
6
Figure 11: Examples of interpolation for the MNIST dataset where each block correspond to the interpolation of a different data point along the 10 dimensions axis represented by the rows. The dimensions are sorted with respect to their KL divergence in the VAE latent space, so the higher rows carry more information while the last rows should leave the image unchanged.
28

---

Published in Transactions on Machine Learning Research (03/2023)
Initial latent space
FFJORD
FF-JRDD GP EULER
9
9
9
9
9
0
9
9
9
9
9
9
0
9
9
9
9
0
8
9
9
9
9
9
9
7
7
7
7
9999
9
9
0
0
7
7
9
9
9
4
9
9
8
8
8
3
7
9
9
9
9
0
7
7
9
8
8
3
3
4
4
9
9
9
7
7
9
9
9
9
777
7
6
6
9
9
9
7
7
9
9
9
9
9
9
9
7
7
9
9
9
9
9
7
9
9
9 9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
7
9
9
9
9
9
9
9
9
8
9
9
9
9
9
9
9
4
7
7
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
5
5
3
0
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
2
8
0
9
9
9
9
9
9
9
9
9
9
9
Initial latent space
FFJORD
FFJORD
GP
EULER
9
9
9
9
9
5
8
9
9
9
9
O
/
/
/
/
9
7
2
2 7 9
4
9
7
5
9
/
9
/
9
8
8
7
7
/
I
8
8
5
9
4
Initial latent space
FFJORD
FFJORD
GP
EULER
/
7
7
7
7
5
5
5
5
7
7
0
5
5
7
7
7
9
9
9
4
4
7
9
9
9
9
9
9
5
7
7
7
9
9
4
9
5
5
8
2
2
7
7
7
0
7
5
5
8
2
2
2
4
7
7
7
7
6
6
0
7
5
4
4
4
8
7
7
7
7
7
7
7
7
7
7
5
5
0
7
7
9
9
5
5
5
7
7
7
7
7 7
7
7
7
7
7
7
7
7
9
9
9
7
7
7
7
7
7 7 7
7
7
7
7
7
7
7
7
9
9
9
9
7
7
7
7
7 7
7
7
5
7
7
7
7
7
7
7
7
0
6
5
5
5
5
5
7
777779
7
9
7
7
7
7
7
7
7
5
5
5
7
7
7
7
5
5
7
7
7
7 7
7
7
7
7
7
7
/
5
6
7
4
4
5
5
5
7
7
7
7
Figure 12: Examples of interpolation for the MNIST dataset where each block correspond to the interpolation of a different data point along the 10 dimensions axis represented by the rows. The dimensions are sorted with respect to their KL, divergence in the VAE latent space, so the higher rows carry more information while the last rows should leave the image unchanged.
29

---

Published in Transactions on Machine Learning Research (03/2023)
Initial latent space
FFJORD
FFJORD GP EULER
h
h
4
h
h
h
h
h
h
H
h
h
h
#
h
h
d
H
A
h
h
h
h
h
h
h
h
h
h
h
4
4
h
R
h
h
h
h
h
h
h
4
h
h
h
h
h
+
h
h
h
h
h
h
h
Initial latent
space
FFJORD
FFJORD
GP
EULER
Initial latent space
FFJORD
FFJORD
GP EULER
A
R
h
R
R
R
R
R
R
R
R
R
R
Figure 13: Examples of interpolation for the chairs dataset where each block correspond to the interpolation of a different data point along the 10 dimensions axis represented by the rows. The dimensions are sorted with respect to their KL, divergence in the VAE latent. space, so the higher rows carry more information while the last rows should leave the image unchanged.
30

---

Published in Transactions on Machine Learning Research (03/2023)
Initial latent space
FFJORD
FFJORD GP EULER
$
R
F
+
+
R
4
+
R
+
+
FR
%A
+
4
h
4
4
Initial latent
space
FFJORD
FFJORD
GP EULER
#
+
R
A
R
R
A
H
R
H
#
18
R
R
Initial latent space
FFJORD
FFJORD
GP EULER
A
*
4
Figure 14: Examples of interpolation for the chairs dataset where each block correspond to the interpolation of a different data point along the 10 dimensions axis represented by the rows. The dimensions are sorted with respect to their KL, divergence in the VAE latent. space, so the higher rows carry more information while the last rows should leave the image unchanged.
31


Reviews:
[{"review_id": "PNPpibe7ID", "reviewer": "Reviewer_XNgV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* I find the paper to be interesting and quite well-written. It should also be of interest for both the Normalizing Flow (NF) and the Optimal Transport (OT) community. Another benefit of this paper is to popularize the Brenier polar factorization theorem [1] which could also be used in other settings where OT is involved. \n* I enjoyed the originality and flexibility of the framework. Not training yet another normalizing flow and instead training a \"straightening\" of the normalizing flow via a Gaussian preserving mapping is an exciting area of research. The fact that the proposed methodology does not depend strongly on the data structure of the problem by only focusing on changes of the \"latent\" Gaussian distribution (although there is of course some interaction with the normalizing flow $f$ during the training) is definitely an advantage.\n* The numerics confirm that the methodology help reducing the OT cost of the normalizing flow which can help with disentanglement.\n\nWeaknesses:\n* In my opinion the main current problem of the paper is its numerical experiments which are quite low dimensional (I might have missed something but could not find the dimension of the latent space in the VAE example). A lot of operation seem to be quite costly and to scale badly with the dimension. For example imposing the fact that the vector field is divergence free is done by computing the Jacobian of some $\\mathbb{R}^d \\to \\mathbb{R}^d$ mapping which will require at least $O(d^2)$ operations. This prevents any application in image processing for example (where for small resolution $d=64\\times64$ and therefore $d^2 \\approx 16M$). It is also not clear how sensitive the loss is to the Euler's equation regularization and how this scales with the dimension, the robustness with respect to this regularization is not studied in depth. The authors seem to be aware of these limitations and propose in the discussion to drop the divergence free requirement and replace that with a penalization.\n* On the theoretical perspective I think the link with OT could be better stated. Indeed after looking at the Brenier polar factorization formula in [1] the connection with OT was not immediately clear to me. I could find the uniqueness of s and its expression as a maximiser in Theorem 1.2 in [1]. The expression as a minimizer of the quadratic formula comes from p.382 [1]. From my understanding of Theorem 1 as stated by the author the uniqueness of s was not clear. The connection between Theorem 1, i.e. the Brenier polar factorization formula and OT should be explicitly stated as well. As far as I understand we get that after the rearrangement $g^{-1} \\circ s$ also pushes $\\nu$ towards $\\mu$ (this is stated by the authors) and can be written as the gradient of a convex function. This is the transport plan for the quadratic loss using the uniqueness property of the rearrangement (with $u = \\mathrm{Id}$ this time). It would be great to explain and detail a bit more these ideas before diving into the construction of the velocity field.  The uniqueness of $s$ the measure preserving maps also begs the question of the usefulness of the Euler equation regularizer. Indeed by uniqueness of the mapping $s$ we should recover only one mapping (which is the one that makes $g$ a gradient of a convex function). More broadly speaking I think the motivation of the Euler equation regularization is a bit loose. Even in practice it is not clear what are the benefits of these regularizations. Why is it important to have a geodesics in the space of diffeomorphisms?\n* While encompassing many settings such as generative modelling and density estimation (at least in the vanilla setting) restriction to Gaussian measures can be harmful if one is trying to solve tasks in which both the distributions are only accessible via datasets (for example in the case of interpolation between two datasets). While this is not the primary goal of the paper, these applications cannot be considered in this setting which is solely available for the Gaussian case. It is not clear (at least to me) how one could extend this approach outside of the Gaussian distribution setting.\n\n\n[1] Brenier (1991) -- Polar Factorization and Monotone Rearrangement of Vector-Valued Functions \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a novel approach to design Normalizing Flows (NFs) in which the transport mapping satisfies the minimization property of Optimal Transport (OT) (for a squared distance cost). In particular, contrary to many existing approaches the authors do not train a novel NF with a specific architecture or a specific training procedure but instead rely on the Brenier polar factorization theorem [1] which ensures that any function (under mild assumptions) can be decomposed into the gradient of a convex function (the transport plan) and a measure-preserving map denoted $s$. The mapping $s$ is obtained via the minimization of a functional related to the Wasserstein cost.  \n\nIn this work the authors focus on Gaussian preserving mappings. This framework already encompasses applications in generative modeling and density estimation. In order to define a Gaussian preserving mapping, the authors define a Lebesgue preserving mapping and show how that this equivalently define a Gaussian preserving mapping through a tractable transformation. Flows with velocity fields with zero divergence correspond to Lebesgue preserving mapping. Therefore, the authors parameterize the velocity field of the Lebesgue preserving flow. There exist many solutions to such flows and in order to pick one the authors add a regularization term corresponding to the Partial Differential Equation (PDE) part of the Euler's Equation which defines geodesics in the space of Lebesgue preserving diffeomorphisms.\n\nCombining this regularization term, the original squared cost loss and the special architecture for the zero divergence velocity field the authors then train the Gaussian preserving transformation. They illustrate their approach on toy 2D data and show how OT maps and their approach can improve disentanglement preservation for several datasets (dSprites and MNIST) in the latent space of a VAE, following the setting of [2].\n\n[1] Brenier (1991) -- Polar Factorization and Monotone Rearrangement of Vector-Valued Functions \n\n[2] Dubois (2019) -- Disentangling VAE", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Changes:\n* Address the weaknesses highlighted in the previous comment. In particular, more details about the Brenier polar factorization theorem and the motivation of the Euler's equation regularization. I would like to see more ablations/discussions regarding this regularization term.\n* The numerics should be improved. It would be nice to showcase how this approach could be made to work in higher dimensional settings. In particular, I think that adding the regularization of the divergence term (instead of fixing it to zero) would be a great addition to the paper.\n* I would like to see more discussion regarding the limitation with a Gaussian source.\n\nMinor remarks:\n* Why approximating the time-derivative in (12) ? If we have parameterized $v$ with a neural network can we take the gradient w.r.t. the time variable instead?\n* Extension to non quadratic  not clear to me \n* Abstract: \"further constrain\" --> \"further constrained\"\n* Looking at (13) would it be possible to jointly (or iteratively) train the flow and the measure preserving mapping $s$? (In that case one would need to add the training loss of the flow $f$ to (13)). That way one could fine-tune the flow to the measure preserving mapping as well. \n* End of page 2: \"which make it\" --> \"which makes it\"\n* Middle of page 2: \"require to constraint\" --> \"require to constrain\"\n* Page 7: \"comparison.As shown in\" --> \"comparison. As shown in\"\n* Equation (21) there is a minus sign missing in the Gaussian in the definition of erf.\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GJtfwvRxRv", "reviewer": "Reviewer_v55J", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths:* To the best of my knowledge, the submission is the first work which presents volume-orientation preserving maps in the context of Machine Learning and corresponding applications. In spite of the presented method is somehow complicated, the idea is fresh and worth to be presented in ML community. Secondly, the problem statement (improving NFs by rearranging the latent points) is also new. \n\n*Weaknesses:* At first, the proposed method seems to be rather difficult, both in terms of computational complexity and algorithmic implementation. Additionally, the scalability of the approach is also questionable (and it is mentioned by authors as a limitation). Secondly, the method seems to have rather limited scope. The only practically-sound problem under consideration is related to NFs in latent space of VAEs and the idea is to restore disentanglement properties of original VAE latent space corrupted by NF.  The setup considered in the paper prescribes that at first we train a VAE-based model, then we train NF on the latent space of the obtained VAE, and finally we train the GP flow proposed. However, it seems that the last two steps could be easily merged by considering recent OT solvers (including CP-Flow) mentioned in the points 1-2 of Requested Changes section. It is unclear how the proposed methods performs compared to these conventional ways to solve OT.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper under consideration proposes a new idea as well as a method capable to improve a pretrained Normalizing Flow model from the perspectives of Optimal Transport theory. The idea is to train an additional model in the latent space of a NF (which is supposed to be Gaussian) which preserves the latent distribution and forms together with the NF the optimal transport map between the latent distribution and data distribution. The proposed approach is built on specifically-designed volume-orientation preserving maps trained with additional Euler regularization, which improves the mass transformation properties of a learned map. The method is shown to be restoring the disentanglement properties of VAEs with disentangled latent space destroyed by prior NF models. \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The first question concerns the comparison between the proposed approach and CP-Flow model [2]. The CP-Flow is build to be a composition of gradients of convex functions, which is not obligatory to be gradient of a convex function, i.e., OT map. And it is not clear from the paper whether the authors utilize a single building block of CP-Flow (single ICNN model) when reporting OT costs (Table 2 and Figure 3). This question should be clarified.\n* Moreover, it is interesting to compare the proposed approach (pre-trained NF + GP flow) with alternative methods (apart from CP-Flow) which solve the OT problem between latent data distribution and Gaussian distribution directly. Of the special interests are OT solvers based on dual formulation of OT combined with ICNNs [4] or utilizing min-max formulation (MM:R solver from [1]). Note, that min-max based solver presents promising results for OT-related problems [5].\n* It is not clear from the paper why does the authors utilizes rather difficult Euler equations-based regularization (eqs. (11)) instead of direct penalization of Energy (eq. (9)). This choice should be explained, especially since the (eq. (9)) - based regularization are successfully used [3].\n* What do the different shades of green mean in Figure 4?\n* It is not clear for unprepared reader why does the property $\\text{div} v = 0$ (eq. 5) imply $\\text{det} \\nabla \\phi = 1$. Some links/clarifications should be added.\n* Where does pressure $p(t, x)$ come from in the equations (8). Is it an arbitrary scalar function?\n* To my point of view, it is not clear, whether the obtained $s_{\\theta}$ combined with NF recovers the true OT map between the latent distribution and data distribution. There should be more convincing practical validation, at least in toy 2D scenarios. This concern is directly related to the question raised in the point 1: If the CP-Flows used contain more than one ICNN blocks, then the OT costs reported in Figure 3 could be different from real OT costs. My recommendation is to consider more extensive practical (low-dimensional) validation, given that there are methods which recover true OT mappings mentioned in the point 2.\n\n[1] Korotin et. al., Do Neural Optimal Transport Solvers Work? A Continuous Wasserstein-2 Benchmark https://openreview.net/forum?id=CI0T_3l-n1\n\n[2] Huang et. al. Convex potential flows: Universal\nprobability distributions with optimal transport and convex optimization. https://arxiv.org/pdf/2012.05942.pdf\n\n[3] Onken et. al., OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport https://arxiv.org/pdf/2006.00104.pdf\n\n[4] Makkuva et. al., Optimal transport mapping via input convex neural networks https://proceedings.mlr.press/v119/makkuva20a.html\n\n[5] Korotin et. al., Neural Optimal Transport, https://openreview.net/forum?id=d8CBRlWNkqH\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "z0EAoG2cvi", "reviewer": "Reviewer_XMTg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n\n1. As pointed out by the authors, OT-based NF work considering Brenier's polar factorization theorem often focuses on the gradient of the convex function and not on the measure preserving function $s$. I agree with the authors that there is value in bringing this point to the community's attention.\n\n2.  I believe the paper is technically sound, although I have some doubts about whether the method is actually solving the OT problem as claimed in the paper, or just reducing (but not minimizing) the transport cost. I will detail my doubt in the weaknesses section of my review.\n\n3. Despite some minor points that I will also outline in the weaknesses section of my review, the paper is well-written and understandable.\n\n## Weaknesses:\n\n4. My biggest issue with the paper lies in its motivation: I am not convinced that the problem being solved is of any practical relevance. While I can understand the motivation behind training NFs with an OT-based loss in the first place (e.g. the loss might be better behaved than maximum-likelihood, and the achieved models somehow improved; or the architectural constraints might not require efficient computation of the log determinant term) as is done is some of the related works mentioned in the paper, I am really struggling to see how improving the transportation cost of an NF -- while not actually changing the learned distribution -- can be used for anything. \n\n5. I believe the issue I raised above in point 4 is reflected in an unconvincing experiments section: (a) Figures 2 and 3 and Table 2 show the proposed method indeed better matches the Monge map than baseline NF models, but no explanation as to why this might be desirable is provided. (b) The comparisons on disentanglement are similarly unilluminating as to why one should care about the proposed method: Table 1 shows that applying the proposed method on disentangled representations obtained with a VAE achieves better disentanglement than applying FFJORD on the same disentangled representations. It is certainly not common practice to train an NF on disentangled representations from a $\\beta$-TCVAE, and it is once again unclear why anyone would do so in practice. The experiment feels very forced, trying to showcase benefits where there might be none. I think it should also be pointed out that the proposed method gets better disentanglement than this forced baseline, but does not improve upon the disentanglement from the initial VAE's representations, once again highlighting the issue I have with the paper's motivation. Additionally, I am in general skeptical of results claiming disentanglement [1].\n\n6. I do not believe the current experiments convincingly show that, even if one decides to only care about the OT cost and not about how well the NF maps a Gaussian to its target (an assumption that, as discussed above, I do not agree with), then the proposed method is the way to go. The comparison against CPFlow on Table 2 seems to be the only attempt at establishing this, yet: (a) it is odd to carry out the comparisons on the latent space of a VAE rather than on actual data, which I suspect is due to scaling issues due to the parameterization of the vector field, (b) it is unclear that the comparisons are computationally fair, i.e. is the compute used for the FFJORD+GP+EULER or FFJORD+GP models comparable to that of the CPFlow model? The proposed models essentially use two NFs, the pre-trained FFJORD model, plus the Gaussian preserving GP model, which could easily result in a more flexible model than CPFlow, and thus explain the good empirical performance of the proposed method, and (c) it is also fair to wonder if all the complications introduced by the authors are worth it: one can obtain volume-preserving flows in a much easier way (albeit not continuous normalizing flows), e.g. NICE [2]. Showing that the introduced measure-preserving flows that the authors use outperform this naive choice should also be carried out.\n\n7. The authors claim they seek to find the optimal map in OT cost, yet it is unclear to me if this is actually what their objective does, or if the objective can merely decrease but not actually recover the Monge map. Through Theorem 1, we know that the optimal $g$ should be of the form $\\nabla \\psi (s(x))$, where $\\psi$ is a convex function. Later, the trained NF is plugged in as $\\nabla \\psi$, so that only $s$ is learned by minimizing OT cost. I do not think that the pre-trained NF needs to match the gradient of a convex function (let alone that of an optimal $\\psi$), meaning that the set of functions expressible as applying the pre-trained NF to $s(x)$ need not actually contain $g$ (even if one assumes that NFs can represent any diffeomorphism). Similarly, Lemma 2 shows how one can obtain a vector field satisfying a desired condition, but not that any vector field satisfying the required condition can be written as in Porposition 2, which might also mean that the Monge map is not recoverable. Could the authors please elaborate on these points? Am I missing something, or am I correct that the proposed method will only reduce OT cost but not necessarily recover the Monge map? To be clear, even if I am indeed correct, I would not see this as a fundamental issue, just one requiring to tone down language about what the proposed method achieves.\n\n8. In remark 1, the authors point out that the assumptions to Brenier's theorem are not exactly fullfiled as the considered support of the Gaussian is unbounded. Yet, when constructing Gaussian-preserving maps, the authors map back to the bounded set $(-1, 1)^d$ and construct measure-preserving maps (i.e. \"uniform preserving\" maps) there. Can we not reinterpret this as trying to find Monge maps between the uniform distribution on $(-1, 1)^d$ (rather than a Gaussian) and the target distribution? In other words, can we not still rely on Brenier's theorem? If not, it should be explained why not, and if yes, this should be explicitly mentioned in the paper.\n\n9. Finally, I point out some typos/minor errors throughout the manuscript:\n- citations are incorrectly typed in LaTeX, please use both \\citet{} and \\citep{}, depending on context\n- \"A third popular class...\" -> \"A fourth popular class...\"\n- \"smooth diffeomorphism\" is somewhat tautological, as diffeorphisms are $C^1$ by definition, specially if the exact meaning of \"smooth\" is not specified in the paper\n- please introduce notation before using it, e.g. with the pushforward operator\n- \"In practice the flow must satisfy the change...\" -> \"In practice the flow satisfies the change...\"\n- \"\\mu := g_{#} \\nu\" in section 1.1 is not how \\mu is defined\n- \"gaussians\" -> \"Gaussians\" (several times throughout the manuscript)\n- the change from $f$ to its inverse $g$ halfway through the manuscript is confusing, and a single direction should be picked from beginning to end\n- Theorem 1 is not formally stated: (a) a probability space is a triplet of a set, a $\\sigma$-algebra, and a probability measure, not a pair as in the theorem, more importantly (b) the assumption that $g$ is \"non-degenerate\" should be formalized, and (c) the meaning of measure preserving function should be defined before stating Theorem 1\n- I believe there is a missing negative sign in the definition of $\\beta(x)$ in remark 1, and in the definition of the erf function (which by the way should be bolded for consistency) before proposition 1\n- $\\Omega$ is introduced as $(-1,1)^d$ halfway through, you should either denote this set as $\\Omega$ from proposition 1, or not use it at all\n- in the definition of SDiff($\\Omega$), the first comma should be a colon\n- the definition of $\\Omega$ is changed in Lemma 2 from an open to a closed set\n- it is not clear what the pressure $p$ is in equation 8\n- I believe stronger statements than those made in section 4.2 hold: I believe $w_{t,x} = \\nabla p(t,x)$ if and only if $\\nabla w_{t,x}$ is symmetric (rather than only one direction), and similarly $M$ is symmetric if and only if $y^\\top M z - z^\\top M y = 0$ for all $y, z$. I think this is important to highlight, as it means that the used regularizer actually encourages the desired properties\n- \"comparison.As shown in Figure 3\" -> \"comparison. As shown in Figure 3\"\n- the claim that your method \"does not constrain the NF architecture to obtain the Monge map\" is a bit misleading, as constraints are incorporated in the construction of $v$\n\n\n[1 ] Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations, Locatello et al., 2018\n\n[2] NICE: Non-linear Independent Components Estimation, Dinh et al., 2015", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method to transform a given trained normalizing flow (NF) mapping a Gaussian to a distribution of interest (or vice-versa, depending on whether the NF or its inverse is considered) in such a way that the updated flow (1) still maps a Gaussian to the same target distribution, and (2) the optimal transport (OT) cost of the updated NF is now optimal (i.e. the updated flow should be a Monge map between the target distribution and the Gaussian, that is, out of all the transformations mapping the Gaussian to its target, the one achieving minimal transportation cost).\n\nIn order to do this, the authors leverage Brenier's polar factorization theorem, which states that, under suitable conditions, the Monge map can be written as a composition of two functions: the first one a measure-preserving function $s$ (in the setting of interest, this just means that $s$ maps standard Gaussians to standard Gaussians), and the second one which can be written as the gradient of a convex function. The authors then treat the given pre-trained NF as the gradient of the convex function, and aim to find learn the measure-preserving function $s$ which will minimize the transport cost. The authors then take $s$ as a continuous normalizing flow, given by the solution of an ODE defined by a parameterized vector field $v$. With the help of Arnold's theorem, the authors characterize the vector fields which result in a measure-preserving solution to the ODE, thus obtaining constraints on the vector field $v$. The authors enforce these constraints both through architectural choices and a regularization term in the training objective.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Address the concerns I raised in the weaknesses above, particularly points 4-6.", "broader_impact_concerns": "I have no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has already undergone a revision, which has addressed the reviewers' main concerns, and meets TMLR's criteria. Therefore it is ready for publication, subject to the minor revision described below.

Reviewer v55J, in their final comment, writes:

> Still, I believe, that it is worth to compare with or at least mention the alternative OT-solvers listed in my review. In particular, probably, the min-max solvers such as [5] can provide more accurate, compared to one block of CP-Flow, restoration of ground truth Monge maps.

I would like the authors to mention in the final version of the paper the alternative OT solvers that the reviewer has listed in their review, and comment on them as appropriate (no experimental comparison is necessary).
Decision: Accept (poster)