Efficient Gradient Flows in Sliced-Wasserstein Space
Authors: Clément Bonet
Nicolas Courty
François Septier
Lucas Drumetz
Venue: Accepted by TMLR ()

Abstract: Minimizing functionals in the space of probability distributions can be done with Wasser-
stein gradient flows. To solve them numerically, a possible approach is to rely on the
Jordan–Kinderlehrer–Otto (JKO) scheme which is analogous to the proximal scheme in
Euclidean spaces. However, it requires solving a nested optimization problem at each it-
eration, and is known for its computational challenges, especially in high dimension. To
alleviate it, very recent works propose to approximate the JKO scheme leveraging Brenier’s
theorem, and using gradients of Input Convex Neural Networks to parameterize the density
(JKO-ICNN). However, this method comes with a high computational cost and stability is-
sues. Instead, this work proposes to use gradient flows in the space of probability measures
endowed with the sliced-Wasserstein (SW) distance. We argue that this method is more flex-
ible than JKO-ICNN, since SW enjoys a closed-form differentiable approximation. Thus,
the density at each step can be parameterized by any generative model which alleviates the
computational burden and makes it tractable in higher dimensions.

Published in Transactions on Machine Learning Research (11/2022)
Efficient Gradient Flows in Sliced-Wassersteii Space
Clément Bonet
demenu..oncc(@unii-ubb...
Université Bretagne Sud, CNRS, LMBA, Vannes, Fronce
Nicolas Courty
nicolas. corolaas.ourry@iiris.....
Université Brelagne Sud, CNRS, IRISA, Vannes, France
François Septier
franccis..cphhirr@unnioooe
Université Brelagne Sud, CNRS, LMBA, Vannes, France
Lucas Drumetz
lucas. wow..ddrmmettmmmttttttttigo
IMT Allantique, CNRS, Lab-STICC, Brest, France
Reviewed on OpenReview: https: // openreview. net/ forum? id= Au1LNKmRuh
Abstract
Minimizing functionals in the space of probability distributions can be done with Wasser- stein gradient flows. To solve them numerically, possible approach is to rely on the Bondan-Kinddelehrer--OP
(JKO) scheme which is analogous to the proximal scheme in Euclidean spaces. However, it requires solving a nested optimization problem at each it- eration, and is known for its computational challenges, especially in high dimension. To alleviate it, very recent works propose to approximate the JKO scheme leveraging Brenier's theorem, and using gradients of Input Convex Neural Networks parameterize the density (JKO-ICNN). However, this method comes with a high computational cost and stability is- sues. Instead, this work proposes to gradient flows in the space of probability measures endowed with the sliced- Wasserstennnnnnn (SW) distance. We argue that this method is more flex- ible than JKO-ICNN, since SW enjoys a closed-form differentiable approximation. Thus, the density at each step can be parameterized by any generative model which alleviates the computational burden and makes tractable in higher dimensions.
1 Introduction
Minimizing functionals with respect to probability measures is a ubiquitous problem machine learning. Important examples are generative models such GANs (Goodfellow et al., 2014; Arjovsky al., 2017; Lin et al., 2021), VAEs (Kingma & Welling, 2013) normalizing flows (Papamakarios et al., 2019).
To that aim, one can rely on Wasserstein gradient flows (WGF) (Ambrosio et al., 2008) which are curves decreasing the functional as fast as possible (Santambrogio, 2017). For particular functionals, these curves are known to be characterized by the solution of some partial differential equation (PDE) (Jordan et al., 1998). Hence, to solve Wasserstein gradient flows numerically, we can solve the related PDE when is available. However, solving a PDE can be difficult and computationally costly task, especially in high dimension (Han al., 2018). Fortunately, several alternatives exist in the literature. For example, one can approximate instead counterpart stochastic differential equation (SDE) related to the PDE followed by the gradient flow. For the Kullback-Leibler divergence, it comes back to the so called unadjusted Langevin algorithm (ULA) (Roberts & Tweedie, 1996; Wibisono, 2018), but it has also been proposed for other functionals such as the Sliced-Wasserstien Wasserstein distance with an entropic regularization (Liutkus el al., 2019).
Another way to solve Wasserstein gradient flows numerically to approximate the curve discrete time. By using the well-known forward Euler scheme, particle schemes have been derived for diverse functionals such the Kullback-Teiiba divergence (Feng et al., 2021; Wang et al., 2021; 2022), the maxinum mean discrepancy

---

Published in Transactions on Machine Learning Research (11/2022)
(Arbel et al., 2019), the kernel Stein discrepancy (Korba et al., 2021) or KALE (Glaser et al., 2021). Salim et al. (2020) propose instead a forward-backward discretization scheme analogously to the proximal gradient algorithm (Bauschke et al., 2011). Yet, these methods only provide samples approximately following the gradient flow, but without any information about the underlying density.
Another time discretization possible is the so-called JKO scheme introduced in (Jordan et al., 1998), which is analogous in probability space to the well-known proximal operator (Parikh & Boyd, 2014) in Hilbertian space and which corresponds to the backward Euler scheme. However, a nested minimization problem, it is difficult problem to handle numerically. Some works use discretization in space (e.g. grid) and the entropic regularization of the Wasserstein distance (Peyré, 2015; Carlier et al., 2017), which benefits from specific resolution strategies. However, those approaches do not scale to high dimensions, as the discretization of the space scales exponentially with the dimension. Very recently, it was proposed in several concomitant works (Alvarez-Melis et al., 2021; Mokrov et al., 2021; Bunne et al.; 2022) to take advantage of Brenicr's theorem (Brenier, 1991) and model the optimal transport map (Monge map) as the gradient of convex function with Input Convex Neural Networks (ICNN) (Amos al., 2017). By solving the JKO scheme with this parametrization, these models, called JKO-ICNN, handle higher dimension problems well. Yet, drawback of JKO-ICNN is the training time due to number of evaluations of the gradient of cach ICNN that is quadratic in the number of JKO iterations. It also requires to backpropagate through the gradient which is challenging in high dimension, even though stochastic methods were proposed in (Huang et al., 2020) to alleviate it. Moreover, it has also been observed in several works that ICNNs have poor expressiveness (Rout et al., 2021; Korotin et al., 2019; 2021) and that we should rather directly estimate the gradient of convex functions by neural networks (Saremi, 2019; Richter-Powell et al., 2021). Other recent works proposed to use the JKO scheme by cither exploiting variational formulations of functionals in order to avoid the evaluation of densities and allowing to use more general neural networks in (Fan et al., 2021), or by learning directly the density in (Hwang et al., 2021).
In parallel, it was proposed to endow the space of probability mcasures with other distances than Wasser- stein. For example, Gallouët & Monsaingeon (2017) study a JKO scheme in the space endowed by the Sewttooooicc--Fishe---eeee
distance. However, this still requires a costly JKO step. Several particles schemes were derived as gradient flows into this space (Lu et al., 2019; Zhang et al., 2021). We can also cite Kalman- Wasserstein gradient flows (Garbuno-Ingge et al., 2020) the Stein variational gradient descent (Tiu & Wang, 2016; Liu, 2017: Duncan et al., 2019) which can be seen as gradient flows in the space of probabilities endowed by a generalization of the Wasserstein distance. However, the JKO schemes of these different metrics are not casily tractable in practice.
Contributions. Tn the following, we propose to study the JKO scheme in the space of probability dis- tributions endowed with the Wasse---asnrnttein (SW) distance (Rabin et al., 2011). This novel and simple modification of the original problem comes with several benefits, mostly linked to the fact that this distance is easily differentiable and computationally more tractable than the Wasserstein distance. We first derive some properties of this new class of flows and discuss links with Wasserstein gradient flows. Notably, we observe empirically for both gradient flows the same dynamic, up to a time dilation of parameter the dimension of the space. Then, we show that it is possible to minimize functionals and learn the stationary distributions in high dimensions, on toy datasets as well as real image datasets, using e.g. neural networks. In particular, we propose to use normalizing flows for functionals which involve the density, such as the negative entropy. Finally, we exhibit several examples for which our strategy performs better than JKO-ICNN, either w.r.l. to computation times and/or w.r.t. the quality of the final solutions.
2 Background
In this paper, we are interested in finding a numerical solution to gradient flows in probability spaces. Such problems generally arise when minimizing functional F defined on P(Rd), the set of probability measures Rd:
min (EP(
F(u),
(1)
2

---

Published in Transactions on Machine Learning Research (11/2022)
but they also be defined implicitly through their dynamics, expressed partial differential equations. JKO schemes are implicit optimization methods that operate on particular discretizations of these problems and consider the natural metric of P(RR) to be Wasserstein. Recalling our goal is to study similar schemes with an alternative, computationally friendly metric (SW), we start by defining formally the notion of gradient flows in Euclidean spaces, before switching probability spaces. We finally give rapid overview of existing numerical schemes.
2.1 Gradient Flows in Euclidean Spaces
Let : Rd + R be functional. A gradient flow of F is curve (í.e. contimouu function from R4 to Rd) which decreases F as much as possible along it. If F is differentiable, then a gradient flow x: [0, - Rd solves the following Cauchy problem (Santambrogio, 2017)
dx(f) - -VF(2()), dr (x(0) Zig
(2)
Under conditions on F (e.g. VF Lipschitz continuous, F' convex semi-convex), this problem admits a unique solution which can be approximated using numerical schemes for ordinary differential equations such as the explicit or the implicit Euler scheme. For the former, we recover the regular gradient descent, and for the latter, we recover the proximal point algorithm (Parikh & Boyd, 2014): let r > 0,
I - xf
T mig
+ F(x) = proxxp(a]])
(3)
2T
This formulation does not use any gradient, and can therefore be used in any metric space by replacing ||-112 with the right distance.
2.2 Gradient Flows in Probability Spaces
To define gradient flows in the space of probability measures, we first need a metric. We restrict our analysis probability measures with moments of order 2: P2(Rd) {u P(Rd), J [==[[[]u((()) < +00). Then, a possible distance on (R is the Wasserstein distance (Villani, 2008), let μ, P2(R)),
/ 112-v11 2112 dy(x,,),
(), min yEll(a..)
(4)
where IT(H,,) v) is the set of probability measures on Rd x Rd with marginals and v.
Now, by endowing the space of measures with W2, we can define the Wasserstein gradient flow a functional F: (2(R R by plugging W2 in (3) which becomes
(2(MAD + +(U)).
1l+1 E arg min (EPI(RR)
(5)
2r
The gradient flow is then the limit of the sequence minimizers when T - 0. This scheme was introduced in the seminal work of Jordan al. (1998) and therefore referred as the JKO scheme. In this work, the authors showed that gradient flows are linked to PDEs, and in particular with the Pokker-Plande equation when the functional FF is of the form
F(r) / Vdp H(r)
(6)
where is some potential function and H is the negative entropy: let denote the Lebesgue measure,
H(x) ( s p(x) log(p(z)) dæ if du = pda +00 otherwise.
(7)
3

---

Published in Transactions on Machine Learning Research (11/2022)
Then, the limit of (p²), when - is curve t + ae such that for all t > 0, pe has density Pa. The curve p satisfies (weakly) the Fokker-Planck PDE
Op a1 - div(pVV) + Ap.
(8)
For more details on gradient flows in metric space and in Wasserstein space, we refer to (Ambrosio et al., 2008). Note that many other functional can be plugged in equation 5 dcfining different PDEs. We introduce here the Fokker-Planck PDE as a classical example, since the functional is counected to the Kullback-Leibler (KT.) divergence and its Wasserstein gradient flow connected to many classical algorithms such as the unadjusted Langevin algorithm (ULA) (Wibisono, 2018). But we will also use other functionals in Section 4 such as the interaction functional defined for regular enough W
/ʃ W(x - y) dn(r)dp(y).
W(n) =
(9)
which admits as Wasserstein gradient flow the aggregation equation (Santambrogio, 2015, Chapter 8)
ap
div (p((((((V))) x)))
(10)
ot
where denotes the convolution operation.
2.3 Numerical Methods solve the JKO Scheme
Solving (5) is not an easy problem as it requires solve an optimal transport problem each step and hence is composed of two nested minimization problems.
There are several strategics which were used to tackle this problem. For example, Laborde (2016) rewrites (5) as a convex minimization problem using the Benamou-Brenier dynamic formulation of the Wasserstein distance (Benamou Brenier, 2000). Peyré (2015) approximates the JKO scheme by using the entropic regularization and rewriting the problem with respect to the Kullback-Leibler proximal operator. The problem becomes casier solve using Dykstra's algorithm (Dykstra, 1985). This scheme was proved converge to the right PDE in (Carlier al., 2017). It was proposed to the dual formulation in other works such (Caluya & Halder, 2019) or (Frogner & Poggio, 2020). Cancès et al. (2020) proposed to lincarize the Wasserstein distance using the weighted Sobolev approximation (Villani, 2003; Peyro, 2018).
More recently, following (Benamou et al., 2016), Alvarez-Melis et al. (2021) and Mokrov et al. (2021) proposed to exploit Brenier's theorem by rewriting the JKO scheme as
convex 2T / Vu(z)) 2112 (
# arg min
(11)
and model the probability measures as 11+1 (Vul P where # is the push forward operator, defined (Vu)= (((((V))~\A)) for all A B(Rd). Then, to solve it numerically, they model convex functions using ICNNs (Amos et al., 2017):
1
/ [Vzuo(r) 1112 duf(() + F((V #1))
0-1
arg min 0 EIINNN)
- 2r
(12)
In the remainder, this method is denoted as JKO-ICNN. Bunne et al. (2022) also proposed to use ICNNs into the JKO scheme, but with a different objective of learning the functional from samples trajectorics along the timesteps. Lastly, Fan el al. (2021) proposed to learn directly the Monge map T by solving at. each step the following problem:
/ [[T(x) - 2112 dpT(x) + F(T
Th E arg
mig min
(13)
27
and using variational formulations of functionals involving the density. This formulation requires only to use samples from the measure. However, it needs to be derived for each functional, and involves minimax opti- mization problems which are notoriously hard to train (Arjovsky & Bottou, 2017; Bond-Taylor et al., 2021).
4

---

Published in Transactions on Machine Learning Research (11/2022)
3 Wasserseen Gradient Flows
As seen in the previous section, solving numerically (5) is challenging problem. To tackle high-dimensional settings, one could benefit from neural networks, such as generalive models, that are known to model high- dimensional distributions accurately. The problem being not directly differentiable, previous works relied on Brenier's theorem and modeled convex functions through ICNNs, which results in JKO-ICNN. However, this method is very costly to train. For a JKO scheme of & steps, it requires O(k²) evaluations of gradients (Mokrov et al., 2021) which can be a huge price to pay when the dynamic is very long. Moreover, requires to backpropagate through gradients, and to compute the determinant of the Jacobian when we need to evaluate the likelihood (assuming the ICNN is strictly convex). The method of Fan et al. (2021) also requires O(k2) evaluations of neural networks, as well as to solve minimax optimization problem at each step.
Here, we propose instcad to use the space of probability measures endowed with the sliced-Wai Wasserstein (SW) distance by modifying adequately the JKO scheme. Surprisingly enough, this class of gradient flows, which are very easy to compute, has never been considered numerically in the literature. Close to our work, Wasscrstein gradient flows using SW as a functional (called Slicod- cesss-ssssr flows) have been considered in (Liutkus et al., 2019). Our method differs significantly from this work, since we propose to compute sliced- Wasserseenn gradient flows of different. functionals.
We first introduce SW along with motivations to use this distance. We then study some properties of the scheme and discuss links with Wasserstein gradient flows. Since this metric is known in closed-form the JKO scheme is more tractable numerically and can be approximated several ways that we describe in Section 3.3.
3.1 Motivations
Slicod-Wasscrste Wasscrstein Distancc. The Wasserstein distance (4) generally intractable. However, in one dimension, for p, E P2(R), we have the following closed-form (Peyré et al., 2019, Remark 2.30)
(2( = [f (u) Fr ( du
(14)
where F-11 (resp. 1) is the quantile function of μ (resp. v). It motivated the construction of the sliced- Wasserstein distance (Rabin et al., 2011; Bonnotte, 2013), as for u.v E P2(R"),
SW2(knv= - Sge. W ( p d)(0)
(15)
where pd (0(x) (x, f) and the uniform distribution on Sd-1 (0 E Rd. ||0||2 = 13.
Computational Properties. Firstly, SW₂ is very easy to compute by a Monte-Carlo approximation (see Appendix C.I). It also differentiable, and hence using e.g. the Python Optimal Transport (POT) library (Flamary et al., 2021), we can backpropagate w.r.f. parameters or weights parametrizing the distributions (see Section 3.3). Note that some libraries allow to directly backpropagate through Wasserstein. However, theoretically, we only have access to a subgradient in that case (Cuturi & Doucet, 2014, Proposition 1), and the computational complexity is bigger (O(n² log versus (n log n) for SW). Moreover, contrary to W2, its sample complexity does not depend on the dimension (Nadjahi et al., 2020) which is important to overcome the curse of dimensionality. However, it is known to be hard to approximate in high-dimension (Deshpande et al., 2019) since the error of the Monte-Carlo estimates is impacted by the number of projections in practice (Nadjahi et al., 2020). Nevertheless, there exist several variants which could also be used. Moreover, a deterministic approach using a concentration of measure phenomenon (and hence being more accurate in high dimension) was recently proposed by Nadjahi et al. (2021) to approximate SW2-
Link with Wasserstein. The sliced- Wasserstein distance has also many propertics related the Wasser- stein distance. First, they actually induce the same topology (Nadjahi et al., 2019; Bayraktar Guoi, 2021) which might justify to use SW proxy of Wasserstein. Moreover, as showed in Chapter 5 of Bonnotte
5

---

Published in Transactions on Machine Learning Research (11/2022)
(2013), they can be related on compact sets by the following inequalities, let >> O, for all p, P(B(O,
SW2(u W2 S C3sW v),
(16)
with c) = and Ca some constant.
Hence, from these properties, we can wonder whether their gradient flows are related not, or even better, whether they are the same or not. This property was initially conjectured by Filippo Santambrogio'. Some previous works started to gather some hints on this question. For example, Candau-Tilh (2020) showed that, while (P2(R*), SW2) not a geodesic space, the minimal length (in metric space, Definition 2.4 in (Santambrogio, 2017)) connecting two measures is W2 up to a constant (which actually ca). We report Appendix A some results introduced by Candau-Tilh (2020).
3.2 Definition and Properties of Sliced Wasserseen Gradient Flows
Instead solving the regular JKO scheme (5), we propose to introduce a SW-JKO scheme, let μo E P2(()))
(M,
vk 2 0, 11 E arg min
F(H))
(17)
(EP)(K")
2r
in which we replaced the Wasserstein distance by SW..
To study gradient flows and show that they are well defined, we first have to check that discrete solutions of the problem (17) indeod exist. Then, we have check that we can pass to the limit - and that the limit satisfies gradient flows properties. These limit curves will be called Sliced- Wasserstein gradient flows (SWGFs).
In the following, restrain ourselves to measures on P2(K) where K C Rd is compact set. We report some properties of the scheme (17) such as existence and uniqueness of the minimizer, and refer to Appendix B for the proofs.
Proposition 1. F ; ( + R be lower semi continuous functional, then the scheme (17) admits minimizer. Moreover, it is unique if 14 is absolutely continuous and F convex or if F is strictly convex.
This proposition shows that the problem is well defined for convex lower semi continuous functionals since we can find at least a minimizer at each step. The assumptions on F are fairly standard and will apply for diverse functionals such as for example (6) or (9) for V and W regular enough.
Proposition 2. The functional is non increasing along the sequence of minimizers (HŞ))-
As the ultimate goal is to find the minimizer of the functional, this proposition assures us that the solution will decrease along it at each step. If J is bounded below, then the sequence (F(xT) will converge (since it is non increasing).
More generally, by defining the piecewise constant interpolation as n² μi and for all k > 0, L Jr, (k + 1)r], (t( = M5+11 we can show that for all t < s, SWa(H Santambrogio (2017), we can apply the Ascoli-Arzelà thcorem (Santambrogio, 2015, Box 1.7) and extract a converging subsequence. However, the limit when t - possibly not unique and has a priori no relation not a geodesic space, but rather a "pseudo-geodesic" space whose true
S C(ttss s/ +T²). Following
with F. Since (P2(R"), geodesics are caW2 (Candau-Tie Tilh, 2020) (see Appendix A.1), we cannot directly apply the theory introduced in (Ambrosio et al., 2008). We leave for future works the study of the theoretical properties of the limit. Nevertheless, we conjecture that in the limit - SWGFs converge toward the same measure as for WGFs, We will study it empirically in Section 4 by showing that we are able to find as good minima as WGFs for different functionals.
Limit PDE. We discuss here some possible links between SWGFss and WGFs, Candau- Tilh (2020) shows that the Euler-Lagrange equation of the functional (6) has similar form (up to the first variation of the
in private communicaions

---

Published in Transactions on Machine Learning Research (11/2022)
SWGF
WGF
SWGF
WGF
0.6
1.75
5
1.25
0.0
3
-2
-4
0.50
-1
-0.6
-6
0.00
-0.5
0.0
0.3
-0.5
0.0
0.5
(a) F()) = W3 (1,
-/ 8z.)
(b) Interaction Functional (9)
Figure Comparison of the trajectories of (dilated by d = Sliced- Wasserstein gradient flows (SWGF) and Wasserstein gradient flows (WGF) of different functionals. On the left, the stationary solution is a uniform discrete distributions. On the right, the stationary solution is Dirac ring of radius 0.5. Blue points represent the initial positions, red points the final positions, and green points the target particles.
distance, see Appendix A.2). Hence, he conjectures that there is a correlation between the two gradient flows. We identify here some cases for which we relate the Sliced- Wasserstein gradient flows to the Wasserstein gradient flows.
We first notice that for one dimensional supported measures, W2 and SW₂ are the same up to a constant Vd, Le. let μ., E P2(RR) be supported on the same line, then SW3 v) W3 (m. v)/d. Interestingly enough, this is the same constant as between geodesics. This property is actually still true any dimension for Gaussians with covariance matrix the form ela with 0. Therefore, we argue that for these classes of measures, provided that the minimum each step stays in the same class, we would have dilation of factor d between the WGF and the SWGF. For example, for the Pokker-Planck functional, the PDE followed by the SWGF would become de øt d(div(pVV) + Ap). And, by correcting the SW-JKO scheme
d SW2 (1, PD) + F(u),
1k E arg min (EP)(R4) 2r
min
(18)
we would have the same dynamic. For more general measures, it not the case anymore. But, by rewriting SW3 and W2 w.r.t. the centered mcasures Jı and V, as well the mcans my = f x du(xr) and m, = I dv(x),
we have:
(W) || - m SW2(
(2, 1| - m 1 W2
(19)
Hence, for measures characterized by their mean and variance (e.g. Gaussians), there will be constant d between the optimal mean of the SWGF and of the WGF. However, such direct relation is not available between variances, even on simple cases like Gaussians. We report in Appendix B.4 the details of the calculations.
We draw on Figure trajectories of SWGFss and WGF' with F(H) W2 Sx,) and F as in (9) with W chosen in Section 4.2. For the former, the target is discrete measure with uniform weights and, using the same mumber of particles in the approximation An, and performing gradient descent the particles as explained in Section 3.3, we expect the Wasserstein gradient flow to push each particle the closest target particle. This is indeed what we observe. For the latter, the stationary solution is a Dirac ring, as further explained in Section 4.2. both cases, by using a dilation parameter of d, we observe almost the same trajectories between SWGF and WGF, which an additional support of the conjecture that the trajectorics of the gradient flows in both spaces are alike. We also report in Appendix D.1 evolutions along the approximated WGF and SWGF of different functionals.
3.3 Solving the SW-JKO Scheme in Practice
As a Monte- Carto approximate of SW can be computed in closed-form, equation 17 is nested minimiza- tion problem anymore and is differentiable. We present here few possible parameterizations of probability
7

---

Published in Transactions on Machine Learning Research (11/2022)
Algorithm 1 SW-JKO with Generative Models
Input: Ho the initial distribution, K the number of SW-JKO steps, the step size, F the functional, N the number of epochs to solve each SW-JKO step, N the batch size for == to K do
Initialize a neural network gø e.g. with gó
for 1 to Ne do
Sample 2j ; (k)
(k) (k+1)
pz i.i.d
1j D0 (2j (k) (++1) 90 (2)
k+1 (z)ht2)
2j
// Denote AT N
E==1d
Egess plat1
8
N - 82p-
J 1 SW3 AT+1)
Backpropagate through J w.r.t.
Perform gradient step using e.g. Adam end for end for
distributions which we can use in practice through SW-JKO to approximate the gradient flow. We further state, as an example, how to approximate the Fokker-Planck functional (6). Indeed, classical other function- als can be approximated using the same method since they often only require to approximate an integral w.r.t. the measure of interests and to evaluate its density as for (6). Then, from these parameterizations, we can apply gradient-basse optimization algorithms by using backpropagation over the loss at cach stop.
Discretized Grid. A first proposition is to model the distribution on a regular fixed grid, as is done e.g. in (Peyré, 2015). we approximate the distribution by a discrete distribution with a fixed grid on which the different samples are located, then only have to learn the weights. Let us denote M N (k) we use N samples located at (2:) and E =. Let EN denote the simplex, then the optimization problem (17) becomes:
P; 8e, where
SW
1 i= Piò..
(4)) + F
min (pt)eEE
prog.).
(20)
2T
The entropy is only defined for absolutely continuous distributions. However, following (Carlier et al., 2017: Peyré, 2015), we can approximate the Lebesgue measure as: L = - 6z, where I represents a volume of each grid point (we assume that each grid point represents a volume element of uniform size). In that case, the Lebesgue density can be approximated by (4), Hence, for the Fokker-Plande (6) example, we approximate the potential and internal energies as
N v(u) - v(z)p(o)de =e VV(a)) H(p)= (uw-| / log (p(x))p(z)dz= =
N
> log (t)
(
(21)
To stay on the simplex, we use a projected gradient descent (Condat, 2016). A drawback of discretizing the grid that it becomes intractable in high dimension.
With Particles.
We can also optimize over the position of a set of particles, assigning them uniform weights: 14 = + E 8 ggy. The problem (17) becomes:
SW3
EN Sx, N
1
1 + FF( Eon))
min (xc)
(22)
2T
In that case however, we do not have access the density and cannot dircctly approximate H. (or more generally internal energies). A workaround is to use nonparametric estimators (Beirlant al., 1997), which is however impractical in high dimension.
8

---

Published in Transactions on Machine Learning Research (11/2022)
Generative Models. Another solution to model the distribution is to use generative models. Let denote go 2 + X' such a model, with 2 a latent space, 0 the parameters of the model that will be learned, and let pz be a simple distribution (e.g. Gaussian). Then, we will denote pi+1 = (99 (++1) #PZ. The SW-JKO scheme (17) will become in this case
SW2 + ) + K+ F((g))))+p2)
min
(23)
2T
To approximate the negative entropy, we have to be able to evaluate the density. straightforward choice that we use in our experiments is use invertible neural networks with a tractable density such normalizing flows (Papamakarios et al., 2019; Kobyzev et al., 2020). Another solution could be to use the variational formulation as in (Fan et al., 2021) as we only need samples in that case, but at the cost of solving minimax problem.
To perform the optimization, we can sample points of the different distributions at each step and use a Monte- Carlo approximation in order to approximate the integrals. Let 24 ~ pz ii.d, then ge())) ~ (ga) = μ and
1
1 Σ (log(PZ<z)) log (2.()))))))))
V(n) = Σ V(oo(z)), H(N) N
(24)
using the change of variable formula in H.
We sum up the procedure when modeling distributions with generative models in Algorithm We provide the algorithms for the discrotized grid and for the particles in Appendix C.2.
Complexity. Denoting by d the dimension, K the number of outer iterations, Ne the number of inner optimization step, N the batch size and L the number of projections to approximate SW, SW-JKO has a complexity of O(KN, LN lo) versus O(KNe((K + d)N + d))) for JKO-ICNN (Mokrov et al., 2021) and O(K2N,NN, for the variational formulation of Fan al. (2021) where Nm denotes the number of maximization iteration. Hence, we that the SW-JKO scheme is more appealing for problems which will require very long dynamics.
Direct Minimization. eterize the distributions as described this scction and then to perform a direct minimization of the functional by performing a gradient descent on the weights, i.e. for instance with generative model, solv- ing ming #()))) While it is viable solution, papers implementing Wasserstein gradient flows with neural networks via the JKO scheme. This problem is theoretically not well defined as a gradient flow on the space of probability measures. And hence, it has less theoretical guarantees of convergence than Wasserstein gradient flows. In our experiments, we noted that the direct minimization suffers from more numerical instabilities in high dimensions, while SW acts as a regularizer. For simpler problems however, the performances can be pretty similar.
straightforward way to minimize a functional μ + F(u) would be to param-
noted that this is not much discussed in related
4 Experiments
In this section, we show that by approximating sliced-Wassee Wasscrstein gradient flows using the SW-JKO schome (17), we are able to minimize functionals well Wasserstein gradient flows approximated by the JKO- ICNN scheme and with a better computational complexity. We first evaluate the ability to learn the station- ary density for the Fokker-Planck equation
(8) in the Gaussian case, and in the context of Bayesian Logistic Regression. Then, we evaluate it on an Aggregation equation. Finally, we use SW as a functional with image datasets as larget, and compare the results with Wasserreessnnneinn flows introduced in (Liutkus et al., 2019).
For these experiments, we mainly use generative models. When it is required to evaluate the density (e.g. to estimate H), we use Real Non Volume Preserving (ReaINVP) normalizing flows (Dinh al., 2016). Our experiments were conducted using PyTorch (Paszke et al., 2019).
9

---

Published in Transactions on Machine Learning Research (11/2022)
Stationary distribution (t=8d)
d=10
EM 1K
EM 50K
2
JKO-ICNN
EM 10K
ReaINVP-SWGF
1991.SymMLLuum. (oLL(a,, (4)
-1
-2
-2
d 12
Figure On the left, SymKT. divergence between solutions at time t = 8d (using 0.1 and 80 steps oquation 17) and stationary measure. On the right, SymKL between the true WGF 14 and the approximation with JKO-ICNN μ., run through 3 Gaussians with 0.1. We observe instabilities at some point.
4.1 Convergence to Stationary Distribution for the Fokker-Planck Equation
We first focus on the functional (6). Tts Wasserstein gradient flow is solution of a PDE of the form of (8). In this case, it is well known that the solution converges as t - оo towards unique stationary mcasure H² x e-v (Risken, 1996). Hence, we focus here on learning this larget distribution. First, we will choose as target a Gaussian, and then in a second experiment, we will learn a posterior distribution in a bayesian logistic regression sctting.
Gaussian Case. Taking V of the form V(x) - - b) for all x E Rd with A symmetric positive definite matrix and m E Rd, then the stationary distribution is μ = =((m,A-) We plot in Figure 2 the symmetric Kullback-Teibles (SymKT.) divergence over dimensions between approximated distributions and the true stationary distribution. We choose т = 0.1 and performed 80 SW-JKO steps. We take the mean over 15 random gaussians for dimensions d E ...,.. 12] for randomly generated positive semii-definit matrices using "make_ __dd__________________ matrix" from scikit-learn (Pedregosa et al., 2011). Moreover, we use RealNVPs in SW-JKO. We compare the results with the Unadjusted Langevin Algorithm (ULA) (Roberts & Tweedie, 1996), called Euler-Maruyane, (EM) since it is the EM approximation of the Langevin equation, which corresponds to the counterpart SDE of the PDE (8). We see that, in dimension higher than 2, the results of the SWGF with RealNVP are better than with this particle scheme obtained with a step size of 10-3 and with either 103, 10a or 5 10* particles. We do not plot the results for JKO-ICNN as we observe many instabilities (right plot in Figure 2). Moreover, we notice a very long training time for JKO-ICNN. We add more details in Appendix D.2. We further note that SW acts here as regularizer. Indeed, by training normalizing flows with the reverse KL (which is equal to equation 6 up to constant), obtain similar results, but with much more instabilities in high dimensions.
Curse of Dimensionality. Even though the Aisser-Wassrsstein distance sample complexity does not suffer from the curse of dimensionality, it appears through the Monte-Carlo approximation (Nadjahi al., 2020). Here, since SW plays reg- ularizer role, the objective is not necessarily to ap- proximate it well but rather to minimize the given functional. Nevertheless, the number of projections can still have an impact on the minimization, and we report on Figure+ the evolution of the found minima w.r.l. the number of projections, averaged over 15 random Gaussians. We observe that we do not need much projections to have fairly good results, even in higher dimension. Indeed, with more than 200 projections, the performances stay pretty stables.
0.5
6-20
6=30
d-150
-1.5
200
400
600
800
1000
Number projections
Figure 3: Impact of the number of projections for a fixed number of epochs.
10

---

Published in Transactions on Machine Learning Research (11/2022)
23
39
39
19
60
do
(a) Steady state on the discretized grid
(b) Steady state for the fully connected neural network
(c) Steady state for parti- cles
(d) Steady state for JKO- ICNN (with T = 0.1)
Figure Steady state the aggregation equation for a= 4, b 2. From left to right, we plot for the discretized grid, for the FCNN, for particles and for JKO-ICNN. We observe that JKO-ICNN does not recover the ring correctly as the particles are not evenly distributed on it.
Bayesian Logistic Regression. Following the experiment of Mokrov al. (2021) in Section 4.3, propose to tackle the Bayesian Logistic Regression problem using SWGFs. For this task, we want to sample from D) where D represent data and x = (w, log a) with to the regression weights on which we apply a Gaussian prior po(wla)= = (W; 0, !) and with Po(a) = I(a; 1, 0.01). In that case, we use (a)) - log DD to learn p(a] D). We refer to Appendix D.2 for more details on the experiments, well as hyperparaameters. We report in Table the accuracy results obtained on different datasets with SWGFs and compared with JKO-ICNN. We also report the training time and see that SWGFs allow to obtain results as good as with JKO-ICNN for most of the datasets but for shorter training times which underlines the better complexity our scheme.
Table 1: Accuracy and Training Time for Bayesian Logistic Regression over 5 runs
JKOJICNN
SWGF+RealNVE
Dataset
Are
Acc
covtype
0.755 +5 10
33702s
U.755
+3 10
103s
german
0.679
15-10 10-3
2123s
0.68
+5 10
82s
diabetis
0.777 +7 17-5
4913s
0.778
+2 10-3
122s
twonorm
0.981
+2-10 10
6551s
0.981
+6-10
301s
ringuorm
0.736 110
1228s
0.741
16 10
82s
banana
0.55 #10
1229s
0.559
110
66s
splice
0.847 #2 10-3
2290s
0.85
#2 10-3
113s
waveform
0.782 18 10
856s
0.776
18 10
120s
image
0.822 110
1947s
0.821
+3 10
72x
4.2 Convergence to Stationary Distribution for an Aggregation Equation
We also show the possibility to find the stationary solution different PDEs than Fokker-Planck. For example, using an interaction functional of the form
W(A) - If W(x - y) dp(x)dp(v).
(25)
We notice here that we do not need to evaluate the density. Therefore, we can apply any neural network. For example, in the following, we will use a simple fully connected neural network (FCNN) and compare the results obtained with JKO-ICNN. We also show the results when learning directly over the particles and when learning weights over regular grid.
Carrillo et al. (2021) use a repulsive-attractiv interaction potential W(x) = |=ll |=ا In this case, they showed empirically that the solution a Dirac ring with radius 0.5 and centered at the origin when starting from μ (N, 0.25 With 7 0.05, we show on Figure 4 that we recover this result with SWGFs for different parametrizations of the probabilities. More precisely, first use discretized grid of 50 x 50 samples of [-1, Then, we show the results when directly learning the particles and when using FCNN. We also compare with the results obtained with JKO-ICNN. The densities reported for the last three methods are obtained through a kernel density estimator (KDE) with a bandwidth manually chosen since we either do not have access to the density, or we observed for JKO-ICNN that the likelihood exploded (see Appendix D.4). It may be due to the fact that the stationary solution does not admit a density with respect the Lebesgue measure. For JKO-ICNN, we observe that the ring shape is recovered, but the samples are not evenly distributed it.
11

---

Published in Transactions on Machine Learning Research (11/2022)
Table 2: FID scores on some datasets (lower is better)
9
3
7
0
6
9
7
6
4
2
+
Methods
MNIST
Fashion
CelebA
SWF (Liutkus et al., 2019)
225.1
207.6
5
2
3
O
Ambinnt
Space
SWGF
ReaINVP
88.1
95.5
/
/
6
SWGP CNN
69.3
102.3
$
6
8
5
AE (golden score)
15.55
31
77
Latent Space
SWCF + AE + ReaINVP
17.8
40.6
90.9
Figure 5: Generated sample obtained through a pre- trained decoder + RealNVP.
SWGF + FCNN
18.3
11.7
SWF
22.5
56.4
91.2
We report the solution at time 7 = 10, and used T = 0.05 for SW-JKO and T = 0.1 for JKO-ICNN. As JKO-ICNN requires O(k2)) evaluations of gradients of ICNNs, the training is very long for such a dynamic. Here, the training took around 5 hours on a RTX 2080 TI (for 100 steps), versus 20 minutes for the FCNN and 10 minutes for 1000 particles (for 200 steps).
This underlines again the better training complexity of SW-JKO compared to JKO-ICNN, which is especially appealing when we are only interested in learning the optimal distribution. One such task is generative mod- eling in which we are interested in learning a target distribution v from which we have access through samples.
4.3 Application on Real Data
In what follows, we show that the SW-JKO scheme can generate real data, and perform better than the associated particle scheme. To perform generative modeling, we can use different functionals. For example, GANs use the censen-Shannon divergence (Goodfellow et al., 2014) and WGANs the Wasserstein-1 distance (Arjovsky et al., 2017). To compare with an associated particle scheme, we focus here on the regularized SW distance as functional, defined as
1 SW XH(P), F(R) - 2
(26)
where V is some target distribution, for which we should have access to samples. The Wasserstein gradient flow of this functional was first introduced and study by Bonnotte (2013) for À x= 0, and by Liutkus et al. (2019) with the negative entropy term. Liutkus et al. (2019) showcased a particle scheme called SWF (Sliced Wasserstein Flow) to approximate the WGF of equation 26. Applied on images such as MNIST (LeCun & Cortes, 2010), FashionMNIST (Xiao et al., 2017) or CelebA (Liu et al., 2015), SWFs need a very long convergence due to the curse of dimensionality and the trouble approximating SW. Hence, they usod instead pretrained auloencoder (AE) and applied the particle scheme in the latent space. Likewise, we use the AE proposed by Liutkus et al. (2019) with latent space of dimension d = 48, and we perform SW-JKO steps thoses images. We report on Figure 5 samples obtained with RealNVPs and on Table 2 the Fréchet Inception distance (FID) (Heusel et al., 2017) obtained between 10 samples. We denote "golden score" the FTD obtained with the pretrained autoencoder. Hence, we cannot obtain better results than this. We compared the results in the latent and in the ambient space with SWFs and see that we obtain fairly better results using generative models within the SW-JKO scheme, especially in the ambient space, although the results are not really competitive with state-of of-the-art methods. This may be due more to the curse of dimensionality in approximating the objective SW than in approximating the regularizer SW.
To sum up, advantage of the SW-JKO scheme is to be able to use easier, yet powerful enough, archilectures to learn the dynamic. This is cheaper in training time and less memory costly. Furthermore, we can tune the architecture with respect to the charactoristics of the problem and add inductive biases (e.g. using CNN for images) or learn directly over the particles for low dimensional problems.
5 Conclusion
In this work, we derive a new class of gradient flows in the space of probability measures endowed with the sliced-Wassersee Wasserstein metric, and the corresponding algorithms. To the best of our knowledge, and despite its
12

---

Published in Transactions on Machine Learning Research (11/2022)
simplicity, this is the first time that this class of flows is proposed in machine learning context. We showed that it has several advantages over state-of-the-art approaches such as the recent JKO-ICNN. Aside from being less computationally intensive, it is more versatile w.r.t. the different practical solutions for modeling probability distributions, such as normalizing flows, generative models or sets evolving particles.
Regarding the theoretical aspects, several challenges remain ahead: gradient flows are still unclear. Second, one needs to understand if, regarding the optimization task, con- vergence speeds guarantecs are changed with this novel formulation, revealing potentially interesting practical properties. Lastly, it is natural to study if popular variants of the as Max-sliced (Deshpande et al., 2019), Distributional sliced (Nguyen et al., 2021), Subspace robust (Paty & Cuturi, 2019), generalized sliced (Kolouri et al., 2019) or projection Wasserstein distances (Rowland et al., 2019) can also be used in similar gradient flow schemes. The study higher-order approximation schemes such as BDF2 (Matthes & Plazotta, 2019; Plazotta, 2018) could also be of interest.
First, its connections with Wasserstein
distance such
Acknowledgmentt
This research was funded by project DynaLearn from Labex CominLabs and Région Bretagne ARED DLearnMe, and by the project OTTOPIA ANR-20-CHIA-0030 of the French National Research Agency (ANR).
References
David Alvarez-Melis, Yair Schilf, and Youssef Mroueh. Optimizing functionals on the space of probabilities with input convex neural networks. arXiv preprint arXiv:2106.0077 00704 2021. (Cited on 2, 4, 29)
Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media, 2008. (Cited on p. 1, 4, 6)
Brandon Amos, Lei Xu, and J Zico Kolter. Input convex neural networks In International Conference Machine Learning, pp. 146-155. PMLR, 2017. (Cited on p. 2, 4)
Michael Arbel, Anna Korba, Adil Salim, and Arthur Gretton. Maximum mean discrepancy gradient flow. Advances in Neural Information Processing Systems, 32, 2019. (Cited on p. 2)
Martín Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: /ttp:://ppeeeeeeeeee Hk4_ 2k5x_e (Cited on p. 4)
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference machine learning, pp.
PMLR, 2017. (Cited p. 1, 12)
Daniel Balagué, José Carrillo, Thomas Laurent, and Gaël Raoul. Nonlocal interactions by repulsive- attractive potentials: radial ins/stability. Physica Nonlinear Phenomena, 260:5 25, 2013. (Cited on 32)
Heinz H Bauschke, Patrick L Combettes, et al. Convex analysis and monotone operator theory in Hilbert spaces, volume 408. Springer, 2011. (Cited on p. 2)
Erhan Bayraklar and Gaoyue Guoï. Strong equivalence between metrics of wasserstein type. Electrovic Communications in Probability, 26:1-13, 2021. (Cited on p. 5)
Jan Beirlant, Edward J Dudewicz, Tászló Györfi, Edward C Van der Meulen, et al. Nonparametric entropy cstimation: An overview. International Journal of Mathematical and Statistical Sciences, 6(1):17-39, 1997. (Cited on p) 8)
Jean-Davis Benamou and Yann Brenier, A computational fluid mechanics solution to the monge-kantorovide transfer problem. Numerische Mathematik, 84(3):375 -93,, 2000. (Cited on p.
13

---

Published in Transactions on Machine Learning Research (11/2022)
Jean-David Benamou, Guillaume Carlier, Quentin Mérigot, and Edouard Oudet. Discretization of functionals involving the monge-ampent operator. Numerische mathemalik, 134(3):611166) 636, 2016. (Cited on p- 4)
Sam Bond-Taylor, Adam Leach, Yang Long, and Chris G Willcocks. Deep generative modelling: A com- parative review of vaes, gans, normalizing flows, energy-based and autoregressive models. orXiv preprint arXiv:2103.04 2021. (Cited on p. 4)
Nicolas Bonnotte. Unidimensional and evolution methods for optimal transportation. PhD thesis, Paris 11, 2013. (Cited on p. 5, 12, 21)
Yann Brenier. Polar factorization and monotone rearrangement of voctor-valued functions. Communications on pure and applied mathemalics, 44(4):375 417, 1991. (Cited on 2)
Charlotte Bunne, Lactitia Papaxanthos, Andreas Krause, and Marco Cuturi. Proximal optimal transport modeling of population dynamics. In International Conference on Arlificial Intelligence and Stalistics. pp. 6511-6528. PMLR, 2022. (Cited on p. 2, 4)
Kenneth F Caluya and Abhishek Halder. Proximal recursion for solving the fokker-planck equation. In 2019 American Control Conference (ACC), pp. 4098-4103. IEEE, 2019. (Cited p. 4)
Clément Cancès, Thomas O Gallouët, and Gabricle Todeschi. A variational finite volume scheme for wasser- stein gradient flows. Numerische Mathematik, 146(3):437 480, 2020. (Cited on p. 4)
Jules Candau-Tine Tilh. Wasserstein and sliced-wasscrstein distances. Master's thesis, Université Pierre et Maric Curie, 2020. (Cited on p. 6, 20, 22)
Guillaume Carlier, Vincent Duval, Gabriel Peyré, and Bernhard Schmitzer. Convergence of entropic schemes for optimal transport and gradient flows. SIAM Journal Mathematical Analysis, 49(2 1111118851111,,1,811 2017. (Cited on p. 2, 4, 8)
José Carrillo, Alina Chertock, and Yanghong Huang. A finite-volume method for nonlinear nonlocal equations with gradient flow structure. Communiication in Computational Physics, 17(1):233 258, 2015. (Cited p. 36)
Jose A Carrillo, Katy Craig, Li Wang, and Chaozhen Wei. Primal dual methods for wasserstein gradient flows. Foundations of Compulational Mathemalics, pp. 1-5, 2021. (Cited on p. 11, 32, 35, 36)
Yuxin Chen and Theodore Kolokolnikov. A minimal model of predator-swand interactions. Journal of The Royal Society Interface, 20114)200 2014. (Cited on 36)
Laurent Condat. Fast projection onto the simplex and the 11 ball. Mathemalical Programming, 158(1): 575 585,585,44 2016. (Cited on p. 26)
Rob Cornish, Authony Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. In International Conference on Machine Learning, pp. 2133- 2143. PMLR, 2020. (Cited on p. 35)
Marco Cuturi and Arnaud Doucet. Fast computation of wasscrstein barycenters. In International conference on machine learning, pp. 683-693. PMLR, 2014. (Cited on p. 5)
Biwei Dai and Uros Seljak. Sliced iterative normalizing flows. In ICML Workshop on Invertible Neural Networks, Normalising Flows, and Explicit Likelihood Models, 2021. (Cited on p. 37)
Ishan Deshpande, Tuan-Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David Forsyth, and Alexander G Schwing. Max-sliced wasserstein distance and its use for gans. Tn Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recogration, pp. 10648 10656, 2019. (Cited on 5, 13, 31)
Laurent Dinh, Jascha Sohl- Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 4rXiv:1605.0880 2016. (Cited on p. 9, 27, 29, 38)
14

---

Published in Transactions on Machine Learning Research (11/2022)
Andrew Duncan, Nikolas Nüsken, and Lukasz Szpruch. On the geometry of stein variational gradient descent. arXiv preprint 0089, 2019. (Cited on p. 2)
Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pp. 975-984, 1985. (Cited on p. 4)
Jiaojiao Fan, Amirhossein Taghvaei, and Yongxin Chen. Variational wasserstein gradient flow. ar preprint 2XXi:2212202243. 2021. (Cited on p. 2, 4, 5, 9)
Kai- Tai Fang, Samuel Kotz, and Kai Wang Ng. Symmetric maltivariate and related distributions. Chapman and Hall/CRC, 1992. (Cited on p. 25)
Xingdong Feng, Yuan Gao, Jian Huang, Yuling Jiao, and Xu Liu. Relative entropy gradient sampler for unnormalized distributions. arXiv preprint orXiv:2110.02787 2021. (Cited on p. 1)
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Alaya, Aurélie Boisbunon, Stanislas Cham- bon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamoniy, levgen Rodko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python opti- mal transport. Journal of Machine Learning Research, 22(78):1-8 8, 2021. URL http: orglrr.org/gge org/ hatg/paprs////// v22/20-451. html. (Cited on p. 5)
Charlie Frogner and Tomaso Poggio. Approximate inference with wasserstein gradient flows. In International Conference on Arlificial Intelligence and Slalislics, pp. 2581 2590. PMLR, 2020. (Cited on p. 4)
Thomas O Gallouët and Leonard Monsaingeon. A jkk splitting scheme for Cantorovich-fishee fisher-rao gradient flows. STAM Journal on Mathematical Analysis, 49(2):1 1000000,,,,, 2017. (Cited on p. 2)
Alfredo Garbuno- Inigo, Franca Hoffmann, Wuchen Li, and Andrew M Stuart. Interacting langevin diffusions: Gradient structure and ensemble kalman sampler. SIAM Journal on Applied Dynamical Systems, 19(1): 412 441, 2020. (Cited on p. 2)
Clark R Givens and Rac Michael Shortt. A class of wasserstein metrics for probability distributions. Michigan Mathematical Journal, 31 (2):331 240, 1984. (Cited on p. 24)
Pierre Glaser, Michael Arbel, and Arthur Gretton. Kale flow: A relaxed kl gradient flow for probabilities with disjoint support. Aduances in Newral Information Processing Systems, 34, 2021. (Cited on p. 2)
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Aduances neural information processing systems, 27:2672 2680, 2014. (Cited on p. 1, 12)
Jiequn Han, Arnulf Jentzen, and Weinan E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505 8510, 2018. (Cited on p. 1)
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural infor- mation processing systems, 30, 2017. (Cited on 12, 37)
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Picter Abbeel. Flow+ Improving flow-basod generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pp. 7722-2730. PMLR, 2019. (Cited on p. 38)
Chin-Wei Huang, Ricky TQ Chen, Christos Tsirigotis, and Aaron Courville. Convex potential flows: Universal probability distributions with optimal transport and convex optimization. arXiv preprint 2rXiv=2012.05942 2020. (Cited on p. 2, 29)
Hyung Ju Hwang, Cheolhyeong Kim, Min Sue Park, and Hwijac Son. The deep minimizing movement scheme. XrXiv preprint ar orXio:2100.1488 14851, 2021. (Cited p. 23)
15

---

Published in Transactions on Machine Learning Research (11/2022)
Richard Jordan, David Kinderlchrer, and Felix Otto. The variational formulation of the fokker- equation. STAM journal on mathemalical analysis, 29(1):1-17, 1998. (Cited on 2,
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiu:1412 111.111..6690 2.800, 2014. (Cited p. 29)
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiw:1312.6114 2013. (Cited on p. 1)
Tvan Kobyzev, Simon Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020. (Cited on
9)
Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo K Rohde. Generalized sliced wasserstein distances. arXio preprinl Xiv:19900 00434, 2019. (Cited on p. 13)
Anna Korba, Fierre-Cyril Aubin- Frankosikn,,kkii, Szymon Majewski, and Pierre Ablin. Kernel stein discrepancy descent. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proccedings of Machine
Learning Research, pp. 1719-5730, PMLR, 2021. (Cited p. 2, 21)
Alexander Korotin, Vage Egiazarian, Arip Asadulaev, Alexander Safin, and Evgeny Burnaev, Wasserstein-2 generative networks. arXiv preprint arXiv:1909. 13082, 2019. (Cited on p. 2, 28)
Alexander Korotin, Lingxiao Li, Aude Genevay, Justin Solomon, Alexander Filippov, and Evgeny Bur- naev. Do neural optimal transport solvers work? a continuous wasserstein-2 benchmark. ar preprint 0rXiv:2106.0190 2021. (Cited on p. 2)
Maxime Laborde. Interacting particles systems, Wasserstein gradient flow approach. PhD thesis, PSL Research University, 2016. (Cited on p. 4)
Jean-François Le Gall. Brownian motion, martingales, and stochastic calculus. Springer, 2016. (Cited on p. 27)
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http://yann. lecun. con/exxb/mnist/ (Cited on p. 12,37)
Alex Tong Lin, Wuchen Li, Stanley Osher, and Guido Montúfar. Wasserstein proximal of gans, arXiv preprint ar.ie:2102. 06862, 2021. (Cited on 1)
Qiang Liu. Stein variational gradient descent as gradient flow. Advances in neural information processing systems, 30, 2017. (Cited on p. 2)
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algo- rithm. Xixiv preprint arXiv: 1006.88.. 04471, 2016. (Cited on p. 2, 30, 31)
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730, 2015. (Cited on 12, 37)
Antoine Liutkus, Umut Simsekli, Szymon Majewski, Alain Durmus, and Fabian-Roberl Stöter. Sliced- wasserstein flows: Nonparametric generative modeling via optimal transport and diffusions. In Interna- tional Conference on Machine Learning, pp. 4104-4113. PMLR, 2019. (Cited on p. 1, 5, 9, 12, 21, 37, 38, 40)
Yulong Lu, Jianfeng Lu, and James Nolen. Accelerating langevin sampling with birth-death. arXie preprint arXiv:1905.090 2019. (Cited on p. 2)
Michael C Mackey. Time's 's arrow: The origins of thermodynamic behavior. Courier Corporation, 1992.
(Cited on p. 26)
16

---

Published in Transactions on Machine Learning Research (11/2022)
Daniel Matthes and Simon Plazotta. A variational formulation of the bdf2 method for metric gradient flows. ESATM: Mathemalical Modelling and Numerical Analysis, 53(1):145-173 172, 2019. (Cited on p. 13)
Sebastian Mika, Gunnar Ratsch, Jason Weston, Bernhard Scholkopf, and Glaus-Robert Mullers. Fisher discriminant analysis with kernels. In Neural networks for signal processing IX: Proceedings of the 1999 IEEE signal processing society workshop (cat. no. 98th8468), pp. 41-48. Icce, 1999. (Cited p. 31)
Petr Mokrov, Alexander Korotin, Lingxiao Li, Aude Genevay, Justin Solomon, and Evgeny Burnaev. Large- scale wasscrstein gradient flows. In Thirty-Fifth Conference on Newral Information Processing Systems, 2021. (Cited p- 2, 4, 5, 9, 11, 27, 28, 29, 30, 31)
Kimia Nadjahi, Alain Durmus, Umut Simsekli, and Roland Badeau. Asymplotic guarantees for learning generative models with the wasses-wass distance. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. (Cited on p. 5)
Kimia Nadjahi, Alain Durmus, Lénaïc Chizat, Soheil Kolouri, Shahin Shahrampour, and Umut Simsekli. Statistical and topological properties of sliced probability divergences. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, vol- ume 33, pp. 20802-20812. Curran Associates, Inc., 2020. (Cited on p. 5, 10, 31)
Kimia Nadjahi, Alain Durmus, Pierre E Jacob, Roland Badeau, and Umut Şimşekli. Fast approxi- mation of the siced-wasserstert distance using concentration random projections. arXiv preprint arXiv:2106. 15427, 2021. (Cited on p. 5, 24, 25)
Khai Nguyen, Nhat Ho, Tung Pham, and Hung Bui. Distributional sliseeewwasersstein and applications to generative modeling. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 9-7, 2021. OpenReview.ner net, 2021. (Cited on p. 13)
George Papamakarios, Thco Pavlakon, and Tain Murray. Masked autoregressive flow for density estimation. arXiv preprint xix: 070577 2017. (Cited 38)
George Papamakarios, Eric Nalisnick, Danilo Jimencz Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan, Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv:1912.08 1922.096,, 2019. (Cited p. 1,9)
Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends in optimization, 1(3):127-20)) 2014. (Cited on p. 2, 3)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjic Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bune Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. (Cited on p.
François-Pierro Paty and Marco Cuturi. Subspace robust wasserstein distances. In International Conference on Machine Learning, pp. 5072-5082 5081. PMLR, 2019. (Cited on p. 13)
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-loarn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 12-2825-2830, 2830, 2011. (Cited on p. 10, 27)
Gabriel Peyré. Entropic approximation of wasserstein gradient flows. SIAM Journal on Imaging Sciences, 8 (4):2323-28) 2351,, 2015. (Cited on p. 2, 4,
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundalions and Trendso in Machine Learning, 11(556):355-60 607, 2019. (Cited on p. 5)
17

---

Published in Transactions on Machine Learning Research (11/2022)
Rémi Peyre. Comparison between W2 distance and H- norm, and localization of wasserstein distance. ESATM: Control, Oplimisation and Calculus of Variations, 24(4):1489-150) 1501, 2018. (Cited on 4)
Simon Plazotta. A bdf2-approach for the non-linear fokker-planck equation. arXiv preprint 2018. (Cited on 13)
Julien Rabin, Gabriel Peyré, Julie Delon, and Marc Bernot. Wasserstein barycenter and its application to texture mixing. In International Conference on Scale Space and Variational Methods in Computer Vision pp. 435-446. Springer, 2011. (Cited on p. 2, 5)
Jack Richter-Powell, Jonathan Lorraine, and Brandon Amos. Input convex gradient networks. arXiv preprint arXiv:2111.12187 2021. (Cited on p. 2)
Hannes Risken. Fokker-planck equation. In The Fokker-Planck Equation, pp. 63 95. Springer, 1996. (Cited on p. 10)
Gareth Roberts and Richard Tweedie. Exponential convergence of langevin distributions and their discrete approximations. Bernoulli, pp. 341 363, 1996. (Cited on p. 1, 10, 27)
Litu Rout, Alexander Korotin, and Evgeny Burnaev. Generative modeling with optimal transport maps. arXiv preprint ar (Xt::21100029999 2021. (Cited p. 2)
Mark Rowland, Jiri Hron, Yunhao Tang, Krzysztof Choromanski, Tamas Sarlos, and Adrian Weller. Orthog- onal estimation of wasserstein distances. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 186-195. PMLR, 2019. (Cited on p. 13)
Adil Salim, Anna Korba, and Giulia Luise. The wasserstein proximal gradient algorithu. aix e-prints, pp. XiXv2-2002, 2020. (Cited on p. 2)
Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 55(58-633:94, 2015. (Cited on p. 6, 20, 21, 22)
Filippo Santambrogio. (Euclidean, metric, and Wasscrstein] gradient flows: an overview. Bulletin of Math- ematical Sciences, 7(1):87 154, 2017. (Cited on p. 1, 3, 6)
Saeed Saremi. On approximating ff with neural networks. arXiv preprint arXio:1910.127/4 2019. (Cited
on p. 2)
P Vatiwutipong and N Phewchean. Alternative way to derive the distribution of the multivariate ornstein uhlenbeck process. Advances in Difference Equations, 2019(1):1-7, 2019. (Cited p. 27)
Cédric Villani. Topics in optimal transportation, volume 58. American Mathematical Soc., 2003. (Cited on p. 4)
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008. (Cited on p. 3, 22)
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Ev- geni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Fric Jones, Robert Kern, Eric Larson, C J Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SeiPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261 2722, 2020. doi: 10.1038/s41592-00 019-0666-2. 2. (Cited on p. 29)
Yifei Wang, Peng Chen, and Wuchen Li. Projected wasserstein gradient descent for high-dimensional bayesian inference. aix preprint arXim:2102.0000 063.00000 2021. (Cited on p. 1)
18

---

Published in Transactions on Machine Learning Research (11/2022)
Yifei Wang, Peng Chen, Mert Pilanci, and Wuchen Li. Optimal neural network approximation of wasserstein gradient direction via convex optimization. arXio preprint or Xiv:2200. 13098, 2022. (Cited on 1)
Andre Wibisono. Sampling as optimization the space of measures: The langevin dynamics as a composite optimization problem. In Conference on Learning Theory, pp. 1193-3027. PMLR, 2018. (Cited p. 1,
4, 27)
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint nee:1708.0741, 2017. (Cited on p. 12, 37)
Chao Zhang, Zhijian Li, Hui Qian, and Xin Du. Dpvi: A dynamic-weight pettcle-bbsse variational inference framework. Xiv preprint or Xiv: orXiv:2112.00945 2021. (Cited on p. 2)
19

---

Published in Transactions on Machine Learning Research (11/2022)
A Some Results of (Candau-Tilh, 2020)
In this section, report some interesting thcoretical results of (Candan-Tilh, 2020). More preciscly, in Appendix A.1, report the geodesics in SW space. Then, in Appendix A.2, report the Euler-Lagrange equation of the Wasserstein gradient flow and the Sliced-Wassesstei) gradient flow of the Fokker-Plance functional. Finally, in Appendix A.3, we report some theoretical results that we will use in our proofs.
A.1 Geodesics Sliced-Wassersee Wasserstein Space
We recall here first some notions metric spaces. Let (X,d) be some metric space. In our case, we will have X P2( with bounded, open convex set and d = SW2.
We first need to define absolutely continuous curve.
Definition (Absolutely continuous curve). A curve w : [0, + X said be absolutely continuous if there ezists g E L'((0, 1]) such that
Vio < 4, d(w(lo), s { g(s)ds.
(27)
We denote by AC(X, d) the set of absolutely continuous measures. Moreover, denole ,C curves AC(X,d) d) starting x and ending at y. Then, we can define the length of an absolutely continuous curve E AA((X, d)
(X,d) d) the set of
La(w)= = sup d(wo(lx), w(x+1))), n n21, 0 - t0 <h ... < 10b-1.-66111.
(28)
Then, we say that space X is geodesic space if for any f, E X,
d(x, y) min (La(w), w AC(X, d) w()) x. w11 = y)
(29)
Finally, Candau- Tilh (2020) showed in Theorem 2.4 that (P2(S2), SW2) is not geodesic space but rather a pscudo-geodesic space since for n, E P2(2),
inf (w), (w), (P2()
ca. (a(,v)))
(30)
We see that the infimum of the length in the SW space is the Wasserstein distance. Hence, suggests that the geodesics in SW space are related the one in Wasserstein space, which are well known since they correspond to the Me Cann's interpolation (see e.g- (Santambrogio, 2015, Theorem .27))
A.2 Euler-Lagrange Equations in Wasserstein and Sliced- Wassersess Spaces
To prove that the Wasserstein gradient flow of a functional satisfies PDE, one step consists at computing the first variations of the functional in the JKO scheme, í.e. of x + dx u)/2r + F(r). Evaluating the first variation at the minimizer actually gives a corresponding Euler-Lagrange equation (Santambrogio, 2015; Candau-Till Till, 2020).
Hence, Candau- Tilh (2020) also provided a similar form of Lagranne equations for Wasserstein and gradient flows of the Mokker-Planck functional
F()) / Vdu + H(1)).
(31)
Proposition 3 (Proposition 3.9 in (Candau-Tiil 2020)).
20

---

Published in Transactions on Machine Learning Research (11/2022)
Let 1h+1 be the optimal measure in
W24 p4) F()). ()Pa(K)
arg min
(32)
2r
Then, it satisfies the following Euler- Lagrange equation:
log( log(pe+1)+ + constant a.e.,
(33)
where Piet1 the densily of 14t1 and vb the Kanlorovitch potential from 1441 14
Let 1641 be the optimal mensure in
(H,
arg min MEP2(K)
+ F(1).
(34)
2T
Then, it satisfies the following Euler- Lagrange equation:
log log(PE+1)+V
gu-1 vo
Pd dx(0) constant a.c.,
(35)
where for 0 E Sd-1, vh is the Kantorovitch potential form Po M to P M
From this proposition, we see that the Guler-Lagrane equations share similar forms, since only the first variation of the distance changes.
Informally, one method to find the corresponding PDE for Wasserstein gradient flows is to show that the derivative of the first variation of Wasserstein is equal (weakly, i.e. in the sense of distributions by integrating dp In this case, we recover well from (32) the PDE. Indeed, satisfies the PDE w.r.t. test functions) to Đr
Öp Ot div(pVV)+ +
(36)
in weak sense if for all & E C (o)
DE (t.2) +VV()) (z), t V)((t)))) - -E((,))) dp(())dt== - -5((5.0)) E(0, dpo(z).
(37)
By using the Euler- Lagrange equation (33), informally, we have P (log(e) + V + + Up+ + p 5.
Then, we integrate Vg w.r.t. to this distribution and we use that (by integration by parts)
/ (VE(E), vp(x))dx= - -
AE(x)) dp(x),
(38)
and we obtain
JrL (-DE(E) (VE(2), VV(2)) + (VE()). vo())) dpc(x) 0.
(39)
Hence, to obtain the right equation, require to show that in the limit 7 - 0, the term containing the first variation of Wasserstein is equal to the term representing the derivation in time, i.c.
1 /f (VE(s), (z))) // dE
lim 7-50
σt dp((z)dl.
(40)
This is done e.g. in (Bonnotte, 2013, Theorem 5.3.6) or (Liutkus et al., 2019, Theorem S6). For the Wasserseen distance, it still an open question whether we have the same connection or not.
Note that we recover the divergence operator by using an integration by parts (see e.g. (Korba et al., 2021, Appendix A.1))
(<VECe), (V()))d((a== - - ((e)div(p(+)WV(()) dr.
(41)
For complete derivation of the Wasserstein gradient flows of the Fokker-Plann functional, see e.g. (San- fambrogio, 2015, Section 4.5).
21

---

Published in Transactions on Machine Learning Research (11/2022)
A.3 Results on the SW-JKO Scheme
Finally, report here some results about the continuity and convexity of the Sliced- Wasserstein distance as well as on the existence of the minimizer at each step of the SW-JKO scheme. We will use these results in our proofs in Appendix B.
In the following, we restrain ourselves to measures supported on a compact domain K. Proposition 4 (Proposition 3.4 in (Candau-Tilh, 2020)). Let E P2(K). Then, l + SW2 (1, (H, is contin- uous w.r.t. the weak convergence.
Proposition 5 (Proposition 3.5 in (Candau-Tilh, 2020)). Let E P2(K), then μ SW) is conver and stricly convex whenever v is absolutedy comtinuous w.r.t. the Lebesgue mensure.
Proposition 6 (Proposition 3.7 in (Candau-Tilh,, 2020)). Let and 1% PP((K). Then, there exists unique solution p E P2(K) to the minimization problem
SW2(1, 1%)
/
min (EPa(K)
Vdu Vdu++ H((u)
(42)
2T
The solution is even absobulely continuous.
B Proofs
B.1 Proof of Proposition
We refer to the proposition 3.7 in (Candau-Tiie Tilh, 2020).
Let 0, k E N, M P2().. Let's note (()) (W)())))) 2r
+ F(u).
According to Proposition 3.4 in (Candau-Tiil 2020), / SW2(H, Mb) is continuous with respect to the weak convergence. Indeed, let μ E P2(K) and let (un)x converging weakly to n, iee Un - μ. Then, by the reverse triangular inequality, we have
[S22(m,, (W2(,,D) SW2(Un; (1) S W2(pn,
(43)
Since the Wasserstein distance metrizes the weak convergence (Villani, 2008), we have that W2(mn ub) - And therefore, μ + SW2 HQ) and + SW2 (T) are continuous w.r.t. the weak convergence.
By hypothesis, F is lower semi continuous, hence μ + ((4) is lower semi contimuus. Morcover, P2(K) is compact for the weak convergence, thus we can apply the reierstrass theorem (Box 1.1 in (Santambrogio, 2015)) and there exists a minimizer 1h+1 of J.
By Proposition 3.5 in (Candan-Tilh, 2020), μ SW2(u, is convex and strictly convex whenever is absolutely contimuous w.r.t. the Lebesgue measure. Hence, for the uniqueness, if F is strictly convex then J(J) is also strictly convex and the minimizer is unique. And if p% is absolutely continuous, then according to Proposition 3.5 in (Candau-Tilh, 2020), + SW, (%) is strictly convex, and hence p+ (p) also strictly convex since FF was taken convex by hypothesis.
B.2 Proof of Proposition 2
Let N, then since 1h+1 is the minimizer of equation 17,
F(1T+1) SW2(MT MR)
SW3 (MT, M%)
{F(HT)
F(uT).
(44)
2T
2T
Hence, as SW2 (1T+1:14)) 2 0,
F(rTt)) F((())
(45)
22

---

Published in Transactions on Machine Learning Research (11/2022)
B.3 Upper Bound on the Errors
Following Hwang el al. (2021), we can also derive upper bound on the error made at each step.
Proposition 7. Let l E IN, Po E P2(RR)) some constant, and assume that F admits a lower bound, then
SW2((90 k+1,7) (( ((+1) SW2 SW2 ) #P2, MK) + C
(46)
$ (k: + 1)CT + (Po,
Proof. The bound can be found by applying Theorem 3.1 in (Hwang al., 2021) with x = (P2(RA, SW2). and then by applying a straightforward induction. We report here the proof for the sake of completeness.
Let E N, then since 1b41 is the minimizer of equation 17,
F(U 3 F((())
SW3 F(uE).
(47)
2r
2T
Therefore, using that F is non increasing along (mf)s (Proposition 2), we have F(uI) S F(wo). Moreover, using that F admits an infimum, we find
SW2 (T+1) s (T(())
inf inl F(p))- 2T((F(m)) F(uR+1)) S
(48)
1
Let A = F(Ho) - in F(u). By the same reasoning, we have that
(W2 k+1.T #pz, ( S 2TA
(49)
Now, using the triangular inequality, have that
SW2((1) (+1.7 #P2, 1 (Wg 1 (1t+tt))P 4P2, (p2)) SW, (1 SW2
(50)
=2V2TA + SW ((9) K.T 1#P2,16) 146)
Let C = 2222A, then by induction we find
SW2 ( (0 (p22, s (k + 1)C+= +
(51)
B.4 Sliced-Wasseree Wasserstein results
Link for ID supported measures. Let u,v E P(Rd) supported on a line. For simplicity, we suppose that the measures are supported on an axis, i.e. p(x) (1(z)) II do(x;) and u(x) = (x) (x,))) In this case, we have that
S/'
W3
| |Fph (x) -Fp? (x)) dr.
(52)
On the other hand, let E gd-1 then we have
Vy R, F Fpg (V)
1 (x) ((d))
JR
1 (0000)((0,22) p(dx)
Rd
1 (x:10)) in (dxi)
(53)
1 # (x1) #r (dz1)
y
Fps P
0
23

---

Published in Transactions on Machine Learning Research (11/2022)
Therefore, Frg (2) = 01 =FFFpd (2) and
(2)
( 14, Pf = .. 18 pet 1a (2) - -01F pe (2) (2) da
0f /
(54)
|F p (F (z) F Fp. (2)1
_ dz
0g (H,
Finally, using that sst-1 00 dx)) L we can conclude that
SW = 1 ga.. W) = (H,
Closed-form between Gaussians. It is well known that there is a closed-form for the Wasserstein distance between Gaussians (Givens & Shortt, 1984). If we take a = =N Σ) and 3 = -N(m, A) with m. J E Rd and Σ, A E Rdxd two symmetric positive definite matrices, then
W2 = ||m --1== T )
(55)
Let NN and 3 (m,, s2mm, 13) two isotropic Gaussians. Here, have
(a, =I1 m 1 Tr(o² 8 2(0s3
|| m 13 + (a - 8)² Tr(Ta)
(56)
|| - m 11 + d(a - =)²-
On the other hand, Nadjahi et al. (2021) showed (Equation 73) that
(a, || mm + (a s)² W2(a,
(57)
d
In that case, the dilation of factor d between WGF and SWGF clearly appears. For more complicated gaussians, we may not have this equality. For example, let a = N(u, D), 8 = N/m, A) with D and A diagonal. Then, Pt a NN1(,, 0), E 0² Dz); P N((m, 0), E 0, A,) and
(1 _OPD:
W #
P$ D)
R.D:
Σ OZ
(58)
V
with a, 33r the centered measures (noting T x + M, then à a)). a). Hence, we have
(W, |/u m
SW3 (ā, 3)
2
|11 - ml
Σ 82 D2D;
Σ A/4 E
dx(0)
d
S4-1
1141 mummm ml3
d
d
DDD++
+ Σ
0:A: -2 2
02 D.A,
dA(o)
d
84-11
1=1
iJ
|| 11mmm| ml|
(,dx(0) - /
Dj
(2)(()) + Σ A
O.ODD.),
d
(=1
5d
1=1
Su
Su 1
11
ml
2
(D. + - - g
Σ DA,
(59)
24

---

Published in Transactions on Machine Learning Research (11/2022)
using that Sge-1 00" dx L and by applying Proposition in (Nadjahi et al., 2021) to decompose SW2 (a, B) 1 (W) B). On the other hand, we have
W2 (a,
|| m - 2(DA
=1Iæ- m
TrD )
= || - ml
(60)
i=1
d Σ (D. A/P.. 1=1
= -
m
Since (W B) (a, 3), have Ei=1 VD,A; d s V Ei, D,
Let d = 2, 0,8 0 and D = diag(o? //) A = diag( x²
2 2²). In this case, on the one hand, we have
Σ
D = V2os.
(61)
On the other hand,
1
2 g
D.A, V20s 5
109 02)2 + 0400 dA(0)
2
-2os {
V (09 + 02) 0902 dx(0)
(62)
2
/ 51
V20s
(07 +0)+ dA(0)
V2as,
using that absolutely continuous with respect to the Lebesgue measure and hence X(10; = 0 0}) = and the fact that for every 0 E S {01 * 0,02 # 0), / + 02)² + 10003 > ( + 03) + From this strict inequality, deduce that W2 and d SW2 not always equal, even in this restricted case. Hence, even in this simple case, cannot directly conclude that have dilation term of d between Wasserstein and Sliced- Wasscrstein gradient flows.
C Computation of the SW-JKO scheme in practice
C.1 Approximation of SW
For each inner optimization problem
SW2 (1, (R
1 E arg min
+ F(H),
(63)
2T
we need approximate the sliced- Wasscrstein distance. To do that, we used Monte-Carlo approximate by sampling no directions (e following the uniform distribution the hypersphere Sd-1 (which can be
done by using the stochastic representation, f.c. let Z ~ (N(0, Ta), then f // Unif(Sd-1) (Fang et al., 1992)). Let v E P(Rd), we approximate the sliced- Wasserstein distance as
SW-(u,)) =
W2 v).
(64)
25

---

Published in Transactions on Machine Learning Research (11/2022)
Algorithm 2 SW-JKO with Discrete Grid
Input: No the initial distribution with density Po: K the number of SW-JKO steps, т the step size, F the functional, Ne the number of epochs to solve each SW-JKO step, (zz) the grid
N
Let p!0) = (5 Po(ag) Ee Po(ZE)
j=1
for k= == 1 to K do Initialize the weights pl++1) (with for example a copy of (k)) N (1+1) 10 Jny and μ = Σラ= Pj (k) // Denote p === P) for i to No do Compute 1((++1)= 1 (u
5n,
F(1 Backpropagate through with respect p(k+1) Perform gradient step
Project the simplex pl++2) using the algorithm of Condat (2016)
end for end for
In practice, we compute it for empirical distributions ün, and Dm, and approximate the one dimensional Wasserstein distance
(2 PS PfDm) S pgan (2) F (2) du
(65)
by the rectangle method.
Overall, the complexity is in (n log + m log m)) where (resp. m) denotes the number of particles of An (resp. pm).
C.2 Algorithms to solve the SW-JKO scheme
We provide here the algorithms used to solve the SW-JKO scheme (17) for the discrete grid (Section 3.3) and for the particles (Section 3.3).
Discrete grid. We recall that in that case, model the distributions pI =
= ri=l Pi
(k) 82. where use
N samples located at (m:) and (o. N belongs to the simplex Σ,. Hence, the SW-JKO scheme at step
k + rewrites
SW2 pida, + F( p(
min (pt)eEN
(66)
2T
We report in Algorithm 2 the whole procedure.
Particle schome. this case, we model the distributions as empirical distributions and we try to optimize the positions of the particles. Hence, we have 11 = - Din 8 (aa and the problem (17) becomes
SW3(1 1 EN Sa,, M4)
min (z.)
+
F N
(67)
2T
In this case, we provide the procedure in Algorithm 3.
Additional experiments
D.1 Dynamic of Wrreedd-ssersteinn gradient flows
The Fokker-Plante equation (8) is the Wasserstein gradient flow of the functional (6). Morcover, it is well- known have counterpart stochastic differential equation (SDF) (see e.g. Mackey (1992, Chapter 11)) of
26

---

Published in Transactions on Machine Learning Research (11/2022)
Algorithm 3 SW-JKO with Particles
Input: Ho the initial distribution, K the number of SW-JKO steps, the step size, F the functional, N the number of epochs to solve each SW-JKO step, N the batch size
Sample (1²) ) ~ Mo ii.d
for == to K do
Initialize N particles (e(++))) (with for example a copy of (z) ))=1)
// Denote 1 N Zj=1 ó (2ett and p = Φ Σ==1 8 Lag0 for i= 1 to Ne do Compute J((++1)= = (5 F(((+t) Backpropagate through with respect (a) (of(t)))g) /=1 Perform gradient step end for end for
the form
dx, -V(,)dd /28 dW,
(68)
with (W,), a Wiener process. This SDE is actually the well-known Langevin equation. Hence, by ap- proximating it using the Maruraaa scheme, we recover the Unadjusted Langevin Algorithm (ULA) (Roberts Twoodic, 1996; Wibisono, 2018).
For
V(x) = ( mm" A(r-m). m),
(69)
with A symmetric and definite positive, obtain an Drnstcin-Ullebber process (Le Gall, 2016, Chapter 8). If we choose Ni as a Gaussian N N(mo, Eo), then we know the Wasserstein gradient flow i in closed form (Wibisono, 2018; Vatiwutipong & Phewchean, 2019), for all t > 0, μ = Nmm,, 2,) with
m + e m) ΣS = - tdpT + +A : - e (A
(70)
Comparison of the evolution of the diffusion between SWGFs and WGFs. For this experiment, model the density using RealNVPs (Dinh et al., 2016). More precisely, we use RealNVPs with 5 alfine coupling layers, using FCNN for the scaling and shifting networks with 100 hidden units and 5 layers. In both experiments, we always start the schome with (4 = (0, 7) and take ng = 1000 projections to approximate distance. We randomly generate a target Gaussian (using "make_ _id___spiiiss_____eeeeeeeeee matrix" from scikit-learn (Pedregosa et al., 2011) to generate a random covariance with 42 as seed).
the
We look at the evolution of the distributions learned between t 0 and We compare it with the true Wasserstein gradient flow. On Figure 6a, observe that they do not seem to match. However, they do converge to the same stationary value. On Figure 6b, we plot the functional along the true WGF dilated of a factor d = 2. We see here that the two curves are matching and we observed the same behaviour in higher dimonsion. Even though we cannot conclude on the PDE followed by SWGFs, this reinforces the conjecture that the SWGF obtained with a step size of 3 (i.e. using the scheme (18)) is very close to the WGF obtained with a step size of 7. We also report here the evolution of the mean (Fig. 7) and of the variance (Fig. 8). For the mean, follows as expected the same diffusson. For the variance, it is less clear but it is hard to conclude since there are potentially optimization errors.
with a time step of 0.1.
Comparison between JKOICCN and SW-JKO. Following the experiment conducted by Mokrov et al. (2021) in section 4.2, we plot in Figure 9 the symmetric Kullback-Leibler (SymKL) divergence over dimensions between approximated distributions and the true WGF at times t = 0.5 and = 0.9. We take the mean over 15 random gaussians (gencrated using the scikit-learn function (Pedregosa et al., 2011) matrix for the covariance matrices, and generating the means with a standard normal distri- bution) for dimensions d [2....,12)
"make "make_spd _pt___atiii""""""""""a______""""
27

---

Published in Transactions on Machine Learning Research (11/2022)
-1.50
Flue)
-150
Fluzd
F(AA)
F(()))
-1.75
Flu")
-1.75
Flye)
2200
-2.00
-2.5
-.25
0
1
4
0
1
2
3
t
t
(a) Without dilation
(b) With dilation
Figure 6: Evolution of the functional equation 6 along the WGF ie and the learned SWGF A.. We observe dilation of parameter 2 between the WGF and the SWGF.
0.05
0.4
0.00
0.2
m2(2t)
-0.05
m2(2t)
m1(t)
mz(t)
0..0
mi
mi
0.0
0
1
2
4
1
2 t
3
4
Figure 7: Evolution of the mean taking into account the dilation parameter. μ denotes the true mean of WGF, â the mean obtained through SW-JKO (17) with r = 0.05 and μ the mean of the stationary measure. We observe that the mean of approximated mcasure obtained through SW-JKO scems to follow the one of the WGF.
For each target Gaussian, we run the SW-JKO dilated scheme (18) with 7 = 0.05 for a RealNVP normalizing flow. We compare it with JKO-ICNN with also 5 = 0.05 and with Euler-Maruyama with 103, 10a and 10 particles and a step size of 10-3. For JKO-ICNN, we use, as Mokrov al. (2021). DenseICNN with convex quadratic layers introduced in (Korotin et. al., 2019) and available at attps://github.. com/ For the JKO-ICNN scheme, we
implementation.
We compule the symmetric Kullback-Leibler divergence between the ground truth of WGF n² and the distribution fi approximated by the different schemes at times = 0.5 and t = 0.9. The symmetric Kullback- Leibler divergence obtained as
SymKL ( (L(x"IAA) + KL(A)le".
(71)
To approximate it, we generate 10s samples of each distribution and evaluate the density at those samples.
If we note go a normalizing flows, the distribution in the latent space and p (ga) ( pzz, then we can evaluate the log density of p by using the change of variable formula. Let x = go(z), then
log(p(x)) log(pz(=)) log det. (=)l-
(72)
28

---

Published in Transactions on Machine Learning Research (11/2022)
1.0
1200 to
(202 ka
0.8
0.7
-0.6
00.8
1.0
0.0
0.5
1.0
1.
2,0
2.3
3.0
3.5
4.0
0.0
0.5
1.0
13
2.0
2.5
3.0
3.5
4.0
0.0
6-
0.2
(2" bp
0.4
-0.8
0.0
0.5 1.0
1.5
2.0
2.5
3.0
3.5
4,0
1.0
4,0
Figure 8: Evolution of the components of the covariance matrix taking into account the dilation parameter. Σ denotes the true covariance matrix WGF, Σ the covariance matrix obtained through SW-JKO (17) with 0.05 and E the covariance matrix of the stationary distribution. We observe some difference between WGF and SWGF.
We choose RealNVPs (Dinh et al., 2016) for the simplicity of the transformations and the fact that we can compute efficiently the determinant of the Jacobian (since have closed-form). A RealNVP flow is a composition of transformations T of the form
Va E Rd, x == = exx(((2))) t(=²))
(73)
where we write z = (2 and with s and some neural networks. To modify all the components, we use also swap transformations (i.e. (z,,²²) 22) + (=2 2))) This transformation is invertible with log det Jr(2) E, s22)).
For JKO-ICNN, we choose strictly convex ICNNs, and can hence invert them as well as compute the density, In this case, we do not have access to a closed-fon form for the Jacobian. Therefore, we used backpropagation to compute it. As this experiment is low dimension, the computational cost is not too heavy. However, there exist stochastic methods to approximate it in greater dimension. We refer to (Huang et al., 2020) and (Alvarez-Meli al., 2021) for more explanations.
We approximate the functional by using Monte-Cait approximation as in Section 3.3.
For Fuler-Maruyana, as in (Mokrov et al., 2021), we use kernel density estimation in order to approximate the density. We use the scipy implementation (Virtanen et al., 2020) "gaussian kde" with the Scott's rule to choose the bandwidth.
Finally, we report on the Figure the mean of the log of the symmetric Kullback-Leibbe divergence over Gaussians in each dimension and the 95% confidence interval.
For the training of the neural networks, of 10-4 for RealNVP (except for the 1st iteration where we take learning rate of 5. 10-3) and of 5. 10-3 for JKO-ICNN. At each inner optimization step, we start from a deep copy of the last neural network, and optimize RealNVP for 200 epochs and ICNNs for 500 epochs, with a batch size of 1024.
Adam optimizer (Kingma & Ba, 2014) with learning rate
We see on Figure 9 that the results are better than the particle schemes obtainod with Eulor-Maruyama (EM) with step size of 10-3 3 and with either 103, 10a or 10 particles in dimension higher than 2. However, JKO-ICNN obtained better results.
29

---

Published in Transactions on Machine Learning Research (11/2022)
1-0.5
0.3
0.2
3.2
0.6
-1.3
0.4
-2.0
0.2
30 12
10
10
12
10
ReaINVP-SWGF
Figure 9: On the left: SymKL divergence at time 0.5 and 0.9 between the groundtruth of the Pokker-Planck equation at time and the solution of the SW Gradient Flow at time t. On the right: Absolute error between the functionals evaluated for WGF and SWGF at time t.
Stationary distribution estimated at 8
2
1yg1o
1
0
-1
EM 1K
EM 50K ReaINVP-SWGF
-2
EM 10K
50
60
70
100
d
Figure 10: Symmetric KL divergence between the learned distribution at time t and the true stationary solution on Gaussians
D.2 Convergence to stationary distribution
Here, want to demonstrate that, through the SW-JKO scheme, are able to find good minima of functionals using simple generative models.
Gaussian. For this experiment, we place ourselves in the same setting of Section 4.1. We start from No I) and use step size of r = 0.1 for 80 iterations in order to match the stationary distribution. In this case, the functional is
F(w) = v(z)da(x) H(p)
(74)
A(= with V(x) b) - b), and the stationary distribution is (z(x) e ev((), hence p" A(b,
We generate 15 Gaussians for d between 2 and 12, and d E [20,30) 30, 40, 50, 75, 100). Due to the length of the diffusion, and to numerical unstabilities, we do not report results obtained with JKO-ICNN, In Figure 2, we showed the results in low dimension (for E , 12}) and the unstability JKO-ICNN. We report on Figure 10 the SymKT. also higher dimension.
We 200 epochs of each inner optimization and an Adam optimizer with learning rate of 5- 10-3 3 for the first iteration and 10-3 for the rest. We also use batch size of 1000 sample.
Bayesian logistic regression. For the Bayesian logistic regression, we have access covariates Rd with their associated labels Vi.....9% Yn {-1, 1}. Following (Liu & Wang, 2016; Mokrov et al., 2021), put as prior the regression weights w, (wwaa) NWw 0, 2) with po(a:) = T(a; 1,0.1).
Sy $1....,8,
30

---

Published in Transactions on Machine Learning Research (11/2022)
Table 3: Number of features, of samples and batch size of each dataset.
covtype
german
diabetis
twonorm
ringnorm
banana
splice
waveform
image
features
54
20
20
20
60
21
18
samples
581012
1000
768
7400
7400
5300
2991
5000
2086
batch size
512
800
614
1024
1024
1024
512
1024
Table 4: Hyperparameters for SWGFs with RealNVPs. nl: number of coupling layers in RealNVP, nh: number of hidden units of conditioner neural networks, learning rate using Adam, JKO steps: number of SW-JKO steps, Iters by step: number of epochs for each SW-JKO step, т: the time step, batch size: number of samples taken approxinate the functional.
covtype
german
diabetis
twonorm
ringnorm
banana
splice
waveform
image
nl nh 512
2
2
2
5
2
512
512
512
512
512
128
128
512
Ir
2c
le-4
5e
1e
5e
le-4
5e
1e
5e-5
JKO steps
5
5
10
20
5
5
5
5
5
Iters by step
1000
500
500
500
1000
500
500
500
500
0.1
10-6
5-10-6
10
10-6
0.1
10-6
10-8
0.1
batch size
1024
1024
1024
1024
1024
1024
1024
512
1024
Therefore, we aim learning the posterior p(w, oly))
p(w, aly) x plyle, a)po(v))a)pp(o)) = (wl)) II p(u))w,))
where p(y: /w, a.) = or (wTss) s;) resample data uniformly.
1+u
(1 (1-o((l oww s{:)) " with a the sigmoid. To evaluate V(H) = V(x) dp(x), we
In our context, let V(x) know that the limit. of the stationary solution of Fokker-Planck is proportional to e p(w, ocly).
log (1)la)) a)), then using F(w) V H(p) functional,
Following Mokrov et al. (2021); Liu & Wang (2016), use the 8 datascts of Mika et al. (1999) and the html).
covertype dataset hhttps//www.csie. ntu. edu www.tt/ww.pppe http://////bbetttto//aaaaaaasse/
We report in Table 3 the characteristics of the different datasets. The datasets are loaded using the code of Mokrov et al. (2021) htt.:///o///ggtttao..oooeoottotr
We split the dataset between train and test set with 4:1 ratio.
We report in Table 4 the hyperparameters used for the results reported in Table 1. We also tuned the time step since for too big approximation of the SWGF only for small enough T.
observed bad results, which the SW-JKO scheme should be good
Morcover, we reported in Table the mean over 5 training. For the results obtained with JKO-ICNN, we used the same hyperparameters as Mokrov al. (2021).
D.3 Influence of the number of projections
It is well known that the approximation of Sliced-Wias Wasscrstein is subject to the curse of dimensionaly through the Monte-Carlo approximation (Nadjahi et al., 2020). We provide here some experiment to quantify this influence. However, first note that the goal is not to minimize the Sliced- Wasserstein distance, but rather the functional, SW playing mostly a regularizer role. Experiments on the influence of the number of experiments approximate the SW have already been conducted (see e.g. Figure in (Nadjahi et al., 2020) or Figure in (Deshpande et al., 2019)).
31

---

Published in Transactions on Machine Learning Research (11/2022)
100
0.04
100
30025
100
00000
075
00005
30005
000
500
00015
0.55
0.25
0.25
0.010
0001
3
-1.00
0.0000-1
0.0000
00
1.00
1.00
0.005
0.004
00025
00033 850
0.0025 025
00000
000
00015
000
800
-055
0.00550
0001
-075
0.000
0005-00.3
00000
00000-3
00000
Figure 11: Density over time of the solution of the aggregation equation learned over the discre grid.
t=1
38
050
25
050
030
025
20
025
100
025
15
000
15
000
075
0.00
025
-0.75
00
08
1100
0:0
-1.00
00
I-3
I-5
I-10
075
025
075
850
20
058
20
050
20
025
15
025
15
025
15
008
16
50
-0.0
-0
-0.5
-035
110
00
0.5
05
1000
0.0
00
-3.00
-0.
10 00
Figure 12: Density over time of the solution of the aggregation equation by learning particles.
Here, we take the same setting of Section 4.1, i.e. we generate 15 random Gaussians, and then vary the number of ioojections and report the Symmetric Kullback-Leibler divergence on Figure 3. We observe that the results seem improve with the number of projections until reach certain plateau. The plateau seems be attained for bigger number of dimension in high dimension.
D.4 Aggregation equations
Here, we use as functional
W(A) = /f W(x v)dp(r)dp()))
(75)
Carrillo et al. (2021) use repulsive-attractivv interaction potential, for b> 0,
W(x) | ||>| a
(76)
b
using the convention |a In[[]=]]] For some values of and b, there is existence of stable equilibrium 0
state (Balagué et al., 2013).
Dirac Ring. First, for the Dirac ring example, we take and b 2. Then, W a repulsive-attractive interaction potential (repulsive in the short range, and attractive in the long range because b (Balagué
32

---

Published in Transactions on Machine Learning Research (11/2022)
100
100
100
079
035
25
20
15
025
15
025
002
008
000
0.55
0.2
1225
50
05
05
0.
-03
1.00
1.00
25
1.00
26
058
025
15
025
025
15
000
008
000
0.25
25
-025
05
05
-3000
0.0
Figure 13: Density over time of the solution of the aggregation equation approximated with a FCNN.
100
100
100
075
175
150
15
26
000
0.5
025
84
0.5
05
1.00
-1.00
00
-1.00
-05
-10
05
-1.0
60
1.00
1.00
1.00
140
120
100
200
000
080
30
000
0.5
03
0.25
100
1.00
-1.00
1.00
-05
0.5
-1.0
Figure 14: Density over time of the solution of the aggregation equation approximated with JKO-ICNN.
et al., 2013)), we show the donsitics over time obtained on a discrete grid (Figure 11), by learning the position of the particles (Figure 12) and with the FCNN (Figure 13). For the last two, the density reported is obtained with kernel density estimation where we chose by hand the bandwidth match well the sampled points. We also report the evolution of the density for the JKO-ICNN scheme. In Figure 14, we show the evolution of the density obtained by the change of variable formula. We observe that values seem explode which may due to the fact that the stationary solution may not have density to numerical unstabilities. We report in Figure 15 the densities obtained with a kernel density estimation. We also report the particles generated through a FCNN, the particle evolution and JKO-ICNN in Figure 16. We observe that the particles match perfectly the ring. For the FCNN, there seem to be some noise. JKO-ICNN recover also well the ring but particles seem to not be uniformly distributed over the ring.
For the SW-JKO scheme, take 0.05 and run it for 200 steps (Irom to 10) starting from No I). For JKO-ICNN, choose 0.1 and run it for 100 steps as the diffusion really long.
We take grid of 50 x 50 samples on [-1, 1]2 To optimize the weights, we used an SGD optimizer with momentum of 0.9 with a learning rate 10-4 for 300 epochs by inner opiimization scheme. For particles, we optimized 1000 particles (sampled initially from no) with an SGD optimizer with a momentum of 0.9, a lcarning rate of 1 and 500 epochs by JKO step. We take FCNN composed of hidden layers with units and leaky relu activation functions. They are optimized with an Adam optimizer with a learning rate of 10-1 (except for the first iteration where take learning rate of 5. 10-3) for 400 epochs by JKO step. For
33

---

Published in Transactions on Machine Learning Research (11/2022)
100
100
100
079
075
25
15
025
06
025
08
002
000
010
0.25
0.25
04
1225
50
05
02
0.
-0.
1.00
1.00
30
1.00
058
025
025
20
025
000
10
008
000
05
-025
10
05
-3.00
0.0
Figure 15: Density over time of the solution of the aggregation equation approximated with JKO-ICNN and kernel density estimator.
1.0
100
1.0
030
025
02
025
4325
-025
-325
-0.00
-0.55
-1.00
-075 -050 odo ažs aão
-100
-225
-101
ožs 050 075 (c) Samples from JKO-ICNN
-075
-050
(a) Samples from the FONN
(b) Samples learned by optimizing over particles
Figure 16: Samples of the stationary distribution
JKO-ICNN, we choose the same parameters as for the previous experiments. In any case, we take no = 10* projections to approximate SW2 and a batch size of 1000.
On Figure 17, we plot on the two first columns the stationary density learned by the different methods. On the third column, plot the evolution of the functional along the flows.
The functionals seem to converge towards the same value. However, JKO-ICNN seems to not be able to capture the right distribution. We also note that the curve for the FCNN is not very smooth, which can probably be explained by the fact that we take independent samples at each step.
On Figure 18, we show the evolution of the functionals along different learned flows. We observe that for the discretized grid, we obtain the worse results which understandable since we have discretization crror. The particle the most stable as particle's position do not move anymore once the stationary state is reached. For the FCNN, observe oscillations which due to the fact that take independent samples at each time t. Finally, the JKO-ICNN scheme seems to converge toward the same value as the SW-JKO scheme with FCNNs.
We also tried to use normalizing flows with this functional. We observed that the training seems harder than with the FCNN. Indeed, with simple flows such as RealNVP, the model has lot of troubles of learning the Dirac ring, probably because of the hole. More generally, since normalizing flows are bijective transformations, they must preserve topological properties, and therefore do not perform well when the
34

---

Published in Transactions on Machine Learning Research (11/2022)
14
680
-00
099
-10
no 00
(a) Steady state on the discretized grid
(b) Steady state on the discretized grid
(c) Evolution functional
55-
35
-0
(d) Steady state for the fully connected neural
(c) Sicady state for the fully connected neural network
(f) Evolution of the functional
network
201-
(g) Steady state for particles
(h) Steady state for par- ticles
(i) Evolution of the functional
4.00
18
on du an
(j) Steady state for JKO-ICNN with KDE
(k)) Steady state for JKO- ICNN with KDE
()) Evolution of the functional
Figure 17: Steady state and evolution of the functional of the aggregation equation for a 4,
standard distribution and the target distribution do not share the same topology (e.g. do not have the same number of connected components or "holes" as it is explained in (Cornish el al., 2020)). For CPF, it worked slightly better, but was not able to fully recover the ring (at least at time t = 5) as we can see on Figure 19. Morover, the training time for CPF was rcally huge compared to the FCNN.
Other functionals. As in section of Carrillo et al. (2021), we also tried to use
W (x) = ||||| 2 log([<=]))
(77)
as interaction potential with FCNN. We find well that the steady state is an indicator function on the centered disk of radius (Figure 20).
35

---

Published in Transactions on Machine Learning Research (11/2022)
Grid
-0.0822
Grid
-0.05
Particles
Particles
0..822
JKO-ICNN
JKO-ICNN
-0.66
0.0826
FCNN
FCNN
0.0828
-0.07
0.0830
Mmw Mn wwwwmm
0..833
0.08
-0.0834
2
4
6
10
5
8
9
10
Figure 18: Evolution of the aggregation functional along different flows.
0.75
0.75
30
0.00
20
0.00
0.5
-0.00
-0500
-075
05
-075
-1.00
-0.75 -0500
025
100
-0.75
-0.50 -025
050
(a) Density learned by SWGF with CPF at time
(b) Density learned by JKO-ICNN at time
t
Figure 19: Density learned for the aggregation equation for CPF and JKO-ICNN.
Another possible functional without using internal energies is to add drift term
f V(x))(z)dd..
(78)
Then, the Wasserstein gradient flow is solution to
aep: = div(pV(W p)) + div(pVV).
(79)
log([|=|]) and V(z) = -3 log([|=[]) with 1 and B 4. Then, it can be shown (sce Carrillo et al. (2021): Chen Kolokolnikov (2014); Carrillo al. (2015)) that the steady state is an indicator function on a torus of inner radius R and outer radius R. o + which we observe on Figure 21 and Figure 22.
Carrillo et al. (2021) use W(x) |a
For this last experiment, we observed some unstabily issues in the training phase. It may be due to non- locality the interaction potential W as it is stated in (Carrillo et al., 2021).
D.5 Sliced- Wasseestessr flows
In this experiment, we aim at minimizing the following functional:
F(u) = sw, + XH(())
(80)
2
36

---

Published in Transactions on Machine Learning Research (11/2022)
16
64
04
130
(a) Density steady state
(b) Density of steady
(c) Evolution the functional over time
state
Figure 20: Steady state and evolution of the functional for W
|all
log([1<])).
2
t=0
t=0.5
t=1
25
0.4
0.35
10
10
10
0.00
20
03
025
15
0.2
0.2
10
015
0.5
-0.
-0.
-10
05
-1.0
-15
-15
0.00
-15
-10
-05
0.
05
10
15
-15
-10
-05
00
05
10
15
-15
-1.0
-05
0.0
05
10
t=2
3=3
t=4
15
15
15
0.35
0.30
0.30
05
05
05
025
020
00
00
00
0.15
015
0.10
0.10
010
-10
0.05
-10
0.05
0.05
-15
0.00
-15
0.00
-15
0.00
-15
-05
0.0
100
15
-15
-0.
0.0
0.5
-15
-10
-05
00
05
Figure Density over time of the solution of the aggregation-drift equation approximated with FCNN.
where y is some target distribution from which we have access to samples. In this section, we use MNIST (LeCun & Cortes, 2010), FashionMNIST (Xiao et al., 2017) and CelebA (Liu et al., 2015).
We report the Fréchet Inception Distance (FID) (Heusel al., 2017) for MNIST and FashionMINIST between 10th test samples and 10 generated samples. As they are gray images, we duplicate the gray levels into 3 channels. Moreover, we the code of Dai & Seljak (2021) (available at https: / /htbttb....//.g com/biweidai/ SINF) and reported their result for SWF (in the ambient space) and SWAE in Table 2. For SWF in the latent space, we used our own implementation.
With a pretrained autoencoder. First, we optimize the functional in the latent space of pretrained autoencoder (AE). We choose the same AE Liutkus et al. (2019) which available at https: //gs thub. com/ com/aliutkus/ste /wtp://///////// wttbbb/lttot///////////aaaaaaaet wwwaatter/oo///ttorrr/tooooo htwpe//wwworrkestuuueenneeeeeee /tta//attooooer.../ .y.. We report results for latent space of dimension d 48.
In this latent space, we applied FCNN and ReaINVP. In either case, we used no = 10 projections to approximale 5W,. We chose a batch size of 128 samples. The FCNN was chosen with 5 hidden layers of 512 units with leaky relu activation function. The RealNVP was composed of coupling layers, with FCNN as scaling and shifting networks (with layers and 100 hidden units). We trained the networks at cach inner optimization step with learning rate of 5 10-3 for RealNVP (and 1 for the first iteration) and of 10 for the FCNN (and 5 10-3 for the first itration) during 1000 epochs.
37

---

Published in Transactions on Machine Learning Research (11/2022)
-1
45 C
00 19
(a) Density of the steady state
(b) Density the steady state
(c) Evolution of the functional
over time
Figure 22: Steady state and evolution of the functional for the aggrogation-drift equation.
0
3
θ
2
7
8
8
G
6
5
S
6
3
O
9
3
6
8
6
6
5
6
FID=19.3
(b) FID=41.7
Figure 23: Generated sample obtained through a pretrained decoder + FCNN.
Following Liutkus et al. (2019), we choose 0.5 and 0. We run it for 10 outer ilerations.
We report in Figure 23 the result obtained with FCNN on MNIST and FNShionMNIST. Overall, the results seem quite comparable and our method seems to perform well compared to other methods applicd in latent spaces.
On Figure 24, we report samples obtained with RealNVP on MNIST, FashionMNIST and CelebA. For CelebA, we choose the same autoccoder with a latent space of dimension 48 and we ran the SW-JKO scheme for 20 steps with 0.1.
We can also optimize directly particles in the latent space and show it on CelebA on Figure 25 for 0.1 and for 10 steps.
In the Original Space. We report here the results obtained by running the SW-KKO scheme in the original spaces of images, which are very high dimensional (d 784 for MNIST and PashionMNISTT.. We obtained worse results than in the latent space. Notice that we used only 1000 projections, and ran it for 50 outer iterations with step size of r 5
On Figure 26, we use a RealNVP and add a uniform dequantization (Ho et al., 2019) and learned in the logit space as it is done in (Dinh et al., 2016; Papamakarios et al., 2017) because using normalizing flows need continuous data. The RealNVP composed here of 2 coupling layers with FCNN also composed of 2 layers and 512 hidden units. We choose 5, and run it for 20 steps. At each inner optimization problem, the neural networks optimized with Adam optimizer with learning rate of 10-2 and for 500 epochs.
38

---

Published in Transactions on Machine Learning Research (11/2022)
9
3
7
0
6
9
7
6
4
2
7
2
3
O
/
/
9
6
S
8
5
(a)
17.=17.8
(b)
FID=40.6
(c) FID=90.8
Figure 24: Generated sample obtained through a pretrained decoder
+ RealNVP.
Figure 25: Particle through SW-JKO with 7 = 0.1 for 10 steps.
We also report results obtained using convolutional neural network (CNN) in Figure 27. The idea here is that we can capture inductive bias, as it is well known that CNNs are efficient for inage-related tasks. We obtained a slightly better FID on MNIST, but worse results on FaShionMNIST. In term of quality of image, the generated samples do not seem better.
For the CNN, we choose a latent space of dimension 100, and we first apply a lincar layer into a size of 128 7 7. Then we apply 3 convolutions layers of (kernel_sizee stride, padding) being respectively (4, 1), (4,2, 2, 1), (3, 1, 1). All layers followed by leaky ReLU activation, and sigmoid is applied the output.
39

---

Published in Transactions on Machine Learning Research (11/2022)
i
4
7
5
88.1
(b) FID=95.3
Figure 26: Generated sample obtained in the original space with RealNVP.
5
7
6
0
Z
7
-
4
?
4
1
D
3
(a) FID=69
(b) PID=102
Figure 27: Generated sample obtained in the original space with CNN.
Nevertheless, samples obtained in the space of image look better than those obtained through the particle scheme induced in (Liutkus et al., 2019).
40


Reviews:
[{"review_id": "ZbP87iLrlP", "reviewer": "Reviewer_zKX6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strength\nI think that the text describes quite well and in an intuitive manner the current state-of-the-art regarding Wasserstein gradient flow. Furthermore, although I am not an expert in this area and do not know the literature well, the method indeed seems to not have been studied in the literature, with the exception of [LS2019]. The method and applications presented in [LS2019] are very different from what is presented in the manuscript.\n\n# Weaknesses:\n\nthe manuscript describes three possible ways of parametrizing the probability distributions:\n\n* **Grid:** for some fixed samples $x_1, \\ldots, x_N$ consider $\\mu = \\sum \\rho_i \\delta_{x_i}$ for some positive weights $\\rho_i$ that sum-up to one (and optimise these probability weights).\n* **Particles:** consider $\\mu = (1/N) \\sum \\delta_{x_i}$ for some free particle locations $x_1, \\ldots, x_N$\n* **Generative-models:** consider $\\mu = g_{\\theta} \\sharp p_Z$ for some neural-net $g_{\\theta}$ and some (Gaussian) base probability measure $p_Z$. If it is necessary to be able to evaluate the density of $\\mu$ (for example to evaluate its entropy), one can use a normalizing flow.\n\nWith these parametrizations, the minimization of $\\mu \\mapsto \\mathcal{F}(\\mu)$ is quite straightforward. Something that I have not found clear at all in the text is the motivation for the introduction of the \"regularisation\" term $\\frac{1}{2 \\tau}\\text{dist}^2(\\mu, \\mu_k)$. Indeed, after discretization, one can try to directly minimize $\\mu \\mapsto \\mathcal{F}(\\mu)$ with a standard gradient-based optimizer. Indeed, the usual argument is that the regularisation term $\\frac{1}{2 \\tau}\\text{dist}^2(\\mu, \\mu_k)$ helps \"take the geometry of the problem into account\" and it is definitely true in some cases (eg. natural gradient, etc..) that this can greatly improve upon vanilla gradient descent. And I think I was expecting a small discussion, with some appropriate simulations. As a matter of fact, I have tried to reproduce some of the experiments in the paper: one can minimize with batch-gradient-descent the function $\\mathcal{F}(\\mu) = \\iint W(x-y) \\mu(dx) \\otimes \\mu(dy)$ described in the paper with the \"particles\"-parametrization proposed in the paper with 10 lines of JAX. This works very well and I could make things converge with $N=10^3$ particles in less than a second on a laptop (the manuscript reports running experiments for up to 5 hours). And I did obtain a result very similar to what is reported in the paper.\n\n\n# References\n[LS2019] Liutkus A, Simsekli U, Majewski S, Durmus A, Stöter FR. Sliced-Wasserstein flows: Nonparametric generative modeling via optimal transport and diffusions. InInternational Conference on Machine Learning 2019 May 24 (pp. 4104-4113). PMLR.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Consider minimising a function $\\mathcal{F}(\\mu)$ on the space $\\mu \\in P_2$ of probability measures with finite second moment. One can use a proximal-gradient type method and set $$\\mu_{k+1} = \\text{argmin}_{\\mu \\in P_2} \\quad J(\\mu) + \\frac{1}{2 \\tau}\\text{dist}^2(\\mu, \\mu_k)$$ for some appropriate notion of distance $\\textrm{dist}(\\mu, \\nu)$ between probability measures. Using KL-type divergences lead to natural-gradient descent. One can use the Wasserstein distance, but this can be computationally difficult (although there is quite a bit of work to make this practical). The manuscript proposes using the sliced-Wasserstein distance: since the sliced-Wasserstein distance is relatively straightforward to approximate with Monte-Carlo samples, this leads to a practical method. The manuscript focusses on two common situations, $$\\mathcal{F}(\\mu) = \\int V(x) \\mu(dx) + \\textrm{Entropy}(\\mu)$$ and $$\\mathcal{F}(\\mu) = \\iint W(x-y) \\mu(dx) \\otimes \\mu(dy)$$ although the proposed methodology is quite general.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "# critical\nI am not finding the method well motivated. Can the authors demonstrate, with appropriate simulations to back up their claims, that running a standard gradient-based optimizer (eg. ADAM) is much worse than what is proposed? The discussion could for example comment on the final accuracy, and the computational complexity.\n* when comparing the different schemes, why do you make these choices for the number of time-steps $\\tau$, the number of SW-JKO steps, etc.. For example, when comparing the time taken for different methods, how to you choose when to stop the different methods: wouldn't it make more sense to choose less examples and display the accuracy-versus-compute-time curves?\n* Can the author compare to slightly more complicated but tractable situations? For example, mixture of Gaussians in not-so-small dimensions? Also, why choose ULA as competitor when there are many other better methods? Why choose these tuning parameters when using ULA?\n\n# Minor:\n* p4,\"For a diffusion of length $k$, it requires $O(k^2)$ evaluations of gradients\": what diffusion? \n* most of page 6 are conjectures: it would make sense to make it much more explicit.\n* \" for randomly generated positive semi-definite matrices\": what randomness? Would it be more interesting to study some ill-conditioned situations instead?\n* in the Bayesian logistic regression example, what is the \"accuracy\" metric?\n* in Section 4.2, I do not think it is true that the *centred* Dirac ring of radius 0.5 is the only solution: *any* Dirac ring of radius 0.5 is solution.\n* is the example (ie. Aggregation Equation) in Section 4.2 a good example? Are the authors claiming that the proposed scheme converges faster to a solution? Or to a more accurate solution? (if so we would need some metric instead of nice pictures). As mentioned above, 10 lines of JAX and batch-gradient-descent give similar pictures orders of magnitude faster (and, it seems, as accurately).\n\n# minor technical question:\nThe proof of the Proposition 1 and 2 are very simple since most of the hard parts are contained in [Candau-Tilh, 2020]. I am wondering whether one can apply the (reverse) triangular to the **squared** slice-Wasserstein distance $\\text{SW}_2^2$. Should we apply this to the slice-Wasserstein distance $\\text{SW}_2$ instead?\n\n# Conclusion:\nI think that the authors should:\n* motivate much carefully their proposed method. Eg. does it lead to faster convergence when compared to simple baselines? more accurate solutions? etc..\n* describe more carefully how the tuning parameters are chosen in the numerical experiments (and how the competing methods are tuned)\n* choose (tractable) examples where proper metrics (ie. more reliable than FID score or pretty-looking pictures) are available. Also, as acknowledged by the authors, the results presented in Section 4.3 are not really competitive: it may be best to choose better examples where the method shines. As described in the introduction, there are **many** situations in applied mathematics that boil down to minimizing a functional over the space of probability distribution: can the authors choose a few well-chosen examples and convincingly show that the proposed method is useful: it is not the case at the moment.\n\n\n", "broader_impact_concerns": "no concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0zbiISM2nk", "reviewer": "Reviewer_nXjA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1) The method works better/faster than other related methods in some experimental scenarios (JKO-ICNN method in the bayesian logistic regression (table 1), SWF method in generative modeling (Figure 5, Table 2)).\n2) The method is more flexible than JKO-ICNN since SW enjoys a closed form differentiable approximation.\n3) The writing of the paper is clear\n4) Technically the paper seems to be correct with no major flaws\n\nWeaknesses\n1) Broader impact/significance seems to be limited (see the Broader Impact section);\n2) In general, the experimental evidence is insufficient to support some claims;\n3) Some important comparisons to related works are missing;\n4) The (straightforward) direct optimization baseline seems to be missing;\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces the concept of sliced Wasserstein gradient flows and employs them to minimize the functionals on the space of the probability measures. The minimization of functionals is done according to the sliced wasserstein JKO stepping scheme (SW-JKO) which the authors propose.\n\nThe SW-JKO scheme is a natural modification of the original JKO scheme (W distance is simply replaced by SW distance in the JKO stepping). JKO is a procedure (introduced by Jordan-Kinderlehrer-Otto) to compute the (conventional) Wasserstein-2 gradient flows of functionals of probability measures. The latter scheme can be viewed as a time-discretized gradient descent in the spaces of probability measures. While the scheme works in theory, its practical implementation is challenging as each JKO step is itself a non-trivial optimization over probability measures involving the Wasserstein-2 distance term which is hard to estimate. There exist several works establishing numerical precedures for to handle this issue (the authors discuss them in Section 2.3).\n\nIn this paper, the authors propose a numerical procedure based either on using a generative model, particle-based approximation or grid discretization to implement SW-JKO stepping in practice. According to the provided computational results, SW-JKO converges to the minimum of the functionals (sometimes) faster (or better) than the alternative methods which compute (conventional) wasserstein gradient flows such as the JKO-ICNN methods (appearing in several related papers), etc.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Experiments\n\nThe current work mostly considers the problem of optimizing functionals over probability distributions. The authors claim that their methods performs (sometimes) faster or better than existing alternatives to minimize the functionals. However, the experimental evidence of this is limited. Thus, I suppose the following experiments should be conducted in the revision:\n\nIn the setting with **neural networks/generative models** approximation. I would expect here both quantitative and qualitative comparisons, e.g., it would be nice to see some non-trivial multi-modal stationary distributions here (related works do consider such setups).\n1) Comparison with (Fan et. al, 2022) [ICML 2022] paper. In the current paper, I see comparisons only with JKO-ICNN (Mokrov et. al., 2021), but (Fan et. al., 2022) claim to outperform JKO-ICNN by a noticeable margin, see their Figure 5.4. Besides, I wonder why the authors do not compare with JKO-ICNN (and Fan et al.) in recovering non-Gaussian the stationary distributions. For examples, why does Figure 9 contain JKO-ICNN but figure 8 - does not (explanation in Fig. 2b is not very convincing)?\n2) Comparison with the direct optimization baselines. Namely, one may pick a generative model (or a normalizing flow) and directly optimize the functional with it (optimize its parameters with SGD).\n\nComparisons in the **particle-based** setting. In Appendices, the authors formulate their particle-based SW-JKO approach. Thus, it seems reasonable to request to compare it with the direct particle-based optimization of some functionals. This is neede to better understand how does the proposed scheme works compared to the straightfoward solution.\n\nSame applies to the **grid-based** optimization SW-JKO presented in appendices.\n\nIn the above metnioned experiments, could the authors please report both final metrics & computational times?\n\nMissing references\n\nJean-David Benamou, Guillaume Carlier, Quentin Mérigot, and Edouard Oudet. Discretization of functionals involving the Monge–Ampère operator. Numerische mathematik, 134(3):611–636, 2016.", "broader_impact_concerns": "My main concern about this paper is that the concept introduced in the paper – SW gradient flows (and the related SW JKO) – seems to be unnatural and presumably will have limited impact on the machine learning community/practitioners. Let me explain my concern below.\n\nFirst, when we talk about the Wasserstein gradient flows, there are two (related) tasks in view:\n1) Computing the gradient flow dynamics (trajectories);\n2) Computing the “end” of the gradient flow (a minimum of the functional).\n\nThe gradient flow (1) itself seems to be an important tool to analyse the **dynamics** of stochastic processes appearing in a wide range of sciences and their practical applications (especially those fields where the Fokker-Plank equation appears). To the best of my understanding, the prior related works (Fan et al., 2021; Alvarez et al., 2021; Mokrov et. al. 2021) propose algorithms which are capable of recovering the entire dynamics of the flow. As far as I understand, the current work is unable to do so as it recovers Sliced Wasserstein dynamics, which differs from the Wasserrstein gradient flow (the relation is discussen in many places in the paper). Thus, the applicability of the method proposed here to studying dynamics appearing in real-world problems, e.g., those characterized by FPE, seems questionable.\n\nThe current paper focuses only on aspect (2), namely **minimization of the functionals** (computing the end of the flow). This immediately raises the question: why do we need to optimize the functional with SW-JKO (or any other JKO scheme). Why not to just directly pick a generative model (invertible in the case when the functional contains entropy) and directly optimize its parameters to minimize the functional? Or not to pick a set of particles and directly optimize them with the gradient descent? Why is SW-JKO better? Unfortunately, the current paper does not provide a deep discussion/evaluation related to this important question. This makes not very clear whether SW-JKO indeed provides any improvement in optimizing functionals.\n\nTo conclude, in my view, to show the potential superiority of SW-JKO to optimize functionals, it is essential to conduct more detailed experimental evaluation and comparisons, see the requested changes section. Unfortunately, otherwise SW-JKO seems to be just a nice mathematical concept but without potential applications in practice/extensions\n\nNevertheless, since the acceptance criteria for TMLR is technical correctness and clarity of presentation (which are ok in this paper) rather than significance or impact (which are, in my opinion, questionable), I recommend to push this paper forward to the next revision.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "67Wob740Rg", "reviewer": "Reviewer_xUbb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- nice introduction to the topic and proposed approach\n- insight about the sliced Wasserstein flow compared with exact one\n- broad numerical experiments and attention to approximation problems regarding SW\n\nWeakness: \n- the evidence and motivation for using SW was not strong\n- no discussion on the existence of the gradient flow with SW, in the limit as JKO step-size converges to zero (what is the pde that describes this flow?)     ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is concerned with gradient flows on the space of probability distributions, where sliced Wasserstein distance is used as a metric. To define and implement the gradient flow, the JKO procedure is employed where exact Wasserstein distance is replaced with its sliced version. Although there is no explicit relationship between the two gradient flows, the paper provide some insight into this by studying certain examples, supported by numerical evidence. In order to numerically implement the flow, the paper proposes to represent the probability distribution at each step as either, a discrete distribution on fixed set of points, or push-forward of a generator map $ g_\\theta \\sharp p_z$ where $p_z$ is a Gaussian distribution in the latent space. At each JKO step, the discrete distribution or the parameters of the generator are updated to minimize the JKO objective. The proposed algorithm is studied and compared with the related approaches with the aid of several numerical examples involving synthetic and real data. The functionals used in the paper are: relative entropy with respect to a target distribution, or the interaction energy. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Comments: \n\nMy main concern is about the motivation for using SW and its advantages. According to my understating, the main bottleneck in implementing these type of gradient flows is computation of the functional in terms of particles, not approximating the Wasserstein distance. Using ICNN $phi$, the W2 distance between p and $\\nabla \\phi \\sharp p$ is simply $\\int |\\nabla \\phi(x)-x|^2 dp$. In case a map $T(x)$ is used instead of $\\nabla \\phi$, one can use the $\\int |T(x)-x|^2 dp$ which for optimal $T$ becomes the W2 distance. However, computing the functional, specially entropy is not straightforward. The existing approaches either used the approximation suggested in the paper in (24), or its variational form which involves extra maximization step. Eq (24) involves $\\log (\\det (\\nabla g_\\theta(z)))$ which is expensive to compute and need to be approximated. \n\nPutting the application of SW aside, another main difference of the proposed approach is using a single generator map from the latent space, instead of constructing sequence of maps as in the existing works. This is the reason for reduction from $K^2$ to $K$ compared to previous approaches and might be the reason for better performance in numerical experiments. \n\nIn summary, I understand the computational value of SW compared to W2 in general when two distributions are compared, but I do not see its value in the gradient flow context.  \n\nQuestions/Changes: \n- Is it possible to implement your proposed approach, but simply use $\\int |g_\\theta^k(z) - g_\\theta^{k+1}(z)|^2 p_z dz$ as regularizer instead of SW? This might be a good proxy for W2. \n\n- Is it possible to derive the limiting PDE? I see you derive the stationary conditions for JKO step in the appendix. \n\n- Are there any reasons for experiments reported in figure 2 to run till $d=12$? \n\n- I do not understand the emphasis on SW being differentiable. Isn't W2 differentiable too? \n\n- first line in background section: \"bet\" -> \"be\"\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has now been reviewed by three reviewers: two lean towards reject (zKX6 and nXjA) and one leans towards acceptance (xUbb). All the reviewers agree that the method is theoretically grounded and it appears experimentally to perform better than JKO-ICCN. However, when one is interested in finding a minimiser of a functional defined on the space of probability measures (as in the experiments of the paper), there exist alternatives which are much simpler as emphasized by the reviewers and the proposed method does not always outperform them (as demonstrated by the experiments of the authors in the rebuttal). Those alternatives are not gradient flows in the space of probability measures but for most ML applications this is somewhat irrelevant.

I believe the material is worth publishing as it is a nice addition to the computational gradient flows literature and is well written. However the authors should follow closely the recommendations of zKX6 and update the "contribution" section of the paper by
- only stating that the proposed method is somehow more competitive that JKO-ICNN
- add some discussion regarding the competitiveness of the method when compared to the vanilla not-regularised approach.
- if something is about the dynamics of the flow, illustrate it more convincingly.
Decision: Accept (poster)