Do Neural Optimal Transport Solvers Work? A Continuous Wasserstein-2 Benchmark
Authors: Alexander Korotin
Lingxiao Li
Aude Genevay
Justin Solomon
Alexander Filippov
Evgeny Burnaev
Venue: NeurIPS 2021 Poster (2021)
Keywords: optimal transport solvers
benchmark
continuous measures
quadratic cost
wasserstein-2 distance
parametric methods
input-convex neural networks
generative modeling

Abstract: Despite the recent popularity of neural network-based solvers for optimal transport (OT), there is no standard quantitative way to evaluate their performance. In this paper, we address this issue for quadratic-cost transport---specifically, computation of the Wasserstein-2 distance, a commonly-used formulation of optimal transport in machine learning. To overcome the challenge of computing ground truth transport maps between continuous measures needed to assess these solvers, we use input-convex neural networks (ICNN) to construct pairs of measures whose ground truth OT maps can be obtained analytically. This strategy yields pairs of continuous benchmark measures in high-dimensional spaces such as spaces of images. We thoroughly evaluate existing optimal transport solvers using these benchmark measures. Even though these solvers perform well in downstream tasks, many do not faithfully recover optimal transport maps. To investigate the cause of this discrepancy, we further test the solvers in a setting of image generation. Our study reveals crucial limitations of existing solvers and shows that increased OT accuracy does not necessarily correlate to better results downstream.

Do Neural Optimal Transport Solvers Work? A Continuous Wasserstein-2 Benchmark
Alexander Korotin
Lingxiao Massachusetts Institute of Technology Cambridge, Massachusetts, USA
Skolkovo Institute of Science and Technology Artificial Intelligence Research Institute
Moscow, Russia
lingxiao@mit. edu
a. korotin@skoltech
Aude Genevay
Justin Solomon
Massachusetts Institute of Technology Cambridge, Massachusetts, USA aude gwwevay@gmai..ccooooooom
Massachusetts Institute of Technology Cambridge, Massachusetts, USA
jsolomon@mit... edu
Alexander Filippov
Evgeny Burnaev
Huawei Noah's Ark Lab Moscow, Russia
Skolkovo Institute of Science and Technology Artificial Intelligence Research Institute
hallipov..aeexaree
Moscow; Russia
e burnaev@@koltech..r
Abstract
Despite the recent popularity of neural network-based solvers for optimal transport (OT), there is no standard quantitative way to evaluate their performance. In this paper, we address this issue for quadratic-cost -pansport--secifiillly, computation of the Wasserstein-2 distance, commonly-ure machine learning. overcome the challenge of computing ground truth transport maps between continuous measures needed to assess these solvers, convex neural networks (ICNN) to construct pairs measures whose ground truth OT maps be obtained analytically. This strategy yields pairs of continuous benchmark measures in high-dimensional spaces such as spaces of images. We thoroughly evaluate existing optimal transport solvers using these benchmark measures. Even though these solvers perform well in downstream tasks, many do not faithfully recover optimal transport maps. To investigate the cause of this discrepancy, we further test the solvers in setting of image generation. Our study reveals crucial limitations existing solvers and shows that increased OT accuracy does not necessarily correlate to better results downstream.
formulation of optimal transport in
input-
Solving optimal transport (OT) with continuous methods has become widespread in machine learning, including methods for large-scale OT [6] and the popular Wasserstein Generative Adversarial Network (W-GAN) 3 12]. Rather than discretizing the problem [31], continuous OT algorithms use
neural networks or kernel expansions to estimate transport maps or dual solutions. This helps scale OT problems not handled by discrete methods. Notable successes
to large-scale and
of continuous OT are in generative modeling 42 20
7 and domain adaptation 43 37 25).
In these applications, OT is typically incorporated part of the loss terms for a neural network model. For example, in W-GANs, the OT cost is used loss function for the generator; the model incorporates neural network-based OT solver estimate the loss. Although recent W-GANs provide ofate-of-the-art generative performance, however, it remains unclear to which extent this
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

success is connected to OT. For example, 22 32 38 show that popular solvers for the Wasserstein- 1 (W distance in GANs fail to estimate W1 accurately. While W-GANs were initially introduced with W in Rl state-of-the art solvers now use both W and W2 (the Wasserstein-2 distance, i.e.,
OT with the quadratic cost). While their experimental performance on GANs similar, W solvers
tend to converge faster (see [9 Table 4) with better theoretical guarantees [19 26 16].
Contributions. In this paper, we develop generie methodology for evaluating continuous quadratic- cost OT solvers (W2). Our main contributions are as follows:
We use input-convex neural networks (ICNNs 2 construct pairs of continuous measures that
we use as a benchmark with analytically-kna -nnowwlll-oowwnnnno solutions for quadratic-cost OT [$3 84.1).
We use these benchmark measures to evaluate popular quadratic-cost OT solvers in high- dimensional spaces (#443), including the image space of 64 64 CelebA faces ($4.4).
We evaluate the performance of these OT solvers as a loss generative modeling images ($4.5)
Our experiments show that some OT solvers exhibit moderate error even in small dimensions (34.3), performing similarly to trivial baselines 422. The most successful solvers are those using parametrization via ICNNs. Surprisingly, however, solvers that faithfully recover W2 maps across dimensions struggle achieve state-of-the-art performance in generative modeling.
Our benchmark measures can be used to evaluate future W2 solvers in high-dimensional spaces, erucial step to improve the transparency and replicability of continuous OT research. Note the benchmark from [5] does not fulfill this purpose, since it is designed to test discrete OT methods and uses discrete low-dimensional measures with limited support.
Notation. We use P2(RDD to denote the set of Borel probability measures on TR. with finite second moment and (2. to denote its subset of absolutely continuous probability measures. We denote by [IP,, Q) the set of the set of probability measures on R R2 with marginals IP and Q For some measurable map T RRD IR,,, we denote by T:' the associated push-forward operator. For é : RD I. we denote by ő its Legendre-Fenched Fenchel transform [TO] defined by Φ(v) max (x, y) <xx]]. Recall that ø is convex function, even when not.
1 Background on Optimal Transport
We start by stating the definition and some properties of optimal transport with quadratic cost. We refer the reader to 34 Chapter for formal statements and proofs.
Primal formulation. For P, Q P2) Monge's primal formulation of the squared Wasserstein-2 distance, i.e., OT with quadratic cost, is given by
T(x)1 d)(()),
def min TP=Q
where the minimum is taken over measurable functions (transport maps) T RR. - Rd mapping P to Q. The optimal T" is called the optimal transport map (OT map). Note that and this formulation does not allow for mass splitting, i.e., for some P, Q (2 T that satisfies TiP Q. Thus, Kantorovich proposed the following relaxation 14
① is not symmetric,
there is map
1 y/ dr dr(x,)).
W2 (P.
del
min rElI(P.0) JRP
where the minimum is taken over all transport plans i.e., measures on R. R whose marginals
are IP and Q The optimal n e [II Q) is called the optimal transport plan (OT plan). n* is of the form [idgo, T* J e 11(P,, Q) for some T then 1 is the minimizer of
Dual formulation. For IP, Q E P2 (RR the dual formulation of WV is given by 40
[[ [(x))P(x)) + slw)d((1)
W P. !
max 5Bg 11-11
(3)
JRD
JRD
where the maximum is taken over all f CP RD - RR and y L² RD - BR) satisfying
f(x) + s(y) 1 x y/2 for all x, E RDD From the optimal dual potential f*, we can recover the
optimal transport plan (x)
(xx 34 Theorem 1.17].

---

The optimal f* g* satisfy (f*)== g" and (g*) = f*, where w² :R - is the c-transform of u defined by v (y) me [1/2| yl u(x)
We can rewrite (3) as
(F max
f())dP(x) + f(v)dQ())
JR.
where the maximum is taken over all f (P, [P, R). Since f* and g are each other's e- e-transforms, they are both e-concave [3 11., which is equivalent to saying that functions
*/* 1 2|1=| (x) and 6* : ticular, 2 6* and 6² $ Since
2 a g* (z) are convex [34 Proposition 1.21]. In par-
(*) = x- (x* ( |x|2
(x)) - VV"
(5)
we see that the OT maps are gradients of convex functions, a fact known as Brenier's theorem 6d.
"Solving" optimal transport problems. In applications, for given P, Q P2(RP), the W2 optimal
transport problem is typically considered in the following three similar but not equivalent tasks:
Evaluating (P, Q) The Wasserstein-2 distance is geometrically meaningful way to compare probability measures, providing a metric on P2(R.))
Computing the optimal map 7 or plan n* The map T provides intuitive way to interpolate between measures. It is often used as a generative map between measures in problems like domain adaptation [36 43)
and image style transfer [16].
Using the gradient (2)(Fa. to update generative models. Derivatives of W? are used implicitly in generative modeling that incorporates W2 loss [19 33 in which case IP = Pa is parametric measure and Q is the data measure. Typically, I. G is the measure generated from fixed latent measure $ by parameterized function G., e.g., neural network. The goal is to find parameters a that minimize W a. Q) via gradient descent.
In the generative model setting, by definition of the pushforward P.. GatS, we have
(P.. Q) J ((Ga(=))dB(s))
( ()())((())
JRD
where f* and g* are the optimal dual potentials. At each generator training step, f* and 9* are fixed so that when we take the gradient with respect to a, by applying the chain rule we have:
OW2(P... da
J.
(6)
where Ja(a())) is the transpose of the Jacobian matrix of Ga w.r.t. parameters a. This result still holds without assuming the potentials are fixed by the envelope theorem 229 To capture the
gradient, we need a good estimate V = idgo o T by (5). This task somewhat different from computing the OT map T since the estimate of V V is only involved in the gradient update for the generator, it is allowed to differ while still resulting
good generative model.
We will use the generic phrase OT solver to refer to method for solving any of the tasks above.
Quantitative evaluation of OT solvers. For discrete OT methods, benchmark dataset [35 exists but the mechanism for producing the dataset does not extend to continuous OT. Existing continuous solvers typically evaluated without evaluating its actual OT performance. Two kinds of metrics are often used:
set of self- self-generated examples or tested in generative models
Direct metrics compare the computed transport map I with the true one T c.g., by using C Unexplained Variance Percentage (C2-UVP) metric [16] 85.1]. [17 $5]. There relatively few direct metrics available, since the number of examples of IP, Q with known ground truth t is small: it is known that 7 can be analytically derived or explicitly computed in the discrete case 31 $3]. 1-dimensional case 31 $2.6], and dauussian/lcationn scatter cases [
Indirect metrics use an OT solver as a component in larger pipeline, using end-to-end performance as proxy for solver quality. For example, in generative modeling where OT is used as the generator loss 19 27 the quality of the generator can be assessed through metrics for GANs, such as the Fréchet Inception distance (FID) [3]. Indirect metrics do not provide clear understanding about the quality of the solver itself, since they depend components of the model that are not related to OT.
3

---

2 Continuous Dual Solvers for Quadratic Cost Transport
While benchmark might be used to test any continuous solver which computes map T** gradient V in this paper, we perform evaluation only on dual-form continuous solvers based on (3) (4) Such solvers have straightforward optimization procedures and can be adapted to various datasets without extensive hyperparameter search. In contrast, primal-for formal-formmm solvers based on ( e.g.,
18 43 22 23], typically parameterize 7 using complicated generative modeling techniques that depend on careful hyperparameter search and complex optimization procedures [24]
We summarize existing continuous dual form solvers in Table 1 These fit parametric function fe (or (h) to approximate f* (or 6* = idgo -f*) The resulting fa produces an approximate OT map id fe Veb T and derivative fa =ide- Vuy needed to update generative models (6)
Related works
Parameterization of potentials
Quanfitatively tested as QT
Solver
Tested in GANs
Regularized [LS]
| 3d
33
fe. :R." R- NNs
Gaussian case
Ent. regulaaeed
WGAN 33
Maximin [MM]
: RR-NN Hw :R RD. NN
X
Three-playe WGAN
Maximin (Batch-wise)
[MM-B]
f0 :R 00 - B--N
x
(g,p-WGAN
27
Quadratic Cost [QC]
Jo R R NN
WGAN-QC
Maximin ICNN
RP R ICNN
Gaussian
[MMv1]
Maximin + ICNNs
Vo :R R -
[MM+2]
:RD RD. VICNN
Gaussian case 16
x
We R ICNN
Non-Maximin [W2]
16
R R² VICNN
Gaussian case M
Table 1: Comprehensive table of existing continuous dual solvers for OT with the quadratic cost.
To our knowledge, none these solvers has been quantitatively evaluated in a non-Gaussian setting. For [MM], [MM-B]. and JQC), the quality of the recovered derivatives f for @*2(F..Q)eaa has only been evaluated implicitly through GAN metrics. Moreover, these three solvers have not been quantitatively evaluated solving OT tasks. We now overview each solver from Table [
[IS]] optimizes an unconstrained regularized dual form of (3) 36):
f(x)dP(x)+ JRD
JRD s(v) R(f.g).
[
max f.g
The entropic quadratic regularizer R. penalizes potentials f.g for violating the constraint
f < -1.11 1 [36] 33]. practice, f = fo and g Iw are linear combinations kernel
functions [1I neural networks (36). The parameters 0, w obtained by applying stochastic
gradient ascent (SGA) over random mini-batches sampled from P, Q
Most other solvers are based on expansion of
(w)
W (P, Q max
(x))P(()) +
min DER
|
y1 -(x) dQ(y).
(8)
J2D
JBD
2
The challenge of 8 is the inner minimization over x E DR. i.e., evaluating fV()). The main difference between existing solvers the procedure used to solve this inner problem.
[MM-B] uses neural network fa as the potential trained using mini-batch SGA [27]. To solve the inner problem, the authors restrict the minimization of the current mini-batch from IP instead of RD. The strategy is fast but leads to overestimation of the inner problem's solution since the minimum is taken over restricted subset.
[MM-v]] exploits the property that f* = } 12 b* where 3 is convex [39]. The authors parametrize fe tlll 1 the where th is an input convex neural network (ICNN) (2). Hence, for every y IR" the inner problem of (8) becomes convex in x. This problem can be solved using SGA to high precision, but doing so computationally costly 16 SC.4].
[MM] uses a formulation equivalent to
8
s(e((v)) )(((()))0200) (Q())
w2 (P, Q
max
f(x)dP(x)
min
2 (() - vll o12=-((((v)) H
(9)
RRD

---

where the minimization is performed over functions H RD RD. The authors use neural networks fo and H. to parametrize the potential and the minimizer of the inner problem. To train 0,w, the authors apply stochastic gradient ascent/descent (SGAD) over mini-batches from P. Q. [MM] is generic and can be modified to compute arbitrary transport costs and derivatives, not just W3. although the authors have tested only on the Wasserstein-1 (W distance.
Similarly to [MMv1], [Mvv2] parametrizes fo 1 1 the where the is an ICNN [26]. For fixed fo. the optimal solution H is given by H = (Vve) which is an inverse gradient of a convex function, so it is also gradient of a convex function. Hence, the authors parametrize H = V
where ow is an ICNN, and use [MM] to fit w.
[W2] uses the same ICNN parametrization as [26 but introduces cycle-consistengg regularization to avoid solving maximin problem 16, 84].
Finally, we highlight the solver [QC] 19]. Similarly to [MM-B], a neural network fo is used as the potential. When each pair of mini-batches {rn) J, [y..) from IP, Q is sampled, the authors solve
discrete OT problem to obtain dual variables {s#): (. which are used to regress fo(x,) onto f
Gradient deviation. The solvers above optimize for potentials like fo (or ehe), but it is the gradient of fo (or v that used recover the OT map via T = x fe. Even if [f 1 f (|2(0r) is small, the difference optimization process. We call this issue gradient deviation. This issue is only addressed formally for ICNN-based solvers [MMv1]. [MMv2], [W2] [16] Theorem 4. 1]. [26 Theorem 3.6].
V fa Vf
([22(p) Vf='l(22(p)
may be arbitrarily large since Ve is not directly involved in
Reversed solvers. [MM], [MM2]] W2] recover not only the forward OT map Vely = V = T", but also the inverse, given by H., R (T*) - (Vv*)=
V see 126 83] or [16 64.1]. These solvers are asymmetric P. Q and an alternative is to swap and Q during training. We denote such reversed solvers by [MM:R]. [MMv2:R], ] [.2:R]. In $ we show that surprisingly [MM:R] works better in generative modeling than [MM].
3 Benchmarking OT Solvers
In this section, we develop a generic method to produce benchmark pairs, i.e., measures (I,, Q) such that Q = T:P with sample access and an analytically known OT solution T between them.
Key idea. Our method is based on the fact that for a differentiable convex function / : IR R its gradient V is an optimal transport map between any P P2 ac (RD)) and its pushforward V by Ve R D D. This follows from Brenier's theorem [6]. 41 Theorem 2.12]. Thus, for continuous measure IP with sample access and a known convex v. (IP, VetPP) can be used as a benchmark pair. We sample from V by drawing samples from P and pushing forward by Vyb.
Arbitrary pairs (F, Q). It is difficult to compute the exact continuous OT solution for an arbitrary pair (IP, Q) As compromise, we compute approximate transport map the gradient of an ICNN using [2]] That is, we find the parameterized as an ICNN such that V W. a Q. Then, the modified pair (T, VvlotPP can be used to benchmark OT solvers. We choose [W2] because it exhibits good performance in higher dimensions, but other solvers can also be used so long as the is convex. Because of the choice of [W2], subsequent evaluation might slightly favor ICNN- based methods.
Extensions. Convex functions can be modified to produce more benchmark pairs. If di,..... V'N are convex, then or $1, v/x) is convex when R - R is convex and monotone. For example, e- 1h (c > 0), Σn th. max tha are convex, and their gradients produce new benchmark pairs.
Inversion. If V is bijective, then the inverse transport map for (IP VrlgtP) exists and is given by (Vee)= For each y IR." the value - (v)) can be obtained by solving a convex problem [39 86], [16] 83). All ICNNs tig we use have bijective gradients Velg, as detailed in Appendix B.
4 Benchmark Details and Results
We implement our benchmark in PoTorc and provide the pre-trained transport maps for all the benchmark pairs. The code publicly available at
https://github: http:///ww...ttoooaaaaaaaattteaa
The experiments are conducted on 4 GTX 1080ti GPUs and require about 100 hours of computation (per GPU). We provide implementation details in Appendix B
5

---

4.1 Datasets
High-dimensional measures. We develop benchmark pairs to test whether the OT solvers can redis- tribute mass among modes of measures. For this purpose, we Gaussian mixtures in dimensions In each dimension D, we consider a random mixture IP of Gaussians and two random mixtures Q1 Q2 of 10 Gaussians. We train approximate transport maps V Q (i= 1, using the |W2] solver. Each potential is an ICNN with DenseICNN architecture [6] SB.2]. We create a benchmark pair via the half-sum of computed potentials (P, 2/ (Vy 1/2 + V1/2) The first measure IP is a mixture of 3 Gaussians and the second is obtained by averaging potentials, which transforms it to approximate mixtures of 10 Gaussians. See Appendix A. 1 and Figure I for details.
D = 2'.2?,.... 29.
Output 4y
Fitted map Way
Input P
(ay Vaytep
Output 4z
Fitted map Vyyad
Figure 1: An example of creation of a benchmark pair for dimension D 16. We first initialize
3 random Gaussian Mixtures IP and Q1, Q2 and fit 2 approximate OT maps VihtP x Qi i = 1, 2. We use the average of potentials to define the output measure: (Vab + Vt Each scatter plot contains 512 random samples projected to 2 principle components of measure (Vii 1/1 + VP.
Images. We use the aligned images of CelebA64 faces datase to produce additional benchmark pairs. First, we fit 3 generative models (WGAN-QC [19] the dataset and pick intermedi- ate training checkpoints to produce continuous measures Quaty Qina- Qu for the first 2 models (k: = 1, and the final checkpoint of the third model (A: 3) to produce measure IP To make measures absolutely continuous, we add small Gaussian noise to the generator's output. Each checkpoint (Early, Mid, Late, Final) represents images of faces of a particular quality. Next, for ke (1,2) and Cpkt E (Early, Mid, Late), we use [W2] solver to fit an approximate transport map wide Vilcppe for the pair (Pii Qt (x i.e., V HTp3 Final & The potential &ppki convolu- tional ICNN with ConvICNN64 architecture ($B.I) I.. For each Cpkt, we define a benchmark pair (PeoloA @Cpkx)
(TP3 Fimal> [(V &ppr + Vule V 2/2] 2/ Pima) See Appendix A.2 and Figure 2 for details.
4.2 Metrics and Baselines
Baselines. We propose three baseline methods: identity [ID], constant [C] and linear [L]. The identity pro
solver outputs 1nd = idgo as the transport map. The constant solver outputs the mean value of Q
i.e, 7 = Fg[y]] = pQ. The linear solver outputs T (x) Σ (x:) Eg D $ x (x (ee) + pQ:
i.., the OT map between measures coarsened to Gaussians Theorem 2.3].
Metrics. To assess the quality of the recovered transport map T : R2D RD from IP to Q we
unexplained variance percentage (UVP) 16: (2-UVP(7) def 100 17 - 7 12a Var (Q) %. Here
T is the OT map. For values x 0%, / approximates 7 well. For values > 100% map T is far from optimal. The constant baseline ('- 100%.
http://mmlab. 1e. cuhk edu. htt.:///oooocts/alle....tttt

---

Quarr
Generator
Latent space
V V
CelebA Dataset
Qenrn
2 VVEllty + V.
Parry
Pp' Ainal
Latent
Generator
Vitanll
Generator
space
Q Early
Latent space
Figure 2: The pipeline of the image benchmark pair creation. We use checkpoints of generative model: Tp3 (well-fitted) and Q Ep (under-fitted). For k 1,2 fit approximate OT map Final - Ldps by Vel ie. a gradient of ICNN. We define the benchmark pair by def (mm Final 1 (V V In the visualization, Cpkt is Early.
(Pecla @cpli)
To measure the quality approximation derivative of the potential [idgo - -T] ~ V that used to update generative models (6) cosine similarity (cos):
( - id, V 1 id)
cos(id ffiid
T*) def
E e[-1,
| - id (²(F) [T' - cd))
To estimate CV-VPP and cos metrics, we use 214 random samples from
4.3 Evaluation of Solvers High-dimensional Benchmark Pairs
We evaluate the solvers on the benchmark and report the computed metric values for the fitted transport map. For fair comparison, in each method the potential and the map H (where applicable) are parametrized as fa alll
vig and H = Vew respectively, where ehe, o., use DenselCNN architectures [16] BB.2]. In solvers [QC]. [LS], (.MBB.[[MMM we do not impose any restrictions on the weights 0, w. i.c. th, ow are usual fully connected nets with additional skip connections. We provide the computed metric values in Table 2 and visualize fitted maps (for
64) Figure
All the solvers perform well (C2-UVP 2 O, cos & 1) in dimension D 2. In higher dimensions, only [MMv1], [MM], [MMv2], [W]] and their reversed versions produce reasonable results. However, [Mvv1] solver is slow since each optimization step solves a hard subproblem for computing ʃe. also hard optimize: they either diverge from the start (-+) or diverge after converging to nearly-optimal saddle point (9+). This behavior is typical for maximin optimization and possibly can be avoided by more careful choice of hyperparameters.
Maximin solvers [MM]. [MM:R]
For [QC], [MM] ). as the dimension increases, the C2-UVP drastically grows. Only [MM-B] notably outperforms the trivial [L] bascline. The error of [MM-B] is explained by the overestimation of the inner problem in 8). yielding biased optimal potentials. The error of [LS] comes from bias introduced by regularization [36]. In |QC], error arises because a discrete OT problem solved on sampled mini-batches, which is typically biased [5] Theorem 1]. used to update fa. Interestingly, although [QC], [LS] imprecise in terms of 2--UVP, they provide high metric.
Due optimization issues and performance differences, wall-clock times for convergence are not representative. All solvers except [MMv1] converged in several hours. Among solvers that
7

---

Dim
8
16
64
128
256
Dim
2
16
32
128
256
0.2
1.0
13
144
K.1
2.2
2.5
MM.+1
0.99
0.59
0.59
0.34
0.97
0,99
0.59
[MM]
3.2
3.19+
[MM]
0.994+
IMMER
07
4.5
[MMER
0.99
1.80
1.00
[MMv2]
0.1
0.68
2.2
3.1
5.3
10.19+
3.24+
2,74+
[MMv2]
0.99
0.99
0.99
0.99
0.99
0,964
0.99
0.994
[M
0.1
0.7
44
7.7
5.8
6.8
2.1
2.8
[MM+2:R]
0.99
1,00
0.97
0.96
0.99
0.97
0.99
1.00
W2:.
4.0
7.0
2.0
2.7
[W2:R]
0.99
1,00
0.98
0.98
0.99
0.97
1.00
1.00
[M-66]
0.1
0.7
3.1
6.4
12.0
13.9
19.0
22.5
[MM-B]
0.99
1.00
0.98
0.96
0.96
0.94
0.93
0.93
LSS]
5.0
11.6
21.5
31.7
40.3
46.8
56.7
0.86
0.88
0.82
[1]
27.3
639
63.6
67.4
0.75
0.76
0.77
10C
1.3
14.3
28.6
67.2
68.0
75.2
50.3
M8.2
[9C
0.99
0.86
0.788
0.70
0.70
0.90
0.69
0.00
1C
100
100
100
100
100
100
0.29
0.32
0.38
0.46
0.55
0.58
0.60
0.62
Table 2: -VUUVP (%, on the left) and cos [-1, (on the right) metric values for transport maps
fitted by OT solvers on the high-dimensional benchmark in dimensions D 2, 2, 2 Orange
highlights C²-UVP 10% and cos 0.95. Red indicates performance worse than [L] baseline.
Input
Ground truth putput Vp ap
[MM] fitted map Vaysp
[W2] fitted map Vypap
[LS] fitted map Viyap
[OC) fitted map Ppy
[MM-8] fitted map Vayap
[L] baseline map
Figure 3:
Visualization of a 64-dimensional benchmark pair and OT maps fitted by the solvers. Scatter plots contain 512 random samples projected onto principal components of measure V..
substantially outperform the linear baseline, i.e. [MM] MMv1], [MMv2], [W2], [MM-B], the
fastest converging one is [MM-B], but is biased. [M]] MMv2], [W2] require more time.
4.4 Evaluation of Solvers in CelebA 64 64 Images Benchmark Pairs
For evaluation on the CelebA benchmark, excluded [LS]] and [MMv1]: the first is unstable in high dimensions 33 and the second takes too long to converge. ICNN-based solvers [MMv2], [W2] and their reversed versions perform roughly the same in this experiment. For simplicity, treat them as one solver [W2].
In [W2], we parametrize fo = =1 12 12- the and H. V where vhe &. are input-convex neural nets with ConvexICNN64 architecture (BI.). All the other solvers are designed in the generative modeling setting to work with convolutional architectures for images. Thus, in [MM], (QC). [M-BB] we parametrize networks fo as ResNet and H., as U-Net (in [MM]. In turn, in [MM:R] we parametrize To by UNet and fw by ResNet.
We compute the transport map Qcckt Peeleea for each solver three image benchmarks. The results are Figure And Table 3) and echo patterns observed on high-dimensional problems ($4.3). [QC], [MM-B] suffer from extreme bias thanks the high dimension of images, and the derivative of W2 computed by these solvers is almost orthogonal to the true derivative (cos & 0). This means that these solvers do not extract W2. [MM], [MM:R]. [W2] recover the transport maps well. [MM]'s map is slightly noisier than the one by JMMR), minor example of gradient deviation.
4.5 Evaluation of Solvers in Generative Modeling of CelebA 64 x 64 Faces
Based on our previous evaluation, many existing neural OT solvers are notably imprecise. This leads us to ask: To what extent does solver quality matter in real-world applications?

---

Input Rcput
Target -"xo-Feoeea
[W2]
Vyb(x)
[MM:R]
Ta(x)
[MM]
Vfy(x)
[MM-B] Vfa(x)
IQCI
X- -fx((
(a) Fitted maps Quaty - Paataa
(b) Fitted maps Quaa Pcaaaa (c) Fitted maps Quate P'cchh
Figure OT maps fitted by solvers on benchmarks QQcpi, (eleba)) 1st line contains random Qcp-r 2nd line contains samples from PeeehA obtained by pushing forward by OT map T V8 Subsequent lines show x transported by maps fitted by OT solvers.
Cpk
Earby
Mid
Late
Cya
Earby
Late
W2]
1.7
0.25
[W]]
0.99
0.95
0.93
MM.
0.99
DD]
31.2
4.26
206
[D]
0.0
0.00
000
[M--B
47.74
[C]
100
[C]
0.03
-014
-0.20
[QC]
94.7
26 100
100
[QC]
0.17
-001
0.05
Table 3: L2-UVP (%, on the left) and E [-1, (on the right) metric values for transport maps
Qcpke - Celeba fitted by OT solvers on 3 developed CelebA64 W2 benchmarks.
To address this question, evaluate the most promising solvers in the task of generative modeling for CelebA 64 64 images faces. For comparison, we add [QC]. which has good generative performance [19] For each solver, train generative network G. with ResNet architecture from 19 to map a 128-dimensional normal distribution to the data distribution Q. As the loss function for generator, GAN-style training, where gradient updates of the generator alternate with gradient steps of OT solver (discriminator) (3B.2.3). We show sample generated images the top row of each subplot of Figure 5lad report FID 13.. On the bottom row, we show the pushforward of the OT map from P G to extracted from the OT solver. Since the model converged (Pa Q) the map should be nearly equal to the identity.
use W Q) W ( Q) estimated by each solver. We perform
[W2] provides the least quality (Figure 5ap. This can be explained by the use of ConvICNN: the other solvers convolutional architectures and work better. In general, the applicability of ICNNs to image-based tasks is questionable [6, $5.3) which might be a serious practical limitation.
[QC] has strong generative performance (Figure 5b). However, as $4.3 4.4 the recovered map is far from the identity. We suspect this solver has decent generative performance because approximates some non- W? dissimilarity measure in practice.
[MM] results in generative model that produces blurry images (Figure 5cp. The computed transport map idg - fe too far from the identity due the gradient deviation. This leads to inaccurate gradient computation used to update the generator and explains why the generator struggles to improve. We emphasize that in 34.4 [MM] does not notably suffer from the gradient deviation. Probably, this is due to measures being absolutely continuous and supported on the entire RD_ This not the case in our generative modeling setup, where generated and data measures are supported on now-dimensional manifolds in IR

---

x= G2(2) [FID=90.2]
x= G₂(2) [FID=14.4]
y= (x) 89.8=
y= x x- Vfx(x)
(a) [W2] solver (ICNN tha V of ICNN H..).
(b) [QC] solver (ResNet fo).
== Gz(z)) [FID 62.1]
x= Gz(Z)
[FID 58.2)
y= x- Vfx(x)
y= Ty(x) [8.8=18.8]
(c) [MM] solver (ResNet So. UNet H.).
(d) [MM:R] solver (UNet Te, ResNet gw).
Figure 5: Random images produced by trained generative models with OT solvers. The 1st line shows random generated images x = G. (z) ~ I... S. The 2nd line shows computed transport map from the generated x (z) P.. to the data distribution Q
Reversed [MMR] overcomes the problem of [MM] with the gradient deviation but still leads to blurry images (Figure 5ch. Interestingly, the fitted transport map To significantly improves the quality and images Tg Ga (2) are comparable the ones with |QC] solver (Figure 5b).
We emphasize that formulations from [MM], [MM:R] solvers are maximin:
using them in GANs requires solving challenging mhin-max-mil optimization problem. To handle this, we use three In our experiments, the training was not stable and often diverged: the reported results use the best hyperparameters we found, although there may exist better ones. The difficulty in selecting hyperparameters and the unstable training process are limitations of these solvers that need to be addressed before using in practice.
nested loops and stochastic gradient sescent-ascent-desse
5 Conclusion
Our methodology creates pairs of continuous measures with ground truth quadratic-cost optimal transport maps, filling the missing gap of benchmarking continuous OT solvers. This development allows to evaluate the performance of quadratic-cost OT solvers in OT-related tasks. Beyond benchmarking the basic transport problem, our study of generative modeling reveals surprising patterns: bad OT solvers can yield good generative performance, and simply reversing asymmetric solvers can affect performance dramatically.
Limitations. We rely on ICNN gradients as W2 optimal transport maps to generate pairs of bench- mark measures. It is unclear whether analogous constructions can be used for other costs such as W1. We also limit our benchmark pairs to be absolutely continuous measures while limiting the ground truth transport maps to be gradients of ICNNs, which may not have enough representational power. While we reveal a discrepancy between performance in OT-related tasks and performance in generative modeling, in-depth study is needed to answer questions such what exact dissimilarity metrie QCC implies that explains its generative performance while poorly approximating W.
Potential impact. We expect our benchmark to become a standard benchmark for continuous optimal transport as part of the ongoing effort of advancing computational OT, in particular, in its application to generative modeling. As a result, we hope our work can improve the quality and reusability of OT-related research. One potential negative is that benchmark might narrow the evaluation of future OT solvers to the datasets of our benchmark. To avoid this, besides particular benchmark datasets, in 3 we describe generic method to produce new benchmark pairs.
The problem statement was developed in the framework of Skoltech-Mit NGP program. The work of Evgeny Burnaev was supported by the Ministry of Science and Higher Education of the Russian Federation grant No. 17-2011 1-668. The MIT Geometric Data Process- ing group acknowledges the generous support of Army Research Office grants W91 INF2010168 and 1011NF111029, of Air Force Office of Scientific Research award FA9550-19-1-033 of National Science Foundation grants 183-18 and 1955195, from the CSAIL Systems that Learn program, from the MIT-IBM Watson AI Laboratory, from the Toyota-CSAI Joint Research Center, from a gift from Adobe Systems, from an MITnano Immersion Lab/NCSOFT Gaming Program seed grant, and from the Skoltech-MIT Next Generation Program.
CCKNOWLEDGEMENTE
10

---

References
[1] Pedro C Elstren- E Del Barrio, JA Cuesta Alberto, and Matrán. A fixed-point ap- proach to barycenters in Wasserstein space. Journal of Mathematical Analysis and Applications, 441(2):744 762, 2016.
[2] Brandon Amos, Lei Xu, and Zico Kolter. Input convex neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 146-155. JMLR. org. 2017.
[3] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. arXiv preprint arXiv:1701.0787. 2017.
[4] Jonathan T Barron. Continuously differentiable exponential linear units. arXiv preprint arXiv:1704.0743,, 2017.
[5] Mare G Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The cramer distance as solution to biased Wasserstein gradients. arXiv preprint arXiv: 1705 2017.
[6] Yann Brenier. Polar factorization and monotone rearrangement of vector-valued functions. Communications on pure and applied mathematics, 44(4):375-417, 1991.
[7] Jiezhang Cao, Langyuan Mo, Yifan Zhang, Kui Jia, Chunhua Shen, and Mingkui Tan. Multi- marginal Wasserstein GAN, arXiv preprint arXiv: 1et1109911800 2019.
[8] Yucheng Chen, Matus Telgarsky, Chao Zhang, Bolton Bailey, Daniel Hsu, and Jian Peng. gradual, semi-discrete approach to generative network training via explicit Wasserstein minimization. In International Conference on Machine Learning, pages 1071-1080. PMLR, 2019.
[9] Jiaojiao Fan, Amirhossein Taghvaei, and Yongxin Chen. Scalable computations of Wasserstein barycenter via input convex neural networks. arXiv preprint arXiv:2007.04462, 2020.
[10] Werner Fenchel. On conjugate convex functions. Canadian Journal of Mathematics, 1(1):73-77, 1949.
[11] Aude Genevay, Marco Cuturi, Gabriel Peyré, and Francis Bach. Stochastic optimization for large-scale optimal transport. In Advances in neural information processing systems, pages 3440-3448, 2016.
[12] Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, pages 5767-5777, 2017.
[13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time- update rule converge to local nash equilibrium. In Advances in neural information processing systems, pages 6626-6637, 2017.
[14] Leonid Kantorovitch. On the translocation of masses. Management Science, 5(1):1-4, 1958.
[15] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[16] Alexander Korotin, Vage Egiazarian, Arip Asadulaev. Alexander Safin, and Evgeny Burnaev. Wasserstein-2 generative networks. In International Conference on Learning Representations, 2021.
[17] Alexander Korotin, Lingxiao Li, Justin Solomon, and Evgeny Burnaev. Continuous wasserstein- 2 barycenter estimation without minimax optimization. In International Conference on Learning Representations, 2021.
[18] Jacob Leygonie, Jennifer She, Amjad Almahairi, Sai Rajeswar, and Aaron Courville, Adversarial computation of optimal transport maps. arXiv preprint
2019.
11

---

[19] Huidong Liu, Xianfeng Gu, and Dimitris Samaras. Wasserstein GAN with quadratic transport cost. In Proceedings of the IEEE International Conference on Computer Vision, pages 4832- 4841, 2019.
[20] Huidong Liu, GU Xianfeng, and Dimitris Samaras. two-step computation of the exact GAN Wasserstein distance. In International Conference on Machine Learning, pages 3159-3168. PMLR, 2018
[21] Shu Liu, Shaojun Ma, Yongxin Chen, Hongyuan Zha, and Haomin Zhou. Learning high dimensional Wasserstein geodesics. arXiv preprint arXiv:2102.02992, 2021.
[22] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[23] Guansong Lu, Zhiming Zhou, Jian Shen, Cheng Chen, Weinan Zhang, and Yong Yu. Large- scale optimal transport via adversarial training with cycle-consistency. arXiv preprint 20X3::003..06635, 2020.
[24] Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are GANs created equal? a large-scale study. In Advances in neural information processing systems, pages 700-709, 2018.
[25] Yun Luo, Si- Si-Yang Zhang, Wei-Long Zheng, and Bao-Liang Lu. WGAN domain adaptation for eeg-based emotion recognition. In International Conference on Neural Information Processing, pages 275-286, Springer, 2018.
[26] Ashok Vardhan Makkuva, Amirhossein Taghvaei, Sewoong Oh, and Jason D Lee. Optimal transport mapping via input convex neural networks. arXiv preprint arXiv: /0962, 2019.
[27] Anton Mallasto, Jes Frellsen, Wouter Boomsma, and Aasa Feragen. (q. Wasseessnnn GANs: Comparing ground metrics for Wasserstein GANs. arXiv preprint arXiv: 2019.
[28] Anton Mallasto, Guido Montúfar, and Augusto Gerolin. How well do WGANs estimate the Wasserstein metric? arXiv preprint (tt0::1910008877 2019.
[29] Paul Milgrom and Ilya Segal. Envelope theorems for arbitrary choice sets Econometrica, 70(2):583-601, 2002.
[30] Quan Hoang Nhan Dam, Trung Le, Tu Dinh Nguyen, Hung Bui, and Dinh Phung. Threeplayer Wasserstein GAN via amortised duality. In Proc. of the 28th Int. Joint Conf. on Artificial Intelligence (IJCAI), 2019.
[31] Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport. Foundations and Trends@ in Machine Learning, 11(5-6):355-000 607, 2019.
[32] Thomas Pinetz, Daniel Soukup, and Thomas Pock. On the estimation of the Wasserstein distance in generative models. In German Conference on Pattern Recognition, pages 156-170. Springer, 2019.
[33] Maziar Sanjabi, Jimmy Ba, Meisam Razaviyayn, and Jason D Lee. On the convergence and ro- bustness of training GANs with regularized optimal transport. arXiv preprint arXiv:1802.08249, 2018.
[34] Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 55(58- 63):94, 2015.
[35] Jöm Schrieber, Dominic Schuhmacher, and Carsten Gottschlich. Dotmark-a - benchmark for discrete optimal transport. IEEE Access, 5:271 1272, 2016.
[36] Vivien Seguy, Bharath Bhushan Damodaran, Rémi Flamary, Nicolas Courty, Antoine Rolet, and Mathieu Blondel. Large-scale optimal transport and mapping estimation. arXiv preprint arXiv: 2017.
12

---

[37] Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Proceedings the AAAI Conference on Artificial Intelligence. volume 32, 2018.
[38] Jan Stanczuk, Christian Etmann, Lisa Maria Kreusser, and Carola-Bibiane Schonlieb. Wasser- stein GANs work because they fail (to approximate the Wasserstein distance). arXiv preprint arXiv:2103.0167, 2021.
[39] Amirhossein Taghvaei and Amin Jalali. 2- W-ssesesteein approximation via restricted convex potentials with application to improved training for GANs. arXiv preprint arXiv: 1922:0090 2019.
[40] Cédric Villani. Topics in optimal transportation. Number 58. American Mathematical Soc., 2003.
[41] Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
[42] Jiqing Wu, Zhiwu Huang, Janine Thoma, Dinesh Acharya, and Luc Van Gool. Wasserstein divergence for GANs. In Proceedings of the European Conference Computer Vision (ECCV), pages 653-668, 2018.
[43] Yujia Xie, Minshuo Chen, Haoming Jiang, Tuo Zhao, and Hongyuan Zha. On scalable and efficient computation of large scale optimal transport. volume 97 of Proceedings of Machine Learning Research, pages 6882-6892, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
13


Reviews:
[{"review_id": "tjdBU8Fwwe", "reviewer": "Reviewer_jpfi", "summary": "This paper proposes a continuous Wasserstein-2 benchmark to evaluate different neural OT solvers including WGAN varients. Extensive experiments are performed on toy datasets and a real face dataset.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. It's good to see a benchmark that evaluates different neural OT solvers. This could benefit future OT application works that choose the best OT solver. \n2. Comprehensive OT solvers are investigated, including some WGAN variants that can be potentially used as OT solvers. \n3. Various metrics such as L^2-UVP, cos, and FID are used to evaluate OT solvers.\n\nWeakness:\n\nI think two important theoretical problems remain unsolved in this paper:\n\n1. In line 216, the authors mentioned that to compute the L^2-UVP and the cos metrics, they use 2^14 random samples from P. They actually computed the estimated values for the two metrics. \nI would like to know the sample complexity of estimating the two metrics, i.e., the estimation error of L^2-UVP and cos in terms of the number of samples required. Is there a curse of dimensionality problem in estimating the L^2-UVP and cos,\njust like computing the Wasserstein distance for two continuous distributions using sampling methods? This is important because if the number of required samples grows exponentially as the dimensionality increases,\nthen the metrics may not be accurate in evaluating OT solvers in the 64d space but only with 2^14 samples.\n\n2. In line 175, the authors construct a benchmark pair (P, 1/2 (\\nabla \\psi_1 + \\nabla \\psi_2)#P), and the 1/2 (\\nabla \\psi_1 + \\nabla \\psi_2) is used as the ground truth transport map if I understand correctly. \nMy concern is that \"is 1/2 (\\nabla \\psi_1 + \\nabla \\psi_2) really the optimal transport map between P and 1/2 (\\nabla \\psi_1 + \\nabla \\psi_2)#P\"? Remember that \\psi_1 and \\psi_2 are approximated by the W2 solver. \nI think this paper should theoretically show \"how far is 1/2 (\\nabla \\psi_1 + \\nabla \\psi_2) away from the real optimal transport map between P and 1/2 (\\nabla \\psi_1 + \\nabla \\psi_2)#P\", and show how this \ngap affects the metrics that are used to evaluate OT solvers.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed some limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "sxP89rpK8dS", "reviewer": "Reviewer_oJwC", "summary": "This work introduces a continuous Wasserstein-2 benchmark to assess the qualities of different neural optimal transport solvers. The work uses input convex neural networks (ICNN) to construct pairs of measures whose ground truth OT maps can be obtained analytically. This strategy yields pairs of continuous benchmark measures in high-dimensional spaces such as spaces of images. The authors thoroughly evaluate existing optimal transport solvers. The study reveals crucial limitations of existing solvers.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Optimal transportation maps become more and more important and popular in deep learning field. Due to its highly non-linear nature, reliable computation with high precision is challenging. So far, there is no solid benchmark to assess different solvers. This work proposes a benchmark to tackle this problem. Hence it has importance for deep learning fields. \n\nThe manuscript is well written. The introduction part is highly motivated and encouraging. The theoretic foundation part is clearly written, easy to follow, but can be further improved.\n\n1. The background on optimal transport is insufficient. There are many existing methods based on Brenier theorem and solving Monge-Ampere equations. For example, Benamou-Brenier used fluid dynamic formulation to solve L2 optimal transportation map; Yau et al give the intrinsic connection between convex geometry and optimal transport, and solve the OT map problem using Alexandrov formulation, see the work  of Lei et al \"A Geometric Understanding of Deep Learning\",   Engineering, Volume 6, Issue 3, March 2020, Pages 361-374. The algorithm there has theoretic foundation. \n\n2. The benchmark ignores one important aspect of optimal transportation map: the boundary condition. In theory, the image of the OT map of the source domain should equal to the target domain, this is the most difficult constraint to satisfy in practice and affect the computational result crucially. According to Figalli's works, if the target domain is non-convex, then the OT map itself may not be continuous, therefore cannot be represented by the deep nerual networks. Even for the benchmark itself, the ground truth map may not be representable by the network. According to Yau's works, this is the reason for mode collapsing. Hence this need to be emphasized and carefully designed in the bench mark.\n\nThe experimental results are convincing, different solvers are compared. \n\n3. It will be more helpful to add geometric based approaches for the test. It will be more convincing if the convergence error is also added. According to the theoretic analysis, the Brenier potential error is O(h^2) where h is the diameter of the cells in the tessellation. \n\nIn general, the idea of Benchmark is important, it will make the whole field more rigorous. Current design can be further improved by emphasizing the 2nd boundary condition, and considering the regularity of the OT maps. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are more limitations of the current design:\n\n1. Secondary boundary condition should be emphasized, which significantly affect the computational quality.\n2. The regularity of the OT maps need to be carefully considered, if the target measure support is non-convex, the OT maps can not be represented by deep neural networks. This fact should be carefully addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "XdAz0rP2Zqs", "reviewer": "Reviewer_paeq", "summary": "The authors present a continuous Wasserstein-2 benchmark using input convex neural networks to obtain ground truth continuous OT maps. They evaluate continuous OT solvers based on how well they reproduce the ground truth transport map, how well they approximate the derivative of the potential, and how well they perform in a GAN framework. This provides a basis to evaluate current and future continuous 2-Wasserstein solvers.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this was an excellent read and I found the use of the ICNN to generate ground truth OT maps novel and clever. The authors mention \"Because of the choice of [W2], subsequent evaluation might slightly favor ICNN-based methods\" (Line 177) and elaborate in the limitations section. This seems like a major limitation, the fact that the transport maps are not based on any real distribution and are in fact sampled from the set of potentials parameterized by ICNNs is highly biased. Would it be possible to examine this bias in some way? Perhaps some P, Q pairs with tractable transport maps that do not require ICNNs?\n\nThe evaluation of W2 solvers in generative modeling is also extremely useful. While this study has a number of limitations it is well posed and a useful benchmark for future work in W_2 solvers. I found this a fascinating read and applaud the authors for their efforts.\n\n------------------------------\nEdit post response:\nI am satisfied by the authors' response. I keep my score. \n\nI initially had the same understanding as reviewer oJwC made in his point 2. Given that this is a mainly a benchmark paper, the construction of the benchmark is of utmost importance. It would improve the paper to further emphasis the construction of the ground truth based on maps model-able by ICNNs, which is the most significant limitation of this work. Having a clear discussion of the use of DenseICNN parameterizing the ground truth transport maps earlier in this work perhaps in section 3 (rather than just in the limitations paragraph at the end) would both make it clearer how the benchmarks are constructed and what the limitations are. The paragraph titled \"Arbitrary pairs\"on Line 172 is misleading as a ground truth map is not constructed between arbitrary pairs, but between P and something close to Q based on how well the ICNN is fit; I suggest \"Approximating arbitrary pairs\" or similar would be clearer.\n\n While I agree other convex parameterizations could be used, this is an important point and deserves further clarification.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose an interesting approach to evaluating transport plans. There exist some concerns about methodology. However, to my knowledge, I have never seen this type of study before in OT problems. So, the problem itself is new and novel. Thus, I also would like to vote for acceptance. For the camera-ready version, I expect the authors to revise the paper based on the reviewer's comments.
Decision: Accept (Poster)