Neural Monge Map estimation and its applications
Authors: Jiaojiao Fan
Shu Liu
Shaojun Ma
Hao-Min Zhou
Yongxin Chen
Venue: Accepted by TMLR ()

Abstract: Monge map refers to the optimal transport map between two probability distributions and provides a principled approach to transform one distribution to another.  Neural network-based optimal transport map solver has gained great attention in recent years. Along this line, we present a scalable algorithm for computing the neural Monge map between two probability distributions. Our algorithm is based on a weak form of the optimal transport problem, thus it only requires samples from the marginals instead of their analytic expressions, and can be applied in large-scale settings. Furthermore, using the duality gap we prove rigorously \textit{a posteriori} error analysis for the method. Our algorithm is suitable for general cost functions, compared with other existing methods for estimating Monge maps using samples, which are usually for quadratic costs. The performance of our algorithms is demonstrated through a series of experiments with both synthetic and realistic data, including text-to-image generation, class-preserving map, and image inpainting tasks.

Published in Transactions on Machine Learning Research (07/2023)
Neural Monge map estimation and its applications
Jiaojiao Fan"
jaejjaafen edu
Georgia Institute of Technology
Shu Liu'
shuliu@math ucla edu
University of California, Los Angeles
Shaojun Ma Georgia Institute of Technology
Chaojurema@@atechhh eduu
Hao-Min Zhou
mmnhoow@mat.ggaeeed edu
Georgia Institute of Technology
Yongxin Chen
yougchen
Bowlech.acc...... yongcher@@alecch.comm
edu
Georgia Institute of Technology
Reviewed on OpenReview: https: // opwp://veeeeeeee..eemmmm net/ forum? id= 2m2S 2
Abstract
Monge map refers to the optimal transport map between two probability distributions and provides a principled approach to transform one distribution to another. Neural network- based optimal transport map solver has gained great attention in recent years. Along this line, we present a scalable algorithm for computing the noural Monge map between two probability distributions. Our algorithu is based on a weak form of the optimal transport problem, thus it only requires samples from the marginals instead of their analytic expres- sions, and can be applied in large-scale settings. Furthermore, using the duality gap we prove rigorously posteriorí error analysis for the method. Our algorithm is suitable for general cost functions, compared with other existing mot for estimating Monge maps using samples, which are usually for quadratic costs. The performance of our algorithms is demonstrated through a series of experiments with both synthetic and realistic data, including text-to-image generation, class preservne map, and image inpainting tasks.
1
Introduction
The past decade has witnessed great success of optimal transport (OT) Villani
2008
based applications in
machinc learning community (Arjovsky et al. 2017
Krishnan & Martínez, 2018 Li et al. 2019 Inoue et al.
2020
Ma el al. 2020 Fan et al.] 2020 Haas er el al.
2020
Melez-Melis & Fusi 2020
Mvarez Melis el al.
2021 Bunne et al. 2021 Mokrov et al.
2021
Bunne et al. 2022 Yang & Uhler 2018 Fan et al.) 2022a, Fan
& Alvarez-Melis 2022 Nguyen & Ho
2022 Zhu ct al. 2021). The Wasscrstein distance induced by OT is
widely used evaluate the discrepancy between distributions thanks to its weak continuity and robustness.
In this work, given any two probability distributions Pa and Ps defined on RR" and Rm consider the Monge problem
I c(x, ((x))(a)) dr. RR
CMonge (Pa: Po) A
min T.R - T:P.=
(1)
Here c(x,y) denotes the cost of transporting from and T is the transport map. We define the pushforward of distribution Pa by T as (F) pa (T-)(E)) for any measurable set E C m The Monge
* Equal contribution.

---

Published in Transactions on Machine Learning Research (07/2023)
problem seeks the cost-minimizing transport plan T. from Pa to pb- The optimal T, is also known as the Monge map of 1
In this work, we propose a scalable algorithm for estimating the Wasserstein distance as well as the op- timal map in continuous spaces without introducing any regularization terms. Particularly, we apply the Tagrangian multiplier directly to the Monge problem and obtaín a sup-inf problem. Our contribution is summarized as follows: 1) We develop neural network based algorithm to compute the optimal transport map associated with general transport costs between two distributions; 2) Our method is capable of com- puting OT problems between distributions with different dimensions. 3) We provide rigorous posteriori error analysis of the algorithm based on duality gaps; 4) We demonstrate its performance and its scaling properties in truly high dimensional settings through various experiments.
Just like other computational OT methods, our method does not require paired data for learning the transport map. In real-life applications, the acquirement and maintenance of large-scale paired data are laborious. We will show that our proposed algorithm can be applied to multiple culting-edge tasks, where the dominant methods still require paired data. In particular, examples include text-to-ineg to--tmaee generation and image inpainting, which confirm that our algorithm can achieve competitive results even without paired data.
This salad is Taste com-au memoer and it's hard to
"Lowestoft Pastern Widemouth
Grand Marnier French Toast stacked on a with fresh berries
1965 Ford Mustang (CC 1304459) for Misscuri
Garden workroam with mix of windows by Rooms
Interer Coursyard 49-1 Kindesign
Garden
with Blossoms
Sauces
Cheesecaizs
text embetainng
our embeoding
real embedding prtor embedding
real image
Figure 1: Random image samples with size 224 x 224 on Laion art testing prompts. Except for the last row of real images, we show the outputs of the same decoder with different conditions. The condition of decoder includes a text encoding, which we keep unchanged across rows, and an image embedding. In the first row, we feed the decoder the text embedding, and the generated images are unrealistic because the image and the text embeddings are not interchangeable. In the second row, we pass the embedding mapped by our transport map, which is trained on unpaired data. As a baseline method, we pass an image embedding generated by the diffusion prior in HALL-E2-LLain which is trained on paired data. To explicitly show the decoder's recovery ability, we pass the real image embedding the fourth row.
2

---

Published in Transactions on Machine Learning Research (07 (022)
cut jawaryy gold nome word of
this week rounded healthy vegetarian that you can find on ambitiuu kitchen
polticinn delivers state speech wednesday.
elevated sidewelk the times..
texture pleasing winter ready color..
bethroom for house with huge windooss
showing the master hedrnom
sunset in the land by parson
emmeeddding
text eextemetttii
our embedding
Wilr
prior embedding
Wwwtnfo
real embedding
ETIRCTIC
real image
Figure Random image samples on Conceptual Captions 3M prompts. The meaning of each row the same as in Figure
2 brief background of OT problems
Consider X' as a closed subset R" and y as an arbitrary Polish space Suppose the cost function e is defined on IR y. We denote Pa: Pb as Borel probability distributions defined on X, respectively. Then the general OT problem, which can be treated a relaxation of Monge problem 1 can be formulated as
/1 c(z, (z, rEll(pa.ps xxx
C(pas (c) = inf
where we define [[(a. Ps) as the of joint Borel probability distributions on x y with marginals equal to Pa and Pb. Once is continuous and satisfies 15) and Pa is atomless, then optimal values of Monge problem and Kantorovich problem equal each other (Pratelli 2007), i.e., CMonge Pb) C(pa, Pt).
The Kantorovich dual formulation
Villani 2008
Ch. 5) of the primal OT problem is
sup S d(v)po(s) dy / (x)) dr) 0.2
(3)
where the maximization is over all tb E Co(pa); E Co(pb) that satisfy Φ(y) - ((x) S c(x,8) y) for any
x. E N,y E y. For further discussions on the dual problem 3 as well as equivalence to the primal OT problem (2) we refer the reader Appendix A In general, the optimal solution म. of 2 can be treated stochastie transport plan, i.e., we are allowed to break single particle into pieces and then transport each
topological space is Polish space it separahle, complete, metric space. Fuclidean space R and its closed subsets Polish spaces.
3

---

Published in Transactions on Machine Learning Research (07/2023)
piece certain positions according the plan Rx. However, in this study, mainly focus computing the deterministic optimal map T. for the Monge problem
The following theorem states the existence and uniqueness of the optimal solution to the Monge problem. It also reveals the relation between the optimal map T. and the optimal transport plan T.. This theorem is taken from Chapter 10 of (Villani 2008). more complete statement can be found in Appendix A
Theorem (Informal, eristence and uniqueness of Monge map) Assume the boundary 0X of X has dimension no larger than
Suppose the cost satisfies (15), (16), (17, (18). and assume that Pa and Ps are compacily supported and Pa is absolule continuous with respect the Lebesgue mensure R Then there crists unique-in-lass tar transport map T. solving Monge problem
① Furthermore, (Id, T, 1.p. is
optimal solution to the OT problem (2)
3 Proposed method
In order to formulate a tractable algorithm for the general Monge problem ( we first notice that 1 is a constrained optimization problem. Thus, we introduce Lagrange multiplier f for the constraint T:Pa = pb and reformulate (1) as a saddle point problem
sup inf f). JECM TEM(X
Here we denote (2)) as the space of bounded continuous functions on ); and M(X V) as the space of measurable map T X + y. In the following discussion, if not specified, we always treat the optimization space of f and as M, M(), Furthermore, we define L(T. as
[x ((r,T(()) + S. f(v)(p) - TEpa)dy = [looe. F(T(x)) padx+ + f
(5)
It is worth mentioning that if we optimize the inner problem of our proposed problem (4) is also closely tied to the Kantorovich dual problem (20) of primal OT problem. Such observation will be useful on establishing several thcoretical results for our method.
The following theorem reveals the equivalence between the existence of solutions to primal problem well as dual problem 3) and the existence of saddle point (see Definition 3 of Appendix B
1 as
Theorem (Equivalence & Consistency between optima of [ and saddle point... Suppose RR,.""
R", ; Pa: Pb are Borel probability distributions on X,Y; Pa is atomles: Assume the OT distance C(pas Pb) too and the cost is continuous and bounded from below by finâte number c, i.e. c. Then,
(Equivalence) The following two statements are equivalent:
1. There crists at least one saddle point of L(T,
2. Both the primal problem ① and the dual problem 20 admit at least one solution.
(Consistency) For any optimal solution T. ①1. optimal solution é. to (20 and saddle point (f.f) of C(T, f), we have (T,, is a saddle point of L(T,f), f is an optimal solution to I and of is an optimal Parthermore, L(T, = O'Monge Pt).
solution to 20
A direct corollary of Theorem guarantees the existence of the saddle point.
Corollary (Existence of saddle point). Suppose X' = = y Rm Assume that Par are compactly supported Borel probability distributtinns om X, ), and Pa is absolute continnous with respect to the Lebesgue measure on R", Assume e satisfies (16)
18 (23 and there erists a finite constant g such that Z g.
Then the Monge map T. and the optimal solution Φ. to (20 both crist and (T., (.) is a saddle point of f)..f).
2This means T.. another Monge map solving the problem then T. T.. Spt(pa) Eu, where Spi(pa) is the support of Pa and En 3Tiis is the joint distribution X y obtained by pushing forward Pu via the map (Id, (Id,T.) X. 4hhis indicates that no single point in X' carries positive mass Pa-
set.
4

---

Published in Transactions on Machine Learning Research (07/2023)
The existence of the saddle point of L(T, is rather strong assumption and may not always hold (Zhang 2022). We next provide some further assertions only locusing on the existence of sup-inf solution to et al.
instead of the existence of saddle point.
Theorem 3 (Consistency of sup-inf solution). Assume the cost function is continuous and satisfies the
condition 15 Suppose the sup-inf problem (4) admits at least one solution (T. f). Under the condition T:0e= = Pb- I is un oplimal solulion lo Kanlorovich dual (0 then T is oplimal solution to Monge problem
1 and furthermore, L(T. f) (Pngee(PaPPP Pb).
It is hard to theoretically guarantee the condition The Ob for general cost. Empirically, we verify our method can achieve Typa Ps qualitatively (Figure 5 b). and quantitatively (Figure 4(b), Table 1
2 More results consistency without T:po pb presented in Appendix [B.3]
In the implementation, we parametrize both the map T and the dual variable f by the neural networks To, Jup with 0, being the parameters. Consequently, our goal is to solve the following saddle point problem
1 max min L(To, = N
cXx, f(( (Y
(6)
71
where N is the size of the datasets and (Xx). [Y&} are samples drawn by Pa and ps separately. The algorithm is summarized in Algorithm [ The computational complexity our algorithm is similar with GAN-type methods. particular, Algorithum requires O(K(Ki + K2) operations total.
Algorithm Computing optimal Monge map from Pa Ph
Input: Marginal distributions Pe and Pb. Batch size B, Cost function c(z, y).
Initialize To, fn-
3: for K steps do
4: Sample {x ~ Pa. Sample (Yx) Pb.
5: Update to decrease (6) for K1 steps.
6: Update to increase (6) for K2 steps.
7: end for
Output: The transport map Tg.
Mapping between unequal dimensions Our method is still applicable when c(x, is defined R" x Rm and n & m. A lypical example is c(x,,) = x(Q(). v) Pass 2010: McCann & Pass 2020 where 7 is defined on Rm x I and ((x) is an embedding function. To name few, Q(x) can be the upsampling function (Rout et al. 2022), or padding zero function (see Figure as < m, and a linear projection
(Section 6.1 as n > m.
Comparison with GAN are similar in the sense that they are both carrying out minimization over the generator transport map and maximization over the discriminator /dual potential. However, there are two main distinctions, which are summarized in the following. More detailed discussions are postponed to Appendix
worth pointing out that our method and the WGAN (Arjovsky et al., 2017)
(Purpose: arbitrary map vs optimal map) The purpose of WGAN is to find arbitrary map T such that Typa is close to the target distribution Ps: On the other hand, the purpose of our method is two-fold: we not only compute for the map T that pushforwards Pa to pb: but also guarantee the oplimalily of T in the sense of minimizing the total transportation cost Ex-pp, "(,, T(X)).
(Designing logic: minimizing distance vs computing distance itself) In WGAN, one aims at minimizing the OT distance as the discrepancy between T.Pa and the target distribution Pb, the optimal value of the corresponding loss function thus equals to 0; In contrast, our proposed method computes OT distance (as well as the transport map) between source distribution Pa and the target Pb. In this case, the optimal value of the associated loss function equals OMonge (Pa: Pb).
4 Error Estimation via Duality Gaps
5

---

Published in Transactions on Machine Learning Research (07/2023)
In this section, we assume X' = ='= Rd, i.c we consider Monge problem R Suppose solve 4 to certain stage and obtain the pair (T, f), inspired by Hütter & Rigollet (2020) and Makkuva et al. (2020):
we provide an a posteriori estimate to a weighted L error between our computed map T and the optimal Monge map T.. Before we present our result, we need the following assumptions.
Assumption (on cost c(-,)). We assume c E CR x RR)) is bounded from below. Furthermore, for any 2, E R² we assume VzC(I, y) is injective w.r.t. y; V (2ye(z, y), as a matriz, is invertible; and 2 My cl,, y) is independent of
Assumption 2 (on marginals Par ph). Par Pb are compactly supported on Rd, and Pa has densily.
Assumption 3 (on dual variable f). Assume the dual variable (RR() is always taken from c-concave functions, Le., there epists cerlain ip E (²RR)) such that s(.) = infx((((x)) + c(x,,)) (see Defivition 5.7 of Villani (2008)). Furthermore, we assume that there erists at least one minimizer By E argmin,Ap((1 + cx, for any E R And the Hessian of (p-) + el.,y) at Ty is positive definile.
Assumption 4 (Infimum over T achieves its minimizer). Assume that there erists measurable Ty E (C(T,,)
M(RR, R2) such that f E argming(L(T. [].. i.e,, Ty(x) E argmin, (1())-c(x)) for almost every E R2d
For the sake of conciseness, let us introduce two notations. We denote oI. y) = min (V2 c(,, 9)) > 0 as the minimum singular value of V2y c(2, y); and denote
Xyy) Amax (22 + c(a,, v))l==n,)>>
(7)
as the maximum eigenvalue of the Hessian of so.) + c(:,v) at Tyy. We denote the duality gaps
- C(T,) - inf L(T, J), E2(f) = sup inf L(T, inf C(T,
T
f
We can verify that the conditions mentioned in Theorem 1 are satisfied under Assumption and Assumption (1) exists and unique. We now state the main theorem
Thus the Monge map T. the Monge problem on error estimation:
Theorem (A posteriori Error Estimation via Duality Gaps). Suppose Assumption 1 3 and hold. Let
further assume the sup-inf problem 4 admits a solution (J,T) that is consistent with the Monge problem,
i.e. T equals T., Pa almost surely. Then there exists strict positive weight function 3(x) > min
o(aa))?
yER"
2X(w]
such that the weighted 2 error between computed map T and the Monge map T. is upper bounded by
IT --TTlll(((pp)) S (2)
(8)
The exact formulation of 3 is provided in 143 in the appendir B.
Remark 1. The cost c(x, y) 11 y/1 or c(2, v) - satisfy the conditions mentioned above. More specifically, when c(x, y) = - we have a (x,, u) = 1, and A(y) Amm which recovers Theorem 3.6 in Makkuva et al. 2020)) Indeed, in this case, our f, correspond to f. in Makkuva et al. (2020).
Theorem 4 provides an posteriori error bound, i.e., for any computed (T,J), we provide an upper bound 8 on the accuracy of T. The weight function ((-)) relies on f, and (T. could be arbitrary computed pair, thus it is almost impossible to provide uniform lower bound for 3(.), which is also dicussed in Makkuva et al. 22020 Remark 3.7). Further analysis on X(-) and 3(-) can be found in Appendix B.5 We clarify some limitations of our Theorem 4 below.
(Hardness evaluating E2) It could be intractable to calculate & exactly due to the term Cup f), but one can approximate them by empirical values using samples after convergence.
(Restricted form for analytic cost) Suppose that the cost satisfies Assumption and is also analytic, then e is restricted to the specifie form A(x) + F(x)" y B(y), where A, B are analytical functions Idd and F: gd + Rd is analytic with invertible Jacobian DF(z) at any E Rdd.
Some further discussion on enforcing the e-concavity of the dual variable is presented in Appendix B.4
6

---

Published in Transactions on Machine Learning Research (07/ (07/2023) 2202)
5
Related work
Discrcte OT methods (Courty et al. 2016 Feydy al. 2020
Pooladian & Nilos- Weed
2021 Mong et al.
2019) solve the Kantorovich formulation with EMD Nash 2000] or Sinkhorn algorithm (Culuri 2013). I normally computes an optimal coupling of two empirical distributions. Such type of treatment has been widely accepted since it is friendly to high dimensional cases (Altschuler et al. 2017 Genevay et al. 2018 Li et al. 2019; Xie et al. 2020), but the algorithm does not scale well to a large number of samples and is not suitable to handle continuous probability measures. In addition, they cannot map the unseen samples. With an additional parameterization of a mapping function, such as lincar or kernel function space (Perrot et al., 2016), they can map the unseen data. However, this map is not the solution to Monge problem but only approximation of the optimal coupling (Courty et al., 2017] Sec. 2.1). Perrot et al. (2016) requires solving the matrix inverse when updating the transformation map, thus introducing risk of instability. Their function space of the map is also less expressive than neural networks, which makes it difficult to solve the OT problem with complex transport costs and high dimension data. And they still can not handle large scale datasets.
With the rise of neural networks, neural OT has become popular with an advantage of dealing with continuous measure. Seguy et al. (0017) ; Genevay et al. (2016) solve the regularized OT problem, and as such introduce
the bias. Another branch of work (Makkuva et al. 2020 Korotin et al. 2021b Fan et al. 2020 Korotin
ct al.
2021c comes with parameterizing Brenier potential by Input Convex Neural Network (ICNN) 4Amos
2020 is special case of our formula. In particular, their et al. 2017 The notable work (Makkuva et al.
dual formula reads sup, inf, - [x, [,, Vg(z) - h(V ((Vg())))addd - h())p,dy, where g. are ICNN. After a
change of variable Vg= = T = -f || 112/2, becomes the same as our (5) equipped with the cost
c(x, y) = /// Later, the ICNN parameterization was shown to be less suitable (Korotin al.) 2021a
Fan al. 2022b Korotin et al. 2022a h for large scale problems because ICNN is not expressive enough.
Recently, several works have illustrated the scalability of our dual formula with different realizations of the transportation costs. When c(I,z) = ||x - v12/22 our method directly boils down to reversed max- imuun solver (Nhan Dam et al., 2019), which appears to be the best neural OT solvers among multiple baselines [Krottin et al.) 2021a]. Another variant (Rout et al., 2022) of our work obtains comparable per- formance in image generalive models, which asserts the efficacy in unequal dimension lasks. [Gaddieva el al.. (2022) utilizes the same dual formula with more diverse costs in image super-resolution task.
Based on the similar dual formula, several works propose to add random noise an additional input to make the map stochastic, i.e. one-to-many mapping, so it can approximate the Kantorovich OT plan. To make the learning of stochastic map valid, Korotin et al. (2022b) extends the transport cost c(z, y) to be weak cost that depends on the mapped distribution. Typically, they involve the variance in the weak cost to enforce diversity. The stochastic map is, however, not suitable for our problem (I) because of the conditional collapse behavior (Korotin et al., (202bb] See 5.1). Later, Asadulaev et al. (2022) extends the dual formula to more general cost functional. We note that the term general cost in |Aaadulavv et al. (2022) is different from general cost in our paper. They propose to solve the General OT problem infxet((p...)) F(T) through the dual formula
sup inf (F(T)) S [())d(() I (w)ddo(()) f xell(.))
If F(T) c(2, c(a.y)dT(x, y) and assuming optimal π. is of the form n. (Id, T.)ppas then their formula boils down to our 5 However, the assumption n. (Id, T. 120a may not always hold.
6 Experiments
In this section, we specialize 55 with different general costs c(x. to fit in various applications. We will not locus on the quadratic cost c(x, v) = || - -w1 since our formula in this special case has been extensively studied in multiple scenarios, e.g. generative model (Rout et al. 2022), super-resolution (Gazdieva et al, 2022), and style transfer (Korotin et al. 2022b). Instcad, we focus on data with specific structure, which will be revealed in the cost function. Due to a wide span of tasks, it is unrealistic to find universal basclines in this section. We will select the proper baselines according to the scale of the examples.
7

---

Published in Transactions on Machine Learning Research (07 (07/2023)
6.1 Unpaired text to image generation
We consider the task of generating images given text. prompts. Existing successful lexl-lo-image generation algorithms (Ramesh et al. 2021 2022 Saharia et al. 2022) are supervised learning methods, and as such rely on paired data. In real world, can be exhausting to maintain large-scale paired datascts given that they are mostly web crawling data, and the validity period of web data is limited. We intend to learn a map between the text and image embedding space of the CL.IP model without paired data. Our framework is shown in Figure 3 We use the same unpaired data generation scheme as Rout et al. (2022).
map
CUP
unpaied data
ancoder
"Winnic Pooh Fianca"
Cl IP text enodder
Generate image
Texts
Hello world!
Not seen
Train
ombedding by GT map
drfusion
Images
Train
Not seen
Tost
dezxder
(a) Pipeline motivated by DALL-E2.
(b) Unpaired data generation process
Figure 3: (a) During the training process (above the dotted line), our map learns to generate image em- beddings that maximize the expected similarity with imput texts. During the evaluation (below the dotted line), given a text encoding from the CLIP model, our map outputs an image embedding, which conditions pre-trained diffusion decoder generate an image. The CLIP encoder and diffusion decoder are both pre-trained and frozen.
(b) We split a paired text-image dataset training into two parts: S and T, and take the texts from S and images from T as our training data. Finally, we use the paired test data to compare with the real images.
In our algorithm, the sample from source distribution Pa is the text encoding x E IR 77 (x * 0) from CLIP (Radford et al. 2021) model, and the sample from target distribution Pb is the image embedding y E 787 which normalized to have ||æ/|22 1. The CLIP model is pre-trained on 400M (text, image) pairs of web data with contrastive loss, such that the paired texts and images share large cosine similarity, and the non-paired ones present small similarity. As result, we choose the transport cost the negative cosine similarity between Rx and y:
(Rr, v) c(r,y) 1 |
(Rx, v) |x
(9)
The frozen matrix R: 77 - 768 extracted from a linear layer of CLIP model and it projects the text encoding x the same dimension as image embedding y. The normalized projected vector Rx/ is called text embedding. Similar to DALL-E2 (Ramesh et al. 2021 2022). use text encoding instead of text embedding as the input data because the encoding contains more information. The cost. e(x,y) would enforce our map to generate an image embedding relevant to the input text.
We evaluate our algorithm on two datasets: Laion art and Conceptual captions 3M (CC-3M) Laion art dataset is filtered from a billion dataset to have the high aesthetic level, which is suitable for the learning of image generation, while CC-3M is not curated. We use the OpenAT's CLIP (VIT-I/14) model and a DALL-E2 dilfusion decoder) The training of their diffusion decoder requires paired data and its train dataset results CC-3M are heavily based the unpaired data
includes Laion art but not CC-3M. Therefore, because no part in Figure has seen paired data of CC-3M, while on Laion art, the diffusion decoder has some paired data information gained from pre-training.
We show qualitative samples in Figure D and 2 where we compare with DALL-E2 E2-Laion, a DALL-E2 model pro-trained on the paired Laion aesthetic datasot, which includes Laion art datasct. Figure [ confirms that our transport map can generate image embeddings with comparable quality even without paired data. Since CC-3M is zero-shot dataset for DALL-F2--aaio the performance of DALL-F22 F2-Laion clearly drops in
8

---

Published in Transactions on Machine Learning Research (07/2023)
Figure 2 while model still generates reasonable images. We visually verify map achieves T,Pa pb by comparing the generated embeddings with the real image embeddings (on a test dataset).. In Figure 4 (a), we present the results of our method, along with the results of non-linear kernel map given by Perrot cct al. 2016). While the latter method struggles to generalize to unseen data, our method able to generate a distribution that closely matches the ground truth.
Target embedding
Ours
Perrot al. 2016
Simlarit
0.000
0.75
Ours (CC-3M
[Laion
0.600
---
DALLE2 Laion (Laion art)
0.60
0.070
25
50
75
100
0
25
75
100
Epochs
Epochs
(b)
Figure 4: (a) Comparison between the Laion art target image embeddings and fitted measure of generated embeddings by our method and Perrot et al. (016). Samples projected onto 2D plane by PCA. (b) Averaged cosine similarity between the generated image embeddings and (Left) the ground truth image embeddings or (Right) the unrelated text embeddings. The similarily on the left depicts how well the model recovers the real image embedding, and the similarity on the right quantifies the overfitting behavior (the lower the better).
We also quantitatively compare with the baseline in terms of cosine similarity in Figure 4 (b). Note that we also normalize Txx), so T(x) = y & cos-similarrty(((()))..
1. Our method achieves higher cosine similarity w.r.t. the real images on Laion art dataset. In practice, we observe that the relevance of each (text, image) pair in CC-3M is more noisy than Laion art, which makes the model more difficult, to learn the real image embedding. This explains why cosine similarity w.r.t real image embedding CC-3M not converge to the same level as Laion art. In the meanlime, our overfiuting level is low on both dalasets.
6.2 Class-preserving map
We consider the class- daass-preerriig map between two labelled datasets. The input distribution Pa. E/=1 apd is a mixture of J distinct distributions {pd /= Similarly, the target distribution is Σ5= bipl where {aj} and {b;} satisfy Σ = 1 and Σ, bj = 1. Each distribution pa (or p associated with known label/class j. We further assume that the support of {oo), (or {obd)) are disjoint with respect to j. We seek map that solves the problem
R" / || - T(x))1 de, Vj
min Tepl
(10)
where the constraint asks the map to preserve the original class. To approximate this map, replace the constraint by contrastive penalty. This trick transfers [10 to the original Monge problem with cost
c(x,, (1',1/))) || x'112 1( (#*)))
(11)
where y and y' are the labels corresponding to x and x', is the indicator function. To better guide the mapping, we involve the label as an input to the mapping T and the potential f. Denote E(.) Rm - RJ as a pre-trained classifier on the target domain ps. Given a feature from the target domain, E(-) will output a probability vector. Then, our full formula reads
sup supinf ipf / ||0-- f0e-z12-2 y f(z:v))) dp. + f f(z':v/ d'dp,. where ã T= == ([(([(v))))
In practice, y E R is a one-hot label and known because both the source and target datasets are labeled.

---

Published in Transactions on Machine Learning Research (07/2023)
1 2||
Trainable
Pretrained and frozen
A 3 7
2 3 6777788
0
3 4 23446160 989
T
9
10.1 0.2]
R
3
5
6
8
4) 1.1.44550 TAA
3
5
6
7
8
4
11 15711 2
y = J0
]
4
5
le
7
8
43 2 25/Te K
3
5
6
7
7 PIPA
y
3
1
5
6
7
6
17 20A
(a)
(b)
Figure 5: (a) Our cost is composed of the ordinary cost || =||² and the contrastive label cost -y -y (b) Class-preserving mapping by our conditional map. The lop block is the source images, and the bottom block is the pushforward images. Each column represents class. Left: F - M, Right: - K.
Table 1: Class-presedd preesppreerving map result: accuracy of the maps and FID of generated samples. Our map is deterministie. The map by Asadulaev et al. (0022 is non-conditional on the label. M, K, F represent MNIST, KMNIST, FMNIST correspondingly. Our accuracy result is significantly better than Asadulaev
et al.
2022)
thanks to the guidance of contrastive cost
Asadulaev et al. 2022
Asadulaev et al. 2022
Ours with conditional map
Ours with non-conditional map
deterministic map
stochastic map
M - K
17.26
9.69
18.07
17.45
FID
+ M
7.15
5.26
5.78
6.86
M K
79.20
61.91
99.59
93.02
Accuracy
F + M
82.97
83.22
99.73
98.93
Recent two works (Asadulaev et al. 2022 Bunne et al. 2022) are along this line of class-guided OT maps. The former one learns an unconditional map T(x), i.c. no need label information during the inference, and the latter one learns conditional map T(x; y). Our method is different from both them because their costs do not explicitly involve the label, while we include the label in the contrastive cost (11). Our method is designed learn a conditional map, but we can also fit unconditional map to the learned conditional map for the convenience of inference.
Xiao et al. 2017 Clanuwat et al. 2018) datasets, where we use default labels in PyTorch. The visualization of our mapping is presented in Figure (b). We also calculate FID (Housel et al. 2017) between the mapped source tost dataset and the target Lest dataset. To quantify how well the map preserves the original class, we use pre-trained SpinalNet classifier (Kabir et al.) 2022) on the target domain and evaluate the accuracy of the predicted label. If the predicted label of T( y) equals the original label y, then the mapping is correct, otherwise wrong. We show the quantitative result in Table I where the results of Asadulaev et al. (2022) are from their paper. Since Asadulaev et al. (2022) makes inference using an unconditional map, to make a fair comparison, we present the results with fitted unconditional map as well in Table The stochastic map in Asadulaev let al.] (2022) leads to a lower FID because the loss function for the stochastic map incorporates the "variance" of the pushforward distribution, which encourages the generation more diverse data. However, because the stochastic map requires random noise as input, requires more sophisticated network architecture and is more computationally expensive to train.
We compare our algorithm with Asadulaev et al. 2022
*NIST (LeCun & Cortes 2005
6.3 Unpaired image inpainting
In this section, we show the effectiveness our method on the inpainting task with random rectangle masks. We take the distribution of occluded images be Pa and the distribution of the full images to be Pb. In many inpainting works, it's assumed that unlimited amount of paired training data is accessible (Zeng)
10

---

Published in Transactions on Machine Learning Research (07 (07/)023)
et al. 2021]. However, some real-world applications do not involve the paired datasets. Accordingly, we consider the unpaired inpainting task. The training and test data are generated in the same way as Figure (b. We choose cost function to be mean squared error (MSE) in the unmasked area
c(z. v) a. | O M(x) - y O M(x)
(12)
where Mx)); depending on x, binary mask with the same size as the image. M takes the value 1 in the unoccluded region, and 0 in the aakkownnmmisiine region. 3 represents the point-wise multiplication, a is a tunable coefficient, and is dimension of x. Different from Rout et al. (2022). where c(z, y) = || yl1 include the mask in the cost to avoid the penalty in the masked region. Otherwise, it penalizes the generation of non-zero pixels because the pixel values are zeros in the masked region of source image. The map satisfying constraints TaPa- ps and T(x) O M(T(x)) = x O M (x) is the optimal map.
(a) Masked images
(b) Real images
(c) Perrot al. (2016)
(d) Our T(x)
Figure 6: Unpaired image inpainting on test dataset of CelebA 64 x 64.
We conduct the experiments on CelebA 64 x 64 and 128 x 128 datasets (Liu et al. 2015):
The input images (pa) are occluded by randomly positioned square masks. Each of the source Pa and target Pb distributions contains 80k images. We present the empirical results of inpainting in Figure 0 and In Figure g we compare with the discrete OT method |eerrot et al.] (2016) since it also accepts general costs. Tt fits a map that approximates the barycentric projection so it can also provide out-of-sample mapping, although their map is not the solution to our Monge problem. We use 1000 unpaired samples for Perrol et. al. (2016) to learn the optimal coupling. The comparison in Figure confirmm that our recovered images have much better quality than the discrete OT method.
confirms
We also evaluate FID of the generated composite images w.r.t. the original images on the Lest dataset. We use 40k images and compute the score with the implementation provided by Obukhov et al. (2020). We choose transport map with 2 cost and WGAN-GP (Gulrajani et al., 2017) as the basclines and show the comparison results in Table 2 Our essential difference with transport map with L cost is that the latter cost does not involve masks, which jeopardizes the performance when the position of masks are changeable.
Table 2: Quantitative evaluation results on CelebA 64 x 64 test dataset.
Cost 12
Cost a|I /ll2/n
WGAN-GP
a
a 104
FID
9.2857 3.7109
8.9398
6.7479
Theoretically, the change of should not affect the OT map because f should be able to scale arbitrarily. However, in practice, duc to the sclf-rognlarization of neural networks (Martin & Mahoncy] 2021), f can not scale arbitrarily with a. As a result, modifying a will change the loss landscape. During the training, we observe that the magnitude ratio r = [Ee((z (())))/B[((((()) from 10a to 10. We also observe the map learned with a larger a can generate more realistic images with natural transition in the mask border and exhibit more details on the face (see Figure 14 in the appendix). In our experiments, observe that > 10 yields reasonable results.
decreases from 18 to 0.2 when a drops
11

---

Published in Transactions on Machine Learning Research (07 (273))
6.4 Population transportation the sphere
Motivated by the recent spherical transportation works (Cohen et al. 2021 Amos al., 2022, we consider the following population transport problem on the sphere for testing known that the population on earth is not distributed uniformly over the land due to various factors such as landscape, climate, temperature, economy, etc. Suppose we ignore all these factors, and we would like to design an optimal transport plan under which the current population travels along the earth's surface to form a rather uniform (in spherical coordinate) distribution over the earth's landmass. We treat the earth as an ideal sphere with radius 1, then we are able to formulate such problem as a Monge problem defined on D [0, 2 x [0, π with certain cost function c. To be more specific, we consider the spherical coordinate (6, Φ) on D, which corresponds to the point (cos sin o, sin O sin o, cos Φ) on the sphere. For any (01, (02, 22) D, we set the distance e(.,,) function the geodesic dislance on sphere, which is formulated as
proposed method. It is well-
c((01, (1), o2)) = arccos(sin Φ1 sin Φ2 cos(O1 - 02) + cos Φ1 cos Φ2).
(13)
Then assume the current population distribution over the sphere denoted as Pa (in Cartesian coordinate).
which introduces the corresponding distribution paph (in sperical coordinate) on D. Denote D C D Sph the region of the land in the spherical coordinate system. We set the target distribution Pb as the uniform distribution supported on Drand We aim at solving the following Monge problem
U c((0,)), v), T(0,) D)) dbdø {
min Bab Sph
T.T =%
The samples used in our experiment generated from the licensed dataset from Doxsey- W Whitfield et al.) (015)). In our exact implementation, in order to avoid the explosion of the gradient of arccos(-) near =1, we replace the cost with its linearization (In our computation, the constant 2 can be omitted.)
(02. = π - (sin Φ1 sin th cos(O: - 02) + cos o1 cos Φ2). 2
(14)
Furthermore, to guarantee that each sample point is transported onto landmass, we composite the trained map T with a map T D D that remains samples land unchanged but maps samples closest location on land among randomly selected sites. In Figure 7 compare our result with the linear transformation method introduced in Perrot et al. (2016). For more general kernel map, such as Gaussian kernel, their algorithm is not very stable and it is very difficult to obtain valid results.
back to the
Although the ground truth for this example is unknown to us, we refer the reader to Appendix D for more synthetic examples with ground truth such as distributions with unequal dimensions, cost of decreasing functions, and Monge map on sphere.
Figure We show our result on the left and Perrot et (2016) on the right. Each figure plots samples from the source distribution Sphh (blue) and samples from the pushforward distribution TorpSph (green). We also demonstrate the computed transport map (orange) for the first 500 randomly generated points from source p Sph to target Pb
Sph
12

---

Published in Transactions on Machine Learning Research (07 (223))
7 Conclusion and Discussion
In this paper, we present a novel method to compute the Monge map between two given distributions with flexible transport cost functions. In particular, we consider applying Lagrange multipliers on the Monge problem, which leads to a sup-inf saddle point problem. By further introducing neural networks into our optimization, we obtain a scalable algorithm that can handle most general costs.
Broader impact Our method will become a useful tool for machine learning applications such as domain adaption, and image restoration that requires transforming data distributions. Compared to other deep learning methods, our scheme reduces the need for paired data; compared to other OT methods, our scheme can be applied to high dimensional large-scale datasets, and we do not involve regularization which could hurt the quality of transformed data. In the future, it is promising to specialize our method to structured data, such as time scries, graphs, point clouds, and SPD manifolds (Juu & Guan 2022
Limitation Firstly, We observe that when the target distribution a discrete distribution with fixed support, our method tends to be more unstable. We conjecture that this is because the network parame- terization of the diseriminator is too complex for this lype of uncomplicated target distribution. Secondly, there could be gap between the theory and some of our experiments. We make summary table in Section Fl which demonstrates that for some experiments, the existence and uniquencss of the Monge map are not guaranteed. It is either because the space is not regular or the assumptions in Theorem ] are not fully satisfied. We leave the theoretical study for those examples as the future direction.
References
SN Afriat. Theory of maxima and the method of lagrange. STAM Journal on Applied Mathematics, 20(3): 343 357, 1971. (Cited on page 25
J. Altschuler, J. Niles-Weed, and P. Rigollet. Near-linear time approximation algorithms for optimal transport via sinkhorn iteration. Advances in neural information processing systems, pp. 1964 1974, 2017. (Cited on page
David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. Advances in Neural Information Processing Systems, 33:21428 28:21428-21439, 2020. (Cited on page
David Alvarez-Melis, Yair Schiff, and Youssef Mroueh. Optimizing functionals on the space of probabilities with input convex neural networks. arXiv preprint orXiv:2106. 00774, 2021. (Cited on page
B. Amos, L. Xu, and J.Z. Kolter. Input convex neural networks. In International Conference on Machine Learning, pp. 146-155, 2017. (Cited page
Brandon Amos, Samuel Cohen, Giulia Luise, and levgen Redko. Mela optimal transport, 2022. URL
https://arxive wrggabs///ee /2006 05262 (Cited on page 12
Martin Arjovsky, Soumith Chintala, and Téón Bottou. Wasserstein gan. In arxiv preprint arXin: NoN::1001000P 1701077875
2017. (Cited on pages g 5 and 29
Arip Asadulaev, Alexander Korotin, Vage Egiazarian, and Evgeny Burnaev. Neural optimal transport with general cost functionals. arXie preprint arXiv:2205.15403 2022. (Cited on pages and 10.
Charlotte Bunne, Laetitia Meng- Papaxanthos, Andreas Krause, and Marco Cuturi. Jkonet: Proximal optimal transport modeling of population dynamics. arXiv preprint arXiv:2106.066 063464456.66 2021. (Cited on page
Charlotte Bunne, Andreas Krause, and Marco Cuturi. Supervised training of conditional monge maps. anXiv preprint ar XrXiv:2206.14 14262, 2022. (Cited pages 1 and 10
Tarin Clanuwat, Mikel Bober-Trizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01 01711, 2018. (Cited on page
10
13

---

Published in Transactions on Machine Learning Research (07/2023)
Samuel Cohen, Brandon Amos, and Yaron Lipman. Riemannian convex potential maps. In International Conference on Machine Learning, pp. 2028 2038. PMI.R, 2021. (Cited on page 12
N. Courty, R. Flamary, D. Tuia, and A. Rakotomamoniy. Optimal transport for domain adaptation. JEEE transactions on poltern analysis and machine intelligence, 39(9):1853 -865, 2016. (Cited on page 7
Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamoniy. Joint distribution optimal transportation for domain adaptation. Advances in Newral Information Processing Systems, 30, 2017. (Cited on page
M. Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Neural Information Pro- cessing Systems, 2013. (Cited on page
Erin Doxsey- Whitfield, Kytt MacManus, Susana B. Adamo, Linda Pistolesi, John Squires, Olena Borkovska, and Sandra R. Baptista. Taking advantage of the improved availability of census data: A first look at the gridded population of the world, version 4. Papers in Applied Geography, :23) 234, 2015. doi:
10.1080 10.1080/28754911 20151111 URL https://dol wtg/1...100/2374493333 2015//4593112111.... 2015. 1014272 (Cited on
pages and 32
J. Fan, A. Taghvaei, and Y. Chen. Scalable computations of wasserstein barycenter via input convex neural networks. arviv preprint 04X4::2007.0442,, 2020. (Cited on pages and 7
Jiaojiao Fan and David Alvarez-Melis. Generating synthetic datasets by interpolating along generalized geodesics. In NewrIPS 2022 Workshop on Synthetic Data for Empowering ML Research, 2022. (Cited on page
Jiaojiao Fan, Isabel Haasler, Johan Karlsson, and Yongxin Chen. On the complexity of the optimal transport problem with graph-structured cost, In International Conference on Artificial Intelligence and Statistics, pp. 9147-9165. PMLR, 2022a. (Cited on page
Jiaojiao Fan, Qinsheng Zhang, Amirhossein Taghvaei, and Yongxin Chen. Variational Wasserstein gradient flow. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 6185-6215 PMLR, 17-23 Jul 2022b. URL https::/
mlr. press/v162/faa22dd html (Ciled on pages and 27
Jean Feydy, Alexis Glaunès, Benjamin Charlier, and Michael Bronstein. Fast geometric learning with sym- bolic matrices. Advances in Neural Information Processing Systems, 33:14448 14462, 2020. (Cited on page
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Cham- bon, Lactitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamoniy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python opti- mal transport. Journal of Machine Learning Research, 22(78): 8, 2021. URL htttp://jmmr http:///mmnr..ggppprrreeee org/papers/ v22/20-451. html (Cited on pages 33 and 36
Milena Gazdieva, Litu Rout, Alexander Korotin, Alexander Filippov, and Evgeny Burnaev. Unpaired image super-rcsolution with optimal transport maps. arXiv preprint arXiv:2202 011116 2022. (Cited on pages
22 and 29
A. Genevay, M. Cuturi, G. Poyré, and F. Bach. Stochastic optimization for large-scale optimal transport. In Advances in neural information processing systems, pp. 3440 3448, 2016. (Cited on page
A. Genevay, G. Peyré, and M. Cuturi. Learning generative models with sinkhorn divergences. International Conference on Artificial Intelligence and Stalistics, pp. 1608 1617, 2018. (Cited on page
Ishaan Gulrajani, Faruk Ahmod, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017. (Cited on page
14

---

Published in Transactions on Machine Learning Research (07/2023)
I. Haasler, R. Singh, Q. Zhang, J. Karlsson, and Y. Chen. Multi-marginal optimal transport and probabilistic graphical models. In aixiv preprint arx:::0006 14113, 2020. (Cited on page 1
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural infor- mation processing systems, 30, 2017. (Cited on page 10)
Man-Christian Hütter and Philippe Rigollet. Minimax estimation of smooth optimal transport maps, 2020.
(Cited on pages 6 and 20
D. Inoue, Y. Ito, and H. Yoshida. Optimal transport-based coverage control for swarm robot systems: Generalization of the voronoi tessellation-based method. JEFE Control Systems Letters, 5(4):1483-1488) 2020. (Cited on page !
Ce Ju and Cuntai Guan. Deep optimal transport on spd manifolds for domain adaptation. arXiv preprint arXiv:2201.057 2022. (Cited on page 13
HM Dipu Kabir, Moloud Abdar, Abbas Khosravi, Seyed Mohammad Jafar Jalali, Amir F Atiya, Saeid Nahavandi, and Dipti Srinivasan. Spinalnet: Deep neural network with gradual input. IEEE Transactions on Artificial Intelligence, 2022. (Cited on page 10
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. (Cited on pages 33 and 36
A. Korotin, L. Li, A. Genevay, J. Solomon, A. Filippov, and E. Burnaev. Do neural optimal transport solvers work? continuous wasserstein-2 benchmark. In arXiv preprint Xrx:v 10.220000000 01 954, 2021a. (Cited on page
Alexander Korotin, Vage Egiazarian, Arip Asadulaev, Alexander Safin, and Evgeny Burnaev. Wasserstein-
2 gencrative networks. In International Conference on Learning Representations, 2021b. URL https:
EXsal (Cited on page 7
Alexander Korotin, Lingxiao Li, Justin Solomon, and Evgeny Burnaev. Continuous wasserstein-2 barycenter estimation without minimax optimization. ar preprint arXiv:2102.01752 2021c. (Cited on page 7
Alexander Korotin, Vage Egiazarian, Lingxiao Li, and Evgeny Burnaev. Wasserstein iterative networks for barycenter estimation. arXiv preprint 2rXiv:2201.12245 2022a. (Cited on pages 7 and 27
Alexander Korotin, Daniil Selikhanovych, and Evgeny Burnaev. Neural optimal transport. arXiv preprint 1XXi:22011112200 2022b. (Cited on pages and 21
V. Krishnan and S. Martínez. Distributed optimal transport for the deployment of swarms. IEEE Conference on Decision and Control (CDC), pp. 1583-4588 2018. (Cited on page
Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. 2005. (Cited on page 10
R. Li, X. Ye, H. Zhou, and H. Zha. Learning to match via inverse optimal transport. In J. Mach. Learn. Res., pp. 20, pp.80-1, 2019. (Cited on pages and
Huidong Liu, Xianfeng Gu, and Dimitris Samaras. Wasserstein gan with quadratic transport cost. Proceedings of the IEEE/CVF international conference on computer vision, pp. 4832-4841, 2019. (Cited
on page 36
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. Tn Proceedings of the IEEE international conference on computer vision, pp. 3730 3738, 2015. (Cited on
page
S. Ma, S. Liu, H. Zha, and H. Zhou. Learning stochastic behaviour of aggregate data. In arXio preprint arXin:2002.05511 2020. (Cited on page
15

---

Published in Transactions on Machine Learning Research (07/2023)
A. Makkuva, A.and Taghvaci, S. Oh, and J. Lec. Optimal transport mapping via input convex neural networks. In International Conference on Machine Learning, pp. 6672-662260 6681, 2020. (Cited on pages G and
Tudor Manole, Sivaraman Balakrishnan, Jonathan Niles- Weed, and Larry Wasserman. Plugin estimation of smooth optimal transport maps, 2022. (Cited on page 20))
Charles H Martin and Michael W Mahoney. Implicit sell-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. J. Mach. Leorn. Res., 22(165):1-73, 2021.
(Cited on page
Robert J McCann and Brendan Pass. Optimal transportation between unequal dimensions. Archive for Rational Mechanics and Analysis, 238(3):1475-1552, 2020. (Cited on page
Cheng Meng, Yuan Ke, Jingyi Zhang, Mengrui Zhang, Wenxuan Zhong, and Ping Ma. Large-scale optimal transport map estimation using projection pursuit. Advances in Neural Information Processing Systems, 32, 2019. (Cited on page
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. arXiv preprint 1rXiv:8802.05631 2018. (Cited on page 36
Petr Mokrov, Alexander Korotin, Lingxiao Li, Aude Genevay, Justin Solomon, and Evgeny Burnaev. Large- scale wasserstein gradient flows. In Fifth Conference on Neural Information Processing Systems, 2021. URL https://opperreview.cee wwtp:/iiiieeet//rrooe wetp://eww...tttoooommmmmmmmllia
(Cited on page !
John C Nash. The (dantzig) simplex method for lincar programming. Computing in Science & Engineering. 2(1):29 31, 2000. (Cited on page 7
Khai Nguyen and Nhat Ho. Amortized projection optimization for sliced wasserstein generative models. arXiv preprint 2rXii:2203.134111 2022. (Cited on page !
Quan Hoang Nhan Dam, Trung Le, Tu Dinh Nguyen, Hung Bui, and Dinh Phung. Threeplayer wasscrstein gan via amortised duality. In Proc. of the 28th Int. Joint Conf. on Artificial Intelligence (IJCAI), pp. 1-11, 2019. (Cited on page
Anton Obukhov, Maximilian Seitzer, Po- Wei Wu, Semen Zhydenko, Jonathan Kyl, and Elvis Yu-Jing Lin. High-fidelity performance metrics for generative models in pytorch, 2020. URL https://githe: com/ toshas/torch-fice fidelitty Version: 0.3.0, DOI: 10.5281 25281/zenodo.co 4957738. (Cited on page
Brendan Pass. Regularity of optimal transportation between spaces with different dimensions. arXiv preprint arXiv:1008.11 1544, 2010. (Cited page
Michaël Perrot, Nicolas Courty, Rémi Flamary, and Amaur Habrard. Mapping estimation for discrete optimal transport. Advances in Neural Information Processing Systems, 29, 2016. (Cited on pages 9
12 and 36
Aram-Alexandre Pooladian and Jonathan Niles- Weed. Entropic estimation of optimal transport maps. arXiv preprint XrXiv:2100. 1 12004, 2021. (Cited on page
Aldo Pratelli. On the equality between monge's infimum and kantorovich's minimum in optimal mass transportation. Annales de ('Institut Henri Poincare (B) Probability and Statistics, 43(1):1 13, 2007. ISSN 0246-0203. doi:
URL https::/ww..ccinceedii.icc
(Cited on pages and 19
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR,
2021. (Cited pagel) 8
16

---

Published in Transactions on Machine Learning Research (07 (203))
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Nya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821 8831. PMLR, 2021. (Cited on page
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXi::2204.061 061022 2022. (Cited page S
R. Tyrrell Rockafellar. Integral functionals, normal integrands and measurable selections. In Jean Pierre Gossez, Enrique José Lami Dozo, Jean Mawhin, and Lucien Waclbroeck (eds.), Nonlinear Operators and the Calculus of Varialions, pp- 157 207, Berlin, Heidelberg, 1976. Springer Berlin Heidelberg. ISBN
978-3-540-38075-77 (Cited on page 21
R. Tyrrell Rockafellar and Roger J-B Wets. Variational analysis, volume 317. Springer Science Business Media, 2009. (Cited on page 21
Litu Rout, Alexander Korotin, and Evgeny Burnaev. Generative modeling with optimal transport maps. In International Conference on Learning Represenlations, 2022. URL httpaattpp//ppeemiiiieeeee////// wttpr//rwwee/oommmmiiiii
5JdLZg346L# (Cited on pages
and 36
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to- image diffusion models with deep language understanding. arXiv preprint arXiv:2905 2022. (Cited
on page
Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 55(58-63):94, 2015. (Cited on page 37
V. Seguy, B.B. Damodaran, R. Flamary, Rolet Courty, N., A., and M. Blondel. Largo-scale optimal transport and mapping estimation. arXiv preprint Xrxi::171 1. 02228,, 2017. (Cited on page
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56): 1929-1958, 2014. URL http://jmlr http:////oooorr//////////// wwwppapers/////ooootoot..t..esst paappers/v15/sree wwt//rraastaaallaaaa htmml (Cited on page 33
C. Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008 (Cited on pages
6 and 20
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmist: novel image dataset for benchmarking
machine learning algorithms. arXiv preprint Nii::7008.0747, 2017. (Cited on page 10
Y. Xie, X. Wang, R. Wang, and H. Zha. A fast proximal point method for computing exact wasserstein distance. In Uncerlainly in Arlificial Intelligence, pp. 433 453, 2020. (Cited on page
Karren D Yang and Caroline Chler. Scalable unbalanced optimal transport using generalive adversarial networks. arxiv preprint arXiv:1810.11447, 2018. (Cited on page !)
Yu Zeng, Zhe Lin, Huchuan Lu, and Vishal M Patel. Cr-fill: Generative image inpainting with auxiliary contextual reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14164 14173, 2021. (Cited on page 10
Peiyuan Zhang, Jingzhao Zhang, and Suvrit Sra. Minimax in geodesic metric spaces: Sion's theorem and algorithms. Xiv preprint ar arXiv:2202.06950 2022. (Cited on page
Jiacheng Zhu, Aritra Guha, Dat Do Mengdi Xu, XuanLong Nguyen, and Ding Zhao. Functional optimal transport: map estimation and domain adaptation for functional data. ar preprint orXiv:2102.0888) 2021. (Cited page
17

---

Published in Transactions on Machine Learning Research (07 (07/202) 2023)
Contents
Introduction
1
2
A bricf background of OT problems
3
3 Proposed method
4
Error Estimation via Duality Gaps
5
Related work
7
6
Experiments
7
6.1
Unpaired text to image generation
8
6.2
Class-reserving map
9
6.3
Unpaired image inpainting
Population transportation on the sphere
12
Conclusion and Discussion
13
A Backgrounds of optimal transport problem with general cost function
19
B Proof of theoretical results
21
B. Proof of Theorem 2
21
B.2 Proof of Theorem
22
B.3 Consistency in optimal values
23
Prool of Theorem
24
B.5 Further analysis X(-) and ((.)
27
C Relation between our method and generative adversarial networks
29
D Additional results
29
E Implementation details and hyper-parameters
33
E.1 Synthetic datasets
................................
33
E.2 Text to image generation
................................
33
E.3 Unpaired inpainting
36
E,
Class preserving mapping
36
F
heoretical justification of all the examples considered in this article
37
Notations used in the paper
38
18

---

Published in Transactions on Machine Learning Research (07/2023)
A Backgrounds of optimal transport problem with general cost function
We present some fundamental results of OT problem with general cost in this section. Most of the results are collected and summarized from Villani (2008).
Suppose M is a smooth, complete connected Riemmannian manifold. In most of our discussions, we treat M as the Euclidean space R for the sake of convenience. Let us denote X as a closed subset of M. Thus X is Polish space, meaning X' is separable complete, metric space. We assume Y general Polish space. Suppose the cost function is defined on X' x V. Suppose X,D) are equipped with their Borel o-algebra. Denote (X), ())) as the space of Borel probability mcasures on X.Y. We mainly focus on the Monge problem from Pa (X) to E (V) with cost function c.
Before the detailed discussion, we list some useful conditions that may be used later.
A) e is bounded below, Le., there exists E (2), I. ()), s.t. c(x,y) 2 a(x) bly) on X y;
(15)
B) is locally Lipschitz function of int( locally in y;
(16)
C) c is everywhere superdifferentiaabe as a function of æ E int(A), for all y;
(17)
D) For arbitrary x E (2), V=c(x,-)) y + R,, is injective, i.e. suppose for x, y. Vac(z, v) = v',,z,)), then y = y//
(18)
Here denote (x' as the interior of the set X, i.e., (x' is the union of all open sets in M contained in X. And we define the local Lipschitz property and superdifferentiabiity as follows.
Definition 1 (Locally Lipschitz). Let U c I= be open and let f : R² + R be given. Then (1) f is Lipschitz if there erists < oo such that
E If(z) f(x) s L|r 2 2/-
(2) f is said to be locally Lipschitz if for any To E RR", there is neighbourhood of xn in which f is Lipschitz. Definition 2 (Superdiferretiaabitt)). f Rn - IR supertifferentiable at x, with supergradient p, if
f(z) S fx) + (p, -0) x) + o(|z xz).
Here the little-o notation indicates that lim
o||=-1|) z- 0.
The relation between optimal values of Monge problem is studied in Pratelli (2007) fc.f. Bibliographical notes of Chap 4 in Villani (2008)). We summarize the result below.
I and its relaxation, namely, the general OT problem
Theorem 5. Suppose X, y are two Polish spaces; Pas Ps are Borel probability distributions on X,Y. Suppose Pa is atomless (t.e. no single point carries a positive mass). Assume further is continuous and satisfies Then (Monge(Pas Pb) C(pa: Pb).
15
Recall that the Kantorovich dual problem of the primal OT problem 2 is formulated as
{ o(u)p(()) dy f ((z))a(T)) do
K(Pa>Pb) =
sup ()) e C2(2) Cal)) *(v) v(x) c(2 x.
ey
We denote the optimal value of this dual problem as K ((pa.Pa)
It is not hard to tell that the above dual problem is equivalent any of the following two problems
U V (v)po()) sup (EC.(a)
(ort))))))))))))))))))) dy - S (()))(()) de)
(19)
A topological space is separable it contains countable dense Since R separable, subset equipped with the subspace topology also separable.
19

---

Published in Transactions on Machine Learning Research (07/2023)
Us )u)esss) dy / (x)())))))) dx (EC+())
sup
(20)
Here the c-transform vet, 0. are defined via infimum/supremum convolution as:
we+ = inf((()) + c)x,))); 6 (x) = sup(ó(v) c(z, v)).
(21)
(22)
If we denote vi., Φ. as the optimal solutions
19
20 respectively. Then both (v., + and (o_ .6.)
are the optimal solutions to (3); v o² are also solutions to 20
19 respectively.
Remark 2 (c.f. Theorem 5.10 of Villani (008)). It is worth mentioning that one can relax the Cx))), Cx())
spaces in
19 and 20 to L (X) and L po):
The following theorem states the equivalent relationship between the primal OT problem 2 and its Kan- torovich dual (3). The proof of a more general version can be found in Theorem 5.10 of Villani (0008).
Theorem 6 (Kantorovich Duality). Suppose N are two Polish spaces. Suppose is a cost function defoned on X y, c. is continuous and satisfies (15 Recall that C(pa: Pb) denotes the infimuum value of ( and
((pa, Pb) denotes the marimum value of (3 (or equivalently. 19
20)).
Then C(pas pb) = K (Pox Pb).
Furthermore, if we assume that e is bounded from above by
c(x v) S u(xx) + v(y), with u E U
(23)
Then the optimal transport plan ne for 2 and optimal potential da for 20 both exist.
The following result states the existence and uniqueness of the optimal solution to the Monge problem. It also reveals the relation between the optimal map T. of (1) and the optinal transport plan n. of (2). is a simplified version of Theorem 10.28 combined with Theorem 10.26 taken from Villani (0008).
Theorem 1 (Existence, uniqueness and characterization of the optimal Monge map). Assume X,Y, Po: Pa: are defined as in the beginning of section 2 Assume the dimension of boundary 0X is no larger than n 1. Suppose the cost с is continuous and satisfies 15 16), [7], (18). we further assume that Pa and pb are compactly supported and Pa absolutely continuous with respect to the Lebesgue measure on RR. Then there ewists unique-in--ee in-law transport map T. solving the Monge problem (1. And the joint distribution (Id, T. PPP on Ruxm is the optimal transport plan of the general OT problem Furthernore, there erists a c-conved""
th., which can be proved to be differentiable Pa. almost surel and (1) + VaclE, (x(x)) = 0, Po. almost surely.
Remark 3. The differentiable property of c-convez vh. a direct result of Theorem 10.26 from Villand 2008).
Remark 4. The condition C(pa Pb) < +00 in Theorem 10.28 of Villand 2008) is automatically satisfied since we assume that Pa. are compactly supported and the cosl e is continuous. It is also worth mentioning that the conditions in Theorem 1 can be modified and we can still obtain the existence and uniqueness of the Monge map. We refer the readers to Remark 10.33 Villand (2008) for more details.
Remark 5 (Discussion on regularity of T., V.). From the above theorem, one can tell that the regularity of the Monge map T, is closely related to the regularity of th,, which is a very important topic in OT and the Monge-Amppre equation associated to the OT problem. We refer the reader to Theorem 12.50, 12.51 and 12.52 Villani (2005) for details. Furthermore, in recent research such as Hütter & Rigollet (2020), Manole et al. 2022). stronger Ck.o regularity on v.. were used to establish bounds on mindmaz rates for certain types of estimators under the statistical selting. In our poper, we did not consider the aforementioned statistical problem, so we omit the detailed discussion on the regularities of both T, and tha.
C..F. the definition in Assumption
79p is differentiable at x for all Spi(pa) \ Fo: where Fi a zero measure set.
20

---

Published in Transactions on Machine Learning Research (07 2022))
B Proof of theoretical results
We first define the saddle point of f) to the sup-inf problem 4 as following.
Definition 3 (Saddle point). Suppose Y ore two Polish spaces. We say (f,f) is a saddle point of the
Junctional f) f' to the sup-inf problem defined in 5 if (f, salisfy
T E argmin (C(T, TEM(X.))
f argmax {C(T, fEC(()
Before we introduce the main results, we present useful lemma that enables one switch the optimization function parameters and the integral. Such result corollary of Theorem 14.60 in Rockafellar & Wets
2009
(also e.f. Theorem 3A in Rockafellar 1176 and it is also used in (Korotin et al. 2022b Gazdieva
ct al. 2022).
Lemma 1. Let : x R. - R is continuous function. Let μ be a Borel measure defined on RR. Denote M(R", RT) as the space of all measurable functions T RR" + Rmm. Then we have
[ f(z, [())((=))))
[ inf f(z du(z). SERm
inf TEM(R" RT)
B.1 Proof of Theorem 2
Proof. (1 + 2) First denote T, as a solution to 1 and de as a solution to 20 We prove (Te, ø.) is the saddle point f). We first prove T. argmin TEM(X.))
By using lemma
one gets
/ o (v)
o (v))o(v))d / 6- (z)
inf ((T = --
Since d. solves 220p, the right-hand side of the above equals K(pa, Pt). Now by Theorem one obtain (Pa Pb) = C(pa: Ph) = K (pas ph), which leads to the strong duality of such Monge problem. Thus we have o.)} = OMouge (Pa, Pb). Since T. solves [ we have
inf (n(( = -J (I,T,(s))P,())d L(T.,
The second equality above is due to the constraint T.pPa = Pb- This proves T. E argmin {C(T, d.))- TEM(X.)
The proof of o. E argmaa((C(T., 6)} is straightforward, since L(T.,0) = Jx (t)) is indepen- dent of á.
(EC,1))
Thus we have proved that (T, is saddle of C(T.f).
(2 + 1) Let us first prove that T an optimal solution to Monge problem I T:po = pb. Suppose this not true, then one can find a function Cb()) such that Jy (x))))) A Sy ((v))o(())d)) Without loss of generality, assume Jy (V)))(())d - Jx test function upp Ci))") with E R as an arbitrary real number, then consider
We first claim that
0. We pick the
c(f, / clx, (2))) + vl plu)po(o)dy- - ((((2)) (x) 2 c pd.
(24)
denote 8>0
Recall 7 E argmin(((T)) f)) use Lemma 1 then we have
C(T,)- = inf / fwipotlldt / fa- S K(pas C(pa, Pb) < +00.
(25)
8rThis is due to the fact that when y a metric space, and pas E (P)) (Borel measure), if Sy dpa Sy dn E Cx(D)ss then pa = pb-
21

---

Published in Transactions on Machine Learning Research (07/2023)
Combine 24 and (5),, one obtain C(T, f) L(T,, vp) I C(pas pb) - vò. We can increase v large enough so that C(T, ((T, This contradiction to the fact that f argmax([( f)}. Thus we have SEC,())
proved that Typa = Pb- As a result,
inf / (t) TePo=Po J (() C(f, f) inf( S inf (C(T,
Topu=Po
This indicates that I solves the Monge problem (Pongo Pb).
The above calculation also leads to L(T,
At last, prove that f is the optimal solution Kantorovich dual problem 20 By applying Lemma just notice that for any E Co()), one can verify
c(T,N) 2 C(F, 2 igfte(T, = SS (lo)oolldd - / - (x)ps(u)dy
On the other hand, we have
f(o)poludd - -n JWl 1 jar (c)(())o())dg
This proves that f is the optimal solution to (20).
A simialr proof of T. E argmin {C(T, o.)3 as part of our proof to Theorem also established in section
TEM(X.))
6.1 of Gazdieva el al. 2022
To prove the Corollary I
it suffices to check the conditions in this corollary imply the conditions in Theorem
B.2 Proof of Theorem 3
Proof- As mentioned before in the proof of Theorem 2 by Lemmn I one can verify
inf L(T, J, f()) / f (x))a(x)dd,,
(26)
T
Then the sup-inf problem sup, inf, L(T,f) can be formulated as
sup { Flieeassdy - (W))aola
This is exactly the Kantorovich dual problem 20 Since (T is the solution to sup-inf problem, f is an optimal solution to (20 This verifies the first assertion of the theorem.
On the other hand, at (T,f), we have
T(x) (f( (), Pa almost surely.
This leads
[o))))= c(., (xx)), Pa almost surely.
Then we have
[ c(xxT(x))p dr f(T()))pa(z) dr / fo- (x-(x))a(x) de
[ (W)ppa dy je- (x)pp()) dr
-Ihr (F(u) (x)()))((z), S I clx, (dx..))
22

---

Published in Transactions on Machine Learning Research (07/2023)
for any E Pb). Here the second equality is due to the assumption TatPa pb. The last incquality is due to the definition of (x) = sup,(((v) c(x, v))-
We now take the infimuum value of J/xxy cdT and we obtain
Jx cx, (x))(=))) de S C(Pa.P)).
(27)
Now, for any transport map T' satisfying Tipo pb, by denoting (Id, T) we have
f c(,, (()))(()) dx= = 11 cIz (d,.))d((2 v) 2 C(pa. Pb).
(28)
Combining 27
and 28
we obtain
J
ca, (())) S / c(x, ((z))()))aa for any T, T:Po Pb.
x
This indicates the existence of the Monge map, and T is the Monge map. At last, we have
c(T,f) / cd,, (x.:(t))pd())d+++ /, ((T)(T - pb)dy / ((x){(=)))((x)) de CMonge (Pa: Pb).
B.3 Consistency in optimal values
As discussed in Theorem we require the condition TyPa = Ps to establish the consistency between the sup-inf solution and the optimal transport solution. However, even without the assumption Typa = Pb: one can directly prove the consistency in optimal value, i.e., sup, inf, L(T, = C'pa. Pt). This is demonstrated in the following theorem.
Theorem 7 (Consistency in optimal value). Assume the cost function is continuous and satisfies the con- Suppose the sup-inf problem 44 admits at least one solution (f,)), then L(T, = C( Pt).
dition
15
Proof. The proof of the first part of this theorem is the same Theorem 3 Thus it is not hard to verify
sup inf L(T, sup { f())pol)dy - S 1 (x)((()(()))))) K(pa, Pb).
T
Since c(x,y) y) satisfies
15 recall Theorem G the optimal value of Kantorovich dual problem equals general OT distance C(pa- Po). This proves our asscrtion.
Theorem 7 indicates that although for some general cases in which our proposed mothod 4 fails to computo for a valid transport map T, ( is still able to recover the exact optimal transport distance C(par Pt). the following example, although the Monge map does not exist, the proposed method can still capture the exact OT distance.
Example 1. Consider the OT problem on It with c(x, v) = |x - yy and pa = Sn (point distribution at 0), Ps = N 1) (normal distribution). Our method yields
sup inf { 10 10-T(0)) T(0)
/ ((0))o()dd (
23

---

To prove this lemma, we Published in Transactions on Machine Learning Research (07 2023)
By setting Mo(y) f(y) -lv1?, sup-inf problem becomes
| ( 1w1) Hoy)dyy (f(I(()) - -10 3
sup inf
Mo(u)
Vo(2 (0))
/
(Volu) + (o))dyy (ood:)ddy-oo((OO) vo((()))) = / pu()pdyyddy sup Wo
sup in!
(Volv) sup V Woles(y)dy =1
Wo
The supreme over Vo is obtained when Vo Const, i.e., when f(0) 1yl + Const. Thus have supy infT L(T, C(p,,Pb).
B.4 Proof of Theorem 4
We firstly verify the following lemma.
Lemma 2 (Existence and uniqueness of Monge map under Assumption Suppose Assumplion and hold, then the Monge map T. erists and unique in law.
Lemma a corollary of Theorem Once assumption and hold, the conditions on c(.-)) and Pa: Pb further need the following two lemmas.
mentioned in Theorem are all satislied. To prove Theorem
Lemma Suppose n x n matria A invertūhle with minimam singular value Fminn (A) > Also assume matriz H is self-adjoint and satisfies Aln 2 H 0 Then TH-AAY amin(A) In.
Proof of Lemma One can first verify that H-1 = In by digonalizing H-1 only need to verily that for arbitrary TR
Aoo?
mmin (A)2
- (Av) H (AA Av
|
Thus A Gmnn (A)² In is positive-semidefniit.
The following lemma is crucial since it analyzes the concavity of the target function f(-) cl-,3) when f is c-concave.
Lemma 4 (Concavity of f(-) c(z.-) )) if ff c-concave). Suppose the cost function c(a,, y) and f satisfy the conditions mentioned in Theorem 4 Denole the function vz(y)) f(v) - c(x,1) v), then we have
(2 o(x y)² I.. A(y)
Proof of Lemma First, notice that is e-convex, thus, there exists & such that f(y) inf (nf(()())) c(x, 3) Let us also denote ( v) x0x) + c(x,
Now for a fixed E R", We pick one
Ty E argmin (p(xx) + c(x.
Since we assumed that &o E (R") and ce CR t R"), have
V ( v) V V V2c(Ky, v)
(29)
2 $ (Ig. y) is positive definite, thus is also invertible. We can now apply the implicit function theorem to show that the equation V, v) determines an implicit function x(-),
Now recall Assumption
3
Here matrix Mj M3 Mj positive-definite matrix, and M1 M1 M2 positive-emmidefnite matrix.
24

---

Published in Transactions on Machine Learning Research (07 (07/2023)
which satisfies z(y) Ty in small neighborhood U R" containing y. Furthermore, show that xr()) is continuously differentiable at We will denote By as s(y) in our following discussion.
Now differentiating
29
with respect to yields
V (x))((())) + ( = 0.
(30)
On one hand,
30 tells us
Vz(y) = - (
(31)
Now we directly compute In order to compute 42 first compute V
(v) V ((r,))
(32)
Vf(y) ((p((()) + ((x(v)))) ()) V,c(x(y),
(33)
the second equality is due to the envelope theorem
Afriat 1971 Then f2f(v) can be computed as
f2y()) = V v) V ()(((()))))
(34)
Plugging
31 into
34
recall
132
this yields
cg Vi ((r))) y). V clx(((())))) v) + ((x()) (4t
Recall the Assumption
notice that C C² )" by symmetry of second derivatives, one can verify wy cl, v) is independent of x. has V c( (I, 0. Thus obtain
2
V2
nn C; Since
v (v) - V ((r())
(a(())
(35)
By the positive definite assumption made in Assumption 3 as well as have X V2 ((x()))))
v)
On Recall that min V2 e(z, v)) v). Now apply lemma to 35
this yields
( 3 o(x 8)
In-
A(v)
Now we prove the main result of Theorem
Proof of Theorem In this proof, we denote as Sax for simplicity.
We first recall
C(T,F) / clx, (()))()) dr / ((u)po(()) dy ((T(x)))o()) dr / f(v)po(v) dy - - - c(z, ((x)))aa())dd
then we write
- --fT)=--(((--))------ - clx, c(c,. T(x))lpa dr + sup
((r -(( T<x))e dr
(TT = C(T,)) inf C(T,
T
Recall Ty defined in Assumption have, for almost every E Rdd
(x) argmax, (s(r) v)) argmax,, (v))
(36)
25

---

Published in Transactions on Machine Learning Research (07/2023)
recall that we denote (z( f(y) c(x, v), then, for almost every RR, have
VU O.
(37)
One can also write:
(T. / (cx, ((((aa))))(((()))))))((((()))))) c(a, T())))
/ [Ix)(T(())) WI.TT=V)( dr
For any Rd, since (-()) (RR), and according to the previous Lemma have
- v -T)))(((())))))))))))) + 2 - v
with
w(x) (x) + (xT
(38)
for certain 0g E 10. By (37 and Lomma
we have
- ( < v(),w(x))² 22(x(x))
((x)) - T-
Thus we have:
(T. (V))(T(()) - (=(((=)))laaa) dx 2
{ (T(x) [Ty(x)) pa(x) dr
v(),w(x))) 2\(w(x))
(39)
On the other hand, recall a solution of our sup-inf problem
4 is denoted as (T, Then we have
sup inf C(T,)) = f dx. (x)))a dr / [(w)(Tpa - pb)dy = / (xx,((x)) dr, T
the second equality is due to the assumption on the consistency between T and Monge map T., thus T:po = Pb. Thus we have
EC) = / clx, (x))pa do inf (ʃ (foo, T(x)))a(a)dr + / f(v)pa(e) dy - / ((T(6))) 7
=- ((T(x)) - cxx, (()))a dr + sup / ((f((E)) c(z, (())
The second equality is due to T4pa Pb. Similar to the previous treatment, we have
/IV.(T((s)) v. (x()))))) dr
E2(
Apply similar analysis as before, we obtain
Exf) /
((x)2 2A(E(z))
(T(x) - (T) dr
(40)
with
= (2)
(41)
26

---

Published in Transactions on Machine Learning Research (07/2023)
for certain Tu [0, 1]. Since T = Tx, Pa almost surely, 40 leads to
c(x,((x))²
E())
2X(E(x)) (T.(x) (a(x)) de
(42)
Now we set
(x)) a(x,5(a))
B(x) = min
(43)
2M(w(x)) 21(E(x))
where and are defined in 38 and
41
Combining
39 and 42
obtain
((x)((((=)-T) (T)() + (T. - (1(x))1)pp da
((x) (T(x) (x) Pa dx
2
This leads I T.11 & (2(
Discussion on enforcing the c-concave property of the dual variable Theorem 4 is established the assumption of e-concavity of the dual variable f. In practice, the c-concave property be approximated via soft-min convolution, i.c., one can set fo(y) K log (u exp(-K((pp())) + c(X, v)))) with K > 0 large enough, (e an arbitrary neural network and U as a uniform distribution over a sufficiently large ball. However, according to the existing work (Fan et al. 2022b Korotin et al. 2022a on applying ICNN to solve OT, restricting f to the c-concave functions may undermine the performance of our computation. The related implementation and comparison may serve as good directions for future rescarch.
B.5 Further analysis on A(-) and B(-)
We can carry out some further calculations so that we may gain a better understanding of the value A(y) evaluated at y (xx). We can then tell that the quantity 8 defined in (43) will be impossible be bounded from below as the maps T, T'y are very close to the sup-inf solution T. Before our calculation, we need the following lemma.
Lemma 5. We assume the assumplions mentioned in Theorem 4 all hold. We also assume that for the
cost function the map vyc(;,y) : + Vye(z, y) is injective. We further strengthen Assumption 4 such
that (x) minimizes f(y) c(x, y) for all E Rdd Similarly, we denote map T TT( z(y) e
arg min, (x(()) + u)} for all E Rd. Then we have (T(T(()) y for all y.
Proof. To prove this, we notice that v,(s()) clx, ))))))(()))))) = 0. This yields
V)(T(()) V ye (x))) 0, for arbitrary E Rd. By similar argument carried out in the proof of Lemma
(44)
able to show that
Vf(x) = V VV:==V,cTTTo ())(T(v))v),
(45)
for all y E Rd. Then, for any we can set y y= (x) in 45 and plug this into 44 to obtain
(T(T,((T(E))) - ,,c(E, T()) =0
(46)
Now due to the fact that cfy y) is invertible in Assumption [ one can tell that the map Vyc(.,) : Vzc(x, y) is injective. Thus (46) yields (z) = for all x E Rd.
+
Remark 6 (T,, T o as inverse map to cach other). If we further assume that yp c-conver, i.c., pp())
supylf(y) c(z, v)}. Then by similar aryument, we can also prove that (T)()y)) = y for all y E Rd. Thus, Ty and To are inverse map to each other.
27

---

Published in Transactions on Machine Learning Research (07/2023)
Theorem 8 (Estimation on X(+) at Ty(x)). Suppose assume all the assumptions in Lemma 5 Let T
defined in Assumption 4 and X(-) defined in 7 Denole r= c2 Ty(x)). Then we have X(Ty(x))
( Furthermore, X(Ty(x)) 2 (T)]ID)
Proof. Let us recall that we denote Ty(x) = arg max, (f(v) - cla, v)} = arg m (v
Using the
optimality condition, we obtain (T)(T)(z)) = O.
Vyclz, VI(T)(<))
(47)
Let us now differentiate 47
W.r.t. T.
0,
which can also be formulated as
V cl, V V (1)
(48)
Let us recall equation 35 and the definition of Top substitute 35
into 48 to obtain
my (x,
0.
(49)
Recall that the previous Lemma yields T. (T) for all Rdd Let us denote Vay [x for simplicity. Then equation 49 then leads to
(50)
Since V2 + is self-adjoint, then Amax (2(T(((())) (r) (T 1l. We can bound this matrix 2-norm
[TDT, (x) min DT))
(D
Z
Thus, when y equals to the minimizer (x(2) e arg (f()) c1., y)}, the quantity A(y) A(T)(z)) o mmin (+))
Now suppose that method is about to converge to the sup-inf solution, iee. the current minimizer Ty of T(f,T) is vory close to the sup-inf solution T and our computed T is very close to Ty. Then the w(x),{()) mentioned in the proof of Theorem 4 will both be very close to (T),,, thus, 3(x) approximately equals
(xx,T(()) A(Ty(x))
Recall the inequality A(T(())) 2 (()) by Theorem S and miin (I') amin (V2 (ay ((, = (x, (z)) by definition, we obtain that 8(x) approximately bounded from above by [DTT(=)] Notice that we can casily construct Pa with very small support and Ps with large support so that the norm of the gradient of the approxinated OT map (x) is large possible for all supp(p.). Then B(x) can be an arbitrarily small value.
Example Consider quadratic cost c(ar, y) = |e pec-yl y/2 and two uniform distributions on R, Pa U((-,, c) and Ps U[[ L, L)) with c, L > 0. Then the optimal transport map Tx(x) Lx. Since we assume the consistency, T(x) ,x, thus [T =.
In Example suppose T.T are very close T, it is likely that (Ty(x) also possesses large norm for most of the E <]. which leads to very small value of 3(x) < f.
28

---

Published in Transactions on Machine Learning Research (07/2023)
C Relation between our method and generative adversarial networks
Tt worth pointing out that our scheme and Wasserstein Generative Adversarial Networks (WGAN) (Ar- jovsky et al. 2017 are similar in the sense that they are both doing minimization over the generator map and maximization over the discriminator/dus potential. However, there are two main distinctions between them. Such differences are not reflected from the superficial aspects such as the choice of reference distributions Pay but come from the fundamental logic hidden behind the algorithms.
We want to first emphasize that the mechanisms of the two algorithms are different: Typical Wasser- stein GANs (WGAN) are usually formulated as
/ D(y)po(y)dy - D(G(s)
min G
max 10ps1
(51)
--Wasserstei distance Wi (GGPa.pph))
and ours reads
max min / SC)nalldd / -(TW))aaaodddd + [(xxT(()))
(52)
T
general Wasserstein distance (P.
The inner maximization of 51 computes W1 distance via Kantorovich duality and the outer loop minimize the W1 gap between desired Ph and GyPa: However, the logic behind our scheme 52) is different: the inner optimization computes for the c-transform of f, ie. (2) sup: (xx,))); And the outer maximization computes for the Kantorovich dual problem C(Pa: = sup, (J ((y))o())dy f (x)
Even under W, circumstance, one can verify the intrinsic difference between two proposed methods: when setting the cost c(x, | 1rr-1 - and Pa C2p. in (52), the entire 'sup-inf" optimization of 52) (underbraced part) is equivalent to the inner maximization problem of [51] (underbraced part), but not for the entire saddle point scheme.
It is also important to note that WGAN aims to minimize the distance between the generated distribution and the target distribution and the ideal value for (51) is O. On the other hand, one of our goals to estimate the optimal transport distance between the initial distribution Pa and the target distribution Pb. Thus the ideal value for (52) should be C(pa. P6). which is not in most of the cases.
We then argue about the optimality of the computed map G and T: In 51 one is trying to obtain a map G by minimizing Wi(ps GyPa) w.r.t. G, and hopefully, GyPa can approximate Ps well. However, there isn't any restriction exerted on G, thus one can not expect the computed G to be the optimal transport map between Pa and Ps; On the other hand, in (22,, we not only compute T such that Tepa approximates Ps but also compute for the optimal T that minimizes the transport cost E.p. T(X)) In (52). the computation of T naturally incorporated in the sup-inf scheme, and there exists the theoretical result (recall Theorem in the paper) that guarantees T to be the optimal transport map.
In summary, even though the formulation of both algorithms are similar, the designing logic (minimizing distance vs computing distance itself) and the purposes (computing arbitrary pushforward map vs computing the optimal map) of the two methods are distinct. Thus the theoretical and empirical study of GANs cannot be trivially translated to the proposed method. In addition to the above discussions, should also refer the readers to Gazdieva et al. (2022), in which comparison between similar saddle point method and the regularized GANs are made in section 6.2 and summarized in Table 1.
Additional results
OT between 1D manifolds We consider the source distribution be an 1D uniform distribution, and the target incomplete ellipse. In this case, the source dimension 1 and target dimension 2.
29

---

Published in Transactions on Machine Learning Research (07 (07/2023)
As discussed in Section 3 the cost reads
choose the embedding function to be padding-zero function Q(x) [a., 0),
- =(Q()),)) [1 -
As shown in Figure S our algorithm able learn a symmetrie map from an 1D uniform distribution towards the incomplete ellipse. This example also shows that our method can deal with the case that both Pa and Ps are not absolute continuous w.r.t. Lebesgue measure.
(a) Pa
(b) Pb
(c) TaPa
(d) T(-) map
Figure 8: Qualitative results for learning unequal dimension maps.
Decreasing function as the cost We consider the cost function c(z, v) = o( 6(1x=yl)) with d as monotonic decreasing function. We test our algorithm for a specific example 6(s) = 3. In this example, we compute for the optimal Monge map from Pe to Pb with Pa = (g, 6] x [0, 2x)) and = h,U((1.2 x [0, 2n)). Here we define h: IR, [0, 2T) + IR² as the transform from polar coordinates to Cartesian coordinates, i.e. h(r, 0) = (r cos sin 0). We denote (E) as the uniform distribution over the measurable set. E C IR. Let denote the outer annulus as Qg, which is the image of [4, 6] [0, 2 under h; similarly, the inner annulus is denoted as Ωz One can verify that the density function Pa (x) 4m [a] Xn, and pb(x:) 27||a|| Xax Here if x E E and X=(x) = if x & E.
XE is the indicator function measurable set E, ie. X=(ø)
Under polar coordinate, we define the map T, that maps cach point (r, on [4,6] [0, 2T)) to (4 2, (0 + v) mod (2x)) [1,2] 2T). Then the ground truth Monge map is T. h h-1 Ω. - Ωr. Now denote OT distance using the above reciprocal cost as W-2. Then it is not hard to compute W-2(Pa Po) JSn, Pal dx -0" J (4+ dar rdrd0 f (4+))=dr=
We also compute the same problem for I cost, i.c. c(x,y) || y| Under polar coordinate, the corresponding Monge map is T. h o o h-1 where of the problem should map each point (r, on
[4, 6) (0, 2m) to (5-1,00. Similar to the above discussion, Wa(Pa. (1 2m Sa (1 + 5)²² 4 rdrdd
37
Figure
shows the transported samples as well as the differences between the two cost functions.
4
(a)
(b)
(c)
(d)
(e)
Figure 9: Panel (a) shows the samples from computed Tip-- When c(æ, y) 1/1|= y| (panels (b) (c)), we show T(x) maps the circle with radius 6 to the circle with radius 1, and maps the circle with radius 4 to the circle with radius c(z, y) = I| - y|l which is the case of the panels (d) (e), it is exactly opposite.
30

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Uniform distribution sphere We use the geodesic distance sphere 13 Pa (T) x [0. F)) and Ps U ( 2n) dor m]), and consider solving
the cost. We set
/ c((0,)),((,0)) o))p, d0dø.
min T:
(53)
To mitigate the gradient blow-up of arecos near +1, we slightly modify the original cost function as
cx ((0(,, (02, (b)) = arccos(A(sin) Φi sin cb2 cos(0) - 02) + cos Φ1 cos (2))
(54)
with cost cx. We also translate our computed map back to the 3D sphere S via spherical- Carttesian coordinate transform (0, o, r) (r sin cos 0, sin @ sin 0, cos o) where we fix the radius = 3.
with A = 0.999. We apply our algorithm to solve (33
We demonstrate the numerical results in Figure 10 In this example, the ground truth Monge map T, of (53 maps each (0, Φ) on [0, 2T) x [0, 41 to (0.
m). Furthermore, it is not hard to verify that the OT
distance between Pas Pb W geodesic (Pa. Pb)
(a)
(b)
(c)
(d)
Figure 10: Monge map from Pa to Ps on the sphere: (a) blue samples from Pa and orange samples from ps: (b) blue samples from orange samples are obtained from Tips, grey curves are geodesics connecting each transporting pairs; (c) our computed Monge map maps blue ring (ó = {) to the orange curve (ground truth
is & T); ); (d) our computed Monge map maps blue ring (Ø 7) to the orange curve (ground truth the south pole)
Our computational method also yields to an estimation on the OT distance. To be more spocific, for any com- puted Te, we can use Monte Carlo method to estimate the OT distance as W (Pa, Pb) = 인 Σ c(Ta, To(xx)) )), choose N 50000.
where the N samples {a*} are i..d. samples drawn from Pa. In exact computation, We summarize the comparison between estimated OT distances and exact OT distances of previous examples in the following Table 3
W-2 (annulus example)
W2 (annulus example) Figure 9 (d,e)
W grodessi (sphere example)
Figure (b.c)
Figure 10
real distance W our ostimation W
1/42 & 0.0238
/ /3/3 R 3.5119
9 /4 RJ 7.0686
0.0241
3.4019
7.1041
relative error
WW-W W W
1.3149%
3.1325%
0.5018%
Table 3: Comparison between the estimated OT distances and the exact OT distances.
31

---

Published in Transactions on Machine Learning Research (07/202) 2023)
Figure 11: In this figure, we plot N 40000 samples {0k, 0b k== (blue) randomly drawn from Pa and their mapped samples (To (be) N (green).
Population transportation We have described this example in Section 6.4 Here we make further ex- planation on the map-to-land transform T, it is defined as follows.
(), if (0, Φ) E Land; T(0,0) argmin {1((0, Φ) - (0, (0, () Sea. (P.
Here we choose P a finite set consists of 2000 samples randomly selected from p
Sph.
We further plot in Figure 11 with sufficiently large amount of source samples and mapped samples. Among the 40000 mapped samples, 7718 are located on the sea, and apply to map these samples back to a rather close location on land. It is worth mentioning that the map used as the background in our figures has several small regions removed from the actual land. This is due to the lack of data points in the dataset provided in Doxsey- Whitfield et al. (2015).
We also tested our algorithm on this population transport example with different costs. One of the costs we consider the ex cost defined before in (4). with X = 0.99. We can treat Cλ as a reasonable approximation of the exact geodesic distance sphere. The other cost consider is the L cost c((11, Φ1), (02, (2)) (02 - 02 + (Ø2 2 We present the numerical results for both costs in Figure 12 One can tell that there are more transport trajectories passing through the Northern Arctic Circle with cost than the trajectories with L cost. This makes sense since the cx distance of trajectory passing through the North Arctic Circle won't be necessarily large when comparing with L distance. As a result, the transport plan with сλ cost is available accommodate more transport plans going across the Arctic Circle.
Text to image generation We show additional generation results in Figure We also compared the cos-sim cost 9 and the ordinary inner product cost c(z,y) -(Rx, y). The latter converge to the same level as cos-sim cost, measured both qualitatively via generated images and quantitatively via cosine similarity w.r.t. real images (see Figure 4 (b).) The only difference is the method with inner product cost will converge slightly slower. After investigating the statistics of [xx], we find this result is reasonable because |Rx] concentrates around with large probability. Meanwhile, we emphasize that this text-to-image generation task has not been considered in previous neural OT methods, which is itself a contribution, even though using cos-sim cost does not distinguish from inner product cost significantly.
32

---

Published in Transactions on Machine Learning Research (07 (07/202) 2023)
(a) Population transport with cost ex
(b) Population transport with L cost
Figure 12: Comparison of transport plans using different cost functions.
E Implementation details and hyper-parameters
E.1 Synthetic datasets
We follow Algorithm [ and apply Adain method Kingma & Ba (0044) to train both Ta and Sn for all the
following examples this section. Recall the notations B, K, K1, K2 mentioned in Algorithm
OT between 1D manifolds The networks To and Jn each has 5 layers with 10 hidden neurons. The batch size B 100. K1 = =2=1, =
The learning rate 10-3, 3, The number of iterations K 12000.
Decreasing cost. function In this example, we set (x(x) = x + (x() and optimize over For either y- or |zr y/² case we set both I's and the Lagrange multiplier fn as six layers multilayer perceptron (MLP), with PReLU and Tanh activation functions respectively, each layer has 36 nodes. The training batch size 2000. We set K = 8000, K₁ = 8, K2 = 6. The learning rate for both and equals 10-4
Uniform distribution on sphere this example, we set To(x) = x + Fo(x) and optimize over We set both Fg and fn as six layers MLP, with PReLU activation functions, each layer has nodes The training batch size is B 4000. We K 10000, K1 8, K2 4. The learning rate for both A and n equals 0.9 x 10-1
Population transportation In this example, we set To(x) x Fe(x)) and optimize over we choose both fo ResNets with depth equals 4 and hidden dimensions equals 32; choose the activation function as PReLU. We apply dropout technique Srivastava et al. (2014) with p 0.24 to each layer of our networks. The learning rate for both 0 and rj was set as 5 x 10-5 To is computed after K = 200000 steps of optimization with K1 = 8, K2 = and training batchsize B = 400.
For the comparison experiment, we tested our example by slightly modifying the codes presented in OT mapping estimation for domain adaptation of POT library Flamary et al. (202T). We train the linear transformation on 2000 samples from both pSph and ph Sph we choose the hyper parameter μ = 1, = 10 10-4 We plot the second Figure 7 by applying the map-lo-land transform 7 with 2000 newly selected samples. We also tried the same example with Gaussian kernel provided FFammay et al. (2021). but the results are always unstable for various hyper parameters. Generally speaking, it is very hard to obtain valid results with Gaussian kernel.
E.2 Text to image generation
Dataset details Laion aesthetic dataset is filtered from a Laion 5B dataset to have the high aesthetic level. Laion art is the subset of Laion aesthetic and contains 8M most acthetic samples. We download the metadata of Laion art according the instructions of Laion. We then filter only English prompts, and the CLIP retrieval to convert
download the images with fing2dataset. To speed up the training,
33

---

Published in Transactions on Machine Learning Research (07 (0722)23)
2012 ford
barefoot cortessa coconut cake and carton delight decorum coconut cupcaless
g1500 with 1967 Shelby g1500 body.
Free vector backiground of paracise lendscape #2183
Home Builders Small House Tiny Homeis
Feb11-730Guy Wlogins, American, 1893 1962, NYC18000
Furniture Des ddas
Fruits with Blacsoma
Watkins Park User Photo
Pattery
text embedding
Barla
our embedeing
embedding
Image
(a) Random image samples on Laion art prompts
sunroom vintage hame is highly decorated
blue, purple
twisting their
strew black tones.
bern painting artist
tropical beach with palm trees
rustic fermhouse whales.
like this medium length hair style
black diamonds fine chain
errangement.
sun
text embeading
anooz
Cremn BAlles
Rellng
our embedding
real embedding
real image
(b) Random image samples on Conceptual Captions 3M prompts Figure 13: Additional text to image generation results
images to emboddings and use sageddddngddattttt--rrrrreeeeeee reorder the embeddings into the expected format. By doing these two steps, we save the time of calculating embeddings on the fly. After filtering English prompis, we get the Laion art dataset with 2.2M data.
34

---

Published in Transactions on Machine Learning Research (07 (07/202) 2022)
(a) Degraded and original images
(b) Composite images G(z) (c) Pushforward images T()
Figure 14: Unpaired image inpainting on test dataset of CelebA 64 64. We take the composite image G(x) = T()) o M + M as the output image. Additionally, provide the pushforward images T(x) to illustrate the regularization effect of transportation costs. In panels (b) and (c), show the results with a = 10 in the first row and a = 10000 in the second row. A small transportation cost would result that the pushforward map neglects the connection to the unmasked area, which is illustrated by a more clear mask border pushforward images.
Figure 15: Unpaired image inpainting on test dataset of CelebA 128 128, We take the composite image G(x) = T(x) M + M as the output image.
We download CC-3M following this instruction We then remove all the images with watermark, which include images downloaded from shutterstock, alamy, getfyimages, and dailymaul. co.uk websites. After re- moving watermark images, the CC-3M only has 0.8M (text, image) pairs left. We then use CLIP retrieval and waabbelding--dttase--reeooo
packages to convert images /tegss/ texts to embeddings.
For each dataset, we let the source and the target distribution contain 0.3M data respectively, and take the rest of dataset as the test data.
Network structure and hyper-parametors We use the DALLE-2 prior network to represent the map T. The potential f using the same network with an additional pooling layer. We use EMA to stabilize the training of the map. The batch size is 225. The numbers of loop iterations are Ki 10, K₂ 1. We
35

---

Published in Transactions on Machine Learning Research (07/2023)
use the learning rates 10-+, Adam (Kingma & Ba 2014 optimizer with weight decay coefficient 0.0602. We train the wetworks for 110 epochs.
On NVIDIA RIX A6000 (48GB), the training time of each experiment is 21 hours.
E.3 Unpaired inpainting
Denote MC as the complement of M, i.c. Mc = in the occluded area and 0 otherwise. The composite image is G(xr) = (x)) @ M + @ M. The loss function we actually use is slightly different with the (5) We modify the f(T(x)) to be f(G(x)) to strengthen the training of
sup inf T / [c(x, [c(,,((x)) ((G(=))) ((G))) dr [ f(v)p((y) dy.
In the unpaired inpainting experiments, the images are first cropped at the center with size 140 and then resized to 64 x 64 128 x 128. We choose learning rate to be 10-3, Adam optimizer with default beta parameters, K2 1. The batch size is 64 for CelebA64 and 16 for CelebA128. The number of inner loop iteration K 5 for CelebA64 and K1 10 for CelebA128.
We use exactly the same UNet for the map T and convolutional neural network for f as Rout et al. 2022 Table 9) for CelebA64 and add one additional convolutional block in ʃ network for CelebA122 On NVIDIA RTX A6000 (48GB), the training time of CelebA64 experiment is 10 hours and the time of CelebA 128 is 45 hours.
We use the POT implementation Flamary al. 2021 of Perrot et 2016)) as the template and implement the masked MSE loss by ourselves. Perrot al. (2016) provides two options the transformation map, one is linear and another is the kernel function. We find the kernel function this example would generate mode-collapse results, so we adopt the linear transformation map. We choose L2 regularization coefficient to be 10-3, and other parameters are the same as default.
E.4 Class preserving mapping
The NIST images are rescaled to size 32 32 and repeated to 3 channels.
We use the conditional UNet to represent map T. We add a projection module (Miyato & Koyama, 2018 on WGAN-QC's ResNet Liu et al. 2019) to represent potential f. We choose learning rate to be 1-10 Adam optimizer with betas (0.5, 0.999). K1 10, K₂ 1. We use EMA stablize the training of the map. The batch size is 64. In practice, we introduce a coefficient À in the cost (2, v), (x',0/) || = # v') and set. À 0.5.
36

---

Published in Transactions on Machine Learning Research (07/2023)
F Theoretical justification of all the examples considered in this article
Table 4: We make a summary table on all the problems considered in this work with their cost function с and distributions Pa: Ps and whether the existence and uniqueness Monge map for each problem are guaranteed. We conclude the last column by checking the conditions Theorem [ Note that the theory of the existence of Monge map relies on the connection to deterministic OT plan (Santambrogio 2015 Section 1.1-1.4), and it is quite common that the OT plan is non-deterministic in deep learning examples. gd is the unit ball in RR, and A is the d dimension probability simplex in Rd
Guaranteo of existence
Problem
x
y
Cost
and uniqueness
[No] V.c(,,)) = [RS] unTy
R
Text image
7772008 (0))
3768
dx.,) 1Ral
may not injective.
Here denote
Noi Cousider M = p22x32x3+1
c((z,)), (..
Chas-preserving map
|- [-1, 82x3223 AJ-1
y
Pa not absolutely contimuous
||r x||² Ay
distribution 3
[=1,1]22x32x3
Existence guaranteed uniqueness not gnaranteed
Image inpainting
y x
c(2 0 x M MIB
c(19,111 01 (62. 2z))
[No] Vac(a, may not be
Population transport
0. 2m) x =]
y =
sin di sin d2 cos(O1 02)
injective, e.f. Remark
c0s cos
c(x,
(r a 6 sin"
[Yes] The conditions
Line ellipse
[-r,
(0.
The parametrization
Theorem are satisfied.
the line and the cllipse
(ir, 0): (acco y, bsin v).
[Yes] One can restrict e on X' x 31, ic., the manifold M as stated in Appendix A choose M. The conditions in Theorem are all satisfied. [Yes] One can verify that the conditions in Thm are satisfied.
Annulus
9
2
c(x,v) = ||
(reciprocal cost)
45r56
1sr$2
Anuulus
91
22
d(x,)) = ||> M/l
(122 cost)
Agrs6
Igrs2
cx((61, (2)) d2))) ++
No) Vacle, may not be injective.
OT sphere
(,,27) [0
y
arccos(X(cos) di cos do
c.f. Remark
sin Φ1 sin d2 cos(O] 02)))
Remark 7.
We discuss the condition
18 for geodesic cost. Consider the cost is the linearized geodesic
distance
14 one can compule
Vo. v ol ((f81, 3, (82, Φ21) ( di sin dh cos(O, 02) sin Φ1 cos Φ2
sin di sin Φ2 sin(O; 02)
Fir 02 01. When di # n, can always choose two different &' d² [0,m] such that sin( )') sin( Φ1 d²),
V Verdi o((0,, 21) (0,, ('2 ( -0) )0))==
0 sin -0") )= -V,
37

---

Published in Transactions on Machine Learning Research (07 (0722)23)
thus V2c(x,-) is not injective [0, 2x) [0, n|. One also similor way to show that for the cost (cf. (54) used in OT on sphere, Vaca( -) is not injective. This is also true for the original are length distance function [3 The cost function c(2, y) may satisfy (18) if it is the square of the geodesic distance, L.., c(x,y) v) = (x. y) where d(x,g) y) denotes the length of the geodesic joining 2, y. The computation w.r.t.
the square cost may serve as one of our fulure research directions.
G Notations used in the paper
Notation
Meaning
c(2, y)
cost. function
X, y
Polish spaces
Pa: Pb
Borel probability measures defined on X.
Measurable map from y
Typ
Pushforward of distribution μ by the map
M(X,3)
Space of measurable maps from X to y
Co(X)
Space of bounded continuous functions defined on X'
(X)
Space of Borel probability distributions defined on X
C(pa: Pb)
OT distance from Pa to Pb. i.e. optimal value to the general OT problem Optimal value to the Monge problem
Omonge (Pa: Pb)
K ((pa:P)) Pb)
Optimal value to the Kantorovich dual problem
19 or 20
C(T,F)
Lagrangian function of the Monge problem, defined in 5
+b+
c-transform defined in 21
Φ6-
c-transform defined in 22
T..0x
Optimal solution (Monge map) to and optimal solution to 20
(f
Saddle point of C(T, f) to 44
(T,))
sup-inf solution to
B,K,K,,K,
Training batch size, total iterations, iterations for To, iterations for fy
R
Frozen matrix
Point-wise multiplication
M, MC
Binary mask with the same size the image, complement of M
U(E)
Uniform probability distribution on measurable set E
38


Reviews:
[{"review_id": "wgsUozyWOm", "reviewer": "Reviewer_HSqP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- Firstly, this paper is well-written, with clear motivation and explanation, along with sufficient experimental evaluation. The organization of the contents is also good.\n- The motivation is strong, as there is clearly a need to improve the understanding of OT map estimation in light of the emergence of deep-learning-based large models. In particular, the example of estimating transport without text-image pair labels will be very useful given the fact that large language models are being deployed in various ML areas.\n- The theoretical results are well-presented. Although the idea of transforming the Kantorovich dual into the saddle point has appeared in existing studies, the approach to understanding the duality gap is quite novel.\n\nWeakness:\n\n- Questions regarding theoretical results:\n    - The theoretical results in this work are established on Theorem 1. I would like to learn more about the regularization conditions for the Monge map itself. Specifically, when estimating the transport map between two distributions, the existence and uniqueness of the optimal transport map will require a certain degree of smoothness (more than that of the source and target measure). Should we also include regularization for the map in this work? What could be the effect of regularization, and is it possible to do it for a Monge map that is parameterized as a neural network?\n- Questions regarding experimental results:\n    - In section 6.1, the authors displayed qualitative results on the Laion Art dataset and the CC-3M dataset, respectively. Since the CC-3M dataset does not have text-image-pair labels, how were the \"real image\" and the \"real embedding\" generated in Figure 2?\n    - In Figure 4, is it possible to show the pushforward embedding distribution heat-map obtained by some baseline OT methods such as the linear map [Perrot et al]?\n    - In Figures 1 and 2, it seems that the decoded images from the proposed method have better fidelity than those from the diffusion prior. What could be the cause? Is it possible to elaborate more on the advantages and disadvantages between a deterministic map and a stochastic map?\n    - The authors could provide more information regarding the label-preserving map. How is the correspondence between labels selected?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a framework for estimating the Monge map that is parameterized as neural networks between probability measures of large-scale datasets. To derive algorithms that are practically feasible, the authors first reformulate the Kantorovich dual of OT into a maximin saddle point problem. Specifically, the authors provided equivalence and consistency theoretical results to connect the proposed saddle point objective with the established results regarding the existence and uniqueness of Monge maps. Thus, the optimal solution to the saddle point objective corresponds well to optimal map estimation. From an algorithmic perspective, the authors parameterized both the transport map and the dual potential as neural networks to obtain a practical algorithm for training a neural Monge map. The proposed method can employ different kinds of ground cost metrics. Moreover, it can transport probability measures of different dimensions using predefined embedding functions.\n\nFrom a theoretical perspective, the authors provided a detailed analysis regarding the duality gap between the estimated map and an optimal Monge map. The $L^2$ error is guaranteed to be upper-bounded. The authors have presented a list of experimental results to validate the effectiveness of their proposed method. Specifically, the authors have used multiple cost metrics for different data tasks, including large-scale unlabeled text-to-image transfer, label-conditioned transfer, image in-painting, and population transport.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, I think this draft has satisfactory quality, and I recommend acceptance particularly if the points in the weaknesses are addressed:\n\n- When adding more discussions regarding the smoothness regularization of the optimal transport map estimation, the authors may want to refer to Theorem 3 of Manole et al. (2021).\n- More about the advantages and disadvantages between deterministic maps and stochastic maps.\n- Could you provide more clarification on the text-to-image generation experimental results?\n- How are the label correspondences selected in the class-preserving map?\n\n[1] Manole, T., Balakrishnan, S., Niles-Weed, J., & Wasserman, L. (2021). Plugin estimation of smooth optimal transport maps. *arXiv preprint arXiv:2107.12364*", "broader_impact_concerns": "It appears that the authors have not discussed the broader impact of their work in the current version of the draft. In my opinion, the proposed method can serve as a powerful generative model, but there may be associated risks. On the other hand, using optimal transport in the formulation provides more interpretability. Therefore, I encourage the authors to include some discussion on the broader impact of their work in the draft.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "L5FoCCR66C", "reviewer": "Reviewer_u8z1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly written and easy to follow. The method proposes to approximate the Monge map for general costs by solving the sup-inf problem on the Lagrangian. While it has already be done for the quadratic cost, the method seems novel for more general costs. Moreover, experiments demonstrate well that the method works for different costs and in large scale settings.\n\nStrengths:\n- A method to solve the Monge problem for general costs\n- Theoretical results with duality gap\n- Several experiments demonstrating that the methods works in large scale settings\n\nWeaknesses:\n- It lacks a quantitative study of the quality of the Monge map learned, and of the Wasserstein distance approximated\n- For some experiments, the superiority of using the proposed cost over the classical quadratic cost is not shown (e.g. in Unpaired text to image generation)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this articles, authors propose to compute the Monge map and the associate transport cost using a general cost. They reformulate the Monge problem using the Lagrangian and derive conditions for which having solutions to the sup-inf problem on the Lagrangian is equivalent with having a Monge map. Then, they propose to solve this problem in practice by modeling the transport map and the dual potential with neural networks, and by solving the sup-inf problem with the Lagrangian as objective. This objective can be approximated provided samples from the source and the target distributions.\n\nOn the theoretical side, they provide a duality gap result. On the experimental side, they provide results for different problems using different ground cost, and hence demonstrating the versatility of their method.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have some quesions about the experiments, in particular on what they aim to show. If I understand well, these experiments aim at showing that this method is well able to approximate the OT map and the Wasserstein distance for general costs in different large scale settings. But no comparison with a ground truth is provided. Hence, I believe that an experiment comparing the map obtained and the value of the objective with a groundtruth Monge map and Wasserstein distance is missing. A possibility would be to use the benchmark introduced in [1] for 2-Wasserstein. However, since the proposed method coincides with other methods in this case, and hence has already been benchmarked, another toy case might also be of interest. Closed-form for other costs than the quadratic cost might be intricate to find. Maybe a possiblity would be to use the construction of [2] to approximate a c-concave map on the sphere, and hence build a Monge map for the Wasserstein distance on the sphere with geodesic cost. Another possibility between discrete samples might be on the manifold of SPDs with log-euclidean metric as in Theorem 1 of [3].\nThen, I am not always convinced with the baseline used in the experiments. For example, in the unpaired text to image generation, it is written \"We use the same unpaired data generation scheme as [4]\" but it is not compared with [4]. Providing such comparison would allow to further assess the superiority of using the negative cosine similarity as cost over the quadratic distance.\nIn the experiment on the sphere, the comparison is performed with [5] which is not really comparable since it uses a linear transformation. I believe that an additional comparison with the groundtruth plan obtained e.g. with Sinkhorn as in [6] would allow to better appreciate the map. And instead of [5], a method using NN with 2-Wasserstein might be more comparable, even though we expect that it would not satisfy the spherical geometry.\n\n[1] Korotin, Alexander, et al. \"Do neural optimal transport solvers work? a continuous wasserstein-2 benchmark.\" Advances in Neural Information Processing Systems 34 (2021): 14593-14605.\n\n[2] Cohen, Samuel, Brandon Amos, and Yaron Lipman. \"Riemannian convex potential maps.\" International Conference on Machine Learning. PMLR, 2021.\n\n[3] Ju, Ce, and Cuntai Guan. \"Deep Optimal Transport on SPD Manifolds for Domain Adaptation.\" arXiv preprint arXiv:2201.05745 (2022).\n\n[4] Rout, Litu, Alexander Korotin, and Evgeny Burnaev. \"Generative modeling with optimal transport maps.\" arXiv preprint arXiv:2110.02999 (2021).\n\n[5] Perrot, Michaël, et al. \"Mapping estimation for discrete optimal transport.\" Advances in Neural Information Processing Systems 29 (2016).\n\n[6] Amos, Brandon, et al. \"Meta optimal transport.\" arXiv preprint arXiv:2206.05262 (2022).", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fsqgLXccwU", "reviewer": "Reviewer_LY47", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The proposed method is based on a simple idea, solving the Monge problem by the Lagrange multiplier method. This does not require any regularizers and architectural constraints that degenerate the distance estimation accuracy and numerical stability.\n2. The proposed method does not need to deal with the \"joint\" constraints in the Kantorovich dual $\\\\phi(y) - \\\\psi(x) \\\\le c(x, y)$, which is an important component to make optimization independent from the paired data $(x, y)$, being scalable. Despite that a closely-related work Makkuva et al. (2020) escapes from this constraint by using input-convex neural networks, this restricts our architectural design and hinders the scalability. The proposed method does not have this constraint.\n3. The authors evaluated the effectiveness of the proposed method with a number of applications (as I mentioned above). In particular, I found the application to text-to-image generation is really fascinating one. This is sufficiently persuasive that the proposed method is scalable to the high-dimensional setup.\n\n## Weaknesses\n\n1. The estimation error analysis in Theorem 4 is rather complicated and involves the weight function $\\\\beta$ to measure the estimation error. This is an additional artifact to the previous error analysis given by Makkuva et al. (2020). Further, by looking at the proof, I found that the weight function depends on the transport map $T$ itself (because the definition of the weight function in Eq. (42) involves $\\\\omega$, which depends on $T$ in Eq. (37)). May I ask you to discuss why the weight function is needed? In addition, are there any issues of the weight function being dependent on the transport map?\n2. The authors claim that \"for large scale problems ICNN is not expressive enough\" in Section 5, and advocates for the proposed method. Is it possible to support this claim experimentally? I don't intend to argue that the current experiments are insufficient (indeed, the experiments given by the authors are very thorough, in my opinion!), but simply would like to highlight the advantage of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method to compute the Monge map (modeled by a neural network) for the optimal transport problem. The proposed method is simply based on the Lagrange multiplier method applied to the Monge problem, which induces a saddle point problem, models the Monge map and multipliers with neural networks, and does not rely on any regularization techniques that are often involved in the recent large-scale optimal transport methods. This formulation has the advantages such as (1) the ability to obtain the Monge map, (2) being an accurate estimate of the optimal transport distance because it is free from regularizers, (3) being more scalable than the previous neural-network-based optimal transport methods because the architecture is not restricted to a specific one, and (4) the applicability to broader cost functions not only restricted to the L2 ground cost. Thanks to these benefits, the authors showed fantastic applications such as the unpaired text to image generation (taking the cosine similarity between texts and images as the ground cost), class-preserving maps (taking the distance defined on x and y jointly as the ground cost), unpaired image inpainting (taking the MSE in the unmasked areas as the ground cost), and population transportation estimation (taking the geodesic distance as the ground cost).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Suggestions and questions\n\n- In Section 3, some technical details are deferred to the appendix, and eventually a lot of equations are referred without their formal explanations in the main text. For example, \"the dual problem (19)\" in Theorem 2 is an important concept but not mentioned in Section 3. Can you have a bit more words to explain it here?\n- In Section 3 \"Mapping between unequal dimensions,\" the authors claim that the proposed method can be used even when the dimensions of the source and target space are different. However, I suppose that this is a natural consequence of the usual optimal transport problem when we only manipulate these two spaces via the corresponding cost function. Is it specific to the proposed method?\n- In Section 6.3, the authors claim \"most real-world applications do not involve the paired datasets,\" but in common scenarios of self-supervised learning such as the masked autoencoders, I think it is pretty natural to mask an input and reconstruct the original data from the occluded one, where the pair data are naturally available. Does the author consider other setups?\n- In Section 7, the authors claim that the proposed method is more advantageous than other deep learning methods for being free from paired data, but some unpaired experiments in Section 6 may not be sufficiently natural. For example, the text-to-image generation (Section 6.1) uses the pretrained CLIP model, which requires paired data to fit. The experiments in Section 6.3 may be natural to have paired data (as mentioned above). Can you discuss and be upfront if necessary in this perspective?\n\n### Minor comments\n\n- In the header of Theorem 1, \"Existence\" is better to be in lowercase.\n- In Theorem 1, you have the notation $(Id, T\\_*)$, which is not introduced formally and inconsistent with $[\\mathrm{id}, T_\\*]$ right before Section 6.\n- In Assumptions 3 and 4, \"argmin\" should be denoted by `\\mathrm`. In addition, \"arginf\" may not make sense because the infimum does not necessarily suppose the existence of the minimizer. I think \"argmin\" is sufficient for the purpose.\n- In p.11, the authors cited Fréchet Inception Distance for the first time, but it appears in p.10 (as FID). It seems better to cite there.", "broader_impact_concerns": "Not applicable", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a Monge map estimation using neural networks. The authors first reformulate the Kantorovich dual of OT into a maximin saddle point problem. Then, they propose to solve this problem in practice by modeling the transport map and the dual potential with neural networks. Moreover, a detailed analysis regarding the duality gap between the estimated map and an optimal Monge map is provided. Through experiments, it demonstrates that the proposed method outperforms existing methods in various types of applications. 

The paper works on estimating a Monge map, a fundamental problem in OT, and many OT researchers are interested in working on the topic. Specifically, they proposed a simple yet general framework based on the Lagrange multiplier method. This contribution reminds me of the work done by the work Cuturi 2013 (NIPS), which proposed an entropic regularized OT and proposed a Sinkhorn algorithm. Now, the Sinkhorn algorithm is used in many machine learning applications including document classification, matching, and divergence estimation. Since the contribution of this paper is simple and fundamental, it can be further studied like the Sinkhorn algorithm, and can be useful in many real applications.

Overall, the proposed neural Monge Map estimation approach is an important contribution to the OT community and the reviewers agree to accept the paper. In addition, one of the very experienced reviewers recommended a featured certification; I also vote for acceptance with a featured certification.


Decision: Accept (poster)