Title: Extremal Domain Translation with Neural Optimal Transport
Venue: NeurIPS 2023
Accepted: yes
Keywords: optimal transport

partial optimal transport

neural networks

domain translation
Abstract: In many unpaired image domain translation problems, e.g., style transfer or super-resolution, it is important to keep the translated image similar to its respective input image. We propose the extremal transport (ET) which is a mathematical formalization of the theoretically best possible unpaired translation between a pair of domains w.r.t. the given similarity function. Inspired by the recent advances in neural optimal transport (OT), we propose a scalable algorithm to approximate ET maps as a limit of partial OT maps. We test our algorithm on toy examples and on the unpaired image-to-image translation task. The code is publicly available at https://github.com/milenagazdieva/ExtremalNeuralOptimalTransport

Extremal Domain Translation with Neural Optimal Transport

Abstract

In many unpaired image domain translation problems, e.g., style transfer or superresolution, it is important to keep the translated image similar to its respective input image. We propose the extremal transport (ET) which is a mathematical formalization of the theoretically best possible unpaired translation between a pair of domains w.r.t. the given similarity function. Inspired by the recent advances in neural optimal transport (OT), we propose a scalable algorithm to approximate ET maps as a limit of partial OT maps. We test our algorithm on toy examples and on the unpaired image-to-image translation task. The code is publicly available at https://github.com/milenagazdieva/ExtremalNeuralOptimalTransport

d～x

d～x

W=1

W=1

W=2

#

W=4

#

W=4

#

W=8

#

W=8

(a) Handbag → shoes (128 × 128).

Figure 1: (Nearly) extremal transport with our Algorithm 1. Higher w yields bigger similarity of x and T ( x ) in ℓ 2 .

1 Introduction

The unpaired translation task [72, Fig. 2] is to find a map x ↦→ T ( x ) , usually a neural network, which transports the samples x from the given source domain to the target domain. The key challenge

similarity

here is that the correspondence between available data samples x from the source and y from target domains is not given . Thus, the task is ambiguous as there might exist multiple suitable T .

When solving the task, many methods regularize the translated samples T ( x ) to inherit specific attributes of the respective input samples x . In the popular unpaired translation [72, Fig. 9] and enhancement [69, Equation 3] tasks for images, it is common to use additional unsupervised identity losses, e.g., ∥ T ( x ) -x ∥ 1 , to make the translated output T ( x ) be similar to the input images x . The same applies, e.g., to audio translation [51]. Therefore, the learning objectives of such methods usually have two components.

0.0

-0.5

-1.0

1.5

2.0

Inputx~p

O

2.5

TargetQ

Mappedy=T(x)

1.0

-0.5

0.0

0.5

1.0

1.0

-0.5

0.0

0.5

1.0

The first component is the domain loss (main) enforcing the translated sample T ( x ) to look like the samples y from the target domain. The second component is the similarity loss (regularizer, optional ) stimulating the translated T ( x ) to inherit certain attributes of input x . A question arises: can one obtain the maximal similarity of T ( x ) to x but still ensure that T ( x ) is indeed from the target domain? A straightforward " yes, just increase the weight of the similarity loss " may work but only to a limited extent. We demonstrate this in Appendix C.

Contributions. In this paper, we propose the extremal transport (ET, M 3.1) which is a rigorous mathematical task formulation describing the theoretically best possible unpaired domain translation w.r.t. the given similarity function. We explicitly characterize ET maps and plans by establishing an intuitive connection to the nearest neighbors (NN). We show that ET maps can be learned as a limit ( M 3.3) of specific partial optimal transport (OT) problem which we call incomplete transport (IT, M 3.2). For IT, we derive the duality formula yielding an efficient computational algorithm ( M 3.4). We test our algorithm on toy 2D examples and high-dimensional unpaired image translation ( M 5).

Notation. We consider compact Polish spaces ( X , ∥·∥ X ) , ( Y , ∥·∥ Y ) and use P ( X ) , P ( Y ) to denote the sets of Radon probability measures on them. We use M + ( X ) ⊂ M ( X ) to denote the sets of finite non-negative and finite signed (Radon) measures on X , respectively. They both contain P ( X ) as a subset. For a non-negative µ ∈ M + ( X ) , its support is denoted by Supp ( µ ) ⊂ X . It is a closed set consisting of all points x ∈ X for which every open neighbourhood A ∋ x satisfies µ ( A ) > 0 . We use C ( X ) to denote the set of continuous functions X → R equipped with ∥ · ∥ ∞ norm. Its dual space is M ( X ) equipped with the ∥ · ∥ 1 norm. A sequence µ 1 , µ 2 , · · · ∈ M ( X ) is said to be weakly-* converging to µ ∗ ∈ M ( X ) if for every f ∈ C ( X ) it holds that lim n →∞ ∫ X f ( x ) dµ n ( x ) = ∫ X f ( x ) dµ ∗ ( x ) . For a probability measure π ∈ P ( X × Y ) , we use π x ∈ P ( X ) and π y ∈ P ( Y ) to denote its projections onto X , Y , respectively. Disintegration of π yields dπ ( x, y ) = dπ x ( x ) dπ ( y | x ) , where π ( y | x ) denotes the conditional distribution of y ∈ Y for a given x ∈ X . For µ, ν ∈ M ( Y ) , we write µ ≤ ν if for all measurable A ⊂ Y it holds that µ ( A ) ≤ ν ( A ) . For a measurable map T : X → Y , we use T♯ to denote the associated pushforward operator P ( X ) →P ( Y ) .

2 Background on Optimal Transport

In this section, we give an overview of the OT theory concepts related to our paper. For details on OT, we refer to [62, 65, 56], partial OT - [22, 10].

Standard OT formulation. Let c : X × Y → R be a continuous cost function. For P ∈ P ( X ) , Q ∈ P ( Y ) , the OT cost between them is given by

where inf is taken over measurable T : X → Y pushing P to Q (transport maps), see Fig. 3a. Problem (1) is called the Monge's OT problem, and its minimizer T ∗ is called an OT map .

In some cases, there may be no minimizer T ∗ of (1). Therefore, it is common to consider Kantorovich's relaxation:

where inf is taken over π ∈ P ( X × Y ) satisfying π x = P and π y = Q , respectively. A minimizer π ∗ ∈ Π( P , Q ) in (2) always exists and is called an OT plan . A widely used example of OT cost for X = Y = R d is the Wasserstein-1 distance ( W 1 ), i.e., OT cost (2) for c ( x, y ) = ∥ x -y ∥ .

(a) Monge's OT formulation.

(b) Kantorovich's OT formulation.

c(x1,y)dn(ylx1)

C(x1,T(x1))

X1

T(x1)

X1

n(ylx1)

IP

C(x2,T(x2))

T(x2)

IP

X2

X2

T(xn)

n(ylxn)

Xn

Xn

C(Xn,T(xn))

[c(xn,y) dn(ylxn)

To provide an intuition behind (2), we disintegrate dπ ( x, y )= dπ x ( x ) dπ ( y | x ) :

i.e., (2) can be viewed as an extension of (1) allowing to split the mass of input points x ∼ P (Fig. 3b). With mild assumptions on P , Q , the OT cost value (2) coincides with (1), see [62, Theorem 1.33].

Partial OT formulation. Let w 0 ,w 1 ≥ m ≥ 0 . We consider

where inf is taken over π ∈ P ( X × Y ) satisfying the inequality constraints mπ x ≤ w 0 P and mπ y ≤ w 1 Q . Minimizers π ∗ of (4) are called partial OT plans (Fig. 4).

Here the inputs are two measures w 0 P and w 1 Q with masses w 0 and w 1 . Intuitively, we need to match a m w 0 -th fraction mπ x of the first measure w 0 P with a m w 1 -th fraction mπ y of the second measure w 1 Q (Fig. 4); choosing π x , π y is also a part of this problem. The key difference from problem (4) is that the constraints are inequalities . In the particular case m = w 0 = w 1 , problem (4) reduces to (2) as the inequality constraints can be replaced by equalities.

3 Main Results

First, we formulate the extremal transport (ET) problem ( M 3.1). Next, we prove that ET maps can be recovered as a limit of incomplete transport (IT) maps ( M 3.2, 3.3). Then we propose an algorithm to solve the IT problem ( M 3.4). We provide the proofs for all the theorems in Appendix F.

3.1 Extremal Transport Problem

Popular unpaired translation methods, e.g., [72, M 3.1] and [33, M 3], de-facto assume that available samples x, y from the input and output domains come from the data distributions P , Q ∈ P ( X ) , P ( Y ) . As a result, in their optimization objectives, the domain loss compares the translated T ( x ) ∼ T♯ P and target samples y ∼ Q by using a metric for comparing probability measures , e.g., GAN loss [27]. Thus, the target domain is identified with the probability measure Q .

We pick a different approach to define what the domain is. We still assume that the available data comes from data distributions, i.e., x ∼ P , y ∼ Q . However, we say that the target domain is the part of Y where the probability mass of Q lives. 2 Namely, it is Supp ( Q ) ⊂ Y . We say that a map T translates the domains if Supp ( T♯ P ) ⊂ Supp ( Q ) . This requirement is weaker than the usual T♯ P = Q . Assume that c ( x, y ) is a function estimating the dissimilarity between x, y . We would like to pick T ( x ) ∈ Supp ( Q ) which is maximally similar to x in terms of c ( x, y ) . This preference of T can be formalized as follows:

where the inf is taken over measurable T : X → Y which map the probability mass of P to Supp ( Q ) . We say that (5) is the (Monge's) extremal transport (ET) problem.

mTc

mT

y

C(X1,y1)

X1

y1

Q

WoIP

m

Xn

C(Xn.ym)

(a) Monge's ET formulation.

(x)1'x)

c(x1,y)dn(ylx1)

T(x1)

X1

X1

n(ylx1)

IP

((zx)1'zx)

T(X2)

Q

IP

n(ylx2)Q

X2

X2

n(ylxn)

T(xn)

Xn

Xn

C(Xn,T(xn))

C(xn,y)dn(ylxn)

Problem (5) is atypical for the common OT framework. For example, the usual measure-preserving constraint T♯ P = Q in (1) is replaced with Supp ( T♯ P ) ⊂ Supp ( Q ) which is more tricky. Importantly, measure Q can be replaced with any other Q ′ ∈ P ( Y ) with the same support yielding the same inf . Below we analyse the minimizers T ∗ of (5). We define c ∗ ( x ) def = min y ∈ Supp ( Q ) c ( x, y ) . Here the min is indeed attained (for all x ∈ X ) because c ( x, y ) is continuous and Supp ( Q ) ⊂ Y is a compact set. The value c ∗ ( x ) can be understood as the lowest possible transport cost when mapping the mass of point x to the support of Q . For any admissible T in (5), it holds ( P -almost surely):

Proposition 1 (Continuity of c ∗ ) . It holds that c ∗ ∈ C ( X ) .

As a consequence of Proposition 1, we see that c ∗ is measurable. We integrate (6) w.r.t. x ∼ P and take inf over all feasible T . This yields a lower bound on Cost ∞ ( P , Q ) :

There exists admissible T making (7) the equality. Indeed, let NN ( x ) def = { y ∈ Supp ( Q ) s.t. c ( x, y ) = c ∗ ( x ) } be the set of points y which attain min in the definition of c ∗ . These points are the closest to x points in Q w.r.t. the cost c ( x, y ) . We call them the nearest neighbors of x . From this perspective, we see that (7) turns to equality if and only if T ( x ) ∈ NN ( x ) holds for P -almost all x ∈ X , i.e., T maps points x ∼ P to their nearest neighbors in Supp ( Q ) . We need to make sure that such measurable T exists (Fig. 5a).

Theorem 1 (Existence of ET maps) . There exists at least one measurable map T ∗ : X → Y minimizing (5) . For P -almost all x ∈ X it holds that T ∗ ( x ) ∈ NN ( x ) . Besides,

We say that Cost ∞ ( P , Q ) is the extremal cost because one can not obtain smaller cost when moving the mass of P to Supp ( Q ) . In turn, we say that minimizers T ∗ are ET maps .

One may extend the ET problem (8) in the Kantorovich's manner by allowing the mass splitting and stochastic plans:

where Π ∞ ( P , Q ) are probability measures π ∈ P ( X × Y ) s.t. π x = P and Supp ( π y ) ⊂ Supp ( Q ) . To understand the structure of minimizers in (8), it is more convenient to disintegrate dπ ( x, y ) = dπ ( y | x ) dπ x ( x ) :

Thus, computing (8) boils down to computing a family of conditional measures π ( ·| x ) minimizing (8). As in (7), for any π ∈ Π ∞ ( P , Q ) , it holds (for P -almost all x ∈ X ) that

because π redistributes the mass of P to Supp ( Q ) . By integrating (10) w.r.t. x ∼ P = π x and taking inf over all admissible plans π , we derive that ∫ X c ∗ ( x ) d P ( x ) is a lower bound for (8). In particular,

(a) Monge's IT formulation.

T

h

c(x1,y)dn(ylx1)

X1

T(y/x1)

IP

X2

T(ylxn)

Xn

c(xn,y) dn(ylxn)

T#IP

C(x1,T(x1))

X1

T(x1)

IP

X2

C(x2,T(x2))

wQ

T(Xn)

Xn

C(Xn,T(xn))

the bound is tight for π ∗ ( y | x ) = δ T ∗ ( x ) , where T ∗ is the ET map from our Theorem 1. Therefore, the value (8) is the same as (5) but possibly admits more minimizers. We call the minimizers π ∗ of (8) the ET plans (Fig. 5b).

From (10) and the definition of NN ( x ) , we see that minimizers π ∗ are the plans for which π ∗ ( y | x ) redistributes the mass of x among the nearest neighbors y ∈ NN ( x ) of x (for P -almost all x ∈ X ). As a result, π ∗ ( y | x ) can be viewed as a stochastic nearest neighbor assignment between the probability mass of P and the support of Q .

3.2 Incomplete Transport Problem

In practice, solving extremal problem (5) is challenging because it is hard to enforce Supp ( T♯ P ) ⊂ Supp ( Q ) . To avoid enforcing this constraint, we replace it and consider the following problem with finite parameter w ≥ 1 :

We call (11) Monge's incomplete transport (IT) problem (Fig. 6a). With the increase of w , admissible maps T obtain more ways to redistribute the mass of P among Supp ( Q ) . Informally, when w →∞ , the constraint T♯ P ≤ w Q in (11) tends to the constraint Supp ( T♯ P ) ⊂ Supp ( Q ) in (5), i.e., (11) itself tends to ET problem (5). We will formalize this statement a few paragraphs later (in M 3.3).

As in (1), problem (11) may have no minimizer T ∗ or even may have the empty feasible set. Therefore, it is natural to relax problem (11) in the Kantorovich's manner:

where the inf is taken over the set Π w ( P , Q ) of probability measures π ∈ P ( X × Y ) whose first marginal is π x = P , and the second marginal satisfies π y ≤ w Q (Fig. 6b).

We note that IT problem (12) is a special case of partial OT (4) with w 0 = m = 1 and w 1 = w . In (12), one may actually replace inf with min , see our proposition below.

∗ w

Proposition 2 (Existence of IT plans) . Problem (12) admits at least one minimizer π ∈ Π ( P , Q ) . We say that minimizers of (12) are IT plans . In the general case, Kantorovich's IT cost (12) always lower bounds Monge's counterpart (11). Below we show that they coincide in the practically most interesting Euclidean case.

Proposition 3 (Equivalence of Monge's, Kantorovich's IT costs) . Let X , Y ⊂ R D be two compact sets, P ∈ P ( X ) be atomless, Q ∈P ( Y ) . Then Monge's (11) and Kantorovich's (12) IT costs coincide.

However, it is not guaranteed that inf in Monge's problem (11) is attained even in the Euclidean case. Still for general Polish spaces X , Y it is clear that if there exists a deterministic IT plan in Kantorovich's problem (12) of the form π ∗ = [ id X , T ∗ ] , then T ∗ is an IT map in (11), and the IT Monge's (11) and Kantorovich's (12) costs coincide. Henceforth, for simplicity, we assume that X , Y , c, P , Q are such that (11) and (12) coincide, e.g., those from Prop. 3.

IT problem (12) can be viewed as an interpolation between OT (2) and ET problems (8). Indeed, when w = 1 , the constraint π y ≤ Q is equivalent to π y = Q as there is only one probability measure which is ≤ Q , and it is Q itself. Thus, IT (12) with w = 1 coincides with OT (2). In the next section, we show that for w →∞ one recovers ET from IT.

3.3 Link between Incomplete and Extremal Transport

Now we connect incomplete (12) and extremal (8) transport tasks.

Theorem 2 (IT costs converge to the ET cost when w →∞ ) . Function w ↦→ Cost w ( P , Q ) is convex, non-increasing in w ∈ [1 , + ∞ ) and

Anatural subsequent question here is whether IT plans in (12) converge to ET plans (8) when w →∞ . Our following result sheds the light on this question.

Theorem 3 (IT plans converge to ET plans when w →∞ ) . Consider w 1 , w 2 , w 3 , · · · ≥ 1 satisfying lim n →∞ w n = ∞ . Let π w n ∈ Π w n ( P , Q ) be a sequence of IT plans solving (12) with w = w n , respectively. Then it has a (weakly-*) converging sub-sequence. Every such sub-sequence of IT plans converges to an ET plan π ∗ ∈ Π ∞ ( P , Q ) .

In general, there may be sub-sequences of IT plans converging to different ET plans π ∗ ∈ Π ∞ ( P , Q ) . However, our following corollary shows that with the increase of weight w , elements of any subsequence become closer to the set of ET plans.

Corollary 1 (IT plans become closer to the set of ET plans when w →∞ ) . For all ε > 0 ∃ w ( ε ) ∈ [1 , ∞ ) such that ∀ w ≥ w ( ε ) and ∀ IT plan π w ∈ Π w ( P , Q ) solving Kantorovich's IT problem (12) , there exists an ET plan π ∗ which is ε -close to π w in W 1 , i.e., W 1 ( π ∗ , π w ) ≤ ε .

Providing a stronger convergence result here is challenging, and we leave this theoretical question open for future studies. Our Theorems 2, 3 and Corollary 1 suggest that to obtain a fine approximation of an ET plan ( w = ∞ ), one may use an IT plan for sufficiently large finite w . In Appendix G.1, we empirically demonstrate this observation through an experiment where the ground-truth deterministic ET plan is analytically known. Below we develop a neural algorithm to compute IT plans.

3.4 Computational Algorithm for Incomplete Transport

To begin with, for IT (12), we derive the dual problem.

Theorem 4 (Dual problem for IT) . It holds

where the max is taken over non-positive f ∈ C ( Y ) and f c ( x ) def = min y ∈Y { c ( x, y ) -f ( y ) } .

We call the function f potential . In the definition of f c , min is attained because c, f are continuous and Y is compact. The function f c is called the c -transform of f .

The difference of formula (13) from usual c -transform-based duality formulas for OT (2), see [62, M 1.2], [65, M 5], is that f is required to be non-positive and the second term is multiplied by w ≥ 1 . We rewrite the term ∫ X f c ( x ) d P ( x ) in (13):

Here we use the interchange between the integral and inf [59, Theorem 3A]; in (14) the inf is taken over measurable maps. Since ( x, y ) ↦→ c ( x, y ) -f ( y ) is a continuous function on a compact set, it admits a measurable selection T ( x ) ∈ arg min y ∈Y { c ( x, y ) -f ( y ) } minimizing (14), see [2, Theorem 18.19]. Thus, inf can be replaced by min . We combine (14) and (13) and obtain an equivalent saddle point problem:

where the functional L ( f, T ) is defined by

Functional L ( f, T ) can be viewed as a Lagrangian with f ≤ 0 being a multiplier for the constraint T♯ P -w Q ≤ 0 . By solving (15), one may obtain IT maps.

Theorem 5 (IT maps are contained in optimal saddle points) . Let f ∗ be any maximizer in (13) . If π ∗ ∈ Π w ( P , Q ) is a deterministic IT plan, i.e., it solves (12) and has the form π ∗ = [ id X , T ∗ ] ♯ P for some measurable T ∗ : X → Y , then

Algorithm 1: Procedure to compute the IT map between P and Q for transport cost c ( x, y ) and weight w .

Input : distributions P , Q accessible by samples; mapper T θ : X →Y ; potential f ψ : X → R -; transport cost c : X × Y → R ; weight w ≥ 1 ; number K of inner iterations;

T Output: approximate IT map ( T θ ) # P ≤ w Q ;

Our Theorem 5 states that in some optimal saddle points ( f ∗ , T ∗ ) of (15) it holds that T ∗ is the IT map between P , Q . In general, the arg inf T set for an optimal f ∗ might contain not only IT maps T ∗ , but other functions as well ( fake solutions ), see limitations in Appendix A.

We solve the optimization problem (15) by approximating the map T and potential f with neural networks T θ and f ψ , respectively. To make f ψ non-positive, we use x ↦→-| x | as the last layer. The nets are trained using random batches from P , Q and stochastic gradient ascent-descent. We detail the optimization procedure in Algorithm 1.

4 Related work

OT in generative models. A popular way to apply OT in generative models is to use the OT cost as the loss function to update the generator [4, 28, 26], see [41] for a survey. These methods are not relevant to our study as they do not learn an OT map but only compute the OT cost.

Recent works [43, 42, 61, 20, 5, 24, 29] are the most related to our study. These papers show the possibility to learn the OT maps (or plans) via solving saddle point optimization problems derived from the standard c -transform-based duality formulas for OT. The underlying principle of our objective (15) is analogous to theirs. The key difference is that they consider OT problems (1), (2) and enforce the equality costraints, e.g., T♯ P = Q , while our approach enforces the inequality constraint T♯ P ≤ w Q allowing to partially align the measures. We provide a detailed discussion of relation with these works as well as with the fundamental OT (2) and partial OT (4) literature [22, 10, 62] in Appendix F, see bibliographic remarks after the proofs of Theorems 2, 4 and 5.

For completeness, we also mention other existing neural OT methods [25, 63, 49, 17, 39]. These works are less related to our work because they either underperform compared to the above-mentioned saddle point methods, see [40] for evaluation, or they solve specific OT formulations, e.g., entropic OT [56, M 4], which are not relevant to our study.

The papers [66, 47, 19] are slightly more related to our work. They propose neural methods for unbalanced OT [14] which can also be used to partially align measures. As we will see in Appendix B, UOT is hardly suitable for ET (8) as it is not easy to control how it spreads the probability mass. Besides, these methods consider OT between small-dimensional datasets or in latent spaces. It is not clear whether they scale to high-dimensions, e.g., images.

Discrete OT methods , including partial OT [11], are not relevant to us, see Appendix D for details.

Unpaired domain translation [3] is a generic problem which includes image super-resolution [12], inpainting [71], style translation [34] tasks, etc. Hence, we do not mention all of the existing approaches but focus on their common main features instead. In many applications, it is important to preserve semantic information during the translation, e.g., the image content. In most cases, to do this it is sufficient to use convolutional neural networks. They preserve the image content thanks to their design which is targeted to only locally change the image [18]. However, in some of the tasks

(a) Input, target measures.

(b) IT map,

w

=3

.

(c) IT map,

w

=

3

.

(d) IT map,

w

=1

.

1.5

O

Input x~P

Mapped y=T(x)

Mapped y=T(x)

Mappedy=T(x)

TargetQ

1.0

0.5

0.0

0.5

-1.0-0.5

0.0

0.5

1.0

1.5

2.0-1.0-0.5

0.0

0.5

1.0

1.5

2.0 -1.0 -0.5

0.0

0.5

1.0

1.5

2.0 -1.0-0.5

0.0

0.5

1.0

1.5

2.0

2

(a) Input measure.

(b) IT map,

w

=8

.

(c) IT map,

w

=4

.

(d) IT map,

w

=2

.

(e) IT map,

w

=1

.

(f) Target measure.

100

Input x ~ p

Mapped y = T(x)

Mapped y=T(x)

Mapped y=T(x)

Mapped y=T(x)

Targety~Q

0

100

0

100

100

0

100

100

0

100

100

0

100

100

0

100

100

0

100

additional image properties must be kept, e.g., image colors in super-resolution. Typical approaches to such problems are based on GANs and use additional similarity losses, e.g., the basic CycleGAN [72] enforces ℓ 1 similarity. Such methods are mostly related to our work. However, without additional modifications most of these approaches only partially maintain the image properties, see [44, Figure 5]. As we show in experiments (Appendix C), IT achieves better similarity than popular CycleGAN, StarGAN-v2 [15] methods, both default and modified to better preserve the image content.

5 Evaluation

In M 5.1, we provide illustrative toy 2D examples. In M 5.2, we evaluate our method on the unpaired image-to-image translation task. Technical training details (architectures, learning rates, etc.) are given in Appendix E. The code is written using PyTorch framework and is publicly available at https://github.com/milenagazdieva/ExtremalNeuralOptimalTransport .

Transport costs. We experiment with the quadratic cost c ( x, y ) = ℓ 2 ( x, y ) as this cost already provides reasonable performance. We slightly abuse the notation and use ℓ 2 to denote the squared error normalized by the dimension. Experiments with the perceptual cost are given in Appendix G.3.

5.1 Toy 2D experiments

In this section, we provide 'Wi-Fi' and 'Accept' examples in 2D to show how the choice of w affects the fraction of the target measure Q to which the probability mass of the input P is mapped. In both cases, measure P is Gaussian . In Appendix B, we demonstrate how other methods perform on these 'Wi-Fi' and 'Accept' toy examples.

In 'Wi-Fi' experiment (Fig. 7), target Q contains 3 arcs. We provide the learned IT maps for w ∈ [1 , 3 2 , 3] . The results show that by varying w it is possible to control the fraction of Q to which the mass of P will be mapped. In Fig. 7, we see that for w = 1 our IT method learns all 3 arcs. For w = 3 2 , it captures 2 arcs, i.e., ≈ 2 3 of Q . For w = 3 , it learns 1 arc which corresponds to ≈ 1 3 of Q .

In 'Accept' experiment (Fig. 2), target Q is a two-line text. Here we put w = 2 and, as expected, our method captures only one line of the text which is the closest to P in ℓ 2 .

5.2 Unpaired Image-to-image Translation

Here we learn IT maps between various pairs of datasets. We test w ∈{ 1 , 2 , 4 , 8 } in all experiments. For completeness, we consider bigger weights w ∈ { 16 , 32 } in Appendix G.4.

d～x

d~x

#

L=M

W=1

Z=M

Z=M

(x)

W=4

W=4

w=8

(x)

w=8

Image datasets. We utilize the following publicly available datasets as P , Q : celebrity faces [46], aligned anime faces 3 , flickr-faces-HQ [36], comic faces 4 , Amazon handbags from LSUN dataset [68], shoes [67], textures [16] and chairs from Bonn furniture styles dataset [1]. The sizes of datasets are from 5K to 500K samples. We work with 64 × 64 and 128 × 128 images.

Train-test split. We use 90% of each dataset for training. The remaining 10% are held for test. All the presented qualitative and quantitative results are obtained for test images.

Experimental results. Our evaluation shows that with the increase of w the images ˆ T ( x ) translated by our IT method become more similar to the respective inputs x w.r.t. ℓ 2 . In Table 1a, we quantify this effect. Namely, we show that the test transport cost 1 N test ∑ N test n =1 c ( x, ˆ T ( x ) ) decreases with the increase of w which empirically verifies our Theorem 2.

We qualitatively demonstrate this effect in Fig. 1b, 1a, 9b, 9a and 8. In celeba (female) → anime (Fig. 1b), the hair and background colors of the learned anime images become closer to celebrity faces' colors with the increase of w . For example, in the 4th column of Fig. 1b, the anime hair color changes from green to brown, which is close to that of the respective celebrity. In the 6th column, the background is getting darker. In handbag → shoes (Fig. 1a), the color, texture and size of the shoes become closer to that of handbag. Additionally, for this experiment we plot the projections of the learned IT maps to the first 2 principal components of Q (Fig. 8). We see that projections are close to Q for w = 1 and become closer to P with the increase of w . In ffhq → comics , the changes mostly affect facial expressions and individual characteristics. In textures → chairs , the changes are mostly related to chairs' size which is expected since we use pixel-wise ℓ 2 as the cost function. Additional qualitative results are given in Appendix G.5 (Fig. 24, 25, 9a).

For completeness, we measure test FID [32] of the translated samples, see Table 1b. We do not calculate FID in the handbag → chairs experiment because of too small sizes of the test parts of the datasets (500 textures, 2K chairs). However, we emphasize that FID is not representative when w > 1 . In this case, IT maps learn by construction only a part of the target measure Q . At the same time, FID tests how well the transported samples represent the entire target distribution and is very sensitive to mode dropping [48, Fig. 1b]. Therefore, while the cost decreases with the growth of w , FID, on the contrary, increases. This is expected since IT maps to smaller part of Q . Importantly, the visual quality of the translated images ˆ T ( x ) is not decreasing.

In Appendix C, we compare our IT method with other image-to-image translation methods and show that IT better preserves the input-output similarity.

6 Potential Impact

Inequality constraints for generative models. The majority of optimization objectives in generative models (GANs, diffusion models, normalizing flows, etc.) enforce the equality constraints, e.g., T♯ P = Q , where T♯ P is the generated measure and Q is the data measure. Our work demonstrates that it is possible to enforce inequality constraints, e.g., T♯ P ≤ w Q , and apply them to a large-scale problem. While in this work we primarily focus on the image-to-image translation task, we believe that the ideas presented in our paper have several visible prospects for further improvement and applications. We list them below.

(1) Partial OT. Learning alignments between measures of unequal mass is a pervasive topic which has already perspective applications in biology to single-cell data [66, 47, 19]. The mentioned works use unbalanced OT [14]. This is an unconstrained problem where the mass spread is softly controlled by the regularization. Therefore, may be hard to control how the mass is actually distributed. Using partial OT which enforces hard inequality constraints might potentially soften this issue. Our IT problem (12) is a particular case of partial OT (4). We believe that our study is useful for future development of partial OT methods.

(2) Generative nearest neighbors. NNs play an important role in machine learning applications such as, e.g., image retrieval [6]. These methods typically rely on fast discrete approximate NN [50, 35] and perform matching with the latent codes of the train samples. In contrast, nowadays, with the rapid growth of large generative models such as DALL-E [58], CLIP [57], GPT-3 [9], it becomes relevant to perform out-of-sample estimation, e.g., map the latent vectors to new vectors which are not present in the train set to generate new data. Our IT approach (for w → ∞ ) is a theoretically justified way to learn approximate NN maps exclusively from samples. We think our approach might acquire applications here as well, especially since there already exist ways to apply OT in latent spaces of such models [20].

(3) Robustness and outlier detection. Our IT aligns the input measure P only with a part of the target measure Q . This property might be potentially used to make the learning robust, e.g., ignore outliers in the target dataset. Importantly, the choice of outliers and contamination level are tunable via c ( x, y ) and w , but their selection may be not obvious. At the same time, the potential f ∗ vanishes on the outliers, i.e., samples in Supp ( Q ) to which the mass is not transported.

Proposition 4 (The potential vanishes at outliers) . Under the assuptions of Theorem 5, the equality f ∗ ( y ) = 0 holds for all y ∈ Supp ( Q ) \ Supp ( T ∗ ♯ P ) .

We empirically illustrate this statement in Appendix A. As a result of this proposition, a possible byproduct of our method is an outlier-score f ∗ ( y ) for the target data. Such applications of OT are promising and there already exist works [52, 7, 54] developing approaches to make OT more robust.

Limitations, societal impact . We discuss limitations , societal impact of our study in Appendix A.

ACKNOWLEDGEMENTS. This work was partially supported by Skoltech NGP Program (Skoltech-MIT joint project).

References

D. Aggarwal, E. Valiyev, F. Sener, and A. Yao. Learning style compatibility for furniture. In German Conference on Pattern Recognition , pages 552-566. Springer, 2018.

C. D. Aliprantis and K. C. Border. Infinite dimensional analysis. Technical report, Springer, 2006.

A. Alotaibi. Deep generative adversarial networks for image-to-image translation: A review. Symmetry , 12(10):1705, 2020.

M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In International conference on machine learning , pages 214-223. PMLR, 2017.

A. Asadulaev, A. Korotin, V. Egiazarian, and E. Burnaev. Neural optimal transport with general cost functionals. arXiv preprint arXiv:2205.15403 , 2022.

A. Babenko, A. Slesarev, A. Chigorin, and V. Lempitsky. Neural codes for image retrieval. In European conference on computer vision , pages 584-599. Springer, 2014.

Y. Balaji, R. Chellappa, and S. Feizi. Robust optimal transport with applications in generative modeling and domain adaptation. Advances in Neural Information Processing Systems , 33: 12934-12944, 2020.

M. Bi´ nkowski, D. J. Sutherland, M. Arbel, and A. Gretton. Demystifying mmd gans. In International Conference on Learning Representations , 2018.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

L. A. Caffarelli and R. J. McCann. Free boundaries in optimal transport and monge-ampere obstacle problems. Annals of mathematics , pages 673-730, 2010.

L. Chapel, M. Z. Alaya, and G. Gasso. Partial optimal tranport with applications on positiveunlabeled learning. Advances in Neural Information Processing Systems , 33:2903-2913, 2020.

H. Chen, X. He, L. Qing, Y. Wu, C. Ren, R. E. Sheriff, and C. Zhu. Real-world single image super-resolution: A brief review. Information Fusion , 79:124-145, 2022.

X. Chen and C. Jia. An overview of image-to-image translation using generative adversarial networks. In International Conference on Pattern Recognition , pages 366-380. Springer, 2021.

L. Chizat. Unbalanced optimal transport: Models, numerical methods, applications . PhD thesis, Université Paris sciences et lettres, 2017.

Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8188-8197, 2020.

M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2014.

M. Daniels, T. Maunu, and P. Hand. Score-based generative neural networks for large-scale optimal transport. Advances in neural information processing systems , 34:12955-12965, 2021.

E. de Bézenac, I. Ayed, and P. Gallinari. Cyclegan through the lens of (dynamical) optimal transport. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part II 21 , pages 132-147. Springer, 2021.

L. V. Eyring, D. Klein, G. Palla, S. Becker, P. Weiler, N. Kilbertus, and F. Theis. Modeling single-cell dynamics using unbalanced parameterized monge maps. bioRxiv , 2022.

J. Fan, S. Liu, S. Ma, H.-M. Zhou, and Y. Chen. Neural monge map estimation and its applications. Transactions on Machine Learning Research , 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=2mZSlQscj3 . Featured Certification.

C. Fefferman, S. Mitter, and H. Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society , 29(4):983-1049, 2016.

A. Figalli. The optimal partial transport problem. Archive for rational mechanics and analysis , 195(2):533-560, 2010.

G. B. Folland. Real analysis: modern techniques and their applications , volume 40. John Wiley &Sons, 1999.

M. Gazdieva, L. Rout, A. Korotin, A. Kravchenko, A. Filippov, and E. Burnaev. An optimal transport perspective on unpaired image super-resolution. arXiv preprint arXiv:2202.01116 , 2022.

A. Genevay, M. Cuturi, G. Peyré, and F. Bach. Stochastic optimization for large-scale optimal transport. In Advances in neural information processing systems , pages 3440-3448, 2016.

A. Genevay, G. Peyré, and M. Cuturi. Learning generative models with sinkhorn divergences. In International Conference on Artificial Intelligence and Statistics , pages 1608-1617. PMLR, 2018.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems , pages 2672-2680, 2014.

I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems , pages 5767-5777, 2017.

N. Gushchin, A. Kolesov, A. Korotin, D. Vetrov, and E. Burnaev. Entropic neural optimal transport via diffusion processes. In Advances in Neural Information Processing Systems , 2023.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016.

P. Henry-Labordere. (martingale) optimal transport and anomaly detection with neural networks: A primal-dual algorithm. arXiv preprint arXiv:1904.04546 , 2019.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems , pages 6626-6637, 2017.

X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz. Multimodal unsupervised image-to-image translation. In Proceedings of the European conference on computer vision (ECCV) , pages 172-189, 2018.

Y. Jing, Y. Yang, Z. Feng, J. Ye, Y. Yu, and M. Song. Neural style transfer: A review. IEEE transactions on visualization and computer graphics , 26(11):3365-3385, 2019.

J. Johnson, M. Douze, and H. Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data , 7(3):535-547, 2019.

T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

W. Karush. Minima of functions of several variables with inequalities as side conditions. In Traces and Emergence of Nonlinear Programming , pages 217-245. Springer, 2014.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

A. Korotin, V. Egiazarian, A. Asadulaev, A. Safin, and E. Burnaev. Wasserstein-2 generative networks. In International Conference on Learning Representations , 2021. URL https: //openreview.net/forum?id=bEoxzW_EXsa .

A. Korotin, L. Li, A. Genevay, J. M. Solomon, A. Filippov, and E. Burnaev. Do neural optimal transport solvers work? a continuous wasserstein-2 benchmark. Advances in Neural Information Processing Systems , 34:14593-14605, 2021.

A. Korotin, A. Kolesov, and E. Burnaev. Kantorovich strikes back! wasserstein gans are not optimal transport? In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

A. Korotin, D. Selikhanovych, and E. Burnaev. Kernel neural optimal transport. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum? id=Zuc_MHtUma4 .

A. Korotin, D. Selikhanovych, and E. Burnaev. Neural optimal transport. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum? id=d8CBRlWNkqH .

Z. Li, Y. Xu, N. Zhao, Y. Zhou, Y. Liu, D. Lin, and S. He. Parsing-conditioned anime translation: A new dataset and method. ACM Transactions on Graphics , 42(3):1-14, 2023.

H. Liu, X. Gu, and D. Samaras. Wasserstein GAN with quadratic transport cost. In Proceedings of the IEEE International Conference on Computer Vision , pages 4832-4841, 2019.

Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , December 2015.

F. Lübeck, C. Bunne, G. Gut, J. S. del Castillo, L. Pelkmans, and D. Alvarez-Melis. Neural unbalanced optimal transport via cycle-consistent semi-couplings. arXiv preprint arXiv:2209.15621 , 2022.

M. Lucic, K. Kurach, M. Michalski, S. Gelly, and O. Bousquet. Are GANs created equal? a large-scale study. In Advances in neural information processing systems , pages 700-709, 2018.

A. Makkuva, A. Taghvaei, S. Oh, and J. Lee. Optimal transport mapping via input convex neural networks. In International Conference on Machine Learning , pages 6672-6681. PMLR, 2020.

Y. A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence , 42(4):824-836, 2018.

A. Mathur, A. Isopoussu, F. Kawsar, N. Berthouze, and N. D. Lane. Mic2mic: using cycleconsistent generative adversarial networks to overcome microphone variability in speech systems. In Proceedings of the 18th international conference on information processing in sensor networks , pages 169-180, 2019.

D. Mukherjee, A. Guha, J. M. Solomon, Y. Sun, and M. Yurochkin. Outlier-robust optimal transport. In International Conference on Machine Learning , pages 7850-7860. PMLR, 2021.

Q. H. Nhan Dam, T. Le, T. D. Nguyen, H. Bui, and D. Phung. Threeplayer Wasserstein GAN via amortised duality. In Proc. of the 28th Int. Joint Conf. on Artificial Intelligence (IJCAI) , 2019.

S. Nietert, Z. Goldfeld, and R. Cummings. Outlier-robust optimal transport: Duality, structure, and statistical analysis. In International Conference on Artificial Intelligence and Statistics , pages 11691-11719. PMLR, 2022.

Y. Pang, J. Lin, T. Qin, and Z. Chen. Image-to-image translation: Methods and applications. IEEE Transactions on Multimedia , 2021.

G. Peyré, M. Cuturi, et al. Computational optimal transport. Foundations and Trends® in Machine Learning , 11(5-6):355-607, 2019.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

R. T. Rockafellar. Integral functionals, normal integrands and measurable selections. In Nonlinear operators and the calculus of variations , pages 157-207. Springer, 1976.

O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention , pages 234-241. Springer, 2015.

L. Rout, A. Korotin, and E. Burnaev. Generative modeling with optimal transport maps. In International Conference on Learning Representations , 2022.

F. Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY , 55(58-63):94, 2015.

V. Seguy, B. B. Damodaran, R. Flamary, N. Courty, A. Rolet, and M. Blondel. Large scale optimal transport and mapping estimation. In International Conference on Learning Representations , 2018.

F. Terkelsen. Some minimax theorems. Mathematica Scandinavica , 31(2):405-413, 1972.

C. Villani. Optimal transport: old and new , volume 338. Springer Science & Business Media, 2008.

K. D. Yang and C. Uhler. Scalable unbalanced optimal transport using generative adversarial networks. In International Conference on Learning Representations , 2018.

A. Yu and K. Grauman. Fine-grained visual comparisons with local learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 192-199, 2014.

F. Yu, A. Seff, Y . Zhang, S. Song, T. Funkhouser, and J. Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015.

Y. Yuan, S. Liu, J. Zhang, Y. bing Zhang, C. Dong, and L. Lin. Unsupervised image superresolution using cycle-in-cycle generative adversarial networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 814-81409, 2018.

R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , 2018.

X. Zhang, D. Zhai, T. Li, Y. Zhou, and Y. Lin. Image inpainting based on deep learning: A review. Information Fusion , 2022.

J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision , pages 2223-2232, 2017.

A Limitations

Transport costs . Our theoretical results hold true for any continuous cost function c ( x, y ) , but our experimental study uses ℓ 2 as it already yields a reasonable performance in many cases. Considering more semantically meaningful costs for image translation, e.g., perceptual [70], is a promising future research direction.

Intersecting supports . ET is the nearest neighbor assignment ( M 3.1). Using ET may be unreasonable when X = Y and Supp ( P ) intersects with Supp ( Q ) . For example, if c ( x, y ) attains minimum over y ∈ Y for a given x ∈ X at x = y , e.g., c = ℓ 2 , then there exists a ET plan satisfying π ∗ ( y | x ) = δ x for all x ∈ Supp ( P ) ∩ Supp ( Q ) . It does not move the mass of points x in this intersection. We provide an illustrative toy 2D example in Fig. 10b.

Limited diversity. It is theoretically impossible to preserve input-output similarity better than ET maps. Still one should understand that in some cases these maps may yield degenerate solutions. In Fig. 10a, we provide a toy 2D example of an IT map ( w = 8 ) which maps all inputs to nearly the same point. In Fig. 9a ( texture → chair translation), we see that with the increase of w the IT map produces less small chairs but more large armchairs. In particular, when w = 8 , only armchairs appear, see Fig. 9a. This is because they are closer (in ℓ 2 ) to textures due to having smaller white background area.

1.5

Input x~ P

Mapped y=T(x)

1.0

。

TargetQ

0.5

0.0

0.5

1.0

1.5

-3

2

-1

0

1

-3

2

-1

0

1

1.5

Inputx~P

Input x~P

1.0

D

TargetQ

Mapped y=T(x)

0.5

0.0

0.5

1.0

8

8

8

6

1.5

1.5 1.00.5

0.0

0.5

1.0

1.5-1.5 -1.0 0.5

0.0

0.5

1.0

1.5

Unused samples. Doing experiments, we noticed that the model training slows down with the increase of w . A possible cause of this is that some samples from Q become non-informative for training (this follows from our Proposition 4). Intuitively, the part of Supp ( Q ) to which the samples of P will not be mapped to is not informative for training. We illustrate this effect on toy 'Wi-Fi' example and plot the histogram of values of f ∗ in Fig. 11. One possible negative of this observation is that the training of IT maps or, more generally, partial OT maps, may naturally require larger training datasets.

(a) Surface of

f

∗

(trained with

w

= 1

).

(b) Surface of

f

∗

(trained with

w

=

3

).

(c) Surface of

f

∗

(trained with

w

= 3

).

of

f

∗

(

y

)

for

y

∼

Q

.

2.0

1.5

Mappedy=T（x)

Inputx~P

Mapped y=T(x)

Mappedy=T(x)

0.00

0.45

17.5

15.0

W = 1

W = 3/2

-0.90

W =3

1.0

1.35

12.5

10.0

0.5

-1.80

-2.25

7.5

0.0

2.70

5.0

0.5

3.15

2.5

1.0 +

-3.60

0.0

1.0 0.5

0.0

0.5

1.0

1.5

2.0-1.0-0.50.0

0.5

1.0

1.5

2.0 1.00.5

0.0

0.5

1.0

1.5

2.0

-1.0 -0.8-0.6 -0.4-0.20.0

f*(y)

2

Limited quantitative metrics. In experiments ( M 5), we use a limited amount of quantitative metrics. This is because existing unpaired metrics, e.g., FID [32], KID [8], etc., are not suitable for our setup . They aim to test equalities, such as T♯ P = Q , while our learned maps disobey it by the construction (they capture only a part of Q ). Developing quality metrics for partial generative models is an important future research direction. Meanwhile, for our method, we have provided a toy 2D analysis ( M 5.1) and explanatory metrics ( M 5.2), such as the transport cost (Table 1a).

Fake solutions and instabilities . Lagrangian objectives such as (15) may potentially have optimal saddle points ( f ∗ , T ∗ ) in which T ∗ is not an OT map. Such T ∗ are called fake solutions [42] and may be one of the causes of training instabilities. Fake solutions can be removed by considering OT with strictly convex weak costs functions [43, Appendix H], see Appendix G.2 for examples.

Inexistent IT maps. The actual IT plans between P , Q may be non-deterministic, while our approach only learns a deterministic map T . Nevertheless, thanks to our Proposition 3, for every ϵ > 0 there always exists a 1-to-1 map T ϵ ♯ P ≤ w Q which provides ϵ -sub-optimal cost ∫ X c ( x, T ϵ ( x ) ) d P ( x ) ≤ Cost w ( P , Q ) + ϵ . Thus, IT cost (12) can be approached arbitrary well with deterministic transport maps. A potential way to modify our algorithm is to learn stochastic plans is to add random noise z to generator T ( x, z ) as input, although this approach may suffer from ignoring z , see [43, M 5.1].

Potential societal impact . Neural OT methods and, more generally, generative models are a developing research direction. They find applications such as style translation and realistic content generation. We expect that our method may improve existing applications of generative models and add new directions of neural OT usage like outlier detection. However, it should be taken into account that generative models can also be used for negative purposes such as creating fake faces.

B Toy 2D Illustrations of Other Methods

In this section, we demonstrate how the other methods perform in 'Wi-Fi' and 'Accept' experiments. We start with 'Wi-Fi' . Assume that we would like to map P to the closest 2 3 -rd fraction of Q , i.e., we aim to learn 2 of 3 arcs in Q , (as in Fig. 7c).

In Fig. 12d, we show the discrete partial OT (4) [11] with parameters w 0 = m = 1 , w 1 = 3 2 , corresponding to IT (12) with w = 3 2 . To obtain the discrete matching, we run ot.partial.partial_wasserstein2 from POT 5 . As expected, it matches the input P with 2 3 of Q and can be viewed as the ground truth (coinciding with our Fig. 7c).

First, we show the GAN [27] endowed with additional ℓ 2 loss with weight λ = 0 . 5 (in Fig. 12a). Next, we consider discrete unbalanced OT [14] with the quadratic cost c = ℓ 2 . In Fig. 12b, we show the results of the matching obtained by ot.unbalanced with parameters m = 1 , reg = 0 . 1 , reg m = 1 , numItermax = 200000 . Additionally, in Fig. 12c we show the result of neural unbalanced OT method [66]. 6 To make their unbalanced setup maximally similar to our IT, we set to zero their regularization parameters. The rest parameters are default except for λ 0 = 0 . 02 ( ℓ 2 loss parameter), λ 2 = 5 (input and target measures' variation parameter).

We see that GAN + ℓ 2 (Fig. 12a) and unbalanced OT (Fig. 12b and 12c) indeed match P with only a part of the target measure Q . The transported mass is mostly concentrated in the two small arcs of Q which are closest to P w.r.t. ℓ 2 cost. The issue here is that some mass of P spreads over the third (biggest) arc of Q yielding outliers . This happens because unbalanced OT (GAN can be viewed as its particular case) is an unconstrained problem: the mass spreading is controlled via soft penalization ( f -divergence loss term). The lack of hard constraints, such as those in partial OT (4) or IT (12), makes it challenging to strictly control how the mass in unbalanced OT actually spreads.

For completeness, we also show the results of these methods applied to 'Accept' experiment, see Fig. 13. Here we tested various hyperparameters for these methods but did not achieve the desired behaviour, i.e., learning only the text 'Accept' . Moreover, we noted that GAN + ℓ 2 for large λ yields undesirable artifacts (Fig. 13a). This is because GAN and ℓ 2 losses contradict to each other and still the models tried to minimize them both. We further discuss in Appendix C below.

(a) GAN

+

ℓ

2

.

(b) Discrete unbalanced

OT.

(c) Scalable unbalanced

OT.

(d) Discrete Partial OT (desired ground truth).

(a) GAN

+

ℓ

2

.

(b) Discrete unbalanced

OT.

(c) Scalable unbalanced

OT.

(d) Discrete Partial OT

(desired ground truth).

1.5

Mapped y=T(x)

Mappedy=T(x)

Mapped y=T(x)

Mapped y=T(x)

1.0

0.5

0.0

-0.5

-1.0-0.5

0.0

0.5

1.0

1.5

2.0-1.0-0.5

0.0

0.5

1.0

1.5

2.0-1.0-0.5

0.0

0.5

1.0

1.5

2.0-1.0-0.5

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

-2.0

2.5

Mappedy=T(x)

Mapped y=T(x)

Mapped y=T(x)

Mapped y=T(x)

1.0

-0.5

0.0

0.5

1.0

-1.0

0.5

0.0

0.5

1.0

-1.0

-0.5

0.0

0.5

1.0

-1.0

-0.5

0.0

0.5

1.0

C Comparison with Other Image-to-Image Translation Methods

Recall that ET by design is the best translation between a pair of domains w.r.t. the given dissimilarity function c ( x, y ) . Our IT maps with the increase of w provide better input-output similarity and recover ET when w →∞ ( M 3.3). A reader may naturally ask: (a) How else can we recover ET maps? (b) To which extent one can control the input-output similarity in existing translation methods? (c) Can these methods be used to approximate ET? We discuss these aspects below.

Many translation methods are based on GANs, see [55, 3, 13] for a survey. Their learning objectives are usually combined of several loss terms:

In (17), the domain loss L Dom is usually the vanilla GAN loss involving a discriminator [27] ensuring that the learned map x ↦→ T ( x ) transforms inputs x ∼ P to the samples from the target data distribution Q . The similarity loss L Sim (with λ ≥ 0 ) is usually the identity loss ∫ X ∥ x -T ( x ) ∥ 1 d P ( x ) . More generally, it can be an arbitrary unsupervised loss of the form ∫ X c ( x, T ( x ) ) d P ( x ) stimulating the output sample T ( x ) to look like the input samples x w.r.t. given dissimilarity function c , e.g., ℓ 1 , ℓ 2 , ℓ p , perceptual, etc. The other terms in (17) involve model-specific terms (e.g., cycle consistency loss in CycleGAN) which are not related to our study.

When learning a model via optimizing (17), a natural way to get better similarity of x and T ( x ) in (17) is to simply increase weight λ of the corresponding loss term. This is a straightforward approach but it has a visible limitation . When λ is high, the term λ · L Sim dominates over the other terms such as L Dom, and the model T simply learns to minimize this loss ignoring the fact that the output sample should be from the target data distribution Q . In other words, in (17) there is a nasty trade-off between T ( x ) belonging to the target data distribution Q and input-output similarity of x and T ( x ) . The parameter λ ≥ 0 controls this realism-similarity trade-off, and we study how it affects the learned map T below.

We pick CycleGAN [72] as a base model for evaluation since it is known as one of the principal models to solve the unpaired translation problem. We use c = ℓ 1 as the similarity loss as the CycleGAN's authors originally used in their paper . 7 We consider parameter λ ∈ [0 , 50 , 100 , 200 , 250 , 300 , 350 , 500] .

Additionally, we perform comparison with a more recent StarGAN-v2 [15] model. By default, StarGAN-v2 does not use any similarity loss and does not enforce the output to be similar to the input. Therefore, analogously to CycleGAN, we endow the model with an additional ℓ 1 similarity loss and consider λ ∈ [0 , 1 , 10 , 50 , 100 , 200 , 500] .

We train both models on celeba → anime ( 64 × 64 ) and handbags → shoes ( 128 × 128 ) translation with various λ and report the qualitative and quantitative results below. In all the cases, we report both ℓ 2 and ℓ 1 transport costs and FID on the test samples. Tables 2, 3, 4, 5 show transport costs and FID of GANs. FID and ℓ 2 metrics for our method are given in the main text (Tables 1b, 1a) and ℓ 1 cost is given in Table 6 below. For convenience, we visualize ( FID , Cost ) pairs for our method and GANs in Fig. 14.

Results and discussion (CycleGAN). Interestingly, we see that for CycleGAN adding small identity loss λ = 50 yields not only decrease of the transport cost (compared to λ = 0 ), but some improvement of FID as well. Still we see that the transport cost in CycleGAN naturally decreases with the increase of weight λ . Unfortunately, this decrease is accompanied by the decrease of the visual image quality, see Fig. 15. While for large λ the cost for CycleGAN is really small, the model is practically useless since it poors image quality. For very large λ , CycleGAN simply learn the identity map, as expected.

For λ providing acceptable visual quality , CycleGAN yields a transport cost which is bigger than that of standard OT ( w = 1) . Our result for w = 8 is unachievable for it. Note that in most cases FID of our IT method is smaller than that of CycleGAN.

Results and discussion (StarGAN-v2). In the celeba → anime experiment, StarGAN-v2 results are similar to CycleGAN ones. Our IT with w = 2 easily provides smaller transport cost (better similarity) than StarGAN-v2. In the handbags → shoes translation, we have encountered surprising observations. We see that starting from λ = 10 the model fails to translate some of the handbags to shoes, i.e., these handbags remain nearly unchanged. We notice the similar behaviour for vanilla GAN in 'Accept' experiment, see Fig. 13a. This explains the low cost for StarGAN-v2 model ( λ ≥ 10 ). Surprisingly to us, for λ = 10 , FID metric is also low despite the fact that model frequently produces failures , see the highlighted results in Fig. 16. Note that while FID is a widely used metric, it still could produce misleading estimations which we observe in the latter case.

Additionally, we provide a large set of randomly generated images for λ = 10 to qualitatively show that the stated issue is indeed notable, see Fig. 17. These failures demonstrate the limited practical usage of the model. Our IT method does not suffer from this issue which we qualitatively demonstrate on the same set of images for different weights w , see Fig. 25.

(a) Test

ℓ

2

transport cost.

(b) Test

ℓ

1

transport cost.

celeba→anime

handbag→shoes

60

50

CycleGAN

StarGAN-v2

=8

。

CycleGAN

StarGAN-v2

40

QA=10

IT (ours)

入

。

一

250

IT (ours)

FID

入=100

30

20

-8

A=1

入=50

入=0

入=10

=

入=50

入-0

入=0

10

=1

0.0

0.1

0.2

0.3

0.4

0.0

0.2

0.4

0.6

Q²cost

cost

celeba→anime

handbag→shoes

。

CycleGAN

StarGAN-v2

w=80

。

.

CycleGAN

StarGAN-v2

IT (ours)

IT (ours)

入=10

A-100

8-m

入=50

Q

-10

入-50

入=0

A-0

w=1

0.0

0.2

0.4

0.6

0.1

0.2

0.3

0.4

0.5

elcost

elcost

Concluding remarks. Our algorithm with w →∞ allows us to achieve better similarity without the decrease of the image quality. At the same time, GANs fail to do this when λ →∞ . Why?

We again emphasize that typical GAN objective (17) consists of several loss terms. Each term stimulates the model to attain certain properties (realism, similarity to the input, etc.). These terms, in general, contradict each other as they have different minima T . This yields the nasty trade-off between the loss terms. Our analysis shows that conceptually there is no significant difference between CycleGAN and a more recent StarGAN-v2 model. More generally, any GAN-based method inherits realism-similarity tradeoff issue. GANs' differences are mostly related to the use of other architectures or additional losses. Therefore, we think that additional comparisons are excessive since they may not provide any new insights.

In contrast, our method is not a sum of losses . Our objective (15) may look like a direct sum of a transport cost with an adversarial loss; our method does have a generator (transport maps T ) and a discriminator (potential f ) which are trained via the saddle-point optimization objective max f min T . Yet, this visible similarity to GAN-based methods is deceptive . Our objective max f min T can be viewed as a Lagrangian and is atypical for GANs: the generator is in the inner optimization problem min T while in GANs the objective is min T max f . In our case, similar to other neural dual OT methods, the generator is adversarial to the discriminator but not vice versa, as in GANs. Please consult [43, M 4.3], [24] or [20] for further discussion about OT methods.

GANs aim to balance loss terms L Dom and L Sim. Our optimization objective enforces the constraint T♯ P ≤ w Q via learning the potential f (a.k.a. Lagrange multiplier) and among admissible maps T searches for the one providing the smallest transport cost. There is no realism-similarity trade-off. For completeness, we emphasize that when w →∞ , FID in Table 1b does not drop because of the decrease of the image quality, but because our method covers the less part of Q . FID negatively reacts to this [48, Fig. 1b].

D Relation and Comparison with Discrete Partial OT Methods

The goal of domain translation is to recover the map x ↦→ T ( x ) between two domains P , Q . We approach this problem by approximating T with a neural network trained on the empirical samples X = { x 1 , . . . , x N } , Y = { y 1 , . . . , y M } , i.e., train datasets. Our method generalizes to new (previously unseen, test) input samples x new ∼ P , i.e, our learned map ̂ T can be applied to new input samples to generate new target samples ̂ T ( x new ) .

In contrast, discrete OT methods (including discrete partial OT) perform a matching between the empirical distributions ̂ P N = ∑ N n =1 δ x n , ̂ Q M = ∑ M m =1 δ y m . Thus, they do not provide out-ofsample estimation of the transport plan π ( y | x new ) or map T ( x new ) . The reader may naturally wonder: why not to interpolate the solutions of discrete OT? For example, a common strategy is to derive barycentric projections T ( x ) = ∫ Y ydπ ( y | x ) of the discrete OT plan π and then learn a network T θ ( x ) ≈ T ( x ) to approximate them [63]. Below we study this approach and show that it does not provide reasonable performance.

～x

×～P

(x)1

0=

0=

T(x)。

入=50

入=50

(x)1

入=500 入=350入=300入=250 入=200入=100

(x)

入=100

(x)

T(x)

T(x)

=300 入=200

T(x)

(x)

T(x)

=500

(x)I

(x)↓

(a)

Celeba

(female)

→

anime

(64

×

64).

(b) Handbags → shoes (128 × 128).

d~x

d~x

0=

0=V

(x)

[=

0=

(x

入=10

OS=Y

(x)

(x)

0S=

00L=

(x)

A=100

入=200

(x)

T(x)

入=500

00S=V

T(x)

(x

V

(a) Celeba (female) → anime (64 × 64).

(b) Handbags → shoes (128 × 128).

d~x

T(x,z)

OI=Y

T(x,z)

OI=V

T(x,z)

入=10

We perform evaluation of the BP approach on celeba → anime translation. We solve the discrete partial OT (DOT) between the parts of the train data X (11K samples), Y (50K samples). 8 We use ot.partial.partial_wasserstein algorithm with parameters w 0 = m = 1 and vary w 1 ∈ { 1 , 2 , 4 , 8 } , see the notation in (4). This corresponds to our IT (12) with w = w 1 . Then we regress a UNet T θ to recover the barycentric projection T . We also simulate the case of w = ∞ by learning the discrete NNs in the train dataset using NearestNeighbors algorithm from sklearn.neighbors . We experiment with using MSE or VGG-based perceptual error 9 as the loss function for regression. We visualize the obtained results in Fig. 18 and report average ℓ 2 test transport cost, FID in Table 7.

BP methods are known not to work well in large scale problems such as unpaired translation because of the averaging effect , see Figure 3 and large FID values of BP in Table 1 of [17]. Indeed, one may learn a barycentric projection network T θ ( x ) ≈ ∫ Y ydπ ( y | x ) over the discrete OT plan π ,

yet it is clear that ∫ Y ydπ ( y | x ) is a direct weighted average of several images y , i.e., some blurry average image of poor quality. It is not satisfactory for practical applications.

Our qualitative results indeed show that BP approach for small w leads to the averaging effect . We see that on the train dataset this effect disappears with the increase of w , see Fig. 18b. It is expected, since with the increase of weight the conditional distribution of a plan π ( y | x ) tends to a degenerate distribution concentrated at the nearest neighbor NN ( x ) of x in the train dataset, i.e., π ( y | x ) ≈ δ NN ( x ) when w →∞ . This means, that in the limit the barycentric projection in point x is its nearest neighbor NN ( x ) .

However, the learned network ( w = ∞ ) does not generalize well to unseen test samples and produces images of insufficient quality which is much worse than for the train samples, see Fig. 18a. Despite the fact that test ℓ 2 cost and FID decrease with the increase of w , see

175

DOT+BP

DOT+BP(perc.)

150

IT(ours)

125

D

100

75

50

25

0.10

0.15

0.20

0.25

0.30

Q2

cost

d~x

d~x

(x)↓

W=1

(x)

W=1

W=2

（x)↓

乙=M

(x)

W=4

(x)。

W=4

(X)

8=M

“(x)↓

8=M

(X)↓

∞0=M

(X)↓

W=∞

(x)(perc.)

(x)(perc.)

∞=M

∞=M

(a) Results on test

set.

(b) Results on train set.

Table 7, Fig. 19, generated test images have poor quality for all weights w . Yet, the FID scores as well as the ℓ 2 costs are much bigger than in our method in all the cases.

As the MSE error for regression is known not to work well in image tasks, we also conduct an experiment using the perceptual error function to learn T θ . Unfortunately, training the neural network with perceptual error does not lead to meaningful improvements, see the bottom lines of Fig. 18b, 18a, Table 7. This confirms that the origins of DOT+BP method's poor performance lie not in the regression part, but in the entire methodology based on barycentric projections.

To conclude, discrete OT methods are not competitors to our work as there is no straightforward and well-performing way to make out-of-sample DOT estimation in large scale image processing tasks.

E Experimental Details

Pre-processing. For all datasets, we rescale images' RGB channels to [-1, 1]. As in [40], we rescale aligned anime face images to 512 × 512. Then we do 256 × 256 crop with the center located 14 pixels above the image center to get the face. Finally, for all datasets except the textures dataset, we resize the images to the required size (64 × 64 or 128 × 128). We apply random horizontal flipping augmentation to the comic faces and chairs datasets. Analogously to [42], we rescale describable textures to minimal border size of 300, do the random resized crop (from 128 to 300 pixels) and random horizontal, vertical flips. Next, we resize images to the required size (64 × 64 or 128 × 128).

Neural networks. In M 5.1, we use fully connected networks both for the mapping T θ and potential f ψ . In M 5.2, we use UNet [60] architecture for the transport map T θ . We use WGAN-QC's [45] ResNet [30] discriminator as a potential f ψ . We add an additional final layer x ↦→ -| x | to f ψ to make its outputs non-positive.

Optimization. We employ Adam [38] optimizer with the default betas both for T θ and f ψ . The learning rate is lr = 10 -4 . We use the MultiStepLR scheduler which decreases lr by 2 after [(5+5 · w )K, (20+5 · w )K, (40+5 · w )K, (70+5 · w )K] iterations of f ψ where w ∈ { 1 , 2 , 4 , 8 } is a weight parameter. The batch size is | X | = 256 for toy 'Wi-Fi' , | X | = 4096 for toy Accept, and | X | = 64 for image-to-image translation experiments. The number of inner T θ iterations is k T = 10 . In toy experiments, we observe convergence in ≈ 30K total iterations of f ψ for 'Wi-Fi' , in ≈ 70K for Accept. In the image-to-image translation, we do ≈ 70K iterations for 128 × 128 datasets, ≈ 40K iterations for 64 × 64 datasets. In the experiments with image-to-image translation experiments, we gradually increase w for 20K first iterations of f ψ . We start from w = 1 and linearly increase it to the desired w (2, 4 or 8).

Computational complexity . The complexity of training IT maps depends on the dataset, size of images and weight w . Convergence time increases with the increase of w : possible reasons for this are discussed in the unused samples limitation (Appendix A). In general, it takes from 2 (for w = 1 ) up to 6 (for w = 8 ) days on a single Tesla V100 GPU.

Reproducibility. We provide the code for the experiments and will provide the trained models, see README.md .

F Proofs

Proof of Proposition 1. Since continuous c is defined on a compact set X × Y , it is uniformly continuous on X × Y . This means that there exists a modulus of continuity ω : [0 , + ∞ ) → [0 , ∞ ) such that for all ( x, y ) , ( x ′ , y ′ ) ∈ X × Y it holds

and function ω is monotone, continuous at 0 with ω (0) = 0 . In particular, for y = y ′ we have | c ( x, y ) -c ( x ′ , y ) | ≤ ω ( || x -x ′ || X ) . Thus, for c ∗ ( x ) = min y ∈ Supp ( Q ) c ( x, y ) , we have | c ∗ ( x ) -c ∗ ( x ′ ) | ≤ ω ( || x -x ′ || X ) , see [62, Box 1.8]. This means that c ∗ : X → R is (uniformly) continuous.

Proof of Theorem 1. Since function ( x, y ) ↦→ c ( x, y ) is continuous, there exists a measurable selection T ∗ : X → Y from the set-valued map x ↦→ arg min y ∈ Supp ( Q ) c ( x, y ) , see [2, Theorem 18.19]. This map for all x ∈ X satisfies c ( x, T ( x ) ) = min y ∈ Supp ( Q ) c ( x, y ) = c ∗ ( x ) . As a result,

Lemma 1. (Distinctness) Let µ, ν ∈ M ( Y ) . Then µ ≤ ν holds if and only if for every f ∈ C ( Y ) satisfying f ≤ 0 it holds that ∫ Y f ( y ) dµ ( y ) ≥ ∫ Y f ( y ) dν ( y ) .

Proof of Lemma 1. If µ ≤ ν , then the inequality ∫ Y f ( y ) d ( µ -ν )( y ) ≥ 0 for every (measurable) f ≤ 0 follows from the definition of the Lebesgue integral. Below we prove the statement in the other direction.

Assume the opposite, i.e., ∫ Y f ( y ) d ( µ -ν )( y ) ≥ 0 for every continuous f ≤ 0 but still ν ≱ µ . The latter means there exists a measurable A ⊂ Y satisfying µ ( A ) > ν ( A ) . Let ϵ = 1 2 ( µ ( A ) -ν ( A ) ) > 0 . Consider the negative indicator function f A ( y ) which equals -1 if y ∈ A and 0 when y / ∈ A . Consider a variation measure | ν -µ | ∈ M + ( Y ) . Thanks to [23, Proposition 7.9], the continuous functions C ( Y ) are dense in the space L 1 ( | µ -ν | ) . Therefore, there exists a function f A,ϵ ∈ C ( Y ) satisfying ∫ X | f A ( y ) -f A,ϵ ( y ) | d ∣ ∣ µ -ν | ( y ) < ϵ. We define f -A,ϵ ( y ) def = min { 0 , f A,ϵ ( y ) } ≤ 0 . This is a non-positive continuous function, and for y ∈ Y it holds that | f A ( y ) -f A,ϵ ( y ) | ≥ | f A ( y ) -f -A,ϵ ( y ) | because f A takes only non-positive values { 0 , -1 } . We derive

which is a contradiction to the fact that ∫ Y f ( y ) d ( µ -ν )( y ) ≥ 0 for every continuous f ≤ 0 . Thus, µ ≤ ν .

Proof of Proposition 2. To begin with, we prove that Π w ( P , Q ) is a weak-* compact set. Pick any sequence π n ∈ Π w ( P , Q ) . It is bounded as all π n are probability measures ( ∥ π n ∥ 1 =1 ). Hence by the Banach-Alaoglu theorem [62, Box 1.2], there exists a subsequence π n k weakly-∗ converging to some π ∈ M ( X × Y ) . It remains to check that π ∈ Π w ( P , Q ) . Let ( µ n k , ν n k ) denote the marginals

of π n k and ( µ, ν ) be the marginals of π . Pick any f ∈ C ( Y ) with f ≤ 0 . Since π n k ∈ Π w ( P , Q ) , it holds that 0 ≤ ν n k ≤ w Q , w ∫ Y f ( y ) d Q ( y ) ≤ ∫ Y f ( y ) dν n k ( y ) ≤ 0 (Lemma 1). We have

The latter limit is ≥ w ∫ Y f ( y ) d Q ( y ) and ≤ 0 . As this holds for every continuous f ≤ 0 , we conclude that 0 ≤ ν ≤ w Q . By the analogous analysis one may prove that µ = P and π ≥ 0 . Thus, π ∈ Π w ( P , Q ) and Π w ( P , Q ) is a weak-* compact.

The functional π ↦→ ∫ X×Y c ( x, y ) dπ ( x, y ) is continuous in the space M ( X × Y ) equipped with weak-∗ topology because c : X ×Y → R is continuous. Since Π w ( P , Q ) is a compact set, there exists π ∗ ∈ Π w ( P , Q ) attaining the minimum on Π w ( P , Q ) . This follows from the Weierstrass extreme value theorem and ends the proof.

Bibliographical remark. The results showing the existence of minimizers π ∗ in partial OT (4) already exist, see [10, Lemma 2.2] or [22, M 2]. They also provide the existence of minimizers in our IT problem (12). Yet, the authors study the particular case when P , Q have densities on X , Y ⊂ R D . For completeness , we include a separate proof of existence as we do not require the absolute continuity assumption. The proof is performed via the usual technique based on weak-* compactness in dual spaces and is analogous to [62, Theorem 1.4] which proves the existence of minimizers for OT problem (2). Our proof is slightly more technical due to the inequality constraint.

Proof of Proposition 3. Let π ∗ ∈ Π w ( P , Q ) be an IT plan. Consider the OT problem between P and π ∗ y :

It turns out that π ∗ is a minimizer here. Assume the contrary, i.e., that there exists a more optimal π ′ ∈ Π( P , π ∗ y ) satisfying

This plan by the definition of Π( P , π ∗ y ) satisfies π ′ x = P and π ′ y = π ∗ y ≤ w Q , i.e., π ′ ∈ Π w ( P , Q ) . However, (19) contradicts the fact that π ∗ is an IT plan in (18) as π ′ provides smaller cost. Thus, min in (18) equals Cost w ( P , Q ) in (12).

Thanks to [62, Theorem 1.33], problem (18) has the same minimal value as the inf in the Monge's problem

i.e., for every ϵ > 0 there exists T ϵ : X → Y satisfying T ϵ ♯ P = π ∗ y and ∫ X c ( x, T ϵ ( x ) ) d P ( x ) < Cost w ( P , Q ) + ϵ . It remains to substitute this T ϵ to Monge's IT problem (11) to get an ϵ -close transport cost to Kantorovich's IT cost (12). As this works for every ϵ > 0 , we conclude that min in (12) is the same as inf in (11).

Proof of Theorem 2. The fact that w ↦→ Cost w ( P , Q ) is non-increasing follows from the inclusion Π w 1 ( P , Q ) ⊂ Π w 2 ( P , Q ) for w 1 ≤ w 2 . This inclusion means that for larger values of w , the minimization in (12) is performed over a larger set of admissible plans. As for convexity, take any IT plans π w 1 ∈ Π w 1 ( P , Q ) , π w 2 ∈ Π w 2 ( P , Q ) for w 1 , w 2 , respectively. For any α ∈ [0 , 1] consider the mixture π ′ = απ w 1 +(1 -α ) π w 2 . Note that π ′ y = απ w 1 y +(1 -α ) π w 2 y ≤ αw 1 Q +(1 -α ) w 2 Q = ( αw 1 +(1 -α ) w 2 ) Q . Therefore, π ′ ∈ Π αw 1 +(1 -α ) w 2 ( P , Q ) . We derive

which shows the convexity of w ↦→ Cost w ( P , Q ) .

Now we prove that lim w →∞ Cost w ( P , Q ) = Cost ∞ ( P , Q ) . For every w ≥ 1 and π ∈ Π w ( P , Q ) , it holds that π y ≤ w Q . This means that Supp ( π y ) ⊂ Supp ( Q ) . As a result, we see that Π w ( P , Q ) ⊂ Π ∞ ( P , Q ) , i.e., Cost w ( P , Q ) ≥ Cost ∞ ( P , Q ) . We already know that w ↦→ Cost w ( P , Q ) is nonincreasing, so it suffices to show that for every ϵ > 0 there exists w = w ( ϵ ) ∈ [1 , + ∞ ) such that Cost w ( P , Q ) ≤ Cost ∞ ( P , Q ) + ϵ . This will provide that lim w →∞ Cost w ( P , Q ) = Cost ∞ ( P , Q ) .

Pick any ϵ > 0 . Consider the set S def = { ( x, y ) ∈ X × Y such that y ∈ NN ( x ) } . It is a compact set. To see this, we pick any sequence ( x n , y n ) ∈ S . It is contained in compact X × Y . Therefore, it has a sub-sequence ( x n k , y n k ) converging to some ( x, y ) ∈ X × Y . It remains to check that ( x, y ) ∈ S . Since Supp ( Q ) is compact and y n k ∈ NN ( x n k ) ⊂ Supp ( Q ) , we have y ∈ Supp ( Q ) as well. At the same time, by the continuity of c ∗ (Proposition 1) and c , we have

which means that y ∈ NN ( x ) and ( x, y ) ∈ S , i.e., S is compact.

Since ( x, y ) ↦→ c ( x, y ) -c ∗ ( x ) is a continuous function, for each ( x, y ) ∈ S there exists an open neighborhood U x × V y ⊂ X × Y of ( x, y ) such that for all ( x ′ , y ′ ) ∈ U x × V y it holds that c ( x ′ , y ′ ) -c ∗ ( x ′ ) < ϵ or, equivalently, c ( x ′ , y ′ ) < c ∗ ( x ′ ) + ϵ . Since ⋃ ( x,y ) ∈S U x × V y is an open coverage of the compact set S , there exists a finite sub-coverage ⋃ N n =1 U x n × V y n of S . In particular, X = ⋃ N n =1 U x n . For convenience, we simplify the notation and put U n def = U x n and V n def = V y n . Now we put U ′ 1 def = U 1 and iteratively define U ′ n def = U n \ U ′ n -1 for n ≥ 2 . By the construction, it holds that the entire space X is a disjoint union of U ′ n , i.e., X = ⊔ N n =1 U ′ n . Some of U ′ n may be empty, so we just remove them from the sequence and for convenience assume that each U ′ n is not empty. Now consider the measure π ∈ P ( X × Y ) which is given by

Here for µ, ν ∈ M + ( X ) , M + ( Y ) , we use × to denote their product measure µ × ν ∈ M ( X × Y ) . In turn, for a measurable A ⊂ X , we use µ | A to denote the restriction of µ to A , i.e., measure µ ′ ∈ M ( X ) satisfying µ ′ ( B ) = µ ( A ∩ B ) for every measurable B ⊂ X . Note that ∑ N n =1 P | U ′ n = P and ∑ N n =1 P ( U ′ n ) = ∑ N n =1 P | U ′ n ( U ′ n ) = 1 by the construction of U ′ n . At the same time, for each n it holds that Q | Vn Q ( V n ) is a probability measure because of the normalization Q ( V n ) . Note that this normalization is necessarily positive because V n is a neighborhood of a point in Supp ( Q ) . Therefore, since sets U ′ n are disjoint and cover X , we have π x = ∑ N n =1 P | U ′ n = P . Now let us show that there exists w such that π y ≤ w Q . It suffices to take w = ∑ N n =1 P ( U ′ n ) Q ( V n ) . Indeed, in this case for every measurable A ⊂ Y we have

which yields π y ≤ w Q and means that π ∈ Π w ( P , Q ) for our chosen w . Now let us compute the cost of π :

To finish the proof it remains to note that this plan is not necessarily a minimizer for (12), i.e., ∫ X×Y c ( x, y ) dπ ( x, y ) is an upper bound on Cost w ( P , Q ) . Therefore, we have Cost w ( P , Q ) ≤ Cost ∞ ( P , Q ) + ϵ for our chosen w = w ( ϵ ) .

Bibliographical remark. There exist seemingly related but actually different results in the fundamental OT literature, see [22, Lemma 2.1] or [10, M 3]. There the authors study partial OT problem (4) and study how the partial OT plan and OT cost evolve when the marginals w 0 P and w 1 Q are fixed and the required mass amount to transport changes from 0 to min { w 0 , w 1 } . In our study, the first marginal w 0 P and the amount of mass to transport m are fixed ( w 0 = m = 1 ), and we study how the OT cost changes when w 1 →∞ in the IT problem.

Proof of Theorem 3. Note that P ( X ×Y ) is (weak-*) compact. This can be derived from the BanachAlaoglu theorem analogously to the compactness of Π w ( P , Q ) in the proof of Theorem (2). Therefore, any sequence in P ( X × Y ) has a converging sub-sequence. In our case, for brevity, we assume that π w n ∈ Π w n ( P , Q ) itself weakly-* converges to some π ∗ ⊂ P ( X × Y ) . Since π w n x = P for all n , we also have π ∗ x = P . As lim n →∞ w n = ∞ , we conclude from Theorem 2 that

where the last equality holds since π w n (weakly-*) converges to π ∗ . From (24), we see that the cost of π ∗ is perfect and it remains to check that Supp ( π ∗ y ) ⊂ Supp ( Q ) . Assume the opposite and pick any y ∗ ∈ Supp ( π ∗ y ) such that y ∗ / ∈ Supp ( Q ) . By the definition of the support, there exists ϵ > 0 and a neighborhood U = { y ∈ Y such that ∥ y -y ∗ ∥ Y < ϵ } of y ∗ satisfying π ∗ y ( U ) > 0 and

U ∩ Supp ( Q ) = ∅ . Let h ( y ) def = max { 0 , ϵ -∥ y -y ∗ ∥ Y } . From π w n y ≤ w n Q (for all n ), it follows that Supp ( π w n y ) ⊂ Supp ( Q ) . Therefore, π w n y ( U ) = 0 for all n . Since π w n converges to π ∗ , we have

The left part is zero because h ( y ) vanishes outside U and ∫ U h ( y ) dπ w n ( y ) = 0 as π w n y ( U ) = 0 . The right part equals ∫ U h ( y ) dπ ∗ ( y ) and is positive as π ∗ ( U ) > 0 and h ( y ) > 0 for y ∈ U . This is a contradiction. Therefore, Supp ( π ∗ y ) ⊂ Supp ( Q ) . Now we see that π ∗ ∈ Π ∞ ( P , Q ) is a perfect plan as its cost matches the perfect cost.

Proof of Corollary 1. Assume the inverse. Then ∃ ε such that ∀ w ( ε ) ∃ w ≥ w ( ε ) and ∃ IT plan π w ∈ Π w ( P , Q ) solving (12) such that ∀ ET plan π ∗ , it holds that W 1 ( π w , π ∗ ) ≥ ε . This means that there exists an increasing sequence w 1 , w 2 , ..., w n , · · ·→∞ and the corresponding sequence of IT plans π w 1 , π w 2 , ..., π w n , . . . such that for every ET plan π ∗ it holds that W 1 ( π w n , π ∗ ) ≥ ε for all n . At the same time, from Theorem 3, this sequence of plans must have a sub-sequence { π w n k } which is (weakly-*) converging to some ET plan π ∗ : π w n k → π ∗ . Recall that the convergence in W 1 coicides with the weak-∗ convergence (for compact X , Y ), see [62, Theorem 5.9]. Hence, the sub-sequence should also converge to π ∗ in W 1 but it is not since W 1 ( π w n , π ∗ ) ≥ ε . This is a contradiction.

Proof of Theorem 4. Let Π( P ) ⊂ P ( X × Y ) denote the subset of probability measures π ∈ P ( X × Y ) satisfying π x = P . Consider a functional I : Π( P ) → { 0 , + ∞} defined by I ( π ) def = sup f ≤ 0 ∫ Y f ( y ) d ( w Q -π y ) ( y ) , where the sup is taken over non-positive f ∈ C ( Y ) . From Lemma 1, we have I ( π ) = 0 when π ∈ Π w ( P , Q ) and I ( π ) = + ∞ otherwise. Indeed, if there exists a non-positive function satisfying ∫ Y f ( y ) d ( w Q -π y ) ( y ) > 0 , then function Cf (for C > 0 ) also satisfies this condition and provides C -times bigger value which tends to ∞ with C →∞ . We use I ( π ) incorporate the right constraint π y ≤ w Q in π ∈ Π w ( P , Q ) to the objective and obtain the equivalent to (12) problem:

In transition from (26) to (27) we use the minimax theorem to swap sup and min [64, Corollary 2]. This is possible because the expression in (26) is a bilinear functional of ( π, f ) . Thus, it is convex in π and concave in f . At the same time, Π( P ) is a convex and (weak-*) compact set. The latter can be derived analogously to the compactness of Π w ( P , Q ) in the proof of Theorem 2. In transition from (28) to (29), we use the measure disintegration theorem to represent dπ ( x, y ) as the marginal dπ x ( x ) = d P ( x ) and a family of conditional measures dπ ( y | x ) on Y . We note that

On the other hand, consider the measurable selection T : X → Y for the set-valued map x ↦→ arg min y ∈Y ( c ( x, y ) -f ( y ) ) . It exists thanks to [2, Theorem 18.19]. As a result, for the deterministic plan π T = [ id , T ] ♯ P , the minimum in (31) is indeed attained. Therefore, (31) is the equality. We combine (30) and (31) and obtain

∗

It remains to prove that sup in the right part is actually attained at some non-positive f ∈ C ( Y ) . Let f 1 , f 2 , · · · ∈ C ( Y ) be a sequence of non-positive functions for which lim n →∞ {∫ X f c n ( x ) d P ( x ) + w ∫ Y f n ( y ) d Q ( y ) } = Cost w ( P , Q ) . For g ∈ C ( X ) , we define the ( c, -) -transform g ( c, -) ( y ) def = min [ min x ∈X ( c ( x, y ) -g ( x ) ) , 0 ] ≤ 0 . It yields a (uniformly) continuous non-positive function satisfying | g ( c, -) ( y ) -g ( c, -) ( y ′ ) | ≤ ω ( ∥ y -y ′ ∥ Y ) , where ω is the modulus of continuity of c ( x, y ) . This statement can be derived analogously to the proof of Proposition 1.

Before going further, let us highlight two important facts which we are going to use below. Consider any g ∈ C ( X ) and 0 ≥ h ∈ C ( Y ) satisfying g ( x )+ h ( y ) ≤ c ( x, y ) for all ( x, y ) ∈ X ×Y . First, from the definition of ( c, -) -transform, one can see that for all ( x, y ) ∈ X ×Y it holds that 0 ≥ g ( c, -) ≥ h and

i.e., ( g, g ( c, -) ) also satisfies the assumptions of ( g, h ) . Second, from the definition of c -transform, it holds that h c ≥ g and

i.e., the pair ( h c , h ) satisfies the same assumptions as ( g, h ) .

Now we get back to our sequence f 1 , f 2 , . . . . For each n and ( x, y ) ∈ X × Y , we have f c n ( x ) + f n ( y ) ≤ c ( x, y ) . Next,

where we first used (33) with ( g, h ) = ( f c n , f n ) and then used (34) with ( g, h ) = ( f c n , ( f c n ) ( c, -) ) . In particular, we have f c n ≤ ( ( f c n ) ( c, -) ) c and f n ≤ ( f c n ) ( c, -) . We sum these inequalities with weights 1 and w , and for all ( x, y ) ∈ X × Y obtain

where for convenience we denote h n def = ( f c n ) ( c, -) . Integrating (36) with ( x, y ) ∼ P × Q yields

This means that potential h n provides not smaller dual objective value than f n . As a result, sequence h 1 , h 2 . . . also satisfies lim n →∞ {∫ X h c n ( x ) d P ( x ) + w ∫ Y h n ( y ) d Q ( y ) } = Cost w ( P , Q ) . Now we forget about f 1 , f 2 , . . . and work with h 1 , h 2 , . . . .

All the functions h n are uniformly equicontinuous as they share the same modulus of continuity ω because they are ( c, -) -transforms by their definition. Let v n ( y ) def = h n ( y ) -max y ′ ∈Y h n ( y ′ ) . This function is also non-positive and uniformly continuous as well. Note that v n provides the same dual objective value as h n . This follows from the definition of v c = h c +max y ′ ∈Y h n ( y ′ ) . Here the additive constant vanishes, i.e., v c n ( x ) + v n ( y ) = h c n ( x ) + h n ( y ) . At the same time, v n are all uniformly bounded. Indeed, let y n ∈ Y be any point where v n ( y n ) = 0 . Then for all y ∈ Y it holds that | v n ( y ) | = | v n ( y ) -v n ( y n ) | ≤ ω ( ∥ y -y n ∥ Y ) ≤ ω ( diam ( Y ) ) . Therefore, by the Arzelà-Ascoli theorem, there exists a subsequence v n k uniformly converging to some f ∗ ∈ C ( X ) . As all v n k ≤ 0 , it holds that f ∗ ≤ 0 as well. It remains to check that f ∗ attains the supremum in (32).

To begin with, we prove that v c n k uniformly converges to ( f ∗ ) c . Denote ∥ v n k -f ∗ ∥ ∞ = ϵ k . For all ( x, y ) ∈ X × Y , we have

since | v n k ( y ) -f ∗ ( y ) | ≤ ∥ v n k -f ∗ ∥ ∞ < ϵ . We take min y ∈Y in (38) and obtain ( f ∗ ) c ( x ) -ϵ k ≤ v c n k ( x ) ≤ ( f ∗ ) c ( x ) + ϵ k . As this holds for all x ∈ X , we have just proved that ∥ v c n k -( f ∗ ) c ∥ ∞ < ϵ k . This means that v c n k uniformly converges to ( f ∗ ) c as well since lim k →∞ ∥ v n k -f ∗ ∥ ∞ = lim k →∞ ϵ k = 0 . Thanks to the uniform convergence, we have

∗

We conclude that f is a maximizer of (32) that we seek for.

Bibliographical remark. There exists a duality formula for partial OT (4), see [10, M 2] which can be reduced to duality formula to IT problem (12). However, it is hard to relate the resulting formula with ours (13). We do not know how to derive one formula from the other. More importantly, it is unclear how to turn their formula to the computational algorithm. Our formula provides an opportunity to do this by using the saddle point reformulation of the dual problem which nowadays becomes standard for neural OT, see [43, 20, 61]. We will give further comments after the next proof. The second part of the derivation of our formula (existence of a maximizer f ∗ ) is inspired by the [62, Proposition 1.11] which shows the existence of maximizers for standard OT (2).

Proof of Theorem 5. By the definition of f ∗ , we have

This means that all the inequalities in (40)-(42) are equalities. Since (40) equals (41), we have T ∗ ∈ arg min T : X→Y L ( f ∗ , T ) .

Bibliographic remark (theoretical part). The idea of the theorem is similar to that of [61, Lemma 4.2], [24, Lemma 3], [43, Lemma 4], [20, Theorem 2] which prove that their respective saddle point objectives max f min T L ( f, T ) can be used to recover optimal T ∗ from some optimal saddle points ( f ∗ , T ∗ ) . Our functional (15) differs, and we have the constraint f ≤ 0 . We again emphasize here that not for all the saddle points ( f ∗ , T ∗ ) it necessarily holds that T ∗ is the IT map, see the discussion in limitations (Appendix A).

Bibliographic remark (algorithmic part). To derive our saddle point optimization problem (15), we use the c -transform expansion proposed by [53] in the context of Wasserstein GANs and later explored by [40, 43, 61, 20, 24, 31] in the context of learning OT maps. That is, our resulting algorithm 1 overlaps with the standard maximin neural OT solver, see, e.g., [24, Algorithm 1]. The difference is in the constraint f ≤ 0 and the additional multiplier w ≥ 1 .

Proof of Proposition 4. From the proof of Theorem 5, we see that ∫ Y f ∗ ( y ) d [ w Q -T ∗ ♯ P ] ( y ) = 0 . Recall that f ∗ ≤ 0 . This means that f ( y ) = 0 for y ∈ Supp ( w Q -T ∗ ♯ P ) . Indeed, assume the opposite, i.e., there exists some y ∈ Supp ( w Q -T ∗ ♯ P ) for which f ( y ) < 0 . In this case, the same holds for all y ′ in a small neighboorhood U of y as f is continuous. At the same time, ∫ Y f ∗ ( y ) d [ w Q -T ∗ ♯ P ] ( y ) ≤ ∫ U f ∗ ( y ) d [ w Q -T ∗ ♯ P ] ( y ) < 0 since [ w Q -T ∗ ♯ P ] is a non-negative measure satisfying [ w Q -T ∗ ♯ P ] ( U ) > 0 by the definition of the support. This is a contradiction. To finish the proof it remains to note that Supp ( Q ) \ Supp ( T ∗ ♯ P ) ⊂ Supp ( w Q -T ∗ ♯ P ) , i.e., f ( y ) = 0 for y ∈ Supp ( Q ) \ Supp ( T ∗ ♯ P ) as well.

Bibliographical remark . Treating functional L ( f, T ) in (16) as a Lagrangian, Proposition 4 can be viewed as a consequence of the complementary slackness in the Karush-Kuhn-Tucker conditions [37].

G Additional Experimental Illustrations

G.1 Comparison with the Closed-form Solution for ET

In this section, we conduct a Swiss2Ball experiment in 2D demonstrating that for the sufficiently large parameter w , IT maps become fine approximations of the ground-truth ET map. We define source measure P as a uniform distribution on a swiss roll centered at (0 , 0) . Target measure Q is a uniform distribution on a ball centered at (0 , 0) with radius R = 0 . 5 , i.e., Supp ( Q ) = B (0 , 0 . 5) . We note that the supports of source and target measures are partially overlapping. In the proposed setup, the solution to ET problem (5) has a closed form: T ( x ) = x · 1 x ∈ B ((0 , 0) , 0 . 5) + x · R ∥ x ∥ 2 · 1 x/ ∈ B ((0 , 0) , 0 . 5) , see Fig. 20f. We provide the learned IT maps for w ∈ { 1 , 3 / 2 , 2 , 32 } , see Fig. 20b-20e. The qualitative and quantitative results show that with the increase of w our IT maps become closer and closer to the ground-truth ET one, see Table 8.

(a) Input, target

measures.

(b) IT map,

w

=1

.

(c) IT map,

w

=

3

/

2

.

(d) IT map,

w

=2

.

(e) IT map,

w

=32

.

(f) Extremal

transport map.

0.8

Target y ~Q

Mapped y = T(x)

Mapped y = T(x)

Mapped y= T(x)

Mapped y = T(x)

ET map

0.6

Input x~p

Inputx-p

0.4

0.2

0.0

0.2

0.4

-0.6

0.5

0.0

0.5

0.5

0'0

0.5

0.5

0.0

0.5

0.5

0.0

0.5

0.5

0.0

0.5

0.5

0.0

0.5

w

= 1

w

=

3

/

2

w

= 2

w

= 32

MSE(

ˆ

T, T

∗

)

0.0136

0.0026

0.0009

7.98e-06

G.2 Solving Fake Solutions Issue with Weak Kernel Cost

As we discuss in Appendix A, saddle-point neural OT methods (including our IT algorithm) may suffer from fake solutions issue. As it is proved in [42], this issue can be eliminated by considering OT with the so-called weak kernel cost functions. In this section, we test the effect of using them in our IT algorithm. Specifically, we demonstrate the example where IT algorithm with the cost function c ( x, y ) = ∥ x -y ∥ 2 struggles from fake solutions, while IT with the same cost endowed with weak kernel regularization, i.e., kernel cost [42, Equation (16)] with parameters α = 1 , γ = 0 . 4 , resolves the issue. Following [42], we consider stochastic version of IT map T ( x, z ) using noise z ∼ N (0 , I ) as an additional input.

We design the Ball2Circle example in 2D, where input P is a uniform distribution on a ball and target Q -on a ring embracing the ball. The solution of ET problem is an internal circuit of a ring, see Fig. 21a. We learn IT maps for c ( x, y ) = ∥ x -y ∥ 2 , with ( γ = 0 . 4) or without ( γ = 0 ) kernel regularization for weights w ∈ { 1 , 2 , 32 } .

(a) Input, target

measures.

(b) IT map,

w

=1

(iter=7K)

(c) IT map,

w

=

3

(iter=8K)

(d) IT map,

w

=2

(iter=7K)

(e) IT map,

w

=2

(iter=8K).

(f) IT map,

w

=32

0.50

0.75

ET map

Targety~Q

Inputx~P

Mapped y = T(x)

Mapped y = T(x)

Mapped y = T(x)

Mapped y = T(x)

Mapped y = T(x)

0.25

0.00

-0.25

-0.50

-0.75

0.5

0.0

0.5

0.5

0.0

0.5

0.5

0.0

0.5

0.5

0.0

0.5

0.5

0.0

0.5

0.5

0.0

0.5

2

(a) Input, target measures

and ET map.

(b) IT map,

w

=1

.

(c) IT map,

w

=2

.

(d) IT map,

w

=32

.

0.75

ETmap

Targety~Q

Mappedy=T(x)

Mapped y= T(x)

ETmap

Mapped y = T(x)

0.50

InputxP

0.25

0.00

0.25

0.50

0.75

0.5

0.0

0.5

0.5

0.0

0.5

0.5

0.0

0.5

0.5

0.0

0.5

Discussion. We observe that without kernel regularization training of IT method is highly unstable (for w ∈ { 1 , 2 } ), see Fig. 21b-21d. Interestingly, with the increase of the weight w , the issue disappears and for w = 32 , IT map is close to the ground-truth ET one (Fig. 21f).

The usage of kernel regularization helps to improve stability of the method, see Fig. 22. However, while MSE between learned and ground-truth ET map drops with the increase of w , for w = 32 it is bigger than that of learned IT maps without regularization. It is expected, since using regularizations usually leads to some bias in the solutions.

Thus, the example shows that (a) fake solutions may be a problem, (b) kernel regularization from [42] may help to deal with them. However, further studying this aspect is out of the scope of the paper.

G.3 Perceptual Cost

In this section, we show that the stated conclusions hold true for the transport costs other than ℓ 2 . For this purpose, we perform additional experiments using perceptual transport cost from [24]. We use the same hyperparameters as in our experiments with ℓ 2 cost.

d～x

W=1

#

W=2

(x)

W=4

#

W=8

Experimental results. Qualitative results show that that similarity of input images x and the images ˆ T ( x ) translated by our IT method trained with perceptual cost grows with the increase of the parameter w , see Figure 23. In 10, we quantitatively verify these observations by showing that both perceptual and ℓ 2 mean transport costs between input and translated images decrease with the increase of w . Interestingly, we see that IT trained with perceptual cost yields smaller FID than IT method trained with ℓ 2 cost.

G.4 Bigger Weight Parameters

In this section, we present the results of IT method trained with ℓ 2 cost and parameters w = 16 , 32 . We see that in contrast to CycleGAN which does not translate face images to anime images in case of big parameters λ , our IT method continues to translate faces to anime even for big parameters w . We quantify the obtained results in Table 11. As expected, mean transport costs are decreasing with the increase of w , while FID is slightly increasing. We present the qualitative results for additional weights in Figure 24.

G.5 Additional Results for ℓ 2 Cost

～×

(x)

W=1

(x)

W=2

(x)

W=4

(x)。

W= 8

W = 16

f(x)

W= 32

(x)

d～x

(x)

W=2

#

W=4

f(x)

W=8

d~x

L=M

(x)

W=2

(x)

W=4

f(x)

W=8

d~x

W=1

Z=M

W=4

(x)

8=M

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel OT problem, extremal transport (ET), in the context of domain translation. The authors propose an incomplete transport (IT) problem as a surrogate optimization problem to obtain an approximate solution to the ET problem. The theoretical convergence between IT and ET costs (plans) is proven conditionally. The effectiveness of ET and IT is showcased through experiments on toy examples and public datasets, while also demonstrating the relationship between ET and IT.\n\nSTRENGTHS:\n- The optimal transport formulations in this work, i.e., ET and IT, are something that I have not seen. These formulations, for me, are inspiring and interesting.\n\n- The paper provides a thorough discussion on the technical differences and relationship with other existing methods.\n\n- The overall presentation of the main paper is generally clear. Furthermore, the full paper, including the appendices, appears to be comprehensive and well-prepared. The theoretical results are presented in a comprehensive and self-contained manner.\n\nWEAKNESSES:\n- The scope of and motivation for the work are a little unclear from the abstract and introduction. In particular, the motivation to incorporate ET in the context of domain translation is not clear, which could make readers confused. It should be highlighted at least in the abstract and introduction sections. \n\n- Certain statements lack sufficient explanations or supporting intuitions, which should be explained and provided in the main text. To list a few:\n  - In the lines 85-91, a variant of OT problem is established. The authors mentioned \"we say that the target domain is the part of Y where the probability mass of Q lives\". However, the intuition behind this statement is not well-presented and should be elaborated upon.\n  - In the lines 76-78, the intuition to establish the ET framework in the context of domain translation should be demonstrated. It should also be highlighted in the abstract and introduction sections. Specifically, readers may hope to see why it is essential to construct ET for domain translation task, given thorough related works on weak OT, partial OT, etc... Although you have formulated the technical difference with them, I hope to see analysis on \"why ET outperforms them in your application\".\n  - In the lines 91-92, the authors relaxes the mass preservation condition for the target domain. Why not relax the condition for both the target and source domain, likewise UOT? A concise and simple comparison with current weak / partial OT should be clarified here, to **explain the advantages of ET in domain translation compared with peer methods, (rather than purely formulating the technical differences)**, which is critical for identifying and evaluating ET's role in domain translation.**\n\n\n### ************ Minors ***************\n\n- Figure 3-4 are kind of redundant and largely inherit from current literatures. As such, it is not appropriate to take too much blank space for them. A more proper way would be consendate them in one line or move them to the appendix.\n- In the line 104, the NN could lead to unexpected confusion since it usually represents neural networks.\n- In the lines 80-109, the authors establish an optimization problem to seek for the nearest neighbors of $x\\in\\mathcal{X}$ in Supp(Q). The narrative in this section is kind of redundant.\n- It is essential for readers to acknowledge the technical differences with other OT methods. As such, I suggest authors moving the section of \"Related works\" after the section of \"background on optimal transport\", and highlight your technical difference with each genre of OT methods concisely.\n\nMy current score is a reflection of the weaknesses above as they stand. Since these seem like rather correctable issues, I expect to increase my score pending a positive response from the authors.\n\nQUESTIONS:\n- The authors noted \"the task is ill-posed as there might exist multiple suitable T.\" It is confusing for me due to two reasons: (1) you just mention that there could be multiple maps between source and target; however, the optimization problem has not been formulated here, and it is not proper to say it is ill-posed problem since the word ill-posed seems to be suitable for optimization problems; (2) the optimization problem, such as the Kantorovich problem, could be not ill-posed in some cases given certain metrics, is it right? Please consider to improve the narrative to make it more rigorous and self-contained.\n- The authors noted \"challenge here is that the correspondence between vailable data samples a from the source and y from target domains is not given.\" What if the correspondence is given? Can we get a new optimization problem with intriguing properties?\n\nLIMITATIONS:\nThe limitations have been well discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper proposes extremal transport (ET), a mathematical framework for achieving the best possible unpaired translation between two domains based on a given similarity function, and proving that ET maps can be learned as a limit of specific partial optimal transport (OT) problem. The contributions of the paper include a formalization of ET as a rigorous mathematical task, the characterization of ET maps and plans through a connection to nearest neighbors, and the derivation of an efficient computational algorithm based on partial optimal transport. The proposed algorithm is evaluated using 2D examples and the unpaired image-to-image translation task.

STRENGTHS:
1.The logic of this paper is well organized, and also the mathematical formulas are well defined.
2.The work demonstrates that it is possible to relax equality constraints to inequality constraints especially for some situations where enforcing rigorous equal constraints are difficult, and proposes incomplete transport (IT) to approximate ET maps using partial optimal transport (OT) problems, which is very novel.
3.The experimental part is quite sufficient complete and proves the effectiveness of the proposed method.

WEAKNESSES:
It is not clarified the motivations of defining ET problem and the relationship between specific task such as unpaired image translation and the mathematical ET problem.The process from abstraction of practical problems to theoretical derivation is too steep.

QUESTIONS:
Why the ET problem is defined in the compact polish spaces？Could it transfer to other spaces?

LIMITATIONS:
The ℓ2 and FID metrics have some limitations in characterizing the effectiveness of the method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a mathematical formalization called \"extremal transport\" that aims to achieve optimal translation between unpaired domains based on a given similarity function. Additionally, the paper proposes a scalable algorithm that utilizes neural optimal transport to approximate extremal transport mappings. The algorithm is tested on toy examples and the image-to-image translation task, yielding promising results.\n\nSTRENGTHS:\nThe paper introduces a novel mathematical formalization called extremal transport for achieving optimal translation between unpaired domains. Additionally, the concept of neural optimal transport is introduced and applied in the algorithm. These theoretical foundations provide a solid basis for the algorithm design.\n\nThe algorithm is tested on toy examples and the image-to-image translation task, demonstrating good results. These experiments validate the effectiveness and scalability of the algorithm.\n\nImage-to-image translation is an important problem in computer vision, and the proposed algorithm can be applied to address this problem. Furthermore, the paper mentions other potential application areas, such as single-cell data analysis in biology.\n\nThe algorithm proposed in this paper can be widely applied to solve translation problems between unpaired domains, making it highly generalizable. Additionally, a new method called partial optimal transport is introduced, which can be used to address alignment problems between imbalanced measures, further expanding its potential for generalization.\n\nWEAKNESSES:\nBased on the information provided, the paper primarily focuses on the common main features of existing approaches rather than encompassing all of them. It's possible that due to space constraints, some concepts or details may not be extensively elaborated in the main paper. However, substantial explanations are provided in the Appendix.\n\nQUESTIONS:\nIn lines 121-124, how the lower bound is defined during the process of analyzing and constructing upper and lower bounds to simplify the objective is not mentioned. The reason why value(8) admits more minimizers is also not explained.\n\nIn lines 133-136, an explanation is needed for the replacement mentioned here and why using a finite parameter w can achieve the desired replacement of Supp(T#P) belonging to Supp(Q).\n\nIn lines 218-220, it is asked whether the method presented in the paper scales to high dimensions. If images are defined as high-dimensional, this should be clarified from the beginning of the paper.\n\nLIMITATIONS:\nThe paper has conducted extensive discussions.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the concept of extremal Optimal Transport (OT) and proposes the use of incomplete OT as a solution to the extremal OT problem. The authors present a duality method to address the incomplete OT problem and validate this approach using a toy 2D dataset and image translation tasks.\n\nSTRENGTHS:\nThe authors offer the analysis of the proposed problems and algorithms, including the existence of Extremal Transport (ET), the presence of Incomplete Transport (IT), and the convergence of IT plans to the ET plans.\n\nThey test their algorithms on several image translation datasets, demonstrating the ability to manipulate the similarity between the source image and the generated image by adjusting the weight parameter $w$.\n\nTheir algorithm exhibits the capability to map the data towards a portion of the target distribution's support.\n\nWEAKNESSES:\n## Motivation\n\nFirstly, the paper's logic doesn't fully convince me. The authors initially propose to solve the Extremal Transport (ET) problem, but then shift to solving the Incomplete Transport (IT) problem, presumably because the ET problem is too challenging. However, based on their experimental results, it seems more logical to propose solving the IT problem directly, as most of their visualizations demonstrate that adjusting the weight parameter $w$ controls the similarity between the source image and the generated image. If the authors are indeed intent on solving the ET problem, could they provide experimental evidence to support this? For instance, they could design some distributions that have a closed-form solution for ET, and then compare the ground truth minimum value of problem (5) with their simulation results.\n\nRegarding the motivation for problem (11): the authors state that \"In practice, solving the extremal problem (5) is challenging because it is hard to enforce Supp($T \\sharp P$) $\\subset$ Supp($Q$)\", but isn't this relatively straightforward for image tasks? Since images are typically scaled to a certain range like [-1,1], we can simply scale the images to meet this support requirement.\n\nI believe that the application of incomplete OT or extremal OT could be limited in image tasks, especially extremal OT, as multiple data points could be mapped to the same data point, resulting in generated images with limited diversity.\n\n## Method\n\nRegarding the relaxation from problem 5 to 11, it seems to me that the constraint in 11 is not a softened version, but rather a stronger one. To satisfy the constraint in (11), one must also satisfy the constraint in (5). This is because in the constraint of (11), if Q=0, then regardless of the value of $w$, $T\\sharp P$ must also be zero, which aligns with the constraint in (5). Why is it not feasible to design a dual formula for ET directly?\n\nThe authors mention the issue of fake solutions. Could they consider borrowing ideas from the Kernel Neural OT paper? It seems that using a similar cost from Kernel Neural OT could enhance this paper, and it seems not that difficult?\n\n## Results\n\nGiven that the motivation of this paper is to solve the extremal OT problem, could the authors clarify under what conditions their method can recover the solution to extremal OT? Specifically, how large does the value of $w$ need to be? It's possible that this information is already included, and I may have overlooked it.\n\nThe authors acknowledge that the FID is not particularly meaningful when $w>1$, yet it appears to be the primary metric used in this paper. If the authors believe that FID is not representative, why do they present so many FID results? They also claim that the image quality of the translated image does not decrease with increasing $w$. Is there a way to verify this quantitatively, perhaps with a metric based on the image itself rather than the image distribution?\n\nSome of the results presented in this paper seem less than satisfactory. For instance, in Figure 24, even when $w$ is large, some images fail to preserve hair colors. In Figure 22, as $w$ increases, some images exhibit unrealistic artifacts (e.g., the first grey shoe and the black saddle have strange bands on them), likely because the generated image is forced to share more similarity with the source image. Could this be a limitation of the proposed method?\n\nQUESTIONS:\nFigure 5b may not be as illustrative as intended. In this case, it seems that pi would still uniquely correspond to a certain deterministic transport map. Perhaps it would be more instructive to provide an example where (8) admits multiple minimizers, such as a scenario where the support is not convex?\n\nLIMITATIONS:\n.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel notion of extremal transport, which relaxes the optimal transport problem by only requiring the support of the pushed-forward distribution to be a subset of the support of the target distribution. To solve this problem, the authors propose a novel approximation approach to find the solution from a subsequence of a series of incomplete OT problems (where T#P = Q is relaxed to T#P < wQ, w >= 1, when w=1 this is the traditional OT problem). Then the author derives novel dual formulations of the incomplete OT problems, which the author proposes to solve by using neural networks to approximate the potential and c-transform solutions. The authors apply the method to toy sets and image translation examples to demonstrate the scalability of the method. FID results on the image translation example prove the concept of the method. \n\nThe authors carefully compare their results with related works and establish their theoretic novelty. The presentation of the results are clear and potential impacts are well discussed.\n\nSTRENGTHS:\nThe paper proves several strong theoretical results on the relaxed versions of OT problems (i.e. ET and IT). The theoretic study of the solutions to these problems is fruitful. This opens up a new domain of study area in the optimal transport area.\n\nWEAKNESSES:\nOverall the theoretic results are rigorous and novel. To solve the solution of the dual formulation of IT, the authors propose to use neural networks to approximate the transport map (necessary for solving large-scale problems). One concern I have is that in the traditional OT problem, the OT map is in general discontinuous. This proposes difficulties in using DNN to approximate the OT map. It would be very welcomed if results on the regularity of the IT maps are presented, and/or how well the neural network can approximate such maps.\n\nQUESTIONS:\nOne question: in theorem 3, it only guarantees the existence of a subsequence. In practice, if we found a sequence of solutions by Alg. 1, how can we make sure whether it is the desired ET solution?\n\nLIMITATIONS:\nThe limitation of the paper is adequately addressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}