Wasserstein-2 Generative Networks
Authors: Alexander Korotin
Vage Egiazarian
Arip Asadulaev
Alexander Safin
Evgeny Burnaev
Venue: ICLR 2021 Poster (2021)
Keywords: wasserstein-2 distance
optimal transport maps
non-minimax optimization
cycle-consistency regularization
input-convex neural networks

Abstract: We propose a novel end-to-end non-minimax algorithm for training optimal transport mappings for the quadratic cost (Wasserstein-2 distance). The algorithm uses input convex neural networks and a cycle-consistency regularization to approximate Wasserstein-2 distance. In contrast to popular entropic and quadratic regularizers, cycle-consistency does not introduce bias and scales well to high dimensions. From the theoretical side, we estimate the properties of the generative mapping fitted by our algorithm. From the practical side, we evaluate our algorithm on a wide range of tasks: image-to-image color transfer, latent space optimal transport, image-to-image style transfer, and domain adaptation.

Published as a conference paper at ICLR 2021
WASSERSTEIN-2 GENERATIVE NETWORKS
Alexander Korotin
Vage Egiazarian
Skolkovo Institute of Science and Technology
Skolkovo Institute of Science and Technology Moscow, Russia
Moscow, Russia
eokootinsskolttedei ru
vagg..egiiaarriaaefotoeee......
Arip Asadulaev
Aleksandr Safin
ITMO University
Skolkovo Institute of Science and Technology
Saint Petersburg, Russia
Moscow, Russia a lee.annrr.aaii@sseeeeeee......ee
ar pasaddaaaave tmo ru
Evgeny Burnaev
Skolkovo Institute of Science and Technology
Moscow, Russia
e burnaevvskolteehh
ABSTRACT
We propose a novel end-to-end non-minimax algorithm for training optimal trans- port mappings for the quadratic cost (Wasserstein-2 distance). The algorithm uses input convex neural networks and cycle-consistency regularization to approximate Wasserstein-2 distance. In contrast popular entropic and quadratic regulariz- ers, cycle-consistency does not introduce bias and scales well to high dimensions. From the theoretical side, estimate the properties of the generative mapping fitted by our algorithm. From the practical side, we evaluate our algorithm on a wide range of tasks: image-to-image color transfer, latent space optimal transport, image-to tenee-ttoimmgee style transfer, and domain adaptation.
INTRODUCTION
Generative learning framework has become widespread over the last couple of years tentatively starting with the introduction of generative adversarial networks (GANs) by boooddelooww et al. (2014). The framework aims to define stochastic procedure to sample from given complex probability distribution Q on space C RR",, e.g. a space of images. The usual generative pipeline includes sampling from tractable distribution P on space X' and applying generative mapping X y
that transforms F into the desired Q
In many
for probability distributions P. Q, there may exist several different gen- erative mappings. For example, the mapping in Figure Tb seems to be better than the one in Figure Ta and should be preferred: the mapping in Figure Iblis straightforward, well- structured and invertible.
I
P
g
Existing generative learning approaches mainly do not focus on the structural prop- erties of the generative mapping. For example, GAN-based approaches, such as f-GAN by Nowozin et al. (2016); Yadav al. (2017), W-
(a) An Arbitrary Mapping. (b) The Monotone Mapping.
GAN by Arjovsky et al. (2017) and others Li cet al. (2017); Mrouch & Sercu (2017), aррrox- imate generative mapping by a neural network
Figure 1: Two possible generative mappings that
transform distribution IP to distribution Q
with problem-speciffe architecture.

---

Published as a conference paper at ICLR 2021
The reasonable question is how to find generative mapping g o P Q that is well-structured. Typically, the better the structure of the mapping is, the easier it is to find such a mapping. There many ways to define what the well-structured mapping is. But usually, such mapping is expected to be continuous and, if possible, invertible. One may note that when P and Q are both one-dimensional (X y C IR.),, the only class of mappings g - > satisfying these properties are monotone m i.c. Vx, æ' E E (æ # r') satisfying (g(zz (z) - g(x")) (x-
(t-=x) x') > 0. The intuition of 1-dimensional spaces can be easily extended to X C IR.". . We can require the similar condition to hold true: Var, x' E X (xr x²)
(g(x) ((x)-g(x'),) 22)= 0.
The condition (T) is called monotonicity, and every surjective function satisfying this condition is invertible. In one-dimensional case, for any pair of continuous P, Q with non-zero density there exists a unique monotone generative map given by (xx) = Fe (F)(z) McCann et al. 1995), where F(-) the cumulative distribution function of or Q. However, for one generative monotone mapping. For example, when P Q are standard 2-dimensional Gaussian distributions, all rotations by angles - < < 2 are monotone and preserve the distribution.
there might exist more than
One may impose uniqueness by considering only maximal Peyré (2018) monotone mappings g X' + ) satisfying AN = 2.3... and N distinct points TI,,..,,N EX(N+==)
N
Σ (g(x.)) In In++1 0.
The condition ② is called cycle monotonicity and also implies "usual" monotonicity 1
Importantly, for almost every two continuous probability distributions P. Q on X' =y= there exists a unique cycle monotone mapping 9 X + ) satisfying o P Q, see McCann et al. (1995). Thus, instead searching for arbitrary generative mapping, one may significantly reduce the considered approximating class of mappings by using only cycle monotone ones.
According to Rockafellar (1966), every cycle monotone mapping g is contained of some convex function : X I.. Thus, every convex class of functions may produce cycle monotone mappings (by considering sub-gradients of these functions). In practice, deep input convex neural networks (ICNNs, see Amos et al. (2017) can be used class of convex functions.
sub-gradient
Formally, to fit a cycle monotone generative mapping, one may apply any existing approach, such as GANs Goodfellow al. (2014), with the set of generators restricted to gradients of ICNN. However, GANs typically require solving a minimax optimization problem.
It turns that the eycle monotone generators are strongly related to Wasserstein-2 distance (W2). The approaches by TTahvaai & Jalali (2019); Makkuva et al. (2019) use dual form of W. to find the optimal generative mapping which cycle monotone. The predecessor of both approaches the gradient-descent algorithm for computing W2 distance by Chartrand et al. (2009). The drawback of all these methods is similar the one of GANs their optimization objectives minimax.
Cyclically monotone generators require that both spaces X' and y have the same dimension, which poses no practical limitation. Indeed, it is possible combine generative mapping with decoder of a pre-trained autoencoder, i.c. train a generative mapping into a latent space. It should be also noted that the cases with equal dimensions of X' and y are common in computer vision. The typical example is taage-to-image style transfer when both the input and the output images have the same size and number of channels. Other examples include mmage-to-image color transfer, domain adaptation, etc.
In this paper, we develop the concept of cyclically monotone generative learning. The main contri- butions of the paper are as follows:
1. Developing an end-to-end non-minimax algorithm for training cyclically monotone generative maps, i.e. optimal maps for quadratic transport cost (Wasserstein-2 distance).
2. Proving theoretical bound on the approximation properties of the transport mapping fitted by the developed approach.
3. Developing a class of Input Convex Neural Networks whose gradients are used to approximate cyclically monotone mappings.
We consider only monotone increasing mappings. Decreasing mappings have analogous propertics.
2

---

Published as a conference paper at ICLR 2021
4. Demonstrating the performance of the method in practical problems of image- to-image color transfer, mass transport in latent spaces, image-to-imane style translation and domain adaptation.
Our algorithm extends the approach of Makkuva et al. (201)), eliminates minimax optimization imposing cyclic regularization and solves non-minimax optimization problem. At the result, the algorithm scales well to high dimensions and converges up to 10x times faster than its predecessors.
The paper is structured as follows. Section 2 is devoted to Related Work. In Section we give the necessary mathematical tools on Wasserstein-2 optimal transport. In Section + we derive our algorithm and state our main theoretical results. In Section we provide the results of computational experiments. In Appendix [A] we prove our theoretical results. In Appendix B particular architectures of ICNN that we use for experiments. Appendix C additional experiments and training details are provided.
we describe the
2 RELATED WORK
Modern generative learning is mainly associated with Generative Adversarial Networks (GANs)
Goodfellow et al. (2014); Arjovsky et al. (2017). Basic GAN model consists of two competing networks: generator and discriminator d. Generator takes as input samples x from given distribution IP and tries to produce realistic samples from real data distribution Q. Discriminator d attempts to distinguish between generated and real distributions g IP and Q respectively. Formally. it approximates a dissimilarity measure between g o P and Q (e.g- f-divergence Nowozin et al. (2016) or Wasserstein-1 distance AAjovssy et al. (2017)). Although superior performance is reported for many applications of GANs Karras et al. (211);; ;; Mirza & Osindero (2014). training such models is always hard due to the minimax nature of the optimization objective.
Another important branch of generative learning is related to the theory of Optimal Transport (OT) Villani 2008); Peyré et al. (2019). OT methods seek generative mapping" X +V, optimal in the sense of the given transport cost c: X' x y + R:
Cost(TP Q) min / c(x. ((c.((x)))P(1)) gop=
Equation 3 also known as Monge's s formulation of optimal transportation Villani (2088).
The principal OT generative method Seguy et al. (2017) is based optimizing the regularized dual form of the transport cost (3). It fits two potentials v. v (primal and conjugate) and then uses the barycentric projection to establish the desired (third) generative network g. Although the method
uses non-minimax optimization objective, it is not end-to-end (consists of two sequential steps).
In the case of quadratic transport cost c(x, y) = |اا 2
the value
is known as the square of
Wasserstein-2 distance:
|| 2(x) dz(x).
w (P. !
min gelf
(4)
2
It has been well studied in literature Brenier (1991) McCann et al. (1995); : Villani (2003) 2008) and
has many useful properties which we discuss in Section 3 in more detail. The optimal mapping for
the quadratic cost is cyclically monotone. Several algorithms exist Lei al. (2019) Taghvaei & Jalali
(2019): Makkuva et al. (2019) for finding this mapping.
The recent approach by Taghvaei & Jalali (2019) uses the gradient-descent-based algorithm by Chartrand et al. (2009) for computing W2. The key idea is to approximate the optimal potential s²* by an ICNN Amos et al. (2017), and extract the optimal generator g" from its gradient V The method is impractical due to high computational complexity: during the main optimization cycle. it solves an additional optimization sub-problem. The inner problem is convex but computationally costly. This was noted in the original paper and de-facto confirmed by the lack of experiments with complex distributions. A refinement of this approach is proposed by Makkuva et al. (2019). The inner optimization sub-problem is removed, and a network is used to approximate its solution. This speeds up the computation, but the problem still minimax.
Commonly, in OT it is assumed that dim X' dim y.
3

---

Published as a conference paper at ICLR 2021
3 PRELIMINARIES
In the section, we recall the properties of W2 distance (4) and its relation to cycle monotone mappings.
Throughout the paper, we assume that IP and Q are continuous distributions on X' y = RD with finite second moments This condition guarantees that (3) is well-defined in the sense that the optimal mapping g always exists. It follows from (Villani 2003 Brenier's Theorem 2.12) that its restriction to the support of P is unique (up to the values on the small sets) and invertible. The
symmetric characteristies apply to its inverse (g")-1, which induces symmetry to definition ( for quadratic cost. According to Villani (003), the dual form of is given by
/ ||=|1
/
1uyl2 dQ(u)
[/ (()))P))) J, (v)) (5)
(P Q)
2 (x)
min CCConex
Const (2)
Cor((P
where the minimum is taken over all the convex functions (potentials) tb X - R U (oo), and
(vy) = max (ax y) - +xx)) is the convex conjugate Fenchel (1949) to v/ which also convex function, 6 y + R [20).
We call the value of the minimum in (5) cyclically monotone correlations and denote it by Corr(P, Q). By equating (5) with (4), one may derive the formula
max / (x, (xrg(x))))P(()) goP=Q
Corr(P, Q)
Note that ( Corr(IP, Q)) can be viewed as an optimal transport cost for bilinear cost function c(x; y) - (ax, y), see McCann al. (995).. Thus, searching for optimal transport map g" for W2 equivalent to finding the mapping which maximizes correlations (6).
known for w distance that the gradient g" Vali of optimal potential 2 readily gives the minimizer of (4). see Villani (2003). Being gradient of a convex function, it is necessarily cycle monotone. particular, the inverse mapping can be obtained by taking the gradient w.r.t. input of the conjugate of optimal potential (y)) McCann al. (1995). Thus, we have
(v) (Vv') "w)) = (v)
(v)
In fact, one may approximate the primal potential a by parametric class θ of input convex functions The and optimize correlations
[/ wo(r)dd(()) + /, (o()))
min Corr(P, Q vo) = min DEE
(8)
in order to extract the approximate optimal generator gat X y from the approximate potential thet. Note that in general it is not true that got o P will be equal to Q. However, we prove that if Corr(P, Q ( is close Corr(P. (I, Q)) then got P Q, see our Theorem A.3 in Appendix A.2
The optimization of ( can be performed via stochastic gradient descent. It is possible to get rid of conjugate the and extract an analytic formula for the gradient of (8) w.r.t. parameters @ by using the only, see the derivations in Taghvaei & Jalaii (2019): Chartrand et (2009):
öCorr (P, Q Wo)
@wo(()
dP(x) / Ouio (+)
dQ(y),
00
00
00
where
Hiee in the second integral computed at à = (Viho) (y), i.e. inverse value of y for V 30
In practice, both integrals are replaced by their Monte Carlo estimates random mini-batches from and Q. Yet to compute the second integral, one needs to recover the inverse values of the current mapping Vrba for all y Q in the mini batch. To do this, the following optimization sub-pro sub-problem has to be solved
(V -)))))(()) f arg max ((z, (z))
In practice, the continuity condition can he assumed to hold true. Indeed, widely used heuristics, such adding small Gaussian noise data Sonderby al. (2016), make considered distributions he continuous.
4

---

Published as a conference paper at ICLR 2021
for each y Q in the mini batch. The optimization problem 9 is convex but complex because it requires computing the gradient of ahy multiple times. It is computationally costly since the is in general large neural network. Besides, during iterations over 9. each time new independent batch of samples arrives. This makes hard to use the information on the solution of (9) from the previous gradient descent step over in
AN END-TO-END NON-MINIMAX ALGORITHM
describe novel end-to-end algorithm with non-minimax optimization objective for fitting cyclically monotone generative mappings, In Subsection 4.2 we state our main theoretical results approximation properties of the proposed algorithm.
In Subsection 4.
4.1 ALGORITHM
To simplify the inner optimization procedure for inverting the values of current Vi'a, one may consider the following variational approximation the main objective:
=b(3)
[ (a)dP(())
I max DE
(a, v) - (x(x) dl(w))
min Corr(I,, Q(6)) = @CConvex
min #EConvex
Ux ((z)dP(x) + max TEY*
/ (T(()) .) ((T(V)))((())
min eConves
(10)
where by considering arbitrary measurable functions T. we obtain variational lower bound which matches the entire value for 7 = (Ve) (v) Vi((). Thus, possible approach is to approximate both primal and dual potentials by different networks the and the and solve the optimization problem w.r.t. parameters 0, w, c.g. by stochastic gradient descent/ascent Makkuva et al. (2019). Yet such a problem is still minimax. Thus, suffers from typical problems such as convergence to local saddle points, instabilities during training and usually requires non-trivial hyperparameters choice.
We propose a method to get rid of the minimax objective by imposing additional regularization. Our key idea is to add regularization term Ry,08, w) which stimulates cycle consistency Zhu et al. (2017).
i.e. optimized generative mappings go Vibo and s Vihw should be mutually inverse:
Ry( = -/, I190 g (v) dQl)) /,
V Vele ven(e) - y/1 dQvy)
(11)
From the previous discussion and equation we see that cycle consistency is a quite natural condition for W2 distance. More precisely, Vate and ViL are exactly inverse to each other (assuming Vula injective), then thw is a convex conjugate to the up constant.
In contrast to regularization used in Seguy et al. (2017). the proposed penalties use not the values of the potentials vo., Vh itself but the values of their gradients (generators). This helps to stabilize the value of the regularization term which in the case of [Seuuy et al.) (2017) may take extremely high
values due to the fact that convex potentials grow fast in absolute value
Our proposed regularization leads the following non-minimax optimization objective (A 0):
X [/ (o(z))((z)) + J, [(VV (VVVVVV)))) + Ry(
min B,a0
(12)
Com(lE Vo.VwiA)
The practical optimization procedure is given in Algorithm We replace all the integrals by Monte Carlo estimates over random bini-batches from P and Q To perform optimization, we the stochastic gradient descent over parameters 0, w of primal the and dual th potentials.
We use the automatic differentiation to evaluate V Vel., V Viñe and the gradients of 12 w.r.t. 0, is comparable by a constant factor
w.r.t. parameters
w. The time required compute the gradient
FFor example, in the of identity map g(z)
veh(x)) 2r, we have quadratic growth: ||x]
5

---

Published as a conference paper at ICLR 2021
Algorithm 1: Numerical Procedure for Optimizing Regularized Correlations 12
Input: Distributions PP, Q with sample access; cycle-consistency regularizer coefficient À > 0:
pair of input-conver neural networks We and tws batch size K > 0:
for ( = 1......
1. Sample batches X Pnd Y Q:
Compute the Monte-Cari estimate the correlations:
Loor
Σ xEX
vo(x)) +
[(VVL(v) v) - vo (Ve.(wi)]
K
yEY
Compute the Monte-Care estimate of the cycle-consistency regularizer:
Ccyele = K
Σ
V Vel(u) -1l3:
NEY
4.
Compute the total loss LToul == CCom + 1े¹ Lcyck:
5. Perform gradient step over {0,w} by using ((0,w) end
0Cpal
the time required compute the value of (x)). We empirically measured that this factor roughly equals 8-12, depending on the particular architecture of ICNN the (x). We discuss the time complexity
of gradient step of our method in more detail in Appendix C.2
In Subsection 5.1
we show that our non non-minimax approach converges up to 10x times faster than minimax alternatives by Makkuva al. (2019) and Taghvaei & Jalali (2019).
4.2 APPROXIMATION PROPERTIES
Our messient-descent-t descent-based approach described in Subsection 4.1 computes Corr(IP, Q) by approx- imating it with restricted sets of convex potentials. Let (eb) o be pair of potentials obtained by the optimization of correlations. Formally, the fitted generators g V and (g')
V are byproducts of optimization (12). We provide guarantees that the generated distribution g o P is indeed close to Q as well as the inverse mapping (gt) pushes Q close to
Theorem 4.1 (Generative Property for Approximators of Regularized Correlations). Let IP, Q be two continuous probability distributions on = Rp with finite second moments. Let e² X R be the optimal convex potential:
[/x ((z)dP(x) + ((s)a())
e arg min Corr(P, ()) = arg min VeConvex
(13)
veConvex
Let two differentiable convex functions ret X + R and 1bt y - IR satisfy for some E R:
Corr (P, 3 A) < [/x 1/ (z)d + I th (v)) + < Corr(P, Q) +e.
(14)
Equals B
Assume that ebt is 3 strongly convex (3t > 1 0) and B smooth (Bt 2 3t), Assume that tfft has bijective gradlent V.jt. Then the following inequalities hold true:
1. Correlation Upper Bound (regularized correlations dominate over the true ones)
Corr (P. Q ttt (T;;; A) (P
(i.e. e x20)
Forward Generative Property (mapping g" Vilit pushes P to be O(e)-close to ! W3lg's oP
B (Bi)2..
A
Inverse Generative Property (mapping (g²) pushes Q to Oce)-close to P.)
w QQ w3 Q)P)
3

---

Published as a conference paper at ICLR 2021
Informally, Theorem 4.1 states that the better we approximate correlations between P and Q by potentials v/t v/², the closer expect generated distributions g o TP and (g*)~1 o Q to be Q and respectively in the W2 sense. We prove Theorem 4.1 and provide extra discussion on smoothness and strong convexity Section A.2 Additionally, we derive Theorem A.3 which states analogous generative properties for the mapping obtained by the base method with single potential and no regularization.
Due to the Forward Generative property of Theorem 4.1 one may view the optimization of regularized correlations (1) as a process of minimizing W3 between the forward generated g o and true Q distributions (same applies to the inverse property). Wasserstein-2 distance prevents mode dropping for distant modes due to the quadratic cost. See the experiment Figure in Appendix C.3
The following Theorem demonstrates that we actually can approximate correlations as well as required if the approximating classes of functions for potentials are large enough.
Theorem 4.2 (Approximability of Correlations). Let P, Q be continuous probability distributions on X = y = l2d with finite second moments. Let v²" y - IR be the optimal convex potential.
Let xx Vy be classes of disferentiable corvex functions X R and y IR respectively and
1. B E xx with ex-close gradient to the forward mapping V in 2 + +R,, sense:
V
def
/
( s Ex.
V
Jx
and th is -smoothh
3ap E Vy with cy-close gradient to the inverse mapping
V in c2 + RD,, Q) sense:
def
| | (v) (1
S sy.
Let (1tt)51) be the minimizers of the regularized correlations within x x My
(161 (1) arg min Corr (P, Q ;.
(15)
VEVx.WEVy
Then the regularized correlations for (t), (t) satisfy the following inequality:
B
Corr( (I,, Q wtt (t; < Corr(P, P, Q) クタ (B* Vey V + (B* Vey + vex) (Vey)
2 eyl, (16)
i.e. regularized correlations do not exceed true correlations plus O(ex Ey) term.
By combining Theorem 4.2 with Theorem 4.1 we conclude that solutions v1, oi of 15) push and Q to be O(ex + Ey) -close to Q and IP respectively. In practice, it is reasonable use input- convex neural networks classes of functions Vxx, Vy. Fully-connected ICNNs satisfy universal approximation property Chen et al. (2018).
In Appendix A.3
prove that method can be applied to the latent space scenario. Theorem A.4 A.4 states that the distance between the target and generated (latent space generative map combined with the decoder) distributions can upper bounded by the quality of the latent fit and the reconstruction which demonstrates how our
loss of the auto-encoder In Appendix [A.4] we prove Theorem A.5 A.5) method can be applied non-continuous distributions IP and Q
EXPERIMENTS
In this section, experimentally evaluate the proposed model In Subsection 5.1
apply we consider
method to estimate optimal transport maps in the Gaussian setting. In Subsection 5.2
2 'The code is written on PyTorch framework and is publicly available at htt / thub. com comm wwt.a///mmmaaaaooooaaaccoo Texkorot n/ www./wwssssseeeeeee n2perssteinneennn n2 www.eeerr...eeee........ veNetworks.
7

---

Published as a conference paper at ICLR 2021
experiment with image-to-ing to-image style translation. In Appendix C we provide training details and additional experiments on color transfer, domain adaptation and toy examples. The architectures of input convex networks that ICNNs) are described in Appendix [ The provided results are not intended to represent the state-of- the-art for any particular task the goal is show the feasibility of our approach and architectures.
latent space mass transport. In Subsection 5.3
(Dense/Conv
5.1 OPTIMAL TRANSPORT BETWEEN GAUSSIANS
We test our method in the Gaussian setting P. Q N Mue, Στ), (po, Eg). It is one of the few setups with the ground truth OT mapping existing in closed form. We compare our method [W2GN] with quadratic regularization approach by (Seguy et al. (2017) [LSOT] and minimax approaches by Taghvaei & Jalali (019) [MM-1] and by Makkuva et al. (2019) [MM-2].
To assess the quality of the recovered transport map Vel. we consider unexplained variance per-
centage: L2-UVP(VVL (V = 100 [000[11V' V I1 Var([)] %. Here V is the optimal transport map. For values x 0%, Vabi is a good approximation of OT map. For values 2 100%, map Vest is nearly useless. Indeed, trivial benchmark V (x:) EVly) provides 22 (V- 100%.
As it is seen from Table 1, LSOT leads to high error which grows drastically with the dimension. W2GN, MM-1 and MM-2 approaches perform nearly equally in metrics. It is expected since they both optimize analogous objectives. These methods compute optimal transport maps with low error (C*-UVP<3% even in IR 1096). However, it is seen from the convergence plot in Figure 2 our approach converges several times faster: it naturally follows from the fact that MM approaches contain an inner optimization cycle. We discuss the experiment in more detail in Appendix C.4
Dimension D=4096
2
32
64
128
256|512|
2048
LSOT
|<1
3
7..
14.3
23
31.7
16.9
>50
500
Min-max-v1
MM-1
|<||<|
<1
<1
[1|
1.2
1.4
1.3
1.5|
1.6
1.8
2.7
230
Min-maxx-2
W2GN tours)
MM-2
|<||
<1
[1
<1
1
1.1
1.3
1.5
2.1
0
1000
3000 4000
W2GN
|4|44|<1|
[4
<1
1.1/13]
1.3
1.8
1.5
Time passed, seconds
Table 1: Comparison of C2-UVP (%) for LSOT, MM-1,
Figure 2: Comparison of convergence of W2GN (ours), MM-1, MM-2 approaches.
MM-2 and W2GN (ours) methods in D 2, 4, ...,. 212
5.2 LATENT SPACE OPTIMAL TRANSPORT
We test our algorithm on CelebA image generation (64 x 64). First, we construct the latent space distribution by using non- variational convolutional auto-encoder to encode the images to 128-dimensional latent vectors. Next, we use pair of DenseIC- NNs to fit cyclically monotone mapping to transform standard normal noise into the latent space distribution.
Method
FID
AE: Dec Enc(X))
7.5
Raw Decode: Dec(Z)
31.81
W2GN+AE: Dec(gi(2)
17.21
WGAN-QC Gen(Z)
14.41
In Figure 3 we present images generated directly by sampling from standard normal noise before (Ist row) and after (2nd row) applying out transport map. While our generative mapping does not perform sigmificant changes, its effect is seen visually as well as confirmed by improvement of Frechet Inception Distance (FID, see Heusel et al. (2017) provide the score of a recent Wasserstein GAN by Liu et al. (2019). In Appendix C.5) we provide additional examples and the visualization of the latent space.
Table 2: FID scores for 64 x 64 generated images.
Table 2 For comparison, also
Decoded Z-N(O,1)
Decoded g'(Z)
Figure 3: Images decoded from standard Gaussian latent noise (Ist row) and decoded from the same noise transferred by our cycle monotone map (2nd row).
8

---

Published as a conference paper at ICLR 2021
5.3 MAAGE-TOIMMAGE STYLE TRANSLATION
In the problem of unpaired style transfer, the learner gets two image datasets, each with its attributes, e.g. each dataset consists of landscapes related to particular season. The goal is to fit mapping capable of transferring attributes of one dataset to the other one, e.g. changing winter landscape to a corresponding summer landscape.
The principal model for solving this problem is CycleGAN by Zhu al. (017). It 4 networks and optimizes a minimax objective to train the model. Our method uses 2 networks, and has non-minimax objective.
We experiment with ConvICNN potentials on publicly availaible Winter2Summer and Photo2Cezanne datasets containing 256 256 pixel images. We train our model on mini-batches of randomly cropped 128 x 128 pixel RGB image parts. The results on Winter2Summer and
Photo2Cezanne datasets applied to random 128 128 crops are shown in Figure
Winter (Weal)
Summer (Real)
Photo (nwea))
Casame's Art (Gevertedd
Summee
Winter
(a) Winter2Summer dataset results.
(b) Photo2Cczanne datasct results.
Figure 4: Results of image-to-image style transfer by ConvICNN, 128 128 pixel images.
Our generative model fits cycle monotone mapping. However, the desired style transfer may not be cycle monotone. Thus, our model may transfer only some of the required attributes. For example, for winter-to-summer transfer our model learned to colorize trees to green. Yet the model experiences problems with replacing snow masses with grass.
As Seguy et al. (211)) noted, OT is permutation invariant. It does not take into account the relation between dimensions, e.g. neighbouring pixels or channels of one pixel. Thus, OT may struggle to fit the optimal generative mapping via convolutional architectures (designed to preserve the local structure of the image). Figures 4a and 46 demonstrate highlights of our model. Yet we provide examples when the model does perform well in Appendix C.8
To fix the above mentioned issue, one may consider OT for the quadratic cost defined on the Gaussian pyramid of image Burt & Adelson (1983) or, similar perceptual losses used for super-resolution Johnson et al. (2016), consider perceptual quadratic cost. This statement serves as the challenge for further research.
CONCLUSION
In this paper, we developed an end-to-end algorithm with a non-minimax objective for training eyclically monotone generative mappings. i.e. optimal transport mappings for the quadratic cost. Additionally, we established theoretical justification for our method from the approximation point of view. The results of computational experiments confirm the potential of the algorithm in vari- ous practical problems: latent space mass transport, mrage-to-image color/style transfer, domain adaptation.
ACKNOWLEDGEMEENS
The work was partially supported by the Russian Foundation for Basic Research grant 21-51-12005 NN1O_a

---

Published as a conference paper at ICLR 2021
REFERENCES
Pedro C Alvarez-FEstebnn Del Barrio, JA Cuesta-Albertos and C. Matrán. A fixed-point approach to barycenters in wasserstein space. Journal of Mathematical Analysis and Applications. 441(2): 744-762, 2016.
Brandon Amos, Lei Xu, and Zico Kolter. Input convex neural networks. In Proceedings of the 34th International Conference on Machine Courning-Voluma 70, pp. 146-155. JMLR. org, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07875 2017.
Wasserstein gan.
arXiv preprint
Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas
Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European Conference Computer Vision (ECCV), pp. 447-463, 2018.
Yann Brenier. Polar factorization and monotone rearrangement of vector- functions. Commu- nications on pure and applied mathematics, 44(4):375-411 1991.
Peter Burt and Edward Adelson. The laplacian pyramid as a compact image code. IEEE Transactions communications, 3144)5332-540 1983.
Rick Chartrand, Brendt Wohlberg, Kevin Vixie, and Erik Bollt. gradient descent solution the mongeekanntorvvic problem. Applied Mathematical Sciences, 3(22): (021):071-1080 2009.
Yize Chen, Yuanyuan Shi, and Baosen Zhang. Optimal control via neural networks: A convex approach. arXiv preprint arXiv: 1805.11851.33 2018
Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. IEEE transactions on pattern analysis and machine intelligence, 39(9):15331185, 2016.
Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamoniy. Joint distribution optimal transportation for domain adaptation. In Advances in Neural Information Processing Systems, pp. 3730-3739, 2017.
Werner Fenchel. conjugate convex functions. Canadian Journal of Mathematics, 1(1):33-77, 1949.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- fion processing systems. pp. 2672-2680, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems, pp. 5767-5777, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Justin Johnson, Alexandre Alahi, and Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 694-711. Springer, 2016.
Sham Kakade, Shai Shalev-Shwartz, and Ambuj Tewari. On the duality of strong convexity and strong smoothness: Learning applications and matrix regularization. Unpublished Manuscript, http://ttic. uchicago.
pdf. 2(1), 2009.
Leonid Kantorovitch. On the translocation of masses. Management Science, 5(1):1-4, 1958.
Tero Karras, Timo Aila. Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 2017.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 141266980, 2014.
10

---

Published as a conference paper at ICLR 2021
Na Lei, Kehua Su, Li Cui, Shing-Tung Yau, and Xianfeng David Gu. A geometric view of optimal transportation and generative model. Computer Aided Geometric Design, 68:1-21, 2019.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems, pp. 2203-2213, 2017.
Huidong Liu, Xianfeng Gu, and Dimitris Samaras. Wasserstein gan with quadratic transport cost. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4832 4841, 2019.
Ashok Vardhan Makkuva, Amirhossein Taghvaei, Sewoong Oh, and Jason D Lee. Optimal transport mapping via input convex neural networks. arXiv preprint arXiv: /908. 10962, 2019.
Robert J McCann et al. Existence and uniqueness of monotone measure-preserring maps. Duke Mathematical Journal, 30(2):309-324) 1995.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv:1411.1784 2014.
arXiv preprint
Youssef Mroueh and Tom Sercu. Fisher gan. In Advances in Neural Information Processing Systems, pp. 2513-2523, 2017.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, pp. 271-279, 2016.
François-Pierre Paty, Alexandre d' d'Aspremont, and Marco Cuturi. Regularity as regulariza- tion: Smooth and strongly convex brenier potentials in optimal transport. arXiv preprint arXiv:1905.10812, 2019.
Gabriel Peyré. Mathematical foundations of data sciences. def. 1(2m):2m, 2018
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-6):55-667, 2019.
Julien Rabin, Sira Ferradans, and Nicolas Papadakis. Adaptive color transfer with relaxed optimal transport. In 2014 IEEE International Conference on Image Processing (ICIP), pp. 4852-4856. IEEE, 2014.
levgen Redko, Nicolas Courty, Rémi Flamary, and Devis Tuia. Optimal transport for multi-source domain adaptation under target shift, arXiv preprint 1eXiv:180:. 04899, 2018.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv: arXiv: (t:::::05000777 2015.
Ralph Rockafellar. Characterization of the subdifferentials of convex functions. Pacific Journal of Mathematics, 17(3):97-510) 1966.
Vivien Seguy, Bharath Bhushan Damodaran, Rémi Flamary, Nicolas Courty, Antoine Rolet, and Mathieu Blondel. Large-scale optimal transport and mapping estimation. arXiv preprint
arXiv:1711.02283. 2017.
Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, and Ferenc Huszár. Amortised map inference for image super-resolution. arXiv preprint arXiv:1610.044400 2016.
Amirhossein Taghvaei and Amin Jalali. 2-wasserstein approximation via restricted convex potentials with application to improved training for gans. arXiv preprint arXiv: (XX2::/90007799 2019.
Cédric Villani. Topics in optimal transportation. Number 58. American Mathematical Soc., 2003.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Abhay Yadav, Sohil Shah, Zheng Xu, David Jacobs, and Tom Goldstein. Stabilizing adversarial nets with prediction methods. arXiv preprint 1705 2017.

---

Published as a conference paper at ICLR 2021
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to to-imageeeeeeeee translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference computer vision, pp. 2223-222222 2017.
A PROOFS
In Subsection A.1 we provide important additional properties of Wasserstein-2 distance and related
L -pacce required to prove main theoretical results.
In Subsection A.2 we prove our main Theorems 4.1 and 4.2 Additionally, we show how our proofs
can be translated to the simpler case (Theorem A.3) Le. optimizing correlations with single
potential by using the basic approach of Taghvaei & Jalali 2019).
In Subsection A.3 we justify the pipeline of Figure [3
To do this, we prove theorem that makes it possible to estimate the quality of the latent space generative mapping combined with the decoding part of the encoder.
In Subsection A.4, we prove useful fact which makes it possible to apply our method to distributions which do not have density.
A.1 PROPERTIES OF WASSERSTEIN-2 METRIC AND RELATION TO L SPACES
To prove our results, we need to introduce Kantorovich's 's formulation of Optimal Transport Kan torovitch (958) which extends Monge's formulation ( For given transport cost c: X y and probability distributions P and Q on X' and y respectively, we define
/
Cost(IP Q))
min wEll(P
/xxy czz, y)dp(z.)).
(17)
where [I,,, Q) is the set of all probability measures on X' x whose marginals are PP and Q respectively (transport plans). If the optimal transport solution exists in the form of mapping
g": X + minimizing (3), then the optimal transport plan in is given by μ² [id. g")
Otherwise, formulation (77 can be viewed as a relaxation of
For quadratic cost c(x, 1) 1 || y1|², the root of (17) defines Wasserstein-2 distance (W2). metric on the space of probability distributions. In particular, it satisfies the triangle inequality, i.e. for every triplets of probability distributions F1, P2, P's on X' IR. we have
(P(PI, Pa) S (2(Pi,P2) + (2(22,PPs)
(18)
We will also need the following lemma.
Lemma A.1 (Lipsehitz property of Wasserstein-2 distance). Let TP, Pt be two probability distributions with finite second moments on X1 C CRP: Let T : Xi - X₂ C Dr be measurable mapping with Lipschitz constant bounded by L. Then the following inequality holds true:
W2(T o P.T PP)) < (P,,PP))
(19)
i.e. the distribution distance between P. F' mapped by T does not exceed the initial distribution distance multiplied by Lipschitz constant L.
Proof. Let n E P2) be the optimal transport plan between P, and P2. Consider the distribu- tion on X₂ x X given by a T n², where mapping T is applied component-wing wise. The left and the right marginals of are equal to T PI and T P2 respectively. Thus, μ E II(T o P1, P2) is transport plan between T P, and T P2. The cost of μ not smaller than the optimal cost, i.e.
/
|
x' (u(xxx²).
WT PH
(20)
Next, we use the Lipschitz property of T and derive
r
x'1l
[Txx) T(x'))12
dp(z,z') = 1μ T w=]
dp (x, E
1xxxx
2
X1xX
2
12

---

omitting the Published as a conference paper at ICLR 2021
1,2111 xx'12
| x'112 du (x, = L² 1P)(),PP))
du (x, = 12
(21)
2
XxxX
2
To finish the proof, we combine 20 with and obtain the desired inequality T9
Throughout the rest of the paper, 22 RR, P)) denote the Hilbert space of functions
f: X + IR. with integrable square w.r.t. probability measure P. 'he corresponding inner product for f2 f2 RR,, P) is denoted by
(21,2)) def [ (1100)))(0)))))))))))
We use || 11e /1:-)> denote the corresponding norm induced by the inner product. Lemma A.2 (L" inequality for Wasserstein-2 distance). Let IP be a probability distribution on 2i C R:. Lef 7,, T2 E (X - 1P). Then the following inequality holds true:
2 Tix))) 12) W o P,TT2PT
Proof. We define the transport plan μ = [Ti, o P between Ti o and T2 P and, similar to the previous Lemma A.I use the fact that its cost is not smaller than the optimal cost.
A.2 PROOFS OF THE MAIN THEORETICAL RESULTS
First, we prove our main Theorem Then we formulate and prove its analogue (Theorem A.3) for the basic correlation optimization method ( with single convex potential. Next, we prove our main Theorem 4.2 At the end of the subsection, we discuss the constants appearing in theorems: strong convexity and smoothness parameters.
Proof of Theorem 4.1
We split the proof into three subsequent parts.
Part Upper Bound on Correlations.
First, establish lower bound for regularized correlations Corr(P, Q tt x1; regularization term.
/ (O))(()))) [w. (VT
I
[2y.
(22)
+ / (v. - (Q(2) Z
1
1, [(V
31 V (v)
(23)
J, (s.
(e (e
(v. Ve (z)) (24)
Com(P.
Corr(F
(vel Vest Vi V
V o
V
(idy, V - Velt ) + Corr(IP, Q) = (25)
gt
idy; V
Viit
V
V
+ Corr(IP Q)
gt
2 Vet o V idyll
+ A
V
idy, V V c
+
V
V
2
Corr(Q, P)
V
V
V
idyll
2
13

---

Published as a conference paper at ICLR 2021
Corr(Q, +
1 2
V Vizi -idy]
Bt V 2 Ve=
idy liē (26)
2
In transition to line 22). we use change variables formula IP 3t-strong convexity of function of and then add zero in line 244
1/
Q.
In line (23) we use
Next, for simplicity,
replace integral notation with (Y + R2, Q) notation starting from line (25).
We add the omitted regularization term back (26) and obtain the following bound:
Corr(P, Q vh,, ()
Corr Corr(Q, P)) + 2A ()) V Veli - +
1
1
V idy] + B1 Vb V
(27)
2
31
Since A > gt the obtained inequality proves that the true correlations Corr(P, Q are upper bounded by the regularized correlations Corr(P, Q wt vi;;; A). Note that if the optimal map Vist
is 2 8 strongly convex. the bound 27 is tight. Indeed, it turns into equality when we substitute
V (V - Vea"
Part Inverse Generative Property.
We continue the derivations of part 1. Letu Vuple Vist idy and Vi
Vil
By matching
with (4) we obtain
2
2
71 39 Mull
v 31g
(28)
Bi
2
Now we derive an upper bound for ||ul| For fixed u we have
V
2e (A 3 lull
+
Next, we apply the triangle inequality:
1
IIV s
+ v 8Tu
+ I1
ull S 20 - (A-
luul + Vst
ule- (29)
Vgi
Vgt
gt
The expression of the right-hand side of (29 attains its maximal value conclude that
2c
at
|u/l2
V 1xB1->
2r
Agr
V
V = 110113 <
Bt
Finally, apply C2-inequality of Lemma A.2 distribution Q mappings V and ViFt and obtain W P., Q S -....c. i.c. the desired upper bound on the distance between the generated and target distribution.
Part Forward Generative Property.
We recall the bound (28). Since all the summands are positive, we derive
| < X
(30)
We will use
30 obtain upper bound on Vib** llb
14

---

Published as a conference paper at ICLR 2021
To begin with, we note that since vt is strongly convex, its convex conjugate eit is 3 smooth.
Thus, gradient Vehi is şi -Lipschit. We conclude that for all x, x' A:
1 - B | 2'1..
(z) V (z))
(31)
We raise both parts (31 into the square, substitute x = Vest o V (v) and = Ves o Vala (v)
and integrate over y w.r.t Q The obtained inequality is as follows:
J (vo Ve
1 (Bi)3 Vi vol(y) - Velt (b) (Q) (32)
/
Next, we derive
V - V / |i (v) (v) s
J (BT
Vibi o Ve) - Vut V (v) ||
(33)
(3t
In transition to line
33), we use the previously obtained incquality (32
Next, we use the triangle inequality for lle to bound
V
lq
S
V
of
V
V F
11
2c 37
2c
2c
2c
+
vA)
(34)
31
Bt
1
31
A31
Next, we derive lower bound for the left-hand side of (34) by using Bm-sshots st. For all t E X' we have
V - vot(2') Bt -- ='Il-
(35)
We raise both parts of (35) to the square, substitute x (य) and x' = (v), and integrate over y w.r.t. Q:
(1)( s (B3)²) / -
V Vot(g)
V V
Ve (v)) (36)
Next, we use 36 derive
F 2 || (v) v=(2))
J (Bt)z Vusl V
(Bi / ( (x) V 2d (x)
1
2
F W oP, Q
(37)
(Bi
In line 37 we use the L property of Wasserstein-2 distance (Lemma A.2
We conclude that
2
V S
+ vx)²
Agt
V
and finish the proof.
is quite straightforward to formulate analogous result for the basic optimization method with single potential (8. We summarise the statement in the following
15

---

Published as a conference paper at ICLR 2021
Theorem A.3 (Generative Property for Approximators of Correlations). Let IP, Q be two continuous probability distributions on y X R. with finte second moments. Let vh" + R be the convex minimizer of Corr(P, Q.
Let differentiable dit 3 -troongyy convex and B -smooth (B) 2 B > function vt X - R satisfy
Corr (P.Q + $ [/x ( + / ((V)D(()) +c= Corr(P, @) +
(38)
Then the following inequalities hold true:
Forward Generative Property (map g V pushes be (le)-close Q
Waig e P, Q) = Wa(V 4t o P, 2 S Ble;
Inverse Generative Property (map (g²) = Vipt = (V pushes Q to be (ce)-llose to IP)
(3((g))-1 Q. P) S
é 3r
Proof. First, note that 0 by the definition of 1/.". Next, repeat the first part of the proof of Theorem (4. by substituting whd vt. Thus, by using Vipt Visi idy we obtain the following simple analogue of formula
81 Corr(P, Q +1) 2 Corr(P, Q) VV V
(39)
i.e
1 Iv Visile- Thus, by using Lemma A.2 we immediately derive W3 V Q) S मो. ie. inverse generative property.
To derive forward generative property, note that B Bl-smoothness of e means that की is g -trongly convex. Thus, due to symmetry of the objective, we can repeat all the derivations w.r.t vt instcad of git in order to prove
W o < Bic,
i.e. forward generative property.
Proof of Theorem 4.2 We repeat the first part of the proof of Theorem 4.1 but instead of exploiting
strong convexity of 20 to obtain an upper bound on the regularized correlations, we use B
smoothness obtain lower bound. The resulting analogue (40) to
(27 is as follows:
Cor P. Q th a A) - Corr( (P, Q) 5
(A
o V
idy 11 +
BX
1
V idy]
VB
V
V
S (40)
VB
|/
V
idy
11 +
1
110 2
idylls
VB
V
V
(41)
B
A
B
V|V V
idy
+
V
V
idy
lee
|
+
V
V
13
(42)
2
2
Here in transition from line
40 to
we apply the triangle inequality.
For every y E Y' we have || raise both parts of this inequality to the power 2 and integrate over ) w.r.t. Q. We obtain
(y)
V x V (v)ll - x
| (
V V We
V V
Velet
V
)3*)
V
V 18 S (B ey.
(43)
16

---

Published as a conference paper at ICLR 2021
Now recall that since Vela
P,
we have
V|
Val*
cx.
(44)
idy
Next, we combine 43 and
44
apply the triangle inequality to bound
V Vuik
V - idylle 5
V (1)
V
V +
V
VV
Vp Ve 1
Idy
B Vey + Vex.
(45)
We substitute all the bounds (40):
Corr Q bx 2.. >) Corr(P,, (P, Q) <
A 1B* Voy vxx)² (B* Voy+ vcx) (Vey) + BV
2 cy.
(46)
2
and finish the proof by using
Corr(P, P, Q st 1 () s Corr(P, Q 1 ",,,
which follows from the definition of tt, 33
One may formulate and prove analogous result for the basic optimization method with a single potential 88. However, do not include this in the paper since similar result exists Taghvaei Jalaii 2019
All our theoretical results require smoothness or strong convexity properties of potentials. We note that the assumption of smoothness and strong convexity also appears in other papers on Wasserstein-2 optimal transport, see e.g. Paty et al. (2019).
The property of B-smoothness of a convex function và means that its gradient V has Lipshitz constant bounded by B. In our case, constant B serves as a reasonable measure of complexity of the fitted mapping Vit: it estimates how much the mapping can warp the space.
Strong convexity is dual to smooothness in the sense that convex conjugate : to 3-strongly convex function v -smooth (and vise-versa) Kakade et al. (2009). In our case, 3-strongly convex potential
means that its inverse gradient mapping (V V not significantly warp the space, i.c. has Lipshitz constant bounded by 4.
Recall the setting of our Theorem 4.2 Assume that the optimal transport map Vel between
and Q is gradient of 3-strongly convex (3 > and B-smooth (B < oc) function. In this case, by considering classes V = Vy equal to all min(3, -sttrongyy convex and max(B, 3 -smmoohh functions, by using our method (for any optimal Vul".
})) we will exactly compute correlations and find the
A.3 FROM LATENT SPACE TO DATA SPACE
In the setting of the latent space an autoencoder and combine it with the decoder to obtain a generative model. The natural question is how close decoded distribution is the real data distribution S used to train encoder. The following Theorem states that the distribution distance of the combined model can be naturally divided into two parts: the quality of the latent fit and the reconstruction loss of the auto-encoder.
transport, fit generative mapping the latent space of
Theorem A.4 (Decoding Theorem). Let S be the real data distribution on S C BB Let S + y RDD be the encoder and u y -> R be L-Lipschitz decoder.
Assume that latent space generative model has fitted a map g" XX' - y that pushes some latent distribution P X D to be close Q u in
i.e.
IP, (, c.
17

---

Published as a conference paper at ICLR 2021
Then the following inequality holds true:
W2 VV2 P. (P)
L Esll
-
(47)
Generated data distribuon
Autoencoder's reconstration loss
where u o ge the combined generative model.
Proof. We apply the triangle inequality and obtain
W2(v o g P. & W2(v o g o P, Q + W2(U o Q.S).
(48)
Let Pt = g o P be the fitted latent distribution (I3((P) Q S c). We use Lipschitz Wasserstein-2 property of Lemma A.I and obtain
Ii 2 L. W2 Q) 2 W2(vo Pt, oo Q = W o o
(49)
Next, we apply L -properyy (Lemma A.2 to mappings idg, v o a and distribution S, and derive
Ralls u()) 1 2 W
(50)
The desired inequality
47 immediately results from combining (48 with 449.. 500p
A.4 EXTENSION TO THE NON-EXISTENT DENSITY CASE
Our main theoretical results require distributions P, Q have finite second moments and density on X' - = R.. While the existence of second moments is a reasonable condition, in the majority of practical use-c the density might not exist. Moreover, is typically assumed that the supports of distributions are manifold of dimension lower than D or even discrete sets.
One may artificially smooth distributions PP, Q by convolving them with a random white Gaussian noise 66 NO, a² TD) and find a generative mapping O X + ) between smoothed TP * A and natural that the generative properties of g' as a mapping between IP A and Q A will transfer to generative properties of g¹ as mapping between P. Q. but with some bias depending on the statistics of A.
Q For Wasserstein-2 distances,
Theorem A.5 (De-smoothing Wasserstein-2 Property). Let F. Q be two probability distributions X = y = R with finite second moments. Let A = (O, p²Ip) be a Gaussian white noise. Let P* A and Q A be versions of and Q smoothed by A. Let T X + y be L-Lipschitz measurable map satisfying
W2(T [P A), Q Al) Ve.
Then the following inequality holds true:
- Ve.
D
Proof.
We apply the triangle inequality twice and obtain
W2(T Q S W2(T P. To * A]) + W2(T [P A] A)) W2( * Q).
Consider transport plan p(y.3') E I[ A), Q) satisfying ply y) A(y -y). is given by
v). The cost of j
| /
ll - 1/12 dA dv)(() = 1 dQ(y) = Do²
Do2
2
Since the plan is not necessarily optimal, we conclude that W2( A), Q) Dg Analogously, we conclude that (IP [PP
Al)
< Do²
2 Next, we apply Lipschitz property of Wasserstein-2 (Lemma
A.I and obtain:
D
(T o P, P* A)) L W2 |P A)) Lo
FFrmm the practical point of view, smoothing is equal to adding random noise distributed according A to samples from P. Q respectively.
18

---

Published as a conference paper at ICLR 2021
Finally, combine all the obtained bounds and derive
D W2 c S (L 1j + Vc.
B NEURAL NETWORK ARCHITECTURES
In Subsection B. we describe the general architecture of the input convex networks. this section,
we describe particular realisations of the general architecture that use in experiments: DenseICNN in Subsection B.2 and ConvICNN in Subsection B.3
B.1 GENERAL INPUT-CONVEX ARCHITECTURE
We approximate convex potentials by Input Convex Neural Networks Amos et al. (017). The overall architecture is schematically presented in Figure
Linear Block
Canv2D
Dense
ConvAct
ConvAct
ConvAct
x, Input
Convexity
Preserving Block
PosConw2D
(x),, output
ConvMonAct
ConvMonPoal
Figure 5: General architecture of an Input Convex Neural Network.
Input convex network consists of two principal blocks:
1. Linear (L) block consists of linear layers. Activation functions and pooling operators in the block are also linear, e.g. identity activation or average pooling.
Convexity preserving (CP) block consists of linear layers with non-negative weights (excluding biases). Activations and pooling operators in this block are convex and monotone.
Within blocks it possible to use arbitrary skip connections obeying the stated rules. Neurons of Block can be arbitrarily connected to those of CP block by applying convex activation and adding the result with a positive weight. comes from the convex function arithmetic that every neuron (including the output one) in the architecture of Figure 5 is convex function of the input
In our case, we expect the network to be able easily fit the identity generative mapping
g(z) = (+(()==
iec. (z)) _ ||||² + is a quadratic function. Thus, mainly insert quadratic activations between and CP blocks, which differs from Amos et al. (2017) where no activation was used. Gradients of input quadratic functions correspond to linear warps of the input and are intuitively highly useful as building blocks (in particular, for fitting identity mapping).
We specific architectures which fit to the general scheme shown in Figure 5 ConvICNN is used for image-prooe processio tasks, and DenseICNN is used otherwise. The exact architectures are described in the subsequent subsections.
We use CELU function as a convex and monotone activation (within CP block) in all the networks. We have also tried SoftPlus among some other continuous and differentiable functions, yet this
Unlike activations within convexity preserving block, convex activation between L and CP block may he monotone, e.g. (z)) x² he used as activation.
'It is possible into insert batch norm and dropout to L and CP blocks as well as between them. These layers do not affect convexity since they can be considered (during inference) as linear layers with non-negative weights.
19

---

Published as a conference paper at ICLR 2021
negatively impacted the performance. The usage of ReLU function is also possible, but the gradient of the potential in this case will be discontinuous. Thus, will not Lipschitz, and the insights of our Theorems and 4.2 may not work.
As a convex and monotone pooling (within CP block), is possible to use Average and LogSum pooling (smoothed max pooling). Pure Max pooling should be avoided for the same reason ReLU activation. However, in ConvICNN architecture we use convolutions with stride instead of pooling, Subsection B3
In order to use insights of Theorem 4.1 we impose strong convexity and smoothness on the potentials. As noted in Appendix A.2 B-smoothness of convex function is equal strong convexity of its conjugate function (and vise versa). Thus, we make both networks vha V/. to be = z strongly convex, and cycle regularization keeps ~ 1/2 B smoothness for dho RJ (Vw)) and V... R2 (20) In
practice, we achieve strong convexity by adding extra value @ ||]]|
| to the output of the final neuron of a network. In all our experiments, we set , 1000000 In addition to smoothing, strong convexity guarantees that Viñe and Vihh are bijections, which is used in Theorems 4.1 4.2
2
B.2 DENSE INPUT CONVEX NEURAL NETWORK
For DenselCNN, we implement Convex Quadratic layer each output neuron of which is convex quadratic function of input. More precisely, for each input E Nu it outputs (cq, (z), C9N... (z)) IR,Nas, with
cq, (z) Anx) + (bn: 2) + Cn.
for positive semi-definite quadratic form A N.N....N vector b N,,, and constant R. Note that for large Ni.. the size of such layer grows fast, i.e. 2 ( Noa).. To fix this issue, we represent each quadratic matrix as product A, = F F,, where
R the matrix of rank at most r. This helps to limit optimization to only positive quadratic forms (and, in particular, symmetric), and reduce the number of weights stored for quadratic part to Orr Nin Now). Actually, the resulting quadratic forms A, will have rank at most
The architecture shown in Figure 0 We use Convex Quadratic Layers in DenselCNN for connecting input directly to layers of fully connected network. Note that such layers (even full rank) do not blow the size of the network when the input dimension is low, e.g. in the problem of color transfer.
ConvQuad
ConvQuad
ConvQuad
(x(x), output
PosDense
PosDense
X, Input
CEUU
CELU
Convexity
Preserving Block
Figure 6: Dense Input Convex Neural Network.
The hyperparameters of DenseICNN are widths of the layers and ranks of the convex input-quadratic layers. For simplicity, we use the same rank r for all the layers. We denote the width of the first convex quadratic layer by hg and the width of le + 1-th Convex Quadratic and k-tth Linear layers by hg. The complete hyperparameter set of the network is given by (r; ho; hi....,
'Imosiig smoothness & strong convexity can be viewed as regularization of the mapping: it does not perform large/small warps of the input. See e.g. Paty et al. (209)).
20

---

Published as a conference paper at ICLR 2021
B.3 CONVOLUTIONAL INPUT CONVEX NEURAL NETWORK
We apply convolutional networks to the problem of unpaired image-to-image style transfer. The architecture of ConvICNN is shown in Figure The network takes input image (128 x 128 with 3 RGB channels) and outputs a single value. The gradient of the network w.r.t. the input serves as generator our algorithm.
X. 128x128 Input Image
Conv20. 3x3
Comv20 Sx3
Conv2D, 3x3
Conv20 3x3
PosC PosConv2D, 3x3, stride->
PosConn2D 3x3, stride=2
PosConv2D 3x3, stride-2
PrsConv20 stride->
PosConv20 3x3, stride-2
PosDense: 2048 to 128
(3 RBG channeis)
Bon
PosDense 128 to
PosDense 32
PosDense 32 to
4(x),, output
tBu
cau
cBU
EBu
Comv2D, 3x3
Conv2D 3x3
CCom2D 3x3
(Comv20, 3x3
Figure 7: Convolutional Input Convex Neural Network. All convolutional layers have 128 channels.
Linear and Convexity preserving blocks are successive, and skip connections are used. Block consists of two separate parts with stacked convolutions without intermediate activation. The square of the second part is added the first part and is used as an input for the CP block. All convolutional layers of the network have 128 channels (zero-padding with offset is used).
C EXPERIMENTAL DETAILS AND EXTRA RESULTS
In the first subsection, we describe general training details. In the second subsection, we discuss the computational complexity of single gradient step of our method. Each subsequent subsection corresponds to particular problem and provides additional experimental results and training details: toy experiments in Subsection C.3 and comparison with minimax approach in Subsection C.4 latent
space optimal transport Subsection C.5 image-to-image color transfer in C.6 domain adaptation
in Subsection C.7 Amage-to-image style transfer in Subsection C.8
C.1 GENERAL TRAINING DETAILS
The code is written on PyTTrrh framework. The networks are trained on single GTX 1080Ti. The numerical optimization procedure is provided in Algorithm D of Section 4.1
In each experiment, both primal th and conjugate th,, potentials have the same network architecture. The minimization of (12) done via mini batch stochastic gradient descent with weight clipping (excluding biases) in CP block to the [0, +o)) 10 We use Adam Kingma & Ba (014) optimizer.
For every particular task we pretrain the potential network zhe by minimizing mean squared error to satisfy Via (z) æ and copy the weights to V.. This provides good initialization for the main training, i.e. Vehy and Vehta are mutually inverse.
Doing tests, we noted that our method converges faster if we disable back proppppopagttion through term Viha which appears twice the second line of (12). In this case, the derivative w.r.t. w is computed using the regularization terms only [T] This heuristic allows
additional memory and computational time because smaller computational graph is built. We used the heuristic in all the experiments.
In the experiments with high dimensional data (latent space optimal transport, domain adaptation and style transfer), we add the following extra regularization term the main objective (12)
(x, l1g_. o go(x) x dz) S. Vi V .II| dT))
(51)
We tried to use softplus, exponent on weights and regularization instead of clipping, none these worked well.
"The term (Ver y) becomes redundant for the optimization. Yet it remains useful for monitoring the convergence.
21

---

Published as a conference paper at ICLR 2021
Term (51) is analogous to the term Ry(6, w) given by (II. It also keeps forward ge and inverse 95 generative mappings being approximately inverse. From the theoretical point of view, it is straightforward to obtain approximation guarantees similar to those of Theorems [4.1 and 4.2 for the optimization with two terms: Rx and Ry. However, we do not to include R2 in the proofs in order to keep them simple.
C.2 COMPUTATIONAL COMPLEXITY
The time required to evaluate the value of (12) and its gradient w.r.t. 0, w is comparable up to constant factor that of single evaluation of (x)(
This claim follows from the well-known fact that gradient evaluation (aha(z) of ho RP - RR when parameterized as neural network, requires time proportional to the size of the computational graph. Hence gradient computation requires computational time proportional to the time for evaluating the function (x) itself. The same holds true when computing the derivative with respect to x. Thus, the number of operations required to compute different terms in [12. e.g. Ve) th (Vets (v)) and Vulp V (y). is also linear w.r.t. the computation time of tha (2) or, equivalently, vhw (2)) As
consequence, the time required for the forward pass of (12) is larger than the forward pass for the (z)
only to a constant factor. Thus, the backward pass for (12) with respect to parameters of ICNNs and w is also linear in the computation time of the (1:).
We empirically measured that for our DenselCNN potentials. the computation of gradient of 12 w.r.t. parameters 0, w requires roughly 8-12x more time than the computation of (2)). Evaluating Vioo Vil takes roughly 3-4x more time than evaluating 1/o (xr).
C.3 TOY EXPERIMENTS
In this subsection, we test our algorithm on 2D toy distributions from Guirajani et al. (2017) Seguy al. (2017). In all the experiments, distribution P the standard Gaussian noise and Q is a Gaussian mixture Swiss roll.
Both primal and conjugate potentials the and V.. have DenseICNN [2: 128; 128, 64) architecture.
Each network has roughly 25000 trainable parameters. Some of them vanish during the training because of the weight clipping. For each particular problem the networks are trained for 30000 iterations with 1024 samples in mini batch. Adam optimizer Kingma& & Ba 2014) with Ir 10-3 is used. We put À= = 1 in our cycle regularization and impose additional 10 L regularization on the weights.
For the case when Q is a mixture of 8 Gaussians, the intermediate learned distributions are shown in Figure g The overall structure of the forward mapping has already been learned on iteration 200, while the inverse mapping gets learned only on iteration RJ 2000. This can be explained by the smoothness of the desired optimal mappings V and ViDT The inverse mapping V has large Lipschitz constant because it has unsquee7e dense masses of Gaussians. In contrast to the inverse mapping. the forward mapping V has to squeeze the distribution. Thus, it is expected to have lower Lipschitz constant (everywhere except the neighbourhood of a central point which is fixed point of V due to symmetry).
When Q is mixture of 100 gaussians (Figure 99. our model learns all of the modes and does not suffer from mode dropping. We do not state that the fit is perfect but emphasize that mode collapse also does not happen.
Additional examples (Gaussian Mixtures & Swiss roll) are shown in Figures 10a 1ob and 9
Note that all our theoretical results require distributions P. Q to be smooth. Our method fits con- tinuous optimal transport map via differentiable w.r.t. input ICNNs with CELU activation. At the same time, Makkuva et al. (2019) also uses ICNNs with discontinuous gradient w.r.t. the input, e.g. by using ReLU activations to fit discontinuous generative mappings. We do not know whether our theoretical results can be directly generalized to the discontinuous mappings case (without using smoothing we suggested in Subsection A.4). However, we note that the usage of ICNNs with discontinuous gradient naturally leads to "torn" generated distributions, see an example in Figure [I] While the fitted mapping is indeed close to the true Swiss Roll in W, sense, it clearly suffers from
22

---

Published as a conference paper at ICLR 2021
truth hypa.
Groudd Ouipst
Nevorse Macseng Ve.
TorMard Raspeng Mpo
Initial distributions and Q
(b) Generated distributions after 200 iterations.
Irvers Mapping ing 1Pg
Fo ward Mapping Vay
Wwwrse Happing Vpc
Ferword Magping Sps.
(c) Generated distributions after 2000 iterations.
(d) Generated distributions after 30000 itcrations.
Figure 8: Convergence stages of algorithm applied fitting cycle monotone mappings (forward and inverse) between distributions IP (Gaussian) and Q (Mixture of 8 Gaussians).
Figure 9: Mixture of 100 Gaussians Q and distribution Vaha o IP fitted by our algorithm.
"tom" effect. From the practical point view, this effect seems to be similar to mode collapse, well-known disease of GANs.
C.4 GAUSSIAN OPTIMAL TRANSPORT DETAILS AND DISCUSSION
We consider the Gaussian setting P.O = // Ee ). NO Eq) for which the ground truth OT solution
2016 Theorem 2.3). Considering non-centered PP, Q is unnecessary since (F.., Q) = 1pp #Q (.. Qo). where Fo. Qo are centered copies of
has closed form, see (Á Álvarezz-Esteban al.
F. Q. In D--dimensional space, /Dp (/2 analogously) is initialized as S ASp, where Sp E Op
random rotation, A is diagonal with eigenvalues ...... 1 ,..,, 2). b P-V4. The ground
23

---

Published as a conference paper at ICLR 2021
Gouud Ouipst
Foraadd Mapa Say
all wh te dpoiiir iiii
Morts
(a) Mixture of 49 Gaussians Q and distribution Veda P Q fitted by our algorithm.
(b) Swiss Roll distribution Q and distribution Vide
P fitted by algorithm.
Figure 10: Tov distributions fitted by our algorithm.
2.3
63
-03
-20
Figure 11: An example of a "torn" generative mapping to a Swiss Roll by a gradient of ICNN with ReLU activations.
truth optimal transport map from P to Q is linear and given by
Vsl (z) Σx ( Ep z.
We compare our approach with the method by (Seguy et al. (017) [LSOT] and the minimax approaches
by Taghvaei & Jalali (2019) [MM-1], Makkuva et al. (2019) [MM-2). We recall the details LSOT's regularized optimization of OT maps and distances. The objective is given by
min vo.
vo( (T())))(())) I (v)d(())))
1
/
[(x,)) vo(x) (P
2c xxx
where we defined via = |al match our notation (zho, w.)) We use L2 regularizer (but not entropy) since it empirically works better (as noted in LSOT paper). Potentials the tha are NOT restricted to be convex. The transport plan T' o IP Q is recovered via the barycentric projection, see (Seguy et al. 2017 Section 4).
2 (x(x) and Via Ivll ee (v) to make LSOT notation (ug, v.)
We set À = min (D,50) 50) for our method and e = 0.01 for LSOT (chosen empirically). In all the methods we use DenseICNN[1; D, D, D, L] of Subsection B.2 In LSOT, we do not convexify nets (do not clamp weights), i.e. they are "usual" unrestricted neural networks (empirically selected as the best option) as originally implied in LSOT paper.
It follows from the Table Section that that LSOT leads to high bias error which grows drastically with the dimension. While thcoretically - 0 should solve the bias problem, practically small leads optimization instabilities. LSOT"s drawback is that estimation of the regularizer requires sampling from joint measure P x Q on R2D.D For the majority of pairs (s. y) the 1.2 regularizer vanishes (the oo). In contrast LSOT, our cycle regularizer uses samples only from
effect worsens with marginal measures (Q or P) on RD, The biasing effect is studied theoretically (our Theorems 4.1 4.2), and the bias is inexistent when optimal potentials v.* 2 are contained in the approximating function classes.
24

---

Published as a conference paper at ICLR 2021
Dimension D=64
Dimension D=256
40
Min-max-v1
Min-max-v1
5
20
Min-max-v2
Min-max-v2
W2GN (ours)
W2GN (ours)
0
50
100
150
200
0
50
100
150
200
250
300
Dimension D=1024
Dimension D=4096
750
100
Min-max-v1
Min-max-v1
Min-max-v2
250
Min-max-v2
W2GN (ours)
W2GN (ours)
0
100
200
300
400
0
1000
2000
3000
4000
5000
Time passed, seconds
Time passed, seconds
Figure 12: Comparison of convergence speed of W2GN, MM-1 and MM-2 approaches in dimensions D 64, 256, 1024, 4096.
W2GN (ours). MM-1 and MM-2 methods are capable of computing maps and distances with low error (C²-UVP<3?) even in R-096). However, as it is from the convergence plots in Figure 12 our approach converges several times faster: it naturally follows from the fact that MM-1, MM-2 approaches contain inner optimization cycle.
C.5 LATENT SPACE OPTIMAL TRANSPORT DETAILS
We follow the pipeline of Figure 13 below. The latent space distribution is constructed by using convolution auto-encoder to encode CelebA images into 128-dimensional latent vectors respectively, To train the auto-encoder, we use perceptual loss on features of pre-trained VGG-16 network.
g
distribution
distribution
Figure 13: The pipeline of latent space mass transport.
We use DenselCNN 256; 256; 128; 64] fit eyclically monotone generative mapping to transform standard normal noise into the latent space distribution. For each problem the networks are trained for 100000 iterations with 128 samples a mini batch. Adam optimizer with Ir 3 10-4 is used. We put À = 100 as the cycle regularization parameter.
We provide additional examples of generated images Figure We also visualize the latent space distribution of the autoencoder and the distribution fitted by generative map in Figure 15 The FID scores presented Table 2 are computed via PyTorch implementation of FIDD Score As benchmark score we added WGAN-QC by Liu et al. (2019).
Finally, we emphasize that cyclically monotone generative mapping that we fit is explicit. Similarly to Normalizing Flows Rezende & Mohamed] (2015) and in contrast to other methods, such as Lei et al. (2019), it provides tractable density inside the latent space. Since Velu s (Vvs) differentiable and injective, one may use the change of variables formula for density q(v) [det Vi (y) ((Veis to study the latent space distribution.
25

---

Published as a conference paper at ICLR 2021
Decoded
Z-N(O,))
Decoded g'(Z)
Figure 14: Images decoded from standard Gaussian latent noise (1st row) and decoded from the same noise transferred by our cycle monotone map (2nd row).
2
Ve(22
Latent Space Distribution
PCA Companent No.
-12
-75
-5.0
7.5
10.0
-75
-3.0
-25
2.5
PCA Component No. 2
PCA Component No.
PCA Component No. 2
Figure 15: A pair of main principal components of CelebA Autoencoder's latent space. From left to right: Z ~ (0, I) [blue], mapped Z by W2GN [red], true autoencoders latent space [green]. PCA decomposition is fitted on autoencoders latent space [green].
C.6 COLOR TRANSFER
The problem of color transfer between images is to map the color palette of the image into the other one in order make it look and "feel" similar to the original.
13
Optimal transport can be applied to color transfer, but it is sensitive to noise and outliers. To avoid these problems, several relaxations were proposed Rabin et al. (2014); Paty et al. (209)). These approaches solve a discrete version of Wasserstein-22 OT problem. The computation of optimal transport cost for large images is barely feasible or infeasible all due to extreme size of color palettes. Thus, the reduction of pixel color palette by k-means clustering is usually performed to make OT computation feasible. Yet such reduction may lose color information.
Our algorithm uses mini batch stochastic optimization. Thus, it has no limitations on the size of color palettes. On training, we sequentially input mini-batches of images' pixels (E 13)) into potential networks with DenceICNN [3: 128; 128, 64] architecture The networks are trained for 5000 iterations with 1024 pixels in a mini batch. Adam optimizer with 10-3 30 is used. We put À 3 as the cycle regularization parameter. We impose extra 10- L L'eeatlyty on the weights.
The color transfer results for megapixel images are presented in Figure [6a] The corresponding color palettes are given in Figure 16b Additional example of color transfer are given in Figure 17
C.7 DOMAIN ADAPTATION
The domain adaptation problem is learn model f (e.g. a classifier) from source distribution Q This model has to perform well on different (related) target distribution IP.
Most of the methods based OT theory solve domain adaptation explicitly by transforming distri- bution P into Q and then applying the model f to generated samples. In some cases the mapping g y (which transforms IP Q) is obtained by solving a discrete OT problem Courty al.
Images may have unequal size. Yet they are assumed to have the same number channels, e.g. RGB ones. "Since our model is parametric, the complexity of fitted generative mapping g¹ R3 Rd between the palettes depends the size of potential networks.
26

---

Published as a conference paper at ICLR 2021
(a) Original images (on the left) and images obtained by color transfer (on the right). The sizes of images are 3300 4856 (first) and 2835 x 4289 (second).
(b) Color palettes (3000 random pixels, best viewed color) for the original images (on the left) and for images with transferred color (on the right).
Figure 16: Results of Color Transfer between high resolution images (=: 10 megapixel) by pixel- wise cycle monotone mapping.
(a) Original images (on the left) and images obtained by color transfer (on the right).
(b) Color palettes (3000 random pixels, best viewed in color) for the original images (on the left) and for images with transferred color (on the right).
Figure 17: Results of Color Transfer between images by a pixel-wise cycle monotone mapping.
2016 2017): Redko et al. (2018), while some approaches adopt neural networks to estimate the mapping 9 Bhushan Damodaran et al. (2018); /Seuuy et al. (2017).
We address the unsupervised domain adaptation problem which is the most difficult variant this task. Labels available only in the source domain, do not any information about the
27

---

Published as a conference paper at ICLR 2021
labels. Our method trains g gradient of convex function. It can be applied arriving samples which are not present in the train set.
We test model on MNIST (+= 60000 images; 28 x 28) and USPS (se 10000 images; rescaled to 28 x 26) digits datasets. We perform USPS - MNIST domain adaptation. To do this, we train LeNet > 99%-accuracy classifier h MNIST. Then, we apply h to both datasets, extract 84 last layer features. Thus, we form distributions Q (features for MNIST) and (features for USPS).
To fit eycle monotone domain adaptation mapping, potentials. We train our model on mini-batches of 64 samples for 10000 iterations with cycle regularization 1000. We use Adam optimizer with Ir 10 and impose 10-7 -p-ennlyty the weights of the networks.
DenselCNN [32; 128; 128, 128]
Similar Seguy et (017)), we compare the accuracy of MNIST 1-NN classifier f applied to features x ~ IP of USPS with the same classifier applied to mapped features g (az). 1-NN is chosen as the classification model in order to eliminate any influence of the base classification model on the domain adaptation and directly estimate the effect provided by our cycle monotone map.
The results of the experiment are presented in Table Since domain adaptation quality highly depends on the quality of the extracted features, we repeat the experiment times, i.e. we train 3 LeNet MNIST classifiers for feature extraction. We report the results with mean and central tendency. For benchmarking purposes, we also add the score of 1-NN classifier applied to the features of USPS transported to MNIST features by the discrete optimal transport. can be considered as the "most straightforward" optimal transport map 15
Repeat 75. 7%
Repeat
Repeat 3
Average (14 + 0)
Target features
77%
75.4%
76 + 0.8%
Mapped features (W2GN)
80.6%
80.3% 75.7%
82.7%
81.2 E 1%
Mapped features (Discrete OT)
76%
76.1%
75.9 + 0.4%
Mapped features Seguy al. (2017)
77.92%
Table 3: 1-NN classification accuracy on USPS + MNIST domain adaptation problem.
Our reported scores are comparable to the ones reported by Seguy et al. (2017). We did not reproduce their experiments since (Seguy al. (2017) does provide the source code for domain adaptation. Thus, we refer the reader directly to the paper's reported scores (Table of SSguy et al. (2017). first column with scores).
For visualization purposes, we plot the two main components of the PCA decomposition of feature spaces (for of the conducted experiments) in Figure I8 MNIST features, mapped USPS features by using our method, original USPS features.
C.8 IMAGE-TO-IMAGE STYLE TRANSFER
We experiment with ConvICNN potentials on publicly available Winter2Summer and Photo2Cezanne datasets containing 256 x 256 pixel images.
We train our model on mini batches of 8 randomly cropped 128 x 128 pixel RGB image parts. As an additional augmentation, we use random rotations (#: ( ). random horizontal flips and the addition of small Gaussian noise (a = 0.01). The networks are trained for 20000 iterations with cycle regularization
35000. We use Adam optimizer and impose additional 10-1 L -'penalyyy on the weights of the networks. Our scheme of style transfer between datasets is presented in Figure T9 We provide the additional results for Winter2Summer and Photo2Monet datasets Figures 20b and 20a respectively.
In all the cases, our networks change colors but preserve the structure the image. none of the results did note that the model removes large snow masses (for winter-to-summe transform)
In contrast to our method, can not he directly applied to out-of-train-smmple examples. Morcover, its computation is infeasible for large datasets.
"htpps: : /// /i thub. com/ wtt.b...w..jmmaaa/////ttrrreeeee eGAN- and andi-axa-i-xxppppi
28

---

Published as a conference paper at ICLR 2021
Transported USPS Features
Original USPS Features
15
PCA Campanent
-10
Component
PCA Component #2
PCA Component #2
Figure 18: A pair of main principal components of feature spaces. From left to right: MNIST feature space, mapped USPS features by W2GN, original USPS feature space. PCA decomposition is fitted on MNIST features. Best viewed in color (different colors represent different classes of digits 0 - 9).
y=VyL
ViL
Original
Modiffed
Figure 19: Schematically presented image-to-image style transfer by a pair of ConvICNN fitted by method.
Photo (Real)
Summer (Real)
(Real)
Cazanne's (Generated)
Winter (Genarated)
(Generated)
(a) Results of cycle monotone image-to-image style transfer by ConvICNN on Photo2Cezanne dataset, 128 128 pixel images.
(b) Results of cycle monotone image-to-image style transfer by ConvICNN on Winter2Summer dataset, 128 x 128 pixel images.
Figure 20: Additional results of image-to-image style transfer on Winter2Summer and Photo2Monet datasets.
covers green trees with white snow (for summer-to-winter). We not know the exact explanation for this issue but we suppose that the desired image manipulation simply may not be cycle monotone.
For completeness of the exposition, we provide some results of cases when our model does not perform well (Figure 21). In Figure@lar the the model simply increases the green color component, while in Figure 21b decreases this component. Although in many cases it is actually enough to transform winter summer (or vice-versa), sometimes more advanced manipulations are required.
In the described experiments, we applied our method directly to original images without any specific preprocessing feature extraction. The model captures some of the required attributes to transfer, but sometimes it does not produce expected results. To fix this issue, one may consider OT for the quadratic cost defined features extracted from the image embeddings of images (similar to
29

---

Published as a conference paper at ICLR 2021
Winter (Real)
Summer (Real)
Summer (Generated)
Winter (Generated)
(a) Winter to summer transform.
(b) Summer to winter transform.
Figure 21: Cases when the cycle monotone image-to-ime to--iage style transfer by ConvICNN on Win- ter2Summer dataset works not well, 128 x 128 pixel images.
the domain adaptation in Subsection C.7 or latent space mass transport This statement serves as the challenge for our further research.
30


Reviews:
[{"review_id": "yRMmEAZB2Q8", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Wasserstein-2 Generative Networks (W2GNs) which is an optimal transport framework for learning generative models. Unlike minimax problems of Wasserstein GANs, the proposed approach which is based on minimizing the 2-Wasserstein distance reduces to a single-level optimization problem. The paper numerically shows that the new approach enjoys faster convergence and improves upon the performance scores of Wasserstein GANs and other optimal transport baselines. While the paper's idea on applying optimal transport tools for training generative models seems interesting, the discussed theoretical and numerical results are not supportive enough to show that the proposed approach indeed improves upon WGANs. Also, the theoretical sections have been written in a convoluted way with several weakly supported claims and the final algorithm has not been stated clearly. I, therefore, do not recommend the paper for acceptance.\n\nTo further explain my concerns, let me start with section 3 which reviews the dual formulation to 2-Wasserstein distance in Eq. (8) and also the connection to the convex conjugate optimization in Eq. (9). Here, the paper vaguely mentions that the optimization problem for computing the convex conjugate is \"convex and very complex\" followed by a brief explanation which I do not find satisfactory. Specifically, the paper's way of reasoning does not convince me why it is necessary to switch from Eq. (8)  to the paper's formulation in (12). Both the issues mentioned for problems (8) and (9) also apply to the formulation in (12) as (12) still requires taking the gradient of the convex conjugate \\psi_w. Also, I highly recommend replacing the terms \"very complex\" and \"impossible\" with more solid theoretically or numerically supported statements. \n\nNext, let me refer to the final sentences of the first paragraph of section 4.1: \" Yet such a problem is still minimax. Thus, it suffers from typical problems such as convergence to local saddle points, instabilities during training and usually requires non-trivial hyperparameters choice.\" These sentences argue that every minimax problem suffers from instability and convergence to local solutions. However, the paper's own formulation in (12) also leads to a non-convex optimization problem for which the authors show no convergence guarantees to a global solution. The paper should either remove these sentences or precisely explain why the proposed non-convex problem enjoys better convergence properties than the minimax problems. Let me also add that while Eq. (12) states an optimization problem, it still does not completely characterize a learning algorithm, because it is unclear how one wants to take the gradient of \\psi_w's convex conjugate. I think the paper should include an algorithm clearly stating the steps of learning the generative model.\n\nFinally, the theoretical guarantees in Theorem 4.1 and 4.2 do not analyze the algorithm's performance for the class of input convex neural nets. Theorem 4.1 connects the optimization error to the closeness of the generative and underlying distributions. Yet, it does not provide any guarantee on how large the optimization error could be for convex neural nets. The result of Theorem 4.2 also immediately follows from the assumptions and offers little understanding of the algorithm's performance with convex networks, since it considers the set of all differentiable functions instead. The theoretical guarantees should somehow analyze the algorithms' convergence and approximation properties for convex neural nets rather than all differentiable convex functions. Overall, the paper seems to carry several nice ideas, but the theoretical discussion needs to be significantly improved.  \n\n*****\nReview update: I thank the authors for their response and for revising the paper based on the comments. The revision addresses several of my concerns. I still think the theoretical guarantees should be stronger and therefore change my score to borderline 5.", "extra_scores": {}, "extra_text": {}}, {"review_id": "S4LferGWJrC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "From my perspective, this is a much needed and love-to-see work for the line of neural generative modeling. Previous approaches were dominated by GAN based approaches which require solving a minimax optimization which has technical hurdles in practice. This paper reviews the literature of OT theory, and provides a comprehensive explanation about the relation between W_2 and cyclical monotonic map. In particular, it motivates the opportunity to approximate Eq. (8) with Eq.(12), as which a non-minimax formulation exist. \n\nPros:\n\n- a non-minimax formulation was proposed for neural generative modeling\n- theoretical properties are derived for the proposed formulation Eq.(12)\n- experiments are preliminary but promising. \n\nCons:\n\n- The paper misses a discussion about how gradient-based optimization is done for Eq.(12) in implementation. For example, it is not clear what functions are supposed to be parameterized as a neural network. Is it \\varphi_\\theta and \\bar{\\varphi}_\\omega? If so, how the gradient is calculated for \\Delta \\bar{\\varphi_\\omega}? Would it involve high order auto-differentiation? \n- The authors have not outlined how computational feasible their approach is? This makes me less confident about their approach regarding processing more difficult datasets or tasks. \n- I strongly suggest the authors to re-organize their presentations, and focus more on what was actually calculated in practice. This allows others to follow their work and reproducing their experiments. Some mathematical introduction is nice, but they have to be directly related to the approach of this paper. \n\n\n\n---------------\npost revision:\nI read authors' revision, and it is indeed an improved version with more readabilities. Since I am familiar with the OT literature, the mathematical presentation is clear enough for me to follow. The motivation of this paper is clear. The presentation of algorithm is also clear enough in the revision. But I can not say much for wider audience this paper may target. \n\nGiven the significance and popularity of GAN related work, this paper seems a big deal to the community. This's why I intend to give 8 instead of 7 for the revised paper.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NrCemgimoKR", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new method for learning an optimal pushforward (for the quadratic cost) from a distribution to another distribution based on samples of both distributions. The optimal transport problem is first written equivalently as a minimax problem over set of convex functions, as in Makkuva et al. 19. Then, the convex functions are parametrized as Input Convex Neural Networks (ICNN). To my understanding, the novelty of the approach is to replace the minimax problem over the parameters of the ICNN by an easier to solve a regularized minimization problem. The authors prove the consistency of their approach: loosely speaking, they show that true minimizers of their approximate problem are epsilon minimizers of the original optimal transport problem. They also provide promising numerical experiments.\n\n\n\nThe paper proposes a simple fix to drawbacks of previous works of on the topic, which e.g. propose minimax approaches. Moreover, the consistency of the approach is rigorously proven.\nOn the other hand, one could argue that the novelty is marginal (replacing a minimax problem by a regularized minimization). Moreover, the consistency result assumes access to a true minimizer of the regularized problem (which is highly non convex). Finally, the writing of the paper can be improved.\n\n    \nI rate the paper as marginally below.\n\n    \nRegarding the novelty of the approach. This can be addressed by SOTA numerical experiments, but this is not the case (although the numerical experiments are well explained and detailed). \nRegarding the consistency result. The authors provide an approximation result that does not take into account the optimization of the regularized problem. It is a difficult a problem but the authors could at least mention this difficulty. \nRegarding the overall quality of the paper. \n- The structure of the introduction is strange. Several paragraphs are used to explain points which cannot be understood at this stage or provide too much details. Several paragraphs are used to explain that we look for well structured pushforward mappings. Why not directly mention the equivalent optimal transport problem ? \nIn dimension one all continuous invertible mappings sending P to Q are monotone? I am not sure. What is a maximal monotone mapping (eq 2)? Considerations about the dimension of the space should not be in the intro in my opinion (the reader has the feeling that the authors are defending themselves). Considerations on minimax vs non minimax and end to end are too difficult to understand at this stage (same in Section 2).\n-Section 2. The notation for the Fenchel transform is misleading. It should be with a star...\n-Section 3. Why do we need a positive density? Optimal pushforward mappings exist (in both directions) if both distributions admit a density wrt Lebesgue measure. What is the point of the footnote 4? The set \"Convex\" is not defined. Convex potentials are called discriminators. Why this name? I guess that there is a connection with GANs but I don't see (actually this work is about fitting an optimal pushforward mapping, which is something independent from the GAN setup). \n-Section 4. Check Eq 10 (symbol = and alignment). Consideration on why non minimax is better than minimax should be explained before.  \"Fully-connected ICNNs satisfy universal approximation property\". What does it imply for the assumptions of Th 4.2? Are they satisfied or not?\n\n\n\nMINOR:\n- Question: Are gradients wrt to x of the ICNN easy to compute?\n- parenthesis in the equation after Forward Generative Property\n- \"diffirentiable\" (several times)\n- I have seen similar considerations (using ICNN to fit gradient of convex functions) in https://openreview.net/pdf?id=rklx-gSYPS", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
The reviewers have  different views on the papers but agreed that the paper can be accepted. However, they suggested
some points of improvements including the writing (clarity and style) and experiments showing strong improvements
compared to WGAN.
Decision: Accept (Poster)