How Well Do WGANs Estimate the Wasserstein Metric?
Authors: Anton Mallasto
Guido Montúfar
Augusto Gerolin
Venue:  (2020)
Keywords: Optimal Transport
Wasserstein Metric
Generative Adversial Networks

Abstract: Generative modelling is often cast as minimizing a similarity measure between a data distribution and a model distribution. Recently, a popular choice for the similarity measure has been the Wasserstein metric, which can be expressed in the Kantorovich duality formulation as the optimum difference of the expected values of a potential function under the real data distribution and the model hypothesis. In practice, the potential is approximated with a neural network and is called the discriminator. Duality constraints on the function class of the discriminator are enforced approximately, and the expectations are estimated from samples. This gives at least three sources of errors: the approximated discriminator and constraints, the estimation of the expectation value, and the optimization required to find the optimal potential. In this work, we study how well the methods, that are used in generative adversarial networks to approximate the Wasserstein metric, perform. We consider, in particular, the $c$-transform formulation, which eliminates the need to enforce the constraints explicitly. We demonstrate that the $c$-transform allows for a more accurate estimation of the true Wasserstein metric from samples, but surprisingly, does not

Under review as a conference paper at ICLR 2020
How WELL Do WGANs ESTIMATE THE WASSER- STEIN METRIC?
Anonymous authors Paper under double-blind review
ABSTRACT
Generative modelling is often cast as minimizing a similarity measure between a data distribution and model distribution. Recently, a popular choice for the similarity measure has been the Wasserstein metric, which can be expressed in the Kantorovich duality formulation as the optimum difference of the expected values of potential function under the real data distribution and the model hypothesis. In practice, the potential is approximated with a neural network and is called the discriminator. Duality constraints on the function class of the discriminator are enforced approximately, and the expectations are estimated from samples. This gives at least three sources of errors: the approximated discriminator and constraints, the estimation of the expectation value, and the optimization required find the optimal potential. In this work, study how well the methods, that are used in generative adversarial networks to approximate the Wasserstein metric, perform. We consider, particular, the c-transform formulation, which eliminates the need to enforce the constraints explicitly. We demonstrate that the c-transform allows for more accurate estimation of the true Wasserstein metric from samples, but surprisingly, does not perform the best in the generative setting.
INTRODUCTION
Recently, optimal transport (OT) has become increasingly prevalent in computer vision and machine learning, allows for robust comparison of structured data that can be cast as probability measures, e.g., images, point clouds and empirical distributions (Rubner et al. 2000 Lai & Zhao 2014), or more general measures (Gangho et al.) 2019). Key properties of OT include its non-singular behavior, when comparing measures with disjoint supports, and the fact that OT inspired objectives can be seen as lifting similarity measures between samples to similarity measures between probability measures. This is in stark contrast to the more traditional information theoretical divergences, which rely only comparing the difference in mass assignment. Additionally, when the cost function c is related to a distance function by c(2, y) d (I:, v). p 2 1, the OT formulation defines the called Wasserstein metric, which a distance on the space of probability measures, i.e. symmetric and positive definite function that satisfies the triangle inequality. Despite its success, scaling OT to big data applications has not been without challenges, since it suffers from the curse of dimensionality (Dudley 1969) Weed & Bach 2017). However, significant computational advancements have been made recently, for which a summary is given by Peyré et al. (2019). Notably, the entropic regularization of OT introduced by Cuturr (2013) preserves the geometrical structure endowed by the Wasserstein spaces and provides an efficient way to approximate optimal transport between measures.
Generative modelling, where generators are trained for sampling from given data distributions, is popular application of OT. In this field, generative adversarial networks (GANs) by Goodfellow jet al. (2014) have attracted substantial interest, particularly due to their success in generating photo- realistic images (Karras et al. 2017). The original GAN formulation minimizes the Jensen-Shannon divergence between a model distribution and the data distribution, which suffers from unstable training. Wasserstein GANs (WGANs) (Arjovsky et al. (017) minimize the 1- Wasserrstein distance over the 12 metric, instead, resulting in more robust training.
The main challenge in WGANs is estimating the Wasserstein metric, consisting of estimating expected values of the discriminator from samples (drawn from model distribution and a given data distribution), and optimizing the discriminator maximize an expression of these expected

---

Under review as a conference paper at ICLR 2020
values. The discriminators are functions from the sample space the real line, that have different interpretations in different GAN variations. The main technical issue is that the discriminators have to satisfy specific conditions, such as being 1-Lipschitz in the 1-Wasserstein case. In the original paper, this was enforced by clipping the weights of the discriminator to lie inside some small box, which, however, proved to be inefficient. The Gradient penalry WGAN (WGAN-GP) (Gulrajani et al.. 2017) was more successful at this, by enforcing the constraint through a gradient norm penalization. Another notable improvement was given by the consistency term WGAN (CT-WGAN) (Wei et al. 2018), which penalizes diverging from 1- I-Lipschitzness directly. Other derivative work of the WGAN include different OT inspired similarity measures between distributions, such as the sliced Wasserstein distance (Dehhpande et al.) 2018), the Sinkhorn divergence (Genevay et al. 2018) and the Wasserstein divergence (Wu et al. 2018). Another line of work studies how to incorporate more general ground cost functions than the metrrc (Adler & Lunz 2018 Mallasto et al. 2019 Dukler et al. 2019).
Recent works have studied the convergence of estimates of the Wasserstein distance between two probability distributions, both in the case of continuous (Klein et al. 2017] and finite (Sommerfeld & Munk 2018 Sommerfeld (017) sample spaces. The decay rate of the approximation error of estimating the true distance with minibatches of size N is of order (NN-1/d for the Wasserstein
distances, where d is the dimension of the sample space (Weed & Bach, 2017 Entropic regularized
optimal transport has more favorable sample complexity of order O //VVNV for suitable choices of regularization strength (see Genevay et al. 2019 and also Feydy et al. 2018 Mena & Weed 2019). For this reason, entropic relaxation of the 1- Wassersttii distance is also considered in this work.
Contribution. this work, we study the efficiency and stability of computing the Wasserstein metric through its dual formulation under different schemes presented in the WGAN literature. We present detailed discussion on how the different approaches arise and differ from each other qualitatively, and finally measure the differences in performance quantitatively. This is done by quantifying how much the estimates differ from accurately computed ground truth values between subsets of commonly used datasets. Finally, we measure how well the distance is approximated during the training of a generative model. This results in a surprising observation; the method best approximating the Wasserstein distance does not produce the best looking images in the generative setting.
2 OPTIMAL TRANSPORT
In this section, we recall essential formulations of optimal transport to fix notation.
Probabilistic Notions. Let (X, dx) and (Y, dy) be Polish spaces, i.c., complete and separable metric spaces, denote by P(X) the set of probability measures on X, and let f: X + be a measurable map. Then, given a probability measure μa E P(X'), we write f#1 for the push-forward of μ with respect to f. given by (=)(AA) p(f (A)) for any measurable A S y. Intuitively speaking, if g is a random variable with law μe, then f(E) has law fgp. Then, given v E PP))), we define
ADM(H, v) {r P(X x ))) (#1)#7 n, (T2) =v,,
(1)
where n. denotes the projection onto the 7th marginal. An element y ADM(H, v) is called an admissible plan and defines joint probability between and v.
Optimal Transport Problem. Given a continuous and lower-bounded cost cost function e: X' x + R. the optimal transport problem between probability measures μ E P(X') and v E P())) is defined as (Tr(H, == min [c,
yEADM(μ,*)
where 1f f(x)dμ(z) is the expectation of a measurable function f with respect to J..
Note that (2) defines a constrained linear program, and thus admits a dual formulation. From the perspective of WGANs, the dual is more important than the primal formulation, as it can be approximated using discriminator neural networks. Denote by I.a (z) = (f: X - IR [x] < xc) the set of measurable functions of μ that have finite expectations under μu, and by ADM(c) the set of admissible pairs (i,, v)) that satisfy
(xx) + (v)) I c(2,,) v), v(z,y) E X' x 3, s0 E (1), (²)
(3)
Then, the following duality holds (Peyré et al. 2019
Sec. 4)
sup (T,( (AM)))
[E] + E.[l]
(4)
2

---

Under review as a conference paper at ICLR 2020
When the supremum is attained, the optimal yp" 35* in (4) called Kantorovich potentials, which, in particular, satisfy & (²(x) + (v) = c(z, y) for any (z: y) E Supp(?"), ) where Y solves (2). Given ip, we can obtain admissable & satisfying (4) through the e-transform of yp
8-: R.
y inf {c(z, y) ((x))}
(5)
JE
so that (19; 909) E (D)(c) for any up E T. Moreover, the Kantorovich potentials satisfy tp 4 and therefore can be written as (Villani 2008 Thm. 5.9)
max OTe(u, v) (e:+")EADM(c)
[Eu(p] + [E
(6)
In other words, the (DM(c) constraint can be enforced with the c-transform, and reduces the optimization in (6) to be carried out over single function.
Wasserstein Metric. Consider the case when X' = y and c(2,, y) - (e 2 1, where we refer dx as the ground metric. Then, the optimal transport problem (2) defines the p- Wasserstein metric W Wp(u, v)= = OT (1, w) on the space
Pd (X) | (ME(()) / (x0,
oc for some zo E ex,,
of probability measures with finite p-moments. It can be shown that (P (2), W;) forms a complete, separable metric space (Villani 2008 Sec. Thm 6.16).
Entropy Relaxed Optimal Transport. We can relax (2) by imposing entropic penalization intro- duced by Cuturi 2013). Recall the definition of the Kullback-Leibler (KI.) divergence from v to y as
KT((I|l>> = / log
Pk
Padx:
(8)
Pe
where we assume that μ, V absolutely continuous with respect to the Lebesgue measure X on X' with densities Py Pus respectively. Using the KL-divergence as penalization, the entropy relaxed optimal transport is defined
(1, == min [E + eKL(vIIu @ v)),
yEADM(H.P)
where >> O defines the magnitude of the penalization, and denotes the independent joint distribution of μI and v. We remark that when c - 0, any minimizing sequence (-*) solving ( converges minimizer of (2). and in particular, OT,
OTe (H,
Analogously to (4). the entropy relaxed optimal transport admits the following dual formulation
(Peyré et al. 2019 Feydy al. 2018: Di Marino & Gerolin
2019)
(c+(p - B (10)
(H,
sup REL (E))
Exlpp + Ev[v] eEpor
exp
E
where (z, ((xx + (v)). In contrast to (4), this is an unconstrained optimization problem, where the entropic penalization can be seen as smooth relaxation of the constraint.
As shown by Feydy al. (2018): Di Marino & Gerolin 2019), a similar approach to (6) for computing the Kantorovich potentials can be taken in the entropic case, by generalizing the c-transform. Let Le (1) = {g X IR (exp((())))] )o), and consider the (c, -)tarassfoorm of o L.XP (p),
l ( (1) -e log (Le eepp (-
c(2, v) (-d(s-) - dy((x)
x(x)
(11)
As + 0, plo))( (y) (v), making the (c, -)ttnnsfform consistent with the c-transform. Analo- gously to (6) one can show under mild assumptions on the cost e (Di Marino & Gerolin 2019). that
OT (H,
max WE (M)
(En[sp) Ey (sp(oc)))
(12)
Sinkhorn Divergence. Since the functional OT fails to be positive-definite (e.g. OT u) * 0), is convenient to introduce the (p, -innkhoor divergence St with parameter c > O, given by
1,,,,) OT de (1, 2 (OT; (n. n) + +T, (v, v))))))
(13)
3

---

Under review as a conference paper at ICLR 2020
where the terms Oga ()) and oT (1, are added to avoid bias, as in general OT
24 (1, ()) + O. The Sinkhorn divergence was introduced by Genevay et al. (2018) and has the following properties: (i) it metrizes weak convergence in the space of probability measures; (ii) interpolates between maximum mean discrepancy (MMD),
00, and the p- p-Wasserstein metric, as - O. For more about the Sinkhorn divergence, see Feydy et al. (2018)
3 GENERATIVE ADVERSARIAL NETWORKS
Generative Adversarial Networks (GANs) are popular for learning to sample from distributions. The idea behind GANs can be summarized as follows: given source distribution μas E PP(R")), want to push it forward by parametrized generator gw' IR= + IR" so that a chosen similarity measure o between the pushforward distribution and the target distribution 14, P(R"<) is minimized. Usually the target distribution is only accessible in form of a dataset of samples, and one considers
an "empirical" version of the distributions. Note that n, ny is chosen, which is justified by the manifold hypothesis. This objective can be expressed as
min (( ( Pe).
(14)
The similarity o commonly estimated with a discriminator Yui R R. parametrized by w. whose role will become apparent below.
The vanilla GAN (Goodfellow al. 2014) minimizes an approximation the Jensen-Shannon (JS) divergence between the push-forward and target, given by
JS(v)lp) max (E (z)) + Eyou [log(1 y (v)))}
(15)
for probability measures i and The discriminator (uu is restricted to take values between 0 and 1, assigning a probability to whether a point lies in μ or v. It can be shown (Goodfellow et al.) 2014)..
that optimality the JS JS-divergence is recovered in (5). if optimized over all possible functions.
Substituting = 14 and y = (gw # in (15) yields the minimax objective for the vanilla GAN
min max {Ez-nu [log(pw())) + Ezaa, [log(1 (ul (2)))I}-
(16)
As mentioned above, in practice considers empirical versions of the distributions so that the expectations are replaced by sample averages.
The Wasserstein GANs (WGANs) (Arjovsky et al. 2017) minimize approximation to the 1- Wasserstein metric over the ground metric, instead. T'ee reason why the 1 Wasserstein case is special property of the c-transform of 1-Lipschitz functions, when d
considered is motivated for any metric if f is 1-Lipschitz, then f f== Kantorovich potential yp" solving the dual problem 66 is 1-Lipschitz when e d, and therefore the WGAN minimax objective can be written as
p==ff f (Villani 2008 Sec. 5). It can also be shown, that
min max (Ezup [x)] Egwn, IPw (2))). W
(17)
In the WGAN case, there is restriction on the range of you opposed to the GAN above. The assumptions above require enforcing P to be I-Lipschitz. This poses a main implementational difficulty in the WGAN formulation, and has been subjected a considerable amount of research.
In this work, we will investigate the original approach by weight clipping (Arjovsky et al. 2017) and the popular approach by gradient norm penalties for the discriminator (Gulrajani et al. 2017). We furthermore consider a more direct approach that computes the c- c-transform over minibatches Mall lasto al.) 2019), avoiding the need to ensure Lipschitzness. We also discuss an entropic relaxation approach through (c, é) <)-transforms over minibatches, introduced by Genevay et al. (0166). In the original work, the discriminator yuu is expressed sum of kernel functions, however, this work we will consider multi-layer perceptrons (MLPs), as we do with the other methods we consider.
3.1 ESTIMATING THE 1- WASSERSTEIN METRIC
In the experimental section, we will consider four ways to estimate the 1- I-Wasserstein distance between two measures i and v, these being the weight clipping (WC), gradient penalty (GP), e-transform and
4

---

where Under review as a conference paper at ICLR 2020
(c, -transffrmm methods. To this end, now discuss how these estimates computed in practice by sampling minibatches of size N from μ and v. yielding {z, J.N and {y: //2 1 respectively, at each training iteration. Then, with each method, the distance is estimated by maximizing model specific expression that relates to the dual formulation of the 1 1-Wasserstein distance in (6) over the minibatches. In practice, this maximization is carried out via gradient ascent one of its variants, such as Adam (Kingma & Ba 2014) or RMSprop (Tieleman & Hinton 2012).
Weight clipping (WC). The vanilla WGAN enforces K K--ippchitzzeess of the discriminator each iteration by forcing the weights W of the neural network to lie inside some box - S W S &. considered coordinate- wise, for some small & (É 0.01 in the original work). Here l stands for the hthh layer in the neural network. Then, the identity for the c-transform (with e = d) of 1-Lipschitz maps is used, and so can written as
{
N Pu - N N
N Σ Pel
max
(18)
Gradient penalty (GP). The weight clipping is omitted in WGAN-GP, by noticing that the Lipschitz condition implies that [/V=)=(<))] S holds for almost surely under μr and This condition can be enforced through the penalization term Ez~x max (0, (=) is some reference measure, proposed to be the uniform distribution between pairs of points of the minibatches by Gulrajani al. 2017). The authors remarked that in practice it suffices enforce [[Vay 1, and thus the objective can be written as
N
M
A M
max
N
Σ Pa (1))
N
Σ (v))
Σ (1
(19)
where À is the magnitude of the penalization, which was chosen be À 10 in the original paper, and { are samples from X-
c-transform. Enforcing 1-Lipschitzness has the benefit of reducing the computational cost of the c-transform, but the enforcement introduces an additional cost, which in the gradient penalty case is substantial. The ADM(c) constraint can be taken into account directly, as done in (4. p)- WGANs (Mallasto et al. 2019), by directly computing the c-transform over the minibatches as
(1)) P (v:) mjj min {c(#,, vi) (x;))
(20)
where = d2 in the 1-Wasserstein case. This amounts to the relatively cheap operation of computing the row minima of the matrix AG = c(z,, v:) (z,). The original paper proposes to include penalization terms to enforce the discriminator constraints, however, this is unnecessary as the c-transform enforces the constraints. Therefore, the objective can be written as
N
N
1 N
1 N
max
Σ
(x.) +
PS (S(U))
(21)
(c, --ttransform. As discussed in Section 2 entropic relaxation applied to Wi results in the (1, Sinkhorn divergence Si. which satisfies Si W, as - O. Then, Si can be viewed as a smooth approximation to W,. The benefits of this approach are that 4p,, is not required to satisfy the ADM(c) constraint, and the resulting transport plan is smoother, providing robustness towards noisy samples.
The expression 13 for St consists of three terms, where each results from solving entropy relaxed optimal transport problem. As stated by Feydy et al. (2018) Sec. 3.1), the terms OTς (u, p) and OT₃ (v,,v) v)) are straight- forward to compute, and tend to converge within couple of iterations of the symmetric Sinkhorn- -nopp algorithm. For efficiency, we approximate these terms with one Sinkhorn- iteration. The discriminator is employed in approximating (R, v), which is done by computing the (c, -)ttransform defined in [T] over the minibatches
N
(c.e) (v1) = -e leoog (wx) ~ ppl
1
exp
(pu(zz) c(es,
(22)
N
Ø=1
5

---

Under review as a conference paper at ICLR 2020
and write the objective
12
for the discriminator
N
1 N +
max
Pa (z,)
Σ N
(c.e) (4))
(23)
N
==1
EXPERIMENTS
We now study how efficiently the four methods presented in Section B. estimate the 1-Wasserstein metric. The experiments use the MNIST (LeCun et al. 1998), CIFAR- 10 (Krizhevsky et al.) 2009). and CelebA (Liu et al.) 2015) datasets. On these datasets, we focus on two tasks: ap- proximation and stability. By approximation we mean how well the minibateh-wis distance between two mea- sures can be computed, and by stability how well these minibatch- distances relate to the I-Wasserstein dis- tance between the two full measures.
(c.e) Transform c-Transfrom
GP
S
WC
W,
3.5
3.0
2.5
Implementation. In the approximation task, we model the discriminator as either (i) simple multilayer percep- tron (MLP) with two hidden layers of width 128 and ReLU activations, and linear output, or (ii) convolutional neu- ral network architecture (CNN) used in DCGAN (Radford et al. 2015). In the stability task we use the simpler MLPs for computational efficiency. The discriminator is trained by optimizing the objective function using stochastic batch gradient ascent. For the gradient penalty method, we use the Adam optimizer with learning rate 10-4 and beta values (0, 0.9). J For weight clipping we use RMSprop with learning rate 5 10-5 as specified in the original paper by Arjovsky et al. (017)). Finally, for the c- c-transform and the (c, e)-transform, we use RMSprop with learning rate 10
0.5
0.0
0 100
200
300
400
500
600
Iterations
Figure I: Estimating the distance be-
tween two standard 2-dimensional Gaus- sian distributions that have been shifted by 1[,
The estimated distances diss obtained from the optimization are compared ground truth values dground computed by POTH The (c, -ttransformm might improve on the POT estimates when dast > dground, as both values result from maximizing the same unconstrained quantity. This discrepancy cannot be viewed as error, which we quantify as
(drsst dground) = max(0, dground dest.).
(24)
Note that this is a subjective error based on the POT estimate that can also etr, and not absolute error. In practice POT is rather accurate, and we found this to make only a small difference (see Figure with the ground truth and estimated distances visualized). The weight clipping and the gradient penalty methods might also return a higher value than POT, but in this case it not guaranteed that the discriminators are admissible, meaning that the constraints of the maximization objective would not be satisfied. In the case of the c-transform, the discriminators are always admissible. However, the POT package's ol emd (used to compute the 1-Wasserstein distance ground truth) does not utilize the dual formulation for computing the optimal transport distance, and therefore we cannot argue in the same way as in the (c, -ttannssformm case. As the Sinkhorn divergence (13) consists of three terms that are each maximized, we measure the error as the sum of the errors given in (24) of each term.
Approximation. We divide the datasets into two, forming the two measures μ and v, between which the distance is approximated, and train the discriminators on 500 training minibatches μk C 1a and un v, k 1, ..., 500, of size 64. See Section 3.1 for how the discriminator objectives. Then, without training the discriminators further, we sample another 100 evaluation minibatches μu C approximate the minibatch wise distance
.... 100, and use the discriminators between each jn and 1/. This approximation will then be compared to the ground- -ruttt minibatch- wise distance computed by POT. We run this experiment 20 times, initializing the networks again each time, and report the average error in Table [] Note that the discriminators are not updated for the last 100 iterations. Results for toy example between two Gaussians are presented in Fig.
and 1/ C. 1
Python Optimal Transport, htwps:tpppp....... : http::///ooo....eeeeteeeeee readehedoss..eddeeeo io/

---

Under review as a conference paper at ICLR 2020
MLP
MNIST
CIFARTO
CelebA
WC
14.98 0.32
0..61
48.65 - 1.29
GP
14.89 = 0.38
27.14
48.00 - 2.88
o-transform
0,82 = 0.16
1.53 - 0.29
2.84 0.49
(c, 0.1 ttransttafoorm
0.43 + 0.17
1.29 0.48
2.52 + 1.28
(c. 1) t--rransform
(1.12 + 4.76) x 10
(0.26 + 5.97) x 10-4
0.04 + 0.26
ConvNet
MNIST
CIFARIO
CelebA
WC
20.73 + 18.59
27.28 + 0.63
18.72 + 1.33
GP
14.78 + 0.54
25.20 + 24.32
96.19 + 77.90
c-transform
0.79 = 0.16
1.00 + 0.26
2.11 + 0.46
(c, 0.1) trtansfffom
0.42 =
0.60 0.41
1.74 -
(c, 1) tttansfform
(0.23 + 1.90) 10 (0.40 = 3.60) x 10 0.02 + 0.17
Table 1: Approximation. For each method, the discriminators are trained 20 times for 500 iterations on minibatches of size 64 drawn without replacement, after which training is stopped and the error between the ground truth and the estimate are computed.
MNIST
(c. trraasssooor
c-transform
GP
WC
512,
17.20 + 0.16
13.87
4.25
+ 0.49
2.10
+ 0.26
N = 512, M = 512
16.95
12.64
4.21
2.03
N = 64, M = 64
17.45 + 0.06
14.12 0.13
1.54
+ 0.25
1.12 = 0.13
N = 64, M 512
16.76
11.4
1.49
1.08
Ground truth
14.22
12.65
12.65
12.65
CIFARIO
(e, trannsooor
c-transform
GP
WC
N 512, M 64
29.98 + 0.28
26.44 0.25
11.04 + 1.16
3.85 = 0.67
= 512, = 512
29.41
24.77
11.10
4.00
N = 64, .M = 64
29.67 1 0.41
26.21 + 0.40
3.25 1 0.53
2.19 1 0.23
N = 64, .M- 512
29.18
24.16
3.59
2.34
Ground truth
26.10
24.78
24.78
24.78
CelchA
(c, tratnsstoor
c-transform
GP
WC
N 512, M 64
50.55 1 0.86
46.56 1 0.89
28.07 1 10.61
19.18
1: 73.86
512,
512
48.42
43.06
28.17
20.93
= 64, M 64
50.80
+ 0.91
46.83
0.86
10.24 + 7.31
13.98 + 39.54
N = 64, .M 512
47.60
41.80
10.10
15.20
Ground truth
43.74
43.07
43.07
43.07
Table 2: Stability. The discriminators are trained using two training batch sizes, N = 64 and N 512. Then, the distances between the measures are estimated, by evaluating the discriminators on the full measures (of size M = 512). or by evaluating minibatch-win with batch size M = 64. Presented here are the distances approximated by each way of training the discriminators.
As Table shows, the c-transform approximates the minibatch- wise L-Wasserstein distance far better than weight clipping gradient penalty, and (c, -)trannsform does even better at approximating the 1-Sinkhorn divergence. The low errors in this case are due to the (c, e ))ttransform outperforming the POT library, which results in error of many iterations, which also explains why the error variance is so high in the e = case.
Stability, We train the discriminators for 500 iterations on small datasets of size 512, that form subsets of the datasets mentioned above. We train with two minibatch sizes N = 64 and N = 512. We then compare how the resulting discriminators estimate the minibatch-wise and total distances, that is, the evaluation minibatches are of size M = 64 and M 512. Letting V Method be the objective presented in for given method, we train the discriminator by maximizing V Methoodd. p. F/N). and finally compare V Method (Ø, HMM, PM) for different M. The results are presented in Table 2 An experiment on CIFARIO 10 was carried out to illustrate how the distance estimate between the full measures behaves when trained minibatch- wise, which is included in Appendix
The ground truth values computed using POT are also included, but are not of the main interest in this experiment. The focus is observing how different batch-sizes on training and evaluation affect the resulting distance. For the c-transform and (c, c) -taanssformm the results varies depending on whether the distances evaluated minibatch-win wise gradient penalty and weight clip methods, the training batch-size has more effect on the result, but the minibatch-we wise and full evaluations comparable.
the full datasets. On the other hand, for the
7

---

Under review as a conference paper at ICLR 2020
(c. tetr
c-transfomm
[c. transsorr
e-transform
4.0
3.5
3.0
2.5
2.0
0-
Approx.
True
True
1.5
20000
20000
Iteration
Iteration
Gradient Penalty
Weight Clip
Gredient Penalty
Weight Clip
10
12.5
True
8
7.5
toploh
5.0
Approx.
True
20000
40000
20000
40000
Iteration
Ieration
Figure 2: Approximating the minibatch-wise distances while training generator. Left: generated faces after x 10 generator iterations. Right: True and estimated log-distances between sampled batches. The large values of GP and WC can be attributed to the fact that they susceptible to failure in enforcing the Lipschitz constraints on the discriminator.
WGAN training. Finally, we measure how the different methods fare when training Wasserstein GAN. For this, we use the architecture from DCGAN (Radford et al. 2015) to learn a generative model on the CelebA dataset. During this training, we compute the POT ground truth distance between presented minibatches, and compare these to the estimated distances given by the discriminators. This is carried out for a total of 5 10a generator iterations, each of which is accompanied by 5 diseriminator iterations. For (c, e)- and c-transform the discriminators are evaluated on the fake samples, which gave similar results to evaluating on the real samples. The results are presented in Fig. 2 The results clearly show how the (c. e--trnnfform and c-transform estimate the Wasserstein distances each iteration better than gradient penalty or weight clipping. However, the resulting images are blurry and look like averages of clusters of faces. The best quality images are produced by the gradient penalty method, whereas weight clipping does not yet converge to anything meaningful. We include the same experiment ran with the simpler MLP architecture for the discriminator, while the generator still is based on DCAN, in Appendix
5 DISCUSSION
Based on the experiments, (c, -ttranssmoooom and c-transform are more accurate at computing the minibatch distance and estimating the batch distance than the gradient penalty and weight clipping methods. However, despite the lower performance of the latter methods, in the generative setting they produce more unique and compelling samples than the former. This raises the question, whether the exact 1- Wassersttii distance between batches is the quantity that should be considered in generative modelling, not. On the other hand, an interesting direction is to study regularization strategies in conjunction with the (c. c)- and c-transforms to improve generative modelling with less training.
The results of Table 2 indicate that the entropic relaxation provides stability under different training schemes, endorsing theoretical results implying more favorable sample complexity in the entropic case. In contrast what one could hypothetise, the blurriness in Fig. 2 seems not to be produced by the entropic relaxation, but the c-transform scheme.
Finally, it is interesting to see how the gradient penalty method performs so well in the generative setting, when based on our experiments, it is not able to approximate the 1- 1-Wasserstein distance so well. What then, that makes such good objective in the generative case?
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jonas Adler and Sebastian Lunz. Banach Wasserstein GAN. In Advances in Neural Information Processing Systems, pp. 6755-6764, 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein GAN. arXiv preprint
arXiv:1701.0723 07875 2017.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in neural information processing systems, pp. 2292-2300, 2013.
Ishan Deshpande, Ziyu Zhang, Alexander Schwing. Generative modeling using the sliced Wasserstein distance. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 3483-3491, 2018.
Simone Di Marino and Augusto Gerolin. An Optimal Transport approach for the Schrödinger bridge problem and convergence of Sinkhorn algorithm. in preparation, 2019.
Richard Mansfield Dudley. The speed of mean Glivenko-Cantelli convergence. The Annals of Mathematical Statistics, 40(1))40-50, 1969.
Yonatan Dukler, Wuchen Li, Alex Lin, and Guido Montúfar. Wasserstein of Wasserstein loss for learning generative models. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 Proceedings of Machine Learning Research, pp. 1716-1725, 2019.
Jean Feydy, Thibault Séjourné, François-Xavs Xavier Vialard, Shun-ichi Amari, Alain Trouvé, and Gabriel Peyré. Interpolating between optimal transport and MMD using Sinkhorn divergences. arXiv 2018.
preprint arXiv:1810.088
Wilfrid Gangbo. Wuchen Li, Stanley Osher, and Michael Puthawala. Unnormalized optimal transport. arXiv preprint arXiv: (tt::::922.3330 2019.
Aude Genevay, Marco Cuturi, Gabriel Peyré, and Francis Bach. Stochastic optimization for large- scale optimal transport. Advances in Neural Information Processing Systems, pp. 3440-3448, 2016.
Aude Genevay, Gabriel Peyre, and Marco Cuturi. Learning generative models with sinkhorn di- vergences. In Amos Storkey and Fernando Perez-Cruz (eds.), Proceedings of the Twenty-Fird International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pp. 1608-1617, 2018.
Aude Genevay, Lénaic Chizat, Francis Bach, Marco Cuturi, and Gabriel Peyré. Sample complexity of sinkhorn divergences. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of Machine Learning Research, volume 89 of Proceedings of Machine Learning Research, pp.
1574-1583, 2019.
Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- 2014.
fion processing systems, pp.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, pp. 5767-5777, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 100.v96710.10196, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 6980, 2014.
Thieny Klein, Jean-Claud Claude Fort, and Philippe Berthet. Convergence of an estimator the wasserstein distance between two continuous probability distributions. 2017.

---

Under review as a conference paper at ICLR 2020
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Rongjie Lai and Hongkai Zhao. Multi-scale non-rigid point cloud registration using robust sliced- Wasserstein distance via Laplace-Beltrami Beltrami eigenmap. arXiv preprint arXiv:1406.3373 2014.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-bass based learning applied to document recognition. Proceedings of the IEEE, 36(11):2278-2333 1998.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Anton Mallasto, Jes Frellsen, Wouter Boomsma, and Aasa Feragen. (q. aa-sassenntein GANs: Comparing ground metrics for Wasserstein GANs. arXiv preprint /902 036422 2019.
Gonzalo Mena and Jonathan Weed. Statistical bounds for entropic optimal transport: sample complexity and the central limit theorem. arXiv preprint arXiv: 1r0iv 1882, 2019.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport. Foundations and Trends@ in Machine Learning, 1 (55-6)3355-607, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.0644 2015.
Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance metric for image retrieval. International journal of computer vision, 40(2):99-121, 2000.
Martin Sommerfeld. Wasserstein distance on finite spaces: Statistical inference and algorithms. 2017.
Max Sommerfeld and Axel Munk. Inference for empirical wasserstein distances finite spaces. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 80(1):2 (0(1):219-236 2018.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-1,, 2012.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Jonathan Weed and Francis Bach. Sharp asymptotic and finite-sample rates of convergence of empirical measures Wasserstein distance. arXiv preprint arXiv: 2017.
Xiang Wei, Boqing Gong, Zixia Liu, Wei Lu, and Liqiang Wang. Improving the improved training of wasserstein gans: consistency term and its dual effect. In International Conference on Learning Representation (ICLR), 2018.
Jiging Wu, Zhiwu Huang, Janine Thoma, Dinesh Acharya, and Luc Van Gool. Wasserstein divergence for GANs. In Computer Vision ECCV 2018, pp. 673-688, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01128--1
A APPENDIX: STABILITY DURING TRAINING
Related to the Stability experiment, we train discriminators modelled by the simple MLPs (see Section 4 approximate the distance between two measures μl and v, which both are subsets the CIFARIO dataset of size 512. We train for 15000 iterations with training minibatch size of 64, and report the estimated minibatch-wise distances with the estimated distances between the full measures in Fig.
The experiments demonstrate, how (c, e) -tttansfoorm and c-transform converge rapidly compared to gradient penalty and weight clipping method, which have not entirely converged after 15000 iterations. Also visible is the bias resulting form minibatch wise computing the distances compared to the distance between the full measures.
10

---

Under review as a conference paper at ICLR 2020
(c. 1)transform
c-transform
30
32
28
26
22
2500
5000
7500
10000
12500
15000
2500
5000
7500 Iteration
10000
12500 15000
Iteration
Gradient Penalty
Weight Clip
Minibatch Estimate
25
Minibatch True
Full Estimate
20
40
Full True
10
2500
5000
7500
10000
12500
15000
2500
5000
7500
10000 12500 15000
Iteration
Iteration
Figure 3: Stability. While training the discriminators on minibatches, taken from two measures consisting of 512 samples from CIFAR1O, we also report the estimated distance between the full measures.
B APPENDIX: WGAN TRAINING WITH MLP
We repeat the WGAN training experiment with the simpler MLP architecture (see Section for the discriminators. The distance estimates at each iteration are given in Fig. and generated samples in Fig.
The training process for (c, e)- and c-transforms is more unstable with the MLP. as notable in the sudden jumps in the true distance between minibatches. This seems caused when the discriminator underestimates the distance. The fluctuation between estimated distances is much higher the gradient penalty and weight clipping cases, the decrease the true distance between minibatches is still consistent. Notice how the fluctuation decreases considerably when use ConvNet architecture for the gradient penalty method Fig. 2

---

Under review as a conference paper at ICLR 2020
(c, 1)-transform
c-transform
4.5
4.0
3.5
3
log(d)
log(d)
3.0
2
2.5
2.0
Approx.
Approx.
True
True
1.5
20000
40000
20000
40000
Iteration
Iteration
Gradient Penalty
Weight Clip
log(d)
log(d)
-2
-5
Approx.
Approx.
True
-4
True
20000
40000
20000
40000
Iteration
Iteration
Figure 4: Repeating the experiment presented in Fig. but with the simpler MLP architecture for the discriminator. Presented here are the estimated batchwise distances at each itcration against the ground truth.
12

---

Under review as a conference paper at ICLR 2020
(c, t)-rsffoo
c-transform
Gradient Penalty
Weight Clip
Figure 5: Repeating the experiment presented in Fig. but with the simpler MLP architecture for the discriminator. Presented here are generated samples after x 10 iterations.
13


Reviews:
[{"review_id": "SygEdKHnKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper has studied the efficiency and stability of computing the Wasserstein metric through its dual formulation under weight clipping, gradient penalty, c-transform and (c- ϵ)-transform. The results show that (c- ϵ)-transform and c-transform give more estimation of the Wasserstein distances than the gradient penalty and weight clipping methods in the given experiments, but the gradient penalty method produces more compelling samples in the generative setting.\n\nThe paper is well written and the experiment section is extensive. However, it is more like an extended experiment report to me which is very valuable but lacks sufficient technical novelty expected at ICLR.\n\nAnother comment is that when the authors mentioned \"... are compared to ground truth values d_ground computed by POT\", there needs more explanations on what that library actually does to compute the Wasserstein distance to make the paper self-contained, e.g. what exact algorithms it uses as there are also dozens of different algorithms implemented in that library.\n\nIn Section 3.1, the authors state that \"it tends to converge within couple of iterations of the symmetric Sinkhorn-Knopp algorithm. For efficiency, we approximate these terms with one Sinkhorn-Knopp iteration\". What is the extent of sacrifice in accuracy due to this approximation? The authors should provide more evidences to justify the approximation.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1llP9f6tB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n[Summary]\nThis paper provides an empirical evaluation of commonly used discriminator training strategies in estimating the Wasserstein distance between distributions. The paper finds that methods motivated from optimal transport theory, e.g. c-transform and (c,\\eps)-transform, perform better in evaluating the Wasserstein distance than methods commonly used in WGAN practice such as weight clipping and gradient penalty. However, when deployed in WGANs as the discriminator training strategy, these methods do not generate images as high-quality as the gradient penalty method. \n\n[Pros]\nThe question considered in this paper, i.e. how well does various discriminator strategies (as proxies of the infeasible all 1-Lipschitz discriminator) perform for *evaluating* the Wasserstein distributions, is important for strengthening our understanding of generative models. The main result that methods that are better at computing W may not be better at generating images is interesting, and agrees with the theoretical insight (e.g. Arora et al. 2017) that *non-parametric* minimization of W may not be a good explanation for the generative power of WGANs.\n\n[Cons]\nI have concerns about the specific setting of “mini-batch distance” considered in this paper, in that whether it is really a sensible task (and can say anything about computing W) given the curse of dimensionality of estimating W from samples. The paper did not really discuss this issue, and from my own thoughts I don’t think the task avoids this issue. \n\nFrom my understanding, the “Approximation” experiment does the following:\n(1) Set (\\mu, \\nu) to be a random split of a dataset and consider them to be the populations. Let’s let f_\\star denote the (ground truth) optimal discriminator between (\\mu, \\nu).\n(2) Train discriminators (f_wc, f_gp, f_c, f_ceps) (using the different algorithms) from training batches from (\\mu, \\nu).\n(3) Evaluate <f, \\mu’_l - \\nu’_l> where (\\mu’_l, \\nu’_l) are fresh test batches from (\\mu, \\nu) and f is one of the above trained discriminators.\nIn comparison, the “ground truth” computes W(\\mu’_l, \\nu’_l) = <f_l, \\mu’_l, \\nu’_l> from the POT package (though not necessarily through explicitly computing f_l.)\n\nThe issue with this is that we expect the method to perform well if f ~= f_l, which can be achievable if all the f_l’s are similar (and hopefully they’re all approximately equal to f_\\star.) However I don’t think this is true -- as (\\mu’_l, \\nu’_l) are samples, and because of the curse of dimensionality, we should expect the f_l’s to be quite different from each other. (Otherwise if they’re really just ~= f_\\star, then we can use standard concentration to show the W(\\mu’_l, \\nu’_l) ~= W(\\mu, \\nu), which we know is not true from curse of dim.)\n\nGiven this, I don’t think the task of comparing <f, \\mu’_l, \\nu’_l> with the POT results really says anything about their power in computing W. I would be glad though to hear back from the authors to see if my understanding is accurate, and adjust my evaluation from there.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gjYpiaFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper empirically evaluates different variants of WGAN (with weight clipping, gradient penalty, c-transform, and the generalized c-transform under entropy relaxation). The experiments, mainly performed over three datasets (MNIST, CIFAR10, CelebA), are designed to evaluate how well the Wasserstein distance is approximated, how much these approximations depend on batch sizes, and how good are the obtained generative models.\n\nI find the presentation of the different background work and models to be excellent, especially for someone who's not expert on WGANs like me. However, they may want to check the writing, like the sentence just after (17) or the penalization term between (18) and (19).\n\nThe contributions of the paper are experimental. The authors argue that they obtain a surprising observation, which is that \"the method best approximating the Wasserstein distance does not produce the best looking images in the generative setting \". \nHowever, the goodness of the approximation is measured with (24), which the authors called \"subjective error\". I think the authors may want to comment more on this measure, which seems to favor the different transforms.\nAlso, the quality of the generative models seems to strongly depend on the architectures used in WGAN. The authors' conclusions are based on DCGAN. However, the results obtained with simple MLP and presented in the appendix have not the same clear distinction as with DCGAN.\n\nOverall, although I liked the presentation very much, I feel the experimental results may be a bit too light for a publication in a venue such as ICLR.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is insufficient support to recommend accepting this paper.  Generally the reviewers found the technical contribution to be insufficient, and were not sufficiently convinced by the experimental evaluation.  The feedback provided should help the authors improve their paper.
Decision: Reject