Title: GENERATIVE MODEL BASED ON MINIMIZING EXACT EMPIRICAL WASSERSTEIN DISTANCE

Abstract: Generative Adversarial Networks (GANs) are a very powerful framework for generative modeling. However, they are often hard to train, and learning of GANs often becomes unstable. Wasserstein GAN (WGAN) is a promising framework to deal with the instability problem as it has a good convergence property. One drawback of the WGAN is that it evaluates the Wasserstein distance in the dual domain, which requires some approximation, so that it may fail to optimize the true Wasserstein distance. In this paper, we propose evaluating the exact empirical optimal transport cost efficiently in the primal domain and performing gradient descent with respect to its derivative to train the generator network. Experiments on the MNIST dataset show that our method is significantly stable to converge, and achieves the lowest Wasserstein distance among the WGAN variants at the cost of some sharpness of generated images. Experiments on the 8-Gaussian toy dataset show that better gradients for the generator are obtained in our method. In addition, the proposed method enables more flexible generative modeling than WGAN.

1. INTRODUCTION:

Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are a powerful framework of generative modeling which is formulated as a minimax game between two networks: A generator network generates fake-data from some noise source and a discriminator network discriminates between fake-data and real-data. GANs can generate much more realistic images than other generative models like variational autoencoder (Kingma & Welling, 2014) or autoregressive models (van den Oord et al., 2016) , and have been widely used in high-resolution image generation (Karras et al., 2018) , image inpainting (Yu et al., 2018) , image-to-image translation (Isola et al., 2017) , to mention a few. However, GANs are often hard to train, and various ways to stabilize training have been proposed by many recent works. Nonetheless, consistently stable training of GANs remains an open problem.

GANs employ the Jensen-Shannon (JS) divergence to measure the distance between the distributions of real-data and fake-data (Goodfellow et al., 2014) . Arjovsky et al. (2017) provided an analysis of various distances and divergence measures between two probability distributions in view of their use as loss functions of GANs, and proposed Wasserstein GAN (WGAN) which has better theoretical properties than the original GANs. WGAN requires that the discriminator (called the critic in Arjovsky et al. (2017) ) must lie within the space of 1-Lipschitz functions to evaluate the Wasserstein distance via the Kantorovich-Rubinstein dual formulation. Arjovsky et al. (2017) further proposed implementing the critic with a deep neural network and applied weight clipping in order to ensure that the critic satisfies the Lipschitz condition. However, weight clipping limits the critic's function space and can cause gradients in the critic to explode or vanish if the clipping parameters are not carefully chosen (Arjovsky et al., 2017; Gulrajani et al., 2017) . WGAN-GP (Gulrajani et al., 2017) and Spectral Normalization (SN) (Miyato et al., 2018) apply regularization and normalization, respectively, on the critic trying to make the critic 1-Lipschitz, but they fail to optimize the true Wasserstein distance.

In the latest work, Liu et al. (2018) proposed a new WGAN variant to evaluate the exact empirical Wasserstein distance. They evaluate the empirical Wasserstein distance between the empirical distributions of real-data and fake-data in the discrete case of the Kantorovich-Rubinstein dual for-mulation, which can be solved efficiently because the dual problem becomes a finite-dimensional linear-programming problem. The generator network is trained using the critic network learnt to approximate the solution of the dual problem. However, the problem of approximation error by the critic network remains. In this paper, we propose a new generative model without the critic, which learns by directly evaluating gradient of the exact empirical optimal transport cost in the primal domain. The proposed method corresponds to stochastic gradient descent of the optimal transport cost. Arjovsky et al. (2017) argued that JS divergences are potentially not continuous with respect to the generator's parameters, leading to GANs training difficulty. They proposed instead using the Wasserstein-1 distance W 1 (q, p), which is defined as the minimum cost of transporting mass in order to transform the distribution q into the distribution p. Under mild assumptions, W 1 (q, p) is continuous everywhere and differentiable almost everywhere.

2. WASSERSTEIN GAN:

The WGAN objective function is constructed using the Kantorovich-Rubinstein duality (Villani, 2009, Chapter 5) as

EQUATION

to obtain min

EQUATION

where D is the set of all 1-Lipschitz functions, where P r is the real-data distribution, and where P g is the generator distribution implicitly defined by y = G(z), z ∼ p(z). Minimization of this objective function with respect to G with optimal D is equivalent to minimizing W 1 (P r , P g ). Arjovsky et al. (2017) further proposed implementing the critic D in terms of a deep neural network with weight clipping. Weight clipping keeps the weight parameter of the network lying in a compact space, thereby ensuring the desired Lipschitz condition. For a fixed network architecture, however, weight clipping may significantly limit the function space to a quite small fraction of all possible 1-Lipschitz functions representable by networks with the prescribed architecture.

3 RELATED WORKS Gulrajani et al. (2017) proposed introduction of gradient penalty (GP) to the WGAN objective function in place of the 1-Lipschitz condition in the Kantorovich-Rubinstein dual formulation, in order to explicitly encourage the critic to have gradients with magnitude equal to 1. Since enforcing the constraint of unit-norm gradient everywhere is intractable, they proposed enforcing the constraint only along straight line segments, each connecting a real-data point and a fake-data point. The resulting learning scheme, which is called the WGAN-GP, was shown to perform well experimentally. It was pointed out, however (Miyato et al., 2018) , that WGAN-GP is susceptible to destabilization due to gradual changes of the support of the generator distribution as learning progresses. Furthermore, the critic can easily violate the Lipschitz condition in practice, so that there is no guarantee that WGAN-GP optimizes the true Wasserstein distance.

SN, proposed by Miyato et al. (2018) , is based on the observation that the Lipschitz norm of a critic represented by a multilayer neural network is bounded from above by the product, across all layers, of the Lipschitz norms of the activation functions and the spectral norms of the weight matrices, and normalizes each of the weight matrices with its spectral norm to ensure the resulting critic to satisfy the desired Lipschitz condition. It is well known that, for any m × n matrix W = (w ij ), the max norm W max = max{|w ij |} and the spectral norm σ(W ) satisfy the inequality W max ≤ σ(W ) ≤ √ mn W max . This implies that the bound of the Lipschitz constant provided via weight clipping can be loose compared with that via SN. In other words, SN is expected to provide a much tighter bound for the Lipschitz condition than weight clipping, and accordingly, the function space for the critic under SN is larger than that under weight clipping. The function space under SN is, however, still a subset of the set of all functions satisfying the Lipschitz condition, and consequently, the resulting estimate for the Wasserstein distance is a lower bound of the true Wasserstein distance. Furthermore, one cannot tell within the framework of SN how good the estimate is. Liu et al. (2018) proposed a new formulation to evaluate the Wasserstein distance, which is equivalent to the discrete case of the Kantorovich-Rubinstein dual formulation under a mild assumption and is more tractable due to obviating the need for the Lipschitz condition. This problem is solved in a two-step fashion, and thus the method proposed in Liu et al. (2018) is called the WGAN-TS. First, one estimates the Wasserstein distance on the basis of finite real-and fake-data points. The empirical Wasserstein distance is evaluated exactly via solving the linear-programming version of the Kantorovich-Rubinstein dual to obtain the optimizer. Second, one approximates the optimizer obtained in the first step via regression using a deep neural network to parameterize the critic and obtains its gradient. WGAN-TS can evaluate the Wasserstein distance more accurately than WGAN, WGAN-GP and WGAN-SN (WGAN with SN), but there are not only approximation errors from using finite samples to evaluate the empirical Wasserstein distance but also those from deep regression in the second step, resulting in not being able to minimize the Wasserstein distance directly.

4. PROPOSED METHOD:

The proposed method in this paper is based on the fact that the optimal transport cost between two probability distributions can be evaluated efficiently when the distributions are uniform over finite sets of the same cardinality. Our proposal is to evaluate empirical optimal transport costs on the basis of equal-size sample datasets of real-and fake-data points.

The optimal transport cost between the real-data distribution P r and the generator distribution P g is defined as

EQUATION

where c(x, y) is the cost of transporting one unit mass from x to y, assumed differentiable with respect to its arguments almost everywhere, and where Π(P r , P g ) denotes the set of all couplings between P r and P g , that is, all joint probability distributions that have marginals P r and P g .

Let D = {x j |x j ∼ P r (x)} be a dataset consisting of independent and identically-distributed (iid) real-data points, and F = {y i |y i ∼ P g (y)} be a dataset consisting of iid fake-data points sampled from the generator. Let P D and P F be the empirical distributions defined by the datasets D and F , respectively. We further assume in the following that |D| = |F | = N holds. The empirical optimal transport cost Ĉ(D, F ) = C(P D , P F ) between the two datasets D and F is formulated as a linear-programming problem, as

EQUATION

EQUATION

EQUATION

It is known (Villani, 2003) that the linear-programming problem ( 4)-( 7) admits solutions which are permutation matrices. One can then replace the constraints M i,j ≥ 0 in (7) with M i,j ∈ {0, 1} without affecting the optimality. The resulting optimization problem is what is called the linear sum assignment problem, which can be solved more efficiently than the original linear-programming problem. As far as the authors' knowledge, the most efficient algorithm to date for solving a linear sum assignment problem has time complexity of O(N 2.5 log(N C)), where C = max i,j c(x j , y i ) when one scales up the costs {c(x j , y i )|x j ∈ D, y i ∈ F } to integers (Burkard et al., 2012, Chapter 4) .

This is a problem to find the optimal transport plan, where M i,j = 1 is corresponding to transporting fake-data point y i ∈ F to real-data point x j ∈ D, and where the objective is to minimize the average transport cost

N -1 N i=1 N j=1 M i,j c(x j , y i ).

Figure 1 shows a two-dimensional example of this problem and its solution. One requires evaluations not only of the optimal transport cost C(P r , P g ) but also of its derivative in order to perform learning of the generator with backpropagation. Let θ denote the parameter of the generator, and let ∂ θ C denote the derivative of the optimal transport cost C with respect to θ. Conditional on z, the generator output G(z) is a function of θ. Hence, in order to estimate ∂ θ C, in our framework one has to evaluate ∂ θ Ĉ. In general, it is difficult to differentiate (4) with respect to generator output y i , as the optimal transport plan M * can be highly dependent on y i . Under the assumption |D| = |F | = N which we adopt here, however, the feasible set for M is the set of all permutation matrices and is a finite set. It then follows that, as a generic property, the optimal transport plan M * is unchanged under small enough perturbations of F (see Figure 1 ). We take advantage of this fact and regard M * as independent of y i . Now that differentiation of (4) becomes tractable, we use (4) as the loss function of the generator and update the generator with the direct gradient of the empirical optimal transport cost, as

∂ θ Ĉ = N -1 N i,j=1 M * i,j ∂ yi c(x j , y i )∂ θ G(z i ).

Although the framework described so far is applicable to any optimal transport cost, several desirable properties can be stated if one specializes in the Wasserstein distance. Assume, for a given p ≥ 1, that the real-data distribution P r and the generator distribution P g have finite moments of order p. The Wasserstein-p distance between P r and P g is defined in terms of the optimal transport cost with c(x, y) = x -y p as W p (P r , P g ) = C(P r , P g ) 1/p .

Due to the law of large numbers, the empirical distributions P D and P F converge weakly to P r and P g , respectively, as N → ∞. It is also known (Villani, 2009, Theorem 6.9 ) that the Wasserstein-p distance W p metrizes the space of probability measures with finite moments of order p. Consequently, the empirical Wasserstein distance Ŵp (D, F ) is a consistent estimator of the true Wasserstein distance W p (P r , P g ). Furthermore, with the upper bound of the error of the estimator

EQUATION

which is derived on the basis of the triangle inequality, as well as with the upper bounds available for expectations of W p (P D , P r ) and W p (P F , P g ) under mild conditions (Weed & Bach, 2017) , one can see that Ŵp (D, F ) is an asymptotically unbiased estimator of W p (P r , P g ).

Note that our method can directly evaluate the empirical Wasserstein distance without recourse to the Kantorovich-Rubinstein dual. Hence, our method does not use a critic and is therefore no longer a GAN. It is also applicable to any optimal transport cost. We summarize the proposed method in Algorithm 1.

5. EXPERIMENTS:



5.1. RESULTS ON MNIST WITH CONVOLUTIONAL NEURAL NETWORK:

We first show experimental results on the MNIST dataset of handwritten digits. In this experiment, we resized the images to resolution 64 × 64 so that we can use the convolutional neural networks Algorithm 1 The proposed method.

1: Input: Real-data samples X real , batch size N , Adam parameters α, β 1 , β 2 2: Output: G θ 3: Initialize θ. 4: while θ has not converged do 5:

Sample {x i } i∈{1,...,N } ∼ X real from real-data.

6::

Sample {z j } j∈{1,...,N } ∼ p(z) from random noises.

7::

Let y j = G θ (z j ), ∀j ∈ {1, . . . , N }.

8::

Solve (4)-7 to obtain M * . 9: described in Appendix A.1 as the critic and the generator. In all methods, the batch size was set to 64 and the prior noise distribution was the 100-dimensional standard normal distribution. The maximum number of iterations in training of the generator was set to 30,000. The Wasserstein-1 distance with c(x, y) = x -y 1 was used. More detailed settings are described in Appendix B.1.

g θ ← ∂ θ Ĉ = N -1 N i,j=1 M * i,j ∂ yi c(x j , y i )∂ θ G θ (z i ) 10: θ ← Adam(g θ , θ, α,

Although several performance metrics have been proposed and are commonly used to evaluate variants of WGAN, we have decided to use the empirical Wasserstein distance (EWD) to compare performance of all methods. It is because all the methods adopt objective functions that are based on the Wasserstein distance, and because EWD is a consistent and asymptotically unbiased estimator of the Wasserstein distance and can efficiently be evaluated, as discussed in Section 4. Table 1 shows EWD evaluated with 256 samples and computation time per generator update for each method. For reference, performance comparison with the Fréchet Inception Distance (Heusel et al., 2017) and the Inception Score (Salimans et al., 2016) , which are commonly used as performance measures to evaluate GANs using feature space embedding with an inception model, is shown in Appendix C. The proposed method achieved a remarkably small EWD and computational cost compared with the variants of WGAN. Our method required the lowest computational cost in this experimental setting mainly because it does not use the critic. Although we think that the batch size used in the experiment of the proposed method was appropriate since the proposed method achieved lower EWD, if a larger batch size would be required in training, it will take much longer time to solve the linear sum assignment problem (4)-( 7).

We further investigated behaviors of the methods compared in more detail, on the basis of EWD. WGAN-SN failed to learn. The loss function of the critic showed divergent movement toward -∞, and the behaviors of EWD in different trials were different even though the behaviors of the critic loss were the same (Figure 2 (a) and (b)). WGAN training never failed in 5 trials, and EWD improved stably without sudden deterioration. Although training with WGAN-GP proceeded favorably in initial stages, at certain points the gradient penalty term started to increase, causing EWD to deteriorate (Figure 2 (c) ). This happened in all 5 trials. Since gradient penalty is a weaker restriction than weight clipping, the critic may be more likely to cause extreme behaviors. We examined both WGAN-TS with and without weight scaling. Whereas WGAN-TS with weight scaling did not fail Liu et al. (2018, Section 4.1) . If β ≤ 0, the critic satisfies the 1-Lipschitz constraint. (c) Regression error of the critic.

D(yi)-D(G(zj )) ||yi-G(zj )|| , ||y i -G(z j )|| > 0}} in

in training but achieved higher EWD than WGAN, WGAN-TS without weight scaling achieved lower EWD than WGAN at the cost of the stability of training (Figure 3 ). The proposed method was trained stably and never failed in 5 trials.

As mentioned in Section 3, the critic in WGAN-TS simply regresses the optimizer of the empirical version of the Kantorovich-Rubinstein dual. Thus, there is no guarantee that the critic will satisfy the 1-Lipschitz condition. Liu et al. (2018) pointed out that it is indeed practically problematic with WGAN-TS, and proposed weight scaling to ensure that the critic satisfies the desired condition. We have empirically found, however, that weight scaling exhibited the following trade-off (Figure 3 ). Without weight scaling, training of WGAN-TS suddenly deteriorated in some trials because the critic came to not satisfy the Lipschitz condition. With weight scaling, on the other hand, the regression error of the critic with respect to the solution increased and the EWD became worse. The proposed method directly solves the empirical version of the optimal transport problem in the primal domain, so that it is free from such trade-off.

Figure 4 shows fake-data images generated by the generators trained with WGAN, WGAN-GP, WGAN-TS, and the proposed method. Although one can identify the digits for the generated images with the proposed method most easily, these images are less sharp. Among the generated images with the other methods, one can notice several images which have almost the same appearance as real-data images, whereas in the proposed method, such fake-data images are not seen and images that seem averaged real-data images belonging to the same class often appear. This might imply that merely minimizing the Wasserstein distance between the real-data distribution and the generator distribution in the raw-image space may not necessarily produce realistic images.

5.2. GRADIENT OPTIMALITY OF GENERATOR:

We next observed how the generator distribution is updated in order to compare the proposed method with variants of WGAN in terms of the gradients provided. Figure 5 shows typical behavior of the generator distribution trained with the proposed method on the 8-Gaussian toy dataset. The 8-Gaussian toy dataset and experimental settings are described in Appendix B.2. One can observe that, as training progresses, the generator distribution comes closer to the real-data distribution. Figure 6 shows comparison of the behaviors of the proposed method, WGAN-GP, and WGAN-TS. We excluded WGAN and WGAN-SN from this comparison: WGAN tended to yield generator distributions that concentrated around a single Gaussian component, and hence training did not progress well. WGAN-SN could not correctly evaluate the Wasserstein distance as in the experiment on the MNIST dataset.

One can observe in Figure 6 that directions of sample updates are diverse in the proposed method, especially in later stages of training, and that the sample update directions tend to be aligned with the optimal gradient directions. These behaviors will be helpful for the generator to learn the realdata distribution efficiently. In WGAN-GP and WGAN-TS, on the other hand, the sample update directions exhibit less diversity and less alignment with the optimal gradient directions, which would make the generator distribution difficult to spread and would slow training. One would be able to ascribe such behaviors to poor quality of the critic: Those behaviors would arise when the generator learns on the basis of unreliable gradient information provided by the critic without learning sufficiently to accurately evaluate the Wasserstein distance. If one would increase the number n c of critic iterations per generator iteration in order to train the critic better, the total computational cost of training would increase. In fact, n c = 5 is recommended in practice and has commonly been used in WGAN (Arjovsky et al., 2017) and its variants because the improvement in learning of the critic is thought to be small relative to increase in computational cost. In reality, however, 5 iterations would not be sufficient for the critic to learn, and this might be a principal reason for the critic to provide poor gradient information to the generator in the variants of WGAN.

6. CONCLUSION:

We have proposed a new generative model that learns by directly minimizing exact empirical Wasserstein distance between the real-data distribution and the generator distribution. Since the proposed method does not suffer from the constraints on the transport cost and the 1-Lipschitz constraint imposed on WGAN by solving the optimal transport problem in the primal domain instead of the dual domain, one can construct more flexible generative modeling. The proposed method provides the generator with better gradient information to minimize the Wasserstein distance (Section 5.2) and achieved smaller empirical Wasserstein distance with lower computational cost (Section 5.1) than any other compared variants of WGAN. In the future work, we would like to investigate the behavior of the proposed method when transport cost is defined in the feature space embedded by an appropriate inception model. 

C EVALUATION WITH FID AND IS ON MNIST DATASET:

We show the result of evaluation of the experimented methods with FID and IS in Table 4 . Both FID and IS are commonly used to evaluate GANs.

FID calculates the distance between the set of real-data points and that of fake-data points. The smaller the distance is, the better the fake-data points are judged. Assuming that the vector obtained from a fake-or real-data point through the inception model follows a multivariate Gaussian distribution, FID is defined by the following equation:

EQUATION

)

where (µ i , Σ i ) is the mean vector and the covariance matrix for dataset i, evaluated in the feature space embedded with inception scores. It is nothing but the square of the Wasserstein-2 distance between two multivariate Gaussian distributions with parameters (µ 1 , Σ 1 ) and (µ 2 , Σ 2 ), respectively.

IS is a metric to evaluate only the set of fake-data points. Let x i be a data point, y be the label of x i in the data identification task for which the inception model was trained, p(y|x i ) be the probability of label y obtained by inputting x i to the inception model. Letting X be the set of all data points used for calculating the score, the marginal probability of label y is p(y) = 1 |X| xi∈X p(y|x i ). IS is defined by the following equation:

EQUATION

)

where KL is Kullback-Leibler divergence. IS is designed to be high as the data points are easy to identify by the inception model and variation of labels identified from the data points is abundant.

In WGAN-GP, WGAN-SN and WGAN-TS*, we observed that training suddenly deteriorated in some trials. We thus used early stopping on the basis of EWD, and the results of these methods shown in Table 4 are with early stopping.

The proposed method marked the worst in FID and the best in IS among all the methods compared.

Certainly, the fake-data generated by the proposed method are non-sharp and do not resemble realdata points, but it seems that it is easy to distinguish them and they have diversity as digit images. If one wishes to produce higher FID results using the proposed method, transport cost should be considered in the desired space corresponding to FID.

:



Figures and Tables:

Figure 1: Two-dimensional example of optimal transport problem (4)-(7) with N = 8. Circles • represent real-data points in D and triangles represent fake-data points in F . Arrows between circles • and filled triangles show the optimal transport plan M * with c(x, y) = x -y 2 , which is an identity matrix. Arrows between open △ and filled triangles show small perturbations of F , which do not change M * .

Figure 2: Training behaviors of the methods experimented. (a) Critic loss versus the number of generator iterations, and (b) EWD versus the number of generator iterations in four trials of WGAN-SN. (c) EWD versus the number of generator iterations, averaged over 5 trials, in WGAN, WGAN-GP, WGAN-TS, and the proposed method.

Figure 4: Examples of fake-data images generated by the generators trained on the MNIST dataset with the four methods compared.

Figure 6: Behaviors of updating in training on the 8-Gaussian toy dataset with (a) the proposed method, (b) WGAN-GP, and (c) WGAN-TS. Eight large circles represent positions of Gaussian components of real-data distribution. Filled circles • are real-data points, filled triangles are preupdated fake-data points, and open triangles △ are post-updated fake-data points. Solid and dotted arrows indicate optimal gradient directions of fake-data points and directions of their actual updates, respectively. Left, center, and right panels are at iterations 10, 20, and 30, respectively.

β 1 , β 2 ) 11: end while Comparison on MNIST. Empirical Wasserstein distance (EWD), and computation time per generator update for each method. Each metric represents the average and standard deviation over 3 out of 5 trials, excluding the maximum and minimum. Lower is better for EWD. WGAN-TS* represents WGAN-TS without weight scaling. In WGAN-GP, WGAN-SN and WGAN-TS*, training can suddenly deteriorate. Thus, we used early stopping based on EWD.

Table content: <table><tr><td>Method</td><td>EWD</td><td>Time [ms/iter]</td></tr><tr><td colspan="3">WGAN WGAN-GP WGAN-SN WGAN-TS* 714.5 ± 17.7 744.8 ± 5.2 811.9 ± 31.7 888.5 ± 30.1 WGAN-TS 761.2 ± 4.5 Proposed 600.5 ± 10.8 148.4 ± 3.58 1547 ± 18.2 2867 ± 43.6 1839 ± 44.9 886.0 ± 10.0 896.8 ± 9.44</td></tr></table>

Comparison on the MNIST dataset. Fréchet Inception Distance (FID), Inception Score (IS) Each metric represents the average and standard deviation over 3 out of 5 trials, excluding the maximum and minimum. Lower is better for FID. Higher is better for IS. FID and IS were calculated with 50, 000 samples. WGAN-TS* represents WGAN-TS without weight scaling.All the numerical in paper were executed on a computer with an Intel Core i7-6850K CPU (3.60 GHz, 6 cores) and 32 GB RAM, and with four GeForce GTX 1080 graphics cards installed. Linear sum assignment problems were solved using the Hungarian algorithm, which has time complexity of O(N 3 ). Codes used in the experiments were written in tensorflow 1.10.1 on python 3.6.0, with eager execution enabled.

Table content: <table><tr><td>Method</td><td>FID</td><td>IS</td></tr><tr><td colspan="3">WGAN WGAN-GP WGAN-SN WGAN-TS* 0.74 ± 0.23 0.41 ± 0.02 2.15 ± 0.02 3.97 ± 2.49 2.15 ± 0.09 4.01 ± 1.36 2.16 ± 0.19 2.07 ± 0.05 WGAN-TS 0.68 ± 0.06 2.16 ± 0.01 Proposed 5.57 ± 0.14 2.33 ± 0.02</td></tr><tr><td>B.3 EXECUTION ENVIRONMENT</td><td/><td/></tr></table>

Peer review (iclr2019):
 The paper proposed to use the exact empirical Wasserstein distance to supervise the training of generative model. To this end, the authors formulated the optimal transport cost as a linear programming problem. The quantitative results-- empirical Wasserstein distance show the superiority of the proposed methods. My concerns come from both theoretical and experimental aspects: The linear-programming problem Eq.(4)-Eq.(7) has been studied in existing literature. The contribution is about combining this existing method to supervise a standard neural network parametrized generator, so I am not quite sure if this contribution is sufficient for the ICLR submission. In such a case, further experimental or theoretical study about the convergence of Algorithm 1 seems important to me. As to the experiments, firstly, EWD seems to be a little bit biased since EWD is literally used to supervise the training of the proposed method. Other quantitative metric studies can help justifying the improvement. Also, given that the paper brings the WGAN family into comparison, the large scale image dataset should be included since WGAN have already demonstrated their success. Last things, missing parentheses in step 8 of Algorithm 1 and overlength of url in references.