This paper introduces an algorithm for learning Wasserstein GANs for discrete distributions.  Taking the dual form of the discrete Wasserstein distance introduced by [Evans 1997] (which produces a constrained optimization problem) and using this as a basis of a GAN training algorithm.  The key algorithmic distinction from conventional GAN approaches is that the critic takes pairs of real and simulated datapoints as inputs and returns a measure of which it thinks is the real datapoint, namely f(x_r)–f(g(z)) where x_r is the data point and g(z) the generator output, rather than the critic corresponding to f directly.   An architecture is proposed for this framework that guarantees that the constraints from the formulation of [Evans 1997] as satisfied. The topic of this paper is timely and of clear interest to the ICLR community.  The underlying idea is interesting  and the architecture seems appropriate for the chosen target.  Further, the paper is relatively clear and easy to follow.  However, the experimental evaluation is not sufficiently strenuous and produces very underwhelming results.  Relatedly, I think the motivation behind using the discrete Wasserstein distance, though seemingly reasonable, needs more careful consideration.  Unfortunately, the serious shortfalls in the experiments mean that the paper, in my opinion, falls noticeably below the acceptance threshold for ICLR.   Having said that, my stance might change substantially if more impressive experimental results can be obtained;  without this, I am unconvinced the method actually behaves as intended.  Regretfully, this is probably beyond the scope of a revision during the rebuttal period  though as it will most likely require significant algorithmic changes. %%%% Shortcomings with Experiments %%%%% Put simply, I do not think that the experiments demonstrate that the approach works and actually suggest the opposite.  The so-called “mode collapse” issue is effectively a sugar-coated way of saying that the method has learned to return a single output rather than learning a generative model.  This is a huge issue and needs sorting before the paper can be seriously considered for publication.  The attempted at a fix at the end of the paper might be a step in the right direction but is not evaluated sufficiently  and the preliminary results provided are not particularly promising. Going past this issue, there are still a lot of problems with the experimental evaluation.  More numerous and more difficult problems need to be considered.  For example, an NLP problem would fit well with the motivation for the approach given in the intro.  It is also necessary to demonstrate that the GAN is doing something different to just memorizing previous examples.  For example, this is a problem where one can actually reasonably compare to just sampling from the data by checking that more unseen correct samples are generated then incorrect samples.  It is worrying that the convergence plots show a single line that ostensibly comes from the “best result”.  This is not a reasonable scientific comparison method  and should be replaced by mean (or median) performance with uncertainty estimates (i.e. +/- some numbers of standard deviations or quantiles).  The small number of samples from the MNIST problem really doesn’t convey anything meaningful  – even if this looked exceptional, a small number of unqualified samples is hardly a serious evaluation metric. More iterations for Figure 4a or needed to see if the WGAN keeps improving beyond on the DWGAN, noting that this has not converged at 100%.  All four methods should be added to Figure 5. %%%% Is the Discrete Metric Always Better for Training the Generative Model? %%%%% Though I think it is probably true that the discrete metric should more beneficial from the point of the view of the critic,  I am not completely convinced it is always beneficial from the point of view of training the generator, which is at the end of the day what really matters.  Note that I am not trying to argue that discrete metric is worse,  just that you haven’t done enough to convince me that it is always better.  If indeed it is always better, please tear apart my following argument,  which will hopefully provide stimulation for improving the motivation of the metric in the paper.  If it is not always better, the paper should be updated to outline some of the potential pathologies and the cases were you expect the approach to work well and when you do not. To demonstrate my argument, consider the training sample [0, 0, 1; 0, 1, 0] and the following four example generated outputs (1) [0.333, 0.333, 0.334; 0.333, 0.334, 0.333] (2) [0, 0, 1; 0, 1, 0] (3) [0, 0, 1; 0.334, 0.333, 0.333] (4) [0.333, 0.334, 0.333; 0.334, 0.333, 0.333] giving respective discrete distances to of 0, 0, 1, and 2; and continuous distances of 1.330668, 0, 0.667334, and 1.334668.  Now imagine we are training a GAN with the one example datapoint [0, 0, 1; 0, 1, 0].  Even though (1) will lead to the target sample [0, 0, 1; 0, 1, 0] after passing through the argmax function and (3) will not,  (1) is also very close to (4) which has the maximum possible discrete distance.  Consequently, the generator for (1) is most likely not at a stable optimum and very close in the space of neural net weights to some very poor generators.  During training, we would thus like to guide our network towards generating (2),  as this is a stable solution, particularly given we are using stochastic optimization methods.  From this perspective, then (3) is perhaps a better generated output than (1), a fact conveyed by the continuous metric but not the discrete metric.  In other words, even though (1) is arguably a better final solution than (3), from the perspective of effective training, it may be favorable to use a target function that prefers (3) to (1) to better guide the training to a stable solution. Once we consider the fact that our aim is not to replicate a single datapoint but learn a generator that in some way interpolates between datapoints,  it becomes even less clear if the discrete metric is better.  For example, small changes to the input z for (1) are likely to lead to generating samples that are very different to anything seen in the training data (which is, in this case, a single point). Though I do not think this argument undermines the suggested approach,  I do think it highlights why the supplied motivation for the approach is insufficient and needs more explicitly linking back to the training procedure.  At the very least, I think the above argument shows why it is not immediately clear cut that the suggested approach will perform better  and so needs backing up with strong empirical evidence,  which unfortunately the paper does not currently have, and/or a more convincing argument for why the discrete metric is better. %%%% Other points %%%% - Tables 1, 2, and 3 are not worth spending more than a few lines on, let alone nearly a page.  They should be substantially compressed or preferably just cut (particularly Tables 2 and 3).  The point they convey is obvious and actually somewhat tangential to the key questions. - It worries me that the method learns an h that explicitly takes y as input, not x.  This is a notational issue in the exposition, but also, more importantly, raises questions about whether the original linear programming problem is actually being solved  because the inputs from the generative method are not discrete variables. - It should be made clear that the Appendices are directly following the derivation of Evans 1997, rather than being a new derivation. - At times the paper is little sloppy at making distinguishing between y ∈ [0, 1] and y ∈ {0, 1}.  This makes it a bit confusing at times what is output from the generator, particularly when you talk about it being one-hot encoded.  As I understand it, the output is always the former,  while data lives in the latter  but this is not always clear.  I think you should also make a bigger deal of this in terms of the problem with previous methods that ignore that the critic might be able to distinguish based on the fact that the training and generated data points have an explicitly different type (discrete and continuous respectively). - On the fourth line of the abstract, both instances of GAN should be GANs. - What do you mean “explained in the sequel”? - Does k need to be the same for each variable? %%%% References %%%% Lawrence C Evans. Partial differential equations and Monge-Kantorovich mass transfer. Current developments in mathematics, 1997(1):65–126, 1997.