Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper empirically evaluates different variants of WGAN (with weight clipping, gradient penalty, c-transform, and the generalized c-transform under entropy relaxation). The experiments, mainly performed over three datasets (MNIST, CIFAR10, CelebA), are designed to evaluate how well the Wasserstein distance is approximated, how much these approximations depend on batch sizes, and how good are the obtained generative models.

I find the presentation of the different background work and models to be excellent, especially for someone who's not expert on WGANs like me. However, they may want to check the writing, like the sentence just after (17) or the penalization term between (18) and (19).

The contributions of the paper are experimental. The authors argue that they obtain a surprising observation, which is that "the method best approximating the Wasserstein distance does not produce the best looking images in the generative setting ". 
However, the goodness of the approximation is measured with (24), which the authors called "subjective error". I think the authors may want to comment more on this measure, which seems to favor the different transforms.
Also, the quality of the generative models seems to strongly depend on the architectures used in WGAN. The authors' conclusions are based on DCGAN. However, the results obtained with simple MLP and presented in the appendix have not the same clear distinction as with DCGAN.

Overall, although I liked the presentation very much, I feel the experimental results may be a bit too light for a publication in a venue such as ICLR.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: 
[Summary]
This paper provides an empirical evaluation of commonly used discriminator training strategies in estimating the Wasserstein distance between distributions. The paper finds that methods motivated from optimal transport theory, e.g. c-transform and (c,\eps)-transform, perform better in evaluating the Wasserstein distance than methods commonly used in WGAN practice such as weight clipping and gradient penalty. However, when deployed in WGANs as the discriminator training strategy, these methods do not generate images as high-quality as the gradient penalty method. 

[Pros]
The question considered in this paper, i.e. how well does various discriminator strategies (as proxies of the infeasible all 1-Lipschitz discriminator) perform for *evaluating* the Wasserstein distributions, is important for strengthening our understanding of generative models. The main result that methods that are better at computing W may not be better at generating images is interesting, and agrees with the theoretical insight (e.g. Arora et al. 2017) that *non-parametric* minimization of W may not be a good explanation for the generative power of WGANs.

[Cons]
I have concerns about the specific setting of “mini-batch distance” considered in this paper, in that whether it is really a sensible task (and can say anything about computing W) given the curse of dimensionality of estimating W from samples. The paper did not really discuss this issue, and from my own thoughts I don’t think the task avoids this issue. 

From my understanding, the “Approximation” experiment does the following:
(1) Set (\mu, \nu) to be a random split of a dataset and consider them to be the populations. Let’s let f_\star denote the (ground truth) optimal discriminator between (\mu, \nu).
(2) Train discriminators (f_wc, f_gp, f_c, f_ceps) (using the different algorithms) from training batches from (\mu, \nu).
(3) Evaluate <f, \mu’_l - \nu’_l> where (\mu’_l, \nu’_l) are fresh test batches from (\mu, \nu) and f is one of the above trained discriminators.
In comparison, the “ground truth” computes W(\mu’_l, \nu’_l) = <f_l, \mu’_l, \nu’_l> from the POT package (though not necessarily through explicitly computing f_l.)

The issue with this is that we expect the method to perform well if f ~= f_l, which can be achievable if all the f_l’s are similar (and hopefully they’re all approximately equal to f_\star.) However I don’t think this is true -- as (\mu’_l, \nu’_l) are samples, and because of the curse of dimensionality, we should expect the f_l’s to be quite different from each other. (Otherwise if they’re really just ~= f_\star, then we can use standard concentration to show the W(\mu’_l, \nu’_l) ~= W(\mu, \nu), which we know is not true from curse of dim.)

Given this, I don’t think the task of comparing <f, \mu’_l, \nu’_l> with the POT results really says anything about their power in computing W. I would be glad though to hear back from the authors to see if my understanding is accurate, and adjust my evaluation from there.

review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper has studied the efficiency and stability of computing the Wasserstein metric through its dual formulation under weight clipping, gradient penalty, c-transform and (c- ϵ)-transform. The results show that (c- ϵ)-transform and c-transform give more estimation of the Wasserstein distances than the gradient penalty and weight clipping methods in the given experiments, but the gradient penalty method produces more compelling samples in the generative setting.

The paper is well written and the experiment section is extensive. However, it is more like an extended experiment report to me which is very valuable but lacks sufficient technical novelty expected at ICLR.

Another comment is that when the authors mentioned "... are compared to ground truth values d_ground computed by POT", there needs more explanations on what that library actually does to compute the Wasserstein distance to make the paper self-contained, e.g. what exact algorithms it uses as there are also dozens of different algorithms implemented in that library.

In Section 3.1, the authors state that "it tends to converge within couple of iterations of the symmetric Sinkhorn-Knopp algorithm. For efficiency, we approximate these terms with one Sinkhorn-Knopp iteration". What is the extent of sacrifice in accuracy due to this approximation? The authors should provide more evidences to justify the approximation.
review_title: Official Blind Review #1

Meta-review: There is insufficient support to recommend accepting this paper.  Generally the reviewers found the technical contribution to be insufficient, and were not sufficiently convinced by the experimental evaluation.  The feedback provided should help the authors improve their paper.
Decision: Reject