Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper provides a benchmark to evaluate approximators for Wasserstein-1 distances as loss functions in the generative adversarial network setting.
strengths: - **{S1}** While previous works use discrete distributions for benchmarking solvers, this work suggests continuous distributions, which is a novel aspect for benchmarking W_1.
weaknesses: - **{W1}** The benchmark contains only one image dataset with a single mode (faces). The addition of more image datasets, especially multi-modal ones (e.g. CIFAR-10), would improve the versatility of the benchmark and extend it to conditional models.
correctness: I am not an expert and have been working on the practical side before, so I cannot assess the correctness of this paper.
clarity: The writing itself is good, however, as someone coming from a different background it was hard to follow.
relation to prior work: This benchmark suggests comparing continuous distributions, while former works compared discrete distributions.
additional feedback: See {W1}.
review_title: Review of Kantorovich Strikes Back!

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: Authors propose a generic methodology to construct benchmark pairs with ground truth OT plan, OT cost, and OT gradient. We can use this tool to evaluate the performance of the neural dual OT solvers approximating the Wasserstein-1 distance or the gradient of Wasserstein-1 distance.

Specifically, the authors employ the 1-Lipschitz MinFunnel functions to compute transport rays and define the ray monotone map. With them, we can define a target distribution $\mathbb{Q}$ and compute OT cost and OT gradient based on the original distribution $\mathbb{P}$
strengths: The authors provide an elaborate introduction to the Wasserstein-1 and its neural dual OT solvers. Followed by compact math proof about their benchmark pairs. Experiments are also reasonable. It is also a good point of view to consider the gradient of the Wasserstein-1 distance.
weaknesses: Some minor concerns.

Is it hard to turn hyperparameters for this method? For example, when you compute the High-dimensional benchmark pairs, you choose $b_n \sim \mathcal{N}(0,0.1)$ and p = 8. How do you choose it? How long does it cost for the hyperparameter search?

The dimension of images ,in reality, is higher than $2^7$. Can this tool handle higher dimensions?

If we carefully choose MinFunnel function u, instead of randomly picking, will the performance be better?

What will be the effect of increasing N and D?

Paper mentions "in WGANs, the solvers move the generated distribution (bad images, $\mathbb{Q}$ in our construction) to the real distribution (good images, $\mathbb{P}$)". However, $\mathbb{P}$ is synthetic distribution and $\mathbb{Q}$ is computed ground truth 'real image' distribution, in the case of images benchmark. Why do the solvers move $\mathbb{Q}$ to $\mathbb{P}$, instead of the opposite?

Authors mention solvers MM, MM:R takes longer for training, compared with GP, SO, and LP. Is the time gap significant? 
correctness: The claims and method are correct, based on my knowledge.
clarity: The paper is well written.
relation to prior work: It is clearly discussed how this work differs from previous contributions. It also introduces important background knowledge.
additional feedback: N/A
review_title: Is it hard to turn hyperparameters? Can it be applied to real-world images?

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Motivated by the lack of benchmarks for W1 dual methods (other than perceptual measures such as FID or IS), this paper proposes to create a (semi-)synthetic set of benchmark datasets with known optimal transport plans, maps, and distance.
To do this, the paper first develops theory about maps that are optimal by construction.
Then, the paper proposes concrete methods for constructing the necessary functions and computing the necessary plans, maps and gradients.
Finally, synthetic dataset pairs are generated from truncated Gaussian data and CelebA data at various dimensionalities and used to evaluate and discuss many existing W1 methods.

strengths: - Discusses good overview of W1 methods.
- Proves theoretical results about how to construct maps that are optimal w.r.t. W1.
- Proposes novel way to construct ground-truth (semi-)synthetic benchmarks for evaluating Wasserstein-1 dual solvers.
- Provides code and datasets for benchmark datasets and algorithms.
- Evaluates the gradient of the W1 w.r.t. the parameters, which is actually most important for most generative methods.
weaknesses: - Only one real-world dataset (celebA) is considered. And the synthetic datasets are quite simple (i.e., truncated Gaussians). It seems including more real-world datasets (even MNIST or CIFAR10) would be useful or using interesting real-world tabular data for smaller dimensions (e.g., even something like iris).

- (This limitation is mentioned in the text but does seem to be the main limitation) It seems the benchmark only considers maps where the samples are grouped more closely together (or the reverse). Maps that expand parts of the space or where some expand and some contract would be better. It is unclear whether the benchmark maps properly represent real-world OT maps.

- (Minor but nonetheless important for final paper) All result tables are in the appendix. And the figures are in odd places with nonstandard captions. At least some summary table of the results and your recommendations for suggested methods based on context would be important to include. What methods would you recommend and why?  The answer may be a combination of ease-of-use, convergence behavior, and overall performance.

correctness: - Shouldn't MM and MM:R (Eqs 9 and 10) have the min outside the integral?  Or is it meant to find a new $T$ or $H$ for every $x$ or $y$? Maybe I'm missing something about this notation.

clarity: Overall, I found the paper was clearly written but could use some improvements in a few areas.

- Why do you truncate to the hypercube? The reason stated is "For convenience" but I don't understand why it's convenient. Why not just sample from uniform distribution?  Why truncated Gaussian?
- I did not understand why you could choose any point along the ray (lines 194-195).  Does this mean you could choose a different point along the ray for every $x$?  For example suppose two unique points both lie along the same ray, could you choose two different points to map them to (e.g., that switches their order)?  Maybe I'm missing something here. (It seems you choose a smooth, deterministic way in Sec 3.3 Eq 12.)
- The discussion section is fairly hard to parse without a legend for all the methods.  Maybe a small inline table summarizing each method (e.g., "GP - Gradient Penalty") would be helpful. Otherwise it is very hard to remember all the abbreviations.

*Minor Clarity*
- An early illustration in 1D of L1 functions with "rays" and corresponding transport maps could help introduce the main idea.  I found Figure 3 helpful but it might have helped to anchor intuition before diving into the math.
- Figure captions look centered rather than justified and are nonstandard.
- It might be interesting to see overlaid t-SNE plots of Figure 4 that project both samples of P and Q via t-SNE.

relation to prior work: Overall, the paper does a good job of placing the contributions w.r.t. related work. However, the related W2 benchmark should probably be mentioned much earlier on and at least some brief explanation of how these are related but different.  The first mention of this was in the experiments---and was actually a question I had while reading the first part.
additional feedback: See clarity above.
review_title: Semi-synthetic W1 benchmark with ground truth

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a benchmark to evaluate the methods of computing the Wasserstein-1 distance. The authors construct 1-Lipschitz functions and use them to build ray monotone transport plans, which yield pairs of continuous benchmark distributions in high-dimensional spaces. Some WGAN dual form solvers are evaluated using these benchmark pairs.
strengths: 1. This paper proposed a benchmark to evaluate the methods of computing the Wasserstein-1 distance. The problem is interesting to the community. 

2. This paper is well-written and technically sound. The method uses 1-Lipschitz functions to construct pairs of continuous distributions, which is well designed. 

3. This paper thoroughly evaluates popular WGAN dual form solvers in high-dimensional spaces using these benchmark pairs.
weaknesses: 1. The title of this paper is ambiguous and may lead to inappropriate reviewers.

2. The theoretical analysis and the intuition of the proposed method is weak. It is unclear why the proposed method works well than previous methods.

3. Evaluating the Wasserstein-1 distance does not directly validate the superiority of the methods on specific tasks, which may need more explanations.
correctness: The evaluation methods and experiment design are generally appropriate and performed correctly.
clarity: The paper is clearly written and well-organized.
relation to prior work: See Weaknesses.
additional feedback: NA
review_title: A generic benchmark to evaluate dual OT solvers for the Wasserstein-1 distance.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a benchmark for computing the Wasserstein-1 distance. The authors first propose to use 1-Lipschitz functions to build ray monotone transport plans and obtain known OT maps. These ground truth maps are then used to benchmark dual OT solvers used in particular in the Wasserstein GAN framework.
strengths: - This papers proposes a method to build **known** OT maps using 1-Lipschitz MinFunnet functions. This choice is clearly justified as these functions are universal approximator of 1-Lipschitz functions (Prop.2). Having known OT maps allows to faithfully compare the OT solvers 
- They carefully build transport ray of these functions.
- The paper is well written and easy to follow.
- The authors tackle an interesting problem and having more comparison like this one is crucial 
weaknesses: - I regret that the results of the benchmarks are only available in the Appendices. I would recommend the authors to include some of them in the main paper since those are the main results of the paper.
- The restriction to 1-Lipschitz *MinFunnet* functions seems to be a main limitation of this work.
- It seems that in the experiments only one random start is considered. Is there any reasons why the authors did not perform multiple runs? This seems to impede to assess the methods stability and robustness with regard to the random start and the parameters $a_n$ and $b_n$ in the *funnel*.
correctness: - The benchmarks pairs are well designed and well justified.
- Not including error bars in the experiments prevents from quantifying the methods stability and robustness to the random start. This should be part of the benchmark.
clarity: This paper is well written and easy to follow.
relation to prior work: To the best of my knowledge this paper clearly discusses and states how it differs from prior works.
additional feedback: - I would suggest the authors reshaping a bit the paper to include some of the metrics from the benchmarks in the main paper to better support the claims made when discussing the results in Sec.4.
- If there any reason why the authors did not include any error bars on the metrics they computed?
review_title: An interesting benchmark of dual OT solver for W1 distance

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a benchmark for methods computing the
Wasserstein-1 distance. Section 1 summarizes background
information on computing W1, often with
the dual in eq (4) and (5), and how the W1 is used
in GAN training.
Section 2 summarizes methods estimating the dual
potentials and transport maps.
Section 3 describes the benchmark distributions,
and Section 4 shows the results of evaluating the
methods on the results, which are quantified in
Section D of the appendix.
strengths: + Approximating W1 computations is widely used and a difficult
  setting to benchmark because the ground-truth transport
  maps and distances are often not known.
  I am not aware of an established W1 benchmarks and papers
  often have to rely on downstream tasks (such as inception scores)
  to justify an algorithmic improvement to the W1 approximation.
+ This paper presents non-trivial settings where the
  ground-truth transport map is known and uses it to
+ The experimental results are thorough and the paper
  strongly shows that minimax methods solve
  the benchmark tasks in most settings, at least for
  obtaining a gradient that approximates the true gradient.
weaknesses: + While the paper proposes a new benchmark for approximating
  the W1, it unfortunately does not present results
  in established GAN settings as the ground-truth maps
  are not known. Thus research that is ultimately focused on
  improving the W1 computations in settings such as GANs
  may be able to use these benchmarks for preliminary
  experiments, but these benchmark tasks may not
  reflect the true difficulties.
  of these methods thus established and powerful
+ It is not clear how "solved" W1 OT is, how much work
  remains in the field, and how many new directions
  this benchmark will enable. In other words, better
  solutions to this benchmark will not directly enable
  new methods (or new GAN results).

correctness: The benchmark and methods are correct and the evaluation is properly done.
clarity: The paper is well-written and easy to follow
relation to prior work: This paper properly discusses and connects together all of the relevant work
additional feedback: n/a
review_title: Important setting quantifying W1 approximations

Meta-review: metareview: This paper proposes a new benchmark to evaluate the solution of optimal transport problems. The reviewers concur that the benchmark is well-executed and novel. Some are concerned that a better benchmark for OT problems will not drive progress, as the successes of Wasserstein GANs occur despite their failure to solve OT. However, it seems like a useful intermediate check to deepen understanding of why Wasserstein GANs (and models to come!) work by (at least) eliminating non-explanations.
confidence: 3: The area chair is somewhat confident
Decision: Accept