Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a continuous Wasserstein-2 benchmark to evaluate different neural OT solvers including WGAN varients. Extensive experiments are performed on toy datasets and a real face dataset.
main_review: Strength:
1. It's good to see a benchmark that evaluates different neural OT solvers. This could benefit future OT application works that choose the best OT solver. 
2. Comprehensive OT solvers are investigated, including some WGAN variants that can be potentially used as OT solvers. 
3. Various metrics such as L^2-UVP, cos, and FID are used to evaluate OT solvers.

Weakness:

I think two important theoretical problems remain unsolved in this paper:

1. In line 216, the authors mentioned that to compute the L^2-UVP and the cos metrics, they use 2^14 random samples from P. They actually computed the estimated values for the two metrics. 
I would like to know the sample complexity of estimating the two metrics, i.e., the estimation error of L^2-UVP and cos in terms of the number of samples required. Is there a curse of dimensionality problem in estimating the L^2-UVP and cos,
just like computing the Wasserstein distance for two continuous distributions using sampling methods? This is important because if the number of required samples grows exponentially as the dimensionality increases,
then the metrics may not be accurate in evaluating OT solvers in the 64d space but only with 2^14 samples.

2. In line 175, the authors construct a benchmark pair (P, 1/2 (\nabla \psi_1 + \nabla \psi_2)#P), and the 1/2 (\nabla \psi_1 + \nabla \psi_2) is used as the ground truth transport map if I understand correctly. 
My concern is that "is 1/2 (\nabla \psi_1 + \nabla \psi_2) really the optimal transport map between P and 1/2 (\nabla \psi_1 + \nabla \psi_2)#P"? Remember that \psi_1 and \psi_2 are approximated by the W2 solver. 
I think this paper should theoretically show "how far is 1/2 (\nabla \psi_1 + \nabla \psi_2) away from the real optimal transport map between P and 1/2 (\nabla \psi_1 + \nabla \psi_2)#P", and show how this 
gap affects the metrics that are used to evaluate OT solvers.
limitations_and_societal_impact: The authors have addressed some limitations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present a continuous Wasserstein-2 benchmark using input convex neural networks to obtain ground truth continuous OT maps. They evaluate continuous OT solvers based on how well they reproduce the ground truth transport map, how well they approximate the derivative of the potential, and how well they perform in a GAN framework. This provides a basis to evaluate current and future continuous 2-Wasserstein solvers.
main_review: Overall this was an excellent read and I found the use of the ICNN to generate ground truth OT maps novel and clever. The authors mention "Because of the choice of [W2], subsequent evaluation might slightly favor ICNN-based methods" (Line 177) and elaborate in the limitations section. This seems like a major limitation, the fact that the transport maps are not based on any real distribution and are in fact sampled from the set of potentials parameterized by ICNNs is highly biased. Would it be possible to examine this bias in some way? Perhaps some P, Q pairs with tractable transport maps that do not require ICNNs?

The evaluation of W2 solvers in generative modeling is also extremely useful. While this study has a number of limitations it is well posed and a useful benchmark for future work in W_2 solvers. I found this a fascinating read and applaud the authors for their efforts.

------------------------------
Edit post response:
I am satisfied by the authors' response. I keep my score. 

I initially had the same understanding as reviewer oJwC made in his point 2. Given that this is a mainly a benchmark paper, the construction of the benchmark is of utmost importance. It would improve the paper to further emphasis the construction of the ground truth based on maps model-able by ICNNs, which is the most significant limitation of this work. Having a clear discussion of the use of DenseICNN parameterizing the ground truth transport maps earlier in this work perhaps in section 3 (rather than just in the limitations paragraph at the end) would both make it clearer how the benchmarks are constructed and what the limitations are. The paragraph titled "Arbitrary pairs"on Line 172 is misleading as a ground truth map is not constructed between arbitrary pairs, but between P and something close to Q based on how well the ICNN is fit; I suggest "Approximating arbitrary pairs" or similar would be clearer.

 While I agree other convex parameterizations could be used, this is an important point and deserves further clarification.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work introduces a continuous Wasserstein-2 benchmark to assess the qualities of different neural optimal transport solvers. The work uses input convex neural networks (ICNN) to construct pairs of measures whose ground truth OT maps can be obtained analytically. This strategy yields pairs of continuous benchmark measures in high-dimensional spaces such as spaces of images. The authors thoroughly evaluate existing optimal transport solvers. The study reveals crucial limitations of existing solvers.
main_review: Optimal transportation maps become more and more important and popular in deep learning field. Due to its highly non-linear nature, reliable computation with high precision is challenging. So far, there is no solid benchmark to assess different solvers. This work proposes a benchmark to tackle this problem. Hence it has importance for deep learning fields. 

The manuscript is well written. The introduction part is highly motivated and encouraging. The theoretic foundation part is clearly written, easy to follow, but can be further improved.

1. The background on optimal transport is insufficient. There are many existing methods based on Brenier theorem and solving Monge-Ampere equations. For example, Benamou-Brenier used fluid dynamic formulation to solve L2 optimal transportation map; Yau et al give the intrinsic connection between convex geometry and optimal transport, and solve the OT map problem using Alexandrov formulation, see the work  of Lei et al "A Geometric Understanding of Deep Learning",   Engineering, Volume 6, Issue 3, March 2020, Pages 361-374. The algorithm there has theoretic foundation. 

2. The benchmark ignores one important aspect of optimal transportation map: the boundary condition. In theory, the image of the OT map of the source domain should equal to the target domain, this is the most difficult constraint to satisfy in practice and affect the computational result crucially. According to Figalli's works, if the target domain is non-convex, then the OT map itself may not be continuous, therefore cannot be represented by the deep nerual networks. Even for the benchmark itself, the ground truth map may not be representable by the network. According to Yau's works, this is the reason for mode collapsing. Hence this need to be emphasized and carefully designed in the bench mark.

The experimental results are convincing, different solvers are compared. 

3. It will be more helpful to add geometric based approaches for the test. It will be more convincing if the convergence error is also added. According to the theoretic analysis, the Brenier potential error is O(h^2) where h is the diameter of the cells in the tessellation. 

In general, the idea of Benchmark is important, it will make the whole field more rigorous. Current design can be further improved by emphasizing the 2nd boundary condition, and considering the regularity of the OT maps. 

limitations_and_societal_impact: There are more limitations of the current design:

1. Secondary boundary condition should be emphasized, which significantly affect the computational quality.
2. The regularity of the OT maps need to be carefully considered, if the target measure support is non-convex, the OT maps can not be represented by deep neural networks. This fact should be carefully addressed.
review_title: null

Meta-review: In this paper, the authors propose an interesting approach to evaluating transport plans. There exist some concerns about methodology. However, to my knowledge, I have never seen this type of study before in OT problems. So, the problem itself is new and novel. Thus, I also would like to vote for acceptance. For the camera-ready version, I expect the authors to revise the paper based on the reviewer's comments.
Decision: Accept (Poster)