Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a method to scalably compute Wasserstein-2 barycenters given samples from input measures. In general the authors also allow for continuous measure settings. Inspired by Li et al. (2020) the paper uses a potential-based approach and recovers the barycenter by using gradients of the potentials as pushforward maps.

In general, I feel this paper is well-written and provides a fast solution to a meaningful problem, thereby supporting the claim of novelty. The theoretical developments in the paper are reasonable and the experiments carried out are quite decent, both in simulation and real-data settings.

The only point that bothers me is the approximation used. It would be great if the authors could give an extensive and detailed understanding of settings where the upper bound in Eq.(10) in the main text is small thereby leading to a good approximation. 
review_title: Upper bound for error needs detailed explanation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work introduces a new Wasserstein-2 barycenter computation method. The authors first derive the dual formulation of the Wasserstein-2 barycenter problem, and then parametrize the convex potentials by ICNNs. The congruent and conjugacy conditions are enforced by regularization terms, respectively. They then show that the algorithm can find a good barycenter if the objective function is properly minimized.

Pros:
1. The algorithm does not introduce bias.
2. The algorithm does not require minimax, which is efficient.
3. The empirical performance is much better than existing methods, probably due to the above two reasons.

Areas to improve:
1. It is good that the empirical analysis include how the performance change w.r.t. D. It would be better if there is a similar analysis to N. Furthermore, since 2N ICNNs are needed to be trained, it would be better if the training time is also reported, so that we can have a more comprehensive understanding of the method. Will there be a setting that discrete method can be faster than the proposed method to enforce comparable approximation error (say, large N for 3D applications)?
2. Since the congruent and conjugacy conditions are enforced by regularizations, they are not guaranteed to be satisfied. Therefore, it would be better if there is an experiment showing that how the conditions are satisfied. 
3. The first section of related work should also briefly include https://arxiv.org/abs/1605.08527 and https://arxiv.org/abs/1905.00158.

After rebuttal:

The additional experiment results provided in the rebuttal stage suggests the efficiency of the proposed method, as well as the congruent and conjugacy conditions are approximately satisfied. I therefore believe this paper should be accepted.
review_title: Neural based method for continuous Wasserstein-2 barycenters, with good performance

final_score: rating: 7: Good paper, accept
review_content: Summary:

The paper considers the Wasserstein Barycenter problems in the continuous setting. In particular, the authors propose an algorithm to compute the Wasserstein-2 barycenter when only samples from the marginals are accessible. Some theoretical analysis of this method is presented. Several numerical examples are carried out to compare this method with two other recently proposed methods.

Reasons for score:

The proposed algorithm utilizes an interesting regularization of the dual formulation of Wasserstein-2 Barycenter, resulting in a single minimization problem instead of a min-max problem. This algorithm is properly justified by theoretical results as well as numerical experiments. 

Pros:

1. The paper provides theoretical results on the consistency of the proposed algorithm.

2. The experiments are overall good and clear. 

2. The paper is well-written and easy to follow.

Cons:

1. High-dimensional examples other than the simple Gaussian setting are missing.

2. There is no analysis of computational complexity of the proposed algorithm. Also, the training expense is not reported.
 
3. The double gradient in the second regularization term could be expensive to evaluate.

Questions: 

1. In both Theorem 4.1 and 4.2, the smoothness of the potentials is crucial. If the smoothness B is too large, then the bound presented is essentially useless. Please comment on it.
review_title: review

final_score: rating: 7: Good paper, accept
review_content: **Summary**
The paper derives the barycenter mapping problem as an optimization over *congruent* convex functions---each convex potential corresponding to a component distribution.  Congruency is a property on the set of optimal potential functions that ties them together.  However, this optimization is quite challenging and so the paper derives an principled objective function that includes two regularization terms.  The first regularization term encourages congruency of the set of convex functions and can be seen as a variational bound on an ideal congruency regularization.  The second regularization term encourages the pairs of convex functions to be conjugate.  The paper proves that the optimal solution of this objective is the true potentials and thus no bias is introduced.  The proposed approach is demonstrated on the tasks of generative modeling (2-256 dimensions), posterior inference, and color pallete barycenters (3D)

**Strengths:**
- Nice problem formulation and setup with respect to prior methods.
- The derivation of the final objective function is clearly laid out and well-motivated.  Each problem that is encountered is explained and then a solution or approximation is introduced.
- The theoretical results give appropriate grounding for the approach.
- The empirical results outperform prior potential-based methods for barycenters.

**Weaknesses:**
- It is unclear if this method can scale in terms of samples and dimensions.  What is the computational cost of estimating these input convex neural networks?  Can you provide (approximate) wall-clock times for the various methods and dimensionalities? What are the key computational bottlenecks either memory-wise or computation-wise?
- The experiments seem small scale with a max dimension of 256.  Barycenters for high dimensional real-world data (e.g., even MNIST (784D)) or some other high-dimensional real-world dataset would improve the paper.
- The paper lacks comparison to methods that do not recover 2N potential functions. What are the closest methods for barycenter that do not use potential functions?  For example, could the algorithms be compared to discretized barycenter algorithms to show the breakdown in higher dimensions?

**Other comments or questions**
- Is $D$ in equation 5 supposed to be $N$?
- Some typos above Eqn. 12.

**Update after author response**

I appreciated the authors response to the scalability and raw computation times.  Thank you also for the additional comparison to a non-potential function method.  This will be a good comparison.  My main concerns were answered, and I still think this is a good paper.
review_title: The paper carefully derives a novel non-adversarial objective via two regularization terms for finding the Wasserstein barycenter maps via convex potential functions.

Meta-review: The authors propose the 2-Wasserstein barycenter problem between measures. The authors propose a novel formulation that leverages a condition (congruence) that the optimal transport (Monge) maps, here parameterized as potentials, must obey at optimality. The introduce various regularizers to encourage that property. The idea is demonstrated on convincing synthetic experiments and on a simple color transfer problem. Although experiments are a bit limited, I do believe, and follow here the opinion of all reviewers, that there is novelty in this approach, and that this paper is a worthy addition to the recent line of work trying to leverage ICNNs/Brenier's theorem to solve OT problems.
Decision: Accept (Poster)