Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to learn the optimal transport map from the latent distribution to the data distribution directly by optimizing the W2 distance. To find the OT map, the authors replace $y$ with $T(x)$ in the Kantorovich dual problem.
Experiments show that the proposed method works well and learns the image distribution successfully.


main_review: Pros:
- The writing is clear and easy to follow.
- The experimental results are solid and convincing.

Cons:
- I am not sure if equation (10) and equation (11) is really equivalent to each other. To apply the Brenier theorem, namely replace $y$ with $T(x)$ in equation (10), we need to add the measure preserving constraint $T_\#\mu = \nu$ in equation (11). Without this constraint, which requires $T$ to be a measure preserving map, equation (11) may goes to $\infty$. For example, $T(x)\equiv 0$ and $\psi(y)=-\infty$. The optimal transport map should be included in the feasible set of equation (11) if it is large enough, but without the measure preserving constraint, it's hard to prove that the learned map $T$ is measure preserving. Therefore, the authors need to make some effort to show the equivalence. 

- Furthermore, to make the conclusion more convincing, I also recommend the authors to experiment on more 2-dimensional toy sets. If the learned map $T$ is an optimal transport map, the samples of the Gaussian distribution should be evenly transported to the Mixture of 8 Gaussians in Fig. 14(f). But it is really difficult to find the 'even' transportation (especially for the southeast samples), which weakens the conclusion that the learned is an optimal transport map.
summary_of_the_review: My main concern comes from the deduction from equation (10) to equation (11). The authors may add more explanation to show the equivalence.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work introduces a method for doing generative modeling, and conditional generative modeling (e.g., image restoration) using the optimal transport map between two distributions.

The paper first discusses the different uses of Optimal Transport (OT) on generative models, and shows that most of the works are on using the OT cost and not the map/plan itself. The paper also discusses some of the work that does show the OT map but only into latent spaces due to the intrinsic complexity of computing the OT map on the original space.

The proposed method uses the Wasserstein-2  distance as a cost measure and assumes the existence of a unique OT deterministic plan, implying the existence of a map.  The formulation of the OT plan estimation problem (Lemma 4.1) ends up addressing a saddle point problem (max-min).

The analysis requires that the dimensions of input and output spaces be the same. The method also proposes to use a deterministic and non-learnable mapping to embed the input signals on a space that has the same dimension of the output space, so the framework can be applied. This embedding needs to be manually defined depending on the application.

The paper presents experiments on image generation and image restoration on classical  datasets: MNIST, CIFRAR, CelebA (64x64). Regarding image restoration three different applications are analyzed: denoising, colorization inpainting.

The method compares similarly to other generative modeling models (in particular similar to WGANs).

main_review: This work addresses an important problem and proposes an interesting solution using the OT map between input and output distributions.
Experimental results show that the method performs similar to other existing methods.

**Paper organization/presentation**

- We need to wait till the middle of Section 4 to understand what the paper proposes. The introduction to the proposed idea is too long (even if  some of the concepts need to be provided,  a lot of that content can probably go to an Appendix).

- Lemma 4.1 seems the relevant one. Maybe this could be better introduced in the paper (there's no  connection to the previous paragraphs, so it's hard to follow the story). 

- The paragraph following Lemma 4.1, that tries to connect the Lemma to the previous work is not particularly clear. The ultimate question is: have people tried this idea of estimating an OT map before? and if so, how is the proposed solution different from previous work (technical difference, algorithmic difference,....). 

- The technical extension to unequal dimensions seems straightforward if you have Q.

- The paper discusses in multiple places the relation to previous work. This makes it hard to follow (the presentation is a little not linear, one needs to jump from one place to the other to get all the pieces). 

**Technical comments**
- Most image restoration problems are ill-posed: there are multiple clean images that could lead to the same low-quality observation. This implies that it won't exist a deterministic OT plan (so no 1-1- map). This is particularly obvious in the case of image colorization: from a gray image estimate a possible RGB image. This raises the question of what is the formulation doing in this case? There seems to be a mismatch between the mathematical formulation and the practical use in inverse problems. 

- The paper claims optimality of the restoration map, but in most inverse problems it is unclear wether this mapping exists (in general there's a loss of information so no 1-1 mapping between low-quality - high-quality image).

- One of the claims is that previous work using OT map didn't address the problem in the input space dimension. The paper does not provide intuition or a mathematical justification why the proposed formulation should work better than the other methods when addressing the problem in the original dimension.

**Experiments**
- The paper claims to be able to work on high-dimensional data, but in terms of image size the experiments consist of images of at most 64x64 RGB pixels. It's true that this could represent a high-dimensional dataset, but in general most of the problems with generated/restored images are visible at larger resolutions (this is particularly the case for image restoration). 


**Typos**
- pag 3. Duality paragraph. $u^c(x)$ → $u^c(y)$.
- pag 4. Alexandroff theory (?)
- pag 4. Figure 3. This figure is not particularly illustrative.
- pag4 - pag5. Indentation and presentation of equations (8) to (11) is confusing.
- pag 5.  const($\mu$,$\nu$) → cost($\mu$,$\nu$)
-pag 6. "large-scale generative models" doesn't seem particularly accurate for the shown experiments (32x32 to 64x64 images).

 

summary_of_the_review: This paper introduces a mathematical formulation to recover the OT map that maps two distributions for doing generative modeling and (unpaired) image restoration tasks. The main weaknesses of the paper are the presentation and experimental validating the approach.  It's hard to understand what is different with respect to previous work also doing OT map estimation. What's particularly interesting about this mathematical formulation that enables it to work on the original dimension?  Experiments are only on small images which implies that the claim of working on high-dimensions and large-scale datasets is a little too strong.

--
After more discussion with the authors and after considering the updated manuscript I'm raising my score (marginally above the acceptance threshold).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, a novel optimal transport (OT) map based generative model is proposed. The OT map is computed as the generator map from noise distribution to data distribution. Different from previous works that mainly focus on manipulating distributions in latent space, the proposed method directly computes the OT map in the data/ambient space. One of the benefits of this approach is that the FID is controlled by the W-2 loss in the data space (i.e., theorem 4.3). Experiment results show that the proposed method has comparable or better performance than competitive methods on various tasks. 
main_review: This paper sheds new light on using OT maps as generators by exploiting the OT map from latent space to data space with unequal dimensions. This is in contrast with previous approaches that manipulate distributions in the latent space. 

The theoretical results of the error analysis of the proposed method are solid and novel in computer vision field. The proposed method does not add a sophisticated mechanism to compute the OT map compared to WGAN. Instead, the structure is very similar to that of WGAN model, which is a strength in terms of computational complexity. 

Extensive evaluation of the proposed method with previous methods is thoroughly conducted on both synthetic and real-world datasets. The proposed method performs comparable or better in terms of visual quality of the generated images as well as FID scores than competitive methods. 

A minor concern is that, as pointed out by previous studies (e.g., An 2020(a)), the OT map is generally highly discontinuous which makes approximating this map by DNNs tricky. I think this is probably one of the reasons why some previous methods avoid using DNNs to compute OT/generator maps (in contrast with the proposed method). A discussion on the singularities of the generator/OT map would be helpful for a deeper understanding of the proposed method. 
summary_of_the_review: Overall the paper is well-presented, with novel theoretical and empirical results. I would like to recommend accepting this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a min-max optimization algorithm to apply OT maps directly in ambient space. And that extends the method the case when the input and output distributions are located in the spaces of different dimensions and derive error bounds for the computed OT map.
Contributions of this paper have three aspects: 1) the end-to-end algorithm is given to fit OT maps for Q-embedded quadratic cost, i.e., if two distributions located on the spaces of equal dimensions, the identity embedding Q(x)≡x (Wasserstein-2 distance); if not, choosing Q-embedded quadratic cost. 2) The theoretical analysis the error bounds is proved. 3) this paper demonstrates large-scale applications of OT maps in CV tasks, image generation and image restoration.

main_review: This paper has some strengths. 1) the proposed method can apply OT maps directly in ambient space, and fit OT maps for Q-embedded quadratic cost between located on the spaces of unequal and equal dimensions. 2) The boundary error of OT maps is analyzed theoretically.
This paper has some weaknesses. 1) Compared with OT maps as loss function and OT maps as generation model, what is computational complexity of this paper. 2) what is the detailed model diagram of this paper, and how do you choose appropriate Q-embedding, what exactly is Q-embedding ?

summary_of_the_review: The idea of this paper is very clear, the theoretical analysis is rigorous, and the experimental results can fully illustrate the performance and effectiveness of the method. But there are a few problems, The c-transform $u^{c}(x)$ on page3 should be $u^{c}(y)$, The computational complexity of the algorithm is not reflected in the text, what is Q-embedding? what is the relationship between it and the generated mapping?
review_title: null

Meta-review: The paper proposes a new method to learn OT maps, and reframes it in the GAN literature. The initial method works when computing maps between equal dimensions, through duality and an identity (10 - 11, amply discussed in the reviewing process). Lemma 4.1 provides the main result. While the discussion right below on the fact that several functions (non-OT maps) might maximize that criterion is not completely satisfactory, the result provides an interesting characterization. The second contribution adds a method to compute OT maps between spaces of unequal dimensions. Overall the contribution sounds a bit ad-hoc, and one wonders whether this does really work (comments such as "we add small gradient penalty (Gulrajani et al., 2017) on potential ψω for better stability. The penalty in not included in Algorithm 1 to keep it simple." are strange and point to instability) but the overall creativity and new ideas in the paper seem to have garnered enough support from reviewers to push for an accept.
Decision: Accept (Poster)