Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes conditional transport as a new divergence to measure the difference between two distributions. The idea is to learn the conditional transport plan of transporting one point in one distribution to the other marginal distribution. This conditional transport plan is modeled using a neural network. The resulting model is then applied to optimal transport formulation. Experiments are shown on image-based generative modeling dataset.

The main idea is to decompose the joint transportation plan $\pi(x, y)$ using conditionals as $p(x)\pi(y|x)$ and $p(y)\pi(x|y)$. The conditional transportation plan $\pi(x|y)$ and $\pi(y|x)$ are then modeled using neural contrastive losses, with the idea being similar points in two distributions are mapped closer. This decomposition is then applied in optimal transport formulation, leading to a new objective for optimizing OT. The authors also derive an empirical version of this objective, that makes it amenable for stochastic mini-batch optimization.

While the idea of this decomposition is interesting, I see the following issues with the formulation.

(1) From the forms of conditional transportation plans $\pi(x|y)$ and $\pi(y|x)$ in Eq 2 and 3, $p(x)\pi(y|x) \neq p(y)\pi(x|y)$. It would be good to have models that satisfy this equality.

(2) I am not sure if the contrastive model assumed in Eq 2 and 3 is expressive enough to model the entire space of marginal constraints in OT. That is, I don’t think $p(x)\pi(y|x)$ will cover $\Pi(\mu, \nu)$. Lemma 1 just shows that under some conditions, $p(x)\pi(y|x)$ lies inside $\Pi(\mu, \nu)$, but it would be interesting to see if then entire space of $\Pi(\mu, \nu)$ is covered by the neural contrastive model.

(3) When critic is used in ground cost function $c(x, y)$, the resulting optimization (Eq 16) has one additional network compared to standard GANs. Also, the adversarial game still exists. So, it looks like this optimization is more harder (or at least equally harder) compared to standard GANs. Is this true? Do you see any optimization benefits compared to standard GANs?

(4) The results of image-based generative modeling is not that impressive. On CIFAR and LSUN datasets, the performance is similar / slightly better than the compared GAN models. Also, many SOTA GAN models are not compared. So, it is very hard to say if the proposed model advances SOTA results.

(5) I would have liked to see more interesting results. The formulation of authors gives an estimate of transportation plan $\pi(x, y)$ in addition to the generative model itself, which is not possible to estimate in dual-based OT GAN optimization. The transportation plan can be used in interesting applications. One possibility is to estimate likelihoods and possibly use in OOD detection. Take a look at Balaji et al., “Entropic GANs meet VAEs: A statistical approach to compute sample likelihoods in GANs” for this. This is just one idea. Other interesting applications could have been demonstrated as well.

Overall, while the idea is interesting, more results could have positioned the paper better. Just selling it as a paper that improves image-based generative modeling is not that great in my opinion. I would encourage authors to think more interesting experiments.
review_title: interesting idea but needs more experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The paper introduces an asymptotic conditional transport divergence to measure the discrepancy between two probability distributions. The new divergence leads to a new adversarial game of generative adversarial networks, which aims at minimizing a distribution-to-distribution transport cost by optimizing the generator distribution and the conditional transport-path distributions of navigators. 

Strength:

Introducing the asymptotic conditional transport divergence to generative modeling looks novel and interesting to me. 
Both the motivation and the technical details seem sound.
The evaluation shows the effectiveness of the proposed method on both toy datasets and some popular image generation datasets.

Weakness:

Fig.3 has shown the superiority of the proposed divergence against the existing divergence (JS, Wasserstein distance, and Sliced Wasserstein distance) to fit the toy data distribution. But readers might doubt whether the comparison is fair or not. Do they use the same network backbone (e.g., DCGAN, WGAN-GP)? As there are many improved variants of SWD like (Deshpande et al., 2018, Deshpande et al., 2019, Wu et al., 2019), I am also wondering which SWD is used. The problem might get more serious for general cases. To address the problem, the paper is suggested to additionally present a clear theoretical study on the comparison, as readers really want to know why the proposed divergence works better than others, while some explanations have been presented in the introduction part.  For instance, it is not clear to me why the sample estimate of the ACT divergence and its gradient stay unbiased. 

As the new divergence requests two additional navigators for bidirectional distribution-to-distribution transport, the adversarial game gets more complex in theory. It is known that GAN models generally suffer from unstable training and mode collapse issues. I am afraid the higher complexity might make such issues more serious. While Fig.1, Fig.2 somehow shows the behaviors of the navigators, I suggest to further study whether the proposed method can overcome or avoid such issues. 

review_title: request deeper studies on the proposed method's properties

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a new transport-based divergence between distributions (CT) and a variant for empirical distributions (ACT). The new divergence is claimed to be more suitable for learning deep generative models than existing divergences like KL, JS (as in the vanilla GAN) and Wasserstein (as used in WGAN and its variants).  The proposed divergence mostly resembles, in my opinion, the Wasserstein divergence variant that uses the  Kantorovich–Rubinstein dual definition (which requires the learned function to be 1-Lipschitz). It seems that the main advantages of ACT over Wasserstein is that there is no constraint on the Lipschitz smoothness (which has to be enforced in WGAN by means of e.g. gradient clipping or gradient penalty), and the fact that ACT provides unbiased gradients that do not require the critic to reach an optimal point (as required in theory in GAN or WGAN).

The paper presents a potentially interesting and significant method, however, it was a bit hard to follow, making it difficult for me to asses the actual significance of the contribution.

Specifically, the distinction between $d(x,y)$ and $c(x,y)$ was not 100% clear to me. $d(x,y)$ seems to be part of the "navigator" $\pi(x | y)$ - an energy-based conditional probability and $c(x, y)$ is the point-to-point transport cost. For the CT to be low, it looks like $c(x,y)$ and $d(x,y)$ should be positively correlated. It was not clear, however (at least from the introduction), how $d$ and $c$ fit the generator-critic GAN scheme. Are they both part of the critic? Do the claims that the critic does not have to reach optimality for the gradients to be unbiased refer just to $c$ or to $d$ as well? I believe this is partially explained, but only towards the end of the paper (eq. 16). Perhaps a figure showing the trained elements in ACT compared to the trained elements in WGAN could help clarify the method.

Specific questions:
- Why are both the forward and backward CT needed? 
- Are the parameters of the forward and backward navigators shared? From eq. 16 it looks like they are. Does it make sense?
- In section 2.3, the authors claim that L2 distance in the original image domain is known not to work well for high-dimensional data that resided on a lower dimensional manifold, but what about L2 on some pre-trained feature-space e.g. perceptual distance [1]?
- The difference in principle between the proposed ACT and OT-GAN and MMD-GAN should be described.

To summarize:

pros:
- Interesting method, seems to be novel
- Potential significance (as an alternative loss for training deep generative models)

cons:
- Writing hard to follow
- Some open questions **update: the authors have answered most of my questions**

[1] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
review_title: A new divergence between probability distributions and its potential application in GANs (replacing Wasserstein)

Meta-review: The paper proposes a new measure of difference between two distributions using conditional transport. The paper considers an important problem.  However, some major concerns remain after the discussion among the reviewers. In particular, the paper focuses on the evaluation on a toy dataset. It is unclear whether the claim carries over to large real datasets. The presentation of the paper also needs substantial improvement.

Decision: Reject