Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to use conditional transport (CT) to measure the difference between two probability distributions. By combining both forward CT and backward CT, the proposed measure shows promising results for generative adversarial model training. 

main_review: After the discussion phase, I increase my overall score from 5 to 6. Thank the authors for their engagement. 

--
Overall, this paper is well-written and the empirical results seem promising. However, I have two concerns:

1) Does the advantage of this new measure come from combining forward and backward CTs? If so, can we look at the results while changing the value of $\rho$?
2) How does the choice of cost function c(x,y) impact the performance of this measure?  Is Euclidean distance used for the all the experiments in this paper? 

limitations_and_societal_impact: No obvious potential negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a new cost function to measure the difference between two probability distributions. The proposed cost function called the conditional transport (CT) is a mixture of two cost functions. The two cost functions of the mixture are derived by factorizing the joint probability density function in two reverse manners. A sample analogue of the CT is given, and it is shown that this sample analogue converges to its population version as the number of samples tends to infinity. Experiments are given to assess the performance of the CT and compare the performance of CT-based methods with that of existing methods.
main_review: Originality: The presented theory seems new. The proposed cost function, CT, comes from an interesting idea to factorize the joint probability density function through the chain rule and Bayes' theorem in two reverse manners. I am impressed with a comprehensive experimental study to demonstrate the performance of the CT and compare the performance of methods based on the CT with that of existing methods. Related works are adequately cited.

Quality: The submission seems technically sound in general. The claims are well supported by experimental results. Compared with great effort on the experiments, relatively little results are given to the theory of the CT. But Lemma 1 proves the convergence of the sample version of the proposed test to its expected version. The advantages of the proposed method are demonstrated through experiments. I do not find much discussion about the weaknesses of the proposed method in the paper.

Clarity: The paper is clearly written in general and well organized. The following are some minor comments on the presentation of the paper:

(i) p.2, l.71: The definition of $\mathbb{R}^H$ should be given.

(ii) p.3, l.86: $p_{\boldsymbol{\theta}}(\boldsymbol{y}) \ $  $\Longrightarrow$ $ \ p_{Y}(\boldsymbol{y})$?

(iii) p.4, l.127: What does "DGM" stand for?

(iv) p.4, equation (7): The definition of $\delta_{ \boldsymbol{x_i} } $ should be given. As it stands, the left-hand side of the equation says $p_{\hat{X}_N}$ is a function of $\boldsymbol{x}$, but the right-hand side of the equation does not involve $\boldsymbol{x}$.

Significance: The thorough experiments given in the paper suggest that the proposed method is useful in practice. Comparatively little work have been done on the theory of the proposed method. Although Lemma 1 is helpful, it would be better to have more theoretical results about the CT. For example, I think the selection of the weight $\rho$ of the cost function (12) would be a possible topic to discuss. If the true distribution is unknown, is it possible to select $\rho$ efficiently in practice?

limitations_and_societal_impact: I do not find much discussion about the the limitations of the proposed method. I am a little worried about whether the estimation of $\rho$ of the cost function (12) is computationally expensive or not. (I ask this because the selection of tuning parameters of a loss function requires a lot of computation in general.)

I do not find any negative societal impact of the work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new distance measure between two probability distributions by exploiting the chain rule and Bayes' theorem. The new distance measure is obtained by minimizing out an additional parameter from the conditional transport (CT) cost. The CT costs is a convex combination of a forward component and a backward one. The forward CT is the expected cost of moving a source data point to a target one, with their joint distribution defined by the product of the source probability density function (PDF) and a surrogate source-dependent conditional distribution, which is related to the target PDF via Bayes' theorem. The backward CT is defined by reversing the direction. The CT cost can be approximated with discrete samples and optimized with existing gradient descent-based methods. The new measure nicely balances the mode-covering property of forward CT and mode-seeking property of backward CT, showing strong robustness against mode collapse. Experiments on a wide variety of benchmark datasets show that substituting the default statistical distance of an existing generative adversarial network with CT consistently improves the performance.
main_review: The paper is very well written and a pleasure to read. The methods in Section 2 are clearly explained and technically sound. The review of related work is informative. PyTorch code is provided to reproduce the experimental results.

Minor comments:
Lines 109-110: It would be useful to briefly explain the concepts "mode-covering" and "mode-seeking".
Eq. (7): Please define the symbol $\delta$. Does $\delta_{x_i}$ denote the Dirac delta function? It should be written as $\delta_{x_i}(x)$ or $\delta(x-x_i)$.
Line 135: "Substituting .... in (2)" should be "Substituting .... in (1)".

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper defines conditional transport (CT), a new divergence between probability distributions (Equation 5), and reports experimental findings where this divergence serve as a good loss function for training generative models over natural images (Table 3, Figure 4 and so on). Among the tunable parameters of the divergence, there is one parameter that modulates between the extremes of mode seeking and mode covering. In real-data experiments, generative models fitted using CT as training loss have low Frechet inception distance (FID) and high inception score, which overall indicate high-quality generated images. 
main_review: Originality. CT uses ideas from optimal transport but possesses novel modifications. In the broadest characterization, Equation 15, the parameter rho aims to balance between mode seeking and mode covering behaviors. For the focus area of generative model training,  CT has additional parameters phi and eta. Eta parametrizes a neural network which transforms the source/target from the natural space into some feature space, while phi parametrizes another network that compares feature representations. CT, when used as training loss, very frequently improves generative modelling quality compared to previous choices of training loss, regardless of the neural network architecture. Each pair of methods in Table 3 correspond to the same architecture: the top rows are image quality metrics under existing training losses, while bottom rows are metrics for CT as training loss. 
	
Quality. The quality of the generated images in Table 3/Figure 4,5 justify the use of phi and eta in the CT definition (although I have a question about the experimental setup that I will say in the Clarity section). The comparisons in Table 3 appear thorough, as the baselines range from very high-quality (like StyleGAN2) to low-quality (like DCGAN). Figure 1 is another justification for the navigator phi. In this synthetic situation, phi governs the curvature of the CT loss as a function of the generator parameters: suitable phi seemingly makes the CT loss more convex in the generator parameters, hence easier to optimize (I also have a question about the setup/result of this experiment). Figure 2 justifies the use of rho in the CT definition. As expected, the extreme setting of rho = 0 correspond to mode seeking behavior while rho = 1 corresponds to mode covering. By picking rho in between, there is a balance between the two. 

Clarity. I have some questions about experimental setups and results. For Section 2.1, what does it mean for the navigator phi to be optimized? Up to this point, we have not talked about changing either rho or phi in definition of CT (Equation 5). Later on in section 2.3, the paper discusses  options like cooperative-trained versus adversarially-trained -- is this what it means in Section 2.1 too? If so, then as early as Section 2.1, it would be good to describe how phi is determined. Still in Section 2.1, for Figure 1, because the four panels on the right have different y-scaling, it's hard to see how the curvature of C(X,Y) change for different phi. Perhaps you can fix the same y-axis across the panels. Or plot the four curves on the same figure. For the "Adversarially-trained CT for natural images" paragraph in Section 4, what is the choice of rho? Perhaps you mention rho in the appendix, but since rho plays a big part in your methodology, and is mentioned in other experiments (like "Forward and backward analysis"), it would be good to know what rho is for these experiments as well. I have some comments about legibility. In general, the figure panels are too small, and it's difficult to see on a printed copy of the paper. Perhaps it's worth removing unnecessary panels, so that the panels chosen for display are easier to read. The panels in Figure 2 are very small and I can't make out what the axis labels/axis ticks say. The panels in Figure 3 are larger, but the axis ticks are not legible. For the panels in Figure 4, there are too many samples, each being too small.  My final comments are about word-choice/phrasing. Mini-max should be changed to min-max (minimax is another concept that is different). 

Significance. Since CT serves as a good as-is training criterion without requiring specialized deep generative model architecture, other researchers are likely to use CT in place of existing training criteria in their research, hence this paper is an important set of results. 
limitations_and_societal_impact: The authors have adequately addressed the limitations of their work. As for potential negative societal impact, for the general field of generative modelling, there is the danger of creating fake material (fake news, etc) that spreads mis-information. Although the paper is primarily concerned with natural images, it is good to be aware of the general dangers.
review_title: null

Meta-review: Three reviewers recommend an accept, one reviewer indicates a reject. The general sentiment is that the presented approach for comparing probability distributions with forward and backward CT is novel, and that the empirical results for generative adversarial model training are promising. However, there were some concerns as to the lack of theoretical results and inadequate discussion about the limitations of the proposed method. A revised paper should incorporate the revisions and clarifications brought up in the rebuttal. 
Decision: Accept (Poster)