Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributions
Authors: Huangjie Zheng
Mingyuan Zhou
Venue: NeurIPS 2021 Poster (2021)
Keywords: Bayesian methods
conditional distribution
generative models
mode-covering
mode-seeking

Abstract: To measure the difference between two probability distributions, referred to as the source and target, respectively, we exploit both the chain rule and Bayes' theorem to construct conditional transport (CT), which is constituted by both a forward component and a backward one. The forward CT is the expected cost of moving a source data point to a target one, with their joint distribution defined by the product of the source probability density function (PDF) and a source-dependent conditional distribution, which is related to the target PDF via Bayes' theorem. The backward CT is defined by reversing the direction. The CT cost can be approximated by replacing the source and target PDFs with their discrete empirical distributions supported on mini-batches, making it amenable to implicit distributions and stochastic gradient descent-based optimization. When applied to train a generative model, CT is shown to strike a good balance between mode-covering and mode-seeking behaviors and strongly resist mode collapse. On a wide variety of benchmark datasets for generative modeling, substituting the default statistical distance of an existing generative adversarial network with CT is shown to consistently improve the performance. PyTorch code is provided.


Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributions
Huangjie Zheng
Mingyuan Zhou
Department of Statistics & Data Science
McCombs School of Business
The University of Texas Austin Austin, TX 78712
The University Texas at Austin Austin, TX 78712
www.geateeenngeaaase...o. edu
mi wwggyuuan.zhouuncccmmmb..... utexas. eddu
Abstract
To measure the difference between two probability distributions. referred to as the source and target, respectively, we exploit both the chain rule and Bayes' theorem to construct conditional transport (CT), which is constituted by both a forward component and backward one. The forward CT is the expected cost of moving source data point target one, with their joint distribution defined by the product of the source probability density function (PDF) and a source- dependent conditional distribution, which is related to the target PDF via Bayes' theorem. The backward CT is defined by reversing the direction. The CT cost be approximated by replacing the source and target PDFs with their discrete empirical distributions supported on mini-batches, making it amenable to implicit distributions and stochastic gradient descent-base based optimization. When applied to train a generative model, CT is shown to strike good balance between mode- covering and mode-seeking behaviors and strongly resist mode collapse. On wide variety of benchmark datasets for generative modeling, substituting the default statistical distance of an existing generative adversarial network with CT shown to consistently improve the performance. PoTor code is provided.
Introduction
Measuring the difference between two probability distributions is fundamental problem in statistics and machine learning [1-3], A variety of statistical distances, such as the Kullback-Leibler (KL) divergence [4], Jensen-Shannon (JS) divergence [5], maximum mean discrepancy (MMD) [6], and Wasserstein distance [7], have been proposed quantify the difference. They have been widely used for generative modeling with different mode covering/secking behaviors [8 (8-13]. The KL divergence, directly related to both maximum likelihood estimation and variational inference [14-16], requires the two probability distributions to share the same support and is often inapplicable if either is an implicit distribution whose probability density function (PDF) is unknown [17-20]. Variational auto-encoders (VAEs) [8], the KI. divergence based deep generative models, are stable to train, but often exhibit mode-covering behaviors its generated data, producing blurred images. The JS divergence is directly related to the min-max loss of generative adversarial net (GAN) when the discriminator optimal [9], while the Wasserstein-1 distance is directly related to the min-max loss of Wasserstein GAN [11], whose critic is optimized under the 1-Lipschitz constraint. However, it is difficult to maintain a good balance between the updates of the generator and discriminator//citic. making (Wasserstein) GANs notoriously brittle train. MMD [6] is an RKHS- statistical distance behind MMD-GANs [10, 21, 22], which have also shown promising results in generative modeling when trained with min max loss. Different from VAEs, these GAN-based models often exhibit mode dropping and face the danger of mode collapse if not well tuned during the training.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this paper, we introduce conditional transport (CT) as new method to quantify the difference between two probability distributions, which will be referred as the source distribution px (a)) and target distribution PY (v). respectively. The construction of CT is motivated by the following observation: the difference between px (a=) and py (t) can be reflected by the expected difference of two dependent random variables æ and y, whose joint distribution <( y) is constrained by both px()) and py (y) in a certain way. Denoting x y) 2 0 as a cost function to measure the difference between points x and y, such as c(æ,, y) = I yl the expected difference is expressed as (u. [c(æ,, y)]. A basic way to constrain T T y) with both px and py (3) is to let π (2, y) = px px(x) (y), which means drawing ae and y independently from px (a:) and py py)),, respectively; this expected difference Exx (x) [e(ar, y) is closely related to the energy distance [23]. Another constraining method is to require both ydyy px (a:) and f IT T y)dx py (y), under which min, () [c(2;, y)]} becomes the Wasserstein distance [7. 24-26].
A key insight of this paper is that by exploiting the chain rule and Bayes' theorem, there exist two additional ways to constrain π (x, y) with both px (a) and py (y): 1) A forward CT that can be viewed as moving the source to target distribution; 2) A backward CT that reverses the direction. Our intuition that given a source (target) point, is more likely to be moved to a target (source) point closer to it. More specifically, if the target distribution does not provide good coverage of the source density, then there will exist source data points that lie in low-density regions of the target, making the expected cost of the forward CT high. Therefore, we expect that minimizing the forward CT will encourage the target distribution to exhibit a mode-covering behavior with respect to (w.rt.) the source PDF Reversing the direction, we expect that minimizing the backward CT will encourage the target distribution to exhibit a mode-seeking behavior w.F.t. the source PDF Minimizing the combination of both is expected to strike good balance between these two distinct behaviors.
To demonstrate the use of CT, we apply it to train implicit (or explicit) distributions model both ID and 2D toy data, MNIST digits, and natural images. The implicit distribution is defined by deep generative model (DGM) that is simple to sample from. We provide empirical evidence to show how to control the mode-covering versus mode- seeking behaviors by adjusting the ratio of the forward CT versus backward CT. To train a DGM for natural images, we focus on adapting existing GANs, with minimal changes to their settings except for substituting the statistical distances in their loss functions with CT. We leave tailoring the network architectures and settings to CT for future study, Modifying the loss functions of various existing DGMs with CT, our experiments show consistent improvements in not only quantitative performance and generation quality, but also learning stability. Our code is available at https: //t///ggtttt..//// . com com JegZheng/ CT -ppytorchh
2 Chain rule and Bayes' theorem based conditional transport
Exploiting the chain rule and Bayes' theorem, we can constrain < y) with both px (a)) and pv (v) in two different ways, leading to the forward CT and backward CT, respectively. To define the forward CT, we use the chain rule to factorize the joint distribution as
π(. y) = px mY (3/ x))
where Ty (y/ ar) is a conditional distribution of y given æ. This construction ensures J (2x, yy px (x) but not f (x y)dæ= (y)) Denote dæ (h1, f22 E IR
function parameterized by Φ. which measures the difference between two vectors h1 h2 R H of dimension H. While allowing I (xx y) # py(Y). to appropriately constraint (अ. y) by py (B), we treat py (3) as the prior distribution, view e- (1) as an unnormalized likelihood term, and follow Bayes' theorem to define
(1) where Q(z) is a normalization term that ensures I T (y x xd = 1. We refer to TT (y x)) as the forward "navigator," which specifies how likely given æ will be mapped to a target point ~ pr (y)) We now define the cost of the forward CT
( x)) = e ( pv (y) (a)), Q(x)= == f ee (y)dy,
C(X - Y) = Ex-px (m) By By (- a) [c( y)).
In the forward CT, we expect large c(æt, y) to typically co-occur with small TT (y (z) as long as py (1) provides good coverage of the density of a. Thus minimizing the forward CT cost expected to encourage py (3) to exhibit mode-covering behavior w.n.t. px (z). Such kind of behavior is also

---

expected when minimizing the forward KL divergence as (L(px = Exppx [In calls for py (x) whenever Px (a) 0.
px (y
which
Reversing the direction, we construct the backward CT, where the joint is factorized as T (x. y) py (y) (a: y) and the backward navigator is defined
=
(x (e y) e (o px (Q)))(y), Q(y)== doll.))))))) px (xxxx(())))))) (z))dx.
(3)
This ensures f ((x,, ry)da py (y): while allowing (ac, y)dy F px (ar), it constrains (ax, y) by treating Px (x) as the prior to construct (x y). The backward CT cost is now defined as
C(X + Y) Bympy (x) (w) (
w))(( y)l.
In the backward CT, we expect large c(æc, y) typically co-oceur with small ι xx y) as long as Px (a:) has good coverage of the density of Thus minimizing the backward CT cost is expected to encourage py (y) to exhibit a mode-seeking behavior w.r.t. px (x). Such kind of behavior is also expected when minimizing the reverse KL divergence as (L [px) = Exwpy [In BY
px ). which allows py (x) 0 when px (z) > 0 and it is fine for py to just fit some portion of px.
In comparison to the forward and revers KLs, the proposed forward and backward CT are more broadly applicable as they don't It require px and py to share the same distribution support and have analytic PDFs. For the cases where the KLs can evaluated, we introduce
D(X, Y) = (I-(xx|lpr) KI(pr|lpx)
as a formal way to quantify the mode seekin and mode-covering behavior of py w.r.t px, with D(X,Y) > 0 implying mode seeking and with D(X, Y) < 0 implying mode covering.
Combining both the forward and backward CTs, now define the CT cost as
C,(X, = pc + Y) + +1 p)0--))(((---- t Y),
(5)
where o [0, 1] parameter that can be adjusted to encourage py (y) exhibit w.r.. px (æ) mode-seeking (p- 0), mode-covering (p 1), or balance of two distinct behaviors (p E (0. 1)). By definition we have C,( 2 O, where the equality can be achieved when px = Py and the navigator parameter & is optimized such that e (x.()))) is equal to one if and only if a = y and zero otherwise. We also have Cp=0.5(X,) Y) (Y, We fix p = 0.5 unless specified otherwise.
2.1 Conjugacy based analytic conditional distributions
Estimating the forward and backward CTs involves Ty (y (y/ and x (xc y). respectively. Both conditional distributions, however, are generally intractable to evaluate and sample from, unless Px px(x:) (ax) and py (y) are conjugate priors for likelihoods proportional to e edd(z.v) i.e., x y) and ( æ:) are in the same probability distribution family as px (æ) and py (yy) respectively. For example, if d(xx3) y) = x y1ī2 and both px (x) and py (y) are multivariate normal distributions, then both x (æ y) and T (y x)) will follow multivariate normal distributions.
To be more specific, we provide a univariate normal based example, with x, y, ó, and
= -N py (y NN(, e.), (s = (x - /)2/((22) c(x.y) (r-
(6)
= sinh which is positive when 0 < 0, implying mode-secking and negative when 0 > 0. implying mode-covering As shown in Appendix C, we have analytic forms of the forward and backward navigators as
Here we have D) = KL[V(O, (0) KOL]](KLL,(C e (O, e
Ty (v/ = (o - d)x o0 - ))e"), (x (v) = -N(a (o)),
where (a) 1/(1 a7) denotes the sigmoid function, and forward and backward CT costs as C(X + Y) á (e a 0)), C(X + Y) ( ()))
As proof of concept, we illustrate the optimization under CT using the above example, for which 0 = 0 is the optimal solution that makes px py. Thus when applying gradient descent to minimize the CT cost Cp (X, Y). we expect the generator parameter - O with proper learning dynamic, as long the learning of the navigator parameter ø is appropriately controlled. This is confirmed by
3

---

CIX, Y) (e ===+=1)
CDX M) (e 2e=
CIX, Y) (e (e-
CIX. Y1 (0=
0X--1
CX--1
10000
cu--1)
CSE+
C
000
220g
Iterationn
Figure/ 1: Illustration of minimizing the CT cost Co,o between (O0, 1) and (V(0, Left: Evolution of CT cost, its parameters, and forward and backward costs; Right: 4 CT cost curves against @ e is being optimized a small value to jointly show the optimized @ provides better learning dynamic for the learning 0.
Fig. 1, which shows that as the navigator & gets optimized by minimizing CT cost, it is more obvious that 6 will minimize the CT cost at zero. This suggests that the navigator parameter & mainly plays the role in assisting the learning of t. The right four subplots describe the log-scale curves of forward cost, backward cost and bi-directional CT costs w.r.t. as & gets optimized to four different values. It is worth noting that the forward cost is minimized at e 1. which implies a mode-covering behavior, and the backward cost minimized at e -> 0, which implies a mode- seeking behavior, while the bi-directional cost is minimized at around the optimal solution e" 1; the forward CT cost exhibits a flattened curve on the right hand side of its minimum, adding to which the backward CT cost not only moves that minimum left, making it closer to 0 = O, but also raises the whole curve the right hand side, making the optimum of & become easier to reach via gradient descent.
To apply CT in general setting where the analytical forms of the distributions are unknown, there no conjugacy, or we only have access to random samples from the distributions, below we show we can approximate the CT cost by replacing both px (x) and py(y) with their corresponding discrete empirical distributions supported on mini-batches. Minimizing this approximate CT cost, amenable to mini-batch SGD based optimization, is found to be effective in driving the target (model) distribution py towards the source (data) distribution px. with the ability to control the mode-seeking and mode-covering behaviors of py w.E.f. px.
2.2 Approximate CT given empirical samples
Below we use generative modeling as an example show how to apply the CT cost in a general setting that only requires access to random samples of both x and y. Denote as a data taking its value in Y In practice, we observe finite set X' { consisting of | data samples assumed to be iid drawn from Px (æ). Given X, the usual task is to learn a distribution to approximate p(c), where Ge is generator that transforms noise e ~ p(c) via deep neural network parameterized by generate random sample y E V.V. While the PDF of the generator, denoted as py (y: 0), is often intractable to evaluate, it is straightforward to draw y ~ py (y: 0) with Go-
Px (x), for which we consider a deep generative model (DGM) defined as y = Go
While knowing neither pxx (a:) nor py (y; f), we can obtain discrete empirical distributions Pxy, and py, supported on mini-batches #i.N and V1:as. as defined below, to guide the optimization of Go in an iterative manner. With N random observations sampled without replacement from X, we define
Px* (w) + E. 8(x =x;,, ( NN C X
as an empirical distribution for æ. Similarly, with M random samples of the generator, we define
Pyn, (4) = M Em 8(y - y,),, yj = Go(e)), 150 p(c)
(8)
Substituting py (y: 0) in (2) with py,, (v). the continuous forward navigator becomes discrete one
Ry ()) Σe fM (3, xx k) AM (v3 xx Φ)
-6mmm))) My -
(9)
Thus given py,," the cost of forward CT can be approximated as
Cx + Yns) = M1.ar 4d pv (w:0) Ex-px(() (x)
Ene clæx, u, F Le, b))
(10)

---

which can be interpreted as the expected cost of following the forward navigator to stochastically transport a random source point to one of the M randomly instantiated "anchors" of the target distribution. Similar to previous analysis, we expect this approximate forward CT to stay small as long as py (y; f) exhibits a mode covering behavior w.r.t px (x))
Similarly, we can approximate the backward navigator and CT cost as
(x y) - Σi= (N; y, 6) (N; y, Φ)
Cx. + Y) = aisa 22 px (a))
E By- (v:))
EN cl y) ( y の)
(1I)
Similar to previous analysis, we expect this approximate backward CT to stay small as long as Py (y; 0) exhibits mode-seeking behavior w.r.f. px (z)).
Combining (10) and (11), we define the approximate CT cost
C. (x YMr) o YMu + p)Co,o(XN -Y), an unbiased sample estimate of which, given mini- mini-batches V1:N and yI can be expressed as
(12)
L p v1.a1) = E ΣE c(2;, Uj) fas Iz. Φ)) + 1-P
M ax yj Φ))
E M U,)
& E -o 2,, + M EN- dalapiis7
dale),))))
dain,
(13)
Lemma 1. Approximate CT in (12) is asymptotic as lim oo Co. (&N, Pas) = Co.o,p(X,1
2.3
Cooperatively-trained or adversarially-traing feature encoder
To apply CT for generative modeling of high-dimensional data, such as natural images, we need to define an appropriate cost function c(2r,) to measure the difference between two random points. A naive choice is some distance between their raw feature vectors, such as c(x, y) x y112. which, however, is known often poorly reflect the difference between high-dimensional data residing on low-dimensional manifolds. For this reason, with cosine similarity h h2
[27] as cos (os(h,, == v /hj hi / we further introduce a feature encoder Tn)(), parameterized by n7 to help redefine the point-to-pont point cost and both navigators as
Cy (2, 3)
cos( (T, (z),, Ty))),, de
(x) (2) (T)))) [T,
(14)
To apply the CT cost to train DGM, we find that the feature encoder (-) can be learned in two different ways: Cooperativell-trrine trained: Training them cooperatively by alternating between two different losses: training the generator under fixed (-) with the CT loss, and training (.) under fixed generator with different loss, such as the GAN discriminator loss, WGAN critic loss, and MMD-GAN [10] critic loss. 2) eegesarialll-triiedd:: Viewing the feature encoder
critic and
training to maximize the CT cost, by not only inflating the point-to-poine cost, but also distorting the feature space used to construct the forward and backward navigators' conditional distributions.
To be more specific, below we present the details for the adversarial way to train Ty, Given training data X, to train the generator Gg. forward navigator T& (y x)), backward navigator T (x yy) and encoder Typ we view the encoder as a critic and propose solve a min-max problem as
min max E 4.0 "
#I.. CX, CHAM sd p(e) (......mm (IINN, (G)(E)
(15)
where Lo.0.pm is defined the same as in (13), except that we replace c(2;; U,) and (. with their corresponding ones shown in (14) and use reparameterizztiio in (8) to draw VI.M == (G)( With SGD, we update & and 0 using V Vo..LL....nn (Go(e;) J and, if the feature encoder is adversarially-train -rainnnd,,, update 77 using - -V, V,,Co. LI.N, (GG(E)))
We find by experiments that both ways to learn the encoder work well, with the adversarial one generally providing better performance. It worth noting that in (Wasserstein) GANs, while the adversarially-trainee discriminator/critic plays similar role as feature encoder, the learning dynamics between the discriminator//cittc and generator need to be carefully tuned to maintain training stability and prevent trivial solutions (e-g., mode collapse). By contrast, the feature encoder of the CT cost based DGM can be stably trained two different ways. Its update does not need to be well synchronized with the generator and can be stopped at any time of the training.
5

---

3 Related work
In practice, variational auto-encoders [8], the KL divergence based deep generative models, are stable to train, but often exhibit mode-covering behaviors and generate blurred images [28-32]. By contrast, both GANs and Wasserstein GANs can generate photo-realistic images, but they often suffer from stability and mode collapse issues, requiring the update of the discriminator/criti to be well synchronized with that of the generator. This paper introduces conditional transport (CT) as a new method to quantify the difference between two probability distributions. Deep generative models trained under CT not only allow the balance between mode-covering and mode-seeking behaviors to be adjusted, but also allow the encoder to be pretrained or frozen at any time during cooperative/adversriill training.
As the JS divergence requires the two distributions to have the same support, the Wasserstein distance often considered as more appealing for generative modeling as it allows the two distributions to have non-overlapping support [24-26]. However, while GANs and Wasserstein GANs in theory are connected to the JS divergence and Wasserstein distance, respectively, several recent works show that they should not be naively understood as the minimizers of their corresponding statistical distances, and the role played by their min-max max training dynamics should not be overlooked [33-35]. In particular, Fedus et al. [34] show that even when the gradient of the JS divergence does not exist and hence GANs are predicted to fail from the perspective of divergence minimization, the discriminator able to provide useful learning signal. Stanezuk et al. [35] show that the dual form based Wasserstein GAN loss does not provide meaningful approximation of the Wasserstein distance; while primal form based methods could better approximate the true Wasserstein distance, they in general clearly underperform Wasserstein GANs in terms of the generation quality for high-dimensional data, such as natural images, and require inner loop to compute the transport plan for each mini-batei leading to high computational cost [12. 35-38J. See previous works for discussions on the approximation error and gradient bias when estimating the Wasserstein distance with mini-batches [10, 23, 39, 40].
MMD-GAN [10, 21, 22] that calculates the MMD statistics in the latent space of feature encoder is the most similar to the CT cost in terms of the actual loss function used for optimization. In particular, both the MMD-GAN loss and CT loss, given mini-batches F1:N and y1:M- involve computing the differences of all NM pairs (æ;; y,). Different from MMD-GAN, there is no need in CT to choose a kernel and tune its parameters. We provide below an ablation study to evaluate both 1) MMD generator + CT encoder and 2) MMD encoder + CT generator, which shows 1) performs par with MMD, while 2) performs clearly better than MMD and on par with CT.
Experimental results
Forward and backward analysis: To empirically verify our previous analysis of the mode cov- ering (seeking) behavior of the forward (backward) CT, we train a DGM with (12) and show the corresponding interpolation weight from the forward CT cost to the backward one, which means СTр reduces from forward CT (p- = 1), to the CT (12) (p E (0, 1)), and to backward CT (p = 0). We consider the squared Fuclidean (i.e. (3) distance to define both cost c(x, y) = || - y and dø (a., y) = (z) - To (1) where Ts denotes neural network parameterized by Φ. We consider ID example of bimodal Gaussian mixture Px (x) = 1xx -5, 1) + 5.((x; 2, 1) and a 2D example of 8-modal Gaussian mixture with equal component weight as Gulrajani et al. [41]. We use an empirical sample set X. consisting of |X 5, 000 samples from both ID and 2D cases, and illustrate in Fig. 2 the KDE of 5000 generated samples y, = Go (E,E) after 5000 training epochs. For the ID case, we take 200 grids in [-10, 10) to approximate the empirical distribution of px and py, and report the corresponding forward KL (KL[px (px D. reverse KL (KL ||xx 1), and their difference D Y)= = KL |py] - KL [py below each corresponding sub-figure in Fig. 2.
Comparing the results of different p in Fig. 2, it suggests that minimizing the forward CT cost only encourages the generator to exhibit cost only encourages mode--eekingg behaviors. Combining both costs provides a contollottol balance between mode covering and seeking, leading to satisfactory fitting performance, as shown in Columns 2-4. Note that for a fair comparison, we stop the fitting at the same iteration; in practice, we find if training with more iterations, both p= 0.75 and p = 0.25 can achieve comparable results as p = 0.5 in this example. Allowing the mode covering and seeking behaviors to be controlled by adjusting p is an attractive property of CT,,
behaviors, while minimizing the backward CT

---

p=1
p=0.75
p=0.5
p=
p=
015
614
-1.
-15
KL(F)
KL())
D
(L((F)
(L
(L((F)
KL(r)
D
(L)
KL())
D
KL(
KL(r)
1.97
3.57
-1.60
0.23
0.40
-0.17
0.12
0.19
-0.07
0.94
0.80
0.14
10.52
3.52
7,00
Figure 2: Forward and backward analysis: (top) Fitting ID bi-modal Gaussian. Quantitative results of estimated
forward KL (KL[áx p D. reverse KL [XLxx)]lpx]) and the difference between the forward and reverse KL (D=KLIix lyy |-LL ( D) shown below each sub-figure. (bottom) 2D 8- 8-Gaussian mixture by
interpolating between the forward CT (p and backward CT (p- 0).
GAN
WGAN-GP
SWD
CT (ours)
True
Y=
V=
Figure 3: Experiments on the resistance to model collapse: Comparison of the generation quality on 8-Gaussian mixture data: one of the modes has weight y and the rest modes have equal weight 1-7
Resistance to mode collapse: We continue to use 8- A-Gaussian mixture to empirically evaluate how well DGM resists mode collapse. Unlike the data in Fig. 2, where 8 modes are equally weighted, here the mode at the left lower corner is set to have weight 7, while the other modes are set to have the same weight of 1=1 We set X' with 5000 samples and the mini-batch size as N = 100. When 7 is lowered to 0.05, its corresponding mode is shown to be missed by GAN, WGAN, and SWD-based DGM, while well kept by the CT-based DGM. As explanation, GANs are known to be susceptible to mode collapse; WGAN and SWD-base DGMs are sensitive to the mini-batch size, when " equals to small value, the samples from this mode will appear in the mini-batches less frequently than those from any other mode, amplifying their missing mode problem. Similarly, when y is increased to 0.5, the other modes are likely to be missed by the baseline DGMs, while the CT-based DGM does not miss any modes. The resistance of CT to mode dropping can be attributed to its forward component's 's mode-covering property. The backward's 's mode-seeking property further helps distinguish the density of each mode component to avoid making components of equal weight.
CT for 2D toy data and robustness in adversarial feature extraction: To test CT with more general cases, we further conduct experiments on representative 2D datasets for generative modeling evaluation [41]: 8- 8-Gaussian mixture, Swiss Roll, Half Moons, and 25-Gaussian mixture. We apply the vanilla GAN [9] and Wasserstein GAN with gradient penalty (WGAN-GP) [41] as two representatives of min-max DGMs that require solving min-max loss. We then apply the generators trained under the sliced Wasserstein distance (SWD) [42] and CT cost as two representatives of min-max-fie free DGMs. Moreover, include CT with an adversarial feature encoder trained with (14) to test the robustness of adversary and compare with the baselines in solving the min-max loss.
On each 2D data, we train these DGMs as one would normally do during the first 5k epochs. We then only train the generator and freeze all the other learnable model parameters, which means we freeze the discriminator in GAN, critic in WGAN, the navigator parameter & of the CT cost, and both
7

---

(a) Adv CT.
(b) GAN.
(c) LD + CT.
(d) SWD.
(c) Slicing + CT.
Figure 4: Ablation of fitting results by minimizing CT in different spaces: (a) CT calculated with adversarially trained encoder. (b-c) GAN vs. CT with feature space cooperatively trained with discriminator loss. (d-f) Sliced Wasserstein distance and CT the sliced space.
Table 1:
FID comparison with differ- cooperative training CIFAR-10
Table 2: FID Comparison with using MMD (Rational quadratic ker- nel/distance kernel) and CT loss in training critic/generator on CIFAR- (lower FID is preferred).
(lower FID is preferred).
Critic space
FID 1
Generator loss
Generator loss
MMD-rq
MMD-dist
Discriminator
29.7
MMD
CT
MMD
CT
Slicing
32.4
Critic
MMD
39.9
24.1
Critic
MMD
40.3
28.8
Adversarial CT
22.1
loss
CT
41.4
23.9
loss
CT
30.9
29.4
(Ø, 17) of CT with an adversarial feature encoder. for another 5 epochs. Figs. 7-10 in Appendix E.1 illustrate this training process on each dataset, where for both min-max baseline DGMs. the models collapse after the first 5k epochs, while the training for SWD remains stable and that for CT continues to improve. Compared to SWD, our method covers all data density modes and moves the generator much closer to the true data density. Notably, for CT with an adversarially trained feature encoder, although it has switched from solving min-max loss to freezing the feature encoder after 5k epochs, the frozen feature encoder continues to guide the DGM to finish the training in the last 5k epochs, which shows the robustness of the CT cost.
A Ablation of cooperatively-traine the adversarially trained feature encoder could provide valid feature space for CT cost, we further study the performance of the encoders cooperatively trained with other losses. Here we leverage, as two alternatives, the space of an encoder trained with the discriminator loss in GANs and the empirical Wasserstein distance in sliced ID spaces [43]. We test these settings on both 8-Gaussian, as shown Fig. 4, and CIFAR-10 data, as shown in Table 1. It is confirmed these encoders are able to cooperatively work with CT, in general producing less appealing results with those trained by maximizing CT. From this view, although CT is able to provide guidance for the generators in the feature space learned with various options, maximizing CT still preferred to ensure the efficiency. Moreover, observed in Figs. 4b-4e, CT clearly improves the fitting with sliced Wasserstein distance. To explain why CT helps improve in the sliced space, we further provide a toy example in ID to study the properties of CT and empirical Wasserstein distance in Appendix E.3.
and irversarially-traine CT: As previous experiments show
Ablation of MMD and CT: As MMD also compares the pair-wise sample relations in a mini-batch, we study if MMD and CT can benefit each other. The feature space of MMD-GAN can be considered as k., where is the rational quadratic distance kernel in Binkowski the combinations of MMD/CT as the generator/encoder criterion to train DGMs. On CIFAR- 10, shown in Table combining MMD and CT generally has improvement over MMD alone in FID. It is interesting to notice that for MMD-GAN, learning its generator with the CT cost shows more obvious improvement than learning feature encoder with the CT cost. We speculate the estimation of MMD relies on a supremum of its witness function, which needs to be maximized w.r./ Tr k and cannot be guaranteed by maximizing CT w.r. Tnp In the case of MMD-dist. using CT for witness function updates shows more clear improvement, probably because CT has similar form MMD when using the distance kernel. From this view, CT and MMD are naturally able to be combined to compare the distributional difference with pair-wise sample relations. Different from MMD, CT does not involve the choice of kernel and its navigators assist to improve the comparison efficiency. Below show more image datasets, CT is compatible with many existing models, and achieve good results to show improvements on a variety of data with different scale.
[10]. Here we evaluate
Adversarially-traine CT for natural images: We conduct a variety of experiments on natural images to evaluate the performance and reveal the properties of DGMs optimized under the CT cost. We consider three widely-used image datasets, including CIFAR-10 [44], CelebA [45], and
8

---

Table Results CT with different deep generative models on CIFAR-IO, CelebA and LSUN. Base model results are quoted from corresponding paper or github page.
Fréchet Inception Distance (FID !)
Inception Score (1)
Method
CIFAR-10 10
CelebA
LSUN-bedroom
CIFAR-10
DCGAN [49]
30.2:10.9
$2.5:22.2
61.7::2.9
6.2.0.11
CT-DCGAN
22.1+1.11
29,4+2,0
32,6+22.5
7.5+0.1
SWG [42]
33.7++1.5
21,9+22.0
67.9+2.7
CT-SWG
25.9 0.9
18.8 = 1.2
39.0 + 2.1
6.9 + 0.1
MMD-GAN [10]
39,911..
20.6110.3
32.0
6.5
CT-MMD-GAN
23.9 + 0.4
13.8 0.4
38.3 + 0.3
7.4 + 0.1
SNGAN [50]
21.5#1.3
21.7=1155
31.1+2.1
8.2+
CT-SNGAN
17.2.1.00
9.2.1..0
16.8
8.8
StyleGAN2 [51]
5.8
5.2
2.9
10.0
CT-SttleGAN2
2.9 + 0.5
4.0 = 0.7
6.3 + 0.2
10.1 = 0.1
Figure 5: Generated samples of the deep generative model that adopts the backbone of SNGAN but is optimized with the CT cost on CIFAR-10, 10, CelebA, and
Sec Appendix E for more results.
LSUN- BUUN-bedroom/ [46] for general evaluation, as well as CelebA-HC - [47]. FFHQ [48] for evaluation in high-resolution. We compare the results of DGMs optimized with the CT cost against DGMs trained with their original criterion including DCGAN [49], Sliced Wasserstein Generative model (SWG) [42], MMD- MAD [10]. SNGAN [50], and StyleGAN2 [51]. For fair comparison, we leverage the best configurations reported in their corresponding paper or Github page. The detailed setups can be found in Appendix D. For evaluation metric, we consider the commonly used Fréchet inception distance (FID, lower is preferred) [52] on all datasets and Inception Score (IS, higher is preferred) [53] on CIFAR-10. 10. Both FID and IS are calculated using pre-trained inception model [54].
The summary of FID and IS on previously mentioned model is reported in Table 3. We observe that trained with CT cost, all the models have improvements with different margin in most cases, suggesting that CT is compatible with standard GANs, SWG, MMD-GANs, WGANs and generally helps improve generation quality, especially for data with richer modalities like CIFAR- 10. CT is also compatible with advanced model architecture like StyleGAN2, confirming that a better feature space could make CT more efficient to guide the generator and produce better results.
The qualitative results shown Fig. are consistent with quantitative results Table To ad- ditionally show how CT works for more complex generation tasks, we show in Fig. 6 example higher-resolution images generated by CT-SNGAN on LSUN bedroom (128x128) and CelebA-HO (256x256), as well as images generated by CT-StyleGAN2 on LSUN bedroom (256x256), FFHQ (256x256), and FFHQ (1224x1024).
On the choice of p for natural images: In previous experiments, we fix 0.5 by default when we prefer neither mode-covering nor mode--eekingg We further tune p as an additional ablation study on CIFAR-I0 dataset with both the CT + DCGAN backbone and CT + SNGAN backbone to see its affects in terms of certain metrics, such the FID score. The results
Table 4: FID of generation results CIFAR- 10, trained with different p.
P
1
0.75
0.5
0.25
0
CT-DCGAN CT-SNGAN
25.1 22.1
22.1
21.4
72.1
23.2
17.5
17.2
17.2
33.2

---

(a) LSUN Bedroomí 128x
(b) CelebA HQ(256x266)
(c) LSUN Bedroom((26x255)
(d) FFHQ 256x25661024xX0222
Figure 6: Generation results in higher-resolution cases, with SNGAN and StyleGAN2 architecture. Top:LSUN- Bedroom (128x and CelebA-HQ (256x256), done with CT-SNGAN. Bottom: LSUN-Bedroom (256x256) FFHQ (256x256/102xx 1024), done with CT-StyleGAN2.
shown in Table 4 suggest that CT is not sensitive to the choice of p as long as 0 < p 1, and the FID) score could be further improved if we choose smaller p to bias towards mode-seeking.
5 Conclusion
We propose conditional transport (CT) as a new criterion to quantify the difference between two probability distributions, via the use of both forward and backward conditional distributions. The forward and backward expected cost are respectively with respect to a source-dependent and target- dependent conditional distribution defined via Bayes' theorem. The CT cost can be approximated with discrete samples and optimized with existing stochastic gradient descent- based methods. Moreover, the forward and backward CT possess mode-covering and mode-seeking properties, respectively. By combining them, CT nicely incorporates and balances these two properties, showing robustness in resisting mode collapse. On complex and high-dimensional data, CT able to be caleulated and stably guide the generative models in a valid feature space, which can be learned by adversarially maximizing CT or cooperatively deploying existing methods. On various benchmark datasets for deep generative modeling, we successfully train advanced models with CT. Our results consistently show improvement over the original ones, justifying the effectiveness of the proposed CT loss.
Discussion: Note CT brings consistent improvement to these DGMs by neither improving their network architectures nor gradient regularization. Thus it has great potential work in conjunction with other state-of-the of-the-art art architectures or methods, such as BigGAN [55], self-attention GANs [56]. partition- partition-guiddd GANs [57], multimodal-DGiMs [58], BigB [59], self-supervised learning [60], and data augmentation [61-63], which we leave for future study. As the paper is primarily focused on constructing and validating a new approach to quantify the difference between two probability distributions, have focused demonstrating the efficacy and interesting properties of the proposed CT on toy data and benchmark image data. We have focused on these previously mentioned models as the representatives in GAN, MMD-GAN, WGAN under CT, and we leave to future work using the CT to optimize more choices of DGMs, such as VAE-based models [8] and neural-SDE [64].
Acknowledgments
The authors acknowledge the support of NSF 181699,,,,, the APX 2019 project sponsored by the Office of the Vice President for Research at The University of Texas at Austin, the support of gift fund from ByteDance Inc., and the Texas Advanced Computing Center (TACC) for providing HPC resources that have contributed to the research results reported within this paper.
10

---

References
families, and [1] Thomas M Cover. Elements of Information Theory. John Wiley Sons, 1999.
[2] Christopher M Bishop. Puttern Recognition and Machine Learning. springer, 2006.
[3] Kevin P Murphy, Machine Learning: A Probabilistic Perspective. MIT Press, 2012.
[4] Solomon Kullback and Richard A Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79-86, 1951.
[5] Jianhua Lin. Divergence measures based on Shannon entropy. IEEE Transactions on Information theory, 37(1):145-151, 1991.
[6] Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the wasssampe-ppbbleee Advances neural information processing systems, 19:513-522 2006.
[7] Leonid V Kantorovich. On the translocation of masses. Journal of Mathematical Sciences, 133 (4):1381-188,, 2006.
[8] Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint 1312v: 14, 2013.
(9] Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672-2680, 2014.
[10] Mikolaj Binkowski, Dougal Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference Learning Representations, 2018. URL https:
/ /oppnneviee....om
[11] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 214-223, 2017.
[12] Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with Sinkhorn divergences. In International Conference on Artificial Intelligence and Statistics, pages 1608- 1617, 2018.
[13] Yogesh Balaji, Hamed Hassani, Rama Chellappa, and Soheil Feizi. Entropic GANs meet VAEs: statistical approach to compute sample likelihoods in GANs. In International Conference on Machine Learning, pages 414 423, 2019.
[14] Martin Wainwright and Michael Irwin Jordan. Graphical models, familiis,,, variational inference. Now Publishers Inc, 2008.
[15] Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research, 14(1):1003-1347, 2013.
[16] David M Blei, Alp Kucukelbir, and Jon McAuliffe, Variational inference: review for statisticians. Journal of the American statistical Association, 11(5118):559-8777 2017.
[17] Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv preprint arXiv::610.03448 2016.
[18] Ferenc Huszár. Variational inference using implicit distributions. urXiv:1702.08233 2017.
arXiv preprint
[19] Dustin Tran, Rajesh Ranganath, and David Blei. Hierarchical implicit models and likelihood- free variational inference. In Advances in Neural Information Processing Systems, pages 5523-5533, 2017.
[20] Mingzhang Yin and Mingyuan Zhou. Semi-implicit variational inference. In International Conference Machine Learning, pages 5660-5669, 2018.

---

[21] Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In Interna- tional Conference on Machine Learning, pages 1718-1727, 2015.
[22] Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. MMD GAN: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems, pages 2203-220 2213, 2017.
[23] Marc G Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The Cramer distance solution to biased Wasserstein gradients. arXiv preprint
2017.
[24] Cédric Villani. Optimal Transport: Old and New, volume 338. Springer Science & Business Media, 2008.
[25] Filippo Santambrogio. Optimal Transport for Applied Mathematicians: Calculus of Variations, PDEs, and Modeling, volume 87. Birkhäuser, 2015.
[26] Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 11 1(5-6))355-6600 607, 2019.
[27] Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving GANs using optimal transport. arXiv preprint arXiv: 1803.0550 2018.
[28] Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Iya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. International Conference on Learning Representation, 2017.
[29] Shengjia Zhao, Jiaming Song, and Stefano Ermon. Info VAE: Information maximizing varia- tional autoencoders. 1706v:1706.000 022622, 2017.
[30] Huangjie Zheng, Jiangchao Yao, Ya Zhang, and Ivor W Tsang. Degeneration in vae: in the light of fisher information loss. arXiv preprint 18Xiv:/802.0660 2018.
[31] Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy, Fixing a broken elbo. In International Conference on Machine Learning, pages 159-168, 2018.
[32] Huangjie Zheng. Jiangchao Yao, Ya Zhang, Ivor W Tsang, and Jia Wang. Understanding vaes in fisher-shannon plane. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 33, pages 5917-5924, 2019.
[33] Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On convergence and stability of GANs. arXiv preprint arXiv: / 170.007215 2017.
[34] William Fedus, Mihaela Rosca, Balaji Lakshminarayanan, Andrew M Dai, Shakir Mohamed, and lan Goodfellow. Many paths to equilibrium: GANs do not need to decrease a divergence at every step. In International Conference on Learning Representutions, 2018
[35] Jan Stanezuk, Christian Etmann, Lisa Maria Kreusser, and Carola-Bibiane Schonlieb. Wasser- stein GANs work because they fail (to approximate the Wasserstein distance). arXiv preprint arXiv:2103.01678 2021.
[36] Akihiro Iohara, Takahito Ogawa, and Toshiyuki Tanaka. Generative model based on minimizing exact empirical wasserstein distance, 2019. URL https:/ www.:/eetffooul/ i BJgTZ3C5FX.
[37] Anton Mallasto, Guido Montúfar, and Augusto Gerolin. How well do WGANs estimate the Wasserstein metric? arXiv preprint arXiv: 1910.0385, 2019.
[38] Thomas Pinetz, Daniel Soukup, and Thomas Pock. On the estimation of the Wasserstein distance in generative models. In German Conference on Pattern Recognition, pages 156-170. Springer, 2019.
[39] Leon Bottou, Martin Arjovsky, David Lopez-Paz, and Maxime Oquab. Geometrical insights for implicit generative modeling. arXiv preprint 17X 2017.
12

---

[40] Espen Bernton, Pierre E Jacob, Mathieu Gerber, and Christian P Robert. On parameter estimation with the Wasserstein distance. Information and Inference: Journal of the IMA, (4):657-676, 2019.
[41] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, pages 5767-5777, 2017.
[42] Ishan Deshpande, Ziyu Zhang, and Alexander G Schwing. Generative modeling using the sliced Wasserstein distance. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3483-3491, 2018.
[43] Jiqing Wu, Zhiwu Huang, Dinesh Acharya, Wen Li, Janine Thoma, Danda Pani Paudel, and Luc Van Gool. Sliced Wasserstein generative models. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3713-3722, 2019.
[44] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
[45] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pages
3730-3738, 2015.
[46] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: /50..0006333 506.03365, 2015.
[47] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representa- rions, 2018.
[48] Tero Karras, Samuli Laine, and Timo Aila, A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4401-4410, 2019.
[49] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 151I::15111.04333 2015.
[50] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representations, 2018.
[51] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020.
[52] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626-6637, 2017.
[53] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung. Alec Radford, and Xi Chen. Improved techniques for training GANs. In Advances in Neural Information Processing Systems, pages 2234-2244 2242, 2016.
[54] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-28266 2016.
[55] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809. 1096, 2018
[56] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International Conference on Machine Learning, pages 7354-7363. PMLR, 2019.
13

---

[57] Mohammadreza Armandpour, Ali Sadeghian, Chunyuan Li, and Mingyuan Zhou. Partition- guided GANs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5099-5109, 2021.
[58] Hao Zhang, Bo Chen, Long Tian, Zhengjue Wang, and Mingyuan Zhou. Variational hetero- encoder randomized GANs for joint image-text modeling. In International Conference on Learning Representations, 2020. URL https://openrevie::
[59] Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Advances in Neural Information Processing Systems, pages 10542-10552, 2019.
[60] Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised GANs via auxiliary rotation loss. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12154-12100 12163, 2019.
[61] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006. 06676, 2020.
[62] Shengyu Zhao, Zhijian Liu, Ji Lin, Yan Zhu, and Song Han. Differentiable augmentation for data-cfficiens GAN training. arXiv preprint arXiv:2006. 10738, 2020.
[63] Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang. Image augmentations for GAN training. arXiv preprint Stti::2006.0259990 2020.
[64] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL https: https://openrevvll openneview. net/forum? htt/forrumid==xTTTII dd=PdIIC1RPPPIISS
[65] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(1 1):2288-2324, Nov 1998. ISSN 0018-9219, doi: 10.1 (0.1199/5.22691.
[66] Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Advances in neural information processing systems, pages 3308-3318, 2017.
[67] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[68] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http:/ www.aiii./..aa......eeee 141.. 6980.
[69] Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badcau, and Gustavo Rohde. Gener- alized sliced Wasserstein distances. In Advances in Neural Information Processing Systems, pages 261-272, 2019.
[70] Soheil Kolouri, Phillip E Pope, Charles E Martin, and Gustavo K Rohde. Sliced Wasserstein auto-encoders. In International Conference on Learning Representations, 2018.
[71] Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power of two samples generative adversarial networks. Advances Neural Information Processing Systems, pages 1498-1507, 2018,
[72] Adji B Dieng, Francisco JR Ruiz, David M Blei, and Michalis K Titsias. Prescribed generative adversarial networks. arXiv preprint arXiv: 2019.
14


Reviews:
[{"review_id": "yth28wstYsy", "reviewer": "Reviewer_M4DG", "summary": "This paper proposes a new distance measure between two probability distributions by exploiting the chain rule and Bayes' theorem. The new distance measure is obtained by minimizing out an additional parameter from the conditional transport (CT) cost. The CT costs is a convex combination of a forward component and a backward one. The forward CT is the expected cost of moving a source data point to a target one, with their joint distribution defined by the product of the source probability density function (PDF) and a surrogate source-dependent conditional distribution, which is related to the target PDF via Bayes' theorem. The backward CT is defined by reversing the direction. The CT cost can be approximated with discrete samples and optimized with existing gradient descent-based methods. The new measure nicely balances the mode-covering property of forward CT and mode-seeking property of backward CT, showing strong robustness against mode collapse. Experiments on a wide variety of benchmark datasets show that substituting the default statistical distance of an existing generative adversarial network with CT consistently improves the performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written and a pleasure to read. The methods in Section 2 are clearly explained and technically sound. The review of related work is informative. PyTorch code is provided to reproduce the experimental results.\n\nMinor comments:\nLines 109-110: It would be useful to briefly explain the concepts \"mode-covering\" and \"mode-seeking\".\nEq. (7): Please define the symbol $\\delta$. Does $\\delta_{x_i}$ denote the Dirac delta function? It should be written as $\\delta_{x_i}(x)$ or $\\delta(x-x_i)$.\nLine 135: \"Substituting .... in (2)\" should be \"Substituting .... in (1)\".\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "rLFZvQjc0Cd", "reviewer": "Reviewer_ausY", "summary": "This paper proposes to use conditional transport (CT) to measure the difference between two probability distributions. By combining both forward CT and backward CT, the proposed measure shows promising results for generative adversarial model training. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "After the discussion phase, I increase my overall score from 5 to 6. Thank the authors for their engagement. \n\n--\nOverall, this paper is well-written and the empirical results seem promising. However, I have two concerns:\n\n1) Does the advantage of this new measure come from combining forward and backward CTs? If so, can we look at the results while changing the value of $\\rho$?\n2) How does the choice of cost function c(x,y) impact the performance of this measure?  Is Euclidean distance used for the all the experiments in this paper? \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No obvious potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "JqXEXf74htb", "reviewer": "Reviewer_aP1u", "summary": "This paper presents a new cost function to measure the difference between two probability distributions. The proposed cost function called the conditional transport (CT) is a mixture of two cost functions. The two cost functions of the mixture are derived by factorizing the joint probability density function in two reverse manners. A sample analogue of the CT is given, and it is shown that this sample analogue converges to its population version as the number of samples tends to infinity. Experiments are given to assess the performance of the CT and compare the performance of CT-based methods with that of existing methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The presented theory seems new. The proposed cost function, CT, comes from an interesting idea to factorize the joint probability density function through the chain rule and Bayes' theorem in two reverse manners. I am impressed with a comprehensive experimental study to demonstrate the performance of the CT and compare the performance of methods based on the CT with that of existing methods. Related works are adequately cited.\n\nQuality: The submission seems technically sound in general. The claims are well supported by experimental results. Compared with great effort on the experiments, relatively little results are given to the theory of the CT. But Lemma 1 proves the convergence of the sample version of the proposed test to its expected version. The advantages of the proposed method are demonstrated through experiments. I do not find much discussion about the weaknesses of the proposed method in the paper.\n\nClarity: The paper is clearly written in general and well organized. The following are some minor comments on the presentation of the paper:\n\n(i) p.2, l.71: The definition of $\\mathbb{R}^H$ should be given.\n\n(ii) p.3, l.86: $p_{\\boldsymbol{\\theta}}(\\boldsymbol{y}) \\ $  $\\Longrightarrow$ $ \\ p_{Y}(\\boldsymbol{y})$?\n\n(iii) p.4, l.127: What does \"DGM\" stand for?\n\n(iv) p.4, equation (7): The definition of $\\delta_{ \\boldsymbol{x_i} } $ should be given. As it stands, the left-hand side of the equation says $p_{\\hat{X}_N}$ is a function of $\\boldsymbol{x}$, but the right-hand side of the equation does not involve $\\boldsymbol{x}$.\n\nSignificance: The thorough experiments given in the paper suggest that the proposed method is useful in practice. Comparatively little work have been done on the theory of the proposed method. Although Lemma 1 is helpful, it would be better to have more theoretical results about the CT. For example, I think the selection of the weight $\\rho$ of the cost function (12) would be a possible topic to discuss. If the true distribution is unknown, is it possible to select $\\rho$ efficiently in practice?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not find much discussion about the the limitations of the proposed method. I am a little worried about whether the estimation of $\\rho$ of the cost function (12) is computationally expensive or not. (I ask this because the selection of tuning parameters of a loss function requires a lot of computation in general.)\n\nI do not find any negative societal impact of the work. ", "review_text": "", "extra_scores": {"ethical_concerns": "There do not seem to be ethical issues with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "2YN85fyKyYs", "reviewer": "Reviewer_cVJH", "summary": "The paper defines conditional transport (CT), a new divergence between probability distributions (Equation 5), and reports experimental findings where this divergence serve as a good loss function for training generative models over natural images (Table 3, Figure 4 and so on). Among the tunable parameters of the divergence, there is one parameter that modulates between the extremes of mode seeking and mode covering. In real-data experiments, generative models fitted using CT as training loss have low Frechet inception distance (FID) and high inception score, which overall indicate high-quality generated images. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality. CT uses ideas from optimal transport but possesses novel modifications. In the broadest characterization, Equation 15, the parameter rho aims to balance between mode seeking and mode covering behaviors. For the focus area of generative model training,  CT has additional parameters phi and eta. Eta parametrizes a neural network which transforms the source/target from the natural space into some feature space, while phi parametrizes another network that compares feature representations. CT, when used as training loss, very frequently improves generative modelling quality compared to previous choices of training loss, regardless of the neural network architecture. Each pair of methods in Table 3 correspond to the same architecture: the top rows are image quality metrics under existing training losses, while bottom rows are metrics for CT as training loss. \n\t\nQuality. The quality of the generated images in Table 3/Figure 4,5 justify the use of phi and eta in the CT definition (although I have a question about the experimental setup that I will say in the Clarity section). The comparisons in Table 3 appear thorough, as the baselines range from very high-quality (like StyleGAN2) to low-quality (like DCGAN). Figure 1 is another justification for the navigator phi. In this synthetic situation, phi governs the curvature of the CT loss as a function of the generator parameters: suitable phi seemingly makes the CT loss more convex in the generator parameters, hence easier to optimize (I also have a question about the setup/result of this experiment). Figure 2 justifies the use of rho in the CT definition. As expected, the extreme setting of rho = 0 correspond to mode seeking behavior while rho = 1 corresponds to mode covering. By picking rho in between, there is a balance between the two. \n\nClarity. I have some questions about experimental setups and results. For Section 2.1, what does it mean for the navigator phi to be optimized? Up to this point, we have not talked about changing either rho or phi in definition of CT (Equation 5). Later on in section 2.3, the paper discusses  options like cooperative-trained versus adversarially-trained -- is this what it means in Section 2.1 too? If so, then as early as Section 2.1, it would be good to describe how phi is determined. Still in Section 2.1, for Figure 1, because the four panels on the right have different y-scaling, it's hard to see how the curvature of C(X,Y) change for different phi. Perhaps you can fix the same y-axis across the panels. Or plot the four curves on the same figure. For the \"Adversarially-trained CT for natural images\" paragraph in Section 4, what is the choice of rho? Perhaps you mention rho in the appendix, but since rho plays a big part in your methodology, and is mentioned in other experiments (like \"Forward and backward analysis\"), it would be good to know what rho is for these experiments as well. I have some comments about legibility. In general, the figure panels are too small, and it's difficult to see on a printed copy of the paper. Perhaps it's worth removing unnecessary panels, so that the panels chosen for display are easier to read. The panels in Figure 2 are very small and I can't make out what the axis labels/axis ticks say. The panels in Figure 3 are larger, but the axis ticks are not legible. For the panels in Figure 4, there are too many samples, each being too small.  My final comments are about word-choice/phrasing. Mini-max should be changed to min-max (minimax is another concept that is different). \n\nSignificance. Since CT serves as a good as-is training criterion without requiring specialized deep generative model architecture, other researchers are likely to use CT in place of existing training criteria in their research, hence this paper is an important set of results. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations of their work. As for potential negative societal impact, for the general field of generative modelling, there is the danger of creating fake material (fake news, etc) that spreads mis-information. Although the paper is primarily concerned with natural images, it is good to be aware of the general dangers.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers recommend an accept, one reviewer indicates a reject. The general sentiment is that the presented approach for comparing probability distributions with forward and backward CT is novel, and that the empirical results for generative adversarial model training are promising. However, there were some concerns as to the lack of theoretical results and inadequate discussion about the limitations of the proposed method. A revised paper should incorporate the revisions and clarifications brought up in the rebuttal. 
Decision: Accept (Poster)