ACT: Asymptotic Conditional Transport
Authors: Huangjie Zheng
Mingyuan Zhou
Venue:  (2021)
Keywords: Statistical distance
Divergence
Optimal Transport
Implicit Distribution
Deep Generative Models
GANs

Abstract: We propose conditional transport (CT) as a new divergence to measure the difference between two probability distributions. The CT divergence consists of the expected cost of a forward CT, which constructs a navigator to stochastically transport a data point of one distribution to the other distribution, and that of a backward CT which reverses the transport direction. To apply it to the distributions whose probability density functions are unknown but random samples are accessible, we further introduce asymptotic CT (ACT), whose estimation only requires access to mini-batch based discrete empirical distributions. Equipped with two navigators that amortize the computation of conditional transport plans, the ACT divergence comes with unbiased sample gradients that are straightforward to compute, making it amenable to mini-batch stochastic gradient descent based optimization. When applied to train a generative model, the ACT divergence is shown to strike a good balance between mode covering and seeking behaviors and strongly resist mode collapse. To model high-dimensional data, we show that it is sufficient to modify the adversarial game of an existing generative adversarial network (GAN) to a game played by a generator, a forward navigator, and a backward navigator, which try to minimize a distribution-to-distribution transport cost by optimizing both the distribution of the generator and conditional transport plans specified by the navigators, versus a critic that does the opposite by inflating the point-to-point transport cost. On a wide variety of benchmark datasets for generative modeling, substituting the default statistical distance of an existing GAN with the ACT divergence is shown to consistently improve the performance. 

Under review as a conference paper at ICLR 2021
ACT: ASYMPTOTIC CONDITIONAL TRANSPORT
Anonymous authors Paper under double-blind review
ABSTRACT
We propose conditional transport (CT) as a new divergence to measure the dif- ference between two probability distributions. The CT divergence consists of the expected cost of a forward CT, which constructs a navigator to stochastically trans- port data point of one distribution the other distribution, and that of a backward CT which reverses the transport direction. To apply it to the distributions whose probability density functions are unknown but random samples are accessible, we further introduce asymptotic CT (ACT), whose estimation only requires access to mini-batch based discrete empirical distributions. Equipped with two navigators that amortize the computation of conditional transport plans, the ACT divergence comes with unbiased sample gradients that are straightforward to compute, making it amenable to mini-bated stochastic gradient descent based optimization. When applied to train a generative model, the ACT divergence is shown to strike a good balance between mode covering and seeking behaviors and strongly resist mode collapse. To model high-dimensional data, we show that it is sufficient to modify the adversarial game of an existing generative adversarial network (GAN) to a game played by generator, forward navigator, and a backward navigator, which try minimize distribution- -o-to-ttt--diiiiiionn transport cost by optimizing both the distribution of the generator and conditional transport plans specified by the navi- gators, versus a critic that does the opposite by inflating the point-to-point transport cost. On a wide variety of benchmark datasets for generative modeling, substituting the default statistical distance of an existing GAN with the ACT divergence is shown to consistently improve the performance.
INTRODUCTION
Measuring the difference between two probability distributions is fundamental problem in statistics and machine learning (Cover) 1999. Bishop [2066] Murphy 2012). variety of statistical distances have been proposed to quantify the difference, which often serves the first step to build a gener- ative model. Commonly used statistical distances include the Kullback-Leibler (KL) divergence (Kullback and Leibler 1951). -aasen-Shannon (JS) divergence (Lin) 1991). and Wasserstein dis- tance Kantorovichi] 2006). While being widely used for generative modeling (Kingma and Welling 2013 Goodfellow al. 2014 Arjovsky al.. 2017 Balaji al. 2019), they all have their own limitations. The KI. divergence, directly related to both maximum likelihood estimation and varia- tional inference, is amenable mini-batch stochastic gradient descent (SGD) based optimization (Wainwright and Jordan, 2008; Hoffman et al. 2013 Blei et al. (017)). However, it requires the [wo probability distributions to share the same support, and hence is often inapplicable if either of them is an implicit distribution whose probability density function (PDF) is unknown (Mohamed and Lakshminarayanan 2016 (uszáá [017 Tran al.] 2017: (in and Zhou 2018). The JS divergence directly related to the mini-max loss of generative adversarial net (GAN) when the discriminator optimal (Goodfellow et al., 2014). However, it is difficult to maintain a good balance between the generator and discriminator, making GANs notoriously brittle to train. The Wasserstein distance is widely used metric that allows the two distributions to have non-overlapping supports (Villani 2008 Santambrogio 2015: Peyré and Cuturi 2019). However, it is challenging estimate in its primal form and generally results Its in biased sample gradients when its dual form is employed (Arjovsky et al.
2017 Bellemare et al. 2017 Bottou et al. 2017 Binkowski et al. 2018 Bernton et al. 2019).
To address the limitations of existing measurement methods, we introduce conditional transport (CT) new divergence quantify the difference between two probability distributions. We

---

Under review as a conference paper at ICLR 2021
refer to them as the source and target distributions and denote their probability density functions (PDFs) as Px (z) and py (y), respectively. The CT divergence is defined with a bidirectional iatrrbbutioonto--istibbbtttitttt
transport. It consists of a forward CT that transports the source to target
distribution, and a back ward CT that reverses the transport direction. Our intuition is that given a source (target) point, it is more likely to be transported to a target (source) point closer to it. Denoting d(æ, y) = d(y, x) as a learnable function and c(æ, y) c(y, x) 2 O, where the equality is true when æ y. as the point-to-point transport cost, the goal is minimize the transport cost between two distributions. The forward CT is constructed in three steps: 1) We define forward "navigator" as π ( xx) = e -d pv (y)/ /e-((a.)) py (1) ddy. a conditional distribution specifying how likely given source point will be transported to distribution py (y) via path æ - y: 2) We define the cost of a forward x æ-transporting CT as f f(( y) T r)dy. the expected cost of employing the forward navigator forward CT as px pxx( f c(æ,, y) æ)dydaæ which is the expectation of the cost of forward æ --transporting CT with respect to p.x
transport æ to a random target point; 3) We define the total cost of the
px(x)) (x)) Similarly, we construct the backward CT by first defining
a backward navigator as (< yy) = E d(n-1)) px (x)/ f e e=d(=.v)p px (z) dd and then its total cost as y)dxryy. Estimating the CT divergence involves both (< y) and x x)). py (y) fc(x, (<
which, however, are generally intractable to evaluate and sample from, except for few limited settings where both px (2) and py()) are exponential family distributions conjugate to e d(ar.))
To apply the CT divergence in general setting where we only have access random samples from the distributions, we introduce asymptotic CT (ACT) as a divergence measure that is friendly to mini-batch SGD based optimization. The ACT divergence is the expected value of the CT divergence, py (y) are both replaced with their discrete empirical distributions, respectively supported on N independent, and identically distributed (iid) random samples from px (œ) and M iid random samples from py (z). The ACT divergence is asymptotically equivalent to CT divergence when both N - oo and M + co. Intuitively, can also be interpreted as performing both a forward one-to-M stochastic CT from the source to target and a backward one-to-N stochastic CT from the target to source, with the expected cost providing an unbiased sample estimate of the ACT divergence.
whose Px (a) and py
We show that similar to the KI. divergence, ACT provides unbiased sample gradients, but different from it, neither px (x) nor py (y) needs to be known. Similar to the Wasserstein distance, it does not require the distributions to share the same support, but different from it, the sample estimates of ACT and its gradients are unbiased and straightforward to compute. In GANs or Wasserstein GANs (Arjovsky et al. 2017). having an optimal discriminator critic is required to unbiasedly estimate the IS divergence or Wasserstein distance and hence the gradients of the generator (Bottou et al. 2017). However, this is rarely the case in practice, motivating common remedy stabilize the training by carefully regularizing the gradients, such as clipping or normalizing their values (Gulrajani et al. 2017 Miyato et al. 2018). By contrast, in an adversarial game under ACT, the optimization of the critic. which manipulates the point-to-point transport cost c(æ, y) but not the navigators' conditional distributions for æ -> y and æ + y. has no impact on how ACT is estimated. For this reason, the sample gradients stay unbiased regardless of how well the critic is optimized.
To demonstrate the use of the ACT (or CT) divergence, we apply it to train implicit (or explicit) distributions to model both ID and 2D toy data, MNIST digits, and natural images. The implicit distribution is defined by a deep generative model (DGM) that is simple to sample from. We focus on adapting existing GANs, with minimal changes to their settings except for substituting the statistical distances in their loss functions with the ACT divergence. We leave tailoring the network architectures to the ACT divergence to future study. More specifically, we modify the GAN loss function to an adversarial game between a generator, a forward navigator, and a backward navigator, which try to minimize the distribution- toautiionnto-iisiiuuutiio transport cost by optimizing both the fake data distribution py py) and two conditional point-to-point navigation-path distributions ( () and [< y), versus critic that does the opposite by inflating the point-to-point transport cost c(2e, y). Modifying an existing (Wasserstein) GAN with the ACT divergence, our experiments show consistent improvements in not only quantitative performance and generation quality, but also learning stability.
2 CONDITIONAL TRANSPORT WITH GENERATOR, NAVIGATORS, AND CRITIC
Denote a data taking its value in RV In practice, we observe finite set X'= {z, ) | consisting of |X| data samples assumed to be rid drawn from px (æ). Given X, the usual task is to learn a distribution to approximate px (æ)), explaining how the data in X' are generated. To approximate
2

---

Under review as a conference paper at ICLR 2021
Px (x) we consider a DGM defined y -o(()) e <~ <>p(e), where Gg is generator that transforms noise E ~ p(e) via a deep neural network parameterized by to generate random sample y E RVV While the PDF of the generator, denoted as Po (z)) is often intractable to evaluate, it is straightforward to draw y ~ Po po(y) with Go. Denote both (((2x)) = px dz and v(dy) = poly)de dy as continuous JgY Px (2r (x))dæ= and v = Jev po po(y)dy 1.
probability measures over TR. V2 with (a(
The Wasserstein distance in its primal form can be defined with Kantorovich 's optimal transport
problem (Kantorovich] 2006 Villani 2008 Santambrogio 2015 Peyré and Cuturi 2019):
W(2 2/) = min J. xxv c(z, 3) dy)) mim.. ( (((m.)))=<(x /c(z, y)]}. (1)
where the minimum is taken over [I, v), defined as the set of all possible joint probability measures т on V x JRV, with marginals π (A,, R = ({AA) and π T(R A) - 1/ (xA) for any Borel set A CRY
When c(x, y) || yll. we obtain the Wasserstein-1 distance, also known the Earth Mover's distance, for which there exists a dual form according to the Kantorovich duality as
Wi (1, suPfeLip' (xx-xx (z) [=)) E. Eurrr (w) (f()))
where f is referred to as the "critic" and Lip' denotes the set of all 1 I-Lipschitz functions Villann 2008). Intuitively. the critic f plays the role of "amortizing" the computation of the optimal transport
plan. However, as is difficult to ensure the 1-Lipschitz constraint, one often resorts to approximations
(Arjovsky al.) (2177) Gulrajani et al. 2017: Wei et al. [208]] MMyato al. 2018) that inevitably
introduce bias into the estimation of Wi and its gradient (Bellemare et al. 2017 Bottou et al. 2017)
2.1 FORWARD AND BACKWARD NAVIGATORS AND CONDITIONAL TRANSPORT PLANS
Constraining स E I(,,, the Wasserstein distance satisfies W(n, v) W(v, 1t). By contrast, the proposed divergence allows & 11(,, v). Denote T E RR as a neural network based function, transforming its input in RY into a feature vector in R" and d(hi, h2) as function that measures the difference between h1, h2 E RRH, We introduce forward CT, whose transport cost is defined as
Co + = Ex Ey-- m)) m//e(2 8)], (3 x) def.
(m)) To Ta( po (w)
(2)
fe-diT (m)) (m) To (1)) po (y)dy"
where (y (2) will be analogized the forward navigator that defines the forward conditional transport plan. Similarly, we introduce the backward CT, whose transport cost is defined as
4( (o To()) px (xr)
Co. + v) = Ey-ro(s) Exwns v) lc(c,, y)], T [x:
(3)
c al ( px (a))dr"
where (x y) will be analogized to backward navigator. We now define the CT problem as ming,e (Co, Co.o(u, v) del CC + v) + Co.o + v), where C v) Co. 1)) will be referred to the CT divergence between μ and v.
(4)
Lemma 1. If y ~ (3) is equal to x ~ Px (x) in distribution and To is chosen such that
e TT) = 1(x y). where 1--) is an indicator function, then both the the joint probability measure x defined with px (y x)
(2) and that with po po(y) T (xx y) in (3) are in II v).
Lemma 2. y (y) is equal ~ px px(x:) in distribution, then Co. Co.0(1, v) = Co.0 v) Co.o(n + v) W(n u) 0, where the equality can be achieved if e di d(Ta((e).T((
1(x y).
The proofs are deferred to Appendix A Note in general, before both 0 and & reach their optimums, the conditions specified in Lemmas and 2 are not satisfied and the joint probability measure ı defined with px ( æ)) or po po polyy)T IT a 1)) is not restricted to be in I[ v). and hence it is possible for Co.e(u ( + v). Co < v). or Co.e(H,)) v) to go below W(p, during training.
2.2 ASYMPTOTIC CONDITIONAL TRANSPORT
Computing the CT divergence requires either knowing the PDFs of both navigators (5(1)) x) and (2 y), or being able to draw random samples from them. However, usually neither is true unless both pe(y) and px (x) are known and conjugate to e dd(Ta(().TT(v)) For example, if (TT) Ty(y)) = ||ø <yyl3
(x - ya ox y). where & E VVV is full-rank matrix, and both Px (z) and po ( are multivariate Gaussian distributions, then one may show that both n (y|l a)) and (a( y) are multivariate Gaussian distributions. In the experimental results
3

---

Under review as a conference paper at ICLR 2021
section, we will provide a univariate normal distribution based toy examples for illustration. Below we show how to apply the CT divergence in a general setting that only requires access to random samples of both and y. While knowing neither px (x(x) nor Po(y), can obtain mini-batch based empirical probability measures j.y and DM, as defined below, to guide the optimization of Go in an iterative manner. With N random observations sampled without replacement from X, define
AN * Ei=1d òz,, {æI,..., ØN S X
(5)
as an empirical probability measure for æ. Similarly, with M random samples of the generator, we define an empirical probability measure for y as
PM M اه òy, y,, Go(Ey), Ej th p(c)
(6)
Substituting po (e) in 2 with x (y), the continuous forward navigator becomes a discrete one as Ta(Ta( Tainy))
( x)
Σ}= f=1 fM (y o, h) AM xx Φ))
Thus the cost of forward CT becomes
Cp. - Um) = Ex (x) E Bur [c(2, y)1 = Exxpx (e) (Co - Dw)],
where Cæ.0(# + PM)
del Σ cxe, y ) (y, Φ)).
(8)
Similarly, we have the backward navigator and the cost of backward CT as (x () = Ei= (xx y: ))
(x; y: (b) det.
a dlTT(ee)) EN di derglayrh
(9)
Co.o(Ri + v) = Byr-olo)
v) wlle(a, b) = E By--o(t) [Co.llān + -y1.
where CaN + y) def Σ d( y) A (z; y, Φ).
(10)
Combining both the forward and backward CTs, we define the asymptotic CT (ACT) problem as ming. (Co. U, M)), where Co v4 M) is the ACT divergence defined as Copo(1, U, N, M) = 1 VI.M 1n poln) (Co.(l vM )
(11)
1 2NN 15 px (w) [C + v]]. (12)
Lemma 3. The ACT divergence is asymptotic that limn Co.0 u, N, M) = (1,
Lemma 4. With xI.N i2t pxx (x) and yIMM is po(y) and drawing x ~ (x(()) and y ~ UMs (y), an unbiased sample estimator of the ACT divergence can be expressed as
(x(x)
Co.e(EIN,, Ml.M) = د c(x, f2) (y, x, Φ) + -. c(2,, f y, Φ)
(13)
Intuitively, the first term in the summation can be interpreted as the expected cost of following the forward navigator to stochastically transport a random source point æ to one of the M randomly instantiated "anchors" of the target distribution. The second term shares similar interpretation. Note in optimal transport, the Wasserstein distance W(4, z/) in its primal form, shown in ① is in general intractable to compute. To use the primal form, one often resorts to the sample Wasserstein distance defined as W(i.. (M), computing which, however, requires solving combinatorial optimization problem (Peyré and Cuturi 2019). To make W(ÂN, PM)) practical to compute, one remedy is to smooth the optimal transport plan between üu and PM with an entropic regularization term, resulting in the Sinkhorn distance that still requires to be estimated with an iterative procedure, whose convergence is sensitive to the entropic regularization coefficient (Cuturi 2013 Genevay et al.) 2016 2018. Xie et al. 2020). When the entropic regularization coefficient goes to infinity, we recover maximum mean discrepancy (MMD), which is considered as the metric for minimization, evaluated in a kernel space found by the adversarial mechanism in MMD-GAN (Li et al., 2013 2017). By contrast, equipped with two navigators, the ACT can directly compute forward monnt-o-distribution transport cost, denoted as Cg + 2as) in (8) and backward one, denoted as Co.e (ā + y) in (T).. which are then combined to define an unbiased sample estimator, as shown in [3], of the ACT divergence. Intuitively, the navigators play the role of "amortizing" the computation of the conditional transport plans between two empirical distributions, removing the need of using an iterative procedure to estimate the transport cost. From this amortization perspective, the navigators for ACT are analogous to the critic for the Wasserstein distance in its dual form.
Lemma S. Another unbiased sample estimator fully using the data in mini-batchee æ1.N and y1:M0 computing an amortized transport cost between two empirical distributions, can be expressed as L..0 (XINN, V1MM = En Σزم c( v,) (2 ( (y, Di Φ) + 22M7 NN (2x; 1y,,
(14)
4

---

Under review as a conference paper at ICLR 2021
2.3 CRITIC BASED ADVERSARIAL FEATURE EXTRACTION
A naive definition of the transport cost between æ and y is some distance between their raw feature vectors, such c(ar, y) x y [ which, however, often poorly reflects the difference between high-dimensional data residing on low-dimensional manifolds. For this reason, with cosine dissim- ilarity (Salimans al., 2018), we introduce a critic (n))), parameterized by n7 help define an adversarial transport cost between two high- nigh-dimensional data points, expressed as
On (n y) = cos( cos(TT (z), (n
cos(f1,
def.
1h hal
(15)
h h1 //
Intuitively, to minimize the distribution-tor tiriibution---diissibuutinn transport cost, the generator tries to mimic true data and both navigators try to optimize conditional path distributions. By contrast, the critic does the opposite by inflating the point-to-point transport cost.
In summary, given the training data set X' to train the generator Go. forward navigator <<(y| backward navigator (x y). and critic Ty, we propose to solve a mini-max problem as
min max E Φ.0
#INEX, CLIM we p(c) (n {Go (Ge(E))))=1)
(16)
where Cs.0. is defined the same as in 13 or except that we replace c(z,, yy) in them with (,, yr) shown in (1)) and draw 9I:M using reparamcterization as in (6. which means def. {Go(e;)) We train & and 0 with SGD using V $..of..0,, (Go(e,) } ()) and, the critic is employed, train with stochastic gradient ascent using V vn ( {Go(e;)))m
91:M
Note in existing britec- GANs, how well the critics are optimized are directly related to how accurate and stable the gradients can be estimated. By contrast, regardless of how well the critic optimized to inflate Cr (zz. y). Lemma shows the sample estimate L (2z (ZI.N: {Go(c,)) ( of the ACT divergence and its gradients stay unbiased. Thus in ACT one can also train the critic parameter 7y using a different loss other than (16), such as the cross-entropy discriminator loss used in vanilla GANs to discriminate between x and Go (c). This point will be verified in our ablation study.
}=1)
EXPERIMENTAL RESULTS
CT divergence for toy data: As proof of concept, we illustrate optimization under the CT divergence in ID, with z, y, o, 0 e R. We consider a univariate normal distribution based example:
y)² p(x) (O 1), po(y) 6"), cxx,y) v) (a u) d(To(r), = 2
(17)
Thus 0 = 0 is the optimal solution that makes v = μu. Denote (a) = 1/(1 + +e a-)aa as the sigmoid
forward and backward navigators as To x) = (a 0)2, (0 ))e²) and T x y) = -N(((o))) (o)). and forward and backward CT costs as Co. - D) = a(o - (e + +a( - 0)) and Co.o(n < v) = (1) + (Ø)ee) (see Appendix B.1 for more details). Thus when applying gradient descent to minimize the CT divergence Co.o (1,, (1,,), v/), we expect the generator parameter 0 + 0 as long as the learning rate of the navigator parameter á is appropriately controlled to prevent ex + 0 from happening too soon. This confirmed by Fig. which shows that long before et approaches zero, has already converged close to zero. This suggests that the navigator parameter 6 mainly plays the role in assisting the learning of 8. It is also interesting to observe that the CT divergences keep descending towards zero even when W2 (M, 2)2 has already reached close zero. As 0 and Φ converge towards their optimal solutions under the CT divergence, we can observe that Co.o (p - v) and Co. + v) getting closer. Moreover, Co Co. v) initially stays above W2 (2 1/2 (1 e j but eventually becomes very close to W2),,,)? which agrees what Lemma 2 suggests. The second and third subplots describe the descent trace on the gradient of the CT cost with respect to (w.r.t) and o, respectively, while the fourth to six subplots show the forward, backward, and bi-directional CT costs, respectively, against 9 when e is optimized close to its optimum (see Fig. S for analogous plots for additional values of e.). It is interesting to notice that the forward cost is minimized at e" > 1, which implics mode covering, and the backward cost is minimized at e" -> 0, which implics mode seeking, while the bi-directional cost is minimized at around the optimal solution e" 1; the forward CT cost
function, we have analytic forms of the Wasserstein distance (2, "0 (1-e8)2
=1
5

---

Under review as a conference paper at ICLR 2021
Ciu 5)
Cip- (c)
(e
Op
Steration
0
-0
Illustration of minimizing the CT divergence Co.a(p, v) between (0 1) and c(0.c"). Left: Evolution of the CT divergence, its parameters and forward and backward costs, and corresponding Wasserstein distance; Middle: Gradients of the CT w.r.t. 0 or ø. The 2D trace of ((,e²) is marked with red arrows. Right: The forward, backward, and CT values against 0 when ex is optimized to small value, which show combining forward and backward balances mode covering and seeking, making easier for 0 to move towards its optimum.
Figure 1:
exhibits flattened curve on the right hand side of its minimum, adding the backward CT cost to which not only moves that minimum left, making it closer to 0 = O, but also raises the whole curve on the right hand side, making the optimum of & become easier to reach via gradient descent.
We further consider a ID example to illustrate the properties of the conditional transport distributions of the navigators, and analyze the risk for them to degenerate to point mass distributions when optimized under the CT or ACT divergence. We defer the details to Appendix B.2
[.2annd and Fig. 9
We move on to model the empirical samples from a true data distribution, for which it is natural to apply the ACT divergence. To parameterize ACT, we apply deep neural network to generator Gg and another one to To that is shared by both navigators. We consider the squared Euclidean (i.e. C3) distance to define both cost c(x, y) and distance ((1,h22)
ACT for 1D toy data:
We first consider a 1D example, where X consists of | = 5, 000 samples 2; E of a bimodal Gaussian mixture px (2)
Nx; -5 1) + (z; 2, 1). We illus- trate in Fig. 2 the training with unbiased sample gradients Vo.eLo.o (x 31.M) of the ACT divergence shown in (14). where yj = Go(e)). The top panel shows the ACT divergence, its backward and forward costs, and Wasserstein distance between the empirical probability measures D.N and Pa defined as in (5) and (6). We set M = N and hence (2xx,, by can be exactly computed by sorting the 1D ele- ments of iiNN and VI.N (Peyré and Cuturi 2019). We first consider N X 5000. Fig. (Top) shows that the ACT divergence converges close Wa )2 and the forward and backward costs move closer to each other and can sometime go below W2(xx, ) Fig. 2 (Bottom) shows that minimizing the ACT divergence successfully drives the generator distribution towards true data density: From the left to right, we can observe that initially the generator is focused on fitting a single mode; at around the 500 iteration, as the forward and backward navigators are getting better, they start help the generator locate the missing mode and we can observe a blue density mode starts to form over there; as the generator and both navigators are getting optimized, we can observe that the generator clearly captures both modes and the fitting is getting improved further; finally the generator well approximates the data density. Under the guidance of the ACT divergence, the generator and navigators are helping each other: An optimized generator helps the two navigators to train and realize the missing mode, and the optimized navigators help the generator locate under- fitted regions and hence better fit the true data density. Given the same X, below we further consider setting N = 20, 200, or 5000 train the generator, using either the Wasserstein distance W2 ) or ACT divergence ( 91:N) ) as the loss function.
4
Figure 2: Illustration of how minimizing the ACT diver- genee between the empirical distribution of generator and that of bimodal Gaussian mixture, whose 5000 random samples are given, helps optimize the generator distribu- tion towards the true one. Top: Plots of the ACT diver- genee C(âN, Exz), forward ACT cost C(pN + Par), back- ward ACT cost C(âx + Par), and Wasserstein distance Wa(āN, )²s)², where of the true data distribution (dxx) = px (=)dx (red) and the generator distribution r(dy) po (y)dy (blue, visualized via kernel density estimation) at different training iterations.
- 5000. Bottom: The PDF
As shown in the right column of Fig. 3 when the mini-batch size N is large 5000, both Wasserstein and ACT lead to a well-tramne generator. However, as shown in the left and middle columns, when N is getting much smaller, we can observe that the generator trained with Wasserstein

---

Under review as a conference paper at ICLR 2021
Rine Rool
-20
Figure 3:
Top: Plot of the sample Wasserstein distance W2(0soo0, P5000) ) against the number of training epochs, where the generator is trained with either W2(āv, Px)² or the ACT divergence between f.N and Pv. with the mini mini-batch size set N = 20 (left), .V 200 (middle), or N 5000 (right); one epoch consists of 5000/N SGD iterations. Bottom: The fitting results of different configurations, where the KDE curves of the data distribution and the generative one are marked red and blue, respectively.
clearly underperforms that trained with ACT, especially when the mini- size becomes as small as N = 20. While the Wasserstein distance W( in theory can well guide the training of generative model, the sample Wasserstein distance W(jx,, DN), whose optimal transport plan is locally re-computed for each mini-batch, could be sensitive to the mini-hatch size N, which also explains why in practice the sample Wasserstein-basee generative models are difficult to train and desire large mini-batch size (Salimans et al.) 2018). On the contrary, ACT amortizes its conditional transport plans through its navigators, whose parameter & is globally updated across mini-batches, leading to a well-traines generator whose performance has low sensitivity to the mini-batch size.
ACT for 2D toy data: We further conduct experiments on four representative 2D datasets: 8-Gaussian mixture, Swiss Roll, Half Moons, and 25-Gaussian mixture, whose results are shown in Figs. 13 of Appendix B.3 We apply the vanilla GAN (Goodfellow et al.] 2014) and Wasserstein GAN with gradient penalty (WGAN-GP) (Gulrajani et al. (017) as two representatives of mini- max DGMs that require solving a mini-max loss to train the generator. We then apply the generators trained under the sliced Wasserstein distance (SWD) (Deshpande et al. 2018)) and ACT divergence as two representatives of mini-max- max-free DGMs. Compared to mini-max DGMs, which require an adversarially learned critic in order to train the generator, one clear advantage of mini-max-fii DGMs is that the generator is stable train without the need of an adversarial game. On each 2D data, we train these DGMs as one would normally do during the first 15k iterations. We then only train the generator and freeze all the other learnable model parameters, which means we freeze the discriminator in GAN, critic in WGAN, and the navigator parameter & of the ACT divergence, for another 15k iterations. Fig. 10 illustrates this training process on the 8- B-Gaussian mixture dataset, where for both mini-max max DGMs, the mode collapse issue deteriorates after the first 15k iterations, while the training for SWD remains stable and that for ACT continues to improve. Compared to SWD, our method covers all 8 data density modes and moves the generator much closer to the true data density, On the other three datasets, the ACT divergence based DGM also exhibits good training consistency, high stability, and and close-to close-to-optimal data generation, shown in Appendix B.3
Resistance to mode collapse: We use 8-Gaussian mixture to empiri- cally evaluate how well a DGM re- sists mode collapse. Unlike the data in Fig. 10 where 8 modes are equally weighted, here the mode the left lower corner is set to have weight while the other modes are set to have the same weight of Lif. We set X' with 5000 samples and the mini-batch size as N = 100. When is low- ered to 0.05, its corresponding mode is shown to be missed by GAN, WGAN, and SWD-based DGM, while well kept by the ACT-based DGM. As an explanation, GANs are known to be susceptible to mode collapse; WGAN and SWD- based DGMs are sensitive to the mini-batch size, when p equals this mode will appear in the mini-batches less frequently than those from any other mode, amplifying their missing mode problem. Similarly, when P is increased to 0.5, the other modes are likely to be
GAN
WGAN-GP
SWD
ACT (ours)
True
0-00 p-0.05
Figure 4: Comparison of the generation quality on 8 Gaussian mixture data: one of the 8 modes has weight p and the rest modes have equal weight -_p.
small value, the samples from
7

---

Under review as a conference paper at ICLR 2021
missed by the baseline DGMs, while the ACT-based DGM does not miss any modes. The resistance of ACT to mode dropping can be attributed to the amortized computation of its conditional transport plans provided by the navigators, whose parameter is optimized with SGD over mini-batcher and, as indicated by Fig. 3 is robust to estimate across a wide rage of mini-hatch sizes.
Forward and backward analysis: To empiri- cally analyze the roles played by the forward and backward ACTs in training DGM, we mod- ity (12) to define ACT,, where y E [0. 1] is the interpolation weight from the forward ACT cost to the backward one, which means ACT, reduces to the forward ACT when 7 = 1, to backward ACT when 7 0. and to the ACT in 112 when y 0.5. Fig. 5 shows the fitting results of ,CT, on the same ID bi-modal Gaus- sian mixture used in Fig. 2 and 2D 8-Gaussian mixture used in Fig. TO: the other experimental settings are kept the same. Comparing the results of different ? in Fig. 5 suggests that minimizing
+=0.25
Figure 5: Fitting ID bi-modal Gaussian (top) and 2D 8-Gaussian mixture (bottom) by interpolating between the forward ACT (y = 1) and backward ACT (7 = 0).
the forward transport cost only encourages the generator exhibit mode covering behaviors, while minimizing the backward transport cost only encourages mode seeking/dropping behaviors; by contrast, combining both costs provides a user-controllable balance between mode covering and seeking, leading to satisfactory fitting performance, as shown in columns 2 to 4. Note that for a fair comparison, we stop the fitting at the same iteration; practice, we find if training with more iterations, both 7 = 0.75 and y = 0.25 can achieve comparable results as mode covering and seeking behaviors to be controlled by tuning y is an attractive property of ACT,- We leave the theoretical analysis of the mode covering/seeking behaviors of ACT, for future study,
0.5. Allowing the
We conduct variety of experiments on natural images to evaluate the performance and reveal the properties of DGMs optimized under the ACT divergence. We consider three widely-used image datasets, including CIFAR- 10 (Krizhessky et al.) 2009). CelebA (Liu et al.) 2015), and LSUNNbedroom (Yu et al. 2015), and compare the results of DGMs optimized with the ACT divergence against DGMs trained with the vanilla GAN and its various generalizations.
ACT for natural images:
Note different from previous experiments on toy data, where the transport cost c(2, y) can be defined by directly comparing x and y. for natural images, whose differences in raw pixel values are often not that meaningful, we need compare T,,()) and Tr (y), where (.)( is a critic that plays the role of adversarial feature extraction, as discussed in Section 2.3 In particular, we use (15) to define the transport cost C, (ar, y)
cos( (Ty, (z) Tn())) To parameterize the navigators, we also set (a), To (y)) = 1 - cos (x)) (z)). We test with the architecture suggested in Radford et al. (2015) as standard CNN backbone and also apply the architecture in Miyato et al. (2018) as the ResNet (He et al., 2016) backbone. Specifically, we use the same architecture for the generator, and slightly modify the output dimension of the discriminator architecture as 2048 for both T,, of the critic and To used by the two navigators. We train this model with (16) and (1). and to keep close to the corresponding backbone's 's original experiment setting, we set M 64 for all experiments.
We summarize in Table T the Fréchet inception distance (FID) of Heusel et al. 2017) on all datasets and Inception Score (IS) of [Salimans et al. (2016) on CIFAR- Both FID and IS are calculated using a pre-trained inception model (Szegedy al. 2016). Lower FID and higher IS scores indicate better image quality. We observe that ACT-DCGAN and ACT-SNGAN, which are DCGAN and SNGAN backbones optimized with the ACT divergence, convincingly outperform DCGAN and SNGAN, respectively, suggesting that ACT is compatible with standard GANs and WGANs and generally helps improve generation quality. Moreover, ranked among the Top 3 all datasets, ACT-SNGAN performs on par with the best benchmarks on CIFAR-10 and CelebA, while slightly worse on LSUN. The qualitative results shown in Fig. 6 are consistent with quantitative results in To additionally show how ACT works for more complex generation tasks, we show in Fig. 7 example higher-resolution images generated by ACT-SNGAN on LSUN bedroom and CeQ
Table I
Note ACT brings consistent improvement to DCGAN and SNGAN by neither improving their network architectures nor gradient regularization. Thus it has great potential work in conjunction with other state-of-the the--rrt architectures or methods, such as BigGAN (Brock et al. 2018), self-attention GANs (Zhang et al. 2019) BigBiGAN (Donahue and Simonyan 2019), progressive training (Karras et al.
8

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison of generative models on CIFAR- CelebA LSUN. (Top-3 3 are bold)
Method
Fréchet Inseption Distance (FID, beller)
Inception (higher beller)
WGAN Anjonky at al 2017
51.3
37.1219
73.312.
5.910.11
WGAN-GP GGann momein dur 2017
19:0=0x
18.0=0.7
26.9+1.1
7.9+0.11
MMD GAR La 2013
73.9
6.2s:0.1
Crumér GAN
2017
40.3.002
31.3:02
51,210.4
6.410.1
CTGAN
17.6-0.7
15.8=06
19,5+1.2
5.1+0.1
19:4=3.0
N0.55.3
SWG Destoandd orai 2015
13.7
21.9220
67,912.7
Max-SWer (peopiiee o 2010
23.6-0.5
10.1=0.6
40.1+4.5
SWGAN
Wo of 2019
17,0=1.0
13.2=0.7
1499
DOGAN Radioni al 2015
30.2.0099
52.51222
61.712,8
6.2.0.1
DCGAN
Miyato el T-DOGAN)
248::10
29 29:2::20
37,4+2.5
7.5-0.1
SNGAN backbons
XCT divergenee SNGAA)
18.0....7
11.31.0
24.21188
8.1.0.1
Figure 6: Generated samples of the deep generative model that adopts the hackbone of SNGAN but optimized with the ACT divergence on CIFAR-10, CelchA, and -UN-Bedroom. See Appendix for more results.
Figure 7: Analogous plot to Fig. for Left:
128) Right: CelebA-HQ (256x256).
2018), self supervisd learning CCen al. 2019 and data augmentation (Karras 2020 Zhao
et al. 2020a b), which we leave for future study. As the paper is primarily focused on constructing and validating a new divergence measure, we have focused on demonstrating its efficacy on toy data and benchmark image data with moderate resolutions. We have focused on adapting DCGAN and SNGAN under ACT, and we leave to future work using the ACT to optimize a big DGM, such as BigGAN, that is often trained for nigh-resolution images with substantially bigger network and larger mini-batch size and hence requires intensive computation that is not easy to afford.
CONCLUSION
We propose conditional transport (CT) as a new divergence to measure the difference between two probability distributions, via the use of both forward-pan and backward-patt point-to-poin condi- tional distributions. To apply CT to two distributions that have unknown density functions but are easy sample from, introduce the asymptotic CT (ACT) divergence whose estimation only requires access to empirical samples. ACT amortizes the computation of its conditional transport plans via its navigators, removing the need of separate iterative procedure for each mini-batch, and provides unbiased mini-batch based sample gradients that are simple to compute. Its minimization, achieved with the collaboration between the generator, forward navigator, and backward navigator, is shown to be robust to the mini-batch size. In addition, empirical analysis suggests the combination weight of the forward and backward CT costs can be adjusted to encourage either model covering or seeking behaviors. We further show that a critic can be integrated into the point-to-point transport cost of ACT to adversarially extract features from high-dimensional data without biasing the sample gradients. We apply ACT train both vanilla GAN and Wasserstein GAN. Consistent improvement is observed in our experiments, which shows the potential of the ACT divergence in more broader settings where quantifying the difference between distributions plays essential role.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Cooningg-Voluee 70, pages 214-223, 2017.
Yogesh Balaji, Hamed Hassani, Rama Chellappa, and Soheil Feizi. Entropic GANs meet VAEs: statistical approach to compute sample likelihoods in GANs. In International Conference on Machine Learning, pages 414-423, 2019.
Marc G Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The Cramer distance as solution to biased Wasserstein gradients. arXiv preprint 1eX.v::705..000
2017.
Espen Bernton, Pierre E Jacob, Mathieu Gerber, and Christian P Robert. On parameter estimation with the Wasserstein distance. Information and Inference: Journal of the IMA, 8(4):657-676, 2019.
Christopher M Bishop. Pattern Recognition and Machine Learning. springer, 2006.
Mikolaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018. URL https
//openreview.. net httt://foommiiiddde oor 10
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians.
Journal of the Americun statistical Association, 1125518):559-87, 2017.
Leon Bottou, Martin Arjovsky, David Lopez-Paz, and Maxime Oquab. Geometrical insights for implicit generative modeling. arXiv preprint arXiv:1712.07822, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised GANs via auxiliary rotation loss. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 12154-12163, 2019.
Thomas M Cover. Elements of Information Theory. John Wiley Sons, 1999.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, pages 2292-2300, 2013.
Ishan Deshpande, Ziyu Zhang, and Alexander G Schwing. Generative modeling using the sliced Wasserstein distance. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3483-3491, 2018.
Ishan Deshpande, Yuan-Ting -ing Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David Forsyth, and Alexander Schwing. Max-sliced Wasserstein distance and its use for GANs. In Proceedings of the IEEE conference computer vision and pultern recognition, pages 10648-10656, 2019.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. Advances in Neural Information Processing Systems, pages 10542-10552, 2019.
Aude Genevay, Marco Cuturi, Gabriel Peyré, and Francis Bach. Stochastic optimization for large- scale optimal transport. In Advances in Neural Information Processing Systems, pages 3440-3448, 2016.
Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with Sinkhorn divergences. In International Conference on Artificial Intelligence and Statistics, pages 1608-1617, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672-2680, 2014.
10

---

Under review as a conference paper at ICLR 2021
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, pages 5767-5777, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626-6637, 2017.
Matthew Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. 2013.
The Journal of Machine Learning Research,
Ferenc Huszár. Variational inference using implicit distributions. arXiv preprint arXiv: 2017.
Leonid Kantorovich. On the translocation of masses. Journal of Mathematical Sciences, 133(4): 1381-1382, 2006.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.06677 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL
http:/ /ttri///xiiiv or..rrg//.. abs 1412.4114 6980
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. 1312.61312.61 14, 2013.
arXiv preprint
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The Annals of Mathemati- cal Statistics, 22(1))79-86, 1951.
Y. Lecun, L. Bottou, Y. Bengio, and Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233) Nov 1998. ISSN 0018-9219. doi: 10.1 10.1109/5.726791
Chun-Liang Li, Wei-Chen, Chang. Yu Cheng, Yiming Yang. and Barnabás Póczos. MMD GAN: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems, pages 2203-2213, 2017.
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International Conference Machine Learning, pages 1718-1727, 2015.
Jianhua Lin. Divergence measures based the Shannon entropy. IEEE Transactions on Information theory, 37(1):145-151, 1991.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of the IEEE international conference computer vision, pages 3730-3738. 2015.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. 2018
Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv preprint ar 1611v: 2016.
Kevin P Murphy. Machine Learning: Probabilistic Perspective. MIT Press, 2012.

---

Under review as a conference paper at ICLR 2021
Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 1 11(5-6)3355-600 0003355667,-00 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv preprint arXiv:1511.0645: 2015.
Tim Salimans, lan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Advances in Neural Information Processing Systems, pages 2234 2242, 2016.
Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving GANs using optimal transport. arXiv preprint arXiv: / 803.05573, 2018.
Filippo Santambrogio. Optimal Transport for Applied Mathematicians: Calculus Variations, PDEs, and Modeling, volume 87. Birkhäuser, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-22122 2818-2826, 2016.
Dustin Tran, Rajesh Ranganath, and David Blei. Hierarchical implicit models and likelihood-free variational inference. In Advances in Neural Information Processing Systems, pages 5523-5533, 2017.
Cédric Villani. Optimal Transport: Old and New, volume 338. Springer Science & Business Media, 2008.
Martin J Wainwright and Michael Irwin Jordan. Graphical models, exponential families, and variational inference. Now Publishers Inc, 2008.
Xiang Wei, Boqing Gong. Zixia Liu, Wei Lu, and Liqiang Wang. Improving the improved training of Wasserstein GANs: A consistency term and its dual effect. In International Conference on Learning Representations, 2018.
Jiging Wu, Zhiwu Huang, Dinesh Acharya, Wen Li, Janine Thoma, Danda Pani Paudel, and Luc Van Gool. Sliced Wasserstein generative models. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3713-3722, 2019.
Yujia Xie, Xiangfeng Wang, Ruijia Wang. and Hongyuan Zha. A fast proximal point method for computing exact Wasserstein distance. In Uncertainty in Artificial Intelligence, pages 433-453. PMLR, 2020.
Mingzhang Yin and Mingyuan Zhou. Semi-implicit variational inference. In International Conference on Machine Learning, pages 5660-5669, 2018.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of large- scale image dataset using deep learning with humans in the loop. arXiv preprint 100.:v::5066.33..
2015.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International Conference on Machine Learning, pages 7354-7363
PMLR, 2019.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Shengyu Zhao, Zhijian Liu, Ji Lin. Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient GAN training. arXiv preprint arXiv:2006. 10738, 2020a.
Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang. Image augmentations for GAN training. arXiv preprint arXiv:2006.022 02595, 2020b.
12

---

Under review as a conference paper at ICLR 2021
A PROOFS
Proof of Lemma
1 If y ~ po(y) is equal to a ~ px((a) in distribution, then px (a:) = po( for
any x E RV. For we have (x x)dy - px T x)dy - px (x) and
/ (x x)dx= px(x))
Tol dx
=po(y) /
pe dx.
If (T=dd(T((a):Tall))) 1(x v), then further have
6 { (ar) dx = /
1 1(x y)px (z) dx = / 1(x y) Px px((x) dx po(xx)
(Te po (y)
f1(x= = u)
and hence it true that
/ (x) ax)d poly)-
Similarly, for B we have I po (g) (x) y)dx = po and can prove f po(y))= ( yydy Px(x) given these two conditions.
Proof of Lemma 2 Since c(x, y) 2 by definition, we have Co.o(H - +) 2 0 and Co.olu + v) 0. When f V, it is known that W(r, v) If y po is equal to æ px px(xx) in distribution, which means px (x) = (xz) and px (y) = po (y) for any x, y E RV and μ = v, then we have
c (x, (x))) dddddy / ʃ
Co.0(1 + v)
= / c(x; y)
d(Ta( dx drdy
/ c(y,
(e drdy
/ c(y,x) x)
po dxdy
/ c(x, c(r, (po))) T (TT[) px (x (x)dx
drdy
/ c( ((r,))rl))) yrddy
Co.ol1 + v)
and hence Co.elu, v) = Co. + v) = Co.e(u t v) 2 - W(,,
If e (m) = 1(x =y), since c(x, x)) = -y definition, we have
Co.olu + = / c(x, y)
1(x = y)lx (x))(()) dady
f 1
(y)dd
= / / c(x, y)
1(x: = y)px (ar (ar dady
(x)
/
clxx 1)1(<= = y))o(y)dady
c(æ, ()))d
0.
13

---

Under review as a conference paper at ICLR 2021
Proof of Lemma M
According the strong law of large numbers, when M - 0o, PM
M 2==1 1 A) converges almost surely
M Ey po (y E A)] = poly)ddy v(A)
M
and hence C - PM) converges to Co - v). Therefore, VIM w polu) Coolu + PM)) converges to Co.0( + v). Similarly, we can prove that as N + oc, ai 1n (x) Co t v) converges to @ f 1/) Therefore, (o, N, M) defined in (12) converges to CC v) C t v) = Co.o(p,)) v) as N,M M + oc.
Proof of Lemma
C,.0(1, N, M) = 1
VL 18 poly) Co Co + im))l + 1 anw v px)
(x Co.olix + v)]
wmxx (w), Un us po(v) (0, UM)] + E Min w px(e), vmpall) CCol(NN y)]
2 Bengn (w) Mi 2h (x)) yaar 1e po(y) (0o. ++ +Mm]
+ 1 By MIN in (m), vae 12 po(y) Co + y)]
(x), (roparr)) (v)
E
ind (o C2 - ins) + Co.ollin y)] (18)
(m)) va.3r
Plugging 8 and 100 into the above equation concludes the proof.
Proof of Lemma 5
Solving the first expectation of (18). we have
E
En
Co.e PM) + 2M En Co.elān + +,,)
#1. ud (xx),, M 1
(o(v)
2N
Plugging and 10 into the above equation concludes the proof.
B SUPPLEMENTARY EXPERIMENT RESULTS
B. ADDITIONAL DETAILS FOR THE UNIVARIATE NORMAL TOY EXAMPLE SHOWN IN
For the toy example specified in (77 exploiting the narmal-norma conjugacy, we have an analytical conditional distribution for the forward navigator as
Th (vl x)) & 200 N(y:0.c²)
x NN(Ev,
=N -00%
and an analytical conditional distribution for the backward navigator as
( v) xe (m-w)
0,
x e
ed
=N
1 の 1 1+c cτ
14

---

Under review as a conference paper at ICLR 2021
Plugging them into 2 and 33. respectively, and solving the expectations, we have
es
e
Co.o(u - v) BonN((.1)
c
+
+ es 2²
e +
e
ex
e
c + +
c + c
es
ed
Ca.l11 t 1/)=1B 1 Ey-N((
1+
+ c 82)
1 +
ex
es
e
1+
6
- EB
C(z-w))
0.50u -) SC
3.0
C(u-o))
0.50g-11- SCpr-vi
14-
1.4
1.2
12
44
Clu-+1)
Clr-v)
0 0.500 1 a) SC
5
33
37
30
300
316
c(ume))
SCC
20
344
33
42
Figure 8: For the univariate normal based toy example specified in [77. we plot the forward, backward, and CT values against 0 at four different values of Φ. which show combining forward and backward balances mode covering and seeking, making it easier for A to move towards its optimum.
15

---

Under review as a conference paper at ICLR 2021
B.2 ANALYSIS OF CONDITIONAL TRANSPORT PLANS ON ID MIXTURE
We consider a ID example to illustrate the properties of the conditional transport plans of the navigators, and analyze the risk for them degenerate point mass distributions when optimized under the CT or ACT divergence. We consider two representative scenarios that both seem to pose a high risk for the conditional transport distributions to degenerate. In the first scenario, we consider both the source and target distributions as mixture of a point mass and a Gaussian distribution, where the location of the point mass and the center of the Gaussian are far away from each other; we obtain the analytic forms of the conditional transport plans under the CT divergence, and analyze the properties of the empirical conditional transport plans under the ACT divergence. In the second scenario, we consider both the source and target distributions as discrete distributions; we make the support of each distribution contain an outlier supporting point far away from all the other supporting points. In this scenario, we are essentially learning how to transport between two discrete sets.
B.2.1 CONDITIONAL TRANSPORT BETWEEN TWO POINT-MASS- DISTRIBUTIONS
MIXTURE
Below we consider the first scenario, where we assume
px(x) = p + (1 ()N0(- 1)
and
Py (v) pó + (1 p) 1),
where p [0, is the probability for x = -1 and y = We construct this specific example to check whether there is a danger that the forward CT distribution T(y (z) will degenerate to a point mass
distribution that concentrates its probability mass at y = 1. Setting d(Ta To))) = 2. via
the definition of the conditional transport distributions and the property of the normal distribution, we can show that
pr(y. x))
1-p
<U 2r, Φ) pr(y, ()) +1-p 51 +
x) +1 x)( N(,,
N(v; 14, o²)
eb
x) Na: 1)'
del 1000
+x 1
o² del
(19)
1 1 ed
1 et
Similarly, for the backward CT, we can show that
(2) pr x)) + - 8
1-p
Fxx 1., の)
+ N(kl,0
(y, x)
def. W(zzp',0²2)
es
11 del. 1000
+ y
def.
(20)
N(:: 1000, 1)¹
1 eσ
1 +
c
1 cx
As shown in the top panel of Fig. 9 it is clear that when o + 0o, we have F 1|x) p and
when é + oc, we have π (y- = r) = 0. Assuming/ p = 0.01 and x = 1000, when @ = 15, we
have To = |
1000) 0.0157 and To =1 = -1) = -1) 0.0116; when 6 = 110-11.9900 1.095945,
we have T = 1 1 = - 1000) = 0.0626 and T = 1
-1) = 1. These analyses suggest that as long the navigator parameter & is chosen appropriately, the conditional transport distributions T (ar y, Φ) and πty| x. Φ) will not degenerate to point mass distribution.
Next we analyze the degeneration risk using the ACT between px (x:) = -1__ 1) and (v) = p5 + (1 P)N 1), in which case we assume we don't 't know (x) and py (v) but can access random samples from them. In ACT, we would approximate T 1) with
M
(0-2))2
AM (y
x
Dj
itd ~ pr (v)-
(21)
(K-Ny) 2
Even if à is set at a value such that the forward CT distribution Fe (y = 1 1|x) - 1, under ACT, as long as $ (yi,..., "ar...,,,r). }. x will surely not be transported to = 1 and will instead be transported to a By that drawn from ((100,, 1): note & (v1..... yar) } happens with probability I P(# + 1) = (1 M)M, which is as large as 36.6% when p = 0.01 and M = 100. Therefore, banc- based optimization under the ACT divergence would further reduce the risk for the conditional transport plans to degenerate.
16

---

Under review as a conference paper at ICLR 2021
Caidlie myacc more mortrr
www.eetoooooooooooooooooo trwphhrrreeee 3800
Con Siaue bumpet For N0 byw3)
Condiiir murenorte 11
Gww.tnal wwrqun naceing NogC
Concitional wwwpprtt mazz #o50
Corditona earape mappingg $050
Cordsore Garger muppiig $350
Figure 9: Top: The risk for the forward and backward conditional transport plans to degenerate w.t.. the value based on CT analysis. Bottom: The forward conditional transport plan between two discrete sets of S = 500 points; among the S data points, one point is an outlider. The ACT navigators are optimized with SGD over mini-batches, whose elements are randomly sampled (with replacement) from their corresponding discrete sets and the mini-batch size varies from 50 to 500.
B.2.2 ACT BETWEEN TWO DISCRETE DISTRIBUTIONS WITH OUTLIER SUPPORTS
Below we consider the second scenario, where we assume two discrete distributions supported on 500 data points as
$
S-1
s
px(x) = 5 8-1
sid where ~ NV-1000, 1), and
S 1
pr()) 5 +
S
where yi sid N(0000, 1). this scenario, we will be optimizing the navigator parameters using SGD over mini-batches, and apply the navigators optimized with mini-batch size of N to calculate the conditional transport plans between the two discrete sets of S = 500 data points. More specifically, we use (1) the loss to train the navigator parameter & (note here both distributions are fixed and there is no generator to train), with the mini-batch size set as N M = 50, 200, 400, or 500. Each mini-batch consists of two sets of M data points iid drawn from their corresponding discrete distributions (i.e., sampled without replacement from their corresponding discrete sets). With the navigator parameter & optimized under the ACT divergence, the bottom panel of Fig reports the forward conditional transport plan from the Gaussian to py, i.e., (T(()) x. Φ),,.., ा (ys cc, Φ)), where x (-11000, 1) and é learned with four different mini-batch sizes. There results suggest that even though for two discrete distributions whose supporting points contain outliers, there is low risk for the conditional transport plans optimized under ACT to degenerate.
17

---

Under review as a conference paper at ICLR 2021
B.3 MORE RESULTS ON 2D TOY DATASETS
We visualize the results on the 8-Gaussian mixture and three additional 2D toy datasets. Compared to the 8-Gaussian mixture dataset, the mode collapse issue of both GAN and WGAN-GP becomes more severe on the Swiss-Roll, Half-Moon, and 25-Gaussian datasets, while ACT consistently shows good and stable performance on all of them.
5k iters
10k iters
15k iters
20k iters
25k iters
30k iters
True
GAN
WGAN-GP
SWD
ACT (Ours)
Figure 10: On 8-Gaussian mixture data, comparison of generation quality and training stability between two mini-max deep generative models (DGMs), including vallina GAN and Wasserstein GAN with gradient penalty (WGAN-GP), and two mini-max- -ree DGMs, whose generators are trained under the sliced Wasserstein distance (SWD) and the proposed ACT divergence, respectively. The critics GAN and WGAN-GP and the navigators of ACT are fixed after 15k iterations. The first column shows the true data density.
5k iters
10k iters
15k iters
20k iters
25k iters
30k iters
True
GAN
WGAN-GP
SWD
ACT (Ours)
Figure 11: Analogous plot Fig. for the Swiss-Ros Roll dataset.
18

---

Under review as a conference paper at ICLR 2021
5k iters
10k iters
15k iters
20k iters
28k iters
30k iters
True
GAN
WGAN-GP
SWD
2
2
2
ACT (Ours)
2
Figure 12: Analogous plot to Fig. 10 for the Half-Moon dataset.
5k iters
10k iters
16k iters
20k iters
25k iters
30k iters
True
GAN
WGAN-GP
SWD
ACT (Ours)
Figure 13: Analogous plot to Fig. 10 for the 25-Gaussian mixture dataset.
19

---

Under review as a conference paper at ICLR 2021
We also illustrate the data points and generated samples with empirical samples, shown in Fig. The first column shows the generated samples (marked in blue) and the samples from data distribution (marked in red). To visualize how the feature extractor T used by both navigators works, set its output dimension as and plot the logits the third and fifth columns and map the corresponding data (in the second column) and generated samples (in the fourth column) with the same color.
Similarly, we visualize the GAN's generated samples and logits produced by its discriminator in Fig. 15 We can observe that the discriminator maps the data very close values. Specifically, in both the 8-Gaussian mixture and 25-Gaussian mixture cases, when the mode collapse occurs, the logits of the missed modes have similar value to the those in the other modes. This property results in GAN's mode collapse problem and it is commonly observed in GANs. Different from the GAN case, the navigator in our ACT model maps the data with non-saturating logits. We can observe in various multi-mode cases, different modes are assigned with different values by the navigator. This property helps ACT well resist the mode collapse problem and stabilize the training.
40 COOM
44
44
46
Figure 14: Visual results of ACT for generated samples (blue dots) compared to real samples (red dots) on Swiss Roll, Half Moons, 8 8-Gaussian mixture, and 25- Gaussian mixture. The second and third columns map the data points and their corresponding navigator logits by color; The fourth and fifth columns map the generated points and their corresponding navigator logits by color.
20

---

Under review as a conference paper at ICLR 2021
Visual results of GAN for generated samples (blue dots) compared to real samples (red dots) on Swiss Roll, Half Moons, 8-Gaussian mixture, and 25- Gaussian mixture. The second and the third columns map the data points and their corresponding discriminator logits by color; The fourth and fifth columns map the generated points and the corresponding discriminator logits by color.
Figure 15:
21

---

Under review as a conference paper at ICLR 2021
B.4 RESULTS FOR ABLATION STUDY
Transport cost in pixel space vs. feature space We visualize the difference of using the transport cost the pixel space and the feature space here. In both Figs.I6. 17 we test with MNSIT and CIFAR-10 data and with the L2 distance and cosine dissimilarity as the transport cost, respectively. For the MNIST dataset, due its simple data structure, ACT can still be trained generate meaningful digits, though some digits appear blurry. On the CIFAR-10, 10, we can observe the model fails to generate any class of CIFAR images. As the dimensionality of the input space increases, using the distance in the pixel space transport cost might lose the essential information for the transport and increases the training complexity of the navigator.
Pixel-L2: (IS)
Pixel-cos: (IS)
Pixel-2:: (FID)
Pixel-cos: 138.4 (FID)
9
8
5
3
1
I
O
3
3
O
7
O
3
7
5
5
8
6
2
6
1
2
1
8
9
7
7
7
/
5
4
9
0
6
0
7
6
1
9
9
9
6
3
9
0
0
5
6
9
8
3
0
6
6
9
8
6
9
7
/
0
8
8
6
0
7
6
1
9
3
2
2
6
6
5
2
0
7
5
4
0
/
9
6
7
3
2
2
3
Figure 16: Visual results of generated samples on MNIST and CIFAR-10 using pixel-wise transport
cost, with DCGAN (standard CNN) backbone.
Pixel-12
Pixel-cos
Pixel-L2
Pixel-cas
O L
RK
0
4
v u
o
((:)) u
<
0
h 1) 0no
6 61
Figure 17: Visual results of generated samples MNIST and CIFAR-10 using pixel-wise transport cost, with SNGAN (ResNet) backbone. The Inception and FID scores are not shown due to poor visual quality.
Percoptual-ACT (fix)
Perceptual- energy distance (fix)
Perceptual-ACT (initialiiationn)
Percaptual-ACT (fix after training)
4
2
3
9
7 3
4
2
3
9
1
5
0
8
0
3
0
3
:
O
0
4
3
,
6
3 0
6
0 0
0
6
3
Q
5
7
6
2
3 Q
7
2
6
8
0
8
1
3 0
7
8
0
8
0
G
Q
8
3 5
Q
6
G
Q
L
8 5
1 5
5
7
5
8
8
6
O
5
4
D
8
4
D
}
Figure 18: Visual results of generated samples with the perceptual similarity ZZang et al. with four different training configurations.
2018
We also consider using the perceptual similarity (Zhang et al. 2018) to define the cost function. Here test with four configurations: 1) Apply fixed and pre-trained perceptual loss calculate the distance between the data and generated samples, use that distance as the point-to to to-point cost, and calculate ACT train the generator; 2) Apply fixed and pre-trained perceptual loss calculate
22

---

Under review as a conference paper at ICLR 2021
Table FID comparison for ACT-DCGAN and ACT-SNGAN on CIFAR-IO with different cost and architecture.
transport cost c(x,3) y) La
Stannardd CNN-
-iiie
cos-pixel
cos-feature
Navigator cost
C
171.9
138.4
27.1
24.8
d(Ta(a)) TT)))
COs
157.9
136.2
25.3
25.0
transport cost c(æ:, y) L3-feature
-ResNet-
Cappixel
cos-pixel
cos-feature
Navigator cost (()) To(y))
C3
237.2
297.1
22.8
21.6
cos
293.1
252.7
19.8
18.0
the energy distance between the data and generated samples to train the generator; 3) Apply the pre-trained perceptual loss as cost and fine-tune with ACT to train the generator; 4) Apply the pre-trained perceptual loss cost, fine-tune it with ACT to train the generator, and then fix it for 20 more training epochs. We report the visual results of these 4 configurations in Fig. T8 As shown, fixing the metric and calculate the distance (either ACT energy distance) in the feature space does not show good generation results. An explanation could be the pretrained perceptual loss is not trained on the generation task and hence might be compatible with the learning objective. Thus the cost could not feedback useful signal to guide the generator. Using the energy distance as in Bellemare et al. 2017) shows similar results. We further use the perceptual network as the initialization of our critic T7, and train with ACT by maximizing the cost for 40 epochs on MNIST, which produce good-quss quood-qulity generations as shown in the third column. Then we fix the training of this critic and train the generator for 20 more epochs, which lead to degraded generation quality. We expect the generation quality will get worse as the training under the fixed critic continues.
Using alternative cost function We also test ACT with different configurations. As discussed in previous sections, the defined cost may also affect the training of the model. Here we vary the choice of transport cost G, (1, y) and the navigator cost d(T (z) To (y)). For both cost, we test with L3 distance and cosine dissimilarity. Moreover, we also compare the effects of distance in the original pixel space and the feature space (equipped with critic Tn). The results in Table highlight the importance of the cost in the feature space when dealing with high-dimensional image data. Moreover, compared to the T. distance, the cosine dissimilarity observed to improve the model when applied as transport and navigator cost, especially with ResNet architecture.
Training the critic with the diseriminator loss of a vanilla GAN Contrary to the existing critic- based GANs, the sample estimates of ACT divergence and its gradient are unbiased regardless of how well the critie trained. We thus keep the same experiment settings and train the ACT-
DCGAN model's critic with the discriminator loss of standard GANs, i.e. Ex-P |- log( (x)] [x)]]]] The results in Fig. 19 shows that ACT works well in conjunction with
Ex-r, log the alternative critic training. The quantitative and qualitative
MNIST, CIFAR-10 10, CelebA, and LSUN are shown in Fig. 19 We can observe the quality of generated samples, while clearly not as
good as training the critic with the ACT divergence, can still catch up with some of the benchmarks in Table
Inception Score: 8.3
FID: 43.5
FID: 34.3
FID: 39.4
3
0
7
6 9
}
7
7
d
0
5
6
7
8
7
4
4
5
6
1 9
3
5
0
3
2
7
3
0
3
0 3
9 0 7
3
7
3
4
8
4
2
6
8
7
9
6
0
3
Figure 19: Visual results of using a standard cross-entropy discriminator loss in lieu of ACT diver- gence to train the critic of ACT.
23

---

Under review as a conference paper at ICLR 2021
B.5 MORE RESULTS ON IMAGE DATASETS
For the experiments on the image datasets, we provide more visual results in this part. Apart from the datasets described in the experiment part, we also test the capacity of single-channel image generation with the MNIST dataset. Considering the inception score and the FID score are designed for RGB natural images, we also calculate the inception score the real testing sets for reference. The presented methods are all able to generate meaningful digits on MNIST. If we take a closer look the digits, the digits generated with L2 cost is less natural than the one with cosine cost. Moreover, we show both unconditional and conditional generation results on CIFAR-10. For both unconditional and conditional generation, our proposed method achieves good quantitative and qualitative results.
Real:
DCGAN:
DCGAN+ ACT-12:88
DCGAN ACT-cos:
7
3
4
5
3
9
0
4
6 5 6 0 0
7
9 8 7 9 4 8 6 0 7
7-
+ 3
9
7
Q
3 9 8 8 9 7 0
&
9
'1
6
0
4
7 6
1
8 8
O
0
7
9
7
4/
O
7
5 3
5
9
5
b
7
8
9
7
8
0 4
9
7
9
4
5
3
8
2 0
7
5
4
4
5
9
4
J 3
0 6 0
7 4
4
9
4
1
7
3
3
f
9
6
7
f
6
9
9 7
2
8
0
1
4
5
2
6
7
3
3 4 7
9
(s D
9
9
4
6
6
8
7
0
4
2
8 2
3
7
6
1
3
0
0 8
9
b
3
7 8
3
7
9 8
7
2
0
0
3
5
4
8
7
7
D
6
8
5
4
3
7
3
0
8
7 2
/
0
8
3
7
6
> 6
4
0
4
6
2
5 2
8
0
3
Figure 20: Unconditional generated samples and inception scores of MNIST, with DCGAN (standard CNN) backbone.
Real
DOGAN: 30.4
DCGAN ACT-L2:
DCGAN ACT-cos: 25.0
Figure 21: Unconditional generated samples and FIDs of CIFAR-10, with DCGAN (standard CNN) backbone.
Real
SNGAN (ResNet): 24.2
SNGAN (ResNet) + ACT-L2: 22.7
SNGAN (ResNet) + ACT-cos: 19J1
Figure 22: Unconditional generated samples and FIDs of CIFAR-10, with SNGAN (ResNet) back- bone.
24

---

Under review as a conference paper at ICLR 2021
Real
SNGAN (ResNet): 21,5
SNGAN (ResNet) ACT-L2: 19.8
SNGAN (ResNett ACT-cos: 18.0
Figure 23: Conditional generated samples and FIDs of CIFAR-10, with SNGAN (ResNet) backhone.
Real
DCGAN: 52.5
DOBAN + ACT-12: 33.2
DCGAN ACT-cos: 29.2
Figure 24: Generated samples and FIDs of CelebA, with DCGAN (standard CNN) backbone.
Real
SNGAN: 21.7
SNGAN (ResNet) ACT-L2: 18.3
SNGAN (ResNet)) ACT-cos: 11.3
Figure 25: Generated samples and FIDs CelebA, with SNGAN (ResNet) backbone.
Real
DOGAN: 6L7
DCGAN + ACT-L2: 52.4
DCGAN ACT-cos: 37A
Figure 26: Generated samples and FIDs LSUN, with DCGAN (standard CNN) backbone.
25

---

Under review as a conference paper at ICLR 2021
Real
SNGAN:
SNGAN (ResNet) ACT-L2::20
SNGAN (ResNet) ACT-cos: 24.2
Figure 27: Generated samples and FIDs of LSUN, with SNGAN (ResNet) backbone.
26

---

Under review as a conference paper at ICLR 2021
B.6 EXPERIMENT DETAILS
Preparation of datasets We apply the commonly used training set of MNIST (50K images, 28 x 28 pixels) (Lecun al. 1998). CIFAR-10 (50K images, 32 32 pixels) (Krizhevsky al. 2009), CelebA (about 203K images, resized to 64 x 64 pixels) (Liu et al. 2013). and LSUN bedrooms (around million images, resized to 64 x 64 pixels) (Yu et al. 2015). The images were scaled range [- 1, 1]. For MNIST, when calculate the inception score, we repeat the channel to convert each gray-scale image into RGB format.
Network architecture and hyperparameters For the network architectures presented here, the slopes of all IReLU functions in the networks are set to 0.1 by default. For toy experiments, typically 10, 000 update steps sufficient. However, our experiments show that the DGM optimized with the ACT divergence can be stably trained at least over 500, 000 steps (or possibly even more allowed to running non-stop) regardless of whether the navigators are frozen or not after a certain number of iterations, where the GAN's discriminator usually diverges long before reaching that many iterations even if we do not freeze it after a certain number of iterations. For all image experiments, the output feature dimension of the navigators and that of the critic (i.e., To (-)()) E R²') are set to m 2048. All models are able to be trained on single GPU, such as NVidia GTX 1080-T1 in our experiments, with 150, 000 generator updates (for CIFAR- 10 we apply 50, 000 iterations).
To keep close to the configuration of the DCGAN and SNGAN experiments setting, we use the the Adam optimizer (Kingma and Ba 2015) with learning rate a 2 x 10-4 and 31 0.5, 32 = 0.99 for the parameters of the generator, navigators, and critic. On the DCGAN backbone, we let all the modules update with the same frequency; while on the SNGAN backbone, the critic is updated once per 5 generator updating steps. The performance might be further improved with more careful fine-tuning. For example, the learning rate of the navigator parameter could be made smaller than that of the generator parameter. The true data minibatch size is fixed to N = 64 for all experiments. Moreover, with this batch-size we let the generated sample size M the same minibatch size N for ACT computation and we have monitored the average time for each update step on a single NVidia GTX 1080-TI GPU: On CIFAR- 10, each update step takes around O. and 0.2s for DCGAN and SNGAN, respectively; For DCGAN and SNGAN backbone trained with ACT divergence each update step takes around 0.4s and 0.7s. On CelebA and LSUN, each update takes 0.6s and 0.7s for DCGAN and SNGAN, respectively; when trained with ACT, the elapsed time for each update increases to 3.3s and 3.6s, respectively.
Table Network architecture for toy datasets (V indicates the dimensionality of data).
(a) Generator Go
(b) Navigator T Discriminator D
E 50 ((00 1)
x E RV
50- - 100, dense, IReLU
2 + 100, dense, IReLU
100 + 50, dense, IReLU
100 - 50, dense, IReI.U
50 + V, dense, linear
50 1, dense, linear
27

---

Under review as a conference paper at ICLR 2021
Table 4: DCGAN architecture for the CIFAR-10 dataset 4).
(a) Generator Go
(b) Navigator Tø. Critic Ty
E
28
(0,
E [-1, 1] 32x32x3
x 512, dense, linear 4 X 4, stride=2 deconv. BN 256 ReLU
128 + 4 x
x 3, stride=1 conv 64 IReLU
x 4, stride=2 conv 64 IReLU
4 x 4, stride=2 deconv. BN 128 ReLU
3 x 3, stride=1 conv 128 IReLU
4 x stride=2 conv 128 IReLU
4 x 4, stride=2 deconv. BN 64 ReLU
stride=1 conv 256 IReLU
3 3, stride=1 conv. Tanh
4 4, stride=2 conv 256 IReLU
3 x 3, stride=1 conv. 512 IReLU
h x 512 m, dense, linear
Table 5: DCGAN architecture for the CelebA and LSUN datasets (h 4).
(a) Generator Go
(b) Navigator Tg Critic Tn
C
(0,
a E [-1, 64x64x3
128 + 4 x 1024, dense, linear
x 4, stride=2 conv 64 IReLU
x 4, stride=2 deconv. BN 512 ReLU
4 X 4, stride=2 cony BN 128 IReLU
4 X 4, stride=2 deconv. BN 256 ReLU
4 x 4, stride=2 conv BN 256 IReLU
3 x 3, stride=1 cony BN 512 IReLU
4 x 4, stride=2 deconv. BN 128 ReLU
h x w x 512 > m, dense, linear
4 x 4, stride=2 deconv. BN 64 ReLU
3 3, stride=1 conv. Tanh
Table 6: ResNet architecture for the CIFAR-10 dataset.
(a) Generator Ge
(b) Navigator Tg Critic T
R 128
10,
æ E [-1, 32x32x3
128
4
256, dense, linear
ResBlock down 128
ResBlock up 256
ResBlock down 128
ResBlock up 256
ResBlock 128
ResBlock up 256
ResBlock 128
BN, ReLU, 3 conv, 3 Tanh
ReLU
Global sum pooling
h 128 - m, dense, linear
28

---

Under review as a conference paper at ICLR 2021
Table 7: ResNet architecture for the CelebA and LSUN datasets.
(a) Generator Go
(b) Navigator TΦ Critic Ty
R 28
(0,
I E [[1,, 1 64x04x3
128 - 4 x 4 1024, dense, linear
ResBlock down 128
ResBlock up 512
ResBlock down 256
ResBlock up 256
ResBlock down 512
ResBlock up 128
ResBlock down 1024
ResBlock up 64
ReLU Global sum pooling
BN, ReLU, 3 conv, 3 Tanh
1024 + m, dense, linear
Table ResNet architecture for the LSUN-122 128 dataset.
(a) Generator Ga
(b) Navigator Ty. Critic Tn
6 E 128
10,
a E 1 61x64x3
128 + 4 x
1024, dense, linear
ResBlock down 128
ResBlock up 1024
ResBlock down 256
ResBlock up 512
ResBlock down 512
ResBlock up 256
ResBlock down 1024
ResBlock up 128
ResBlock 1024
ResBlock up 64
ReLU Global sum pooling
BN, ReLUU, 3 3 cony, 3 Tanh
1024 m, dense, linear
Table 9: ResNet architecture for the CelebA-HQ dataset.
(a) Generator Go
(b) Navigator Tg Critic T
R 128
10,
@ E [[1, 64x64x3
128
4 x
1024, dense, linear
ResBlock down 128
ResBlock up 1024
ResBlock down 256
ResBlock up 512
ResBlock down $12
ResBlock up 512
ResBlock down 512
ResBlock up 256
ResBlock down 1024
ResBlock up 128
ResBlock 1024
ResBlock up 64
ReLU Global sum pooling
BN, ReLU, 3 3 conv, 3 Tanh
h 1024 + m, dense, linear
29


Reviews:
[{"review_id": "v6Ke_-8VcY", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new transport-based divergence between distributions (CT) and a variant for empirical distributions (ACT). The new divergence is claimed to be more suitable for learning deep generative models than existing divergences like KL, JS (as in the vanilla GAN) and Wasserstein (as used in WGAN and its variants).  The proposed divergence mostly resembles, in my opinion, the Wasserstein divergence variant that uses the  Kantorovich–Rubinstein dual definition (which requires the learned function to be 1-Lipschitz). It seems that the main advantages of ACT over Wasserstein is that there is no constraint on the Lipschitz smoothness (which has to be enforced in WGAN by means of e.g. gradient clipping or gradient penalty), and the fact that ACT provides unbiased gradients that do not require the critic to reach an optimal point (as required in theory in GAN or WGAN).\n\nThe paper presents a potentially interesting and significant method, however, it was a bit hard to follow, making it difficult for me to asses the actual significance of the contribution.\n\nSpecifically, the distinction between $d(x,y)$ and $c(x,y)$ was not 100% clear to me. $d(x,y)$ seems to be part of the \"navigator\" $\\pi(x | y)$ - an energy-based conditional probability and $c(x, y)$ is the point-to-point transport cost. For the CT to be low, it looks like $c(x,y)$ and $d(x,y)$ should be positively correlated. It was not clear, however (at least from the introduction), how $d$ and $c$ fit the generator-critic GAN scheme. Are they both part of the critic? Do the claims that the critic does not have to reach optimality for the gradients to be unbiased refer just to $c$ or to $d$ as well? I believe this is partially explained, but only towards the end of the paper (eq. 16). Perhaps a figure showing the trained elements in ACT compared to the trained elements in WGAN could help clarify the method.\n\nSpecific questions:\n- Why are both the forward and backward CT needed? \n- Are the parameters of the forward and backward navigators shared? From eq. 16 it looks like they are. Does it make sense?\n- In section 2.3, the authors claim that L2 distance in the original image domain is known not to work well for high-dimensional data that resided on a lower dimensional manifold, but what about L2 on some pre-trained feature-space e.g. perceptual distance [1]?\n- The difference in principle between the proposed ACT and OT-GAN and MMD-GAN should be described.\n\nTo summarize:\n\npros:\n- Interesting method, seems to be novel\n- Potential significance (as an alternative loss for training deep generative models)\n\ncons:\n- Writing hard to follow\n- Some open questions **update: the authors have answered most of my questions**\n\n[1] Zhang, Richard, et al. \"The unreasonable effectiveness of deep features as a perceptual metric.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "cjR6FfFsTX7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes conditional transport as a new divergence to measure the difference between two distributions. The idea is to learn the conditional transport plan of transporting one point in one distribution to the other marginal distribution. This conditional transport plan is modeled using a neural network. The resulting model is then applied to optimal transport formulation. Experiments are shown on image-based generative modeling dataset.\n\nThe main idea is to decompose the joint transportation plan $\\pi(x, y)$ using conditionals as $p(x)\\pi(y|x)$ and $p(y)\\pi(x|y)$. The conditional transportation plan $\\pi(x|y)$ and $\\pi(y|x)$ are then modeled using neural contrastive losses, with the idea being similar points in two distributions are mapped closer. This decomposition is then applied in optimal transport formulation, leading to a new objective for optimizing OT. The authors also derive an empirical version of this objective, that makes it amenable for stochastic mini-batch optimization.\n\nWhile the idea of this decomposition is interesting, I see the following issues with the formulation.\n\n(1) From the forms of conditional transportation plans $\\pi(x|y)$ and $\\pi(y|x)$ in Eq 2 and 3, $p(x)\\pi(y|x) \\neq p(y)\\pi(x|y)$. It would be good to have models that satisfy this equality.\n\n(2) I am not sure if the contrastive model assumed in Eq 2 and 3 is expressive enough to model the entire space of marginal constraints in OT. That is, I don’t think $p(x)\\pi(y|x)$ will cover $\\Pi(\\mu, \\nu)$. Lemma 1 just shows that under some conditions, $p(x)\\pi(y|x)$ lies inside $\\Pi(\\mu, \\nu)$, but it would be interesting to see if then entire space of $\\Pi(\\mu, \\nu)$ is covered by the neural contrastive model.\n\n(3) When critic is used in ground cost function $c(x, y)$, the resulting optimization (Eq 16) has one additional network compared to standard GANs. Also, the adversarial game still exists. So, it looks like this optimization is more harder (or at least equally harder) compared to standard GANs. Is this true? Do you see any optimization benefits compared to standard GANs?\n\n(4) The results of image-based generative modeling is not that impressive. On CIFAR and LSUN datasets, the performance is similar / slightly better than the compared GAN models. Also, many SOTA GAN models are not compared. So, it is very hard to say if the proposed model advances SOTA results.\n\n(5) I would have liked to see more interesting results. The formulation of authors gives an estimate of transportation plan $\\pi(x, y)$ in addition to the generative model itself, which is not possible to estimate in dual-based OT GAN optimization. The transportation plan can be used in interesting applications. One possibility is to estimate likelihoods and possibly use in OOD detection. Take a look at Balaji et al., “Entropic GANs meet VAEs: A statistical approach to compute sample likelihoods in GANs” for this. This is just one idea. Other interesting applications could have been demonstrated as well.\n\nOverall, while the idea is interesting, more results could have positioned the paper better. Just selling it as a paper that improves image-based generative modeling is not that great in my opinion. I would encourage authors to think more interesting experiments.", "extra_scores": {}, "extra_text": {}}, {"review_id": "aINGQJ-tFA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper introduces an asymptotic conditional transport divergence to measure the discrepancy between two probability distributions. The new divergence leads to a new adversarial game of generative adversarial networks, which aims at minimizing a distribution-to-distribution transport cost by optimizing the generator distribution and the conditional transport-path distributions of navigators. \n\nStrength:\n\nIntroducing the asymptotic conditional transport divergence to generative modeling looks novel and interesting to me. \nBoth the motivation and the technical details seem sound.\nThe evaluation shows the effectiveness of the proposed method on both toy datasets and some popular image generation datasets.\n\nWeakness:\n\nFig.3 has shown the superiority of the proposed divergence against the existing divergence (JS, Wasserstein distance, and Sliced Wasserstein distance) to fit the toy data distribution. But readers might doubt whether the comparison is fair or not. Do they use the same network backbone (e.g., DCGAN, WGAN-GP)? As there are many improved variants of SWD like (Deshpande et al., 2018, Deshpande et al., 2019, Wu et al., 2019), I am also wondering which SWD is used. The problem might get more serious for general cases. To address the problem, the paper is suggested to additionally present a clear theoretical study on the comparison, as readers really want to know why the proposed divergence works better than others, while some explanations have been presented in the introduction part.  For instance, it is not clear to me why the sample estimate of the ACT divergence and its gradient stay unbiased. \n\nAs the new divergence requests two additional navigators for bidirectional distribution-to-distribution transport, the adversarial game gets more complex in theory. It is known that GAN models generally suffer from unstable training and mode collapse issues. I am afraid the higher complexity might make such issues more serious. While Fig.1, Fig.2 somehow shows the behaviors of the navigators, I suggest to further study whether the proposed method can overcome or avoid such issues. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new measure of difference between two distributions using conditional transport. The paper considers an important problem.  However, some major concerns remain after the discussion among the reviewers. In particular, the paper focuses on the evaluation on a toy dataset. It is unclear whether the claim carries over to large real datasets. The presentation of the paper also needs substantial improvement.

Decision: Reject