AE-OT: A NEW GENERATIVE MODEL BASED ON EXTENDED SEMI-DISCRETE OPTIMAL TRANSPORT
Authors: Dongsheng An
Yang Guo
Na Lei
Zhongxuan Luo
Shing-Tung Yau
Xianfeng Gu
Venue:  (2020)
Keywords: Generative model
auto-encoder
optimal transport
mode collapse
regularity

Abstract: Generative adversarial networks (GANs) have attracted huge attention due to
its capability to generate visual realistic images. However, most of the existing
models suffer from the mode collapse or mode mixture problems. In this work, we
give a theoretic explanation of the both problems by Figalli’s regularity theory of
optimal transportation maps. Basically, the generator compute the transportation
maps between the white noise distributions and the data distributions, which are
in general discontinuous. However, DNNs can only represent continuous maps.
This intrinsic conflict induces mode collapse and mode mixture. In order to
tackle the both problems, we explicitly separate the manifold embedding and the
optimal transportation; the first part is carried out using an autoencoder to map the
images onto the latent space; the second part is accomplished using a GPU-based
convex optimization to find the discontinuous transportation maps. Composing the
extended OT map and the decoder, we can finally generate new images from the
white noise. This AE-OT model avoids representing discontinuous maps by DNNs,
therefore effectively prevents mode collapse and mode mixture.

Published as a conference paper at ICLR 2020
AE-OT: A NEW GENERATIVE MODEL BASED ON EX- TENDED SEMI-DISCRET OPTIMAL TRANSPORT
Dongsheng An', Yang Guo', Na Lei Zhongxuan Luo", Shing-Tung Yau', and Xianfeng Gu'
Department of Computer Science, Stony Brook University
DUT-RU ISE, Dalian University of Technology
Department of Mathematies, Harvard University
Idoan, yangguo) @ cs. stossyyyooooo.ce edu, [nalei, zxluo @ dlut.edu.cr yau@math. mea@@aath.haaae harvard.edu. gu@cc.stonybrook.c
ABSTRACT
Current generative models like generative adversarial networks (GANs) and vari- ational autoencoders (VAEs) have attracted huge attention due to its capability to generate visual realistic images. However, most of the existing models suffer from the mode collapse or mode mixture problems. In this work, we give the- oretic explanation of the both problems by Figalli's regularity theory of optimal transportation maps. Basically, the generator compute the transportation maps between the white noise distributions and the data distributions, which are gen- eral discontinuous. However, deep neural networks (DNNs) can only represent continuous maps. This intrinsic conflict induces mode collapse and mode mixture. In order to tackle the both problems, we explicitly separate the manifold embedding and the optimal transportation; the first part is carried out using an autoencoder (AE) to map the images onto the latent space; the second part is accomplished using GPU-based convex optimization to find the discontinuous transportation maps. Composing the extended optimal transport (OT) map and the decoder, we can finally generate new images from the white noise. This AE-OT model avoids representing discontinuous maps by DNNs, therefore effectively prevents mode collapse and mode mixture.
INTRODUCTION
Generative adversarial networks (GANs) (Goodfellow et al. (2014)) and variational autoencoders (VAEs) (Kingma & Welling (2033) emerge as the dominant approaches for unconditional image generation. When trained on appropriate datasets, they are able to produce realistic and visual appealing samples. GAN methods train an unconditional generator that regresses real images from random noise and a discriminator that measures the difference between generated samples and real images. Despite GANs' advantages, they have critical drawbacks. 1) Training of GANs are tricky and sensitive to hyprraarameters. 2) GANs suffer from mode collapse, in which the generator only learns to generate few modes of data distribution while missing others, although samples from the missing modes occur throughout the training data (see e.g. Goodrellow (2016)) While for the VAEs, the encoder is used to map the data distribution to Gaussian latent distribution, which is then mapped back the data distribution by the decoder. While standard VAEs tend capture all modes, they often generate ambiguous images multimodal real data distributions. We propose that these phenomena relates deeply with the singularities of distribution transport maps.
Manifold Distribution Hypothesis In deep learning, the manifold distribution hypothesis is well accepted, which assumes the distribution a specific class natural data concentrated on a low dimensional manifold embedded in the high dimensional data space Tenenbaum et al. (2000). Therefore, GANs and VAEs implicitly aim to accomplish two major tasks: 1) manifold embedding: to find the encoding/decoding maps between the data manifold embedded in the image space and the latent space; 2) probability distribution transport: to transport a given white noise distribution to the data distribution, either in the latent in the image space.

---

Published as a conference paper at ICLR 2020
hi.h
Js
Jss
Convex support
(b) concave support: single mode
(c) concave support: multi modes
Figure 1: Mode collapse/mixture caused by the discontinuity of the transport map. Top row shows real data distributions, the bottom row gives the noise distributions. On top, each cluster represents mode, the spurious generated samples are red crosses (mode mixture); the bottom, red dotted lines are the singularity set, red crosses are mapped to be spurious samples by DNNs. (a) If the support of the target distribution is convex, DNN (f1) is able to approximate the transport map fi well. When the support of the target distributions are concave, there are two situations: (b) single mode and (c) multi modes. In (b), DNN, represented by f2, cannot approximate the transport map fa well and generates some spurious samples. fa gives the transport map of multi-mode, when approximating it with continuous DNNs, either mode collapse fai mode mixture f32 will happen.
Distribution Transformation The generator of GAN model and the decoder of VAE model trained to compute a transport map that transforms a known continuous distribution (e.g. Gaussian white noise) to the real data distribution. Namely, the transport map pushes forward the given noise distribution to generated distribution to approximate the real data distribution, the similarity between the two distributions determines the generalization ability of the generator Ben- Dea--Dvvdd al. 2010).
Discontinuity and Mode Collapse/Mixture It is common practice among GAN/VAE models that the generatorssdeeoderss are expressed by deep neural networks, which can only represent continuous mappings. Unfortunately, as pointed out by works Nagarajan & Kolter (2017): Khayatkhoei et al.
(2018); Xiao et al./ (2018), the transport maps may be discontinuous when there are multiple modes in the data distribution. This intrinsic conflict can cause mode collapse or mode mixture. The later means that the generated samples are mixtures of multiple modes and look spurious or ambiguous, [T]
Furthermore, even when the real data distribution has a single mode, ambiguous data (e.g. a human face image with mismatched eye colors) can still present. This can be explained by Brenier's polar factorization theorem Brenier (1991b; 1987 1991a) and Figalli's s regularity theorem Figalli (2010); Chen & Figalli] (2017) (Thm. in Appendix B), which asserts that if the support of the target distribution is not convex, then there will be singularity sets on the support of the source distribution, such that the transport map is discontinuous on these sets. This shows the intrinsic difficulties of conventional GANs/VAES cannot be eliminated, as shown in Fig.
1991a) theorem, the optimal transport map be represented as the gradient map of the Brenier potential. At the regular points, the Brenier potential is differentiable, its gradient map (the transport map) is continuous; at the singularities, the Brenier potential is continuous but differentiable, and its gradient map is discontinuous. Conventional GANs and VABS model the gradient map directly and encounter the trouble of discontinuity. In contrast, we propose model the globally continuous Brenier potential to avoid mode collapse/mixture.
Conquering Mode Collapse/Mixture However, according to Brenier 1987
More specifically, our proposed AE-OT model separates the manifold embedding step and the probability distribution transformation step, the former is carried out by an autoencoder (AE), the latter is accomplished by a convex optimization framework (OT). The OT step computes the Brenier potential explicitly and is able to locate the singularity set (the discontinuous points of the gradient map) based on Figalli's 's theory. Our experimental results demonstrate that the proposed method can not only cover all of the modes, but also avoid generating spurious samples (mode mixture).
Contributions (i) From theoretical aspect, this work gives a thorough explanation of mode collapse and mode mixture by the regularity theory of optimal transportation developed by Figalli (2018
For example, generator generates obscure digits mixing and but neither nor 8 on the MNIST dataset.
2

---

Published as a conference paper at ICLR 2020
Fields medalist) and the reasons why standard GANs/VAEs cannot solve this problem perfectly. (ii) From practical aspect, this work proposes a novel model called AE-OT, which first encodes the data manifold into the latent space, then compute the Brenier potential to represent the optimal transportation map in the latent space. The Figalli's singularity set can be located efficiently and avoided when generating new samples. In this way, our model eliminates mode collapse and mode mixture successfully. (iii) The algorithm for finding the Brenier potential and the optimal transportation map can be accelerated with GPU based convex optimization algorithm. The method converges to the unique global optimum with bounded error estimate. (iv) Our experiment results demonstrate the efficiency and efficacy of the proposed method.
RELATED WORK
Optimal Transport Optimal transport (OT) has been successfully applied in the areas like economy, optics and machine learning, as surveyed in [Solomon| (2018) and Peyré & Cuturi (0118) In Gu et al. (2016) the intrinsic connection between Brenier theory in OT and Alexandroff theory in convex geometry was established, and applied for deep learning in Lei et al. (017)) by an convex optimization. Figalli and the collaborators Figalli (2010); Chen & Figalli (2017) 7 proposed that when the support of the data distribution is non-convex, the transport map will be discontinuous.
Generative models Generative model is one of the main tasks in the machine learning field. One of the most used image generation methods is Variational Autoencoders (VAEs), where the decoders approximate real data distributions from Gaussian distribution in a variational approach (e.g Kingma & Welling (2013) and Rezende et al. (014)). Later, Adversarial Autoencoders (AAEs) Makhzani et al. (2015) and Wasserstein Autoencoders (WAEs) Tolstikhin al. (2018) are proposed following the similar scheme. Although VAEs are relatively simple to train, images they generate look blurry. Generative Adversarial Networks (GANs) Goodfellow et al. (2014) proposed by Goodfellow et.al can produce better quality images. While being a powerful tool in generating realistically looking samples, GANs suffer from the mode collapse problem. To solve this problem. huge number of methods, including changing the loss function (e.g. Wasserstein Arjoskyy et al. (2017)) regularizing the discriminators to be Lipschitz (clipping (Arjovsyy et al. (2017), , gradient regularization Gulrajani et al.. (0117). Mescheder et al. (2018) or spectral normalization Miyato et al. (2018)) were proposed.
Besides, various non aov-ssavrlllial methods has also been proposed recently. GLO Bojanowski et al.) (017)) adopts the "encoder-less autoencoder" method to generate new images with non-adversarial loss function. IMLE & Malik (2018) proposed an ICP related generative model training approach. Later GLANN Hoshen Hoshen & Malik 2019) combines the advantages of GLO and IMLE, an embedding from image space to latent space was first found using GLO and then a transformation between an arbitrary distribution and latent code was computed using IMLE.
Mitigating Mode Collapsing Recently, Nagarajan & Kolter 2017 Khayatkhoei et al. 2018): ;
Xiao et al. (0088) also realize the training difficulties of GANs come from the approximation of discontinuous functions with continuous DNNs. By the gradient-based regularization, GDGAN Nagarajan & Kolter (2017) do relieve the mode collapse phenomenon of GANs, but mode mixture still exists. Khayatkhoei et al. (2018) proposes to use multiple GANs to overcome the mode collapse. Xiao al. (2018) proposed to embed the images into latent space according to Bourgain's 's theorem, and train the generator by sampling a Gaussian mixture distribution in the latent space instead of a unimodal Gaussian. The recently introduced normalized diversification by Liu et al. (0188) can also help overcome mode collapse successfully. However, all of them cannot solve the mode mixture well.
All these works Nagarajan & Kolter (2017); Khayatkhoei et al. 2018): : Xiao et al. (2018) explain that if the target data distribution has multiple modes, the transport map is discontinuous, but DNNs can only represent continuous mappings, the intrinsic conflict causes mode collapse.
3 COMPUTATIONAL ALGORIHTM
Our AE-OT model is summarized in Fig. it has two major compo- nents: i) (AE) An autoencoder is trained to encode (fe) the data manifold from the image space X' to the latent space Z, and map the data distribution to the latent code distribution; then the decoder I5 decodes the latent code back to the data manifold. ii) (OT) This module computes the optimal transportation map T from the noise distribution the latent code distribution. First, the Brenier potential is found by a convex optimization process according to Gu et al. (2016), whose gradient is the semi-discrete optimal transport map, where the target is the discrete set of latent codes of training
Overview of AE-OT Model
3

---

Published as a conference paper at ICLR 2020
Image Synce X
Lasent Space
Imnge Space x
Decoder
f
Catent Clde Distrthution
Extendeed f
Geuendor
oT Map
Images
Recon.
Genereted
Pragez
Imagee
Noise Diseshution
Figure 2: AF-OT model. AE: Se and Ig represent the encoding and decoding maps respectively, where and & are their corresponding network parameters. In the latent space Z, the latent codes are clustered into three different modes, represented marks with different shapes (i.e. disks, squares and circles). OT: The singular set between different modes is plotted with dashed lines. Finally, the generator of our model, which generates realistic images from random noise samples, is the composition of the extended OT map T and the decoding map 9c-
samples; then the transport map is pisce-wise linearly extended to global continuous map T, where the image domain becomes simplicial complex obtained by triangulating the above latent codes. Finally, the singularity set in the source domain is located and avoided when generating new samples. As result, given a random noise x, we can get the generated image by 95 e Txx).
Semi-Discrete OT Map Suppose the source measure μ (Gaussian uniform distribution) is ab- solutely continuous defined on a convex domain Ω C R4, the target domain is a discrete set, y= ( 1/,-- ... J. Y/ E Rd, the target measure is a Dirac measure, v -i=1 uzō(y vi), 1.2..... with the equal total mass the source measure, ((22) = Di=1 14. Under semi- discrete transport map T Ω + Y, a cell decomposition is induced Ω = U W, such that every x in each cell W; is mapped to the target yi. T : x E W, + y/. The map T is measure preserving, denoted as Tμ
v, if the p-volume of each cell W, equals to the y-measure of the image T(W) = Yt, p(W)) = 14. The cost function is given by c O x Y + R, where cfx, y) represent the cost for transporting unit mass from y. The total cost of T is given by la ca. (x)) = Σ Sw. c(x, yi (d)(()). Semi-discrete optimal transport map is the measure- prresrrepreserving map that minimizes the total cost, T arg min So c(z, d(z))d
When the cost function is the L distance c(x, 3) 1/2 x y Brenier's theorem claims that the semi-discred OT map is given by the gradient map of a niece-wise (PL) convex function, the so-called Brenier potential un. Ω - R. up (4) == max"_ 1 { ( where (x) = (x, y1) + h;; is the hyperplane corresponding to y: Y. As shown in Fig. aa). the projection of the graph of decomposes into cells W; (h),,, each cell (h(h) is the projection of the supporting plane (x)) The height vector h is the unique optimizer of the following convex energy under the condition that Σ; h4 0,
E(h) = / Σ w. (n)dn
Σ h2
(1)
where (n))) is the p--ooume of (n)... The convex energy E(h) can be optimized simply by gradient descend method with V VE = (o)hh)));)) -
The key is to compute the y-volume w;: of each cell (h).) which can be estimated using conventional Monte Carlo method. We draw N random samples from distribution, {x, mi.idd μ. Vj E J. the estimated y-volume of each cell (h) = # (j Iy, we can find W; in which zy E W; by i = arg max, { (z,, 31) + h4h. =
x. E (h))// Given
n. When N is large enough, w; (h) converges to (h). Then the gradient of the energy is approximated as VE ss (h) - D² Once the gradient is estimated, we can use Adam algorithm Kingma Ba (2055) to minimize the energy. Sampling of x: is independent of each other and finding the cell that
4

---

Published as a conference paper at ICLR 2020
mc(t) mpx
G10
411
Y18
V19
X11
12
610
Y14
sire
y2
(a)
(b)
Figure 3: Illustration of the proposed algorithm with two modes in Y. (a) Brenier potential and the corresponding power diagram. Each cell W; is mapped yi, which is the slope of , The red line in Ω gives the singular set. (b) The extended semi-discrete map. By computing the weighted center of each cell, and then triangulating the centers according to the power diagram, we get the PL map T(x) from Q Y. If the samples 2 is not in the triangles transverse the singular set, we map it to the corresponding T(x1) Y.
x is located only involves matrix multiplication and sorting. Hence the Monte Carlo method has a natural parallel computation implementation on GPUs.
The approximation error is proportional to the inverse of the square root the amount of Monte Carlo samples. Asymptotically the number of Monte Carlo samples increases exponentially with respect to the dimension d (see e.g. WWeed & Bach (2017)) This brings huge computational burdens. To find good balance between precision and speed, we adaptively adjust the number of random samples. In practice, we apply the following strategy: if the energy E(h) ceases decreasing for number of consecutive steps, we double the amount of Monte Carlo samples. The algorithmic details of semi-discrete OT map are summarized in Alg.
Algorithm Semi discrete OT Map
Algorithm 2 Generate latent code
(Ms)EC. empirical latent code dis- tribution TŽT E.ez 6y number Monte Carlo samples
Input: Latent codes
Input: Optimal transport map T(-), number of samples gen- angle threshold é.
N. positive integer a. Output: Optimal transport map T(.). Initialize = (A1, A..... h1z1) + O0.0......0 repeat
2: Output: Generated lasent code P.
2:
3: Compute by Monte Carlo method.
4: repeat
5: Sample x R. Find the smallest di + vertex around a
5: Generate N uniformly distributed samples (1r
[d(ir, C. ).d( df2r, )....... d(z,, 3).
Calculate V = (h) vz)""
6: Compute dihedral angles Hi, between sTiy and mty-
=
mean((h),
7: Select 0ig with 0ig. 6, result in it U,
Update by Adam algorithm with 0.9.,9 92 0.5. 9: if E(h) has not decrensed for a steps then
Ahandon 9: else Generate latent code T())) 2iHh Ax with
vkk H1,, >
N 4- N
11:
end if
Ag d= (z,, 22 E/E3o d= (e,
12: Converge
end
13: OT map T(-) + (Imax 9.) bz).
11: until Genernte new latent code
Piece-wise Linear Extension The remi-discrete OT map Vu, Ω + Y maps all x E Q to the latent codes of training samples {y:} I's and won't generate new samples. Therefore, we extend the semi-discrete OT map T = Vun to piccewise linear (PL) mapping T as follows. The projection of un in the source domain induces a cell decomposition of 2, of which each cell is of p-volume is and is mapped to the corresponding yi- By representing the cells by their jı-mass centers as = Swaas) zd)((x), we can get the point-wise map t : C H Vi. The Poincaré of the cell
decomposition induces triangulation of the centers C (c::: if W2 n W, A 6, then e; is connected with e, to form an edge |c,, cz)- Similarly, if Wi, n W; ... W., * 0, then there is a k-dimennsional simplex [Cig C.y.-.., G- Áll these simplices form triangulation of C (a simplicial complex), denoted as T(C") (the green triangles in the left of Fig. b)) We can triangulate Y in the same way obtain the triangulation T(Y) (the green triangles the right of Fig. 3))). Once random sample is drawn from the distribution μ4, we can find the simplex o in T(C) containing ær Assume the is defined as
simplex has d + 1 vertices {ca, C., ..., C2a }. the bary-centric coordinates of k= ARCig, and Σ o A = 1 with all d non-negative. Then the generated latent code of x
& >i under this piece-wise linear map is given by ?(x) Ek ARlli, (In Fig. b)), the green dot .:1 is
5

---

Published as a conference paper at ICLR 2020
mapped to be T(x1).. Because all of the y&8 are used to construct the simplicial complex T(Y) in the support of the target distribution, we can guarantee that no mode is lost.
In practice, the u-mass center is approximated by the mean value of all the Monte-Carlo samples inside Wh(h), 2 = Σxyew, 23/ # {zy= E Wi). where 2; ~ p. The connectivity information T(C) is too complicated to construct and to store in high dimensional space, thus T(C) is not explicitly built. Instead, we find the simplex a E TCC) containing x as follows: given a random point a E Ω evaluate and sort its Euclidean distances to the centers d(x, C, Suppose the first d + items are {d(z, C,)), d(x, C,2 C.,)...,, d(z,, &,a) }. then o is formed by {cr, J. The bary-centric coordinates Åig are estimated as A = =d='(x, (x, 2/ Σi= d may generate some spurious samples. To overcome it, we need further to detect the singular set.
1,2,- ................................ in the ascending order.
d''(r, (I, 2.2). However, this
Singular Set Detection According to Figalli's theory Figalli (2000); Chen & Figalli (2017), if there are multiple modes or the support of the target distribution is concave, there will singular sets E C Ω2 where the Brenier potential is continuous but not differentiable, making its gradient map, i.e. the transport map, discontinuous.
As shown in Fig. aa), the source distribution is uniformly defined on Ω2 and the target empirical distribution has two modes. There is one ridge (the red line) on the Brenier potential u/. whose projection is the singular set Σ (the red line in 12). Ω \ E consists of two connected components, each of them is mapped onto a single mode. Σ consists of codimension 1 facets of cells. If (i(h) n (h)
E, then the dihedral angle between Un is prominently large. Therefore, on the graph of Brenier potential, we pick the pairs of facets whose dihedral angles are larger than given threshold, the projection of their intersection gives co-dimension 1 cell in the singular set Σ. During the generation process, if a random sample x is around Σ. it will be mapped by T to the gaps among the modes. When generating new latent codes, we just abandon such samples. And this help our method prevent the mode mixture phenomenon.
supporting planes Th, and snj of
Given the extended OT map f(x). some of the polyhedrons transverse the singular set (the red lines of Fig b)). which means that different vertices of the polyhedron belongs to different mode. If the sample falls into such a polyhedron (the dotted red triangle), we just abandon it (as shown in Fig. 3b)), the red dot x2 is just abandoned). Specifically, given x, we can detect if it belongs to the singular set by checking the angles fr, between Tio and Ti k = 1, 1.2..... ..... d as 6i,, = (Wio Vi.))/ /ll9///llll l1ve, ll- If all of the angles 0g, is larger than a threshold 6, we say x belongs to the singular set and just abandon it. Or we just select a subset { J with 0z2- S 0, denoted as {ITi, k = O,1,...,ddy d1 Then we can compute A = dx. O22) // (x, and f(x) Σdi >a TC Intuitively, T(-) smooths the discrete function T(.) in regions where latent codes dense and keep the discontinuity of T(-) where latent codes are very sparse. In this way we avoid generating spurious latent code and thus improve the generation quality. The algorithm to generate new code is shown in Alg. 2na the
effect of threshold filtering is further investigated in Appendix 4.1
EXPERIMENTS
In order to validate that the proposed method can solve the mode collapse/mixture problems and generate controllable high quality images, several experiments are conducted.
The first experiment explores the influence of the angle thresholds for the singularity detection on synthetic dataset.
The second experiment focuses on toy sets, so that the complexity of the tasks can be manually controlled and the mode and quality of the generated samples can be accurately computed. [in et al. (2018) did a large-scale comparison with previous methods that explicitly proposed to mitigate mode collapse and thus established bascline for comparison. For consistent evaluation, set up experiment on the same benchmark dataset as theirs, and make the comparison.
In the last experiment, we run the proposed method mainly on 4 public datasets, MNIST LeCun & Cortes (2010), MINIST-FANSINN CelebA Zhang et al. (2018), just like the authors of Hoshen & Malik 2019) 9 Saajadi al. (2018) Lucic et al. (2018) did in their papers. Besides, the architecture of the decoder is the same as Lucic et al. (2018). in which the authors did large-scale study to evaluated the best performance of
Han Xiao & Vollgraf (2017). CIFAR- 10 Krizhevsky (2009) and

---

Published as a conference paper at ICLR 2020
different generative models including various GAN models and VAE, and the encoder is set to be the mirror of decoder.
4.1 SINGLE PARAMETER SELECTIVE INTERPOLATION
On synthetic datasets, effects of angle threshold filtering can be visually inspected. As illustrated in Fig. 4 number of mode is a monotonically increasing function with respect to angle threshold 0. Quality of generated samples is effected directly by choosing different 0. Generally, small encourages interpolation in between closely related real samples while too large will result in interpolation between samples from different modes, which might turn lower generation quality. On synthetic datasets, where modes are isotropic and different modes are clearly separable, an ideal that captures all modes while avoids generating low quality samples can be chosen within relatively wide band. For real world datasets of unknown modes, ideal Ô needs to be hand tuned as the separability of different modes depends largely on input data pattern and quality of the embedding map.
(a) Target Distribution
(b) i too small
(c) 0 too large
(d) Proper 9
Figure 4: Effect of increasing angle threshold 0. (a) shows target distribution. (b) and (c) shows AE-OT results when is too small (as in (b)) or too large (as in (c)). (d) shows a proper choice of that precisely captured and generalized all modes.
4.2 MITIGATION OF MODE COLLAPSE AND MODE MIXTURE IN SYNTHETIC DATASET
Since synthetic dataset consists of explicit distributions and known modes, mode collapse and the quality of the generated sample can accurately measured. We choose the same synthetic datasets and metrics as in Lin et al. (20T8). Specifically, we use 2D>ring and 2D-grid for test sets and Number of modes, Percentage of high-quality samples, reverse Kullback-Leibe metrics. Number of modes counts the amount of modes captured by samples produced generative model. Percentage of high-quality samples measures the proportion samples that generated within three standard deviations of the nearest mode. reverse KL divergence measures how well generated samples balance among all modes regarding the real distribution. In Lin et al. (2018). the authors evaluated GAN, ALI, MD and PacGAN on synthetic sets with above three metrics. Each networks trained under the same generator architecture with a total of approximated 400K training parameters. For AE-OT test, since the source space and the target space are both 2-dimensional, there no need to train any autoencoder. A two dimensional extended OT map is directly computed. Our results are included in table and benchmarks of previous methods are copied from Lin al. (0088) and Xiao et al. 2018). Generally speaking, the samples generated by the proposed method can not only cover all of the modes, the quality of them is also better than others.
(KL) divergence as evaluation
Besides, we experiment on stack MNIST dataset and CelebA dataset to further illustrate the perfor- mance of the proposed method, and the results are shown in Section C.
and C.2 in the Appendix.
4.3 QUANTITATIVE COMPARISON WITH FID
FID computed by: (1) extract the visual-meant meaual-meaningful features of both the generated and real images through the inception network, (2) fit the features in both the generated and real feature spaces with Gaussian distribution, and then (3) compute the distance between the two Gaussian distributions with the following formula FID = ||,,, Hg 11 + Tr(E, + Eg 2(E,E,) ) ). where f4r: My mean the means of the real and generated features, Σ, E, represent the variances of both distributions.
We report our results in Tab. (2). in which the compared data comes from Lucic et al. Hoshen & Malik 2019). In general, the proposed model achieves better than comparable scores to other
7

---

Published as a conference paper at ICLR 2020
Table I: Experiments on synthetic datasets. Under standard benchmark settings, AE-OT achieves best performances over an average of 10 independent experiment results in terms of modes captured, probability of high quality samples and reverse KL divergence. The mean values and standard deviations of the experiment results are reported here.
2D-ring
2D-grid
Modes (Max 8)
high quality
Modes (Max 25)
high quality samples
samples
reverse KL
reverse KL
GAN
6.3:10.5
98.211.2%
0.45:10.09
17.3
94.8=1.7%
0.70+10.07
ALI
6.6+0.33
97.6=004%
0.36=t0.04
24.1+0.4
95.7=00.%
0.14
MD
4.3
36.6+.88%
1.93:1.111
23.8+00.5
79.9=3.2%
0.18+0.03
PacGAN2
7.940.1
95.6112.0%
0.07+10.03
23.8:10.7
91.3++0.8%
0.13+00.04
PacGAN3
7.8+0.1
97.7+.03%
0.10+00.22
24.6:10.4
94.2++0.4%
0.06-10.02
PacGAN4
7.810.11
95.9+1.4%
0.07
24.8+002
93.6=0.6%
0.04
BourGAN
8.0+0..0
99.8+2.9%
4c-4:2e-4
24.9
95.9-10.2%
0.01
AE-OT
8.0++0.0
99.6+.03%
0.004++00001
25.0++..0
99.8++.2%
0.007++0.002
Table 2: Quantitative comparison with FID
Adversarial
Non-Adversarial
Reference
Dataset
NS GAN
LSGAN
WGAN
BEGAN
VAE
GLANN
AE
Ours
MNIST
6.810..5
7.830.66
6.740..4
13.141.0
23.8:0.6
8.6:0.11
3.5
6.2.10.2
Fansion
26.5H1.6
30,7+22.
21.541.66
22,9+0.99
58.7+1.2
13.0+0.1
4.7
10.1:00.3
CIFAR-10
58.5+119
87.1
55.241..3
7..4.11.6
65.4.10.2
46.54:0.2
28.2
3..
ColebA
55.0
53,94228
41.3+2.0
38.9
85.743.8
46.340.1
67.5
68.41 0.5
state-of-the-arr generative models. Theoretically, the FID scores of proposed generative models should be close to that of the pre-trained autoencoders, and this is also validated in our experiments.
The autoencoder architecture the limited capacity. But the FID score of the generation model is still approach to the autoencoder. In order to verify that with appropriate capacity of autoencoder, the proposed model works. We use the generator of DCGAN Radford et al.] (2016) as the decoder of the autoencoder, then the reported FID score is 28.6, outperforming other models. Also, some of the generated images are displayed in Fig. 7 of the Appendix.
use here cannot find a good encoding for the CelebA dataset due
also display the generating results for the four dataset in Fig. [ includes the original images, the best generating results of Lucic et al. (2018), including various GANs and VAE, the results of Hoshen Malik (2019) and ours, row by row.
5 CONCLUSION
This work gives theoretic explanation for mode collapse/mixture by Brenier's theory and Figalli's regularity theory optimal transport maps. When the target measure has concave support, the OT map is discontinuous on the signular sets. But DNNs can only represent continuous functions, this conflict causes the both problems. In order to solve this problem, the AE-OT model proposed by separating manifold embedding and measure transformation. The former step is computed using an autoencoder, the latter is carried out using the extended semi-discrete model is tested thoroughly and extensively by both synthetic and real data sets. The experimental results validates the discontinuity the OT maps and demonstrate the advantages comparing to the state-of-the-aus
map based on GPUs. The
Acknowledgements
The project is partially supported by the National Natural Science Foundation of China (61936002, 61772105, 61720106005), NSF CMMI-1762287 collaborative research: computational framework for designing conformal stretchable electronics, Ford URP topology optimization of cellular mesostruc- tures' nonlinear behaviors for crash safety, and NSF DMS-1 1737812 12 collaborative research: ATD: theory and algorithms for discrete curvatures network data from human mobility and monitoring.
8

---

Published as a conference paper at ICLR 2020
5
2
3
3
3
9
3
2
0
8
/
5
5
9
3
8
1
8
7
4
0
2
3
3
6
9
0
3
6
3
0
6
5
8
9
7
3
3
9
6
7
G
3
6
4
Z &
2
5
9 0
0 6
3
7
1
3
2
4 477 6 5 8
Q
0
7
5
3
7
8
9
6
2
8 1 9 3 0 6
3
8
4
4
6
O
4
3
1
0
3
8
0
9
7
0
3
4
3 4 8
/
7
8
6
8
7 2 2
9
7
3
9
O
9
6 30 9
5
O
2
6
7
0 6
6
5 1 5 9
6 1 9 O 9 2
7
2
0
5
5
6
5
0
7
0
7
4
5
6
7 6
7
4
5
4 4
6
2
6
5
3
Figure 5: The comparison between the proposed method and the state-of-the thrt on MNIST LeCun & Cortes 2010). Fashion MNISTHan Xiao & Vollgraf (2017). Cifar Krizhevsky (2009) and CelebA Zhang et al. (2018). The first row shows some of the real images in each dataset. The second row corresponds the best results of Lucic et al. (2018); The third row gives the results Hoshen & Malik 2019); then we display our generating results in the last row.
REFERENCES
Martin Arjovsky. Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML, pp. 214-223, 2017.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1-2)1151-175, 2010.
Piotr Bojanowski, Armand Joulin, David Lopez- Paz, and Arthur Szlam. Optimizing the latent space of generative networks. arXiv:1707.05776 2017.
Y. Brenier. Polar decomposition and increasing rearrangement vector fields. R. Acad. Sci. Paris Sr. / 305 808, 1987.
Y. Brenier. Polar factorization and monotone rearrangement of vector-valued functions. Comm. Pure Appl. Math., 44(4):375 417, 1991a.
Yann Brenier. Polar factorization and monotone rearrangement of vector- functions. Commu- nications on pure and applied mathematics, 44(4):375-417, 1991b.

---

Published as a conference paper at ICLR 2020
L.A. Caffarelli. A localization property of viscosity solutions to the monge-amper equation and their strict convexity. Ann. of Math., 131(2):129-1) 114,, 1990a.
L.A. Caffarelli. Interior w2.p estimates for solutions ofthe monge-ampére equation. Ann. of Math., 131(2):135-150 1990b.
L.A. Caffarelli. Some regularity properties solutions of monge ampére equation. Comm. Pure Appl. Math., 4468-9);955-96, 1991.
Shibing Chen and Alessio Figalli. Partial w2.p regularity for optimal transport maps. Journal of Functional Analysis, 272:4588 4605, 2017.
Alessio Figalli. Regularity properties of optimal maps between nonconvex domains in the plane. Communications Partial Differential Equations, 35(3)466--47, February 2010.
Ian Goodfellow. Nips 2016 tutorial: Generative adversarial networks. arXiv: 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, pp. 2672-2680, 2014.
David Xianfeng Gu, Feng Luo, jian Sun, and Shing-Tung Yau. Variational principles for minkowski type problems, discrete optimal transport, and discrete monge-ampère equations. Asian Journal of Mathematics, 2016.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. In NIPS, pp. 5769-5779, 2017.
Kashif Rasul Han Xiao and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. http:://arii.orgaa////08880 707.077777777777 2017.
Yedid Hoshen and Jitendra Malik. Non-adversarial image synthesis with generative latent nearest neighbors. CVPR, 2019.
Mahyar Khayatkhoei, Maneesh K. Singh, and Ahmed Elgammal. Disconnected manifold learning for generative adversarial networks. Advances in Neural Information Processing Systems, 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. 2015.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv: 1312.61 14, 2013.
Jun Kitagawa, Quentin Mérigot, and Boris Thibert. Convergence a newton algorithm for semi- discrete optimal transport. JOURNAL OF THE EUROPEAN MATHEMATTICAL
2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Tech report, 2009.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http:/ /est:://yann
lecun. wwt.coocceee.eeeee httmm/xxb/nniiii
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86(1) (6(1)):277823222 1998.
Na Lei, Kehua Su, Li Cui, Shing-Tung Yau, and David Xianfeng Gu. A geometric view of optimal transportation and generative mode. Computer Aided Geometric Design, 2017.
Na Lei, Dongsheng An, Yang Guo, Kehu Su, Shixia Liu, Zhongxuan Luo, thing-Tung Yau, and Xianfeng Gu. A geometrie understanding of deep learning. Engineering, 2020.
Ke Li and Jitendra Malik. Implicit maximum likelihood estimation. arXiv:1809,0m 00000009087,.. 2018.
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power of two samples in generative adversarial networks. In Advances in Neural Information Processing Systems, pp. 1505-1514, 2018
Shaohui Liu, Xiao Zhang, Jianqiao Wangni, and Shi Jianbo. Normalized diversification. In CVPR, 2018.
10

---

Published as a conference paper at ICLR 2020
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. In Advances in neural information processing systems, pp. 698-707, 2018.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial 2015.
autoencoders.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In ICML, pp. 3478-3483 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
Vaishnavh Nagarajan and J. Zico Kolter. Gradient descent gan optimization is locally stable. In Advances in Neural Information Processing Systems, 2017.
Gabriel Peyré and Marco Cuturi. Computational Optimal Transport. arxiv: 1803.00567, 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In JCLR, 2016.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv: 1401 2014.
Mchdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. arXiv:1806.000355, 2018.
Justin Solomon. Optimal Transport on Discrete Domains. https://arxiv.orr/abb/// 1001 801107775, 2018.
J B Tenenbaum, V Silva, and J C Langford. A global geometric framework for nonlinear dimension- ality reduction. Science, 29065500) 2000.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders.
In ICLR, 2018
Cédric Villani. Optimal transport: old and new. Springer Science Business Media, 2008.
Jonathan Weed and Francis Bach. Sharp asymptotic and sinite-sample rates of convergence of empirical measures in wasserstein distance. D00iiv:v7770000877 2017.
Chang Xiao, Peilin Zhong, and Changxi Zheng. Bourgan: Generative networks with metric embed- dings. In Advances Neural Information Processing Systems, 2018.
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. From facial expression recognition to interpersonal relation prediction. International Journal of Computer Vision, 2018.

---

Published as a conference paper at ICLR 2020
A BRENIER'S s THEORY
In this subsection, we briefly introduce the basic concepts and theorems in optimal transport theory, which comes from Brenier theory Villani 2008); Brenier [1987 1991a), and discrete theory Gu et al. (2016).
Optimal transport Problem Suppose X,Y C Id are two subsets of n-dimensional Euclidean space, n1 are two probability measured defined on X and Y respectively, with equal total measure, (X) v(Y). A map T : X + Y is measure preserving. denoted as Tи = v. if for any measurable set B C Y, (B) = v(B). Given a cost function c(z, y) X Y + Rgo- indicating the cost of moving each unit mass from the source the target, the total transport cost of the map 7' is defined to be Jx c(a, (d)))d(())
The Monge's 's problem of optimal transport arises from finding the measure-preserving map that minimizes the total transport cost.
(MP) We(m,v) = min / c(a,T(=))dp(x))
Typ=u
The solutions to the Monge's problem is called the optimal transport map, whose total transport cost called the Wasserstein distance between μ and v,, denoted (,,
Brenier's Approach Brenier Brenier 1987 1991a) proved the following theorem:
Theorem (Brenier Brenier 1987 1991b)). Suppose and Y are the Euclidean space IR and
the transport cost is the quadratic Euclidean distance c(a, y) = 1/2[|> yl1². Furthermore μ is absolutely continuous and y and v have finite second order moments, then there exists convex function u X + R. the so-called Briener potential, gradient map Vu gives the solution to the Monge's 's problem. The Brenier potential is unique up to constant.
Brenier's polar factorization theorem claims that: for any measure preserving map T사 = v, T can be uniquely decomposes into the forms T = Vu where X + X is volume preserving map and Vu is the optimal transport map under L cost. Therefore, the regularity of T can be determined by that of Vu.
Brenier theorem can be directly generalized to discrete target measure. Suppose the source measure f is defined on compact convex set Ω2 the target measure Σ m10(y - y.), u(l)) = Σ, pt. The discrete Brenier potential is piecewise linear function.
Discrete Brenier's Theorem
un (x) mäx (ma, (x)) mäx (ar, ve) + hz) 1=1
s=1
As shown in Fig. 3(a), the projection of the Brenier potential induces a cell decomposition of S2 each cell Wh(h) == [p E Ω Vu, (p) ),), whose ju-measure is denoted as w:(h))
Theorem 2 (Discrete Brenier Theorem Gu et al. (0166). For any V1, V2...., Un > (S2), there exists h (h1, h2,.... .... h.,) R", unique up adding constant (c, (c,c,.... C). so that wx(h) = vi. for all i. The vector h is the unique minimum argument of the following convex energy
with Σ =
E(h)
/
h Σ
w: (n)d))
Σ havs
defined open convex set H. {h- R wh(h) 0. 1.2..... n... Furthermore, Vun
minimizes the quadratic cost Jn || - (dd)()))((()) among all transport maps T+ V. The
gradient of above energy is given by EE()) (wi (h) - V1, we (h) 12, wn)h) - vn)) The
Hessian of the energy is given by
Ow: Th
p(W, n W,,) Ow:
Ow;
(5)
11 1 ah
Th
The optimal transport map can be obtained by convex optimization. Furthermore, the optimization
can be carried out using Newton's 's method. The global linear convergence rate guaranteed by the following theorem:
12

---

Published as a conference paper at ICLR 2020
Theorem 3 (Kitagawa-Mérgge Meiieot-Thhibrrt Kitagawa et al. (019 Assume the cost function is quadratic distance, ii has convex support and also that (i) The probability density of μ. is (9t) for a in (0; 1]. (ii) has positive Poincaré- Poincaré-Wirtinge constant. Then the Newton algorithm for semi discreee optimal transport converges globally with linear rate and locally with rate a.
Though our method the gradient descend method applied, the above theorem also ensures its convergence because of the convexity of the energy function E(h) we adopted.
B FIGALLI'S THEORY
In this section, we show the fact that even for the case of single mode, the transport map may still be discontinuous, which will cause the instability of the training process GANs. The arguments are mainly based on the regularity theory of transport maps developed by Figalli Chen & Figalli (2017); Figalli (2110) and so on.
According to Brenier's Theorem Brenier (1987
1991a), any transport map can be decomposed into measure preserving map and a solution to the Monge- Monge-Ampére equation, which the optimal transport map under the L cost function. Therefore, the continuity of the transport map be reduced to the regularity (smoothness) of the solution to the Monge-Ampére equation. When the support of the target measure is convex and the density functions differentiable; otherwise the target domain is not convex, Figalli showed the map is discontinuous, and gave precise description of the singularity theory, and conduct an experiment using CelebA data set to show the existence of the singularity set, hence demonstrate the fact that the transport maps computed GANs are discontinuous.
smooth, Caffarelli showed the map is
In this section, we briefly introduce Figallis'
B. CONVEX DOMAINS CAFFARELLI THEOREM
Let Ω and A are two bounded open sets in R", and let f: : R and g: RR" R be two positive functions such that f R Ω. = in R" A, and
[
-1
According to Brenier's Theorem Brenier (1987 1991a), there exists globally Lipschitz convex function 3 : R." R such that Vi+./= = g and and Vip(x) for aae a E R.". We say yp weakly solves the Monge-Ampére equation
S
dell del(D' 9) go V
in R
(6)
together with the boundary condition (R())) C A. cp is called the Briener potential.
As shown by Caffarelli [9], if is convex, then yo is strictly convex, and it solves the Monge-Ampére
equation 6 The regularity theory has been estabilished (see Caffarelli (1990a 1991 )) such as
if S. 1/A for some À > 0, then & E C
(.l.a(A) (12).
2. if f o (It) and g E Clao( then & E C (Ω)), (k: e (0,
B.2 NON-CONVEX DOMAINS FIGALLI THEOREM
However, if A is not convex, the regularity of the Brenier potential can not be guaranteed. Figalli gave examples in Figalli (2010), such that
1. Ω is convex, A is simply connected, but non-convex;
2. the density functions f and smooth, f E C (!2) and g E C (A):
3. the Brenier potential & C (S)), the transport map Vip is not continuous.
In this scenario, the transport map can not be learned using DNNs, and training process unstable or the GAN model generates unrealistic samples.
13

---

Published as a conference paper at ICLR 2020
Figalli's construction Let 4 R + R be convex function. Its subdifferential at a point is defined by
@p(x)) (v (p())))) ((x) +y (z RR).
differentiable at a point x: if and only if 0p(x)) a singleton. Figalli decomposes the set of non-differentiabiiiy points according to the dimension of the singular set:
Σx(e) == fir E I" dim((pp) For any l: = O,..., n, the set (p) (n -eetttifie The set of reachable subgrudients at a as
k), ...
v.p lim Ve((x))ax E Eo, + 40+000
is known that the convex hull of Vx)(=), coincides with 0s((z).
Theorem 4 (Figalli), Assume that there exists À > O such that X S. A, and that 02 and 0A are continuous. Then yp is strictly convex inside Q. Moreover there exist two open sets St' Ω and N' C A, with [ \ Dt L² N') O, such that yp E (S'), V is a bi Ri-Hölder homeomorphism between Q' and A' and yp is an Alexandrov solution of inside Q. . In particular, Caffarelli's 's regularity theory for strictly convex Alexandrov solutions of the Monge-Ampére equations applies to y inside Dt'
1/A in
Figalli studies the singular set of ip in Ω i.e. the of points Ω where şo is differentiable, denoted as Sing. Figalli shows the singularity set has the following characterization,
Sing {a E (2)p(()) NA = 0, V.)(x)) aA, 0p(x:) & A}.
it can be decomposed into connected components Sing U.S., For planar case.
Theorem 5 (Figalli Singularity Set). The number of connected components of Sing is at most countable. Moreover:
1. either S; coincides with an isolated point {ri) for some Ii E Q and in this case the boundary of d(ri) entirely contained inside 0A (so that de(lri) completely fills a hole in A);
2. or S can be written as disjoint union as follows:
S: U Tig
where "lis : 1ij + Sing are embedded Lipschitz curves parameterized by are-lengh, Iig is interval.
B.3 ELEMENTARY EXPERIMENTS
We have designed a numerical experiment to verify Figalli's theorems in low dimensional case.
Fig. 6 shows another computational result, which demonstrates the singularity structure in Figalli's theorem. The source domain Ω is the unit disk, the target domain A is with complicated geometry. The singularity the optimal transport map satisfies the description of Figalli's theorem
Σ
U The De i=0
U r.
j=0
0x(xx) fills the hole on A. For any interior point E m. ap(p) is a line segment connecting two points the boundary of A.
C ADDITIONAL EXPERIMENTS
C.1 STACK MNIST EXPERIMENT
Experiments varisou GAN models on stacked MNIST dataset are consistent with Lin et al. (2018). For AE-OT model, the architecture shown in table and table 4 with the decoder
14

---

Published as a conference paper at ICLR 2020
71
0e
Ω
Figure 6: Figalli's example: Singularity structure of an optimal transport map, Fig.3.2 in Figalii (2010). Lei et al. (2020).
Table 3: Encoder architecture for stack MNIST
layer Input x ~ Paata 28*28*3
number of outputs
kernel size
stride
BN
activation
Convolution 14*14*16
5*5
2
LeakyReLU
Convolution 7#7*32
5*5
2
Yes
LeakyReLU
Convolution
4"4"64
5*5
2
Yes
LeakyReLU
Convolution
2*2"128
5*5
2
Yes
LeakyReLU
Fully connected
100
architecture same the consistent generator architectures in GANs, and encoder having mirrored architecture.
We test diversity of generated samples from AE-OT method stack MNIST dataset that consists of 128,000 samples 1,000 modes with each sample stacking three handwritten digit images from MNIST dataset LeCun et al. (1998). Number of modes counts the amount of modes captured by samples produced a generative model. The reverse KI. divergence is computed by first assign each samples to their nearest mode, and compute the KI. divergence between histogram of sample count on each mode and the histogram of real data. We choose angle threshold Ô 0.5 for AE-OT method. Details of network architectures are listed in supplementary materials. Experiments results are summarized in table which show our method achieves best performance in terms of modes captured and reverse KI. divergence on stacked MNIST dataset.
C.2 CELEBA EXPERIMENT
we evaluate our method on CelebA dataset by measuring collision probability batch of 1024 generated images of size 64-by-64. If pair of identical images appear, collision is declared, and thus higher collision probability means lower generation diversity. The same metrie has been used in Lin et al. (2018) for evaluation of PacGAN. To make consistent comparison, we design our autoencoder network with encoder having the same architecture as previous work and decoder having a mirrored architecture of encoder. Angle threshold Ó chosen to be 0.7 for AE-OT test. Results are listed in table 6 with corresponding images be downloaded here Results have shown that our method achieves best result in terms of probability of collision. Autoencoder network structures can be found table 7 and
15

---

Published as a conference paper at ICLR 2020
Table 4: Decoder architecture for stack MNIST
layer
number of outputs
kernel size
stride
BN
activation
Input 2 Platent
100
Fully connected
2*2*128
Yes
ReLU
Transposed Convolution
4*4*64
5*5
Yes
ReLU
Transposed Convolution
7*7*32
5*5
2
Yes
ReLU
Transposed Convolution
14*14
5*5
Yes
ReLU
Transposed Convolution
28*28*3
5*5
Tanh
Table Experiments stacked MNIST. Results have shown that method achieves best results
in terms of mode captured and reverse KL. divergence. (*) In WGAN, PacWGAN and AE-OT experiments, number of feature maps each network layer a quarter of those in other experiments.
Stacked MNIST
Modes
KL
DCGAN
99.0
3.40
ALI
16.0
5.40
Unrolled GAN
48.7
4.32
VEEGAN
150.0
2.95
MD
24.5 + 7.67
5.49 + 0.42
PacDCGAN4
1000.0 0.00
0.07 0.005
WGAN(*)
314.3 + 38.54
0.170
PacWGAN4(3)
965.7 1 19.07
0.42 1: 0.094
AE-OT(*)
1000.0 0.0
0.03 + 0.0008
C.3 LINEAR INTERPOLATION IN THE LATENT SPACE
Given any two images in the dataset, we can find the images between them by linear interpolation in the noise space because the one to one correspondence between μ mass centers in the noise space and the images in the dataset is provided by the proposed algorithm. For other generation models, though the interpolation can be done successfully in the noise space, they cannot find the correspondence from the noise space and the image space. The results of the linear interpolation are shown in Fig. g
16

---

Published as a conference paper at ICLR 2020
Table 6: Probability of identical images in batch of 1024 generated images from DCGAN, PacGAN2
and AE-OT. Results have shown that our method achives best result in terms collision probability CelebA dataset.
Discriminator size
Probability of collision
(Decoder size)
DCGAN
PacDCGAN2
AE-OT
273K
0.33
4x273K
0.42
0
16 273K
0.86
25 x273K
0.65
0.17
Table 7: Encoder architecture in CelebA experiment
layer
number of outputs
kernel size
stride
BN
activation
Input x ~ Paata
64*64*3 d2*
Convolution
444
2
LeakyReLU
Convolution
16*16*dim
f*2
444
2
Yes
LeakyReLU
Convolution
8*8 f44
4%4
2
Yes
LeakyReLU
Convolution
di4
4+4
2
Yes
LeakyReL.U
Convolution
100
4*4
1
Table 8: Decoder architecture in CelebA experiment
layer
number of outputs
kernel size
stride
BN
activation
Input Piatent
100
Transposed Convolution
4*4*dim_ f*8
Transposed Convolution
8*8 f#4
4
Yes
ReLU
Transposed Convolution
16*16*din f*2
4
Yes
ReLU
Transposed Convolution
32*32"dim_
494
Yes
ReLU
Transposed Convolution
64"64"3
4"4
Tanh
17

---

Published as a conference paper at ICLR 2020
2
7
Figure 7: The generated human faces with the architecture originated from DCGAN Radford et al. 2016
18

---

Published as a conference paper at ICLR 2020
Figure 8: The linear interpolation between given two faces in the dataset.
19


Reviews:
[{"review_id": "ryetcUV7iH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with an important problem of mode collapse and mode mixture. In order to\ntackle the both problems, the paper proposes to separate the manifold embedding and the\noptimal transportation problems; the first part being carried out using an autoencoder to map the\nimages onto the latent space and the second part is accomplished using a GPU-based\nconvex optimization to find the discontinuous transportation maps.\n\nI have some doubts about moving from the \"semi-discrete OT map\" to the piece-wise linear extension. The illustration in Fig. 3, and implicit in all the explanation charts is the fact that discontinuity can be found by a linear separation. This seems to be an extremely simplifying assumption, which leads to not so great visual results from the paper. Although the numerical results seems promising, I feel that fewer images, but larger in size, and analysis of mode collapse phenomenon in real images would have been much better.\n\nSingular set detection seems to be the most tricky part in this paper, which should have been explained further. The Simplex projection assumption, renders this part not that tricky, but that is where I feel the biggest doubt about this paper lies.\n\nThe authors themselves mention the need for a high quality auto encoder model to encode celebA dataset, which has been improved upon by numerous other papers, the claims seems not too strong. Also, the method does not have any adversarial training and hence, it studies the GAN idea from only fixing the generator point of view. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1e5aB-z9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "General Comments:  The generator in Generative Adversarial Networks (GANS) computes an optimal transportation from the noise distribution to the data distribution.  However, such maps are in general discontinuous.  Since deep neural networks can only represent continuous maps, this brings two problems: mode collapse and mode mixture. This paper approaches both problems using Figalli's regularity theory. They separate the manifold embedding (here an autoencoder maps input data to a latent space) from the optimal transportation (this map is found by convex optimization). Composing these two steps yields the proposed method. Their method basically avoids representing discontinuous maps by the generator. Empirically, the proposed method performs similar or better than state-of-the-art.\n\nI think the idea of the paper is nice, and an interesting perspective  on GANs is presented. A new method is proposed. The numerical contributions are certainly significant. Therefore, I believe the paper deserves publication.\n\nNevertheless, I have some comments below.\n\n1) Although this paper brings a new perspective, based on optimal transport theory, as far as I can understand this paper does not establish formal new results. Thus I think some strong claims about providing deep theoretical explanation should be more moderate. In essence, it seems that the paper verifies *numerically* (in section B.3) that Figalli's theorem (stated in Appendix B) holds in this context.\n\n2) This is just a suggestion. I think in some parts a lighter notation and a more intuitive explanation could help.\n\n3) After Eq. (5) in the Appendix the authors mention Newton's method, and Thm 3 is also specific to Newton's method. Then they mention that *Gradient Descent* is used (and in the main part of the paper they mentioned Adam). This is confusing. All these algorithms are different, and Newton's method does not imply convergence results for gradient descent. I don't see how Thm 3 is relevant.\n\n4) This is a simple doubt. To avoid non-differentiability of the gradient, the OT step computes the Brenier potential and is able to locate the singularities. I wonder if using a simpler approach through optimization for nosmooth problems (such as Moreau envelopes or proximal methods) could resolve this issue? In the negative case, why not?\n\n5) Some Minor comments:\n1. Define OT in the abstract (Optimal Transportation?) \n2. What is AE? (not defined also; Auto Encoder?)\n3. There are lots of typos through the text, such as missing \"the\", \"a\", etc. \nand a couple mispelled words. I suggest the authors proofread the draft\nmore carefully.\n4. pp. 4 ... what is a \"PL convex function\". PL is not defined.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SylWjepycS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contributions:\n1. This paper proposes a new problem in GAN distribution mapping: the concavity of support problem.\n2. This paper provides a solution to the concave support together with mode collapsed problem in GAN, via a discrete-continuous optimal transport model, given some post-processing techniques to rule out \"singular points\".\n3. Empirical results show the effectiveness of the proposed method.\n\nTo summarize their method. First, they fit a good auto-encoder model to get embeddings for the observed data as an empirical distribution \\nu on space Z. Second, they use a semi-discrete OT to map a noise distribution \\mu to \\nv. Since OT will be aware of all modes in \\nu, singular points can be detected by checking the angle between \"shards\" and those points that are around the \"ridge\" should be rejected. Thus, the proposed method could handle both the concave support problem and the mode collapse problem.\n\nMy concern is whether the proposed method is overkill because the singular point detection can be very tricky and relies on heavy linear programming. Could you explain why not using the following substitute: \nStep 1. Fit an auto-encoder just as you did in the paper and get an empirical distribution \\nu.\nStep 2. Fit a Gaussian mixture model on \\nu and do model selection over # clusters.\nStep 3. Sample from the Gaussian mixture model to generate fresh images.\n\nSince this method relies on a high-quality auto-encoder model, it is hard to say this paper makes progress in fixing the GAN's mode collapsed problem. Besides, the paper does not involve an adversarial training module. So I will not treat it as a satisfactory improvement over GAN. Overall, the proposed problem in GAN indeed exists. But the solution seems to deviate from the goal the paper aim to achieve.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a different perspective on the mode collapse and mode mixture problems in GAN based on some recent theoretical results. 

This is an interesting work. However, two reviewers have raised some concerns about the results and hence given a low rating of the paper. After reading the reviews and the rebuttal carefully I feel that the authors have addressed all the concerns of the reviewers. In particular, at least for one reviewer I felt that there was a slight misunderstanding on the reviewer's part which was clarified in the rebuttal. The concerns of R1 about a simpler baseline have also been addressed by the authors with the help of additional experiments. I am convinced that the original concerns of the reviewers are addressed. Hence, I recommend that this paper be accepted. 

Having said that, I strongly recommend that in the final version, the authors should be a bit more clear in motivating the problem. In particular, please make it clear that you are only dealing with the generator and do not have an adversarial component in the training. Also, as suggested by R3 add more intuitive descriptions to make the paper accessible to a wider audience.


Decision: Accept (Poster)