Improving Multi-Manifold GANs with a Learned Noise Prior
Authors: Matthew Amodio
Smita Krishnaswamy
Venue:  (2020)
Keywords: GAN
generative adversarial network
ensemble

Abstract: Generative adversarial networks (GANs) learn to map samples from a noise distribution to a chosen data distribution. Recent work has demonstrated that GANs are consequently sensitive to, and limited by, the shape of the noise distribution. For example, a single generator struggles to map continuous noise (e.g. a uniform distribution) to discontinuous output (e.g. separate Gaussians) or complex output (e.g. intersecting parabolas). We address this problem by learning to generate from multiple models such that the generator's output is actually the combination of several distinct networks. We contribute a novel formulation of multi-generator models where we learn a prior over the generators conditioned on the noise, parameterized by a neural network. Thus, this network not only learns the optimal rate to sample from each generator but also optimally shapes the noise received by each generator. The resulting Noise Prior GAN (NPGAN) achieves expressivity and flexibility that surpasses both single generator models and previous multi-generator models.

Under review as a conference paper at ICLR 2020
IMPROVING MULTI-MANIFOLD GANS LEARNED NOISE PRIOR
WITH
A
Anonymous authors Paper under double-blind review
ABSTRACT
Generative adversarial networks (GANs) learn map samples from noise dis- tribution to a chosen data distribution. Recent work has demonstrated that GANs are consequently sensitive to, and limited by, the shape of the noise distribution. For example, a single generator struggles to map continuous noise (e.g. a uniform distribution) to discontinuous output (e.g. separate Gaussians) or complex output (e.g. intersecting parabolas). We address this problem by learning to generate from multiple models such that the generator's 's output is actually the combination of several distinct networks. We contribute novel formulation of multi-generraor models where we learn a prior over the generators conditioned on the noise, pa- rameterized by neural network. Thus, this network not only learns the optimal rate to sample from each generator but also optimally shapes the noise received by each generator. The resulting Noise Prior GAN (NPGAN) achieves expressivity and flexibility that surpasses both single generator models and previous multi- generator models.
INTRODUCTION
Learning generative models of high-dimensional data of perpetual interest, its wide suite of applications include synthesizing conversations. creating artwork, or designing biological agents (Bollepalli et al., 2017: Tan et al. 2017: Blaschke al., 2018). Deep models, especially generative adversarial networks (GANs), have significantly improved the state of the art at modeling these complex distributions, thus encouraging further research (Goodfellow et al. 2014). Whether implicitly explicitly, works that GANs make crucial modeling decision known as the mani- fold assumption (Zhu et al. 2016 Schlegl et al. 2017 7 Reed et al. 2016). This is the assumption that high-ddmensional data lies single low-dimensional manifold which smoothly varies and where local Euclidean distances in the din-dimesiional space correspond to complex transformations in the nigh-dimensional space. While generally true in many applications, this assumption does not always hold (Khayathhoei et al. 2018).
For example, recent work has emphasized situations where the data lies not one single manifold, but on multiple. disconnected manifolds (Khayatkhoci et al. 2018 Gurumurthy et al. [017 Hoang al. 2018). In this case, GANs must attempt to learn a continuous cover the multiple manifolds, which inevitably leads to the generation of off-manifold points which lie in between (Kelley 2017).
The generator tries minimize the number of these off-manifold points, and thus they are gener- ally just small fraction of the total generated distribution. As such, they barely affect the typical GAN evaluation measures (like Inception and FID scores for images), which measure the quality of the generated distribution as a whole. Thus, this problem is usually ignored, as other aspects are prioritized. However, some applications, the presence of these bad outliers more catastrophic than slight imperfections in modeling the most dense regions of the space. For example, consider the goal of an artificial agent acting indistinguishably from a human: the famous Turing Test. In- correctly modeling sentence density by using given sentence structure 60% of the time instead of 40% of the time is relatively harmless. However, generating a single gibberish sentence will give away the identity of the artificial agent.
Moreover, there are serious concerns about the implications this has for proofs of GAN conver- gence (Mescheder al. 2018). These works address the problem of disconnected manifolds by

---

Under review as a conference paper at ICLR 2020
z ~ U(-1 1)
G
NPGAN (Ours)
NP
G
Real/Fake
T
G
The Noise-Prior GAN (NPGAN) architecture. Unlike previous work, the NP network learns a prior over the generators conditioned on the noise distribution This allows it to both control the sampling frequency of the generators and shape the input appropriate to each one, in an end-to-end differentiable framework.
Figure 1:
simultaneously training multiple generators and using established regularizations (Chen al. 2016) to coax them into dividing up the space and learning separate manifolds.
Methods for getting multiple generators generate disconnected manifolds can be divided into two categories: (i) imposing information theoretic losses to encourage output from different generators be distinguishable (Khayatkhoei et al.] 2018 [Hoang et al., 2018) (ii) changing the initial noise distribution to be disconnected (Gurumurthy et al. (017]) Our approach falls into the second cat- egory. Previous efforts to change the noise distribution handle disconnectedness has exclusively taken the form of sampling from a mixture of Gaussians rather than the typical single Gaussian (with sampling fixed and uniform over the mixture).
Our approach differs significantly from those previously. We use multiple generators as before, but instead of dividing up the noise space into factorized Gaussians and sending one each generator, we let an additional neural network determine how best to divide up the noise space and dispatch it each generator. This network learns prior over the generators, conditioned on the noise space. Thus, we call our additional third network noise-prior (NP) network. Previous methods have modeled the data with noise z and generators G; as p(G,(z) G2(2), with p(G,|z) = p(G,) = p(G,) Vi,j. We instead propose framework to incorporate richer p(G.l2) into the generator. This framework is entirely differentiable, allowing us to optimize the NP network along with the generators during training.
We note that with this strategy, we significantly increase the expressivity of each generator over the previous disconnected manifold models. By dividing up the space into four slices 8; and sending to the first generator and 84 to the second generator, we can generate four disconnected manifolds with just two generators. Previous work would have devote precisely four generators to this task, with degradation in performance if fewer or more generators are chosen for the hyperpa- rameter. Here, the prior network learns divide the noise space appropriately for whatever number of generators is chosen, and is thus more expressive as well as more robust than previous models.
Moreover, much existing work has exclusively framed the problem as, and tailored solutions for, the disconnected manifold problem. Our approach is more generalized, addressing any mis- specification between noise distribution and the target distribution. This means that our approach does not become redundant or unnecessary in the case of single complex manifolds, for example.
Our contributions can be summarized as:
1. We introduce the first
ensemble to learn prior over the noise space, using a novel soft, differentiable loss formulation.
2. We present
celli-ggenerttr method that can learn sample generators in proportion to the relative density of multiple manifolds.
3. We show how our model not only improves performance on disconnected manifolds, but also complex-but-connecte manifolds, which are more likely arise in real situations.
RELATED WORK
Several previous works have included multiple generators, mixing and matching few commonly used features. Some use completely distinct generators (Khayatkhoei et al. 2018] Arora et al. 2017), while others tie some or all of their weights (Curumurthy et al.] 2017 Hoang et al. 2018).
2

---

Under review as a conference paper at ICLR 2020
Most use single parametric noise source (e.g. single Gaussian) (Khayatkhoei et al., 2018 Hoang eet al. 2018) while one uses a mixture of Gaussians (Gurumurthy et al. 2017). Most sample the generators randomly with equal probability, but one attempts to find (in a non-differeeniialee way) a sampling scheme to not sample from redundant generators (Khayatkhoei et al. 2018). (Ghosh jet al. 2018) encourages diversity among generator outputs by introducing classifier that tries to identity the generator a data point came from, or whether it is a real data point (reminiscent of an auxiliary classifier (Odena et al., 2017) or the mutual information loss of (Chen et al.) 2016)). A more theoretical analysis of convergence and equilibrium existence in the loss landscape motivated a muuliple-generaton multiple-disse discriminator mixture in (Arora et al. 2017). We discuss detail the works with the most resemblance to our approach here:
DeLiGAN The DeLiGAN (Gurumurthy et al. 2017) was designed to handle diverse datasets with limited amount of datapoints. It used single generator and a Gaussian mixture model latent space. To train, a single random Gaussian, out of the mixture is chosen, and then they added p to the (ormal(O,) 1) noise and multiplied it by ds with both μ4; and o; as learnable parameters. This differs from our work because while the noise is separated into different components, the probability of selecting each component is cut off from the gradient information in the model and is not differentiable (each Gaussian is selected with an equal probability, and this never changes). Also, every component of the noise is parameterized as a Gaussian. Finally, only one component of the noise is trained at time (a single f4; and 0g is randomly selected for each training batch), while our model learns to model the data over the full collection of generators in each minibatch.
MGAN The MGAN (Hoang et al. 2018) focused on the problem of mode collapse and addressed by using multiple generators which are really the same network except for the first linear projection layer. They introduced new loss term into the traditional GAN training: to encourage the generators to learn different parts of the data space, lower bound on mutual information between the generated images and the generator they came from share almost weights between them and otherwise may redundantly use multiple generators to cover the same part of the space. Unlike in work, they use single noise source and let the single first layer of the generators learn to project it to different parts of the space before going through the same convolutions. In our work, this transformation of the noise before going each generator is done with a separate network which gets gradient information through the generator, but is not optimized jointly with the generator weights. Moreover, like the DeLiGAN, the probability over the multiple generators was assumed to be fixed and uniform.
maximized. This is helpful because the generators
DMWGANPL The DMWGANPL (Khayatkhoei et al., 2018) exclusively viewed multi- generator models as solution for disconnected manifolds. Each generator is given the same single noise sample, and the same mutual information criteria (termed ((G](z), the probability that x came from generator G₂) as the MGAN was used to ensure each generator learned different part of the space. Unlike the previous works, they do not assume an equal probability of selecting each generator. Instead, they sample each generator G2 with probability r,. After each step of the generator and discriminator during training, the rj's are updated to maximize mutual information between their distribution and QG, (r). This has the primary effect of not sampling redundant generators whose output is hard to distinguish from another generator's 's output, and is completely disassociated from the minimax GAN game. Each generator gets the same noise sample that takes single parametric form (N or, 1)), and the effect this has on the minimax game and the quality of generated images only indirect and tangential to the objective being minimized.
MODEL
Let X ~ Px,,, I... be sample of N₂ points from d- d-dimensional distribution Px- We seek a generator G that learns to mimic Px by mapping from a noise distribution Z Pz. To do this, we train discriminator D and pit them against each other in the standard GAN framework:
min max LGAN = Ex [log( D(=)) + Ex [Log(1 D(G(()))
G D
where G tries to minimize and D tries to maximize this objective.
Motivated by the success of ensemble models (Miller & Ehret 2002: Wang & Yao 2009 Hinton et al. 2015) Whitehead & Yaeger 2010), our NPGAN represents the generating function G with multiple distinct generators of the same architecture. However, rather than simply averaging equal, independent, randomly initializing models to take advantage of uncorrelated errors, we adopt in- sights from machine teaching (Mei & Zhu 2015) and knowledge distillation (Hinton et al. 2015).
3

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Calculating the loss LGAN optimized during training.
G: Generator, D: Discriminator, N:P Noise-Prior Network
Sample Fezi... X
Sample 2,=1... N(O, 1)
function Loss(æ,
NP(())
Probability point is sampled from generator i
Douic Σ ((G:D(((())))
Dreat + Σ D(ze))
LOAN + Dreaa)) loo(1 Djake)
Expression for G/D to minimize/maximizz
return LGAN
end Tunction
We use teacher network to select a generator G, conditioned on the particular input it sees. By learning prior over the generators conditioned on the noise, this Noise Prior (NP) network dele- gates each input point to the appropriate generator that is optimally prepared to handle it. Thus, our total generator G can be decomposed into:
G(z) NP( G.(z)
When traditionally training a GAN, the generator and discriminator alternate gradient descent steps, allowing gradient information to flow through the other network while keeping fixed and only optimizing with respect to the given network's s parameters. We extend this to our third noise prior network N P, allowing gradient information inator D while optimizing the GAN loss with respect to the parameters of NP. During training, we let both the NP network and the generators use a soft version of the GAN loss, weighting each generator output by the learned probabilities N x() Then, during inference, the choice of G; is sampled from this learned prior. The full details of the training procedure are given in Algorithm
flow through the fixed generators G; and discrim-
The NE network looks at the sample from Z and determines how best to divide it across the gen- crators to achieve the goal of modeling Px. In the special case where Px is disconnected, N can divide Z in multiple ways such giving each generator a continuous area of Z, or giving some generators multiple disconnected areas of Z (as we demonstrate later in the experiments). Nothing our model formulation specifically designed to model disconnectedness or any other specific property in Px, however, so N P can learn to divide the sample from Z in whatever way is most appropriate for the given shape of Px.
Thus, we model a distribution over our generators rather than simply sampling them uniformly and concatenating their output. Moreover, we learn this distribution over the generators with another network which is conditioned on the input noise, allowing it to choose the shape of input each generator receives. This network does not optimize a separate loss that is a heuristic indirectly related to the GAN framework, but directly participates in the GAN minimax game. To summarize, we fully and flexibly incorporate the multiple-generator framework into a GAN such that the model can learn for itself how best to use the generators. This is achieved by modeling prior over the generators that is conditioned on the noise input and optimizing with respect to the GAN loss directly.
EXPERIMENTS
4.1
DISCONNECTED MANIFOLDDS
Our first experimental dataset (Figure consists of a mixture of samples from two-dimensional Gaussians such that the three Gaussians are not sampled with equal probability (7000, 5000, and 3000 points, respectively). We compare our NPGAN's ability to model this distribution to sin- gle generator model, MGAN (Hoang et al., 2018), DMWGANPL (Khayatkhoei et al. 2018), and DeLiGAN (Gurumurthy et al. 2017). The noise distribution for each model was 100-dimensional mniform(-1.. except the DeLiGAN, which requires samples from Normal(O. 1). The genera- tors in each case share the same architecture of three layers with 200-100-20 neurons per layer and Leaky ReLU activations. The discriminator all cases had three layers of 1000-200- 2000200-1000 100 neurons and used minibatch discrimination (Salimans al. 2016
4

---

Under review as a conference paper at ICLR 2020
Origiaal Dara
Sayle Geveaater
Origon
Saylee Geoenaitr
-35
UNIGIAII
DsUGAN
UNIDAME
DHUGAN
22-
46
4
Figure 2: A single generator fails capture the underlying structure without generating points off the support. Only NPGAN learns sample the generators in proportion to the data (the generator is indicated by the point's color).
Figure 3: Only NPGAN is able to learn to create a discontinuity in the support of one generator and capture three manifolds with just two generators (the generator is indicated by the point's color).
Initially obvious that a single generator cannot model this distribution without generating trailing points that connect the manifolds. By looking at the underlying density plots, we see that most of the generated data lies on the manifolds, in terms of proportions of points. However, when densely sampling the noise distribution, these few off-support outliers still arise. We then evaluate all of the multi-generator models with three generators, which we know to be the true underlying number of disconnected manifolds in this synthetic situation. The MGAN and DeLiGAN fail to model each manifold with distinct generator and thus cover multiple manifolds with one of their generators and produce trail of points in between. This failure stems from their sampling the generators with a fixed, equal probability. Since the disconnected manifolds do not have exactly the sample probability, their model formulations cannot effectively manage this situation. The DMWGANPL. does learn prior over the generators, but this prior only learned to eliminate redundant generators. Thus, it does learn an unequal sampling of the generators and each generator produces points that are distinct from the other generators, but does without accurately modeling the data. The NPGAN, however, assigns each manifold to an individual generator and matches the data distribution without generating any off offifoldo points. This is confirmed quantitatively in Table where we measure the percentage of points each model generates that are off the manifold, which we define to be any point farther from the center of any Gaussian than the largest distance of any of the true points. There we see that the NPGAN generates no points off the manifold, while the other models are all forced generate a trail of points connecting two of the Gaussians.
We next demonstrate the improved flexibility of the NPGAN over previous models by choosing two generators, imagining ourselves in the realistic case of not knowing the underlying number of disconnected manifolds in a particular dataset (Figure 3). In this case, all of the other models must inevitably cover two manifolds with a single generator. Since each generator receives a continu- ous, unaltered noise distribution, this means they produces points off the manifold (Table []. The NPGAN alone learns model three disconnected manifolds with two generators without generating off- off-manifold points in between.
To investigate how the NPGAN achieves this, we learn another model with Z ~ Uri formformm(L,, E R² so that we can plot the noise in two dimensions. In Figure 4h and 4c. we see the noise space for two and three generators, respectively. Notably, in both cases there are three partitions, no matter the number of generators. By learning to give generator disconnected input, the NP network effectively models third manifold without having a dedicated generator responsible for it. Viewing the latent space also informs how the NPGAN easily model non-equally sampled manifolds,
5

---

Under review as a conference paper at ICLR 2020
2at Nobe Scace
Itt bata Scace
(C) Noise Seace
HD Data Space
Conerstar GEMPRDD2
Conertter
Genomrer2 Corentor
Figure 4: We investigate the NP network by using two-dimensional uniform noise and plotting the learned prior over the generators. The three unequally sampled Gaussians in the data can have their density matched with three generators (c-d) by creating a discontinuity with two generators (a-b).
2Gen Gaussinn
3Gen Goussian
2Gen Parabolas
3Gen Parabolas
NPGAN
0.000
0.000
0.003
0.007
MGAN
0.031
0.088
0.083
DMWGANPL
0.037
0.026
0.076
0,082
DeLIGAN
0.134
0.053
0.196
0.320
Table 1: Scores for each model each artificial dataset and the number of generators used (2Gen or 3Gen). For the Gaussian data, the score the percentage of generated points off the manifold. For the parabolas, the score represents the percentage of real points without any generated point in its neighborhood.
as well, as the size of each partition of the noise space expands or contracts to match the underlying data distribution.
4.2 COMPLEX CONNECTED MANIFOLDS
Our next dataset explores the case where the underlying data distribution complex but not neces- sarily disconnected. The other models have design choices to specifically target distinct areas of the data space with each generator. While single generator networks have difficulty with disconnected parts of the data space, there many other ways the data distribution generator to model that have nothing to do with disconnectedness. Since the NPGAN gives the N P network full flexibility shape the input for each generator however it needs to in order to beat the discriminator, it can aid in generating complex shapes of any kind.
be difficult for a single
To investigate this we create distribution of intersecting parabolas and test it with two and three generators for each model (Figures 3 and 6). As before, this complex shape is too difficult for single generator to effectively model. In the DeLiGAN, the equal probability Gaussians trained alternatingly are unable to coordinate with each other and capture any of the tails of the two parabo- las. The MGAN and DMWGANPL. have the mutual information penalty that pushes the generated points for different generators away from each other. This not only keeps them from learning to gen- crate intersecting shapes, but pushes the optimization away from any solution where it requires a complex function to know which generator a particular point came from. The NPGAN, the other hand, effectively models the data distribution with just two generators while finding a different but equally effective way of modeling with three generators. As opposed the previous Gaussian example, the problem here is not generating points off the manifold but leaving parts of the true manifold unmodeled. Thus, quantitatively evaluate this dataset, calculate the percentage of real points that do not have a generated point within an e-ball centered on it (e .001). Table [ confirms that the other models leave significant parts of the tails unmodeled, representing as much as 32% of the data in the most extreme case. The NPGAN's low score with both two and three gen- erators corroborates that can not only help in modeling complex distributions, but that the flexible formulation makes it robust to the specific number of generators chosen.
4.3 CELEBA+PHOOTO
To test the NPGAN's ability to model disconnected, unevenly sampled manifolds on real images, we combine two distinct datasets. We take 1500 images randomly from the CelebA (Liu et al. 2015) dataset and combine them with the 6000 photographs dataset from (Zhu al. 2017). To

---

Under review as a conference paper at ICLR 2020
Oraginal Dark
Sayle Geverater
Original Daga
Single Conosler
OOX
UMIGIANTL
DEUGAN
DeCKSAN
22 4
Figure 5: A single generator is unable to model this non-disconnected data. The disconnected assumption of the other models forces each gen- erator to produce points that are separable from other generators'.
Figure 6: The parabola dataset is naturally solved with two generators, but the NPGAN is robust to the number of generators that are cho- sen. The other models can only work if the pre- cise optimal number of generators known priori.
(x) (ours))ours)
(b) MGAN
(c) DMWGANA
DeLIGAN
Figure 7: Randomly selected images from each model trained on the CelebA+Photo dataset.
effectively match this data distribution with two generators, the models will have to either learn to sample generators at differential rate, or have one generator cover discontinuity (or both).
The images were resized to 32x32, and models use DCGAN architecture (Radford et al. 2015). with three convolutional transpose layers in the generators and three convolutional layers in the discriminator. Each convolution used stride length two, kernel size three, batch normalization on all layers except the first layer of the discriminator, and ReLU activations in the generator with Leaky ReLU activations the diseriminator. Training was performed on minibatches of 32 with an Adam optimizer (Kingma Ba 2014) with learning rate 0.0001. the MGAN and DeLiGAN, the generators are all the same network except for the initial linear projection of the noise (or the adding of the generator-specific mean and the multiplying of the generator-specific standard deviation in the DeLiGAN). In our NPGAN and the DMWGANPL, the generators do not share weights. To compensate for the increased capacity that this would otherwise provide, we decrease the number of filters per generator learned to keep the total number of parameters across all models (within 1%).
Figure shows randomly selected images from each model, and there we can see the consequences of the MGAN and DeLiGAN sampling generators at fixed rate and giving each generator the same continuous noise. each case, one of the generators effectively generates photos, but the other generator gets caught in between generating photos and CelebA images, producing many blurry images. The DMWGANPL samples generators at different rate, but again did so ineffectively: generator makes photos. but the other generator makes both CelebA images and some photos that are not being made by the other generator. Even though is an imperfect measure of capturing outliers, the FID scores reported in Table 2how show that the imbalance affects their ability model the

---

Under review as a conference paper at ICLR 2020
underlying dataset, too. To add uncertainty measure this score, average the last three model checkpoints and report the mean and standard deviation. In Figure we see the NPGAN learns to sample more from the generator that exclusively makes photos, while also using its ability to create discontinuity in its input to allow the other generator to make both CelehA images and a few realistic photos. Only the NPGAN is able to effectively model this disconnected, unevenly sampled dataset.
4.4 COMPLEX-BUT-CONNEEE BUT-CONNECTED BUT-CONNECTEED IMAGE DATASET
In this section, we explore how connected- ness affects the results of the models for im- age datasets. The previous works on multiple generators have emphasized disconnectedness, but we show here that the NPGAN outperforms the alternatives even without disconnected data. The effects of other properties, like class/mode imbalance, dominate the results. We test this notion by modifying the dataset from the previ- ous section create connection between the Face and Photo images. To do this, we randomly choose images in each dataset and perform linear interpolation between them with a mixing coeff- cient chosen from form(() (0, 1) distribution. We add these interpolations to the Photo dataset to make a ConnectedFacee Phoot dataset. Conceptually, ComnectedFace+Phott takes the shape of a "barbell" with a narrow trail connecting two areas of density in data space.
FID Score
Face-Pboto
Comsecied-sooPPooo
NPGAN
58.9 2.4
51.7 4,1
MGAN
65.5 +/. 1.5
63.1 +/: 3.4
Del DeAG
4/
68.6
DMWGANPL
81,0 6.2
83.4 4/ 5.5
WGAN-GP
66.5 4/-4/-99 4.9
69.3 4/- 3.9
Table 2: FID Scores for all models.
We then repeat the experiment of the previous section and report the results. Notably, the quantitative results remain essentially the same. This can be explained with a couple of observations. First, as in the artificial cases, the other models have difficulty dealing with density imbalances, and this difficulty dominates the effects of whether the data disconnected or not. Second, as previously discussed, the FII) scores in Table @ are affected most strongly by model performance where most of the data density is as opposed to a few bad trailing outlier points. Nonetheless, the presence of wrong mffifolniold outliers like those in Figure 7 could be severely problematic in contexts with a higher sensitivity to outliers than the FID score captures.
4.5 CIFAR
Next, we explore the NPGAN's ability to model the canonical CIFARIO dataset (Krizhevsky al.] 2014). Unlike in the previous case where the disconnectedness in the data was drastic enough to measurably affect sample quality as measured by FID, here our NPGAN produced essentially the same FID as our code with one-generaton (26.4 to 25.8). However, as previously discussed, FID is not a good measure of whether model produced outliers or not, since generating 1% bad samples off the manifold will be unnoticed in FID score if coupled with a slight
FaceBed
CIFAR
NPGAN
16.39
18.94
WGANGP
17.84
20.74
DMWGANPL
17.51
19.69
DeLAGAN
17.69
19.61
MGAN
17.43
19.58
Table 3: Outlier manifold distances for all models on FaceBed and CIFAR.
improvement of sample quality on the other 99% of the samples. With that in mind, we introduce new measure of how bad a model's 's worst samples are: outlier manifold distance. Unlike FID, our outlier manifold distance is sensitive to model generating outliers, irrespective of how good its best samples are. We calculate this distance by finding the average distance of the 1 1% furthest generated points from the real data manifold, as measured by the distance the closest real point in the last feature map of the Inception network for each generated point. The outlier manifold distance for each model then the average of the 1% largest distances (the 1% "most anomalous" points). In Table[3] we see that NPGAN has the best outlier manifold distance of all models. As a sanity check, we also calculate on the previous FaceBed data, and show that it confirm quantitatively what we saw qualitatively and with FID score, that other models produce outliers that are worse than NG- PAN's worst samples. For space reasons, a more comprehensive investigation into the NPGAN's use of multiple generators CIFAR we defer to the supplement.
DISCUSSION
We introduced novel formulation of multiple- gegerrato models with prior the generators, conditioned on the noise input. This results in improved expressivity and flexibility by shaping each generator's 's input specifically to best perform that generator's task.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). arXiv preprint arXiv: nee::70730055 2017.
Thomas Blaschke, Marcus Olivecrona, Ola Engkvist, Jürgen Bajorath, and Hongming Chen. Appli- cation of generative autoencoder de novo molecular design. Molecular informatics, 37(1-2): 1700123, 2018.
Bajibabu Bollepalli, Lauri Juvela, Paavo Alku, al. Generative adversarial network-based glottal waveform model for statistical parametric speech synthesis. In Interspeech, pp. 3394-3398, 2017.
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In Advances in Neural Information Process- ing Systems, pp. 742-751, 2017.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. arXiv preprint arXiv: 2018.
Arnab Ghosh, Viveka Kulharia, Vinay P Namboodiri, Philip HS Torr, and Puneet K Dokania. Multi- agent diverse generative adversarial networks. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 85 8513-8521, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R Venkatesh Babu. Deligan: Genera- tive adversarial networks for diverse and limited data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 166-174, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint ar 133500150..02531, 2015.
Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung. Mgan: Training generative adversarial nets with multiple generators. 2018.
John L Kelley. General topology. Courier Dover Publications, 2017.
Mahyar Khayatkhoei, Maneesh Singh, and Ahmed Elgammal. Disconnected manifold learning for generative adversarial networks. arXiv preprint 10Xx:18006000800 2018.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. arXiv preprint arXiv: 07947, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6981/ 2014.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar- 10 dataset. online: http://www. cs. toronto. edu/kriz/cifar. html, pp. 4, 2014.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015.
Shike Mei and Xiaojin Zhu. Using machine teaching to identify optimal training-set attacks on machine learners. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? arXiv preprint arXiv:1801 1000004400 2018.

---

Under review as a conference paper at ICLR 2020
Robert N Miller and Laura L Ehret. Ensemble generation for models of multimodal systems. Monthly Weather Review, 130(9))2313-233 2002.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2642-2651. JMLR. org, 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 1:Xiv::511.064/ 2015.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schicle, and Honglak Lec. Generative adversarial text to image synthesis. arXiv preprint /r05v:/605.05330 2016.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-222222 2016.
Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International Conference on Information Processing in Medical Imaging, pp. 146- 157. Springer, 2017.
Wei Ren Tan, Chee Seng Chan, Hernán E Aguirre, and Kiyoshi Tanaka. Artgan: with conditional categorical gans. In 2017 IEEE International Conference on Image Processing (ICIP), pp. 3760-3764. IEBE, 2017.
Artwork synthesis
Shuo Wang and Xin Yao. Diversity analysis on imbalanced data sets by using ensemble models. In 2009 IEEE Symposium on Computational Intelligence and Data Mining, pp. 324-331. IEEE, 2009.
Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Kdgan: Knowledge distillation with genera- tive adversarial networks. In Advances in Neural Information Processing Systems, pp. 783-794, 2018.
Matthew Whitehead and Larry Yaeger. Sentiment mining using ensemble classification models. In Innovations and advances in computer sciences and engineering, pp. 509-514. Springer, 2010.
Jerry Zhu. Machine teaching for bayesian learners in the exponential family. In Advances Neural Information Processing Systems, pp. 1913, 2013.
Jun Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexci A Efros. Generative visual manipu- lation on the natural image manifold. European Conference on Computer Vision, pp. 597-613. Springer, 2016.
Jun-Ya Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycleccon adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2223-2232, 2017.
10

---

Under review as a conference paper at ICLR 2020
Generators
Generators
Generators
Figure A.1: NPGAN output, with the generator of the image represented by the color of the box, on the CIFAR dataset.
A SUPPLEMENT
A.1 CIFAR
In this section, we elaborate on the CIFAR experiment from the main text. We use a more com- plicated architecture here with spectral normalization, self-attention, and ResNet connections, per the best achieving models to-date. We experimented using two, three, four, and five generators in the NPGAN architecture. Figure A.I shows images generated by the NPGAN with each number of generators. With just two generators, each one creates a wide diversity of images. On the other hand, when increasing the number of generators, each two generator model, one of them creates dogs, cars, and frogs, while in the five-generator model each generator has specialized just birds the sky or just cars.
homogeneous. For example, in the
Qualitatively, the noise prior is obviously learning a sensible split of the data across generators and each generator is outputting quality images. However, when comparing the two-generator, three- generator, four geuereenrr,,r, and five-generator versions of NPGAN to the baseline one- me-generator of the same model, we do not observe any improvement in FID score. This is unsurprising for the reasons mentioned in the main text. The FID scores treat all points equally across a generated dataset, and thus will be most strongly influenced by where the most points are. relatively small number of outliers barely register by this metric.
Even current state-of-the-an image generation on CIFARIO is no where close to perfectly modeling the data. When GANs are able to perfectly model the dataset except for trailing outliers between modes, we expect the NPGAN's improvements to be visible FID scores on this dataset. Until then, the detection of few bad outliers needs to be done with other evaluation techniques on this dataset.
With this caveat, we note that could achieve FID score of 26.4 with our NPGAN, compared to 25.8 with our code and one generator, which demonstrates that the NPGAN can scale to state- of-the-art architecture without suffering in quality. The NPGAN is robust while simultaneously being able to automatically solve the problems of disconnected dataset. Furthermore, this motivated the creation of our new outlier manifold distance metric, designed to be more sensitive to the creation of outliers than the FID score. Using this metric, we see NPGAN outperform all other models.
connected dataset
Relation to Machine Teaching In Zhu 2013), an analogous question is posed: if teacher net- work knows the function its student network is supposed to learn, what are the optimal training points to teach it as efficiently possible? For students following Bayesian learning approach, this is thought of as finding the best data points D to make the desired model 0", or minimizing with respect to D: -log(p(0*1D)) In our framework, the teacher network NP does not know the function its students should learn ahead-of-time because this target changing continually as the discriminator improves simultaneously. Nevertheless, the NP network is still learning to form the optimal curriculum for each individual student such that the collection students best models the target function given the current parameters of the discriminator.
11

---

Under review as a conference paper at ICLR 2020
Relation to knowledge distillation Our NP network also has links to the field of knowledge dis- tillation (Kim & Rush 2016; Chen et al. 2017; Furlanello et al. 2018:) Wang et al. 2018), where a teacher network is trying to compress distill the knowledge it has about particular distribution into one or several (Hinton et al. 2015) smaller models. In the case multiple smaller models, the teacher be thought of generalist whose job it is to find the right specialist for specific problem.
12


Reviews:
[{"review_id": "rkgCn9KRtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to improve GAN by learning a function that splits the latent space in several parts and then feed each part to a different generator, this enables GAN to model distribution with disconnected support. They try the proposed approach on several toy examples where the support of the distribution are disconnected, the data is imbalanced or the support of the distribution intersect,  they in all this cases that the proposed approach improve performance over baseline, in particular the proposed method doesn't produce outliers. They also show how the method is robust to the choice of number of generators used. Finally they show improved performance on more challenging dataset in particular on the CelebA+Photo dataset as measured by FID score.\n\nI'm slightly in favour to accept the paper. I think the idea is well motivated and shows real advantage over other methods on the toy experiments. The major downside of the paper is that the proposed method doesn't seem to improve that much in more realistic setting.\n\nMain Argument:\n+ The idea is well motivated and the paper precisely explain that they try to address the problem of modelling data when the manifold is disconnected and the class are imbalanced. They illustrate how the proposed approach is able to address this problem on some toy example\n\n+ The paper also show how the method is robust to the choice of number of generators\n\n+ Figure 7 and A1 are quite interesting showing how the different generators can learn different part of the data like different class.\n\n- The main counterpoint is that the method doesn't seem to improve the performance that much in more realistic settings. The author point to the fact that this might be due to the fact that the metrics we used are not sensible to outliers. I wish the author had used another metric to show and confirm this hypothesis.\n\n- I found the explanation of the proposed algorithm a bit confusing, it would be nicer if the final loss was clearly defined in the paper and the derivation of the loss explained. In particular in Algorithm 1 I don't understand why $L_{GAN} = D_{real} + (1-D_{fake})$ shouldn't there be some logarithm somewhere ?\n\n- The author propose some measure of standard deviation for the different models but this computed on the three last model checkpoints. It would be much more valuable to compute the standard deviation and the mean with different seeds.\n\nMinor comments:\n- Conflicting notation: In the related work you use $G_i$ to denote a random Gaussian, the same notation is used to denote the generator. \n\n- The part about Machine Teaching and knowledge distillation seem a bit irrelevant to me. I don't really understand what they bring to the paper. Also this would save some space and enable to put the CIFAR10 results in the paper\n\n- In the toy experiments I would be curious to see the results when the different modes have the same probability.\n\n- It would be interesting to have the influence of the number of generators on the FID.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hyx9BI-F_B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall:\nIIUC, the main contribution of this paper is to take previous work on training GANs with\nmultiple generators and add a learned assignment of z -> G_i, so that we have\nG(z) = \\sum_i NP(G_i | z) * G_i(z)\nwhere NP is this learned assignment.\n\nI think that technically this idea is new,\nbut there is important related work [1] that (IMO)\n  - does essentially the same thing\n  - better experimentally validates the thing they do\n  - was published in last ICLR.\n\nI also have a number of issues with the experimental design: see my detailed comments below.\nFinally, the technique seems like it can't easily scale up to larger GANs or data sets \nbecause it requires instantiating many copies of the generator in memory?\n\nFor these reasons, I lean somewhat strongly toward rejection.\n\n\nDetailed comments on draft:\n\nIt's worth noting that conditional GANs also sample from multiple disconnected `manifolds'.\nI guess the value of adding your technique is that it can work without labels?\n\n> Our approach differs significantly from those previously\nstrange sentence.\n\n>  By dividing up the space into four slices...\nI don't get this part. \nIf you fed all the slices to a single generator you could\ngenerate four `disconnected manifolds' with just 1 generator, no?\n\n> , multiple-discriminator mixture in (Arora et al., 2017)\nNit: you don't need the parens there, IMO.\n\n> our model learns to model the data over the full collection of generators in each minibatch\nGiven that modern GAN techniques (e.g. bigGAN), this is going to have some pretty unpleasant\nperformance characteristics, right?\nEach generator has to be instantiated in memory all the time.\n\n> LGAN\nNit: surely, given the state of the GAN literature, this name has been used before.\n\n\n> We compare our NPGAN’s ability to model this distribution to\nThe following baseline seems like it would be much simpler and get the job done:\nTrain a GAN on a prior that's a mixture of like 100 gaussians or whatever. \nThen it seems like it could learn to assign 70, 50, and 30 of those modes\nto each one of the gaussians in your underlying data?\nI guess I haven't tried this myself, but it seems like a more fair comparison.\nThe way the current experiment is set up, you know a priori that your model\nis the only one that can work.\n\nOverall, I don't really understand why it's necessary to have multiple generators.\nIf I want the prior my generator `sees' to be disconnected, can't I just\npass samples from the prior through a standard relu network?\nSurely a relu network can learn to separate one mode into two, and so forth.\nBut once you do that, you've essentially got [1].\n\n> We next demonstrate\nWould be nice to have a new subesection here.\n\nI think your single-generator baseline in Fig 2 and 3 is unrealistically bad.\nSee fig 3 of [2], in which it looks bad but not nearly as bad as you've shown it.\nI think this points to another issue with the experimental design.\nIf the generator is a big enough relu network, it ought to be able to automatically\ngenerate `disconnected bits', but it has to be big enough, and you've compared a\ngenerator with N parameters to 2 and 3 generators with N parameters each.\nMoreover, your learned noise prior has some extra number of parameters (I don't see\nwhere you described these but I may be missing it).\n\n> A single generator is unable to model this non-disconnected data.\nAgain, I have a feeling that this is because of the small generator you use.\nI'm pretty sure that a motivated person could get a normal GAN to model the\ndistribution in fig 5 reasonably well.\n\n> The other models can only work if the precise optimal number of generators is known a\npriori.\nIsn't this not true for the method that throws out redundant generators?\n\nThe experiments described in 4.3 and 4.4 are pretty contrived.\nIt's ok to have some contrived experiments, but it seems like all experiments\non which you were able to provide evidence that your technique was helpful are contrived.\nIt also seems like the baselines you used are not obviously the right choice for these\nexperiments?\n\nRe: your CIFAR experiments:\nI think this supports my earlier claim that the extra parameters and bells and whistles\nin modern GAN techniques *already implicitly do what your method is proposing to do*.\nI think a lot of people miss this point when writing papers about GANs.\nWhen people request comparisons against more modern techniques,\nthey're not (merely) being difficult: \nthey want to know if the technique you propose \"stacks\" with other techniques,\nin the sense that your method is doing something that wasn't already being done implicitly\nby the old methods.\n\nReferences:\n[1] On Self Modulation for Generative Adversarial Networks (https://arxiv.org/abs/1810.01365)\n[2] Discriminator Rejection Sampling (https://arxiv.org/pdf/1810.06758.pdf)\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJlAxDuqdr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Comments: \n\n-As I understand it this technique is relatively simple.  There are K distinct generators with different parameters.  There is a network NP(z) which maps from an input z to a weighting over the generators.  All of these generators are used during training and the loss on the discriminator is reweighted using NP(z).  \n\n-This doesn't involve any explicit discrete decisions so the entire thing along with NP(z) can be trained end-to-end using the usual backprop.  \n\n-It would be interesting to consider simply doing importance sampling after sampling NP(z) to select the most relevant generators to train on, as NP(z) is presumably much cheaper than G(z).  \n\n-One thing that I'd like to see is a variant where the \"different\" generators share almost all of their parameters, but have different batch-norm \"mean/sigma\" parameters.  This would be the same as conditional batch norm.  I think it would be interesting if this could have some of the strengths of this method while still allowing the generators to share most parameters.  \n\nReview: \n\nThis paper presents a simple yet well motivated new method for helping GANs to model disconnected manifolds.  There are a few more explorations that I'd like to see (discussed in comments), but I still appreciate this paper for directly addressing an important challenge.  The 2D toy experiments do a good job of illustrating why the method helps and also the improvements on the \"disconnected face/landscape\" dataset are quite good.  It is a bit disappointing that it doesn't help on CIFAR10, although I think it's reasonable to leave this for future work.  ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a modified GAN architecture that looks a lot like a mixture of experts, to address the problem of learning multiple disconnected manifolds.  They show this method helps on 2D toy experiments, and artificial tasks where different datasets are combined, but not on CIFAR.  They also introduced a new variant of FID that they claim is more sensitive to the improvements made by their model.

R2 didn't seem to think too hard about the paper, and R3 seemed a bit dismissive.

Overall the idea seems sensible but the particulars of this approach aren't all that well-motivated in my opinion, especially since the cost of the generator is increased.  Why not just use a mixture of Gaussians in the original untransformed space?

I also found the toy experiments unconvincing, particularly the claim that a standard GAN couldn't learn a mixture of 3 Gaussians.  Learning a mixture of 8 Gaussians was one of the results in the unrolled GAN paper, for instance.

The results on the mixed datasets experiments seem encouraging, but I'm afraid that proposing a new GAN architecture in 2019 requires even more baselines than the authors compared against, and the fact that the task was artificially constructed undercuts its importance.
Decision: Reject