Improving Multi-Manifold GANs with a Learned Noise Prior

1 INTRODUCTION . Learning generative models of high-dimensional data is of perpetual interest , as its wide suite of applications include synthesizing conversations , creating artwork , or designing biological agents ( Bollepalli et al. , 2017 ; Tan et al. , 2017 ; Blaschke et al. , 2018 ) . Deep models , especially generative adversarial networks ( GANs ) , have significantly improved the state of the art at modeling these complex distributions , thus encouraging further research ( Goodfellow et al. , 2014 ) . Whether implicitly or explicitly , works that use GANs make a crucial modeling decision known as the manifold assumption ( Zhu et al. , 2016 ; Schlegl et al. , 2017 ; Reed et al. , 2016 ) . This is the assumption that high-dimensional data lies on a single low-dimensional manifold which smoothly varies and where local Euclidean distances in the low-dimensional space correspond to complex transformations in the high-dimensional space . While generally true in many applications , this assumption does not always hold ( Khayatkhoei et al. , 2018 ) . For example , recent work has emphasized situations where the data lies not on one single manifold , but on multiple , disconnected manifolds ( Khayatkhoei et al. , 2018 ; Gurumurthy et al. , 2017 ; Hoang et al. , 2018 ) . In this case , GANs must attempt to learn a continuous cover of the multiple manifolds , which inevitably leads to the generation of off-manifold points which lie in between ( Kelley , 2017 ) . The generator tries to minimize the number of these off-manifold points , and thus they are generally just a small fraction of the total generated distribution . As such , they barely affect the typical GAN evaluation measures ( like Inception and FID scores for images ) , which measure the quality of the generated distribution as a whole . Thus , this problem is usually ignored , as other aspects are prioritized . However , in some applications , the presence of these bad outliers is more catastrophic than slight imperfections in modeling the most dense regions of the space . For example , consider the goal of an artificial agent acting indistinguishably from a human : the famous Turing Test . Incorrectly modeling sentence density by using a given sentence structure 60 % of the time instead of 40 % of the time is relatively harmless . However , generating a single gibberish sentence will give away the identity of the artificial agent . Moreover , there are serious concerns about the implications this has for proofs of GAN convergence ( Mescheder et al. , 2018 ) . These works address the problem of disconnected manifolds by simultaneously training multiple generators and using established regularizations ( Chen et al. , 2016 ) to coax them into dividing up the space and learning separate manifolds . Methods for getting multiple generators to generate disconnected manifolds can be divided into two categories : ( i ) imposing information theoretic losses to encourage output from different generators to be distinguishable ( Khayatkhoei et al. , 2018 ; Hoang et al. , 2018 ) ( ii ) changing the initial noise distribution to be disconnected ( Gurumurthy et al. , 2017 ) . Our approach falls into the second category . Previous efforts to change the noise distribution to handle disconnectedness has exclusively taken the form of sampling from a mixture of Gaussians rather than the typical single Gaussian ( with sampling fixed and uniform over the mixture ) . Our approach differs significantly from those previously . We use multiple generators as before , but instead of dividing up the noise space into factorized Gaussians and sending one to each generator , we let an additional neural network determine how best to divide up the noise space and dispatch it to each generator . This network learns a prior over the generators , conditioned on the noise space . Thus , we call our additional third network a noise-prior ( NP ) network . Previous methods have modeled the data with noise z and generators Gi as p ( Gi|z ) · Gi ( z ) , with p ( Gi|z ) = p ( Gi ) = p ( Gj ) ∀i , j . We instead propose a framework to incorporate a richer p ( Gi|z ) into the generator . This framework is entirely differentiable , allowing us to optimize the NP network along with the generators during training . We note that with this strategy , we significantly increase the expressivity of each generator over the previous disconnected manifold models . By dividing up the space into four slices si and sending s1 , s3 to the first generator and s2 , s4 to the second generator , we can generate four disconnected manifolds with just two generators . Previous work would have to devote precisely four generators to this task , with degradation in performance if fewer or more generators are chosen for the hyperparameter . Here , the prior network learns to divide the noise space appropriately for whatever number of generators is chosen , and is thus more expressive as well as more robust than previous models . Moreover , much existing work has exclusively framed the problem as , and tailored solutions for , the disconnected manifold problem . Our approach is more generalized , addressing any misspecification between noise distribution and the target distribution . This means that our approach does not become redundant or unnecessary in the case of single complex manifolds , for example . Our contributions can be summarized as : 1 . We introduce the first multi-generator ensemble to learn a prior over the noise space , using a novel soft , differentiable loss formulation . 2 . We present a multi-generator method that can learn to sample generators in proportion to the relative density of multiple manifolds . 3 . We show how our model not only improves performance on disconnected manifolds , but also on complex-but-connected manifolds , which are more likely to arise in real situations . 2 RELATED WORK . Several previous works have included multiple generators , mixing and matching a few commonly used features . Some use completely distinct generators ( Khayatkhoei et al. , 2018 ; Arora et al. , 2017 ) , while others tie some or all of their weights ( Gurumurthy et al. , 2017 ; Hoang et al. , 2018 ) . Most use a single parametric noise source ( e.g . a single Gaussian ) ( Khayatkhoei et al. , 2018 ; Hoang et al. , 2018 ) while one uses a mixture of Gaussians ( Gurumurthy et al. , 2017 ) . Most sample the generators randomly with equal probability , but one attempts to find ( in a non-differentiable way ) a sampling scheme to not sample from redundant generators ( Khayatkhoei et al. , 2018 ) . ( Ghosh et al. , 2018 ) encourages diversity among generator outputs by introducing a classifier that tries to identify the generator a data point came from , or whether it is a real data point ( reminiscent of an auxiliary classifier ( Odena et al. , 2017 ) or the mutual information loss of ( Chen et al. , 2016 ) ) . A more theoretical analysis of convergence and equilibrium existence in the loss landscape motivated a multiple-generator , multiple-discriminator mixture in ( Arora et al. , 2017 ) . We discuss in detail the works with the most resemblance to our approach here : DeLiGAN The DeLiGAN ( Gurumurthy et al. , 2017 ) was designed to handle diverse datasets with a limited amount of datapoints . It used a single generator and a Gaussian mixture model latent space . To train , a single random Gaussiani out of the mixture is chosen , and then they added µi to the Normal ( 0 , 1 ) noise and multiplied it by σi , with both µi and σi as learnable parameters . This differs from our work because while the noise is separated into different components , the probability of selecting each component is cut off from the gradient information in the model and is not differentiable ( each Gaussian is selected with an equal probability , and this never changes ) . Also , every component of the noise is parameterized as a Gaussian . Finally , only one component of the noise is trained at a time ( a single µi and σi is randomly selected for each training batch ) , while our model learns to model the data over the full collection of generators in each minibatch . MGAN The MGAN ( Hoang et al. , 2018 ) focused on the problem of mode collapse and addressed it by using multiple generators which are really the same network except for the first linear projection layer . They introduced a new loss term into the traditional GAN training : to encourage the generators to learn different parts of the data space , a lower bound on mutual information between the generated images and the generator they came from was maximized . This is helpful because the generators share almost weights between them and otherwise may redundantly use multiple generators to cover the same part of the space . Unlike in our work , they use a single noise source and let the single first layer of the generators learn to project it to different parts of the space before going through the same convolutions . In our work , this transformation of the noise before going to each generator is done with a separate network which gets gradient information through the generator , but is not optimized jointly with the generator weights . Moreover , like the DeLiGAN , the probability over the multiple generators was assumed to be fixed and uniform . DMWGANPL The DMWGANPL ( Khayatkhoei et al. , 2018 ) exclusively viewed multi-generator models as a solution for disconnected manifolds . Each generator is given the same single noise sample , and the same mutual information criteria ( termedQ ( Gi|x ) , the probability that x came from generator Gi ) as the MGAN was used to ensure each generator learned a different part of the space . Unlike the previous works , they do not assume an equal probability of selecting each generator . Instead , they sample each generator Gi with probability ri . After each step of the generator and discriminator during training , the ri ’ s are updated to maximize mutual information between their distribution and Q ( Gi|x ) . This has the primary effect of not sampling redundant generators whose output is hard to distinguish from another generator ’ s output , and is completely disassociated from the minimax GAN game . Each generator gets the same noise sample that takes a single parametric form ( Normal ( 0 , 1 ) ) , and the effect this has on the minimax game and the quality of generated images is only indirect and tangential to the objective being minimized . 3 MODEL . Let X ∼ PX , xi ∈ Rd , i = 1 ... Nx be a sample of Nx points from a d-dimensional distribution PX . We seek a generator G that learns to mimic PX by mapping from a noise distribution Z ∼ PZ . To do this , we train a discriminator D and pit them against each other in the standard GAN framework : min G max D LGAN = Ex∼Px [ log ( D ( x ) ) ] + Ez∼Pz [ log ( 1−D ( G ( z ) ) ) ] where G tries to minimize and D tries to maximize this objective . Motivated by the success of ensemble models ( Miller & Ehret , 2002 ; Wang & Yao , 2009 ; Hinton et al. , 2015 ; Whitehead & Yaeger , 2010 ) , our NPGAN represents the generating function G with multiple distinct generators of the same architecture . However , rather than simply averaging equal , independent , randomly initializing models to take advantage of uncorrelated errors , we adopt insights from machine teaching ( Mei & Zhu , 2015 ) and knowledge distillation ( Hinton et al. , 2015 ) . Algorithm 1 Calculating the loss LGAN optimized during training . G : Generator , D : Discriminator , NP : Noise-Prior Network Sample xk=1 ... n ∼ X Sample zj=1 ... n ∼ N ( 0 , 1 ) function LOSS ( x , z ) πi=1 ... kj ← NP ( zj ) . Probability point j is sampled from generator i Dfake ← ∑ ij ( πij ·D ( Gi ( zj ) ) ) Dreal ← ∑ k D ( xk ) LGAN ← log ( Dreal ) + log ( 1−Dfake ) . Expression for G/D to minimize/maximize return LGAN end function We use a teacher network to select a generator Gi conditioned on the particular input it sees . By learning a prior over the generators conditioned on the noise , this Noise Prior ( NP ) network delegates each input point to the appropriate generator that is optimally prepared to handle it . Thus , our total generator G can be decomposed into : G ( z ) = NP ( Gi|z ) ·Gi ( z ) When traditionally training a GAN , the generator and discriminator alternate gradient descent steps , allowing gradient information to flow through the other network while keeping it fixed and only optimizing with respect to the given network ’ s parameters . We extend this to our third noise prior network NP , allowing gradient information to flow through the fixed generators Gi and discriminator D while optimizing the GAN loss with respect to the parameters of NP . During training , we let both the NP network and the generators use a soft version of the GAN loss , weighting each generator output by the learned probabilities NP ( Gi|z ) . Then , during inference , the choice of Gi is sampled from this learned prior . The full details of the training procedure are given in Algorithm 1 . The NP network looks at the sample from Z and determines how best to divide it across the generators to achieve the goal of modeling PX . In the special case where PX is disconnected , NP can divide Z in multiple ways such as giving each generator a continuous area of Z , or giving some generators multiple disconnected areas of Z ( as we demonstrate later in the experiments ) . Nothing in our model formulation is specifically designed to model disconnectedness or any other specific property in PX , however , so NP can learn to divide the sample from Z in whatever way is most appropriate for the given shape of PX . Thus , we model a distribution over our generators rather than simply sampling them uniformly and concatenating their output . Moreover , we learn this distribution over the generators with another network which is conditioned on the input noise , allowing it to choose the shape of input each generator receives . This network does not optimize a separate loss that is a heuristic indirectly related to the GAN framework , but directly participates in the GAN minimax game . To summarize , we fully and flexibly incorporate the multiple-generator framework into a GAN such that the model can learn for itself how best to use the generators . This is achieved by modeling a prior over the generators that is conditioned on the noise input and optimizing it with respect to the GAN loss directly .

Summary: This paper proposes to improve GAN by learning a function that splits the latent space in several parts and then feed each part to a different generator, this enables GAN to model distribution with disconnected support. They try the proposed approach on several toy examples where the support of the distribution are disconnected, the data is imbalanced or the support of the distribution intersect,  they in all this cases that the proposed approach improve performance over baseline, in particular the proposed method doesn't produce outliers. They also show how the method is robust to the choice of number of generators used. Finally they show improved performance on more challenging dataset in particular on the CelebA+Photo dataset as measured by FID score.