Unsupervised Conditional Generation using noise engineered mode matching GAN

1 INTRODUCTION . Unsupervised learning or extraction of semantically relevant information from unlabelled data is a problem of great interest . It is useful for a wide range of tasks such as feature learning for supervised and semi-supervised classification , data augmentation and reconstruction ( Hastie et al . ( 2009 ) ; Hinton et al . ( 2006 ) ; Hinton & Salakhutdinov ( 2006 ) ; Bengio et al . ( 2013 ) ; Yann ( 1987 ) ; Radford et al . ( 2015 ) ; Kingma et al . ( 2014 ) ; Cheung et al . ( 2014 ) ) . From a generative model ’ s perspective , unsupervised learning amounts to divide the data manifold into semantically interpretable regions and sample new points from each of the regions ( Jaakkola & Haussler ( 1999 ) ; Hastie & Tibshirani ( 1996 ) ) . However , distributions of most of the naturally occurring data such as images and speech lie over complex high-dimensional manifolds and do not have tractable forms . Deep generative neural networks , in particular Generative Adversarial Networks ( GANs ) ( Goodfellow et al . ( 2016 ) ) , are shown to be effective in modelling complex distributions for several real-world tasks ( Radford et al . ( 2015 ) ; Isola et al . ( 2017 ) ; Van Den Oord et al . ( 2016 ) ; Odena et al . ( 2016 ) ; Denton et al . ( 2015 ) ) . Basic idea of a GAN is to implicitly learn sampling from the data generating distribution without explicitly estimating it . It is done by adopting an adversarial learning principle where the problem of sampling from an unknown data distribution is formulated as a min-max game between two function approximators called generator and discriminator . A generator learns to map an input noise distribution to the data generating distribution . This is facilitated by the discriminator which is trained to distinguish between the generated and the true samples . At equilibrium , the distribution of the generated samples converges to the data generating distribution . There are two major differences between GANs and conventional generative models - First , in GANs , the posterior of the latent variable conditioned on the data is intractable . This is due to the fact that GANs are not designed to explicitly model the relation between the latent and data space . If this relation can be inferred in a GAN setting , they can be readily used to perform a variety of unsupervised learning tasks . Second , in conventional generative models , the prior on the latent variable is chosen in accordance with some guess on the structure of the data . For example , number of clusters in K-means is chosen to be ten if there are ten ‘ classes ’ in the data ( Jain et al . ( 1999 ) ) . As a result , the generated space inherits the properties of the latent space . Motivated by the these observations , in this paper , we propose a GAN model in which latent space , engineered in accordance with the data space , is inverted back from the generated space . This enables the generation to be such that the modal properties of the data generating distribution are matched to those of the latent distribution . The idea of inverting the latent space while training a GAN has been incorporated in different contexts ( Chen et al . ( 2016 ) ; Donahue et al . ( 2016 ) ; Dumoulin et al . ( 2016 ) ; Mescheder et al . ( 2017 ) ; Larsen et al . ( 2015 ) ; Makhzani et al . ( 2015 ) ; Kim et al . ( 2017 ) ; Springenberg ( 2015 ) ; Bao et al . ( 2017 ) ; Sohn et al . ( 2015 ) ) . It is shown to stabilize the training and prevent a degenerative solution for the generator ( known as mode collapse ) ( Che et al . ( 2016 ) ; Srivastava et al . ( 2017 ) ; Arjovsky et al . ( 2017 ) ; Salimans et al . ( 2016 ) ) . Although our work has a similar ambit , the following are our unique contributions : • We propose to engineer the noise space according to the desired objective for unsupervised learning . • We combine the latent space engineering with an inverse mapping from the data space to the latent space in order to enforce the matching the modal properties of generated data distribution with the engineered latent space . • We include an activity regularizer on latent space reconstruction to avoid intra-mode collapse . • Unlike some of the recent approaches ( Chen et al . ( 2016 ) ; Srivastava et al . ( 2017 ) ) , which rely on minimizing a lower bound on the actual cost via variational inference , the loss terms in the proposed method are exact . We demonstrate the use of the proposed method in several unsupervised learning tasks such as conditional generation , inference and attribute discovery through experiments on multiple real-world datasets1 . 2 PROPOSED METHOD . Let PX denote the data generating distribution from which the samples are to be drawn and X denote its support . If X is a disconnected set , which is a union of k non-empty disjoint connected open subsets ( Xi , i = { 1 , 2 ... k } ) , each representing a mode , then PX is termed as a multimodal distribution with k modes . Following we describe our method to induce multiple modes in the data generating space by engineering the latent space along with inversion . For simplicity , we assume that PX is a bimodal distribution , albeit all the analysis holds equally well for multimodal distributions . Let z denote a latent variable with a distribution z ∼ PZ and Z be the support of PZ . Let g : Z→ X be a continuous function mapping from z to x which serves as an estimator for PX . Let X0 ⊆ X and X1 ⊆ X represent two modes of PX . Proposition 1 . Let Z0 ⊆ Z and Z1 ⊆ Z respectively be the inverse images of X0 and X1 under g , then X0 ∩ X1 = Φ only if Z0 ∩ Z1 = Φ , where Φ is an empty set . Proof : In Appendix A . From proposition 1 , to get a bimodal generated distribution , it is necessary to have a bimodal latent distribution , however it is not a sufficient condition . If there exists a mapping h : X→ Ŷ which maps x , the output of g , to another random variable ŷ in such a way that PŶ is bimodal , one can apply proposition 1 again on h and achieve bimodality on PX . The following corollary to proposition 1 is stated to affirm this fact . Corollary 1.1 . Let Ŷ0 ⊆ Ŷ and Ŷ1 ⊆ Ŷ be two subsets of Ŷ such that h : X0 → Ŷ0 and h : X1 → Ŷ1 then Ŷ0 ∩ Ŷ1 = Φ only if X0 ∩ X1 = Φ . Given Z0 ∩ Z1 = Φ , Ŷ0 ∩ Ŷ1 = Φ is a sufficient condition for X0 ∩ X1 = Φ. Corollary 1.1 asserts if latent distribution is bimodal and h maps x to any bimodal distribution , the generated distribution ( PX ) will also be bimodal . However , since both g and h are continuous mappings , conditional generation of samples from PX ( conditioned on modes of z ) is possible only if h is in accordance with the modal properties of z . Thus , we introduce a new random variable y that is an indicator of the modes of z . Formally , y : = 1z∈Z1 , which implies that PY ( y = 1 ) = ∫ Z1 PZ dz 1Code for implementation can be found at - https : //github.com/NEMGAN/NEMGAN and PY ( y = 0 ) = ∫ Z0 PZ dz . With this , we show that if PY and PŶ are close , then the modal properties of PX will match with those of PZ . Proposition 2 . Define a random variable that is an indicator of modes of PX as follows : x̂ : = 1x∈X1 , PX̂ ( x̂ = 1 ) = ∫ X1 PX dx and PX̂ ( x̂ = 0 ) = ∫ X0 PX dx . The minimization of DKL ( PŶ ||PY ) is equivalent to the minimization of DKL ( PŶ ||PX̂ ) . Proof : In appendix A. Corollary 2.1 . Minimizing DKL ( PŶ ||PX̂ ) leads to matching of modal properties of PX̂ and PY , that is , ∫ Xi PX dx = PY ( y = i ) , i ∈ { 0 , 1 } . Proof : In Appendix A . Proposition 2 and Corollary 2.1. state an important fact - modal properties of the latent space and the data generating space can be matched by choosing h which would minimize DKL ( PŶ ||PY ) . This also implies that the imbalance ( if any ) in the modes of the latent distribution is reflected in the data generating distribution . This is especially useful for datasets containing imbalanced classes , which is often the case in real-world applications . The latent space Z can be constructed following the presumed class imbalance ratio in the real data to conditionally generate imbalanced classes . Using proposition 1 and 2 , one can make the data generating distribution to be bimodal , however , produced modes might be degenerated in a sense that Xi ’ s reduce to singletons ( mode collapse ) . To avoid this degenerative case , we propose to decompose h as a composite of two mappings h1 : X → Ẑ and h2 : Ẑ → Ŷ . Minimizing a norm distance between the samples of Z and Ẑ prevents degenerative modes in PX . This is because h1 enforces a unique reconstruction of every sample of z which in turn ensures that a unique sample of x is generated by a unique sample of z . The function h1 can be seen as an activity regularizer that would force every unique noise sample within each mode to map to a unique samples in the inversion and the generated spaces . In the above formulation , however , the generated distribution is not constrained to be close to the distribution of the real data . For that , one has to rely on a technique ( such as GANs ) that would enforce g ( z ) to be close to the distribution of the real data . Thus training a GAN according to the proposed formulation , simultaneously enforces the generated data distribution to be close to the real data distribution and match the modal properties of the latent distribution . Conversely , if the latent distribution is chosen in accordance with the modal properties of the real data distribution , the generated data space will be clustered conditioned on the modes of the real data . This is a desirable property because if the modes of the real data signify semantic separation , then by adopting the proposed procedure one can achieve unsupervised semantically meaningful conditional generation . In the following section , we detail the procedure to realize the proposed method in a GAN framework . 2.1 REALIZATION USING GENERATIVE ADVERSARIAL NETWORKS . GAN realizes the mapping g using a deep neural network . Additionally , to ensure that PX is close to the real data distribution a second neural network , the discriminator denoted by d , is adversarially trained to differentiate between a true data sample and generated data sample . At the optimum , the generated distribution , PX approaches the distribution of the real data , PXr . The inversion network h = h2 ◦ h1 is also modelled using deep neural networks . All neural networks , g , d and h are simultaneously trained to optimize the following objective function : min g , h2◦h1 max d [ Exr [ log d ( xr ) ] + Ez [ log ( 1− d ◦ g ( z ) ) + ||z− h1 ◦ g ( z ) ||p ] + DKL ( PŶ ||PY ) ] ( 1 ) The proposed method can be seen as establishing a chain of continuous functions of random variables - y → z → x → ẑ → ŷ and imposing a similar modal structures on all of them . The model being completely unsupervised offers several advantages - ( a ) Once trained , sampling z from a particular mode confines x to a unique mode leading to conditional generation , ( b ) when training by imposing a fixed prior on z , unseen attributes in the data can be discovered , ( c ) post training , the h network can be used independently to cluster the data , ( d ) if z is well-reconstructed by h1 , this architecture can be used for inference ( Note that this is not guaranteed since the model convergence can be defined only via convergence of h2 even if z is not reconstructed back by h1 ) , ( e ) Mode-collapse when generated PX becomes independent of the latent random variable z is avoided in our formulation since g is explicitly forced to produce a multi-modal distribution by the inversion network h .

Summary: This paper is concerned with the so-called conditional generation, which was descried as the task of sampling from an unknown distribution conditioned on semantics of the data. This proposed method aims at achieving this by using a latent distribution engineered according to a certain data prior. Experimental results showed that the proposed method seems to produce good results.