Learning Latent Semantic Representation from Pre-defined Generative Model

1 INTRODUCTION . Developing generative model is a crucial issue in artificial intelligence . Creativity was a human proprietary , but many recent studies have attempted to make machines to mimic it . There has been an extensive research on generating data and one of them , generative adversarial network ( GAN ) , has led to significant achievements , which might be helpful to deep learning model because , in general , lots of data result in good performance ( LeCun et al. , 2015 ) . Many approaches to creating data as better quality as possible have been studied : for example , variational auto-encoder ( VAE ) ( Kingma & Welling , 2013 ) and GAN ( Goodfellow et al. , 2014 ) . The former constructs an explicit density , resulting in an explicit likelihood which can be maximized , and the latter constructs an implicit density ( Goodfellow , 2016 ) . Both can generate data from manifold which is hidden to us so that we can not control the kind of data that we generate . Because it is costly to structure data manually , we need not only data generation but also automatically structuring data . Generative models produce only data from latent variable without any other information so that we can not control what we want to generate . To cope with this problem , the previous research generated data first and found distributions of features on latent space by investigating the model with data , since the manifold of data is hidden in generative models . This latent space is deceptive for finding an area which represents a specific feature of our interest ; it would take a long time even if we can find that area . Besides , in the most of research , generative models had a large latent space , resulting in a low compression rate which leads to poor scalability . To work out these problems , we propose a model which can generate the data whose type is what we want and learn a representation of data with a higher compression rate , as well . Our model is based on VAE and GAN . We pre-define distributions corresponding to each feature and modify the loss function of VAE so as to generate the data from the latent variable which follows the specific distribution according to its features . However , this method makes the latent space to become a more complex multimodal distribution which contains many distributions , resulting in an instability in training the LSC-GAN . We prove that this problem can be solved and even made more efficiently by using an auto-encoder model with the theorem in Section 3 . Although the proposed model compresses the data into small manifold , it is well-defined with Euclidean distance as shown in Fig . 1 , which compares the manifolds in general models and in our model . The distance can be calculated with Euclidean distance in adjacent points but not in far points at the left manifold in Fig . 1 . However , in the right manifold , we can calculate the distance between points regardless of the distance of them , where we can recognize the manifold more easily as shown in the left side . Thanks to a relatively simple manifold , it can produce neutral features regardless of their location in latent space , so that all features can be said as independent to each other . Our main contribution is summarized as follows . • We propose a method to improve the stability of a LSC-GAN with LSC-VAE by performing the weight initialization , and prove it theoretically . • We achieve conditional generation without additional parameters by controlling the latent space itself , rather than adding additional inputs like the existing model for condition generation . • We propose a novel model that automatically learns the ability to process data continuously through latent space control . • Finally , we achieve an efficient compression rate with LSC-GAN based on weight initialization of LSC-VAE . The rest of the paper is organized as follows . Section 2 reviews the related works and the proposed LSC-GAN model is illustrated in Section 3 . In Section 4 , we evaluate the performance of the proposed method with some generated data . The conclusion and discussion are presented in Section 5 . 2 RELATED WORKS . Many research works have been conducted to generate data such as text , grammar , and images ( Yang et al. , 2017 ; Kusner et al. , 2017 ; Denton et al. , 2015 ) . We divide the approaches for data generation into three categories : only generation , conditioned generation , and transforming data to have different features . Several researchers proposed generative models of VAE and GAN ( Kingma & Welling , 2013 ; Goodfellow et al. , 2014 ) . These are basis of the generative models . Both use maximum likelihood approach , but they have different policies to construct density : explicitly and implicitly . There are lots of variations of these models . Radford et al . ( 2015 ) constructed deep convolutional GAN ( DCGAN ) with convolutional neural networks ( CNN ) for improving the performance with the fact that CNN had been huge adoption in computer vision applications . Zhao et al . ( 2016 ) introduced energy-based GAN ( EBGAN ) using autoencoder in discriminator . Kim et al . ( 2017a ; 2018a ; b ) proposed transferred encoder-decoder GAN ( TED-GAN ) for stabilizing process of training GAN and used it to classify the data . These studies focused on high productivity in generation so that they could not control the type of generated data . Recently , some researchers began to set conditions on the data they generate . Sohn et al . ( 2015 ) and Walker et al . ( 2016 ) inputted data and conditions together into VAE and generated data whose type is what they want , called conditional VAE ( CVAE ) . van den Oord et al . ( 2017 ) set discrete embedding space for generating a specific data using vector quantized variational auto-encoder ( VQ-VAE ) , but because of discrete space , they could not control latent space continuously . Larsen et al . ( 2015 ) used both VAE and GAN in one generative model . As they just mixed two models and did not analyzed a latent space , so that the manifold of data was hidden to us . To generate image with a specific feature , they extracted a visual attribute vector which is a mean of vector in latent space . Mirza & Osindero ( 2014 ) inputted not only data but also conditions into GAN to create data that we want , called conditional GAN ( CGAN ) . Chen et al . ( 2016 ) used mutual information for inducing latent codes ( InfoGAN ) and Nguyen et al . ( 2017 ) added a condition network that tells the generator what to generate ( PPGN ) . These two models needed an additional input to generate the type of data we want . These studies make us to generate data with condition , but we still do not know about latent space and it is hard to find the location of a specific feature in the latent space . Therefore , we propose a model that learns to generate concrete features that we want from the latent space determined when LSC-VAE is trained . Some studies attempted to transfer the given data to others which have different features or even in different domain . Tran et al . ( 2017 ) proposed disentangled representation learning GAN ( DRGAN ) for pose-invariant face recognition . Reed et al . ( 2016a ; b ) tried matching latent space of text and images and finally they translated text to image . Zhang et al . ( 2017 ) also translated text to image and generated photo-realistic images conditioned on text by stacking models ( StackGAN ) . Zhu et al . ( 2017 ) and Kim et al . ( 2017b ) discovered cross-domain relations with CycleGAN and DiscoGAN . They can translate art style , face features , and bags to shoes . While other models could only do one conversion task , Choi et al . ( 2017 ) proposed StarGAN that could do multiple translation tasks with one model . These studies have been conducted to transform the data into those in other domains . However , they could not generate new data without input data . In addition , the size of latent space of most of them was too large . We aim to generate conditioned data even with a small size of latent space . 3 THE PROPOSED METHOD . In this section , we present a method to generate the data with the corresponding characteristics by inputting the latent variable which follows the specific distribution in latent space . As the instability caused by the larger scale of latent space in this process , we use the modified VAE , called LSC-VAE1 . As shown in Fig . 2 ( a ) , we train the LSC-VAE with Lprior for the data to be projected by the encoder into the desired position in the latent space according to the characteristics of the data . The trained decoder of the LSC-VAE is used as a generator of LSC-GAN so that the LSC-GAN generates the data with the corresponding features by using latent variables sampled from a specific distribution . The proposed model is divided into two phases : initializing latent space ( Fig . 2 ( a ) ) and generating data ( Fig . 2 ( b ) ) . In the first phase , latent semantic controlling VAE ( LSC-VAE ) is trained to project data into a specific location of latent space according to its features , and it learns to reconstruct data which is compressed . The decoder of LSC-VAE is used in the generator ( G ) of LSC-GAN in the second phase . G and discriminator ( D ) are trained simultaneously so that G can produce data similar to real data as much as possible and that D can distinguish the real from the fake . The architecture of the generation process is shown in Fig . 2 ( b ) . 3.1 INITIALIZING THE LATENT SPACE WITH LSC-VAE . Auto-encoder has been traditionally used to represent manifold without supervision . In particular , VAE , one type of auto-encoders , is one of the most popular approaches to unsupervised learning of complicated distributions . Since any supervision is not in training process , the manifold constructed is hidden to us . As we mentioned in Section 1 , this is usually too complex to generate the conditioned 1Intuitively , as opposed to GAN , VAE constructs the distribution of data explicitly , resulting in efficient training of LSC-GAN . We deal with this theoretically in Section 3.2. data . Therefore , we allow LSC-VAE to learn a representation of data with supervision . It compresses data into a particular place on latent space according to its features . The proposed model consists of two modules that encode a data xi to a latent representation zi and decode it back to the data space , respectively . zi ∼ Enc ( xi ) = Q ( zi | xi ) ( 1 ) x̃i ∼ Dec ( zi ) = P ( xi | zi ) ( 2 ) Index i means a feature which is included in data x and latent space z . The encoder is regularized by imposing a prior over the latent distribution P ( z ) . In general , z ∼ N ( 0 , I ) is chosen , but we choose zi ∼ N ( µi , I ) for controlling latent space . In addition , if we want to produce data which has multiple features i , j , we generate data from zij ∼ N ( µi + µj , I ) 2 . The loss function of LSC-VAE is as follows . LLSC−V AE = −Ezi∼Q ( zi|xi ) [ logP ( xi | zi ) ] +DKL [ Q ( zi | xi ) || P ( zi ) ] = LV AE + Lprior ( 3 ) whereDKL is the Kullback-Leibler divergence . The first term of equation 3 is related to reconstruction error and the second term is related to appropriate projection of data to the latent space . For example , when LSC-VAE projects the data with i− and j−features into the latent space , it is trained to map the data into the pre-defined latent space ( N ( µi + µj , I ) ) with Lprior in equation 3 so as to locate the reconstructed data as similar to the input data according to its characteristics using LV AE . Therefore , LSC-VAE can be used in initializing GAN and it is demonstrated that LSC-VAE is valid and efficient for LSC-GAN in the next section .

Summary: The paper proposes a generative model that combines VAE and GAN. The main idea of the paper is to replace the standard normal distribution used in VAE with a normal distribution centered at a feature representation of the input image. In other words, the prior distribution is data adaptive. The paper compares the proposed generative model to DCGAN and EBGAN for image generation quality using the CelebA dataset and reports better human preference score.