Title: LEARNING LATENT SEMANTIC REPRESENTATION FROM PRE-DEFINED GENERATIVE MODEL

Abstract: Learning representations of data is an important issue in machine learning. Though GAN has led to significant improvements in the data representations, it still has several problems such as unstable training, hidden manifold of data, and huge computational overhead. GAN tends to produce the data simply without any information about the manifold of the data, which hinders from controlling desired features to generate. Moreover, most of GAN's have a large size of manifold, resulting in poor scalability. In this paper, we propose a novel GAN to control the latent semantic representation, called LSC-GAN, which allows us to produce desired data to generate and learns a representation of the data efficiently. Unlike the conventional GAN models with hidden distribution of latent space, we define the distributions explicitly in advance that are trained to generate the data based on the corresponding features by inputting the latent variables that follow the distribution. As the larger scale of latent space caused by deploying various distributions in one latent space makes training unstable while maintaining the dimension of latent space, we need to separate the process of defining the distributions explicitly and operation of generation. We prove that a VAE is proper for the former and modify a loss function of VAE to map the data into the pre-defined latent space so as to locate the reconstructed data as close to the input data according to its characteristics. Moreover, we add the KL divergence to the loss function of LSC-GAN to include this process. The decoder of VAE, which generates the data with the corresponding features from the pre-defined latent space, is used as the generator of the LSC-GAN. Several experiments on the CelebA dataset are conducted to verify the usefulness of the proposed method to generate desired data stably and efficiently, achieving a high compression ratio that can hold about 24 pixels of information in each dimension of latent space. Besides, our model learns the reverse of features such as not laughing (rather frowning) only with data of ordinary and smiling facial expression.

1. INTRODUCTION:

Developing generative model is a crucial issue in artificial intelligence. Creativity was a human proprietary, but many recent studies have attempted to make machines to mimic it. There has been an extensive research on generating data and one of them, generative adversarial network (GAN), has led to significant achievements, which might be helpful to deep learning model because, in general, lots of data result in good performance (LeCun et al., 2015) . Many approaches to creating data as better quality as possible have been studied: for example, variational auto-encoder (VAE) (Kingma & Welling, 2013) and GAN (Goodfellow et al., 2014) . The former constructs an explicit density, resulting in an explicit likelihood which can be maximized, and the latter constructs an implicit density (Goodfellow, 2016) . Both can generate data from manifold which is hidden to us so that we cannot control the kind of data that we generate.

Because it is costly to structure data manually, we need not only data generation but also automatically structuring data. Generative models produce only data from latent variable without any other information so that we cannot control what we want to generate. To cope with this problem, the previous research generated data first and found distributions of features on latent space by investigating the model with data, since the manifold of data is hidden in generative models. This latent space is deceptive for finding an area which represents a specific feature of our interest; it would Figure 1 : Examples of the manifold. Left: a complex manifold which can be seen in general models, Right: a relatively simple manifold in the proposed model. The midpoint M of A and B can be easily calculated in the right manifold, but not in the left one. The midpoint of A and B is computed as N in the left manifold, which is incorrect. take a long time even if we can find that area. Besides, in the most of research, generative models had a large latent space, resulting in a low compression rate which leads to poor scalability. To work out these problems, we propose a model which can generate the data whose type is what we want and learn a representation of data with a higher compression rate, as well. Our model is based on VAE and GAN. We pre-define distributions corresponding to each feature and modify the loss function of VAE so as to generate the data from the latent variable which follows the specific distribution according to its features. However, this method makes the latent space to become a more complex multimodal distribution which contains many distributions, resulting in an instability in training the LSC-GAN. We prove that this problem can be solved and even made more efficiently by using an auto-encoder model with the theorem in Section 3. Although the proposed model compresses the data into small manifold, it is well-defined with Euclidean distance as shown in Fig. 1 , which compares the manifolds in general models and in our model. The distance can be calculated with Euclidean distance in adjacent points but not in far points at the left manifold in Fig. 1 . However, in the right manifold, we can calculate the distance between points regardless of the distance of them, where we can recognize the manifold more easily as shown in the left side. Thanks to a relatively simple manifold, it can produce neutral features regardless of their location in latent space, so that all features can be said as independent to each other. Our main contribution is summarized as follows.

• We propose a method to improve the stability of a LSC-GAN with LSC-VAE by performing the weight initialization, and prove it theoretically.

• We achieve conditional generation without additional parameters by controlling the latent space itself, rather than adding additional inputs like the existing model for condition generation.

• We propose a novel model that automatically learns the ability to process data continuously through latent space control.

• Finally, we achieve an efficient compression rate with LSC-GAN based on weight initialization of LSC-VAE.

The rest of the paper is organized as follows. Section 2 reviews the related works and the proposed LSC-GAN model is illustrated in Section 3. In Section 4, we evaluate the performance of the proposed method with some generated data. The conclusion and discussion are presented in Section 5.

2. RELATED WORKS:

Many research works have been conducted to generate data such as text, grammar, and images (Yang et al., 2017; Kusner et al., 2017; Denton et al., 2015) . We divide the approaches for data generation into three categories: only generation, conditioned generation, and transforming data to have different features.

Several researchers proposed generative models of VAE and GAN (Kingma & Welling, 2013; Goodfellow et al., 2014) . These are basis of the generative models. Both use maximum likelihood approach, but they have different policies to construct density: explicitly and implicitly. There are lots of variations of these models. Radford et al. (2015) constructed deep convolutional GAN (DCGAN) with convolutional neural networks (CNN) for improving the performance with the fact that CNN had been huge adoption in computer vision applications. Zhao et al. (2016) introduced energy-based GAN (EBGAN) using autoencoder in discriminator. Kim et al. (2017a; 2018a; b) proposed transferred encoder-decoder GAN (TED-GAN) for stabilizing process of training GAN and used it to classify the data. These studies focused on high productivity in generation so that they could not control the type of generated data.

Recently, some researchers began to set conditions on the data they generate. Sohn et al. (2015) and Walker et al. (2016) inputted data and conditions together into VAE and generated data whose type is what they want, called conditional VAE (CVAE). van den Oord et al. ( 2017) set discrete embedding space for generating a specific data using vector quantized variational auto-encoder (VQ-VAE), but because of discrete space, they could not control latent space continuously. Larsen et al. ( 2015) used both VAE and GAN in one generative model. As they just mixed two models and did not analyzed a latent space, so that the manifold of data was hidden to us. To generate image with a specific feature, they extracted a visual attribute vector which is a mean of vector in latent space. Mirza & Osindero (2014) inputted not only data but also conditions into GAN to create data that we want, called conditional GAN (CGAN). Chen et al. ( 2016) used mutual information for inducing latent codes (InfoGAN) and Nguyen et al. (2017) added a condition network that tells the generator what to generate (PPGN) . These two models needed an additional input to generate the type of data we want. These studies make us to generate data with condition, but we still do not know about latent space and it is hard to find the location of a specific feature in the latent space. Therefore, we propose a model that learns to generate concrete features that we want from the latent space determined when LSC-VAE is trained.

Some studies attempted to transfer the given data to others which have different features or even in different domain. Tran et al. (2017) 2017) proposed StarGAN that could do multiple translation tasks with one model. These studies have been conducted to transform the data into those in other domains. However, they could not generate new data without input data. In addition, the size of latent space of most of them was too large. We aim to generate conditioned data even with a small size of latent space.

3. THE PROPOSED METHOD:

In this section, we present a method to generate the data with the corresponding characteristics by inputting the latent variable which follows the specific distribution in latent space. As the instability caused by the larger scale of latent space in this process, we use the modified VAE, called LSC-VAE1 . As shown in Fig. 2 (a), we train the LSC-VAE with L prior for the data to be projected by the encoder into the desired position in the latent space according to the characteristics of the data. The trained decoder of the LSC-VAE is used as a generator of LSC-GAN so that the LSC-GAN generates the data with the corresponding features by using latent variables sampled from a specific distribution.

The proposed model is divided into two phases: initializing latent space (Fig. 2(a) ) and generating data (Fig. 2(b) ). In the first phase, latent semantic controlling VAE (LSC-VAE) is trained to project data into a specific location of latent space according to its features, and it learns to reconstruct data which is compressed. The decoder of LSC-VAE is used in the generator (G) of LSC-GAN in the second phase. G and discriminator (D) are trained simultaneously so that G can produce data similar to real data as much as possible and that D can distinguish the real from the fake. The architecture of the generation process is shown in Fig. 2 (b).

3.1. INITIALIZING THE LATENT SPACE WITH LSC-VAE:

Auto-encoder has been traditionally used to represent manifold without supervision. In particular, VAE, one type of auto-encoders, is one of the most popular approaches to unsupervised learning of complicated distributions. Since any supervision is not in training process, the manifold constructed is hidden to us. As we mentioned in Section 1, this is usually too complex to generate the conditioned data. Therefore, we allow LSC-VAE to learn a representation of data with supervision. It compresses data into a particular place on latent space according to its features. The proposed model consists of two modules that encode a data x i to a latent representation z i and decode it back to the data space, respectively.

EQUATION

Index i means a feature which is included in data x and latent space z. The encoder is regularized by imposing a prior over the latent distribution P (z). In general, z ∼ N (0, I) is chosen, but we choose z i ∼ N (µ i , I)for controlling latent space. In addition, if we want to produce data which has multiple features i, j, we generate data from z ij ∼ N (µ i + µ j , I)2 . The loss function of LSC-VAE is as follows.

L LSC-V AE = -E zi∼Q(zi|xi) [logP (x i | z i )] + D KL [Q(z i | x i ) || P (z i )] = L V AE + L prior (3)

where D KL is the Kullback-Leibler divergence. The first term of equation 3 is related to reconstruction error and the second term is related to appropriate projection of data to the latent space. For example, when LSC-VAE projects the data with iand j-features into the latent space, it is trained to map the data into the pre-defined latent space (N (µ i + µ j , I)) with L prior in equation 3 so as to locate the reconstructed data as similar to the input data according to its characteristics using L V AE . Therefore, LSC-VAE can be used in initializing GAN and it is demonstrated that LSC-VAE is valid and efficient for LSC-GAN in the next section.

3.2. GENERATING DATA WITH LSC-GAN:

GAN has led to significant improvements in data generation (Goodfellow et al., 2014) . The basic training process of GAN is to adversely interact and simultaneously train G and D. However, because the original GAN has a critical problem, unstable process of training (Radford et al., 2015) , the least squares GAN (LS-GAN) is proposed to reduce the gap between the distributions of real data and fake data by Mao et al. (2017) . Equation 4shows the objective function of the LS-GAN. p data is the probability distribution of the real data. G(z) is generated data from a probability distribution p z , and it is distinguished from the real by D.

min D E x∼p data (x) [(D(x) -1) 2 ] + E z∼pz(z) [(D(G(z))) 2 ] min G E z∼pz(z) [D(G(z)) -1) 2 ] (4)

The main differences of the proposed model with VAE-GAN and LS-GAN is that LSC-GAN is based on LSC-VAE for initializing a latent space to control it. To produce the type of data we want, we just input latent variable z i ∼ N (µ i , I) to G, if the data has i-feature. Besides, we add the encoder of LSC-VAE into LSC-GAN to make sure that the generated data actually have the desired features. The encoder projects back to latent space so as to be trained to minimize the difference between latent space where data is generated and the space where the compressed data is projected. Equation 5 is about loss of D and loss of encoder and G.

min

D E xi∼p data (xi) [(D(x i ) -1) 2 ] + E zi∼pz(zi) [(D(G(z i ))) 2 ] min Q,G E zi∼pz(zi) [D(G(z i )) -1) 2 ] + D KL [Q(z i | G(z i )) || N (µ i , I)]

(5)

3.2.1. PRE-TRAINED GENERATOR:

Since the original GAN has disadvantage that the generated data are insensible because of the unstable learning process of the G, we pre-train G with decoder of LSC-VAE. The goal of the learning process of generating data of G is the same as equation 6 from equation 5, and it is equivalent to that of equation 7. However, it is not efficient to pre-train the G, because it depends on the parameters of the D. Therefore, we change this equation to equation 8 again, and it is represented only by the parameters of G. In this paper, to train the G with equation 8, we use the decoder of LSC-VAE, which is trained by using Dec(Enc(x)) ≈ x. The result of LSC-VAE is that

EQUATION

where X i is real dataset with i-feature.

3.2.2. VALIDITY AND EFFICIENCY OF LSC-VAE:

From the game theory point of view, the GAN converges to the optimal point when G and D reach the Nash equilibrium. In this section, let p G be the probability distribution of data created from G.

We show that if G(z) ≈ x, i.e., p data ≈ p G , the GAN reaches the Nash equilibrium. We define

J(D, G) = x p data (x)D(x)dx + z p z (z)(1 -D(G(z)))dz and K(D, G) = x p data (x)(1 - D(x))dx + z D(G(z))dz 3 .

We train G and D to minimize J(D, G) and K(D, G) for each. Then, we can define the Nash equilibrium of the LSC-GAN as a state that satisfies equation equation 9 and equation equation 10. Fully trained G and D are denoted by G * and D * , respectively.

J(D * , G * ) ≤ J(D * , G)∀G (9) K(D * , G * ) ≤ K(D, G * )∀D (10)

Theorem 1. If p data ≈ p G almost everywhere, then the Nash equilibrium of the LSC-GAN is reached.

Before proving this theorem, we need to prove the following two lemmas.

Lemma 1. J(D * , G) reaches a minimum when p data (x) ≤ p G (x) for almost every x.

Lemma 2. K(D, G * ) reaches a minimum when p data (x) ≥ p G * (x) for almost every x.

The proof of Lemma 1 and 2 were discussed by Kim et al. (Kim et al., 2018b) . We assume that p data ≈ p G . From Lemma 1 and Lemma 2, if p data ≈ p G , then J(D, G) and K(D, G) both reach minima. Therefore, the proposed GAN reaches the Nash equilibrium and converges to optimal points. By theorem 1, GAN converges when p d ≈ p g , and it is done to some extent by the modified VAE, i.e.

| p LSC-V AE d -p LSC-V AE g |≤| p d -p g | since the one goal of modified VAE is P (x | Q(z | x)) ≈ x.

Therefore, the proposed method is useful to initialize the weight of the generative model. However, it shows only validity of using VAE when learning GAN. We prove that it is also efficient by proving theorem 2. Assume that a model f is well-trained if and only if ∇L f ≈ 0, where L f is the loss function of f . Theorem 2. Let en k , de k be k epoch-trained encoder and decoder whose goal is de k (en k (x)) ≈ x ∈ X . D and de's are linear functions. 4 Let

L k G = E z∼Q(z|x) [(D(de k (z)) -1) 2 ], then L f → 0 as k → ∞.

Proof. Notice that the derivative is unique, and a derivative of linear function is itself. Since en and de are trained with L V AE and L prior , the following statement is satisfied.

EQUATION

By differentiating the formula,

∇L k G ≈ 0 as k → ∞ (12) ⇔ [p(z)(D(de k (z)) -1)DD(de k (z))](z) → 0 as k → ∞ where z ∼ Q(z | x). Since the derivative of linear function is itself, it derives to ⇔ p(z)(D • de k (z) -1)D • de k (z) → 0 as k (13)

With the fact that D(x) = 1, ∀x ∈ X and equation 11, it finally derives to

EQUATION

By theorem 1 and 2, the proposed learning process is valid and efficient.

4. EXPERIMENTS:



4.1. DATASET AND EXPERIMENTAL SETTING:

To verify the performance of the proposed model, we use the celebA dataset (Liu et al., 2015) . It is a large-scale face attributes dataset. We crop the initial 178×218 size to 138×138, and resize them as 64×64. We use 162,769 images in celebA and 14 attributes: black hair, blond hair, gray hair, male, female, smile, mouth slightly open, young, narrow eyes, bags under eyes, mustache, eyeglasses, pale skin, and chubby. We assign 20 dimensions to each feature and set mean of the i th -20 dimensions as 1. For example, if an image has i-feature, the elements of i * 20 th to (i + 1) * 20 th of the image's latent variable are 1 in average and 0 in the remainder and we denote that latent variable as n i .

4.2. GENERATED IMAGES:

As shown in Fig. 3 , we generate images from a specific latent space by using LSC-GAN. The images in the first column are generated to have 'female' and 'blond hair' features. We confirm that the condition works well. The images in the remaining columns are transformed using equation 15 for the features listed below. For example, if we generate an image x i which has i-feature from the latent variable z i , we add n j to add j-feature into the image.

EQUATION

where x ij is an image which has iand j-features, and z i is the latent variable for i-feature. To show that the proposed model does not simply memorize data but understand features of data and generate them, we generate images from a series between two random images as in DCGAN. As shown in Fig. 4 , the change between images is natural so that we can say that the latent space of LSC-GAN is a manifold. Besides, the images in the middle column have both features of images in leftmost and rightmost, resulting in more simple manifold as shown in Fig. 1 .

Unlike other GAN models, the LSC-GAN fully understands features of data so as to generate data including inverse-feature. We only train the model about the presence of the 'pale skin' and 'smile' features, but the model also learned about the reverse of 'pale skin' and 'smile' automatically as shown in the fourth and the ninth column of Fig. 5 . Besides, if we assign a value of 2 rather than 1 to the average of latent variable which is related to 'mustache', we can see that more mustaches are created in the last column in Fig. 5 . Therefore, our model can automatically infer and generate the To verify the proposed model, we conduct subjective test about the quality of the generated data. We generate data by using DCGAN, EBGAN, and the proposed GAN. We randomly choose 25 generated data for each model. We perform the subjective test on 30 subjects and ask them to evaluate the quality of the generated data in 5 ways: very low, low, medium, high, and very high. We collect the results of 750 questionnaires, which are the evaluated result of 25 generated images by 30 subjects, and summarize them in Table 1 . We score 1,2,3,4, and 5 points for each evaluation result which is shown in the last column in Table 1 . 

5. CONCLUSION:

In this paper, we address some of significant issues in generative models: unstable training, hidden manifold of data, and extensive hardware resource. To generate a data whose type is what we want, we propose a novel model LSC-GAN which can control a latent space to generate the data that we want. To deal with a larger scale of latent space cause by deploying various distributions in one latent space, we use the LSC-VAE and theoretically prove that it is a proper method. Also, we confirm that the proposed model can generate data which we want by controlling the latent space. Unlike the existing generative model, the proposed model deals with features continuously, not discretely and compresses the data efficiently.

Based on the present findings, we hope to extend LSC-GAN to more various datasets such as ImageNet or voice dataset. In future work, we plan to conduct more experiments with various parameters to confirm the stability of model. We will also experiment by reducing the dimension of the latent space to verify that the proposed model is efficient. Besides, since the encoder can project the data to the latent space according to the features inherent in data, it could be used as a classifier.

:

Intuitively, as opposed to GAN, VAE constructs the distribution of data explicitly, resulting in efficient training of LSC-GAN. We deal with this theoretically in Section 3.2.

More precisely, N (µi + µj, 2I) is correct but calculated as N (µi + µj, I) for convenience in computation and scalability.

Since (1 p data (x)>p G (x) ) 2 = 1 p data (x)>p G (x) and (1 -1 p data (x)>p G (x) ) 2 = (1 -1 p data (x)>p G (x)), we eliminate the square.

In fact, it is enough that D does not satisfy DD = D a , where a ≤ -1.

Figures and Tables:

proposed disentangled representation learning GAN (DRGAN) for pose-invariant face recognition. Reed et al. (2016a;b) tried matching latent space of text and images and finally they translated text to image. Zhang et al. (2017) also translated text to image and generated photo-realistic images conditioned on text by stacking models (StackGAN). Zhu et al. (2017) and Kim et al. (2017b) discovered cross-domain relations with CycleGAN and DiscoGAN. They can translate art style, face features, and bags to shoes. While other models could only do one conversion task, Choi et al. (

Figure 2: (a) The process of pre-defining a latent space. The LSC-VAE is trained to project the data into the appropriate position on latent space. (b) Generating process of the proposed method. The latent space is pre-defined in the process of (a).

Figure 3: The generated images. The images are shown in each column according to the features below the columns.

Figure 4: Interpolation between a series between images in leftmost and rightmost columns.

Figure 5: The result of generating inverse-features. Our proposed model automatically learns inverse features such as dark skin (inverse of 'pale skin') and frown (inverse of 'smile').

The results of subjective test about the quality of the generated data by DCGAN, EBGAN, and the proposed model.Our model not only generates images according to input conditions, but also compress efficiently. We calculate the compression rate with rate= size inputdata /size bottleneck /#classes. As shown in Table2, our proposed model has the best compression rate compared to others. This proves experimentally that LSC-VAE, theoretically proven with theorems 1 and 2, has been helpful in initializing the weights of the LSC-GAN, and it can achieve good performance even in small latent spaces.

Table content: <table><tr><td>Model</td><td colspan="2">DCGAN EBGAN</td><td>Ours</td></tr><tr><td>Very low (1)</td><td>38.8%</td><td>18.2%</td><td>3.6%</td></tr><tr><td>Low (2)</td><td>38.5%</td><td>37.6%</td><td>20.8%</td></tr><tr><td>Medium (3)</td><td>15.3%</td><td>28.9%</td><td>36.7%</td></tr><tr><td>High (4)</td><td>4.3%</td><td>11.6%</td><td>24.8%</td></tr><tr><td>Very high (5)</td><td>3.1%</td><td>3.6%</td><td>14.1%</td></tr><tr><td>Score</td><td>1.943</td><td>2.447</td><td>3.251</td></tr><tr><td>4.3 COMPRESSION RATE</td><td/><td/><td/></tr></table>

The compression rate of models with 14 classes.

Table content: <table><tr><td>Model</td><td>U-NET</td><td colspan="3">VQ-VAE DiscoGAN CycleGAN StarGAN LSC-GAN</td></tr><tr><td colspan="3">Compression rate 0.504/14 48.006/14 40.922/14</td><td>0.378/14</td><td>0.188</td><td>24.546</td></tr></table>

Peer review (iclr2019):
 The paper proposes a generative model that combines VAE and GAN. The main idea of the paper is to replace the standard normal distribution used in VAE with a normal distribution centered at a feature representation of the input image. In other words, the prior distribution is data adaptive. The paper compares the proposed generative model to DCGAN and EBGAN for image generation quality using the CelebA dataset and reports better human preference score. Overall, the paper is poorly written with incorrect technical descriptions and vague expositions. The two baselines (DCGAN and EBGAN) are also quite out-dated. Beating these two baselines are insignificant, particularly there are GAN methods that can generate high quality images without an encoder. For example, the Progressive GAN by Terro et. al. (ICLR 2018), SNGAN by Miyato et. al.(ICLR 2018), and GAN with zero-center gradient penalty by Mescheder et. al. (ICML 2018). The paper also fails to give a literature overview of effort in combining VAE and GAN. For example, Zhiting et. al. ICLR 2018 and Liu et. al. NIPS 2017. Technical errors - In the related works section, the paper states that VAEs and GANs are both based on maximum likelihood. This statement is incorrect as GANs are based on distribution matching. Vague exposition - In Section 2, the paper states that "Larsen et al. (2015) used both VAE and GAN in one generative model. As they just mixed two models and did not analyzed a latent space, so that the manifold of data was hidden to us." Isn*t the data manifold in this case a multivariate Gaussian distribution. The paper fails to explain what it means by the sentence. - In Section 3.1, the paper states that "Since any supervision is not in training process, the manifold constructed is hidden to us." Again, the reviewer fails to understand what the paper means. - In Section 3.2.1, the paper states that "it is not efficient to pre-train the G , because it depends on the parameters of the D." This sentence is confusing. Isn*t pretraining just meaning using a pretrained decoder weight to initialize G?