Title: DUAL IMPORTANCE WEIGHT GAN

Abstract: Generative Adversarial Networks (GAN) are trained to generate a sample image of interest. To this end, generative network of GAN learns implicit distribution of true dataset from the classification samples with candidate generated samples. However, in real implementation of GAN, training the generative network with limited number of candidate samples guarantees to properly represent neither true distribution nor the distribution of generator outputs. In this paper, we propose dual importance weights for the candidate samples represented in the latent space of auto-encoder. The auto-encoder is pre-trained with real target dataset. Therefore, the latent space representation allows us to compare real distribution and the distribution of generated samples explicitly. Dual importance weights iteratively maximize the representation of generated samples for both distributions: current generator outputs and real dataset. Proposed generative model not only resolves mode collapse problem of GAN but also improves the convergence on target distribution. Experimental evaluation shows that the proposed network learns complete modes of target distribution more stable and faster than state of the art methods.

1. INTRODUCTION:

Recently, generative networks have been widely studied thanks to the explosive and successful applications of Generative Adversarial Networks (GAN) proposed by Goodfellow et al. (2014) . Main difficulty of training step in the generative model is evaluating high-dimensional original probability distribution. Estimation of the probability distribution of target dataset from sparse samples is not a trivial task that plays a critical role in the quality of generated samples.

GAN is composed of two main networks: discriminator and generator. Core idea is learning the target distribution through two-player minimax game theory. The discriminator helps generator to learn the representation of the target distribution by distinguishing the difference between generated and real samples. As a result, generator is able to produce samples which resemble target real data. Although GAN has been successfully implemented in many applications, it suffers from ill-training problems such as oscillation between modes, mode collapsing, etc. Especially, mode collapsing results in the generation of samples only from a single or a few modes of target data distribution losing diversity of target dataset. Main reason of mode collapsing problem is that the discriminator is incapable of delivering any information about samples' diversity. Once the generator finds optimal point of a fixed discriminator in each generator training step, the network produces same samples driving the expectation value in objective function becomes minimum regardless of input noise vectors. Consequently, generator repeatedly generates certain good instance rather than diverse instances from entire data distribution. Metz et al. (2016) proposes unrolled GAN that trains generator using unrolled multiple samples rather than single target at the generator output, which alleviate mode collapsing. VEEGAN in Srivastava et al. (2017) employs a reconstruction network which is not only mapping real data distribution to a Gaussian in encoded space but also approximating reverse action of the generator. Because VEEGAN estimates implicit probability of real dataset, it prevents mode collapsing problem and produces more realistic samples. Bang & Shim (2018) propose a manifold guided generative adversarial network(MGGAN) for guiding a generator by adding another adversarial loss in manifold space transformed by pre-trained encoder network. This enables the generator to learn all modes of target distribution without impairing the quality of image. Although real data distribution is represented in the manifold loss of their second discriminator, it is not able to estimate explicit distance between the distributions. OT-GAN (Salimans et al. (2018) ) expands generator loss function using optimal transport theory. They define a new metric, M inibatchEnergyDistance, over probability distribution in a adversarially learned feature space. Although they make the condition of GAN to be more stable than other state-of-the-art research, the computational cost of their large mini-batches is expensive to be practical in real applications.

In all such previous approaches, using increased number of samples or larger mini-batches for the training of generator network allows more stable and improved performance. On the other hand, they require hugely increased computation cost and training time. Our idea is that given limited number of generated samples in the training step, representation ability of the samples can be evaluated and maximized for both distributions (current generator outputs and real dataset) adopting importance weights estimation of the samples.

In this paper, we propose dual importance weights for the generated candidate samples in generator network training step represented in the latent space of auto-encoder. The auto-encoder is pre-trained with real target dataset. We assume that auto-encoder trained with target dataset has ability to represent the distribution of target dataset in the latent space under optimally minimized dimensions. Therefore, the latent space representation allows us to compare real distribution and the distribution of generated samples explicitly. Our dual importance weights on generated samples iteratively maximize the representation ability for the generator and guide the generator to find complete modes of real data distribution. To this end, we expand generator objective function to save the diversity of target distribution via adopted auto-encoder which works as a bridge between data and manifold space. As a result, proposed dual importance weight GAN not only resolves mode collapse problem of GANs but also improves the convergence on target distribution. Transformed distributions to the latent space enables explicit representation of generated and real data. We calculate actual distance between the two distributions and achieve fast and robust convergence to optimal states avoiding mode collapse problem.

2. GAN WITH AUTO-ENCODER:

GAN is motivated by game theory that two players (generator and discriminator) compete with each other in a zero-sum game framework. The generator learns the target distribution via an adversarial training process with discriminator. When the generator transform a noise vector z into a data vector G(z) on target space, the discriminator tries to estimate the probability if input sample comes from target distribution P (x) or not. At the end of the training process, the generator produces outputs of P (x). GAN defines this adversarial problem as follow:

EQUATION

where the goal of GAN is to find the optimal parameter set of G and D in equation ( 1) minimizing the generator loss and maximizing the discriminator loss.

Auto-encoder is a neural network that learns optimal representation of unlabeled input data in encoded space. It generates original input through decoding network. Auto-encoder not only decreases the dimensionality of input data but also finds optimal abstraction of input data for better discrimination between subjects. Generative networks with auto-encoder and adversarial learning algorithms have been proposed to improve GAN. Adversarial auto-encoder (AAE) 

3. SAMPLING WITH DUAL IMPORTANCE WEIGHT:

Based on our GAN with auto-encoder, obtaining good samples is a critical for improved performance. Importance sampling is a statistical technique for estimating characteristics of distribution We define a new distance D a (s r , s g ) which measures the distance between paired real and generated samples in the latent space of auto-encoder.

EQUATION

where s r ∈ S r , s g ∈ S g denote samples from real and generated sample sets in encoded space.

Real sample s r is extracted from the distribution of real data in the encoded space and the the distribution of real data is obtained using Kernel density estimation (KDE) on the complete real dataset transformed to the encoded space of pre-trained auto-encoder.

We define the expectation of euclidean distance over mini-batch samples as the distance between real and generated samples. w k r and w k g are target importance weight and generator importance weight, respectively. Target importance weight is assigned to each target real sample and generator importance weight is assigned to each generated sample. First, generator importance weight w g indicates how well current generated sample represents generator outputs. In order to obtain the weight for each sample, we accumulate past N generated sample sets constructing the distribution of generator outputs using Kernel density estimation (KDE). A generated sample of high generator importance weight contributes more to the distance calculation in equation ( 2). When the current generated samples poorly represent the current generator distribution, they gets smaller weights and are ignored in the training. In this manner, iterative sample quality evaluation makes stable environment of GAN training procedure. Secondly, target real importance weight w r indicates how well all generated samples contribute to the estimation of current real data sample. We assign the weight for each target real sample based on the degree of how much the real sample is covered by generated samples. Calculated distance of a real sample from paired generated samples in the past iterations add importance to the real sample. In other words, if a real sample has higher distances to all generated samples during the past training steps, the real sample gets higher real importance weight. And then, in the next training step, network is updated to generated this isolated real sample due to the higher loss value. Finally, both importance weights encourage generated samples to cover entire target real data samples. At each training step, both weights are updated adjusting the importance of generated and target samples. Due to the elaborate evaluation of the generated sample quality at each training step, proposed network converges to target real distribution faster without mode collapse problem.

Our generator loss function is shown in equation ( 3). If GAN unexpectedly generates samples from single mode of real dataset, second term in equation ( 3) increases forcing the network to learn other modes.

min

EQUATION

Overall training procedures are summarized in Figure 6 . First we train auto-encoder with target real dataset. The target data distribution in the encoded space is estimated by KDE(Kernel density estimation) from entire dataset transformed to the encoded space. And then, we obtain same number of good real samples of batch size which reflects target distribution. Note that the real samples are extracted once and used for all iterations of the training. For optimal generated and real sample pair matching in distance D a calculation, we calculate distances among all real samples to all generated samples and assign pair one by one minimizing average distance. Based on generated candidate samples, our approach tries to find all aspects of real distribution qualifying generated samples.

Better generated samples evaluated by generator importance weight understand generator better.

Better target importance understands target real dataset better. Finally, our method let the generator understand target real dataset better.

4. EXPERIMENTAL EVALUATION:

We perform quantitative and qualitative evaluation on three datasets: Mixture of Gaussians, (stacked) MNIST, and Cifar10. Several similarity metrics are used to quantify generation performance.

4.1. MIXTURE OF GAUSSIANS:

We have created several test distributions made of Gaussians: mixture of eight 2D Gaussians located in a ring, 25 Gaussians located in a grid and 25 Gaussians randomly located, 27 Gaussians in a 3D cube. 25 random Gaussians are anisotropic. VEEGAN (Srivastava et al. (2017) ) and Unrolled GAN (Metz et al. (2016) ) are compared with our method. Identical network architecture is used for fair comparison. Generator has three layers of fully connected MLPs with 128 nodes without dropout and batch normalization and discriminator has two layers of fully connected MLPs without dropout and batch normalization. We choose the dimension of encoded space of auto-encoder that properly reproduces original data. Following Srivastava et al. (2017) , we employ two metrics for quantitative evaluation. First, number of modes found are counted. Secondly, the quality of samples are measured by high quality sample ratio (HQS). However it's hard to evaluate how the generated samples are able to cover the entire real distribution just using number of modes and percentage of high quality samples. Thus we measure the distance between estimated target and generated distribution. We map the sample to canonical space and count the number of points within each canonical unit. By measuring Jensen-Shannon divergence(JSD) between the distributions (P r , P g ), we evaluate how well generator follows target distribution. Figure 3 shows results on our mixture of Gaussians dataset. Proposed method outperforms over two state of the art methods showing much faster convergence without mode collapse problem. Table 1 summarizes quantitative results showing outstanding performance of our generative network. 

4.3. CIFAR 10:

We also evaluate our method using CIFAR-10 dataset which includes 32x32 color images with 10 classes collected by Krizhevsky & Hinton (2009) . The architecture is same to 4.2. Generated results of VEEGAN and DCGAN are collected from Srivastava et al. (2017) . As shown in Figure 6 , VEEGAN and DCGAN frequently include identical samples suffering from mode collapsing (see yellow circles).

5. CONCLUSION:

In this paper, we propose dual importance weights for the candidate samples represented in the latent space of auto-encoder. Dual importance weights iteratively maximize the representation of generated samples for both distributions: current generator outputs and real dataset. Evaluation and comparison are extensively performed on three datasets showing promising performance of the proposed method. On the other hand, our method involves additional computation for sample pair matching. Structure of auto-encoder and the dimension of encoded space can be further optimized for improved quality of generated samples.

:



Figures and Tables:

Figure 1: Proposed generative network with dual importance weights

Figure 3: Experimental results compared to VEEGAN(Srivastava et al. (2017)) and Unrolled GAN(Metz et al. (2016)): 2D Ring, 2D Grid(isotropic and unisotropic), and 3D Cube testset. Proposed method converges faster than state-of-the-art methods.

MNISTIn this experiment, we expand MoG dataset to image space with MNIST and stacked MNIST. Evaluation on MNIST data (Figure4) shows that proposed method generates more number of digits than original GAN. Stacked MNIST is designed for high complexity evaluation with extended number of modes. Stacked MNIST is synthesized by stacking different MNIST digits in respective colors. This synthesized dataset has 1000 modes which are the combinations of 10 classes in 3 channel. We use implemented generator architecture of standard DGGAN. For the discriminator, we use same architecture with DCGAN and Unrolled GAN. For VEEGAN, the architecture of discriminator is designed followingSrivastava et al. (2017). Auto-encoder has three convolutional layers with 5 by 5 filters, 2 fully connected layers for encoder part and one fully connected layer, 3 transposed convolutional layers are used for decoder. First, we train the classifier using MNIST dataset which assigns the mode to generated images. In this evaluation, we use 20,000 generated samples and they are given the mode from pre-trained classifier. For example, one sample has 3 channels which represent different digit. Digit for each sample in channel is determined by MNIST classifier.

Figure 4: Experimental results on MNIST dataset. First row is original GAN and second row is proposed method.

Figure 5: Generated stacked MNIST samples from trained models. We refer these images from Srivastava et al. (2017) paper except our result.

Figure 6: Experimental results are compared to VEEGAN and DCGAN using CIFAR 10 dataset: VEEGAN and DCGAN frequently suffer from mode collapsing in yellow circles.

Makhzani et al. (2015) is a type of auto-encoder combined with a discriminator and learns target data distribution using both reconstruction and adversarial losses. The adversarial loss let the encoder learn a posterior which follows imposed prior during adversarial training. More recently,Rosca et al. (2017) suggests a α-GAN which combines variational auto-encoder (VAE, Kingma & Welling (2013)) and GAN. Using reconstruction and adversarial losses, α-GAN tries to address mode collapse problem of GAN producing blurry images of VAE. Proposed generative network extends originalGAN Goodfellow  et al. (2014)  with auto-encoder. Auto-encoder trained with real dataset optimally reduces the dimension of the dataset. In the encoded space, auto-encoder extracts essential feature set to represent all modes and aspects of original data distribution which is ideal to guide generative network.

Table content: <table/>

Quantitative Evaluation: Number of modes found, HQS(High Quality Sample), and JSD(Jensen-Shannon divergence) between real and generated sample distributions. Best result is indicated bold face font.

Table content: <table><tr><td/><td>METRIC</td><td colspan="3">Unrolled GAN(std) VEEGAN(std) Proposed(std)</td></tr><tr><td/><td>Modes(Max 8)</td><td>8(0)</td><td>8(0)</td><td>8(0)</td></tr><tr><td>2D Ring</td><td>% HQS</td><td>30.9(0.006)</td><td>60.4(0.005)</td><td>85.5(0.006)</td></tr><tr><td/><td>JSD(real generated)</td><td>0.254(0.004)</td><td>0.19(0.005)</td><td>0.172(0.004)</td></tr><tr><td/><td>Modes(Max 25)</td><td>25(0)</td><td>24.1(0.31)</td><td>25(0)</td></tr><tr><td>2D Grid</td><td>% HQS</td><td>14.4(0.006)</td><td>65.4(0.006)</td><td>82.5(0.004)</td></tr><tr><td colspan="2">(uniform) JSD(real generated)</td><td>0.47(0.006)</td><td>0.21(0.004)</td><td>0.12(0.004)</td></tr><tr><td/><td>Modes(Max 25)</td><td>24.4(0.5)</td><td>24.3(0.637)</td><td>25(0)</td></tr><tr><td>2D Grid</td><td>% HQS</td><td>15.9(0.005)</td><td>63.7(0.007)</td><td>83.4(0.006)</td></tr><tr><td colspan="2">(random) JSD(real generated)</td><td>0.38(0.007)</td><td>0.32(0.007)</td><td>0.15(0.004)</td></tr><tr><td/><td>Modes(Max 27)</td><td>27(0)</td><td>26.6(0.5)</td><td>27(0)</td></tr><tr><td>3D Cube</td><td>% HQS</td><td>85.0(0.348)</td><td>43.3(0.007)</td><td>80.0(0.005)</td></tr><tr><td/><td>JSD(real generated)</td><td>0.194(0.004)</td><td>0.31(0.005)</td><td>0.125(0.004)</td></tr></table>

Peer review (iclr2019):
 The paper proposes a novel approach to address the issue of mode collapse in the GANs. I see two interesting ideas here: (a) Since the GAN formulation seeks to achieve distributional alignment by the generator PDF end the PDF corresponding to the real, target data, the paper proposes to represent the real data using the embedding from an AE encoder, and, (b) achieve distributional alignment via an importance sampling formulation. The paper claims that their implementation of the above ideas helps improves the issues of mode collapse and mode oscillation, thereby (?) improving sample diversity and speeds up convergence. The ideas seem promising and I was hoping that the paper carefully implemented an approach based on the above, compared with the existing approaches and reported the results carefully. I find that the paper falls short of doing this and does not merit a publication at this point. A major revision of the work is needed. The main shortcomings of the paper follow: 1. Clarity: The paper greatly suffers from a loose handling of technical concepts and terminology; statements are not made in a careful and measured manner; and, grammatical mistakes abound. There are too many to list them all. Following are exemplary but symptomatic of the way the entire paper is written. a. “Dual importance weights iteratively maximize the representation of generated samples for both distributions”. Maximize the representation? b. “Experimental evaluation shows that the proposed network learns complete modes of target distribution more stable and faster than state of the art methods.” What is meant by ‘more stable’? Please provide the metric used to measure the stability and indicate which experiment validates this claim. c. “Main reason of mode collapsing problem is that the discriminator is incapable of delivering any information about samples’ diversity. Once the generator finds optimal point of a fixed discriminator in each generator training step, the network produces same samples driving the expectation value in objective function becomes minimum regardless of input noise vectors.” Loose and overly broad statements about the underlying problems and speculations about the mechanisms and underlying behavior of the algorithms – both the state of the art ones and the proposed one, shouldn’t be made unless there is a strong justification provided via mathematical proofs or experiments. d. “Auto-encoder not only decreases the dimensionality of input data but also finds optimal abstraction of input data for better discrimination between subjects.” Optimal abstraction of input data for better discrimination between subjects? e. Equation (2) should use the 2-norm since s_r and s_g are not scalars. 2. Implementation and technical details: a. The distance defined in Equation (2) and using it to modify the generator loss function in (3) seems to be based on a heuristic. The authors should formally derive it from importance sampling. This will help formally ground the proposed modifications to the objective function and help understand the expected benefits and the convergence properties better. b. Secondly, the proposed distance function in (2) is supposed to be obtained using a kernel density estimator. KDEs bring their own challenges, especially when used in high-dimension spaces. It is not clear what the bias/ variance properties of the proposed KDE estimator are and how it impacts the overall objective. Further, no details are shared about the dimensionality of the embedding space and the bandwidths used to compute the KDE. c. Beyond the KDE, there are other mathematical / implementation details missing making it practically impossible to replicate results. For example, what are the mathematical expressions for the two set of weight? d. High quality sample ratio (HQS) metric is neither defined nor cited from other works. It is not clear how the quality is measured and how the high-quality samples are obtained. 3. Insufficient/ inadequate experimental validation: a. While experiments on toy dataset (mixture of gaussian) shows slight improvements compared to baselines, the experiments on CIFAR and MNIST only report qualitative results in Figure 4,5 and 6. This is not sufficient to establish that the proposed approach works better than approaches compared with. b. Quantitative evaluation on real datasets using the IS or the FID score (or even the metrics in Table 1) to establish the claimed improvement in performance – sample diversity and the quality of generated images. Perceptual measures such as SSIM and the one proposed in Zhang et al (CVPR 2018, arXiv 1801.03924) should be used to measure image quality. Precision, Recall and F1 can be used to provide a measure of similarity between the real and generated data. c. The claim about improved convergence is not adequately validated. d. Aside: Since the real target data is represented by an AE embedding, the time taken to train the AE should be used when comparing baselines that do not have pre-training, in Figure 4. I’m willing to revise my opinion if the authors address the above issues in the forthcoming discussion phase.