Difference-Seeking Generative Adversarial Network--Unseen Sample Generation

1 INTRODUCTION . Unseen data1are not samples from the distribution of the training data and are difficult to collect . It has been demonstrated that unseen samples can be applied to several applications . Dai et al . ( 2017 ) proposed how to create complement data , and theoretically showed that complement data , considered as unseen data , could improve semi-supervised learning . In novelty detection , Yu et al . ( 2017 ) proposed a method to generate unseen data and used them to train an anomaly detector . Another related area is adversarial training Goodfellow et al . ( 2015 ) , where classifiers are trained to resist adversarial examples , which are unseen during the training phase . However , the aforementioned methods only focus on producing specific types of unseen data , instead of enabling the generation of general types of unseen data . In this paper , we propose a general framework called difference-seeking generative adversarial network ( DSGAN ) , to generate a variety of unseen data . The DSGAN is a generative approach . Traditionally , generative approaches , which are usually conducted in an unsupervised learning manner , are developed for learning the data distribution from its samples , from which subsequently , they produce novel and high-dimensional samples , such as the synthesized image Saito et al . ( 2018 ) . A state-of-the-art approach is the so-called generative adversarial network ( GAN ) Goodfellow et al . ( 2014 ) . GAN produces sharp images based on a game-theoretic framework , but it can be difficult and unstable to train owing to multiple interaction losses . Specifically , GAN consists of two functions : generator and discriminator . Both functions are represented as parameterized neural networks . The discriminator network is trained to determine whether the inputs belong to the real dataset or fake dataset created by the generator . The generator learns to map a sample from a latent space to some distribution to increase the classification errors of the discriminator . 1In traditional machine learning scenarios , `` unseen '' data corresponds to data that is not used or seen during the training stage but rather the testing stage . The distribution of `` unseen '' data could be same as or different Nevertheless , if a generator can learn to create unseen data , then a traditional GAN requires numerous training samples of unseen classes for training , leading to a contradiction with the definition of the unseen data . This fact motivates us to present the DSGAN , which can generate unseen data by adopting seen data as training samples ( see Fig . 9 , which illustrates the difference between GAN and the DSGAN , in Appendix A ) . The key concept is to consider the distribution of the unseen data as the difference between two distributions that are relatively easy to obtain . For example , the out-of-distribution examples in the MNIST dataset , from another perspective , are found to belong to the differences between the sets of examples in MNIST and the universal set . It should be noted that in traditional GAN , the target distribution is identical to the training data distribution ; however , in the DSGAN these two distributions , are considered to be different . This paper makes the following contributions : ( 1 ) We propose the DSGAN to generate any unseen data only if the density of the target ( unseen data ) distribution is the difference between those of any two distributions , pd̄ and pd . ( 2 ) We show that the DSGAN possesses the flexibility to learn different target ( unseen data ) distributions in two key applications , semi-supervised learning and novelty detection . Specifically , for novelty detection , the DSGAN can produce boundary points around the seen data because this type of unseen data is easily misclassified . For semi-supervised learning , the unseen data are linear combinations of any labeled data and unlabeled data , excluding the labeled and unlabeled data themselves2 . ( 3 ) The DSGAN yields results comparable to a semi-supervised learning but with a short training time and low memory consumption . In novelty detection , combining both the DSGAN and variational auto-encoder ( VAE , Kingma & Welling ( 2014b ) ) methods achieve the state-of-the-art results . 2 PROPOSED METHOD-DSGAN . 2.1 FORMULATION . We denote the generator distribution as pg and training data distribution as pd , both in an N - dimensional space . Let pd̄ be the distribution decided by the user . For example , pd̄ can be the convolution of pd and normal distribution . Let pt be the target distribution that the user is interested in , and it can be expressed as ( 1− α ) pt ( x ) + αpd ( x ) = pd̄ ( x ) , ( 1 ) where α ∈ [ 0 , 1 ] . Our method , the DSGAN , aims to learn pg such that pg = pt . Note that if the support set of pd belongs to that of pd̄ , then there exists at least an α such that the equality in ( 1 ) holds . However , even if the equality does not hold , intuitively , the DSGAN attempts to learn pg such that pg ( x ) ∼ pd̄ ( x ) − αpd ( x ) 1− α with the constraint , pg ( x ) ≥ 0 . Specifically , the generator will output samples located in the high-density areas of pd̄ − αpd . Furthermore , we show that the DSGAN can learn pg , whose support set is the difference between those of pd̄ and pd in Theorem 1 . First , we formulate the generator and discriminator in GANs . The inputs , z , of the generator are drawn from pz ( z ) in an M -dimensional space . The generator function , G ( z ; θg ) : RM → RN , represents a mapping to the data space , where G is a differentiable function with parameter θg . The discriminator is defined as D ( x ; θd ) : RN → [ 0 , 1 ] , which outputs a single scalar . D ( x ) can be considered as the probability that x belongs to a class of the real data . Similar to traditional GAN , we train D to distinguish the real data from the fake data sampled from G. Concurrently , G is trained to produce realistic data that can mislead D. However , in the DSGAN , the definitions of “ real data ” and “ fake data ” are different from those in traditional GAN . The samples from pd̄ are considered as real , but those from the mixture distribution between pd and pg are considered as fake . The objective function is defined as follows : from the `` seen '' data , according to applications . In this paper , we focus on the scenario that the two distributions are different . 2The linear combination of any labeled data and unlabeled data probably belongs to the set of seen data ( labeled data and unlabeled data ) , which contradicts the definition of unseen data . Thus , the samples generated by the DSGAN should not include the seen data themselves . V ( G , D ) : = Ex∼pd̄ ( x ) [ logD ( x ) ] + ( 1− α ) Ez∼pz ( z ) [ log ( 1−D ( G ( z ) ) ) ] + αEx∼pd ( x ) [ log ( 1−D ( x ) ) ] . ( 2 ) We optimize ( 2 ) by a min–max game between G and D , i.e. , min G max D V ( G , D ) . During the training procedure , an iterative approach , like traditional GAN , is to alternate between k steps of training D and one step of training G. In practice , minibatch stochastic gradient descent via backpropagation is used to update θd and θg . Thus , for each pg , pd , and pd̄ , m samples are required for computing the gradients , where m is the number of samples in a minibatch . The training procedure is illustrated in Algorithm 1 in Appendix A . The DSGAN suffers from the same drawbacks as traditional GAN , ( e.g. , mode collapse , overfitting , and strong discriminator ) so that the generator gradient vanishes . There are literature Salimans et al . ( 2016 ) ; Arjovsky & Bottou ( 2017 ) ; Miyato et al . ( 2018 ) focusing on dealing with the above problems , and such concepts can be readily combined with the DSGAN . Li et al . ( 2017 ) and Reed et al . ( 2016 ) proposed an objective function similar to ( 2 ) . Their goal was to learn the conditional distribution of training data . However , we aim to learn the target distribution , pt , in Eq . ( 1 ) , and not the training data distribution . 2.2 CASE STUDY ON VARIOUS UNSEEN DATA GENERATION . To achieve a more intuitive understanding about the DSGAN , we conduct several case studies on two-dimensional ( 2D ) synthetic datasets and MNIST . In Eq . ( 1 ) , α = 0.8 is used . 1 7 1 7 Figure 4 : Illustration of the difference-set seeking in MNIST . Figure 5 : DSGAN learns the difference between two sets . Complement samples generation Fig . 1 illustrates that the DSGAN can generate complement samples between 2 circles . Denoting the density function of the two circles as pd , we assign the samples drawn from pd̄ as linear combinations of the two circles . Then , by applying the DSGAN , we achieve our goal of generating complement samples . In fact , this type of unseen data is used in semi-supervised learning . Boundary samples generation Fig . 2 illustrates that the DSGAN generates boundary points between four circles . This type of unseen data is used in novelty detection . In this case , we assign pd and pd̄ as “ the density function of four circles ” and “ the convolution of pd and normal distribution , ” respectively . The basis of our concept is also illustrated by a one-dimensional ( 1D ) example in Fig . 3 . Difference-set generation We also validate the DSGAN on a high-dimensional dataset such as MNIST . In this example , we define pd as the distribution of digit “ 1 ” and pd̄ as the distribution containing two digits “ 1 ” and “ 7 ” . Because the density , pd ( x ) , is high when x is digit “ 1 , ” the generator is prone to output digit “ 7 ” with a high probability . More sample qualities of DSGAN on CelebA can be refer to Appendix G. From the above results , we can observe two properties of the generator distribution , pg : i ) the higher the density of pd ( x ) , the lower the density of pg ( x ) ; ii ) pg prefers to output samples from the high-density areas of pd̄ ( x ) − αpd ( x ) . 2.3 DESIGNING pd̄ Thus far , we have demonstrated how the DSGAN can produce various types of unseen data by choosing a specific pd̄ . In this section , we introduce a standard procedure to design pd̄ , and illustrate each step with pictures . Step 1 . First , the training data , pd , are collected ( Fig . 6 ( a ) ) . Step 2 . Second , based on the applications , the desired unseen data distribution is defined ( e.g. , complement samples for semi-supervised learning ) ( Fig . 6 ( b ) ) . In the above procedure , the most important step is to determine which types of unseen data are suitable for a specific problem ( Step 2 ) . In this paper , we show two types of unseen data , which are useful in semi-supervised learning and novelty detection . However , determining all types of unseen data for all applications is beyond the scope of this study , and we leave this for future work . Furthermore , we provide a method ( see Appendix B in supplementary materials ) by reformulating the objective function ( 2 ) , so that it is more stable to train the DSGAN .

Summary: This paper provides an interesting application of GAN which can generate the outlier distribution of training data which forces generator to learn the distribution of the low probability density area of given data. To show the effectiveness of the method, the author intuitively shows how it works on 2-D points data as well as the reconstructed Mnist dataset. Additionally, this approach reaches a comparable performance on semi-supervised learning and novelty detection task.