Difference-Seeking Generative Adversarial Network--Unseen Sample Generation

1 INTRODUCTION . Unseen data1are not samples from the distribution of the training data and are difficult to collect . It has been demonstrated that unseen samples can be applied to several applications . Dai et al . ( 2017 ) proposed how to create complement data , and theoretically showed that complement data , considered as unseen data , could improve semi-supervised learning . In novelty detection , Yu et al . ( 2017 ) proposed a method to generate unseen data and used them to train an anomaly detector . Another related area is adversarial training Goodfellow et al . ( 2015 ) , where classifiers are trained to resist adversarial examples , which are unseen during the training phase . However , the aforementioned methods only focus on producing specific types of unseen data , instead of enabling the generation of general types of unseen data . In this paper , we propose a general framework called difference-seeking generative adversarial network ( DSGAN ) , to generate a variety of unseen data . The DSGAN is a generative approach . Traditionally , generative approaches , which are usually conducted in an unsupervised learning manner , are developed for learning the data distribution from its samples , from which subsequently , they produce novel and high-dimensional samples , such as the synthesized image Saito et al . ( 2018 ) . A state-of-the-art approach is the so-called generative adversarial network ( GAN ) Goodfellow et al . ( 2014 ) . GAN produces sharp images based on a game-theoretic framework , but it can be difficult and unstable to train owing to multiple interaction losses . Specifically , GAN consists of two functions : generator and discriminator . Both functions are represented as parameterized neural networks . The discriminator network is trained to determine whether the inputs belong to the real dataset or fake dataset created by the generator . The generator learns to map a sample from a latent space to some distribution to increase the classification errors of the discriminator . 1In traditional machine learning scenarios , `` unseen '' data corresponds to data that is not used or seen during the training stage but rather the testing stage . The distribution of `` unseen '' data could be same as or different Nevertheless , if a generator can learn to create unseen data , then a traditional GAN requires numerous training samples of unseen classes for training , leading to a contradiction with the definition of the unseen data . This fact motivates us to present the DSGAN , which can generate unseen data by adopting seen data as training samples ( see Fig . 9 , which illustrates the difference between GAN and the DSGAN , in Appendix A ) . The key concept is to consider the distribution of the unseen data as the difference between two distributions that are relatively easy to obtain . For example , the out-of-distribution examples in the MNIST dataset , from another perspective , are found to belong to the differences between the sets of examples in MNIST and the universal set . It should be noted that in traditional GAN , the target distribution is identical to the training data distribution ; however , in the DSGAN these two distributions , are considered to be different . This paper makes the following contributions : ( 1 ) We propose the DSGAN to generate any unseen data only if the density of the target ( unseen data ) distribution is the difference between those of any two distributions , pd̄ and pd . ( 2 ) We show that the DSGAN possesses the flexibility to learn different target ( unseen data ) distributions in two key applications , semi-supervised learning and novelty detection . Specifically , for novelty detection , the DSGAN can produce boundary points around the seen data because this type of unseen data is easily misclassified . For semi-supervised learning , the unseen data are linear combinations of any labeled data and unlabeled data , excluding the labeled and unlabeled data themselves2 . ( 3 ) The DSGAN yields results comparable to a semi-supervised learning but with a short training time and low memory consumption . In novelty detection , combining both the DSGAN and variational auto-encoder ( VAE , Kingma & Welling ( 2014b ) ) methods achieve the state-of-the-art results . 2 PROPOSED METHOD-DSGAN . 2.1 FORMULATION . We denote the generator distribution as pg and training data distribution as pd , both in an N - dimensional space . Let pd̄ be the distribution decided by the user . For example , pd̄ can be the convolution of pd and normal distribution . Let pt be the target distribution that the user is interested in , and it can be expressed as ( 1− α ) pt ( x ) + αpd ( x ) = pd̄ ( x ) , ( 1 ) where α ∈ [ 0 , 1 ] . Our method , the DSGAN , aims to learn pg such that pg = pt . Note that if the support set of pd belongs to that of pd̄ , then there exists at least an α such that the equality in ( 1 ) holds . However , even if the equality does not hold , intuitively , the DSGAN attempts to learn pg such that pg ( x ) ∼ pd̄ ( x ) − αpd ( x ) 1− α with the constraint , pg ( x ) ≥ 0 . Specifically , the generator will output samples located in the high-density areas of pd̄ − αpd . Furthermore , we show that the DSGAN can learn pg , whose support set is the difference between those of pd̄ and pd in Theorem 1 . First , we formulate the generator and discriminator in GANs . The inputs , z , of the generator are drawn from pz ( z ) in an M -dimensional space . The generator function , G ( z ; θg ) : RM → RN , represents a mapping to the data space , where G is a differentiable function with parameter θg . The discriminator is defined as D ( x ; θd ) : RN → [ 0 , 1 ] , which outputs a single scalar . D ( x ) can be considered as the probability that x belongs to a class of the real data . Similar to traditional GAN , we train D to distinguish the real data from the fake data sampled from G. Concurrently , G is trained to produce realistic data that can mislead D. However , in the DSGAN , the definitions of “ real data ” and “ fake data ” are different from those in traditional GAN . The samples from pd̄ are considered as real , but those from the mixture distribution between pd and pg are considered as fake . The objective function is defined as follows : from the `` seen '' data , according to applications . In this paper , we focus on the scenario that the two distributions are different . 2The linear combination of any labeled data and unlabeled data probably belongs to the set of seen data ( labeled data and unlabeled data ) , which contradicts the definition of unseen data . Thus , the samples generated by the DSGAN should not include the seen data themselves . V ( G , D ) : = Ex∼pd̄ ( x ) [ logD ( x ) ] + ( 1− α ) Ez∼pz ( z ) [ log ( 1−D ( G ( z ) ) ) ] + αEx∼pd ( x ) [ log ( 1−D ( x ) ) ] . ( 2 ) We optimize ( 2 ) by a min–max game between G and D , i.e. , min G max D V ( G , D ) . During the training procedure , an iterative approach , like traditional GAN , is to alternate between k steps of training D and one step of training G. In practice , minibatch stochastic gradient descent via backpropagation is used to update θd and θg . Thus , for each pg , pd , and pd̄ , m samples are required for computing the gradients , where m is the number of samples in a minibatch . The training procedure is illustrated in Algorithm 1 in Appendix A . The DSGAN suffers from the same drawbacks as traditional GAN , ( e.g. , mode collapse , overfitting , and strong discriminator ) so that the generator gradient vanishes . There are literature Salimans et al . ( 2016 ) ; Arjovsky & Bottou ( 2017 ) ; Miyato et al . ( 2018 ) focusing on dealing with the above problems , and such concepts can be readily combined with the DSGAN . Li et al . ( 2017 ) and Reed et al . ( 2016 ) proposed an objective function similar to ( 2 ) . Their goal was to learn the conditional distribution of training data . However , we aim to learn the target distribution , pt , in Eq . ( 1 ) , and not the training data distribution . 2.2 CASE STUDY ON VARIOUS UNSEEN DATA GENERATION . To achieve a more intuitive understanding about the DSGAN , we conduct several case studies on two-dimensional ( 2D ) synthetic datasets and MNIST . In Eq . ( 1 ) , α = 0.8 is used . 1 7 1 7 Figure 4 : Illustration of the difference-set seeking in MNIST . Figure 5 : DSGAN learns the difference between two sets . Complement samples generation Fig . 1 illustrates that the DSGAN can generate complement samples between 2 circles . Denoting the density function of the two circles as pd , we assign the samples drawn from pd̄ as linear combinations of the two circles . Then , by applying the DSGAN , we achieve our goal of generating complement samples . In fact , this type of unseen data is used in semi-supervised learning . Boundary samples generation Fig . 2 illustrates that the DSGAN generates boundary points between four circles . This type of unseen data is used in novelty detection . In this case , we assign pd and pd̄ as “ the density function of four circles ” and “ the convolution of pd and normal distribution , ” respectively . The basis of our concept is also illustrated by a one-dimensional ( 1D ) example in Fig . 3 . Difference-set generation We also validate the DSGAN on a high-dimensional dataset such as MNIST . In this example , we define pd as the distribution of digit “ 1 ” and pd̄ as the distribution containing two digits “ 1 ” and “ 7 ” . Because the density , pd ( x ) , is high when x is digit “ 1 , ” the generator is prone to output digit “ 7 ” with a high probability . More sample qualities of DSGAN on CelebA can be refer to Appendix G. From the above results , we can observe two properties of the generator distribution , pg : i ) the higher the density of pd ( x ) , the lower the density of pg ( x ) ; ii ) pg prefers to output samples from the high-density areas of pd̄ ( x ) − αpd ( x ) . 2.3 DESIGNING pd̄ Thus far , we have demonstrated how the DSGAN can produce various types of unseen data by choosing a specific pd̄ . In this section , we introduce a standard procedure to design pd̄ , and illustrate each step with pictures . Step 1 . First , the training data , pd , are collected ( Fig . 6 ( a ) ) . Step 2 . Second , based on the applications , the desired unseen data distribution is defined ( e.g. , complement samples for semi-supervised learning ) ( Fig . 6 ( b ) ) . In the above procedure , the most important step is to determine which types of unseen data are suitable for a specific problem ( Step 2 ) . In this paper , we show two types of unseen data , which are useful in semi-supervised learning and novelty detection . However , determining all types of unseen data for all applications is beyond the scope of this study , and we leave this for future work . Furthermore , we provide a method ( see Appendix B in supplementary materials ) by reformulating the objective function ( 2 ) , so that it is more stable to train the DSGAN .

Summary: This paper proposed DSGAN which learns to generate unseen data from seen data distribution p_d and its somehow “broad” version p_{\hat d} (E.g., p_d convolved with Gaussian). The “unseen data” is the one that appears in p_{\hat d} but not in p_d. DSGAN is trained to generate such data. In particular, it uses samples from p_d as fake data and samples from p_{\hat d} as the real one.