Difference-Seeking Generative Adversarial Network

1 INTRODUCTION . In machine learning , how to learn a probability distribution is usually conducted in a unsupervised learning manner . Generative approaches are developed for learning data distribution from its samples and thereafter produce novel and high-dimensional samples from learned distributions , such as image and speech synthesis ( Saito et al . ( 2018 ) ) . The state-of-the-art approaches is so-called Generative Adversarial Networks ( GAN ) ( Goodfellow et al . ( 2014 ) ) . GAN produces sharp images based on a game-theoretic framework , but can be tricky and unstable to train due to multiple interacting losses . Specifically , GAN consists of two functions : generator and discriminator . Both functions are represented as parameterized neural networks . The discriminator network is trained to classify whether or not inputs belong to real data or fake data created by the generator . The generator learns to map a sample from a latent space to some distribution to increase the classification errors of discriminator . GAN corresponds to a minimax two-player game , which ends if the generator actually learns the real data distribution . The generator is of main interest because the discriminator will be unable to differentiate between both distributions once the generator has been trained well . 1.1 MOTIVATIONS . In reality , it is difficult to collect training samples from unseen classes , which none of their samples involves the training phase , but their samples could be encountered in the testing phase . How to reject or recognize unseen data as ” abnormal “ ( not belonging to the training data ) is an important issue known as one-class classification ( Ruff et al . ( 2018 ) ) . Due to the absence of unseen data , most of algorithms are unsupervised ( Scholkopf & Smola ( 2001 ) ) . Based on the assumption that there do exist very few unseen examples , some approaches ( Wu & Ye ( 2009 ) ) focus on supervised learning using unbalanced data . In addition to one-class classification , Dai et al . ( 2017 ) theoretically show that complementary data , which is also considered as unseen data , can improve semi-supervised learning . Another related issues is adversarial attack , where classifiers may be vulnerable to adversarial examples , which are unseen during the training phase . Fig . ( 8 ) in Appendix A illustrates the applications regarding unseen data . Apparently , if we can generate unseen data via GAN , it is helpful for those applications . But , traditional GAN requires preparing plenty of training samples of unseen classes for training , leading to the contradiction with the prerequisite . This fact motivates us to design the proposed DSGAN , which can generate unseen data by taking seen data as training samples shown in Fig . ( 1 ) . The nuclear idea is to consider the distribution of unseen data as the difference of two distributions , which both are relatively easy to obtain . For example , out-of-distribution examples in the MNIST dataset , from another point of view , are found to belong to the difference of the set of examples in MNIST and the universal set . Examples in both sets are relatively easy to obtain . It should be noted that the target distribution is equal to the training data distribution in traditional GANs ; nevertheless , both distributions are considered different in DSGAN . 1.2 CONTRIBUTIONS . In this paper , we make the following contributions : ( 1 ) We propose a novel algorithm , Difference-Seeking Generative Adversarial Network ( DSGAN ) , where the density of target distribution pt is the difference between those of any two distributions , pd̄ and pd . Nevertheless , the differences of two densities being negative are not well-defined . Thus , instead of learning the target distribution pt directly , the generator distribution approximates pt by minimizing the statistical distance between the mixture distribution of pd and the generator distribution , and pd̄ . ( 2 ) Theoretical results based on traditional GAN are extended to the case of mixture distribution considered in this paper . With enough capacity of the generator and the discriminator , we show DSGAN can learn the generator distribution pg under mild condition where the support set of pg is the difference of support sets of pd̄ and pd . ( 3 ) We show that DSGAN possesses the flexibility to learn different target distributions in two key applications : semi-supervised learning and adversarial training . Samples from target distribution in semi-supervised learning must satisfy two conditions : i ) are linear combination of any label data and unlabel data ; ii ) do not belong to neither label data nor unlabel data . For adversarial training , samples from the target distribution are assigned as out-of-distribution examples with bounded distortion . Experiments validate that DSGAN can learn these two kinds of distributions well in various datasets . The paper is structured as follows . In Sec . 2 , we introduce DSGAN , including the algorithm in the training procedure . Theoretical results are described in Sec . 3 . In Sec . 4 , two applications are taken as examples to show the effectiveness of DSGAN . Finally , we present the experimental results in Sec . 5 and conclude by pointing out some promising directions for future work in Sec . 7 . 2 PROPOSED METHOD-DSGAN . 2.1 THE SCHEME OF DSGAN . We denote the generator distribution as pg and training data distribution as pd , both in a N - dimensional space . Let pd̄ be the distribution decided by user . For example , pd̄ can be the convolution of pd and normal distribution . Let pt be the target distribution which the user is interested in , and it can be expressed as ( 1− α ) pt ( x ) + αpd ( x ) = pd̄ ( x ) , ( 1 ) where α ∈ [ 0 , 1 ] . Our method , DSGAN , aims to learn pg such that pg = pt . Intuitively , our method tries to learn pg such that pg ( x ) ∼ pd̄ ( x ) − αpd ( x ) 1− α . In other words , the generator is going to output samples located in high-density areas of pd̄ and low-density areas of pd . At first , we formulate the generator and discriminator in GANs . The inputs z of the generator are drawn from pz ( z ) in an M -dimensional space . The generator function G ( z ; θg ) : RM → RN represents a mapping to data space , where G is a differentiable function with parameters θg . The discriminator is defined as D ( x ; θd ) : RN → [ 0 , 1 ] that outputs a single scalar . D ( x ) can be considered as the probability that x belongs to a class of real data . Similar to traditional GAN , we train D to distinguish the real data and the fake data sampled from G. Meanwhile , G is trained to produce realistic data as possible to mislead D. But , in DSGAN , the definitions of “ real data “ and “ fake data “ are different from those in traditional GAN . The samples from pd̄ are considered as real but those from the mixture distribution between pd and pg are considered as fake . The objective function is defined as follows : min G max D V ( G , D ) = Ex∼pd̄ ( x ) [ logD ( x ) ] + ( 1− α ) Ez∼pz ( z ) [ log ( 1−D ( G ( z ) ) ) ] + αEx∼pd ( x ) [ log ( 1−D ( x ) ) ] . ( 2 ) During the training procedure , an iterative approach like traditional GAN is to alternate between k steps of training D and one step of training G. In practice , minibatch stochastic gradient descent via back propagation is used to update θd and θg . In other words , for each of pg , pd and pd̄ , m sample are required for computing gradients , wherem is the number of samples in a minibatch . Algorithm 1 illustrates the training procedure in detail . DSGAN suffers from the same drawbacks with traditional GAN ( e.g. , mode collapse , overfitting , and strong discriminator such that the generator gradient vanishes ) . There are literatures ( Salimans et al . ( 2016 ) , Arjovsky & Bottou ( 2017 ) , and Miyato et al . ( 2018 ) ) focusing on improving the above problems , and their ideas can be combined into DSGAN . Li et al . ( 2017 ) and Reed et al . ( 2016 ) proposed the similar objective function like ( 2 ) . Their goal is to learn the conditional distribution of training data . Nevertheless , we aim to learn the target distribution pt in Eq . ( 1 ) , not the training data distribution . Algorithm 1 The training procedure of DSGAN using minibatch stochastic gradient descent . k is the number of steps to apply to the discriminator . α is the ratio between pg and pd in the mixture distribution . We used k = 1 and α = 0.8 in experiments . 01. for number of training iterations do 02. for k steps do 03 . Sample minibatch of m noise samples z ( 1 ) , ... , z ( m ) from pg ( z ) . 04 . Sample minibatch of m samples x ( 1 ) d , ... , x ( m ) d from pd ( x ) . 05 . Sample minibatch of m samples x ( 1 ) d̄ , ... , x ( m ) d̄ from pd̄ ( x ) . 06 . Update the discriminator by ascending its stochastic gradient : ∇θd [ 1 m m∑ i=1 logD ( x ( i ) d ) + log ( 1−D ( G ( z ( i ) ) ) ) + log ( 1−D ( x ( i ) d̄ ) ) ] 07. end for 08 . Sample minibatch of m noise samples z ( 1 ) , ... , z ( m ) from pg ( z ) . 09 . Update the generator by descending its stochastic gradient : ∇θg 1 m m∑ i=1 [ log ( 1−D ( G ( z ( i ) ) ) ) ] 10. end for 2.2 CASE STUDY ON SYNTHETIC DATA AND MNIST . To get more intuitive understanding about DSGAN , we conduct several case studies on 2D synthetic datasets and MNIST . Those results validate that DSGAN can learn the distribution we desire . Low-density samples generation Fig . 2 illustrates that DSGAN is able to generate boundary samples on the 2D swissroll . Given the density function of the swissroll as pd , we assign pd̄ as the convolution of pd and the normal distribution . Then , by applying DSGAN , we achieve our goal to generate boundary samples . The intuition of our idea is also illustrated by a 1D example in Fig . 4 . In general , our idea will lead DSGAN to generate low-density samples like another example in Fig . 3 . In this case , the density is low in the center of the circle , and our generator can not only create the boundary samples but also the samples located in low-density area . Difference-set generation We also validate DSGAN on high dimensional dataset such as MNIST . In this example , we define pd be the distribution of digit “ 1 ” and pd̄ be the distribution contains both digits “ 1 ” and “ 7 ” . Since the density pd ( x ) is high when x is digit “ 1 ” , the generator is prone to output digit “ 7 ” with high probability . From the above results , we can observe two properties of generator distribution pg : i ) the higher density of pd ( x ) , the lower density of pg ( x ) ; ii ) pg prefers to output samples from high-density areas of pd̄ ( x ) -pd ( x ) . In those case studies , α = 0.8 in Eq . ( 1 ) is used . In the next section , we will show that the objective function is equivalent to minimizing the JensenShannon divergence between the mixture distribution ( pd and pg ) and pd̄ as G and D are given enough capacity . Furthermore , we provide a trick ( see Appendix C in details ) by reformulating the objective function ( 2 ) such that it is more stable to train DSGAN .

Summary: The paper presents DS-GAN, which aims to learn the difference between any two distributions whose samples are difficult or impossible to collect. To this end they simply model the target distribution such that adding it to one of the distribution results in another, and propose a min-max objective based on it. To show the effectiveness of the proposed DS-GAN, the authors validate it on semi-supervised learning and adversarial training tasks, on which it performs reasonably well in generating the difference between the two distributions.