Differential-Critic GAN: Generating What You Want by a Cue of Preferences

1 INTRODUCTION . Learning a good generative model for high-dimensional natural signals , such as images ( Zhu et al. , 2017 ) , video ( Vondrick et al. , 2016 ) and audio ( Fedus et al. , 2018 ) has long been one of the key milestones of machine learning . Powered by the learning capabilities of deep neural networks , generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) have brought the field closer to attaining this goal . Currently , GANs are applied in a setting where the whole training dataset is of user interest . Therefore , regular GANs no longer meet our requirement when only partial instead of the entire training dataset possesses the desired properties ( Killoran et al. , 2017 ) . It is more challenging when the given dataset has a small number of desired data . Adapting vanilla GAN to this setting , a naive way is to first select the samples possessing the desired properties and then perform regular GAN training only on the selected samples to derive the desired distribution . However , vanilla GAN fails when the desired samples are limited . FBGAN overcomes the limited data problem by iteratively introducing desired samples from the generation into the training data . Specifically , FBGAN is pretrained with all training data using the vanilla GAN . In each training epoch , the generator first generates certain amounts of samples . The generated samples possessing the desired properties are selected by an expert selector and used to replace the old training data . Then , regular WGAN is trained with the updated training data . Since the ratio of the desired samples gradually increases in the training data , all training data will be replaced with the desired samples . Finally , FBGAN would derive the desired distribution when convergence . However , bluntly eliminating undesired samples may lead to a biased representation of the real desired data distribution . Because the undesired samples can also reveal useful clues about what is not desired . Suppose we want to generate old face images , however the training data contains only a few old face images whereas it has many young face images . In this case , the young face images can be used as negative sampling ( Mikolov et al. , 2013 ) to learn the subtle aging features ( e.g . wrinkles , pigmented skin , etc . ) , which guides the generation of the desired old face images . The conditional variants of GAN , such as CGAN ( Mirza and Osindero , 2014 ) and ACGAN ( Odena et al. , 2017 ) can be also applied in this setting by introducing condition variables to model the conditional desired data distribution . However , the generation performance of condition-based GAN is governed by the respective conditions with sufficient training observations . When the desired data is limited , the conditional modeling is dominated by the major classes , i.e. , undesired data , resulting in a failure to capture the desired distribution . All the literature methods require user-defined criteria to select the desired data in order to learn the distribution of the desired data , which may not exist in real applications . Instead of soliciting a ready-to-use criteria , we consider a more general setting where GAN can be guided towards the distribution of user-desired data by the user preference . In particular , pairwise preferences are the most popular form of user preference due to their simplicity and easy accessibility ( Lu and Boutilier , 2011 ) . Therefore , our target is to incorporate pairwise preferences into the learning process of GAN , so as to guide the generation of the desired data . Relativistic GAN ( RGAN ) ( Jolicoeur-Martineau , 2019 ) is a variant of regular GAN and is proposed to learn the whole data distribution . It considers the critic value as the indicator of sample quality and defines the discriminator using the difference in the critic values . The critic value in RGAN is similar to the ranking score , but it is used to describe sample quality . Motivated by this , we consider taking the critic value as the ranking score and define the ranking loss for pairwise preferences based on the critic value directly . In particular , the difference in critic values for each pair of samples reflects the user ’ s preference over the samples . This is why we call our critic the differential critic , and we propose Differential-Critic GAN ( DiCGAN ) for learning the user-desired data distribution . As shown in Fig . 1 , the differential critic incorporates the user preference direction , which pushes the original critic direction towards the real desired data region instead of the entire real data region . The main contributions are summarized as follows : • We propose DiCGAN to learn the distributions of the desired data from the entire data using pairwise preferences . To the best of our knowledge , this is the first work to promote the ratio of the desired data by incorporating user preferences directly into the data generation . • We introduce the differential critic by defining an additional pairwise ranking loss on the WGAN ’ s critic . It endows the difference in the critic values between each pair of samples with user preferences . • The empirical study shows that DiCGAN learns the distribution of user-desired data and the differential critic can derive the preference direction even from a limited umber of preferences . 2 GENERATIVE ADVERSARIAL NETWORKS . Generative Adversarial Network ( GAN ) ( Goodfellow et al. , 2014 ) performs generative modeling by learning a map from low-dimensional input space Z to data space X , i.e. , Gθ : Z → X , given samples from the training data distribution , namely , x ∼ pr ( x ) . The goal is to find θ which achieves pθ ( x ) = pr ( x ) , where pθ ( x ) is the fake data distribution x = Gθ ( z ) . Let p ( z ) be the input noise distribution and G indicate Gθ . GAN defines a discriminator D that is trained to discriminate real data from fake data to guide the learning of G. Wasserstein GAN ( WGAN ) ( Arjovsky et al. , 2017 ) proposes to use the Wasserstein metric as a critic , which measures the quality of fake data in terms of the distance between the real data distribution and the fake data distribution . The Wasserstein distance ( W-distance ) is approximated by the difference in the average critic values between the real data and the fake data . The empirical experiments show that the W-distance between two distributions corresponds well to the quality of the generated data . WGAN ’ s objective function is defined as follows : min G max D Epr ( x ) [ D ( x ) ] − Epθ ( x ) [ D ( x ) ] , ( 1 ) where D is the critic and satisfies 1-Lipschitz . 3 DICGAN FOR USER-DESIRED DISTRIBUTION . No longer learning the distribution of the whole dataset , GAN is applied in a new scenario , where the distribution of the partial dataset is what we desire . User-desired data may refer to some certain class of data among multiple class datasets , or observations with/without some particular attributes . Such data can be induced from the user preference , which can be represented as an ordering relation between two or more samples in terms of the desired properties . We propose differential-critic GAN to learn the desired data distribution from the user preferences along with the whole dataset . 3.1 LEARNING THE DISTRIBUTION OF USER-DESIRED DATA . Following the score-based ranking literature , we suppose that there exists a numeric score associated with each sample , reflecting the user ’ s preference for the sample . A higher score indicates that its corresponding sample is preferred by the user . In detail , let f denote a score function that maps sample x to score f ( x ) . Then , if sample x is desired by the user , its score f ( x ) exceeds a predefined threshold , namely , I ( f ( x ) > ) = 1 . I is a sign function , which equals 1 if its condition is true and 0 otherwise . For the sake of explanation , we use pr ( x ) , pd ( x ) , pu ( x ) to denote the distribution of the whole data , the user-desired data and the undesired data , respectively . FBGAN ( Gupta and Zou , 2019 ) was proposed to learn the distribution of the desired data pd ( x ) . FBGAN executes alternatively between two steps : ( 1 ) construct the desired dataset Xd = { x|I ( f ( x ) > ) = 1 , x ∼ pr ( x ) } ; ( 2 ) train GAN on Xd to derive pd ( x ) . However , the assumption that the score function f is predefined in FBGAN may be too restrictive for real applications , where no universal and explicit criteria exists . Further , the definitions of the desired/undesired samples are highly dependent on the choice of the threshold . The removal of the so-called undesired samples may result in a biased representation of real desired data distribution . Instead of relying on a predefined score function , we propose to learn the desired data distribution in a straightforward manner from the user preference . Here , we consider a general auxiliary information , i.e. , the pairwise preferences , to represent the user preference , due to its simplicity and easy accessibility . For any two samples x1 , x2 ∼ pr ( x ) , let x1 x2 denote that x1 is preferred over x2 according to the user-defined criteria . Let X be the training samples , i.e. , X = { xi ∼ pr ( x ) } . A collection of pairwise preferences S is obtained by : S = { s = ( x1 , x2 ) |x1 x2 , x1 , x2 ∈ X } . ( 2 ) Definition 1 ( Problem Setting ) . Given the training samples X and the pairwise preferences S , the target is to learn a generative model pθ ( x ) that is identical to the distribution of the desired data pd ( x ) , i.e. , pθ ( x ) = pd ( x ) . 3.2 DIFFERENTIAL CRITIC GAN . Instead of WGAN ’ scritic for quality assessment , we present the differential critic for modelling pairwise preferences . The differential critic can guide the generation of the user-desired data . 3.2.1 PAIRWISE PREFERENCE . In this section , we consider incorporating the pairwise preference into the training of GAN . The score-based ranking model ( Zhou et al. , 2008 ) is used to model the pairwise preferences . It learns the score function f , of which the score value , called ranking score in the model , is the indicator of the user preference . Further , the difference of ranking scores can indicate the pairwise preference relation . That is , for any pair of samples x1 , x2 , if x1 x2 then f ( x1 ) − f ( x2 ) > 0 and vice versa . For any pairwise preference s : x1 x2 , the ranking loss we consider is as follows : h ( s ) = max ( 0 , − ( f ( x1 ) − f ( x2 ) ) +m ) , ( 3 ) where m is the ranking margin . For other forms of ranking losses , the reader can refer to ( Zhou et al. , 2008 ) . Instead of learning the score function independent of GAN ’ s training , we consider incorporating it into GAN ’ s training , guiding GAN towards the generation of the desired data . The critic in RGAN ( Jolicoeur-Martineau , 2019 ) is similar to the score function , where the critic values are used to describe the sample quality . We are motivated to take the critic value as the ranking score and define the ranking loss on the critic value directly . In particular , the difference in the critic values for each pair of samples reflects the user ’ s preference over the samples . Remark 1 ( Pairwise regularization to the generator ) . It is possible to consider a pairwise regularization to the generator . As the target is to learn the desired distribution , the regularization to the generator can be used to make the critic values of the generated samples larger than those of the undesired samples . We construct the regularization with the principle similar as FBGAN . Specifically , a selector is first applied to give a full ranking for the training data and then bottom K samples are picked up as the undesired samples . The pairwise preferences are then defined over the generated samples and the undesired samples .

Summary: The motivation of this study is to estimate the distribution of desired data from the entire data distribution. And the proposed solution extends existing GAN solutions by introducing an additional pairwise loss on the discriminator, e.g., its scores on the desired instances should be higher than the undesired ones. The idea is natural and neat, and it is also proved to be effective in the reported experiments.