A Fair Generative Model Using Total Variation Distance

1 INTRODUCTION . High-quality realistic samples synthesized thanks to recent advances in generative models ( Brock et al. , 2019 ; Goodfellow et al. , 2014 ; Karras et al. , 2019 ) have played a crucial role to enrich training data for a widening array of applications such as face recognition , natural language processing , and medical imaging ( Wang et al. , 2019 ; Chang et al. , 2018 ; Yi et al. , 2019 ) . One challenge concerning fairness arises when generative models are built upon biased training data that preserve unbalanced representations of demographic groups . Any existing bias in the dataset can readily be propagated to the learned model , thus producing generations that are biased towards certain demographics . The unbalanced generated samples may often yield undesirable performances against underrepresented groups for downstream applications . One natural way to ensure fair sample generation is to exploit demographic labels ( if available ) to build a fair generative model , e.g. , via conditional GAN ( Mirza & Osindero , 2014 ; Odena et al. , 2017 ; Miyato & Koyama , 2018 ) which employs such labels to easily generate an arbitrary number of samples for minority groups . In many practically-relevant scenarios , however , such labels are not often available . To address the challenge , one pioneering work ( Choi et al. , 2020 ) develops a novel debiasing technique that employs the reweighting idea ( Ren et al. , 2018 ; Kamiran & Calders , 2012 ; Byrd & Lipton , 2019 ) to put more weights to underrepresented samples , thereby promoting fair sample generation across demographic groups . One key feature of the technique is to identify the bias ( reflected in the weights ) via a small and unlabelled reference dataset . While it enjoys significant fairness performance for moderate sizes of the reference dataset , it may provide a marginal gain for a more practically-relevant case of a small set size where the weight estimation is often inaccurate , as hinted by the meta-learning literature ( Ren et al. , 2018 ; Shu et al. , 2020 ) . We also find such phenomenon in our experiments ; see Table 2 for details . On the other hand , one recent study ( Roh et al. , 2020 ) sheds lights on addressing the small set size issue . Roh et al . ( 2020 ) propose a robust training approach by employing the Jensen-Shannon divergence ( or equivalently mutual information ( Majtey et al. , 2005 ) ) between poisoned training and clean reference samples . It then takes the divergence as a regularization term in order to promote robust training . One key benefit of Roh et al . ( 2020 ) in light of the reweighting-based approaches ( Ren et al. , 2018 ; Choi et al. , 2020 ) is that the robustness performance is guaranteed even for a small size of the clean reference set down to 5 % relative to the training set size . This implies that the divergence-based regularization approach makes a more efficient use of reference data for robustness , as compared to the reweighting technique . Contribution : Inspired by the key benefit featured in Roh et al . ( 2020 ) , we address the small set issue by another well-known statistical measure , in particular , total variation distance ( TVD ) . See Section 3.2 for details on the rationale behind the use of TVD . Similarly we introduce a reference dataset which is balanced and unlabelled . We then employ it to formulate the TVD between the generated and reference sample distributions , which can serve as quantifying the degree of unfairness . We then promote fair sample generation by adding the TVD into a conventional optimization ( e.g. , GAN-based optimization ( Goodfellow et al. , 2014 ; Nowozin et al. , 2016 ; Arjovsky et al. , 2017 ) ) . Motivated by the variational optimization technique w.r.t . TVD ( Villani , 2009 ; Nowozin et al. , 2016 ; Arjovsky et al. , 2017 ) , we translate the TVD-regularization term into a function optimization . We also conduct extensive experiments on three benchmark real datasets : CelebA ( Liu et al. , 2015 ) , UTKFace ( Zhang et al. , 2017 ) , and FairFace ( Karkkainen & Joo , 2021 ) . We demonstrate via simulation that the proposed framework can significantly boost up the fairness performance while offering high-quality realistic samples reflected in low FID . We also find that our approach outperforms the state of the art ( Choi et al. , 2020 ) , particularly being robust to the balanced reference set size : the significant improvements preserve for a wide range of the reference set size down to 1 % relative to training data ( more preferable in reality ) . Related works : In addition to Choi et al . ( 2020 ) , Tan et al . ( 2020 ) propose a different way that promotes fair sample generation by smartly perturbing the input distribution of a pre-trained generative model with the help of a classifier for sensitive attributes . The key distinction w.r.t . ours is that it relies upon the additional classifier which requires the use of demographic labels to obtain . Yu et al . ( 2020 ) employ demographic labels for minority groups to generate a wide variety of samples with improved data converge by harmonizing GAN and MLE ideas . A distinction w.r.t . ours is that it requires the knowledge on demographic labels . Jalal et al . ( 2021 ) consider a fair generative model yet in a different context , image reconstruction . The goal of the task is to ensure fair sample generation of restored images from degraded versions . Since it relies upon the degraded images , it is not directly comparable to ours . Another line of fair generative modeling focuses on label bias , instead of representation bias ( Xu et al. , 2018 ; 2019a ; b ; Sattigeri et al. , 2019 ; Jang et al. , 2021 ; Kyono et al. , 2021 ) . The goal therein is to develop a generative model such that the generated decision labels are statistically independent of the given demographic labels . Again , these are not directly comparable to ours , as they require the use of demographic labels . The variational optimization technique w.r.t . TVD that gives an inspiration to our work has originated from Villani ( 2003 ; 2009 ) , wherein the author shows that the TVD can be expressed as a function optimization in the context of transport theory . The technique was recently applied to the GAN context ( Nowozin et al. , 2016 ) . The TVD has also served as a useful tool for quantifying various fairness measures in fair classifiers that pursue individual fairness ( Dwork et al. , 2012 ; Dwork & Ilvento , 2018 ) and group fairness ( Gordaliza et al. , 2019 ; Wang et al. , 2020a ; Farokhi , 2021 ) . 2 PROBLEM FORMULATION . Setup : Fig . 1 illustrates the problem setting for a fair generative model that we focus on herein . We consider a challenging yet practically-relevant scenario wherein demographic information ( or that we call sensitive attribute ) , say z ∈ Z , is not available . Under this blind setting , the goal of the problem is to construct a fair generative model that ensures the produced samples to have the same size across distinct demographics . We assume that there are two types of data given in the problem : ( i ) training data Dbias : = { x ( i ) bias } mbias i=1 ; ( ii ) reference data Dref : = { x ( i ) ref } mref i=1 . Since we consider the setting where training data is potentially biased , we use the word “ bias ” in the associated notations . Here mbias denotes the number of training examples . Let Pbias be data distribution which each training data x ( i ) bias ∈ X is generated upon . In a biased scenario having female-vs-male sensitive attribute , e.g. , z = 0 ( female ) and z = 1 ( male ) , we may have Pbias ( Z = 0 ) > Pbias ( Z = 1 ) . As in Choi et al . ( 2020 ) , we also employ a balanced yet small reference dataset for the purpose of promoting fair sample generation , which can be obtained via a set of carefully-designed data collection protocols employed in organizations like World Bank and biotech companies ( Choi et al. , 2020 ; 23 & me , 2016 ; Hong , 2016 ) . Let Pref be the corresponding data distribution defined on X : i=1 ( potentially biased ) and balanced reference data { x ( i ) ref } mref i=1 . The entire structure of the proposed model will be illustrated in detail in Fig . 2 . Here mbias ( or mref ) denotes the number of training samples ( or reference samples ) . Pref ( Z = 0 ) ≈ Pref ( Z = 1 ) . Since the balanced reference set is often challenging to collect , typically the number of the reference samples is much smaller than that of training examples : mref mbias . Denote by x̂ : = G ( w ) ∈ X the generated sample fed by a random noise input w ∈ W . We assume that the generated samples have the same support X as training and reference samples . Let PG and PW be distributions w.r.t . generated samples and the random noise input respectively . As a fairness measure that will be employed for the purpose of evaluating our framework to be presented in Section 3 , we consider fairness discrepancy proposed by Choi et al . ( 2020 ) . It quantifies how PG differs from Pref w.r.t . a certain sensitive attribute , formally defined below . Definition 1 ( Fairness Discrepancy ( Choi et al. , 2020 ) ) . Fairness discrepancy between Pref and PG w.r.t . a sensitive attribute z ∈ { z1 , . . . , z|Z| } is defined as : F ( Pref , PG ) : = ‖pref ( z ) − pG ( z ) ‖2 ( 1 ) where pref ( z ) : = Pref ( Z = z1 ) Pref ( Z = z2 ) ... Pref ( Z = z|Z| ) and pG ( z ) : = PG ( Ẑ = z1 ) PG ( Ẑ = z2 ) ... PG ( Ẑ = z|Z| ) . Here Ẑ denotes the prediction of the sensitive attribute w.r.t . a generated sample , yielded by a pre-trained classification model which we call attribute classifier as in Choi et al . ( 2020 ) . The attribute classifier is employed only for the purpose of evaluation , and is trained based on another real dataset , e.g. , like the one mentioned in Choi et al . ( 2020 ) : the standard train and validation splits of CelebA ( Liu et al. , 2015 ) . For faithful evaluation , we consider a vast number of generated samples ( i.e. , more than 10,000 ) as well as employ highly-accurate attribute classifiers , around 98 % accuracy of gender classifier for instance . As a measure for the quality of generated samples that may compete with the fairness measure , we employ a well-known measure : Fréchet Inception Distance ( FID ) ( Heusel et al. , 2017 ) . It is defined as the Fréchet distance ( Fréchet , 1957 ) ( also known as the second-order Wasserstein distance ( Wasserstein , 1969 ) ) between real and generated samples approximated via the Gaussian distribution . The lower FID , the more realistic and diverse the generated samples are . For a more precise measure that reveals sample quality of each sensitive group , we consider FID computed within each demographic , called intra FID ( Miyato & Koyama , 2018 ; Zhang et al. , 2019 ; Wang et al. , 2020b ) . Computing intra FID requires the knowledge on group identities of generated samples . Since demographic labels are not available in our context , we rely upon the attribute classifier ( that we introduced above ) for predicting demographic information of the generated samples . GAN-based generative model : Our framework ( to be presented soon ) builds upon one powerful generative model : Generative Adversarial Network ( GAN ) ( Goodfellow et al. , 2014 ) . The GAN comprises two competing players : ( i ) discriminator D ( · ) that wishes to discriminate real samples against generated samples ; and ( ii ) generator G ( · ) that intends to fool the discriminator by producing realistic generated samples . In particular , we consider a general optimization framework ( Tseng et al. , 2021 ) which subsumes many GAN variants as special cases : ( Discriminator ) max D EPbias [ fD ( D ( X ) ) ] + EPG [ fG ( D ( X ) ) ] ; ( Generator ) min G −EPG [ gG ( D ( X ) ) ] ( 2 ) where fD , fG , and gG indicate certain functions that vary depending on an employed GAN approach . For instance , the choice of ( fD ( t ) , fG ( t ) , gG ( t ) ) = ( t , −t , t ) together with Lipschitz-1 condition on D leads to the prominent WGAN optimization ( Arjovsky et al. , 2017 ) . Another choice of ( fD ( t ) , fG ( t ) , gG ( t ) ) = ( min { 0 , −1 + t } , min { 0 , −1− t } , t ) yields a hinge-loss-based GAN ( Lim & Ye , 2017 ; Tran et al. , 2017 ) . We adopt this as a base framework that we will add a fairness aspect into in the next section .

Summary: This paper focuses on the fair generative model problem. In particular, "fairness" lies in the balance of representation among groups in the data. The paper proposes to use TV as the distance metric in order to capture the difference between the generated distribution and the reference distribution. Empirical results are also provided.