DECAF:  Generating Fair Synthetic Data Using Causally-Aware Generative Networks

1 Introduction . Generative models are optimized to approximate the original data distribution as closely as possible . Most research focuses on three objectives [ 1 ] : fidelity , diversity , and privacy . The first and second are concerned with how closely synthetic samples resemble real data and how much of the real data ’ s distribution is covered by the new distribution , respectively . The third objective aims to avoid simply reproducing samples from the original data , which is important if the data contains privacy-sensitive information [ 2 , 3 ] . We explore a much-less studied concept : synthetic data fairness . Motivation . Deployed machine learning models have been shown to reflect the bias of the data on which they are trained [ 4 , 5 , 6 , 7 , 8 ] . This has not only unfairly damaged the discriminated individuals but also society ’ s trust in machine learning as a whole . A large body of work has explored ways of detecting bias and creating fair predictors [ 9 , 10 , 11 , 12 , 13 , 14 , 15 ] , while other authors propose debiasing the data itself [ 9 , 10 , 11 , 16 ] . This work ’ s aim is related to the work of [ 17 ] : to generate fair synthetic data based on unfair data . Being able to generate fair data is important because end-users creating models based on publicly available data might be unaware they are inadvertently including ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . bias or insufficiently knowledgeable to remove it from their model . Furthermore , by debiasing the data prior to public release , one can guarantee any downstream model satisfies desired fairness requirements by assigning the responsibility of debiasing to the data generating entities . Goal . From a biased dataset X , we are interested in learning a model G , that is able to generate an equivalent synthetic unbiased dataset X ′ with minimal loss of data utility . Furthermore , a downstream model trained on the synthetic data needs to make not only unbiased predictions on the synthetic data , but also on real-life datasets ( as formalized in Section 4.2 ) . Solution . We approach fairness from a causal standpoint because it provides an intuitive perspective on different definitions of fairness and discrimination [ 11 , 13 , 14 , 15 , 18 ] . We introduce DEbiasing CAusal Fairness ( DECAF ) , a generative adversarial network ( GAN ) that leverages causal structure for synthesizing data . Specifically , DECAF is comprised of d generators ( one for each variable ) that learn the causal conditionals observed in the data . At inference-time , variables are synthesized topologically starting from the root nodes in the causal graph then synthesized sequentially , terminating at the leave nodes . Because of this , DECAF can remove bias at inference-time through targeted ( biased ) edge removal . As a result , various datasets can be created for desired ( or evolving ) definitions of fairness . Contributions . We propose a framework of using causal knowledge for fair synthetic data generation . We make three main contributions : i ) DECAF , a causal GAN-based model for generating synthetic data , ii ) a flexible causal approach for modifying this model such that it can generate fair data , and iii ) guarantees that downstream models trained on the synthetic data will also give fair predictions in other settings . Experimentally , we show how DECAF is compatible with several fairness/discrimination definitions used in literature while still maintaining high downstream utility of generated data . 2 Related Works . Here we focus on the related work concerned with data generation , in contrast to fairness definitions for which we provide a detailed overview in Section 4 and Appendix C. As an overview of how data generation methods relate to one another , we refer to Table 1 which presents all relevant related methods . Non-parametric generative modeling . The standard models for synthetic data generation are either based on VAEs [ 19 ] or GANs [ 2 , 3 , 20 , 21 ] . While these models are well known for their highly realistic synthetic data , they are unable to alter the synthetic data distribution to encourage fairness ( except for [ 17 , 23 ] , discussed below ) . Furthermore , these methods have no causal notion , which prohibits targeted interventions for synthesizing fair data ( Section 4 ) . We explicitly leave out CausalGAN [ 24 ] and CausalVAE [ 25 ] , which appear similar by incorporating causality-derived ideas but are different in both method and aim ( i.e. , image generation ) . Fair data generation . In the bottom section of Table 1 , we present methods that , in some way , alter the training data of classifiers to adhere to a notion of fairness [ 10 , 11 , 16 , 17 , 22 , 23 ] . While these methods have proven successful , they lack some important features . For example , none of the related methods allow for post-hoc changes of the synthetic data distribution . This is an important feature , as each situation requires a different perspective on fairness and thus requires a flexible framework for selecting protected variables . Additionally , only [ 11 , 23 ] allow a causal perspective on fairness , despite causal notions underlying multiple interpretations of what should be considered fair [ 13 ] . Furthermore , only [ 17 , 22 , 23 ] offer a flexible framework , while the others are limited to binary [ 10 , 11 ] or discrete [ 16 ] settings . Xu et al . [ 23 ] also use a causal architecture for the generator , however their method is not as flexible—e.g . it does not easily extend to multiple protected attributes . Finally , in contrast to other methods DECAF is directly concerned with fairness of the downstream model—which is dependent on the setting in which the downstream model is employed ( Section 4.2 ) . In essence , from Table 1 we learn that DECAF is the only method that combines all key areas of interest . At last , we would like to mention [ 26 ] , who aim to generate data that resembles a small unbiased reference dataset , by leveraging a large but biased dataset . This is very different to our aim , as we are interested in the downstream model ’ s fairness and explicit notions of fairness . 3 Preliminaries . Let X ∈ X ⊆ Rd denote a random variable with distribution PX ( X ) , with protected attributes A ∈ A ⊂ X and target variable Y ∈ Y ⊂ X , let Ŷ denote a prediction of Y . Let the data be given by D = { x ( k ) } Nk=1 , where each x ( k ) ∈ D is a realization of X . We assume the data generating process can be represented by a directed acyclic graph ( DAG ) —such that the generation of features can be written as a structural equation model ( SEM ) [ 27 ] —and that this DAG is causally sufficient . Let Xi denote the ith feature in X with causal parents Pa ( Xi ) ⊂ { Xj : j 6= i } , the SEM is given by : Xi = fi ( Pa ( Xi ) , Zi ) , ∀i ( 1 ) where { Zi } di=1 are independent random noise variables , that is Pa ( Zi ) = ∅ , ∀i . Note that each fi is a deterministic function that places all randomness of the conditional P ( Xi|Pa ( Xi ) ) in the respective noise variable , Zi . 4 Fairness of Synthetic Data . Algorithmic fairness is a popular topic ( e.g. , see [ 13 , 28 ] ) , but fair synthetic data has been much less explored . This section highlights how the underlying graphs of the synthetic and downstream data determine whether a model trained on the synthetic data will be fair in practice . We start with the two most popular definitions of fairness , relating to the legal concepts of direct and indirect discrimination . We also explore conditional fairness [ 29 ] , which is a generalization of the two . In Appendix C we discuss how the ideas in this section transfer to other independence-based definitions [ 30 ] . Throughout this section , we separate Y from X by defining X̄ = X\Y , and we will write X ← X̄ for ease of notation . 4.1 Algorithmic fairness . The first definition is called Fairness Through Unawareness ( e.g . [ 31 ] ) . Definition 1 . ( Fairness Through Unawareness ( FTU ) : algorithm ) . A predictor f : X 7→ Ŷ is fair iff protected attributes A are not explicitly used by f to predict Ŷ . This definition prohibits disparate treatment [ 28 , 32 ] , and is related to the legal concept of direct discrimination , i.e. , two equally qualified people deserve the same job opportunity independent of their race , gender , beliefs , among others . Though FTU fairness is commonly used , it might result in indirect discrimination : covariates that influence the prediction Ŷ might not be identically distributed across different groups a , a′ , which means an algorithm might have disparate impact on a protected group [ 10 ] . The second definition of fairness , demographic parity [ 32 ] , does not allow this : Definition 2 . ( Demographic Parity ( DP ) : algorithm ) A predictor Ŷ is fair iff A ⊥⊥ Ŷ , i.e . ∀a , a′ : P ( Ŷ |A = a ) = P ( Ŷ |A = a′ ) . Evidently , DP puts stringent constraints on the algorithm , whereas FTU might be too lenient . The third definition we include is based on the work of [ 29 ] , related to unresolved discrimination [ 14 ] . The idea is that we do not allow indirect discrimination unless it runs through explanatory factors R ⊂ X . For example , in Simpson ’ s paradox [ 33 ] there seems to be a bias between gender and college admissions , but this is only due to women applying to more competitive courses . In this case , one would want to regard fairness conditioned on the choice of study [ 14 ] . Let us define this as conditional fairness : Definition 3 . ( Conditional Fairness ( CF ) : algorithm ) A predictor Ŷ is fair iff A ⊥⊥ Ŷ |R , i.e . ∀r , a , a′ : P ( Ŷ |R = r , A = a ) = P ( Ŷ |R = r , A = a′ ) . CF generalizes FTU and DP Note that conditional fairness is a generalization of FTU and DP , by setting R = X\A and R = ∅ , respectively . In Appendix C we elaborate on the connection between these , and more , definitions . 4.2 Synthetic data fairness . Algorithmic definitions can be extended to distributional fairness for synthetic data . Let P ( X ) , P ′ ( X ) be probability distributions with protected attributes A ⊂ X and labels Y ⊂ X . Let I ( A , Y ) be a definition of algorithmic fairness ( e.g. , FTU ) . Note , that under CF , I ( A , Y ) is a function of R as well . We propose ( I ( A , Y ) , P ) -fairness of distribution P ′ ( X ) : Definition 4 . ( Distributional fairness ) A probability distribution P ′ ( X ) is ( I ( A , Y ) , P ) -fair , iff the optimal predictor Ŷ = f∗ ( X ) of Y trained on P ′ ( X ) satisfies I ( A , Y ) when evaluated on P ( X ) . In other words , when we train a predictor on ( I ( A , Y ) , P ) -fair distribution P ′ ( X ) , we can only reach maximum performance if our model is fair . Note the explicit reference to P ( X ) , the distribution on which fairness is evaluated , which does not need to coincide with P ′ ( X ) . This is a small but relevant detail . For example , when training a model on data D′ ∼ P ′ ( X ) it could seem like the model is fair when we evaluate it on a hold-out set of the data ( e.g. , if we simply remove the protected attribute from the data ) . However , when we use the model for real-world predictions of data D ∼ P ( X ) , disparate impact is possibly observed due to a distributional shift . By extension , we define synthetic data as ( I ( A , Y ) , P ) -fair , iff it is sampled from an ( I ( A , Y ) , P ) - fair distribution . Defining synthetic data as fair w.r.t . an optimal predictor is especially useful when we want to publish a dataset and do not trust end-users to consider anything but performance.2 Choosing P ( X ) . The setting P ( X ) = P ′ ( X ) corresponds to data being fair with respect to itself . For synthetic data generation , this setting is uninteresting as any dataset can be made fair by randomly sampling or removing A ; if A is random , the prediction should not directly or indirectly depend on it . This ignores , however , that a downstream user might use the trained model on a real-world dataset in which other variables B are correlated with A , and thus their model ( which is trained to use B for predicting Y ) will be biased . Of specific interest is the setting where P ( X ) corresponds to the original data distribution PX ( X ) that contains unfairness . In this scenario , we construct P ′ ( X ) by learning PX ( X ) and removing the unfair characteristics . The data from P ′ ( X ) can be published online , and models trained on this data can be deployed fairly in real-life scenarios where data follows PX ( X ) . Unless otherwise stated , henceforth , we assume P ( X ) = PX ( X ) .

Summary: This work proposes a novel task of generating fair synthetic data by a GAN-based model. The generation process follows the order on a causal DAG and fairness is achieved by debiasing during the inference phase. Experiment results are carried out on Adult and Credit Approval datasets to show the effectiveness of generating fair synthetic data.