Differentially Private Empirical Risk Minimization under the Fairness Lens

1 Introduction . While learning systems have become instrumental for many decisions and policy operations involving individuals , the use of rich datasets combined with the adoption of black-box algorithms has sparked concerns about how these systems operate . Two key concerns regard how these systems handle discrimination and how much information they leak about the individuals whose data is used as input . Differential Privacy ( DP ) [ 14 ] has become the paradigm of choice for protecting data privacy and its deployments are growing at a fast rate . DP is appealing as it bounds the risks of disclosing sensitive information of individuals participating in a computation . However , it was recently observed that DP systems may induce biased and unfair outcomes for different groups of individuals [ 3 , 23 , 27 ] . The resulting outcomes can have significant societal and economic impacts on the involved individuals : classification errors may penalize some groups over others in important determinations including criminal assessment , landing , and hiring [ 3 ] or can result in disparities regarding the allocation of critical funds , benefits , and therapeutics [ 23 ] . While these surprising observations have become apparent in several contexts , their causes are largely understudied and not fully understood . This paper makes a step toward addressing this important knowledge gap . It builds on these key observations and sheds light on the causes of the disparate impacts arising in the problem of differentially private empirical risk minimization ( ERM ) . It focuses on the accuracy disparity arising among groups of individuals in two well-studied DP learning methods : output perturbation [ 11 ] and differentially private stochastic gradient descent ( DP-SGD ) [ 2 ] . The paper analyzes which properties of the model and the data are responsible for the disproportionate impacts , why these aspects are affecting different groups disproportionately , and proposes guidelines to mitigate these effects . In summary , the paper makes the following contributions : 1 . It develops a notion of fairness under private training that relies on the concept of excessive risk . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) 2 . It analyzes this fairness notion in two DP learning methods : output perturbation and DP-SGD . 3 . It isolates the relevant components related with noise addition and gradient clipping responsible for the disparate impacts . 4 . It studies the behaviors and the causes for these components to affect different groups of individuals disproportionately during private training . 5 . Based on these observations , it proposes a mitigation solution and evaluates its effectiveness on several standard datasets . To the best of the authors knowledge , this work represents a first step toward a deeper understanding of the causes of the unfairness impacts in differentially private learning . 2 Related work . The research at the interface between differential privacy and fairness is receiving increasing attention and can be broadly categorized into three main lines of work . The first shows that DP is in alignment with fairness . Notable contribution in this direction include Dwork et al . [ 15 ] seminal work , which highlights the relation between individual fairness and differential privacy , and Khalili et al . [ 19 ] , which shows that the private exponential mechanism can produce fair outcomes in some selection problems . Works in the second category study the setting under which a fair model can leak privacy [ 22 , 18 , 9 , 25 , 28 ] . These works propose learning frameworks that guarantee DP while also encouraging the satisfaction of different notions of fairness . For example , Xu et al . [ 27 ] proposes a private and fair variant of DP-SGD that uses separate clipping bounds for each groups of individuals . Such proposal encourages accuracy parity at the expense of an extra privacy cost ( required to customize the clipping bound for each group ) . Works in the last category show that private mechanisms can have a negative impact towards fairness [ 23 , 27 , 3 , 17 , 25 ] . For example , Cummings et al . [ 13 ] shows that it is impossible to achieve exact equalized odds while also satisfying pure DP . Pujol et al . [ 23 ] observe that decisions made using a private version of a dataset may disproportionately affect some groups over others . Similar observations were also made in the context of model learning . Bagdasaryan et al . [ 3 ] empirically observed that the accuracy of a DP model trained using DP-SGD drops disproportionately across groups causing larger negative impacts to the underrepresented groups . Farrand et al . [ 17 ] reaches similar conclusions . The authors empirically show that the disparate impact of differential privacy on model accuracy is not limited to highly imbalanced data and can occur even in situations where the classes are slightly imbalanced . This paper builds on this body of work and their important empirical observations . It derives the conditions and studies the causes of unfairness in the context of private empirical risk minimization problems as well as it introduces mitigating guidelines . 3 Preliminaries . Differential privacy ( DP ) [ 14 ] is a strong privacy notion used to quantify and bound the privacy loss of an individual participation to a computation . Informally , it states that the probability of any output does not change much when a record is added or removed from a dataset , limiting the amount of information that the output reveals about any individual . The action of adding or removing a record from a dataset D , resulting in a new dataset D′ , defines the notion of adjacency , denoted D ∼ D′ . Definition 1 . A mechanismM : D→R with domainD and range R is ( ϵ , δ ) -differentially private , if , for any two adjacent inputs D ∼ D′ ∈D , and any subset of output responses R ⊆ R : Pr [ M ( D ) ∈ R ] ≤ eϵ Pr [ M ( D′ ) ∈ R ] + δ. Parameter ϵ > 0 describes the privacy loss of the algorithm , with values close to 0 denoting strong privacy , while parameter δ ∈ [ 0 , 1 ) captures the probability of failure of the algorithm to satisfy ϵ-DP . The global sensitivity ∆ℓ of a real-valued function ℓ : D → Rk is defined as the maximum amount by which ℓ changes in two adjacent inputs : ∆ℓ = maxD∼D′ ∥ℓ ( D ) − ℓ ( D′ ) ∥ . In particular , the Gaussian mechanism , defined byM ( D ) = ℓ ( D ) +N ( 0 , ∆2ℓ σ2 ) , where N ( 0 , ∆2ℓ σ2 ) is the Gaussian distribution with 0 mean and standard deviation ∆2ℓ σ 2 , satisfies ( ϵ , δ ) -DP for δ > 45 exp ( − ( σϵ ) 2/2 ) and ϵ < 1 [ 16 ] . 4 Problem settings and goals . The paper adopts boldface symbols to describe vectors ( lowercase ) and matrices ( uppercase ) . Italic symbols are used to denote scalars ( lowercase ) and data features or random variables ( uppercase ) . Notation ∥·∥ is used to denote the L2 norm . The paper considers datasets D consisting of n individuals ’ data points ( Xi , Ai , Yi ) , with i ∈ [ n ] drawn i.i.d . from an unknown distribution . Therein , Xi ∈X is a feature vector , Ai ∈A is a protected group attribute , and Yi ∈Y is a label . For example , consider the case of a classifier that needs to predict the risks associated with a lending decision . The training example features Xi may describe the individual ’ s demographics , education , credit score , and loan amount , the protected attribute Ai may describe the individual gender or ethnicity , and Yi represents whether or not the individual will default on the loan . The goal is to learn a classifier fθ : X → Y , where θ is a vector of real-valued parameters , that guarantees the privacy of each individual data ( Xi , Ai , Yi ) in D. The model quality is measured in terms of a nonnegative loss function ℓ : Y ×Y → R+ , and the problem is that of minimizing the empirical risk ( ERM ) function : min θ L ( θ ; D ) = 1 n n∑ i=1 ℓ ( fθ ( Xi ) , Yi ) . ( L ) For a group a ∈ A , the paper uses Da to denote the subset of D containing exclusively samples whose group attribute A = a . The paper focuses on learning classifiers that protect the disclosure of the individuals ’ data using the notion of differential privacy and it analyzes the fairness impact ( as defined next ) of privacy on different groups of individuals . Importantly , the paper assumes that the attribute A is not part of the model input during inference . Fairness The fairness analysis focuses on the notion of excessive risk , a widely adopted metric in private learning [ 26 , 29 ] . It defines the difference between the private and non private risk functions : R ( θ , D ) = Eθ̃ [ L ( θ̃ ; D ) ] − L ( θ∗ ; D ) , ( 1 ) where the expectation is defined over the randomness of the private mechanism and θ̃ denotes the private model parameters while θ∗ = argminθ L ( θ ; D ) . The paper uses shorthands R ( θ ) and Ra ( θ ) to denote , respectively , the population-level R ( θ , D ) excessive risk and the group level R ( θ , Da ) excessive risk for group a . Fairness is measured with respect to the excessive risk gap : ξa = |Ra ( θ ) − R ( θ ) | . ( 2 ) ( Pure ) fairness is achieved when ξa = 0 for all groups a ∈ A and , thus , a private and fair classifier aims at minimizing the maximum excessive risk gap among all groups . The paper assumes that the private mechanisms are non-trivial , i.e. , they minimize the population-level excessive risk R ( θ ) . All proofs are reported in the Appendix , Section A . 5 Warm up : output perturbation . The paper starts with analyzing fairness under the DP setting induced by an output perturbation mechanism . In this setting the analysis restricts to twice differentiable and convex loss functions ℓ . Output perturbation is a standard DP paradigm in which noise calibrated to the function sensitivity is added directly to the output of the computation . In the context of the regularized ERM problem , adding noise drawn from a Gaussian distribution N ( 0 , ∆2ℓσ2 ) to the optimal model parameters θ∗ ensures ( ϵ , δ ) -differential privacy [ 11 ] . Therein , ∆ℓ = 2/nλ with regularization parameter λ . The following result sheds light on the unfairness induced by this mechanism . Theorem 1 . Let ℓ be a twice differentiable and convex loss function and consider the output perturbation mechanism described above . Then , the excessive risk gap for group a ∈ A is approximated by : ξa ≈ 1 2 ∆2ℓσ 2 ∣∣∣Tr ( Haℓ ) − Tr ( Hℓ ) ∣∣∣ , ( 3 ) where Haℓ =∇2θ∗ ∑ ( X , A , Y ) ∈Daℓ ( fθ∗ ( X ) , Y ) is the Hessian matrix of the loss function , at the optimal parameters vector θ∗ , computed using the group data Da , Hℓ is the analogous Hessian computed using the population data D , and Tr ( · ) denotes the trace of a matrix . The approximation above follows form a second order Taylor expansion of the loss function , linearity of expectation , and the properties of Gaussian distributions . It uses that fact that the excessive risk Ra ( θ ) for a group a can be approximated as 1/2∆2ℓσ 2 Tr ( Haℓ ) . The proof is reported in Appendix A. Theorem 1 sheds light on the relation between fairness and the difference in the local curvatures of the losses ℓ associated with a group and the population and provides a necessary condition to guarantee pure fairness . It suggests that output perturbation mechanisms may introduce unfairness when the local curvatures associated with the loss function of different groups differ substantially from one another . Additionally , the unfairness level is proportional to the amount of noise σ or , equivalently , inversely proportional to the privacy parameter ϵ , for a fixed δ . Finally , it also suggests that groups with larger Hessian traces Tr ( Hℓa ) will have larger excessive risk compared to groups with smaller Hessian traces . An additional analysis on the reasons behind why different groups may have large differences in their associated Hessian traces is provided in Section 8 . Figure 1 illustrates Theorem 1 . The plots show the correlation between the excessive risk 1 and the quantity Tr ( Hℓz ) for each group z ∈ A , at varying of the privacy loss ϵ ∈ [ 0.005 , 0.5 ] , δ = 1e−5 on two datasets . Each data point represents the average of 100 runs of a DP Logistic Regression ( obtained with output perturbation ) on each group z ∈ A . Details on dataset and experimental setting are provided in Appendix B and additional experiments in Appendix C. Note the positive correlation between the excessive risk and the Hessian trace : Groups with larger Hessian traces tend to have larger excessive risks . Note also the inverse correlation between ϵ and the dependency between the excessive risk and the Hessian trace . This is due to that larger ϵ values require smaller σ values , and thus , as shown in Equation 3 , the dependency between the excessive risk and Hessian trace is attenuated . The following illustrates that even a class of simple linear models may not to satisfy pure fairness . Corollary 1 . Consider the ERM problem for a linear model fθ ( X ) def = θT X , with L2 loss function i.e. , ℓ ( fθ ( X ) , Y ) = ( fθ ( X ) − Y ) 2 . Then , output perturbation does not guarantee pure fairness . It follows from the observation that the Hessian of the L2 loss for group a ∈ A , i.e. , Tr ( Haℓ ) = EX∼Da Tr ( XX T ) =EX∼Da∥X∥2 , depends solely on the input norms of the elements in Da2 . Interestingly , this result highlights the relation between fairness and the average input norms of different group elements . When these norms are substantially different one another they will impact their respective excessive risks differently . An additional analysis on this behavior is also discussed in Section 7 . The following is a positive result . Corollary 2 . If for any two groups a , b ∈ A their average group norms EXa∼Da∥Xa∥ = EX∼Db∥Xb∥ have identical values , then output perturbation with L2 loss function provides pure fairness . The above is a direct consequence of Corollary 1 . Note also that pure fairness may be achieved , in this setting , by normalizing the input values for each group independently ( as shown in Appendix C ) although this solution requires accessing the sensitive group attributes at inference time .

Summary: This paper investigates the underlying causes of the disparate impact produced by differentially private machine learning methods. To that end, the paper considers the notion of excessive risk gap, which measures the difference between the population-level and subgroup risks. The paper explores how two different differential privacy mechanisms (output perturbation and DP-SGD) affect the excessive risk gap. For output perturbation, the paper relates the excessive risk gap to the absolute difference between the population-level and subgroup trace of the hessian. For DP-SGD, the paper decomposes the expected loss into a non-private term, a private term due to clipping, and a private term due to noise. Finally, the paper leverages these insights to propose a novel formulation of DP-SGD that empirically decreases the excessive risk gap on real-world datasets.