Fair Differential Privacy Can Mitigate the Disparate Impact on Model Accuracy

1 INTRODUCTION . Protecting data privacy is a significant concern in many data-driven decision-making applications ( Zhu et al. , 2017 ) , such as social networking service , recommender system , location-based service . For example , the United States Census Bureau will firstly employ differential privacy to the 2020 census data ( Bureau , 2020 ) . Differential privacy ( DP ) guarantees that the released model can not be exploited by attackers to derive whether one particular instance is present or absent in the training dataset ( Dwork et al. , 2006 ) . However , DP intentionally restricts the instance influence and introduces noise into the learning procedure . When we enforce DP to a model , DP may amplify the discriminative effect towards the underrepresented and relatively complicated classes ( Bagdasaryan et al. , 2019 ; Du et al. , 2020 ; Jaiswal & Provost , 2020 ) . That is , reduction in accuracy from nonprivate learning to private learning may be uneven for each class . There are several empirical studies on utility reduction : ( Bagdasaryan et al. , 2019 ; Du et al. , 2020 ) show that the model accuracy in private learning tends to decrease more on classes that already have lower accuracy in non-private learning . ( Jaiswal & Provost , 2020 ) shows different observations that the inequality in accuracy is not consistent for classes across multiple setups and datasets . It needs to be cautionary that although private learning improves individual participants ’ security , the model performance should not harm one class more than others . The machine learning model , specifically in supervised learning tasks , outputs a hypothesis f ( x ; θ ) parameterized by θ , which predicts the label y given the unprotected attributes x . Each instance ’ s label y belongs to a class k. The model aims to minimize the objective ( loss ) function L ( θ ; x , y ) , i.e. , θ∗ : = arg min θ E [ L ( θ ; x , y ) ] . ( 1 ) Our work builds on a recent advance in machine learning models ’ training that uses the differentially private mechanism , i.e. , DPSGD ( Abadi et al. , 2016 ) for releasing model . The key idea can be extended to other DP mechanisms with the specialized noise form ( generally Laplacian or Gaussian distribution ) . The iterative update scheme of DPSGD at the ( t+ 1 ) -th iteration is of the form θ̃t+1 = θ̃t − µt · 1 n ( ∑ i∈St gt ( xi ) max ( 1 , ‖g t ( xi ) ‖2 C ) + ξ1 ) , ( 2 ) where n and µt denote the batch size and step-size ( learning rate ) respectively ; St denotes the randomly chosen instance set ; the vector 1 denotes the vector filled with scalar value one ; and gt ( xi ) denotes the gradient of the loss function in ( 1 ) at iteration t , i.e. , ∇L ( yi ; θt , xi ) . The two key operations of DPSGD are : i ) clipping each gradient gt ( xi ) in ` 2-norm based on the threshold parameter C ; ii ) adding noise ξ drawn from Gaussian distribution N ( 0 , σ2C2 ) with a variance of noise scale σ and the clipping threshold parameter C. These operations enable training machine learning models with non-convex objectives at a manageable privacy cost . Based on the result of traditional SGD , we theoretically analyze the sufficient decrease type scheme of DPSGD , i.e. , E [ f ( θt+1 ) ] 6 f ( θt ) + E [ 〈 ∇f ( θt ) , θt+1 − θt 〉 ] + L 2 E [ ∥∥θt+1 − θt∥∥2 ] + τ ( C , σ ; θt ) , ( 3 ) where the last term τ ( C , σ ; θt ) denotes the gap of loss expectation compared with ideal SGD at this ( t+ 1 ) -th iteration , and related with parameters C , and σ . The term τ ( C , σ ; θ ) , which can be called bias-variance term , can be calculated mathematically as 2 ( 1 + 1 µtL ) ‖∇f ( θ ) ‖ · η + η2︸ ︷︷ ︸ Clipping bias + 1 n2 · σ2C2|1|︸ ︷︷ ︸ Noise variance , ( 4 ) where L denotes the Lipschitz constant of f ; |1| denotes the vector dimension ; and we have η : = 1 n ∑ I‖gt ( xi ) ‖ > C ( ‖gt ( xi ) ‖ − C ) , where I‖gt ( xi ) ‖ > C denotes the cardinality number of satisfying ‖gt ( xi ) ‖ > C. The detailed proof of ( 3 ) and ( 4 ) can be found in Appendix A. τ ( C , σ ) is consist of clipping bias and noise variance terms , which means the amount that the private gradient differs from the non-private gradient due to the influence truncation and depending on the scale of the noise respectively . As a result , we call τ ( C , σ ) the bias-variance term . As underrepresented class instances or complicated instances manifest differently from common instances , a uniform threshold parameter C may incur significant accuracy disparate for different classes . In Figure 1 ( a ) , we employ DPSGD ( Abadi et al. , 2016 ) on the unbalanced MNIST dataset ( Bagdasaryan et al. , 2019 ) to numerical study the inequality of utility loss ( i.e. , the prediction accuracy gap between private model and non-private model ) caused by differential privacy . For the unbalanced MNIST dataset , the underrepresented class ( Class 8 ) has significantly larger utility loss than the other classes ( e.g. , Class 2 ) in the private model . DPSGD results in a 6.74 % decrease in accuracy on the well-represented classes , but accuracy on the underrepresented class drops 74.16 % . Training with more epochs does not reduce this gap while exhausting the privacy budget . DPSGD obviously introduces negative discrimination against the underrepresented class ( which already has lower accuracy in the non-private SGD model ) . Further , Figure 1 ( b ) shows the classification accuracy of different sub-classes for τ ( C , σ ; θ ) on the unbalanced MNIST dataset . Larger bias-variance term τ ( C , σ ; θ ) ( determined by C and σ ) results in more serious accuracy bias on different classes , while similar results are also shown in ( Bagdasaryan et al. , 2019 ; Du et al. , 2020 ; Jaiswal & Provost , 2020 ) . Both theoretical analysis and experimental discussion suggest that minimizing the clipping bias and noise variance simultaneously could learn “ better ” DP parameters , which mitigates the accuracy bias between different classes . This motivates us to pursue fairness with a self-adaptive differentially privacy scheme1 . This paper proposes a fair differential privacy algorithm ( FairDP ) to mitigate the disparate impact problem . FairDP introduces a self-adaptive DP mechanism and automatically adjusts instance influence in each class . The main idea is to formulate the problem as bilevel programming by minimizing the bias-variance term as the upper-level objective with a lower-level differential privacy machine learning model . The self-adaptive clipping threshold parameters are calculated by balancing the fairness bias-variance and per-class accuracy terms simultaneously . Our contributions can be summarized as follows : • FairDP uses a self-adaptive clipping threshold to adjust the instance influence in each class , so the model accuracy for each class is calibrated based on their privacy cost through fairness balancing . The class utility reduction is semblable for each class in FairDP . • To our knowledge , we are the first to introduce bilevel programming to private learning , aiming to mitigate the disparate impact on model accuracy . We further design an alternating scheme to learn the self-adaptive clipping and private model simultaneously . • Our experimental evaluation shows that FairDP strikes a balance among privacy , fairness , and accuracy by performing stratified clipping over different subclasses . The following is the road-map of this paper . Section 2 describes the proposed FairDP algorithm . In Section 3 , we provide a brief but complete introduction to related works in privacy-aware learning , fairness-aware learning , and the intersection of differential privacy and fairness . Extensive experiments are further presented in Section 4 , and we finally conclude this paper and discuss some future work in Section 5 . 2 FAIRDP : FAIR DIFFERENTIAL PRIVACY . 2.1 THE BILEVEL FAIRDP FORMULATION . Our approach ’ s intuition is to fairly balance the level of privacy ( based on the clipping threshold ) for each class based on their bias-variance terms , which are introduced by associated DP . The biasvariance terms arise from capping instance influences to reduce the sensitivity of a machine learning algorithm . In detail , a self-adaptive DP mechanism is designed to balance the bias-variance difference among all groups , while the obtained DP mechanism must adapt to the original machine learning problem simultaneously . Recall the definition of the machine learning problem , we assume there are ` classes and according to the bias-variance term ( 4 ) for class k ∈ { 1 , · · · , ` } can be denoted as τk ( Ck , σ ; θ ∗ ) : = 2 ( 1 + 1 µtL ) ‖∇f ( θ∗ ) ‖ · ηk + ηk2 + |Gk|2 n2 · σ2Ck2|1| , ( 5 ) where Ck denotes the clipping parameter for class k ; Gk denotes the data sample set for class k. As motivated by Section 1 , we aim to minimize the associated bias-variance term to obtain a unified clipping parameter for the machine learning problem . However , to mitigate the disparate impact on model accuracy for different classes , we minimize the summation of per-class bias-variance terms . This objective can lead to the self-adaptive clipping threshold among different classes , while the inconsistent DP schemes for different classes should work on the privacy protection on the machine learning model . The self-adaptive clipping threshold parameters should be utilized to learn the original machine learning privately with the DP mechanism . A simple bilevel programming problem2 ( Dempe et al. , 2019 ; Liu et al. , 2019 ) is introduced to model these two goals which influence each 1Note that we do not attempt to optimize the bias-variance bound in a differentially private way , and we are most interested in understanding the forces at play . 2The simple bilevel programming is not to say that the bilevel problem is simple , and it denotes a specific bilevel programming problem . other . The formulation can be denoted as follows , i.e. , min { Ck } , θ ∑̀ k=1 τk ( Ck , σ ; θ ) , ( 6a ) s.t . θ ∈ arg min θ L ( θ ; { Gk } ` k=1 ) , ( 6b ) where the upper-level problem ( 6a ) aims to fairly adjust the clipping threshold parameters for all classes , which is related to the classification model θ ; as for the lower-level problem ( 6b ) , we aim to learn the classification model based on the differential privacy schema with the self-adaptive clipping threshold { Ck } . These two objectives are coupled together , although the model of the lower-level problem is determined only by θ . The effect of clipping is reflected through the DP calculation procedure . Guided by the bias-variance term in ( 6a ) , the parameters of the DP learning can be finely updated simultaneously with the learning process of the classifiers in ( 6b ) . Algorithm 1 : The FairDP Method Input : Instances { ( x1 , y1 ) , · · · , ( xN , yN ) } , objective function L ( θ ; x , y ) , learning rate µt ; 1 Initialize θ0 ; 2 for t← 1 to T do 3 Randomly sample a batch of instances St with probability |S t| N ; Compute gradient 4 for xi ∈ St do 5 Compute gt ( xi ) ← ∇θtL ( yi ; θt , xi ) ; 6 end Minimize bias-variance 7 Ct+1k ← arg minCk τk ( Ck , σ ; θ t ) ; 8 for xi ∈ St and yi = k do Clip gradient 9 ḡt ( xi ) ← g t ( xi ) max ( 1 , ‖gt ( xi ) ‖2 C t+1 k ) ; 10 end Add noise 11 g̃t ← 1n ( ∑ i ḡ t ( xi ) + ξI ) ; Noise Gradient Descent 12 θt+1 ← θt − µtg̃t ; 13 end Output : θT , accumulated privacy cost ( , δ ) .

Summary: The paper introduces an algorithm for mitigating disparate impact of private learning (DP-SGD) on different groups of a given population. In each iteration of DP-SGD, instead of using a uniform gradient clipping threshold for all groups, the proposed Fair DP-SGD algorithm uses an optimal clipping threshold (one that minimizes the bias-variance tradeoff) for each group separately. The authors include experimental results to show how well their algorithm performs compared to state-of-the-art algorithms.