Fair Differential Privacy Can Mitigate the Disparate Impact on Model Accuracy

1 INTRODUCTION . Protecting data privacy is a significant concern in many data-driven decision-making applications ( Zhu et al. , 2017 ) , such as social networking service , recommender system , location-based service . For example , the United States Census Bureau will firstly employ differential privacy to the 2020 census data ( Bureau , 2020 ) . Differential privacy ( DP ) guarantees that the released model can not be exploited by attackers to derive whether one particular instance is present or absent in the training dataset ( Dwork et al. , 2006 ) . However , DP intentionally restricts the instance influence and introduces noise into the learning procedure . When we enforce DP to a model , DP may amplify the discriminative effect towards the underrepresented and relatively complicated classes ( Bagdasaryan et al. , 2019 ; Du et al. , 2020 ; Jaiswal & Provost , 2020 ) . That is , reduction in accuracy from nonprivate learning to private learning may be uneven for each class . There are several empirical studies on utility reduction : ( Bagdasaryan et al. , 2019 ; Du et al. , 2020 ) show that the model accuracy in private learning tends to decrease more on classes that already have lower accuracy in non-private learning . ( Jaiswal & Provost , 2020 ) shows different observations that the inequality in accuracy is not consistent for classes across multiple setups and datasets . It needs to be cautionary that although private learning improves individual participants ’ security , the model performance should not harm one class more than others . The machine learning model , specifically in supervised learning tasks , outputs a hypothesis f ( x ; θ ) parameterized by θ , which predicts the label y given the unprotected attributes x . Each instance ’ s label y belongs to a class k. The model aims to minimize the objective ( loss ) function L ( θ ; x , y ) , i.e. , θ∗ : = arg min θ E [ L ( θ ; x , y ) ] . ( 1 ) Our work builds on a recent advance in machine learning models ’ training that uses the differentially private mechanism , i.e. , DPSGD ( Abadi et al. , 2016 ) for releasing model . The key idea can be extended to other DP mechanisms with the specialized noise form ( generally Laplacian or Gaussian distribution ) . The iterative update scheme of DPSGD at the ( t+ 1 ) -th iteration is of the form θ̃t+1 = θ̃t − µt · 1 n ( ∑ i∈St gt ( xi ) max ( 1 , ‖g t ( xi ) ‖2 C ) + ξ1 ) , ( 2 ) where n and µt denote the batch size and step-size ( learning rate ) respectively ; St denotes the randomly chosen instance set ; the vector 1 denotes the vector filled with scalar value one ; and gt ( xi ) denotes the gradient of the loss function in ( 1 ) at iteration t , i.e. , ∇L ( yi ; θt , xi ) . The two key operations of DPSGD are : i ) clipping each gradient gt ( xi ) in ` 2-norm based on the threshold parameter C ; ii ) adding noise ξ drawn from Gaussian distribution N ( 0 , σ2C2 ) with a variance of noise scale σ and the clipping threshold parameter C. These operations enable training machine learning models with non-convex objectives at a manageable privacy cost . Based on the result of traditional SGD , we theoretically analyze the sufficient decrease type scheme of DPSGD , i.e. , E [ f ( θt+1 ) ] 6 f ( θt ) + E [ 〈 ∇f ( θt ) , θt+1 − θt 〉 ] + L 2 E [ ∥∥θt+1 − θt∥∥2 ] + τ ( C , σ ; θt ) , ( 3 ) where the last term τ ( C , σ ; θt ) denotes the gap of loss expectation compared with ideal SGD at this ( t+ 1 ) -th iteration , and related with parameters C , and σ . The term τ ( C , σ ; θ ) , which can be called bias-variance term , can be calculated mathematically as 2 ( 1 + 1 µtL ) ‖∇f ( θ ) ‖ · η + η2︸ ︷︷ ︸ Clipping bias + 1 n2 · σ2C2|1|︸ ︷︷ ︸ Noise variance , ( 4 ) where L denotes the Lipschitz constant of f ; |1| denotes the vector dimension ; and we have η : = 1 n ∑ I‖gt ( xi ) ‖ > C ( ‖gt ( xi ) ‖ − C ) , where I‖gt ( xi ) ‖ > C denotes the cardinality number of satisfying ‖gt ( xi ) ‖ > C. The detailed proof of ( 3 ) and ( 4 ) can be found in Appendix A. τ ( C , σ ) is consist of clipping bias and noise variance terms , which means the amount that the private gradient differs from the non-private gradient due to the influence truncation and depending on the scale of the noise respectively . As a result , we call τ ( C , σ ) the bias-variance term . As underrepresented class instances or complicated instances manifest differently from common instances , a uniform threshold parameter C may incur significant accuracy disparate for different classes . In Figure 1 ( a ) , we employ DPSGD ( Abadi et al. , 2016 ) on the unbalanced MNIST dataset ( Bagdasaryan et al. , 2019 ) to numerical study the inequality of utility loss ( i.e. , the prediction accuracy gap between private model and non-private model ) caused by differential privacy . For the unbalanced MNIST dataset , the underrepresented class ( Class 8 ) has significantly larger utility loss than the other classes ( e.g. , Class 2 ) in the private model . DPSGD results in a 6.74 % decrease in accuracy on the well-represented classes , but accuracy on the underrepresented class drops 74.16 % . Training with more epochs does not reduce this gap while exhausting the privacy budget . DPSGD obviously introduces negative discrimination against the underrepresented class ( which already has lower accuracy in the non-private SGD model ) . Further , Figure 1 ( b ) shows the classification accuracy of different sub-classes for τ ( C , σ ; θ ) on the unbalanced MNIST dataset . Larger bias-variance term τ ( C , σ ; θ ) ( determined by C and σ ) results in more serious accuracy bias on different classes , while similar results are also shown in ( Bagdasaryan et al. , 2019 ; Du et al. , 2020 ; Jaiswal & Provost , 2020 ) . Both theoretical analysis and experimental discussion suggest that minimizing the clipping bias and noise variance simultaneously could learn “ better ” DP parameters , which mitigates the accuracy bias between different classes . This motivates us to pursue fairness with a self-adaptive differentially privacy scheme1 . This paper proposes a fair differential privacy algorithm ( FairDP ) to mitigate the disparate impact problem . FairDP introduces a self-adaptive DP mechanism and automatically adjusts instance influence in each class . The main idea is to formulate the problem as bilevel programming by minimizing the bias-variance term as the upper-level objective with a lower-level differential privacy machine learning model . The self-adaptive clipping threshold parameters are calculated by balancing the fairness bias-variance and per-class accuracy terms simultaneously . Our contributions can be summarized as follows : • FairDP uses a self-adaptive clipping threshold to adjust the instance influence in each class , so the model accuracy for each class is calibrated based on their privacy cost through fairness balancing . The class utility reduction is semblable for each class in FairDP . • To our knowledge , we are the first to introduce bilevel programming to private learning , aiming to mitigate the disparate impact on model accuracy . We further design an alternating scheme to learn the self-adaptive clipping and private model simultaneously . • Our experimental evaluation shows that FairDP strikes a balance among privacy , fairness , and accuracy by performing stratified clipping over different subclasses . The following is the road-map of this paper . Section 2 describes the proposed FairDP algorithm . In Section 3 , we provide a brief but complete introduction to related works in privacy-aware learning , fairness-aware learning , and the intersection of differential privacy and fairness . Extensive experiments are further presented in Section 4 , and we finally conclude this paper and discuss some future work in Section 5 . 2 FAIRDP : FAIR DIFFERENTIAL PRIVACY . 2.1 THE BILEVEL FAIRDP FORMULATION . Our approach ’ s intuition is to fairly balance the level of privacy ( based on the clipping threshold ) for each class based on their bias-variance terms , which are introduced by associated DP . The biasvariance terms arise from capping instance influences to reduce the sensitivity of a machine learning algorithm . In detail , a self-adaptive DP mechanism is designed to balance the bias-variance difference among all groups , while the obtained DP mechanism must adapt to the original machine learning problem simultaneously . Recall the definition of the machine learning problem , we assume there are ` classes and according to the bias-variance term ( 4 ) for class k ∈ { 1 , · · · , ` } can be denoted as τk ( Ck , σ ; θ ∗ ) : = 2 ( 1 + 1 µtL ) ‖∇f ( θ∗ ) ‖ · ηk + ηk2 + |Gk|2 n2 · σ2Ck2|1| , ( 5 ) where Ck denotes the clipping parameter for class k ; Gk denotes the data sample set for class k. As motivated by Section 1 , we aim to minimize the associated bias-variance term to obtain a unified clipping parameter for the machine learning problem . However , to mitigate the disparate impact on model accuracy for different classes , we minimize the summation of per-class bias-variance terms . This objective can lead to the self-adaptive clipping threshold among different classes , while the inconsistent DP schemes for different classes should work on the privacy protection on the machine learning model . The self-adaptive clipping threshold parameters should be utilized to learn the original machine learning privately with the DP mechanism . A simple bilevel programming problem2 ( Dempe et al. , 2019 ; Liu et al. , 2019 ) is introduced to model these two goals which influence each 1Note that we do not attempt to optimize the bias-variance bound in a differentially private way , and we are most interested in understanding the forces at play . 2The simple bilevel programming is not to say that the bilevel problem is simple , and it denotes a specific bilevel programming problem . other . The formulation can be denoted as follows , i.e. , min { Ck } , θ ∑̀ k=1 τk ( Ck , σ ; θ ) , ( 6a ) s.t . θ ∈ arg min θ L ( θ ; { Gk } ` k=1 ) , ( 6b ) where the upper-level problem ( 6a ) aims to fairly adjust the clipping threshold parameters for all classes , which is related to the classification model θ ; as for the lower-level problem ( 6b ) , we aim to learn the classification model based on the differential privacy schema with the self-adaptive clipping threshold { Ck } . These two objectives are coupled together , although the model of the lower-level problem is determined only by θ . The effect of clipping is reflected through the DP calculation procedure . Guided by the bias-variance term in ( 6a ) , the parameters of the DP learning can be finely updated simultaneously with the learning process of the classifiers in ( 6b ) . Algorithm 1 : The FairDP Method Input : Instances { ( x1 , y1 ) , · · · , ( xN , yN ) } , objective function L ( θ ; x , y ) , learning rate µt ; 1 Initialize θ0 ; 2 for t← 1 to T do 3 Randomly sample a batch of instances St with probability |S t| N ; Compute gradient 4 for xi ∈ St do 5 Compute gt ( xi ) ← ∇θtL ( yi ; θt , xi ) ; 6 end Minimize bias-variance 7 Ct+1k ← arg minCk τk ( Ck , σ ; θ t ) ; 8 for xi ∈ St and yi = k do Clip gradient 9 ḡt ( xi ) ← g t ( xi ) max ( 1 , ‖gt ( xi ) ‖2 C t+1 k ) ; 10 end Add noise 11 g̃t ← 1n ( ∑ i ḡ t ( xi ) + ξI ) ; Noise Gradient Descent 12 θt+1 ← θt − µtg̃t ; 13 end Output : θT , accumulated privacy cost ( , δ ) .

Summary: This paper addresses the problem of an unbalanced data set. In particular, the accuracy on the well-represented classes is higher than the accuracy in underrepresented classes in an unbalanced dataset. This paper shows that DPSGD makes the problem of an unbalanced dataset even worse and decreases accuracy on the underrepresented class significantly. Further, this paper introduces a modification of DPSGD, which can increase the underrepresented class's accuracy.