On Adversarial Bias and the Robustness of Fair Machine Learning

1 INTRODUCTION . Trustworthiness is a crucial requirement of machine learning algorithms in critical decision making processes , as highlighted by many AI regulations and policies as well as technical research papers . Algorithmic fairness is at the core of trust requirements for automated decision making in sensitive domains . Group fairness measures , such as equal opportunity and equalized odds ( Hardt et al. , 2016 ) which is the focus of this paper , suggest equalizing the model ’ s behavior across groups identified based on a protected attribute ( e.g. , race ) to avoid systemic discrimination against protected groups ( Agarwal et al. , 2018 ; Calders et al. , 2009 ; Hardt et al. , 2016 ; Madras et al. , 2018 ) . The main question that we are interested in is whether fair models are trustworthy , and in particular robust with respect to changes to their training data . In this paper , we study if and how achieving group fairness can increase the susceptibility of machine learning models to a small fraction of adversarially-sampled outliers in the training set . A large body of work shows machine learning is vulnerable to noisy and adversarial data ( Biggio et al. , 2012 ; Chen et al. , 2017 ; Jagielski et al. , 2018 ; Koh & Liang , 2017 ; Li et al. , 2016 ; Mei & Zhu , 2015a ; Shafahi et al. , 2018 ; Steinhardt et al. , 2017 ; Suciu et al. , 2018 ) . Recent work studies the performance of fair machine learning in the presence of noisy training data with under-representation and labeling bias ( Blum & Stangl , 2020 ; Calders & Žliobaitė , 2013 ; De-Arteaga et al. , 2018 ; Jiang & Nachum , 2020 ; Kallus & Zhou , 2018 ; Lamy et al. , 2019 ) . Under the assumptions of uniform noise and in the theoretical setting of having an unlimited number of training data , these works analyze the effect of noisy/biased training data on fair models . Interestingly , Blum & Stangl ( 2020 ) show that ERM with equal opportunity can recover the Bayes-optimal classifier from biased data . In other words , fair algorithms are more robust to certain types of bias in the training dataset than standard learning algorithms ( without fairness constraints ) . However , there has been little quantitative analysis of the interaction between group fairness and robustness of the model under realistic settings ( finite training data and non-uniform noise ) . In this paper , we quantitatively measure the impact of group fairness on the robustness of the model under worst-case ( adversarial ) bias , which exactly aims at minimizing the chance of recovering from biased data . To the best of our knowledge , this paper provides the first quantitative analysis for the robustness of fair machine learning algorithms in the adversarial setting . We assume the training data is biased , through adding a small fraction of outliers that are adversarially sampled ( and labeled ) to degrade the test accuracy of fair models . We exploit the fact that algorithms with group-fairness constraints approximately equalize the influence of different groups , in the training set , on the model . Equality of the group influence would consequently change the influence of individual data samples across different groups in a disproportionate way due to differences in size and distribution of the groups . Thus , the model ’ s susceptibility to worst-case outliers is largely dependent on how the outliers are distributed across different subgroups . We extensively evaluate the robustness of fair machine learning on multiple fairness algorithms ( Hardt et al. , 2016 ; Agarwal et al. , 2018 ; Rezaei et al. , 2020 ; Cotter et al. , 2019 ; Zhang et al. , 2018 ) and benchmark datasets ( Dua & Graff , 2017 ; Larson et al. , 2017 ; mep ; ahr ) to investigate how , why , and under what circumstances models with group fairness are more fragile with respect to adversarial bias compared to unconstrained models . We show that group fairness reduces robustness . Models trained using various fair machine learning algorithms are all more susceptible to adversarial bias compared with unconstrained models . We can observe this effect even for the case of the most limited scenario of adversarial data sampling for a small fraction of the training set , without manipulating the data features and labels . We notice this is because that adversarial bias amplifies the cost of fairness on model accuracy by placing the outliers into the smallest group with the least frequent label . It effectively reduces the best achievable accuracy for the smallest subgroup , limiting the fair models ’ accuracy on the minority . In this case , the model sacrifices its accuracy over the majority group to satisfy the fairness constraint . It results in a significant accuracy on the overall dataset . Furthermore , we present the potential trade-offs between robustness and fairness . Finally , adversarial manipulation of the training data prevents the model from generalizing its fairness to clean test data even though it is guaranteed on training data . This results in models that , according to the fairness measure , are even more discriminatory than unconstrained models , but on a different part of the population . This work introduces a significant challenge towards designing trustworthy machine learning algorithms . We emphasize that , as shown in our results , the fair models trained under noisy data could be significantly unfair ( with respect to the same fairness measures ) . Thus , sensitivity to changes in data undermines both fairness and accuracy of models . This calls for designing new fairness measures which are not inherently susceptible to noise . 2 BACKGROUND AND PROBLEM STATEMENT . Machine learning . Consider a classifier fθ : X → Y , that maps the feature space X to labels Y . The model is represented by its parameters θ taken from a parameter space Θ . The model is trained to minimize a loss function ` : Θ×X × Y → R+ over its training set D. We let X and Y denote the random variables associated with the features and the labels , and ( X , Y ) to denote the underlying distribution of the data . We obtain the model parameters by solving minθ∈Θ 1|D|L ( θ ; D ) , where L ( θ ; D ) = ∑ ( x , y ) ∈D ` ( θ ; x , y ) is the cumulative loss of the model over the training set D. Fairness . We assume all data points are split into groups based on a protected attribute S ∈ S ( e.g. , gender ) . This attribute could be part of the feature set X . We focus on equalized odds , which is a widely-used notion for group fairness ( Hardt et al. , 2016 ) . 1 Following previous works ( Agarwal et al. , 2018 ; Donini et al. , 2018 ) , we say a classifier fθ is δ-fair under equalized odds if ∆ ( θ , D ) : = max y∈Y , a , b∈S ∣∣∣∣PrD [ fθ ( X ) 6= y|S = a , Y = y ] − PrD [ fθ ( X ) 6= y|S = b , Y = y ] ∣∣∣∣ ≤ δ , ( 1 ) where the probabilities are computed empirically over the training data set D. We refer to ∆ as the model ’ s empirical fairness gap . A model satisfies exact equalized odds fairness when δ = 0 . In practice , fairness is usually achieved by ensuring δ-fairness empirically on the model ’ s training set , e.g. , through minimizing the model ’ s empirical loss under δ-fairness as a constraint ( Agarwal et al. , 2018 ) or post-processing ( Hardt et al. , 2016 ) . We define the constraint C ( θ , D ) : = ∆ ( θ , D ) − δ ≤ 0 as a fairness constraint . We refer to the models learned with the fairness constraint as fair models , 1Extension and analysis for other group fairness metrics ( i.e. , equal opportunity ( Hardt et al. , 2016 ) ) can be found in Appendix E.7 . to distinguish them from unconstrained models that are learned without any fairness constraint . We quantify the performance of a model based on its test accuracy , and its fairness gap on the test dataset . Problem statement . The primary research question we investigate in this paper is whether , how , and why models with ( equalized odds ) group fairness are less robust to adversarial bias , compared with unconstrained models . We consider a model is robust if changing a small fraction of the training set does not significantly downgrade the predictive power of the model . Towards quantifying the robustness , we assume the biased training set D is composed of the clean datasetDc of size n , and the adversarially chosen datasetDp of size n. The clean training setDc and test set Dtest are sampled from the same underlying distribution ( X , Y ) . So , we investigate the effect of the bias in the training set , which is introduced through Dp . We consider two variations of bias : adversarial sampling , and adversarial labeling . These are the worst-case sampling and labeling bias , whereDp is chosen to maximize the loss of a model . LetDk be a dataset sampled from ( X , Y ) , similar to the clean data . In adversarial sampling , we choose outliers Dp ⊂ Dk , and in adversarial labeling , we choose them also with the possibility of crafting their labelsDp ⊂ { ( x , y′ ) : ( x , y ) ∈ Dk , y′ ∈ Y } . Based on this setting , the maximum vulnerability of a fair model in the presence of adversarial bias can be formulated as a bi-level optimization problem subject to the fairness constraint C ( θ , D ) ≤ 0 : max Dp E ( X , Y ) [ ` ( θ̂ ; X , Y ) ] , where θ̂ : = argmin θ∈Θ L ( θ ; Dc ∪ Dp ) |Dc ∪ Dp| , st. C ( θ , Dc ∪ Dp ) ≤ 0 , ( 2 ) where the expectation is taken over the underlying ( clean ) data distribution . The outer maximization searches for the strongest Dp that maximizes the expected loss of the fair model ( given by θ̂ ) . This fair model is obtained by solving the inner constrained minimization on the biased training dataset Dc ∪ Dp . The expected loss can be measured on a test set Dtest sampled from the data distribution ( X , Y ) . To generate the strongest Dp , we assume the knowledge of the learning algorithm ( e.g. , logistic regression , SVM ) and the clean training datasetDc is available when generatingDp , but the exact fair learning algorithm is unknown . It allows us to obtain an upper bound on the performance degradation incurred by the adversarial bias and serves as a starting point towards understanding the maximal vulnerability . Besides , for investigating the effect of fairness constraints on the model robustness , we evaluate the robustness of unconstrained models as a baseline . Adversarial bias against unconstrained models is equivalent to the problem of data poisoning attacks ( we discuss more in Section 6 ) . 3 ADVERSARIAL BIAS . To find the strongest Dp for evaluating the robustness of fair models , we need to solve the bi-level optimization problem ( 2 ) , which is non-convex and intractable ( Bard , 1991 ; Hansen et al. , 1992 ; Deng , 1998 ) . The fairness constraint makes the problem even more difficult.2 In this section , we explain how to approximate problem ( 2 ) to design effective adversarial strategies . In the problem ( 2 ) , it is hard to track the influence of Dp on the test loss as it can only affect the test loss via the model ’ s parameters . To make progress , we first approximate the loss on the test data by the loss on the clean training data , following the approximations used for designing poisoning attacks against unconstrained models ( Steinhardt et al. , 2017 ) . Specifically , let θ̂ be the solution to the inner optimization problem and we have L ( θ̂ ; Dtest ) /|Dtest| ≈ L ( θ̂ ; Dc ) /|Dc| ≤ L ( θ̂ ; Dc ∪ Dp ) /|Dc| . As long as the model has enough capacity to fit but does not overfit the training dataset ( which can be achieved by appropriate regularization ) , the loss on the biased training dataset ( i.e. , RHS of the inequality ) provides a good approximation for the test loss , which allows us to explicitly measure the impact of the poisoning data . Therefore , we replace the objective of the inner minimization in Eq . ( 2 ) with L ( θ ; Dc∪Dp ) /n , where n = |Dc| . The resulting optimization problem is hard to solve due to the bi-level optimization and the constraints in the inner optimization . To resolve this , we then relax the inner constrained optimization by introducing a Lagrange multiplier λ ∈ R+ : min θ∈Θ [ L ( θ ; D ) n , s.t . C ( θ , D ) ≤ 0 ] = min θ∈Θ max λ∈R+ ( L ( θ ; D ) n + λC ( θ , D ) ) ≥ max λ∈R+ min θ∈Θ ( L ( θ ; D ) n + λC ( θ , D ) ) , 2We would like to point out that , for the unconstrained model , under the convex assumption of the loss function , it is possible to find the approximate solution by replacing the inner optimization with its stationarity ( KKT ) condition ( Biggio et al. , 2012 ; Koh & Liang , 2017 ) . where D = Dc ∪ Dp . The inequality follows from the weak duality theorem . We can now find Dp by maximizing a lower bound provided by the Lagrangian function minθ∈Θ ( L ( θ ; D ) /n + λC ( θ , D ) ) for a fixed λ ∈ R+ . Indeed , maximizing the lower bound provided by the Lagrangian function would result in a solution with a high loss ( which is guaranteed to be at least equal to the loss for the lower bound ) for the original problem . In this optimization procedure , we can also replace the fairness constraint C ( θ , D ) : = ∆ ( θ , D ) − δ with the fairness gap ∆ ( θ , D ) , because the constant value δ ≥ 0 does not affect the solution for the Lagrangian . Finally , by considering all the above-mentioned steps , the new optimization problem is : max Dp min θ∈Θ ( L ( θ ; Dc ∪ Dp ) n + λ∆ ( θ , Dc ∪ Dp ) ) . ( 3 ) Thus , to solve the problem ( 2 ) , the alternative goal is to find Dp that maximizes a linear combination of the training loss and the model ’ s violation from the fairness constraint , where λ controls the penalty for the violation . In the light of those approximations , we design two algorithms to generate Dp . An Approximation for the Fairness Gap . Finding Dp is an intractable combinatorial optimization problem because the fairness gap could not be split into separate functions of individual data points . It is still hard to track the influence of individual data points . To resolve it , we first find an additive proxy for fairness gap . We substitute the fairness gap ∆ by the average of an approximate contribution of each training data point to the fairness gap . This allows us to design an efficient sequential policy . More specifically , let { ( x , y ) } k be a multi-set with k repetitions of a data point ( x , y ) . Consequently , D ∪ { ( x , y ) } k is equivalent to adding k copies of ( x , y ) to D. In this setting , for any data point ( x , y ) ∈ Dp , the fairness gap 1 n∆ ( θ , Dc ∪ { ( x , y ) } n ) is a proxy for measuring the contribution of that data point to the fairness gap ∆ ( θ , Dc ∪ Dp ) . In other words , it measures how the fairness gap changes if n copies of ( x , y ) is added to the clean data . Also , the maximum of ∆ ( θ , Dc ∪ { ( x , y ) } n ) over all data points ( x , y ) ∈ Dp provides an upper bound on the fairness gap of the model , when the size of Dp is n. Given this proxy for the contribution of each data point to the fairness gap , we obtain the following approximation : ∆ ( θ , Dc ∪ Dp ) ≈ ∑ ( x , y ) ∈Dp 1 n∆ ( θ , Dc ∪ { ( x , y ) } n ) . By substituting the fairness gap with its proxy , now we can solve the following optimization problem : max Dp min θ∈Θ ( 1 n L ( θ ; Dc ∪ Dp ) + λ n ∑ ( x , y ) ∈Dp ∆ ( θ , Dc ∪ { ( x , y ) } n ) ) = max Dp M ( Dp ) : = M∗ ( 4 ) where M ( Dp ) is the loss incurred by a poisoning set Dp on the fair model , and M∗ is the maximum loss of the fair model under any choices of Dp . Algorithm 1 , a variant of the no-regret online gradient descent methods ( Hazan , 2016 ) , presents our solution to problem ( 4 ) . It initializes a model θ0 ∈ Θ , and identifies n points for Dp iteratively . The feasible set of points F ( Dk ) is determined by the adversarial bias setting . For adversarial sampling , we have F ( Dk ) = Dk , and for adversarial labeling , F ( Dk ) = { ( x , y′ ) : ( x , y ) ∈ Dk , y′ ∈ Y } . The algorithm iteratively performs the following steps : • Data point selection . ( Algorithm 1 , line 5 ) : It selects a data point with the highest impact on a weighted sum of the loss function and the fairness gap with respect to the model parameter θt−1 . • Parameter update . ( Algorithm 1 , line 7 ) : The parameters are updated to minimize the penalized loss function based on the selected data point ( xt , yt ) . In this way , the algorithm ( through the approximations made by the Lagrange multiplier and the surrogate function ) keeps track of the fair model under the set of already selected data points for Dp . In Theorem 1 , by following the approach proposed by ( Steinhardt et al. , 2017 ) , we relate the performance of Algorithm 1 with the maximum loss in Eq . ( 4 ) . Moreover , in Appendix C.2 , we prove that under some reasonable conditions ( e.g. , by using similar assumptions made by ( Donini et al. , 2018 ) to approximate the fairness gap ) , our algorithm finds the ( nearly ) optimal solution for Eq . ( 4 ) . Theorem 1 . Let D∗p be the data set produced by Algorithm 1 . Let Regret ( n ) be the regret of this online learning algorithm after n steps . The performance of the algorithm is guaranteed by M∗ −M ( D∗p ) ≤ Regret ( n ) n , ( 5 ) where M∗ and M ( D∗p ) are the loss of the fair model under the optimal Dp and D∗p , respectively.3 3The regret of a decision-maker is defined as the difference between the total cost incurred and that of the best-fixed decision in hindsight . Algorithm 1 Online Gradient Descent Algorithm for Generating Dp for Fair Models 1 : Input : Clean data Dc , n = |Dc| , feasible set F ( Dk ) , n ( the size of Dp ) , penalty parameter ( Lagrange multiplier ) λ , learning rate η . 2 : Output : Dp . 3 : Initialize θ0 ∈ Θ 4 : for t = 1 , · · · , n do 5 : ( xt , yt ) ← argmax ( x , y ) ∈F ( Dk ) [ · ` ( θt−1 ; x , y ) + λ ·∆ ( θt−1 , Dc ∪ { ( x , y ) } n ) ] 6 : Dp ← Dp ∪ { ( xt , yt ) } 7 : θt ← θt−1 − η ( ∇L ( θt−1 ; Dc ) n +∇ [ · ` ( θt−1 ; xt , yt ) + λ ·∆ ( θt−1 , Dc ∪ { ( xt , yt ) } n ) ] ) 8 : end for The proof for the theorem is deferred to Appendix C. From the theorem , it is clear that when the average regret Regret ( n ) / n is small , the Algorithm 1 will result in a nearly optimal Dp . A Surrogate Function for the Fair Model . The parameter update step in Algorithm 1 provides an approximation ( through adding the fairness constraint as a penalty and approximating the fairness gap ) for the fair model . Differently , our Algorithm 2 approximates the fair model using the unconstrained model . More specifically , Algorithm 2 iteratively adds data points that maximize a combination of the loss and the fairness gap , however , over the unconstrained model . The reason for this approach is that we hope the points with the largest weighted sum of the loss and the fairness gap on the unconstrained model may still have a large weighted sum on the fair models . Algorithm 2 in Appendix C.3 presents the pseudo-code of this algorithm . An advantage of Algorithm 2 is that it reduces the chance of getting stuck in local minima because , in each parameter update , it makes a step towards the negative gradient of the exact unconstrained loss . This is in contrast with Algorithm 1 , where due to the difficulty of approximating a constrained max-min problem , it might converge to some parameters not close to the fair model at all . We should point out that the algorithm and objectives are similar to the data poisoning attacks against unconstrained models when λ = 0 ( Steinhardt et al. , 2017 ) . In this case , the fairness constraints are not exploited to introduce adversarial bias . In Section 5 , we empirically show that Dp generated by exploiting the fairness gap ( i.e. , λ > 0 ) can incur a higher test loss of fair models compared with the case where λ = 0 .

Summary: Intuitively, the same amount of data poisoning will have a larger impact when the learner is solving a constrained optimization problem than an unconstrained one.  This paper proposed an attack algorithm specifically designed for such fair learners and showed that fair learning algorithms are more vulnerable to adversarial data poisoning attacks.  The problem studied in this paper is of high importance.