A Technical and Normative Investigation of Social Bias Amplification

1 INTRODUCTION . The machine learning community is becoming increasingly cognizant of problems surrounding fairness and bias , and correspondingly a plethora of new algorithms and metrics are being proposed ( see e.g. , Mehrabi et al . ( 2019 ) for a review ) . The gatekeepers checking the systems to be deployed often take the form of fairness evaluation metrics , and it is vital that these be deeply investigated both technically and normatively . In this paper , we endeavor to do this for bias amplification . Bias amplification happens when a model exacerbates biases from the training data at test time . It is the result of the algorithm ( Foulds et al. , 2018 ) , and unlike other forms of bias , can not be solely attributed to the dataset . To this end , we propose a new way of measuring bias amplification , BiasAmp→ 1 , that builds off a prior metric from Men Also Like Shopping ( Zhao et al. , 2017 ) , that we will call BiasAmpMALS . Our metric ’ s technical composition aligns with the real-world qualities we want it to encompass , addressing a number of the previous metric ’ s shortcomings by being able to : 1 ) generalize beyond binary attributes , 2 ) take into account the base rates that people of each attribute appear , and 3 ) disentangle the directions of amplification . Concretely , consider a visual dataset ( Fig . 1 ) where each image has a label for the task , T , which is painting or not painting , and further is associated with a protected attribute , A , which is woman or man . If the gender of the person biases the prediction of the task , we consider this A→ T bias amplification ; if the reverse happens , then T → A . In our normative discussion , we discuss a few topics . We consider whether predicting protected attributes is necessary in the first place ; by not doing so , we can trivially remove T → A amplification . We also encourage the use of confidence intervals when using our metric because BiasAmp→ , along with other fairness metrics , suffers from the Rashomon Effect ( Breiman , 2001 ) , or multiplicity of good models . In deep neural networks , random seeds have relatively little impact on accuracy ; however , that is not the case for fairness , which is more brittle to randomness . 1The arrow in BiasAmp→ is meant to signify the direction that bias amplification is flowing , and not intended to be a claim about causality . Notably , a trait of bias amplification is that it is not at odds with accuracy , unlike many other fairness metrics , because the goal of not amplifying biases and matching task-attribute correlations is aligned with that of accurate predictions . For example , imagine a dataset where the positive outcome is associated at a higher rate with group A than with group B . A classifier that achieves 100 % accuracy at predicting the positive outcome is not amplifying bias ; however , according to metrics like demographic parity , this perfect classifier is still perpetuating bias because it is predicting the positive label at different rates for both groups . While matching training correlations is desired in object detection where systems should perfectly predict the labels , we will explore the nuances of what this means in situations where the validity of the labels , and thus task-attribute correlations themselves , are up for debate . For example , in the risk prediction task which assesses someone ’ s likelihood of recidivism , the label represents whether someone with a set of input features ended up recidivating , but is not a steadfast indicator of what another person with the same input features will do . Here , we would not want to replicate the task-attribute correlations at test time , and it is important to keep this in mind when deciding what fairness metrics to apply . The notion of amplification also allows us to encapsulate the idea that systemic harms and biases can be more harmful than errors made without such a history ( Bearman et al. , 2009 ) ; for example , in images overclassifying women as cooking carries more of a negative connotation than overclassifying men as cooking.2 Distinguishing between which errors are more harmful than others is a pattern that can often be lifted from the training data . To ground our work , we first distinguish what bias amplification captures that standard fairness metrics can not , and then distinguish BiasAmp→ from BiasAmpMALS . Our key contributions are : 1 ) proposing a new way to measure bias amplification , addressing multiple shortcomings of prior work and allowing us to better diagnose where a model goes wrong , and 2 ) providing a technical analysis and normative discussion around the use of this measure in diverse settings , encouraging thoughtfulness with each application . 2 RELATED WORK . Fairness Measurements . Fairness is nebulous and context-dependent , and approaches to quantifying it ( Verma & Rubin , 2018 ; Buolamwini & Gebru , 2018 ) include equalized odds ( Hardt et al. , 2016 ) , equal opportunity ( Hardt et al. , 2016 ) , demographic parity ( Dwork et al. , 2012 ; Kusner et al. , 2017 ) , fairness through awareness ( Dwork et al. , 2012 ; Kusner et al. , 2017 ) , fairness through unawareness ( Grgic-Hlaca et al. , 2016 ; Kusner et al. , 2017 ) , and treatment equality ( Berk et al. , 2017 ) . We examine bias amplification , which is a type of group fairness where correlations are amplified . Bias Amplification . Bias amplification has been measured by looking at binary classifications ( Leino et al. , 2019 ) , GANs ( Jain et al. , 2020 ; Choi et al. , 2020 ) , and correlations ( Zhao et al. , 2017 ) . Wang et al . ( 2019 ) measures this using dataset leakage and model leakage . The difference between these values is the level of bias amplification , but this is not a fair comparison because the 2We use the terms man and woman to refer to binarized socially-perceived gender expression , recognizing these labels are not inclusive , and in vision datasets are often assigned by annotators rather than self-disclosed . attribute classifier gets discrete labels for the former but continuous model outputs for the latter . Jia et al . ( 2020 ) looks at output distributions like we do , but with a different formulation . The Word Embedding Association Test ( WEAT ) ( Caliskan et al. , 2017 ) measures bias amplification in de-contextualized word embeddings , looking at correlations but not causations ( Bolukbasi et al. , 2016 ) . However , with newer models like BERT and ELMo that have contextualized embeddings , WEAT does not work ( May et al. , 2019 ) , so new techniques have been proposed incorporating context ( Lu et al. , 2019 ; Kuang & Davison , 2016 ) . We use these models to measure the directional aspect of these amplifications , as well as to situate them in the broader world of bias amplification . Directionality . Directionality of amplification has been observed in computer vision ( Stock & Cisse , 2018 ) and language ( Qian et al. , 2019 ) . It has also been studied with causality ( Bhattacharya & Vogt , 2007 ; Wooldridge , 2016 ; Pearl , 2010 ; Middleton et al. , 2016 ; Steiner & Yongnam , 2016 ) . We take a deeper and more empirical approach . Predictive Multiplicity . The Rashomon Effect ( Breiman , 2001 ) , or multiplicity of good models , has been studied in various contexts . The variables investigated that differ across good models include explanations ( Hancox-Li , 2020 ) , individual treatments ( Marx et al. , 2020 ; Pawelczyk et al. , 2020 ) , and variable importance ( Fisher et al. , 2019 ; Dong & Rudin , 2019 ) . We build on these discoveries and investigate how fairness measurements also differ between equally “ good ” models . 3 EXISTING FAIRNESS METRICS . In this section we present existing fairness metrics and show how bias amplification can distinguish errors resulting from under- and overclassification in a way that others can not , followed by a discussion of the shortcomings of BiasAmpMALS . 3.1 OVERVIEW OF EXISTING FAIRNESS METRICS . We begin with a review of existing fairness metrics in a concrete classification setting . We consider again the example from Fig . 1 , where on this dataset women ( a0 ) are correlated with painting ( t0 ) , and men ( a1 ) with not painting ( t1 ) , such that there are N images each of ( a0 , t0 ) and ( a1 , t1 ) but only N/2 images of ( a0 , t1 ) and ( a1 , t0 ) . A classifier trained to recognize painting on this data is likely to learn this association and over-predict painting on images of women and under-predict painting on images of men ; however , algorithmic interventions may counteract this effect and in fact result in the opposite behavior . Fig . 2 shows the behavior of fairness metrics under varying amounts of learned amplification of the correlation . The four fairness metrics are : False Positive Rate ( FPR ) and True Positive Rate ( TPR ) difference : the difference in false positive ( true positive ) rate of predicting the label t0 on images of a1 versus on images of a0 ( Chouldechova , 2016 ; Hardt et al. , 2016 ) , accuracy difference : difference between the overall task prediction accuracy on images of a1 versus on images of a0 ( Berk et al. , 2017 ) , and mean accuracy across subgroups : mean task prediction accuracy across the four image subgroups ( ( a0 , t0 ) , ( a1 , t0 ) , ( a0 , t1 ) , and ( a1 , t1 ) ) ( Buolamwini & Gebru , 2018 ) . However , these metrics are not designed to account for the training correlations , and are unable to distinguish between cases of increased or decreased learned correlations , as seen in Fig . 2 . Zhao et al . ( 2017 ) introduced an alternative to these that explicitly captures the notion of bias amplification . Concretely , they consider P ( A = a|T = t ) of the training data as the fraction of times a protected attribute a ∈ A appears on images corresponding to task t ∈ T . They then compare this with the test-time predictions made by the model , P ( Â = a|T̂ = t ) , or the number of times attribute a is predicted on images where the task is predicted as t , which allows them to measure bias amplification in the absence of any additional annotations on the hold-out set . Note that in this formulation they are assuming that the model is making predictions for both the task and the attribute . The full bias amplification metric ( reformulated in our terms ) , is computed as BiasAmpMALS = 1 |T | |T |∑ t=1 |A|∑ a=1 1 ( P ( Aa = 1|Tt = 1 ) > 1 |A| ) ︸ ︷︷ ︸ y ( t , a ) ( P ( Âa = 1|T̂t = 1 ) − P ( Aa = 1|Tt = 1 ) ) ︸ ︷︷ ︸ ∆ ( t , a ) ( 1 ) Fig . 2 empirically demonstrates that this metric is able to capture the level of increasing bias amplification . ( For consistency in comparison with prior metrics , we assume the model always correctly predicts the protected attribute A . ) However , as we discuss in the next section , there are some properties of bias amplification that this metric is not able to capture : for example , it does not distinguish between errors in predicting the protected attribute versus errors in predicting the task . Thus we introduce a new metric ( last graph of Fig . 2 ) which maintains the desirable properties from Zhao et al . ( 2017 ) while including a number of innovations .

Summary: The paper builds on the "bias amplification" aspect of fairness in machine learning literature i.e. the tendency of models to make predictions that are biased in a way that they amplify societal correlations. The paper claims three major contributions: a metric, discussion about the dependence of bias measurements on randomness, and a normative discussion about the use of bias amplification ideas in different domains.