A Technical and Normative Investigation of Social Bias Amplification
Authors: Angelina Wang
Olga Russakovsky
Venue:  (2021)
Keywords: bias amplification
fairness
societal considerations

Abstract: The conversation around the fairness of machine learning models is growing and evolving. In this work, we focus on the issue of bias amplification: the tendency of models trained from data containing social biases to further amplify these biases. This problem is brought about by the algorithm, on top of the level of bias already present in the data. We make two main contributions regarding its measurement. First, building off of Zhao et al. (2017), we introduce and analyze a new, decoupled metric for measuring bias amplification, $\text{BiasAmp}_{\rightarrow}, which possesses a number of attractive properties, including the ability to pinpoint the cause of bias amplification. Second, we thoroughly analyze and discuss the normative implications of this metric. We provide suggestions about its measurement by cautioning against predicting sensitive attributes, encouraging the use of confidence intervals due to fluctuations in the fairness of models across runs, and discussing what bias amplification means in the context of domains where labels either don't exist at test time or correspond to uncertain future events. Throughout this paper, we work to provide a deeply interrogative look at the technical measurement of bias amplification, guided by our normative ideas of what we want it to encompass.

Under review as a conference paper at ICLR 2021
A TECHNICAL AND NORMATIVE INVESTIGATION OF SOCIAL BIAS AMPLIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
The conversation around the fairness of machine learning models is growing and evolving. In this work, we focus on the issue of bias amplification: the tendency of models trained from data containing social biases to further amplify these bi- ases. This problem is brought about by the algorithm, on top of the level of bias already present in the data. We make two main contributions regarding its mea- surement. First, building off of et al. (2017), we introduce and analyze new, decoupled metric for measuring bias amplification, BiasAmp which possesses a number of attractive properties, including the ability to pinpoint the cause of bias amplification. Second, we thoroughly analyze and discuss the normative implica- tions of this metric. We provide suggestions about its measurement by cautioning against predicting sensitive attributes, encouraging the use of confidence inter- vals due to fluctuations in the fairness of models across runs, and discussing what bias amplification means in the context of domains where labels either don't exist at test time or correspond to uncertain future events. Throughout this paper, we work to provide a deeply interrogative look at the technical measurement of bias amplification, guided by normative ideas of what we want it to encompass.
INTRODUCTION
The machine learning community is becoming increasingly cognizant of problems surrounding fair- ness and bias, and correspondingly plethora of new algorithms and metrics are being proposed (see e.g., Mehrabi M.g.. et al. (2019) for a review). The gatekeepers checking the systems to be deployed often take the form of fairness evaluation metrics, and it is vital that these be deeply investigated both technically and normatively. In this paper, we endeavor to do this for bias amplification. Bias amplification happens when a model exacerbates biases from the training data test time. It is the result of the algorithm (Foulds et al.) 2018), and unlike other forms of bias, cannot be solely attributed to the dataset.
To this end, we propose a new way of measuring bias amplification, BiasAmp that builds off a prior metric from Men Also Like Shopping (Zhao al.) 2017). that we will call BiasAmp Our metric's technical composition aligns with the real-world qualities we want it to encompass, addressing a number of the previous metric's shortcomings by being able to: 1) generalize beyond binary attributes, 2) take into account the base rates that people of each attribute appear, and 3) disentangle the directions of amplification. Concretely, consider a visual dataset (Fig. []] where each image has a label for the task, T. which is painting not painting, and further is associated with protected attribute, A, which is woman or man. If the gender of the person biases the prediction of the task, we consider this A + T bias amplification; if the reverse happens, then T A.
In our normative discussion, we discuss a few topics. We consider whether predicting protected attributes is necessary in the first place: by not doing so, can trivially remove T A amplifica- tion. We also encourage the use of confidence intervals when using our metric because BiasAmp.., along with other fairness metrics, suffers from the Rashomon Effect (Breiman 2001). or multiplicity of good models. In deep neural networks, random seeds have relatively little impact on accuracy; however, that not the case for fairness, which is more brittle randomness.
'The arrow in BiasAmp_, is meant to signify the direction that bias amplification is flowing, and not in- tended to be a claim about causality.

---

Under review as a conference paper at ICLR 2021
Dataset
Ground truth
(peinting, woman)
(painting, man)
(not painting, woman)
(not painting, man)
Prediction Errors
Task wrong
(not painting. --)
(not painting....
--)
(painting.
--)
(painting, --)
Blas Amp
T-+A Amp
Amp
Aliribule wrong
(->,, man)
(- womann
(->,, man)
(-.,, woman)
Figure I: Consider an image recognition dataset where the goal to classify the task, T, as painting or not painting, and the attribute, A, as woman or man. Note that in this dataset women are correlated with painting, and men with not painting. In this work we are particularly concerned with errors that contribute to the amplification of bias (red and yellow in the figure), i.e.. those that amplify the training correlation. We further disentangle these errors into those that amplify the attribute to task correlation (i.e., incorrectly predict the task based on the attribute of the person; shown in yellow) versus those that in contrast amplify the task to attribute correlation (shown red).
Notably, a trait of bias amplification is that it is not at odds with accuracy, unlike many other fair- ness metrics, because the goal of not amplifying biases and matching task-attribute correlations is aligned with that of accurate predictions. For example, imagine dataset where the positive outcome is associated at a higher rate with group A than with group B. A classifier that achieves 100% ac- curacy at predicting the positive outcome not amplifying bias; however, according to metrics like demographic parity, this perfect classifier is still perpetuating bias because is predicting the posi- tive label at different rates for both groups. While matching training correlations is desired in object detection where systems should perfectly predict the labels, we will explore the nuances of what this means in situations where the validity of the labels, and thus aattibtte correlations themselves, are up for debate. For example, in the risk prediction task which assesses someone's 's likelihood of recidivism, the label represents whether someone with a set of input features ended up recidivating. but is not steadfast indicator of what another person with the same input features will do. Here, we would not want to replicate the task-attri correlations at test time, and it is important to keep this in mind when deciding what fairness metrics to apply. The notion of amplification also allows to encapsulate the idea that systemic harms and biases can be more harmful than errors made without such history (Bearman et al. 2009); for example, in images overclassifying women cooking car- ries more of a negative connotation than overclassifying men as cooking 2 Distinguishing between which errors are more harmful than others a pattern that can often be lifted from the training data.
To ground our work, we first distinguish what bias amplification captures that standard fairness metrics cannot, and then distinguish BiasAmp from BiasAmPMALS: Our key contributions are: 1) proposing a new way to measure bias amplification, addressing multiple shortcomings of prior work and allowing us to better diagnose where model goes wrong, and 2) providing a technical analysis and normative discussion around the use of this measure in diverse settings, encouraging thoughtfulness with each application.
RELATED WORK
Fairness Measurements. Fairness is nebulous and context-dependent, and approaches to quanti- fying it (Verma & Rubin| 2018: Buolamwini & Gebru 20188) include equalized odds (Hardt et al. 2016 equal opportunity (Hardt al. demographic parity (Dwork al. 2012 Kusner al.
2017 fairesss through awareness (Dwork et al. 2012 Kusner et al. (017). fairness through un-
awareness (Grgic-Hlase Hlacc et al.) [016]] Kusner et 2017). and treatment equality (Berk al.) (017)) We examine bias amplification, which is a type of group fairness where correlations are amplified.
Bias Amplification. Bias amplification has been measured by looking at binary classifica- tions (Leino et al.) 2019). GANs (Jain et al. 2020 Choi et al.) 2020), and correlations (Zhao et al.) 2017 Wang et al. (2019) measures this using dataset leakage and model leakage. The difference between these values is the level of bias amplification, but this is not fair comparison because the
We use the terms man and woman refer to binarized pocially-pereeing gender expression, recognizing these labels inclusive, and in vision datasets often assigned by annotators rather than disl-dis
2

---

Under review as a conference paper at ICLR 2021
attribute classifier gets discrete labels for the former but continuous model outputs for the latter. Jia jet al. (2200) looks at output distributions like we do, but with a different formulation.
The Word Embedding Association Test (WEAT) (Caliskan et al. 2017) measures bias amplification in de-contcxtualized word embeddings, looking at correlations but not causations (Bolukbasi et al. 2016). However, with newer models like BERT and ELMo that have contextualized embeddings, WEAT does not work (My et al.] 2019)) so new techniques have been proposed incorporating context (Lu al., 2019 Kuang & Davison 2016). We use these models to measure the directional aspect of these amplifications, as well as to situate them in the broader world of bias amplification.
Directionality. Directionality of amplification has been observed in computer vision (Stock & Cisse 2018 and language (Qian ct al. 2019). It has also been studied with causality (Bhattacharya & Vogi, 2007 Wooldridge| 2016 Pearl 2010 Middleton et al. 2016; Steiner & Yongnam 2016). We take
a deeper and more empirical approach.
Predictive Multiplicity. The Rashomon Effect (Breiman 2001), or multiplicity of good models, has been studied in various contexts. The variables investigated that differ across good models include explanations (Hancox-Lii (020). individual treatments (Marx et al. 2020; PPaellzyyk et al. 2020), and variable importance (Fisher et al. 2019 Dong & Rudin (019). We build on these discoveries and investigate how fairness measurements also differ between equally "good" models.
3 EXISTING FAIRNESS METRICS
In this section we present existing fairness metrics and show how bias amplification can distinguish errors resulting from under- and overelassification a way that others cannot, followed by a discus- sion of the shortcomings of BiasAMPMALS:
3.1 OVERVIEW OF EXISTING FAIRNESS METRICS
We begin with review of existing fairness metrics in a concrete classification setting. We consider again the example from Fig. [ where on this dataset women (ug) are correlated with painting (la), and men (a1) ) with not painting (41), such that there are N images each of (ao, 40) and (a1, (1) but only N/2 images of (ag. t1) and (a1, to). A classifier trained to recognize painting on this data is likely to Iearn this association and over-predict painting images of women and under- predict painting on images of men; however, algorithmic interventions may counteract this effect and in fact result in the opposite behavior.
Fig. [ shows the behavior of fairesss metrics under varying amounts of learned amplification of the correlation. The four faimess metrics are: False Positive Rate (FPR) and True Positive Rate (TPR) difference: the difference in false positive (true positive) rate of predicting the label lo on images of a versus images of ag (Chouldechoval 2016 Hardt al., 2016), accuracy difference: difference between the overall task prediction accuracy on images of 0₁ versus on images of ao (Berk) al., (017). and mean accuracy across subgroups: mean task prediction accuracy across the four image subgroups ((o,, fo), (a1, to). (ao,1) 11), and (a1, f1)) (Buolamwini & Gebru 2018). However, these metrics are not designed to account for the training correlations, and are unable to distinguish between cases of increased or decreased learned correlations, seen in Fig.
Zhao et al. (017) introduced an alternative to these that explicitly captures the notion of bias am- plification. Concretely, they consider Р(Λ = a|7 L) of the training data as the fraction of times protected attribute a E AA appears on images corresponding to task E T. They then compare this with the test-time predictions made by the model, P(A = a = f), or the number of times attribute is predicted on images where the task is predicted as t, which allows them to measure bias amplifi- cation in the absence of any additional annotations on the hold-out set. Note that in this formulation they are assuming that the model is making predictions for both the task and the attribute. The full bias amplification metric (reformulated in our terms), is computed as BiasAmp
ITI
LA
P(A. = 1 = 1)
P(Á. = 177, 1) 1
(1)
A
(tr.a)
3

---

Under review as a conference paper at ICLR 2021
Difference
TPIL Differncce
Ace
BinsABPALL
BiasAmpa--
0.0
10
0.07
1.1
0.17
0.33
0.67
0.92
0.83
00.25
3.0
0.0
0.75
Different fairness metrics vary in how they respond to model errors. In our image dataset (Fig. [ of predicting someone who is a woman or man be painting or not, consider classifier that always predicts the task correctly for men, but varies for women. The x-axes of the graphs correspond to the fraction of women predicted be painting, where the ground-truth is 2/3, and the model does not predict false positives before this point. The first two metrics, FPR and TPR differ- ence, only capture one of under- or overelassification. The next two metrics are symmetric around 2/3, being unable to differentiate under- or overclassification Both bias amplification metrics are able to distinguish between under- and overelassification.
Figure 2:
Fig. 2 Tempiricall demonstrates that this metric is able to capture the level of increasing bias ampli- fication. (For consistency in comparison with prior metrics, assume the model always correctly predicts the protected attribute A.) However, as we discuss in the next section, there are some prop- erties of bias amplification that this metric is not able to capture: for example, it does not distinguish between errors in predicting the protected attribute versus errors in predicting the task. Thus we introduce new metric (last graph of Fig. 2 which maintains the desirable properties from Zhao et al. 2017) while including number of innovations.
3.2 SHORTCOMINGS OF BIASAMPMALS
Despite the advantages we just documented about BiasAmpyars (Eqn. I and its ability to distin- guish under- and overclassifications of training correlations, this metric also suffers from a number of shortcomings. To ground our discussion, we will work directly with the model outputs released by IZhao et al. (2017) from their Conditional Random Field (CRF) model on COCO (Lin et al., 2014), which has predictions for gender and objects detected for each image.
3.2.1 NON-BINARY ATTRIBUTES
The first shortcoming is that the metric assumes that the protected attributes are binary, limiting its use: the indicator variable (t, a) implicitly chooses only with every task t. Consider a task to E T such that au E A is associated with it, but none of the other a, A are, where # 0. In this scenario, diff(to, a;) is only considered when there is one other a, such that 0; = -a since diff(t, a) - diff(t, -a). A simple addition of ddiff for all a, 's when is 0 ensures that when there are more than two groups, their bias amplification is also counted.
of the attribute E A to be associated
3.2.2 BASE RATES
The second shortcoming of BiasAmpMALL is the fact that the metric does not take into account the base rates of each attribute. Concretely, when determining in y(t, a) of Equ. whether the attribute a is correlated with the task t, P(A = a|T = t) is compared to [AT However, this assumes that all a's within A are evenly distributed, which may not be the case. For example, in COCO there are about 2.5x as many men as women, so it would appear that most objects positively correlate with men simply by nature of there being an overrepresentation of men. Consider the object oven; BiasAmpMALS calculates P(A = man|7 to be correlated with men rather than women. However, computing P(A man, T oven)= 0.0103 < 0.0129 = P(A = P(T = oven) reveals that men are in fact not correlated with oven, and the sceming overrepresentation comes from the fact that men are overrepresented in the dataset more generally. Not surprisingly, the model trained on this data learns to associate women with ovens and underpredicts men with ovens at test time, i.c., P(Å = man i = oven) P(A man T oven) = 0..00. BiasAmp erroneously counts this inverse bias amplification.
over) = 0.56 > Y and thus considers this object
4

---

Under review as a conference paper at ICLR 2021
3.2.3 ENTANGLING DIRECTIONS
Another shortcoming we observe the inability to distinguish between different types of bias am- plification. Zhao et al. (2017) discovers that "Technology oriented categories initially biased toward men such as keyboard ... have each increased their bias toward males by over 0.100." Concretely, from Eqn. P(A m keyboard)
.70 and P(Á = man ' = keybooard) = .83, demonstrating an amplification of bias. However, the direction or cause of bias amplification re- mains unclear: is the presence of man in the image increasing the probability of predicting a key- board, or vice versa? Looking more closely at the model's 's disentangled predictions, we see that:
P(T = keyboard| = man) = 0.0020 < 0.0032 = P(T = keybcard|4 = man) (2)
P(Á = man keyboard) 0.78 > 0.70 P(A = man keyboard)
indicating that keyboards images with keyboards. Thus the root cause of this amplification appears to be in the gender pre- dictor rather than the object detector. Such disentangement allows to properly focus algorithmic intervention efforts. This also highlights the need to consider the ground truth labels on the hold-out set when measuring bias amplification in addition the predictions (since when considering only the predictions, it is impossible to decouple the different sources of bias).
under-predicter
images with men yet men over-predicted on
3.3 THRESHOLD
We also fully replicate the original experiment from Zhao al. (2017) using BiasAmpMALs on their model predictions and measure .040. However, we observe that "man" is being predicted at a higher rate (75.6%) than is actually present (71.2%). With this insight, we tune the decision threshold on the validation set such that the gender predictor is well-calibrated to be predicting the same percentage of images to have men as the dataset actually has. When we calculate AiasAL on these newly thresholded predictions for the test set, we see bias amplification drop from 0.040 to 0.001 just as a result of this threshold change, outperforming even the solution proposed in Zhao et al. (2017) of corpus-level constraints, which achieved a drop only 0.021. Faireess can be quite sensitive the threshold chosen (Chen & Wu 2020), so careful selection should be done in picking the threshold, rather than using the default of 55. In Fig. 3 we show how the amount of bias amplification, as measured by BiasAmpMALS and Bias changes as we vary the threshold, i.c., proportion of people classified to man. We can see that when the threshold is chosen to be the one well- calibrated on the validation set rather than the default threshold, bias amplification is measured to be closer to zero for both metrics. From here on out when a threshold is needed, we will pick it to be well-calibrated on the validation set. Although we do not take this approach, one could also imagine integrating bias amplification across proportion in order to have a threshold-agnostic measure of bias amplification, similar to what is proposed by Chen & Wu (2020). We do not do this in our experiments because at deployment time, it is often the case that discrete predictions are required.
BIASAMP
Now we present our metric, BiasAmp_,, which retains the desirable properties of BiasAmp while addressing the shortcomings noted in the previous section. To account for the need to disen- tangle the two possible directions of bias amplification (Sec. 3.2.3 the metric consists of two values, BiasAmp,, A-- corresponding to the amplification of bias resulting from the protected attribute influ- encing the task prediction, and BiasAmp, corresponding the amplification of bias resulting from the task influencing the protected attribute prediction. Concretely, the metric is defined as:
BiasAmp = [AITI
1
Σ
a(t,a)A(t, a) + (1 - a(),,))(-AA((,..
(4)
1ET
where y(t, a) = 1 [P(T = = =) a)> P(T = t) a)]
A,
(5)
At, a) = { P(f = = P = a) measuring A + T
P(Å - a =t ----P(AA
measuring T - A
Eqn. generalizes BiasAmpMALS to measure the amplification of both positive and negative cor- relations between task i and attribute a, depending on y(t, a), when the attributes are non-binary,
5

---

Under review as a conference paper at ICLR 2021
6.2
Image Cuncition
-0.1-
BiasAmp
0.7
0.8
0.9
1.
BiasAmpr
0287 1 0127
.425 .009
.491 .0092
Proportion Cloosited Men
Figure 3: Bias amplification varying as function of pro- portion of images classified to be men (controlled by classifi- cation threshold).
Figure 4: BiasAmp changes as a result of three conditions of images: full, noisy person mask. full person mask. As less of the person is visible, A-xT decreases from less hu- man attribute visual cues to bias the task prediction. T-+A increases because the model must rely on visual cues from the task to predict the attribute.
as discussed in Sec. 3.2.1 Eqn. identifies the direction of correlation of attribute with task t, properly accounting for base rates as described in Sec. 3.2.2 Finally, Eqn. 6 decouples the two possible directions of bias amplification as Sec. 3.2.3 Since values may
negative, reporting the aggregated bias amplification value could obscure task-attribute pairs that exhibit strong bias amplification; thus, disaggregated results per pair can be returned for a more detailed diagnosis.
Although in much of the examples have and will look at, A {woman, man], this for- mulation allows for any attribute set to be defined, including intersectional identities. This is achieved by having A encompass the cross- pross-product of possible attributes, for example A [Black woman, Black man, white woman, white man].
We introduce scenario for validating the decoupling aspect of our metric, that simultancously serves as inspiration for an intervention approach to mitigating bias amplification. We use baseline amplification removal idea of applying segmentation masks (noisy or full) over the people in an image to mitigate bias stemming from human attributes (Wang et al. 2019). We train on the COCO classification task a VGG16 (Simonyan & Zisserman 2014) model pretrained on ImageNet (Rus- sakovsky et al., 2015) to predict objects and gender, with a Binary Cross Entropy Loss over all outputs, and measure BiasAmpr-..A and BiasAmpAA-71 we report 95% confidence intervals for runs of each scenario. In Fig. 4 we see, as expected, that as less of the person is visible, A-T decreases because there are less human attribute visual cues to bias the task prediction. On the other hand, T-AA increases because the model must lean into task biases to predict the person's s attribute. However, we can also see from the overlapping confidence intervals that this technique of bias amplification mitigation does not appear to be particularly robust; continue discussion of this phenomenon Sec. 5.2 Further mitigation techniques are outside of our scope, but we look to
works like Singh et al. (2020): Wang et al. 2019);; Agarwal et al. (2020).
ANALYSIS AND DISCUSSION
We now discuss some of the normative issues surrounding bias amplification, starting Sec. 5.1 with the existence of T-AA bias amplification, which implies the prediction of sensitive attributes; in Sec. 5.2 about the need for confidence intervals to make robust conclusions; and in Sec. 5.3 about scenarios in which the original formulation of bias amplification as desire to match base correlations may not be what is actually wanted.
5.1 T - A BIAS AMPLIRICATION
If we think more deeply about these bias amplifications, we might come to a normative conclusion that T > A, which measures sensitive attribute predictions conditioned on the tasks, should not exist the first place. There are very few situations which predicting sensitive attributes makes sense (Scheuerman et al. 2020 Larson 2017), we should carefully consider if this is strictly necessary for target applications. For the image domains discussed, by simply removing the notion of predicting gender, we trivially remove all T A bias amplification. In similar vein, there has been great work done on reducing gender bias in image captions (Hendricks et al., 2018 Tang et al. 2020), but it is often focused on targeting T - A amplification rather than A T. When

---

Under review as a conference paper at ICLR 2021
Baseline:
Baseline: a man riding a motarratte
Baseline: man brushing his teeth With tooth booth boush Equalizer: a vesmnan holding glass wine in her hand
Baseline: and steabani standing the saane Equalizer.
standing front
marict
with a dog on the
bananas Equalbeer woman red dress holding umbreta
back
Equalizer:
waraa sitiig dog
woreen a dokin) standing next to dogg
Figure 5: Illustrative captions from the Equalizer model (Hendricks et al. 2018), which decreases T + A bias amplification in these captions from the Baseline, but inadvertently increases A + T. Green underlined words are correct, and red italicized words are incorrect. In the images shown, the Equalizer improves the Baseline for the gendered word, but also introduces additional biased errors in the captions.
disentangling the directions of bias, we find that the Equalizer model (Hendricks et al. 2018), which was trained with the intention of increasing the quality of gender-specific words in captions, inad- vertently increases A + T bias amplification for certain tasks. In Fig. [ we examples where the content of the Equalizer's caption exhibits bias coming from the person's 's attribute. Even though the Equalizer model reduces T + A bias amplification in these images, it inadvertently increases A + T. It is important to disentangle the two directions of bias and notice that while one direction becoming more fair, another is actually becoming more biased. Although this may not always be the case, depending on the downstream application, perhaps this is a setting in which we could consider simply replacing all instances of gendered words like "man" and "woman" in the captions with "person" to trivially eliminate T + A. and focus on A + T bias amplification. Specifically when gender is the sensitive attribute, Keyes (2018) thoroughly explains how we should carefully think about why we might implement Automatic Gender Recognition (AGR), and avoid doing so.
On the other hand, sensitive attribute labels, ideally from self-disclosure, can be very useful. For example, these labels are necessary to measure A + '"' amplification, which is important to discover, as we do not want our prediction task to be biased for or against people with certain attributes.
5.2 LACK OF CONSISTENCY BIAS MEASUREMENT
Evaluation metrics, ours' included, are specific to each model on each dataset. Under common loss functions such as Cross Entropy Loss, some evaluation metrics, like average precision, are not very sensitive to random seed. However, we noticed that bias amplification, along with other fairness metrics like FPR difference, often fluctuates greatly across runs. Because the loss functions that machine learning practitioners tend to default to using are proxies for accuracy, it makes sense that the various local minima, while equal in accuracy, are not necessarily equal in terms other measurements. The phenomena of differences between equally predictive models has been termed the Rashomon Effect (Breiman 2001), or predictive multiplicity (Marx et al. 2020).
Thus, like previous work (Fisher et al., 2019), urge transparency, and advocate for the inclu- sion of confidence intervals. To illustrate the need for this, we look at the facial image domain of CelebA (Liu et al., 2015), defining the tasks of classifying "big nose" and "young" our T, and treating the gender labels as our attribute, A. Note that we are not going to classify gender, for reasons raised in Sec. 5.I, so we only measure A T' amplification. For these tasks, women are more correlated with no big nose and being young, and men with big nose and not being young. We examine two different scenarios, one where our independent variable is model architecture, and another where it is the ratio between number of images of the majority groups (e.g., young women and not young men) and minority groups (e.g., not young women and young men). By looking at the confidence intervals, we can determine which condition allows us to draw reliable conclusions about the impact of the variable on bias amplification.
For model architecture, we train 3 models pretrained on ImageNet (Russakovsky et al., 2015) across 5 runs: ResNet18 (He et al.) 2016), AlexNet (Krizhevsky et al., 2012). 2). and VGGI6 (Simonyan & Zisserman 2014). Training details are in Appendix A.3 In Fig. 6 we see from the confidence intervals that while model architecture does not result in differing enough of bias amplification to conclude anything about the relative fairesss of these models, across-ratio differences are significant enough to draw conclusions about the impact of this ratio on bias amplification. We encourage researchers to include confidence intervals that findings
robust random fluctuations.
7

---

Under review as a conference paper at ICLR 2021
Average Pretboon
0.922
5We beferrece
0.523
1
Bisskmp
D003
0.50
Dassmo,,..
0.06
0.04
1
0.900
0.43
0250
0.00
Acchett Reshiet18 VO016
Aexiet Resiet11
VS016
Aleshitt Reshvet2 V6616
1.5
1.75
2.0
2.25
2.5
Modal Archeecture
Model Archnoccture
Model Architecture
Young
Majority to Minority Groups Rato
Heerage Fredsom
0.24
FPR Oitrrreeeeee
0.04
0.83
-0.10
Blasmoo
0.82
0.22
-0.11
0.020
0.81
0.00
Aeccltt
Reshvet8
VG616
Aeccett Rechee18 VC616
Aleshutt Reshee18 VCC16
1.5
1.75
3.0
2:25
2.5
Model Archrlecture
Model Architecture
Model fecture
Majority to Stmorrty Groups Rabio
Figure 6: We investigate the consistency of various metrics by looking at 95% confidence intervals as we manipulate two independent variables: model architecture (left three graphs), and majority to minority groups ratio (right graph). The top row is for the attribute of "big nose", and bottom row for "young." " For model architecture, across 5 runs, the accuracy measure of average precision retains consistent ranking across models, but two different fairness measures (FPR difference and T bias amplification) have overlapping intervals. This does not allow us to draw conclusions about the differing fairness of these models. However, across-ratio differences in bias amplification are significant enough to allow us to draw conclusions about the differing levels of fairness.
Base Correlation Source
BiasAmp
BiasAmp
Uniform (2 pronouns)
.1875
.0156
Uniform (3 pronouns)
.1250
.0104
Wikipedia (2 pronouns)
.2457
.0088
2016 U.S. Labor Force (WinoBias) (2 pronouns)
-00747
-0001
Table I: BiasAmp_ for different base correlation sources.
5.3 CONSIDERATIONS FOR UNCERTAIN PREDICTION PROBLEMS
A property of bias amplification is that it does not conflict with having perfect accuracy. However, in turn, such model with perfect accuracy would exactly replicate the correlations present in the training data. In this section we will discuss two cases that challenge this desire for matching training correlations: in the first, there will be no ground-truth labels from which to lift these correlations, and in the second, our goal is actually to not match the training correlations.
No Ground Truth. When we don't 't have labels to derive training correlation rates from, bias am- plification becomes harder to measure. Consider the fill-in-the-blank NLP task, where there is no ground-truth for how to fill in a sentence. Given "The [blank] went on a walk", variety of words could be equally suitable. Therefore, to measure bias amplification in this setting, we can manually set the base correlations, c.g., P(T = f = a), P(A = a/T = t). To see the effect that adjust- ing base correlations has, we test the bias amplification between occupations and gender pronouns, conditioning on the pronoun and filling in the occupation and vice versa. In Table T we report measured bias amplification results the FitBERT (Fill in the blanks BERT) (Havens & Stal, 2019 Devlin et al. 2019) model using various sources as our base correlation of bias from which used for each set of pronouns, and
amplification is measured. The same outputs from the model the independent variable we manipulate is the source of: 1) equality amongst the pronouns (using 2 and 3 pronouns), 2) co-occurrence counts from English Wikipedia (one of the datasets BERT was trained on). and 3) WinoBias (Zhao et al.) 2018) with additional information supplemented from the 2016 U.S. Labor Force Statistics data. Details are Appendix A.4 It is interesting note that relative to U.S. Labor Force data on these particular occupations, FIRBETT actually exhibits no bias amplification. For the occupation of hairdresser, the Labor statistics are biased at 92% women while FuBERT is at 80%, reflecting in fact reduction in bias amplification. This demonstrates the importance setting appropriate base correlations, because picking one that already exhibits strong amounts of bias will only flag models that further amplify this. In the next section discuss another manifestation of this phenomenon, where the training correlation itself would be misleading to compare to, because of the strong amount of bias it contains.
8

---

Under review as a conference paper at ICLR 2021
Future Outcome Prediction. Next, we examine the risk prediction setting, where matching the base correlations may not be our desired goal. The labels here do not represent an objective ground truth because they: 1) suffer from problems like historical and selection bias (Suresh & Guttag 2019] Olteanu et al. 2019) Creen, 2020), and 2) will be used for prediction of a future event for which no one knows the answer. We will put aside the significant problems stemming from the first point for this current discussion, and focus on the latter. In this section, when using the word "predict" we will revert from the machine learning meaning and adopt the colloquial sense of the word, as in the forecast of future event. What we had called object prediction, we will now call object labeling.
Consider bias amplification in the context of domain like risk prediction relative to previous do- mains looked at, such as object detection. The difference is not in tabular versus vision, but rather in prediction versus labeling. The notion of "ground-truth" doesn't quite exist the way we might think about it, because given the input features that define particular person, one could imagine an individual with these features who does recidivate, and one who does not (Barocas et al. 2019). The training label is just how someone with these input features once acted, but is not necessarily rigid indicator of how someone else with these features will act. On the other hand, in object label- ing the labels are very much the -oound-truth,, and thus bias amplification is reasonable metric to gauge fairess by. However, we do not know this for risk prediction, and thus, matching the training correlations should not be our intended goal (Wick al. 2019). In Fig. 7 we show the metrics of BiasAmp A-T and FPR difference measured on COMPAS pre- dictions (Angwin al.. 2016) at different thresholds, only looking at two racial groups we will call and 2. It might be surprising see that at certain thresholds, bias amplification is 0 or negative. What this means is that Group 1, which is more correlated with the target label than Group 2, has a (predicted rate- base rate) value that is smaller than Group 2's. If we were looking at bias amplification as way to measure fairness in this domain, this is not a situation where our metric is misreporting, but rather a misspecification of what we are looking for. Like every other fairness metric, ours only captures one perspective, which is that of not amplifying already present biases. It does not require a correction for these biases. As from the continued existence of difference in FPR's between the two groups, there is no threshold that could be picked at which this disparity does not exist (except at the trivial cases of a 10 threshold), even if there does exist thresholds where bias is not being amplified.
Risk Prediction by COMPAS
0.2
0.0
Threshold
Figure 7: How the fair- ness metrics of BiasAmp A-T and FPR difference vary we change the threshold for COMPAS's risk prediction.
For each application, different metrics of faimess are more or less applicable, and Bias is no different. is crucial that we think thoughtfully when deciding how evaluate a model. In previous applications we imagined the direction of systemic bias to be captured by the training data and thus we lifted base correlations of bias from there. However, one could imagine a similarly manual intervention as was done for NLP on other tasks like risk prediction, where domain expert verifies the direction of bias determined by the training set, and even sets the base correlations.
CONCLUSION
In this paper, we take a deep dive into the measure of bias amplification. We introduce a new metric, BiasAmp_,, and through the use of this metric and its directional components, diagnosing models discuss normative considerations, such as
will provide more actionable insights. Additionally, thinking carefully about why we might be performing sensitive attribute prediction, incorporating confidence intervals the norm when reporting fairness metrics, and exercising care when deter- mining which fairness metrics are applicable, and what assumptions they are encoding.
REFERENCES
Vedika Agarwal, Rakshith Shetty, and Mario Fritz. Towards causal VQA: Revealing and reducing spurious correlations by invariant and covariant semantic editing. Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. Propublica, 2016.

---

Under review as a conference paper at ICLR 2021
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning. fairml- book.org, 2019. http::/wwwe fai www.rol..bo...oommm
Steve Bearman, Neill Korobov, and Avril Thorne. The fabric of internalized sexism. Journal of Integrated Social Sciences I(1): 10-47, 2009.
Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. Fairness criminal justice risk assessments: The state of the art. Sociological Methods and Research, 2017.
Jay Bhattacharya and William B. Vogt. Do instrumental variables belong in propensity scores? NBER Technical Working Papers 0343, National Bureau of Economic Research, Inc., 2007.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in Neural Information Processing Systems (NeurIPS), 2016.
Leo Breiman. Statistical modeling: The two cultures. Statistical Science, 16:199-231, 2001.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities commer- cial gender classification. Proceedings of Machine Learning Research, 81, 2018.
Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 2017.
Mingliang Chen and Min Wu. Towards threshold invariant fair classification. Conference on Un-
certainty in Artificial Intelligence (UAI), 2020.
Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon. Fair generative modeling 2020.
via weak supervision.
Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instrument. Big Data, 2016.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. Annual Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HILT), 2019.
Jiayun Dong and Cynthia Rudin. Variable importance clouds: A way to explore variable importance for the set of good models. arXiv: 2019.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. Proceedings of the 3rd Innovations in Theoretical Computer Science Con- ference, 2012.
Aaron Fisher, Cynthia Rudin, and Francesca Dominici. All models are wrong, but many are useful: Learning a variable's importance by studying an entire class prediction models simultaneously. Journal of Machine Learning Research, 20, 2019.
James Foulds, Rashidul Islam, Kamrun Naher Keya, and Shimei Pan. An intersectional definition of fairness. 1rXiv:1807.0883 08362, 2018.
Ben Green. The false promise of risk assessments: Epistemic reform and the limits of fairness. ACM Conference on Fairness, Accountability, and Transparency (ACM FAceT), 2020.
Nina Grgic-Hlaca, Muhammad Bilal Zafar, Krishna Gummadi, and Adrian Weller. The case for process fairness in learning: Feature selection for fair decision making. NeurIPS Symposium Machine Learning and the Law, 2016.
Leif Hancox-Li. Robustness in machine learning explanations: Does it matter? Conference on Fairness, Accountability, and Transparency (FAccT), 2020.
Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. urXiv:16100 1610.02413, 2016.
10

---

Under review as a conference paper at ICLR 2021
Sam Havens and Aneta Stal. Use BERT to fill in the blanks, 2019. URL httts://githhb:
comm http://odbbacoeee/iiiiee fitberrt
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. European Conference on Computer Vision (ECCV), 2016.
Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Overcoming bias in captioning models. European Conference on Computer Vision (ECCV), 2018.
Niharika Jain, Alberto Olmo, Sailik Sengupta, Lydia Manikonda, and Subbarao Kambhampati. Im- perfect imaganation: Implications of gans exacerbating biases on facial data augmentation and snapchat selfie lenses. arXiv:2001.09528, 2020.
Shengyu Jia, Tao Meng, Jieyu Zhao, and Kai-Wei Chang. Mitigating gender bias amplification in distribution by posterior regularization. Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
Os Keyes. The misgendering machines: Trans/HCI implications of automatic gender recognition. Proceedings of the ACM on Sliman-Computn Interaction, 2018.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. Advances in Neural Information Processing Systems (NeurIPS), pp- 1097-1105, 2012.
Sicong Kuang and B. Davison. Semantic and context-aware linguistic model for bias detection. Proc. of the Natural Language Processing meets Journalism IJCAI-16 Workshop, 2016.
Matt Kusner, Joshua R. Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Ad- vances in Neural Information Processing Systems (NeurIPS), 2017.
Brian N. Larson. Gender as variable in natural-language processing: Ethical considerations. Pro- ceedings of the First ACL Workshop on Ethics in Natural Language Processing, 2017.
Klas Leino, Emily Black, Matt Fredrikson, Shayak Sen, and Anupam Datta. Feature-wiss bias amplification. International Conference on Learning Representations (ICLR), 2019.
Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis- Philippe Morency. Towards debiasing sentence representations. Annual Meeting of the Associa- tion for Computational Linguistics (ACL), 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongic, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollar. Microsoft COCO: Common objects in context. European Conference on Computer Vision (ECCV), 2014.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender bias in neural natural language processing. 1807 2019.
Charles Marx, Flavio du Pin Calmon, and Berk Ustun. Predictive multiplicity in elassification. arXiv:1909. 1ttri9999.6677,, 2020.
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. On measur- ing social biases in sentence encoders. Anmual Conference of the North American Chapter of the Association for Computational Linguistics (NACCL), 2019.
Ninarch Mchrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. arXiv:1908.00 09635, 2019.
Joel A. Middleton, Mare A. Scott, Ronli Diakow, and Jennifer Hill. Bias amplification and bias unmasking. Political Analysis, 3:307-323, 2016.

---

Under review as a conference paper at ICLR 2021
U.S. Bureau of Labor Statistics. Employed persons by detailed occupation, sex, race, and hispanic or latino ethmicity. 2016. URL https: www.blw.....o..wwwww.wwwwww...w. gov/ ttwpgg/ww/ppoooo httpp////aaa1/pppaaaala..
pdff
Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Emre Kiciman. Social data: Biases,
methodological pitfalls, and ethical boundaries. Frontiers in Big Data, 2019.
Martin Pawelczyk, Klaus Broelemann, and Gjergji Kasneci. On counterfactual explanations under predictive multiplicity. Conference on Uncertainty in Artificial Intelligence (UAI), 2020.
Judea Pearl. On a class of bias-amplifying variables that endanger effect estimates. Uncertainty in Artificial Intelligence, 2010.
Yusu Qian, Urwa Muaz, Ben Zhang, and Jae Won Hyun. Reducing gender bias in word-level language models with gender-equalizing loss function. ACL-SRW, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJV), 115(3):211225 252, 2015. doi: 10. 1007// 1263-015-0816-y.
Morgan Klaus Scheuerman, Kandrea Wade, Caitlin Lustig, and Jed R. Brubaker. How we' 'vee taught algorithms to see identity: Constructing race and gender in image databases for facial analysis. Proceedings of the ACM on Human- Human-Computer Interaction, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. prXiv:1409) 1556, 2014.
Krishna Kumar Singh, Dhruv Mahajan, Kristen Grauman, Yong Jae Lee, Matt Feiszli, and Deepti Ghadiyaram. Don't judge an object by its context: Learning overcome contextual bias. Con- ference Computer Vision and Pattern Recognition (CVPR), 2020.
Peter M. Steiner and Kim Yongnam. The mechanics of omitted variable bias: Bias amplification and cancellation of offsetting biases. Journal of causal inference, 4, 2016.
Pierre Stock and Moustapha Cisse. ConvNets and ImageNet beyond accuracy: Understanding mis- takes and uncovering biases. European Conference on Computer Vision (ECCV), 2018.
Harini Suresh and John V. Guttag. A framework for understanding unintended consequences of 10002, 2019.
machine learning.
Ruixiang Tang, Mengnan Du, Yuening Li, Zirui Liu, and Xia Hu. Mitigating gender bias in caption- ing systems. (rXXiv20066083/55 2020.
Sahil Verma and Julia Rubin. Fairness definitions explained. ACM/IEEE International Workshop on Software Fairness, 2018.
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. Interna- tional Conference on Computer Vision (ICCV), 2019.
Michael Wick, Swetasudha Panda, and Jean-Baptiste Tristan. Unlocking fairness: a trade-off revis- ited. Conference Neural Information Processing Systems (NeurIPS), 2019.
Jeffrey M. Wooldridge. Should instrumental variables be used as matching variables? Research in Economics, 70:232-237, 2016.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. Conference on Empirical Methods Natural Language Processing (EMNIP)), 2017.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. North American Chapter of the Asso- ciation for Computational Linguistics (NAACL), 2018
12

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1 ADDITIONAL. METRIC DETAILS
We provide additional details here about BiasAmp_,, as defined in Sec.
In practice the indicator variable, y(t, a), is computed over the statistics of the training set, whereas everything else is computed over the test set. The reason behind this, is that the direction of bias is determined by the existing biases in the training set. Additionally, for computing the integra- tion across all thresholds, we sparsely sample from all of the probabilities outputted to compute an approximation.
Comparisons of the values outputted by BiasAmp_ should only be done relatively. In particular, within one of the directions time, either A + T or T + A, on one dataset. In particular, comparing + to + directly is not a signal as to which direction of amplification is stronger.
A.2 WALKING THROUGH THE EQUATIONS FOR FIGURE 1 SCENARIO
In this section we concretely write out the equations for BiasAmpyALs and BiasAmp for the scenario shown in Fig. [ to better clarify what each metric captures. As reminder, in this scenario A = (a,, } and T = to. t }. and ag is correlated with ty, and 01 with 11
BiasAmp MALS
(7)
2
= 1[]; 1) =12; P = 1/, = |
111;=11) 1))
(8)
2 t=1 a=1
1
x (P(Á)- = 11 = 1) - P(1. = 1/T0 = 11) +
2
0x (P(Á) 11 = 1) - P(A1 1/7 = 11)
(10)
0x
(P(Ao 1/71 = 1) - P(A. 1 1))
(11)
1x (P(A, 1 = 1) - P(A1 1 11)
(12)
2 (((A- 1/7 = 1) - P(A, 1/76 11)
1
(13)
+ (P(A, 1/71 = 1) - P(A1 1 11)
(14)
For BiasAmp_, our equation simplifies in the case of discrete predictions as follows:
1
BiasAmp, A-+7
(P(To 1/4, 1) P(T, 1/4, =))
(15)
4
(P(T) 1|A =1) P(To 1/4, =
(16)
(P(T) 1|Ao 1) P(Zi A =1)
(17)
(P(2) 1/A1 1) P( 1A = 1)
(18)
(19)
13

---

Under review as a conference paper at ICLR 2021
1[ 4
BiasAmpr-+A
(P(Ao 1/T 1) 1[ 1)
(20)
(P(Ag- 1 =1 P(A. 1[; 1)
(21)
(P(Á: =17 =1) P(A1 = 1/7 =1)
(22)
+ (P(Á) 1/7 =) 1) P(A1 = 1 = 1)
(23)
(24)
A.3 DETAILS AND EXPERIMENT FROM LACK OF CONSISTENCY IN BIAS
For the models we trained in Sec. 5.2
performed hyperparameter tuning on the validation set,
and ended up using the following: ResNet18 had a learning rate .0001, AlexNet of .0003, and VGG16 of .00014. All models were trained with stochastic gradient descent, batch size of 64, and 10 epochs.
A.4 DETAILS ON MEASURING BIAS AMPLIFICATION IN FITBERT
Here we provide additional details behind the numbers presented in Tbl. iin Sec. 5.3
As noted, and done, by Liang et al. 2020),, large and diverse corpus of sentences is needed to sam- ple from the large variety of contexts. However, that out of scope for this work, where we simply use 2 sentences: "[he/she/(they)] is a(n) [occupation]" or "[he/she/(they)) was a(n) [occupation]" to test.
When calculating the amount of bias amplification when the base rates are equal, we picked the direction of bias based on that provided by the WinoBias dataset. practice, this can be thought of as setting the base correlation, P(4 a = for a men-biased job like "cook" to be .5 + e for "he" and .5 e for "she" when there are two pronouns, and .33 + e for "he" and .33 e for "she" and "they". where in practice used 1e-7. This ensures that the indicator variable, y(t, a) from Eq. 55) is set in the direction fo the gender bias, but the magnitudes of (tt, a) from Eq. 6 are not affected to significant degree.
To generate a rough approximation of what training correlation rates could look like this domain, look
of the datasets that BERT simply counting the cooccurrences of all the occupations along with gendered words such as "man". "he", "him", etc. There flaws with this approach because in sentence like "She went to the doctor.", the pronoun is in fact not referring to the gender of the person with the occupation. However, we leave more accurate measurement of this future work, as our aim for showing these results was more for demonstrative purposes illustrating the manipulation of the correlation rate, rather than in rigorously measuring the training correlation rate.
trained on, the Wikipedia dataset. We do by
We use 32 rather than 40 occupations in WinoBias Zhao et al. (2018). because when we went to the 2016 U.S. Labor Force Statistics data (of Labor Statistics (016) to collect the actual numbers of each gender and occupation in order to be able to calculate P(T t|4 a). since WinoBias only had P(A a/T' = t). we found 8 occupations to be too ambiguous to be able to determine the actual numbers. For example, for "attendant", there were many different attendant jobs listed, such as "flight attendants" and "parking lot attendant", so we opted rather to drop these jobs from the list of 40. The 8 from the original WinoBias dataset that we ignored are: supervisor, manager, mechanician, CEO, teacher, assistant, clerk, and attendant. The first four are biased towards men, and the latter four towards women, that did not skew the distribution of jobs biased towards each gender.
14


Reviews:
[{"review_id": "tS4um5m3eyK", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes a new metric for quantification of bias amplification that aids in interpretability of models. In particular, the paper offers useful insights concerning the meaning of bias amplification across varying contexts or use cases and prescribes the use of confidence intervals for better validation of various fairness metrics.\n\nPositives:\n\n1. Technical Sophistication\nThe paper throughly examines the shortcomings of a previous metric used for computing bias amplification and proposes a new metric that overcomes the limitation of prior metric while still retaining the desirable properties. In particular, the authors clearly state the benefits of their method as described in Sec 3.4. \n2. Presentation: \nThere is sufficient clarity in the paper facilitating easy comprehension. \n3. Experiments and analysis:\nThis is the biggest strength of the paper. A thorough investigation makes this paper compelling. The authors also discuss the limitations of the proposed metric. The running example (fig 1 ) considered in the paper is interesting, especially given that such stereotypes are not considered often. \n\nConcerns:\n\n1. The recommendation for confidence interval is justifiable, but lacks a proper grounding. \n2.  The authors use causal notations for bias amplifications with the arrows from A to T or vice versa. While it is ok in terms of notation, it raises the question of how causal claims can be justified by merely thresholding  as opposed to interventions . In this context, more elaboration of the metric (Sec 3.3) would be useful.\n\nOverall comments:\n\nI enjoyed reading this paper, and it offered some interesting insights. I believe it will be valuable in analysis of bias across different use cases in a manner that is interpretable.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YVWQ68ElYI0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper talks about an interesting aspect in algorithmic fairness meaning bias amplification; however, there are couple of concerns that I will raise below:\n\n1. The paper mainly addresses shortcomings of a previously mentioned simple concept, bias amplification, brought up in Zhao et al 2017. In this regards, I am not sure about extensiveness of novelty and contribution of this paper along with its technical rigor. One suggestion would be to add some theoretical analysis maybe in Validating the metric session.\n\n2. Although Zhao et al's paper is a famous paper in NLP domain, not much attention has not been given in the pure algorithmic fairness and using it as a measure along with other well known measures, such as statistical parity or EO. This makes me even more concerned about an extension to this work.\n\n3. The discussion on error bars and their need sounds like an intuitive concept which authors spent a section on it. Other sections could have been made richer experimentally instead of this section maybe.\n\n4. Analysis of benchmark fairness datasets remains unexplored. Mostly vision datasets are explored, but authors could have done some studies on famous fairness benchmark datasets as well. This can introduce bias amplification concept to the fairness community and make it more comparable to other well known fairness measures and more acceptable to the community.\n\n5. There are some claims that I do not find accurate in the paper as follows:\n5.1. \"By definition, this problem stems from the algorithms, and can not be attributed to the dataset\" -> we can not 100% say this! The bias in the first place is coming from data itself in this case!\n5.2. \"A trait of bias amplification is that it is not at odds with accuracy, unlike many other metrics ...\" -> where is the proof for this? need strong evidence for this claim.\n\nOverall, this paper addresses shortcomings in a previous work based on a simple bias amplification metric. The lack of technical rigor and theoretical guaranties for some claims makes this paper not a strong candidate. Inclusion of some fairness benchmark datasets can make this paper more strong.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XhbXpSxPOrQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper builds on the \"bias amplification\" aspect of fairness in machine learning literature i.e. the tendency of models to make predictions that are biased in a way that they amplify societal correlations. The paper claims three major contributions: a metric, discussion about the dependence of bias measurements on randomness, and a normative discussion about the use of bias amplification ideas in different domains. \n\nOverall I find the metric as the only major contribution of the paper, and below I will explain why.\n\nThe BiasAmp metric makes a significant contribution in terms of fixing the drawbacks of the previously proposed metric from Zhao et al. 2017(BiasAmp_MALS). It would be more effective if the work also included a study such as Zhao et al demonstrating how to mitigate the bias as measured by the BiasAmp measure. \n\nThe discussion around--the usage of error bars because of the Rashomon effect seems incomplete and almost trivial. First, in my opinion, it is indeed necessary to have error bars if there are metrics whose measurements vary across different runs--regardless of whether that measure was being optimized on or not. But a more nuanced idea would be either: (a) usually a fairness metric is used as a guidance of whether a specific model is deployable or not, rather than being a property of a training method unless however, an ensemble of a number models trained by the same method is being used in deployment. (b) Fairness metrics may be used as a model selection criterion: since the fairness metric is not being optimized upon but a proxy of accuracy, it just means that, out of the models with equally high accuracy, we need to choose the model with the least bias (as measured by a trustworthy metric). \n\nI find the discussion around the 'consistency' of different metrics incomplete, where average precision (AP) seems to rank models consistently while BiasAmp and others don't. BiasAmp is a pretty sophisticated metric and because of the use of conditional probabilities of all kinds prone to pitfalls such as Simpson's paradox when measuring bias. I would be curious if it were robustly tested e.g. another experiment such as Fig 4 where the authors control the amount of the bias by tuning at the source of the bias. \n\nThe discussion around the use of bias amplification in terms of prediction problems where the outcome is chance-based is a little confusing and it does not provide a fresher perspective of discussion already in the fairness literature around understanding the significance of inherent 'uncertainty' or 'randomness' in application domains other than vision (and probably language). For example, the Fair ML book (Barocas, Hardt, and Narayanan 2017) does mention uncertainty in the context of such applications (on page 34, 56). \n\nOverall, I am not very convinced that the paper should be accepted unless fellow reviewers think strongly otherwise. However, I think the paper has the potential to be a more complete and important contribution with a more comprehensive study around the technical contributions and clearer discussion about the normative contributions.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors study bias amplification [Zhao et al, 2017] and propose an improved metric for measuring it. The authors also discussed normative issues in bias amplification (predicting a sensitive feature), and how to measure amplification when we do not have labels, or where labels correspond to uncertain future events. While the reviewers acknowledged the importance to study bias amplification, normative measures and social context, they raised several important concerns: 

(1) limited technical contributions (R3 and R4) -- see reviewers’ concerns that the metric is the only technical contribution; one possible suggestion is to propose a mitigation strategy based on the proposed metric similarly to [Zhao et al, 2017]; 

(2) ‘the usage of error bars because of the Rashomon effect seems incomplete and almost trivial’ (R3, R4), ‘lacks a proper grounding’ (R1) -- see two suggestions by R3 how to revise; these suggestions have not been discussed in the rebuttal; 

(3) empirical evidence lacks a controlled scenario of tuning the bias source to evaluate consistency (suggested by R3) and is limited in the context of algorithmic fairness benchmarks (R4) -- this has been partly addressed in the rebuttal; 

(4) normative contributions and broader discussions are oversimplified – see R3’s comments and suggestions on how to better position the paper.

Among these, (3) and (4) did not have a major impact on the decision but would be helpful to address in a subsequent revision. However, (1) and (2) make it very difficult to assess the benefits of the proposed work and were viewed by AC as critical issues. 
A general consensus among reviewers and AC suggests, in its current state the manuscript is not ready for a publication. It needs technical strength, more empirical studies and polish to achieve the desired goal. We hope the detailed reviews are useful for revising the paper.

Decision: Reject