Are My Deep Learning Systems Fair? An Empirical Study of Fixed-Seed Training
Authors: Shangshu Qian
Hung Viet Pham
Thibaud Lutellier
Zeou Hu
Jungwon Kim
Lin Tan
Yaoliang Yu
Jiahao Chen
Sameena Shah
Venue: NeurIPS 2021 Poster (2021)
Keywords: DL software implementation
fairness variance
empirical study

Abstract: Deep learning (DL) systems have been gaining popularity in critical tasks such as credit evaluation and crime prediction. Such systems demand fairness. Recent work shows that DL software implementations introduce variance: identical DL training runs (i.e., identical network, data, configuration, software, and hardware) with a fixed seed produce different models. Such variance could make DL models and networks violate fairness compliance laws, resulting in negative social impact. In this paper, we conduct the first empirical study to quantify the impact of software implementation on the fairness and its variance of DL systems. Our study of 22 mitigation techniques and five baselines reveals up to 12.6% fairness variance across identical training runs with identical seeds. In addition, most debiasing algorithms have a negative impact on the model such as reducing model accuracy, increasing fairness variance, or increasing accuracy variance. Our literature survey shows that while fairness is gaining popularity in artificial intelligence (AI) related conferences, only 34.4% of the papers use multiple identical training runs to evaluate their approach, raising concerns about their results’ validity. We call for better fairness evaluation and testing protocols to improve fairness and fairness variance of DL systems as well as DL research validity and reproducibility at large.

Are My Deep Learning Systems Fair? An Empirical Study of Fixed-Seed Training
Shangshu Qian Purdue University West Lafayette, IN, USA shangshudpurdue. edu
Hung Viet Pham University of Waterloo Vector Institute hypham@uwaterloo..cm
Thibaud Lutellier University of Waterloo Waterloo, ON, Canada tlutelliee wwwllli@uatterloo.oo
Zeou Hu University of Waterloo Vector Institute
Jungwon Kim Purdue University West Lafayette, IN, USA kim32410purdue edu
Lin Tan
Purdue University West Lafayette, IN, USA lintan@purdue.co edu
297hu@uaaterloo. ca
Yaoliang Yu University of Waterloo Vector Institute yaoliang yu@uwaterlo..caa
Jiahao Chen"
Sameena Shah
J. P. Morgan Al Research New York, NY, USA jihao@getparityy... ai
J. P. Morgan Al Research New York, NY, USA sameena hwaaaasshhhhhhaaaaaaaaae
Abstract
Deep learning (DL) systems have been gaining popularity in critical tasks such as credit evaluation and crime prediction. Such systems demand fairness. Recent work shows that DL software implementations introduce variance: identical DL training runs (i.e., identical network, data, configuration, software, and hardware) with a fixed seed produce different models. Such variance could make DI. models and networks violate fairness compliance laws, resulting in negative social impact. In this paper, we conduct the first empirical study to quantify the impact of software implementation on the fairness and its variance of DL systems. Our study of 22 mitigation techniques and five baselines reveals up to 12.6% fairness variance across identical training runs with identical seeds. In addition, most debiasing algorithms have negative impact the model such reducing model accuracy, increasing fairness variance, or increasing accuracy variance. Our literature survey shows that while fairness is gaining popularity in artificial intelligence (AI) related conferences, only 34.4% the papers use multiple identical training runs to evaluate their approach, raising concerns about their results' validity. We call for better fairness evaluation and testing protocols to improve fairness and fairness variance of DL systems well DL research validity and reproducibility large.
Introduction
DL systems, which consist of DL models and DL software (for example libraries such as Tensor-
Flow) [7I), are widely used; and enrusing their fairness is paramount importance [4 59 97 98
Biases in DL systems have been identified as major concern for civil rights by the U.S. govern-
ment 27 Yet, fairness testing for DL systems still an open challenge.
Although the DL community aware of the nondeterminism caused by using different random seeds 72 90 such algorithmic factors are not the sole source of DL nondeterminism. Even with the
Jiahao Chen has since moved to Parity Technologies.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

same seed, e.g., the same initial weights, DL systems are nondeterministic, i.e., multiple training runs with the same network configuration, the same dataset, and the same seed, referred to as fxxe-seed identical training runs (FIT runs), produce different models with different accuracies [71]. Such nondeterminism is caused by software implementation, e-g., software parallelism and floating-point computation of DL software (details in Appendix E.T)
Our previous work 771 shows that software implementation alone causes models' accuracies to vary by up to 2.9%, and most (83.8%) of the DL researchers and practitioners surveyed are unaware or unsure about the nondeterminism caused by software implementation. The small portion that is aware of such implemental variance underestimates it. The survey participants are researchers and practitioners with DL experience from universities and Al companies including Google, NVIDIA, Microsoft, and Facebook. These results suggest that even with sxed-seed identical training runs, one may still need to report error bars or use statistical tests.
Since the focus of the prior study 71 was on the variance of accuracy and training time only, it is unknown how DL software implementation affects the variance of fairness. Various debiasing algorithms 3 47 59 98 [9] optimize DL models toward a specific fairness goal and it is unclear how software implementation affects these debiasing algorithms. In addition, unlike accuracy, which
is a widely-accepted metric, dozens of fairness metrics exist 22 33 46
60 64] [9] [08]. It is unclear
if software implementation causes different levels of variance for different fairness metrics.
In this paper, we study the fairness variance caused by software implementation in training DL networks. Overall, we found that fairness variance is large-up to 12.6% difference (27.3% versus 39.9%) across 16 fixed-seed identical training runs.
First, our results show that one training run may produce fair model but another fixed-seed identical training run may generate an unfair one. For example, we reproduced previous work [47] that proposed a new network and debiasing algorithm and we found that one FIT run generates fair model (i.e., a model that does not discriminate based on gender) based on the threshold (20%) for the normalized disparate impact (DI) metric [29 64 (DI of 17.9 17.9%). A DI of 0% is the fairest, while 100% DI is the least fair. However, a second FIT run produces an unfair model (DI of 20, 20.4%, above the 20% threshold). This 20% threshold for DI has been used in U.S. legal case to determine system's 's fairness [9] where fairness variance could have changed the outcome.
Second, the variance of fairness imposes challenges for the valid evaluation of debiasing algorithms. For example, our experiments show that while a debiasing algorithm [07] reduces the bias amplifica- tion (BA) [99 [08] to 6.1% in one FIT run (compared with BA score of 7.8% for its baseline), this technique increases the bias amplification instead of reducing it when we check the average of 16 FIT runs: average BA score of 8.7% compared to an average BA score of 7, 7.4% for the baseline. The Mann-Whitney U-test shows the results from the 16 runs are statistically significant. The observed variance across 16 FIT runs is solely caused by software implementation.
Finally, many mitigation techniques increase fairness variance or hurt accuracy. With Mann-Wil Main-Whitney U-test and Levene's & test, our experiments show that a majority (68.0%) of the mitigation techniques reduce biases with a cost: they increase fairness variance, increase accuracy variance, or decrease model accuracy.
Since existing debiasing algorithms [7 [5 often do not measure the variance of biases, they could introduce risks. When such debiasing algorithms are applied to online learning, where DL models are trained continuously [53]. the increased variance from debiasing algorithms could cause the resulting models' fairness to miss compliance requirements or service-level agreements. For example,
Microsoft Tay [86] became a "racist chatbot" only after a few hours of deployment.
What's worse, even with fairness auditing before deployment, which is still not a standard practice in the industry (39, model can be fair on pre-deployment data, but unfair on real-world data, due to the gap between training/test data and real-world data. The fairness variance puts the trained models at greater risk of being unfair real-world data. Theoretically, while debiasing techniques on average improve fairness on the training set, they increase the variance of fairness on the test set [2]. In practice, such gaps are the key reason that facial recognition systems (58 76 have a racial bias.
In addition to quantifying fairness variance, we conduct a survey of research papers from top artificial intelligence (Al) conferences of recent years (2016-2020) to understand how researchers

---

handle the fairness variance caused by DL software implementations. Only 34.4% of the 32 papers evaluating or addressing DL fairness issues use multiple training runs in their evaluation.
To the best of our knowledge, we are the first to study software implementations impact on DL fairness variance. The contributions are:
A variance study of DL fairness of 27 techniques (22 mitigation techniques and five baselines). The experiments are performed on four popular datasets (CelebA, MS-COCO, imSitu, and CIFAR-IOS) with three DL networks (ResNet-18 I8, ResNet-50, and NIFR 47),, measured by seven popular bias metrics (Section 3)
Finding 1: Across 16 FIT runs, software implementation causes maximum difference in bias metrics of up to 12.6% in the Demographic Parity (DP) metric and up 11.8% in the DI metric. Yet, most (83.8% researchers and practitioners are unaware or unsure about the DL nondeterminism caused by software implementation (71).
The effectiveness of the debiasing algorithms can be inaccurate with only one fixed-seed training run.
Finding 2:
Finding 3: Most of (15 out of 22) the mitigation techniques tested increase at least one of the bias metrics.
Finding In a majority (68.0%, or 70) 103 experiments, debiasing algorithms have a negative impact on the model.
A literature survey on the fairness research papers of top machine learning (ML), (NeurIPS/NIPS, ICML, AAAI, and FAccTT), and computer vision (CVPR, ICCV, and ECCV) conferences (Section
Implications and suggestions for DL fairness researchers and practitioners (Section 5 including one should use FIT runs (setup details in Section 2.5) with statistical tests (Appendix [B.) to evaluate debiasing algorithms, when nondeterminism from DL software is unavoidable.
Data and code availability: Experiment data and artifact for reproducibility study are available in public GitHub repository
2
Experimental method
We investigate 27 techniques of DL fairness on four popular datasets with three networks by repro- ducing four previous works [7 59
9 99
that fit our scope.
2.1 Scope
We focus on the impact of DL software implementation on the variance of group fairness (66 in classification tasks, where two orthogonal labels are assigned to each instance in the dataset, task label for training classifier, and a protected label that should not be discriminated against (e.g.. gender). The set of instances with the same protected label forms protected group.
Although the protected labels are not used the classification task, the trained models could still be biased because the protected labels can be leaked into the prediction by proxy variables [3] 59 98 [08].
2.2 Bias metrics
Various metrics exist to measure biases of ML algorithms. Yet. it is still an open problem to select these metrics as an optimization goal for real-world problems 44). Therefore, we choose three fairness conditions and seven bias metrics derived from them. Fairness conditions include statistical parity (66 (SP), predictive equality [19 (PE), and equality of opportunity (35) (EOp). Seven bias
metrics include demographic parity [1] (DP), normalized disparate impact [15] (DI), statistical parity
subgroup fairness [46) (SPSF), false positive subgroup fairness [46] (FPSF), equalized odds false
positive 24 (EOFP), equalized odds true positive (35 (EOTP), and bias amplification [99 [08
(BA). For all bias metries, indicates the fairness condition satisfied, i.e., completely fair model, and indicates completely unfair model. Details of these metrics are in Appendix A
httpps//ggtchbb.cmm///aaaaaaaaaa

---

Table 1: Evaluated mitigation techniques and basclines
Task
Technique Debiasing Algorithm
Network
Tgt
S-Base
Baseline
S-RS
Data Rebalancing
S-UC
Uniform confusion loss
S-GR
Gradient reversal
Multi-class classification in CIFAR-IOS 199 (1)
S-DD1
Sum of probabilities
ResNet-18
BA
S-DD2
Max of prob. w/ prior shift
S-DD3
Sum of prob. w/ prior shift
S-DD4
RBA debias
S-DII
Domain independent, conditional
S-D12
Domain independent, sum
C-Base
Baseline
C-R1
Data rebalancing, M/F
Multi-label classification
C-R2
Data rebalancing, M/F 2 Data rebalancing, M/F <
DP
ResNet-50
MS-COCO 98 (2)
C-R3
(BA*)
C-A4
Adv training at 4th conv block
C-A5
Adv training at 5th conv block
[-Base
Baseline
I-R1
Data rebalancing, M/F
Multi-class classification in imSitu [98] (3)
I-R2
Data rebalancing, M/F 2
DP
ResNet-50
I-R3
Data rebalancing, M/F
(BA*)
I-A4
Adv training at 4th conv block Adv training at 5th conv block
I-A5
A- Base
Baseline
Binary classification
A-L2
L2 penalty
ResNet-18
EO
in CelebA 559 (4)
A-ALM
FairALM
Binary classification in CelebA 47 (5)
N-Base
Baseline
NIFR
DP
N-Flow
Conditional Flow
Per-class and overall bias: Bias caleulated on each class is called per-class bias and the bias for each model is the overall bias. For example, there are ten per-class biases in models trained on CIFAR- (one per class). The overall bias of the model the average of all per-class biases.
(task-lta classification: We treat each task label as an independent binary classification, and the overall bias value of the model is the average across all labels.
2.3 Reproduced studies
We conduct a literature survey of published papers from the recent five years (2016 6-2000) of the top ML (NeurIPS/NIPS, ICML, AAAL, and FAceTT), and computer vision (CVPR, ICCV, and ECCV) conferences (Section 4 Out of the 32 papers relevant to the fairness DL systems, only four 447
98 [9] have working replication package so that can reproduce their results. These papers study the fairness of DI. systems for the five tasks described in Table (expanded details in Appendix B.T]. Overall, replicate 27 different techniques from four state-of-the the-atherrr art papers five classification tasks. In the next section, we introduce each technique and detail our experiments.
2.4 Techniques
Table T lists the 27 techniques that we reproduce for our study. Column Task lists the tasks on which these techniques are applied. Column Technique lists the technique abbreviations we used. Column Debiasing Algorithm lists the debiasing algorithms used for each technique. Column Network lists the DL network used to train the models. Finally, column Tgr indicates which bias metric the corresponding debiasing algorithm is designed to reduce.

---

Five techniques (S-Base, C-Base, I-Base, A-Base. and N- N-Base) are baselines (i.e., do not use any debiasing algorithm), and the other 22 use debiasing algorithms. Techniques starting with "S" from [99]; techniques starting with "C" and "I" are from [98] trained for tasks (2) and (3) respectively; techniques starting with "A" are from [59] ; and techniques starting with "N" are from
Apart from five baseline techniques, the remaining algorithms can be grouped into three categories including data rebalancing, fairness through blindness, and fairness through awareness. Details can be found in Appendix
Optimization target for tasks (2) and (3): The study [8] uses "bias amplification" as their opti- mization goal. However, current literature does not use this term consistently. The original study definition (noted as BA*) is different from the one proposed by Zhao et al. [108] (noted as BA) and BA* is more analogous the DP metrie. Therefore, we denote this optimization target DP (BA*) *).
2.5 Variance experiments setup
Following the prior work 77,, we use FIT runs to measure the variance of trained model fairness. For each technique, all the training runs are executed with the same training data (also the original training/test split), prrer-parameters and optimizers. With the fixed seed, all training runs also have the same order of data and the same initial weights. We perform 16 FIT runs with the same random seed for each technique, and then evaluate the fairness of the trained models using seven bias metrics.
Since there is no theoretical bound on the variance caused by DL software implementations, and each set of 16 FIT runs can only be viewed as a sample of the real distribution, we use statistical tests, including Mann-Whitner Whitney U-test, Cohen's 's d value, and Levene's ' test to ensure the statistical significance of our findings. Details of these statistical tests can be found in Appendix B.3
3 Findings
We evaluate the fairness variance of 27 techniques (22 mitigation and five baseline) using seven bias metrics. These are 189 experiments (technique-metric pairs) To evaluate the impact of DL software implementation on different bias metrics, we perform 16 FIT runs for each of the 27 techniques, which produce 432 models. Each experiment generates one bias value per run (i.e., 16 bias values per experiment), which is a total of 3,024 bias values. The training time of the 189 experiments 7,117 hours (about 9.8 months). Details of the hardware and software environment are in Appendix [.4]
To measure the difference across multiple FIT runs, use two metrics: (1) the difference in percentage point as MaxDiff (maximum difference) and (2) the standard deviation in percentage point as STDEV. MaxDiff is the range of the bias values from set of 16 FIT runs. For example, the MaxDiff of 16 runs, whose bias values are between 20.0%-300 -0.0%% inclusive, is 10.0%.
3.1 RQ1: How much variance of fairness does DL software implementation introduce?
Table 2 shows the fairness variance introduced by software implementation. Row Technique lists the techniques we focus on for this RQ Row Metric shows the bias metrics used for evaluation. Row Overall shows the bias for the overall technique. Row Per-class shows the results for the class within each technique that has the most variance.
Table 2 shows that software implementation impacts all bias metrics. For example, we observed maximum difference DP with technique A-L.2 of 12.6% (27.3%-39 (...3%%33999.99 99),, i.e., the most unfair model has 12.6% more gender bias measured by DP than the fairest model. Also, in technique S-GR, we observed a maximum difference in DI of 11.8% (21.3%-3..) 13)....................... Software implementation solely causes such variance, and thus cannot be eliminated by fixing the random seeds.
The per-class variance of fairness and single class fairness failure could be problematic if a class privileged. For example, in a DL model for loan applications, the approval is the privileged class. difference of 100% for DI means that in one run, the model is completely fair for all races (protected groups) to get approval. In another run, none of the applicants of particular race gets approved, which is 100% bias against this minority group, even if the overall fairness variance is small.
5

---

Table 2: Maximum difference (MaxDiff) and standard deviation (STDEV) on the seven bias metrics
for S-GR, A-L22, C-Base, and C-A5. All numbers are percentage points.
SPSF
FPSF
EOTP
SPSF
FPSF
EOFP
BOTT
Metric
DP
DI
BA.
DP
DI
BA
Technique
S-GR
A-1.2
MaxDiff
1.7
11.
0.9
0.5
1.2
7.2
4.2
12.6
8.0
6.0
1.6
7.2
7.2
11.6
Overall (%)
STDEV
0.5
3.5
0.3
0.2
0.4
2.0
1.3
2.9
2.4
1.4
0.5
2.1
2.1
3.1
MaxDiff
7.3
50.1
3.6
2.5
5.7
34.6
19.3
12.6
31.1
6.0
3.5
14.3
14.3
15.9
Per-class (%)
STDEV
1.9
12.7
1.0
0.7
1.4
8.1
5.0
2.9
8.8
1.4
1.2
4.8
4.8
4.2
Technique
C-Base
C-A5
MaxDiff
<0.1
2.6 <0.1
<0.1 <0.1
1.9
1.4
0.1
2.4
<0.1
<0.1
<0.1
2.6
3.1
Overall (%)
STDEV
<0.1
0.8
<0.1
<0.1 <0.1
0.6
0.4
<0.1
0.6
<0.1
<0.1
<0.1
0.6
0.8
MaxDiff
1.6
100.0
0.8
0.5
1.1
33.3
33.3
3.7
100.0
1.8
1.0
2.1
43.9
65.4
Per-class (%)
STDEV
0.5
38.9
0.3
0.2
0.4
11.7
8.1
0.9
28.2
0.4
0.2
0.5 10.8
21.8
Finding I: Across 16 FIT runs, software implementation causes a maximum difference of up to 12.6% the DP bias metric, and up to 1 11.8% the DI metric. A complete single-class fairness failure is possible, where one run produces a model with 0% bias while another with 100% bias.
Additional results of this RQ can be found in Appendix D.I results for researchers and practitioners in Section 5
We discuss the implications of these
3.2 RQ2: How does software implementation affect debiasing algorithms?
To quantifyy the impact of software implementation debiasing algorithms, use the Mann-Whitue U-test and Levene's s test to check whether the average bias value and the variance of bias value after debiasing statistically different from those of its baseline respectively. We normalize each technique's 's bias values before Levene's s test to eliminate the impact of the average value [5] (8 77 91).
For 22.1% 1% (34 of 154) of the experiments (the comparison of 22 mitigation techniques with their baseline techniques evaluated using seven bias metrics), the mitigation techniques' bias values overlap with those of the baselines. For example, the range of FOTP bias values for the A-ALM debiasing is 1.6%%224.33 while baseline's (A-Base) range is 22.6%.24.77%
For 9.1% (14 of 154) of the experiments, the mitigation technique's average bias metric is higher than the baseline, yet single run's bias value is lower than single run of the baseline. Thus, using single run, one may incorrectly conclude that the mitigation technique is fairer, but on average, the mitigation technique is less fair. For example, S-Base (7.4% BA) fairer than S-GR (8. 7% BA) on average of 16 FIT runs (p 0.0011 in U-test). However, one may obtain a fairer model with S-GR from one training run (e-g., BA 6. % S-GR vs. 7.8% for S-Base). Therefore, DL fairness researchers should perform multiple runs and check statistical tests to ensure the effectiveness and validity the debiasing algorithms. Our Levene's test results also confirm that the variance of bias values of S-GR is statistically higher than that of S-Base (p 0.0005).
Finding 2: The effectiveness of the debiasing algorithms can be inaccurate with only one fixed-seed training run.
Since there no consensus on which bias metric to use 19 satisfied simultaneously (50), study if and how debiasing algorithm reduces biases measured by one bias metric but increases biases measured by a different metric.
49
and some bias constraints cannot be
For the 154 experiments above, we conduct a series of Mann-Whitney Whitney U-tests on the bias values obtained. For each test that exhibits a statistically significant difference, we compute effect size using Cohen's s d value between the two sets of bias values (baseline and mitigation) involved. The d values tell us whether each mitigation technique has a meaningful effect or not. Raw values before and after mitigation available in the supplementary material.

---

Huge
Large
Small
Small + Medium
Large Very Large
Huge +
Very Large
Medium
DP
DI
SPSF
FPSF
EOFP
ЕОТР
S
S-UC
S.GR
S.
S.D
S.DDD
S-DDA
S.DI1
S-D12
GR1
C-R2
CR3
C-44
CAS
I-R1
I-R2
I-R3
I-A4
1-A5
A-L2
A-ALM
N-Flow
Figure I: Effect size of bias changes between before and after mitigation application. Blue is bias
decrease (good), orange an increase (bad), and grey not statistically significant (NS).
Figure Shows shows the effect sizes of bias value between 22 mitigation techniques and their corresponding baselines. The x-axis and y-axis labels are the abbreviations for the techniques and bias metrics. If mitigation technique statistically reduces the bias value on bias metric (with respect its baseline), the corresponding block is blue. Oppositely. the block is orange. The block is grey for mitigation techniques that do induce a statistically significant difference from its baseline. The different shades of color show the standard grouping and interpretation of the effect size, such as "Huge" and "Very large". Figure T legend shows the color details. Blocks with a "** are the bias metric that the mitigation technique is designed to reduce in the original research (e.g., BA for S-DI2 and DP for N-Flow). An "o" is reminder that the optimization target of the mitigation techniques that begin with "C" and "I" is better captured by the DP metric (BA* in the original research, see Section 2.4).
Figure shows that most (96 of 103) mitigation techniques that induce a statistically significant improvement on the bias value have huge effect (deep blue block). However, the vast majority (68.2%, 15 of 22) of mitigation techniques fail reduce biases measured by all seven bias metrics. For example, A-L2 and A-ALM aim to improve bias metric EOFP. Although they succeeded, they also increase biases for BA.
This experiment shows that for most of the mitigation techniques, bias metrics may not be compat- ible with each other. Practitioners applying mitigation techniques must understand the techniques' optimization goal and use statistical tests.
Finding 3: Most of (15 out of 22) the mitigation techniques tested increase at least one of the bias metrics: all 15 of them fail to improve the newly introduced bias amplification metric.
3.3 RQ3: How does software implementation affect the cost of mitigation techniques?
The most known cost of mitigation techniques is the trade-off between fairness and model accuracy [25] [00). However, with the software implementation taken into consideration, mitigation techniques could introduce new costs including more variance of model accuracy and fairness.
There three dimensions the cost of mitigation techniques: lower accuracy, higher accuracy variance, and higher fairness variance. We excluded the techniques that increase bias for all metrics compared to its baseline. For accuracy, use U-test to check if the reduction is statistically significant. For the variance of both model accuracy and model bias, we use Levene's test.
Figure shows the cost of mitigation techniques on fairness variance (Levene's test). Labels x-axis and y-axis are the same as those Figure Blue and orange blocks respectively indicate lower and higher variance than the baseline. Grey blocks mean statistically significant change in the variance of bias values, and "X" indicates mitigation technique fails to improve the bias metric tested. For example, the orange [N-Flow, DP] block shows that N-Flow generates models whose variance of DP bias values higher than that of the baseline, and the comparison results are statistically significant.
To study the relationship among the three types of cost, we further examine the 154 experiments on mitigation techniques. Most of the experiments (103) reduce biases. The majority them (70 out of 103) achieve bias reduction with cost in one or more of the three dimensions. Eight experiments come with all three costs: lower accuracy, higher accuracy variance, and higher fairness variance, and from A-L2 and A-M.M. However, there is correlation between accuracy variance and bias

---

NS
DP
X
X
DI
X
X
X
X
SPSF
FPSF
X
X
EOFP
X
X
ЕОТР
BA
X
X
X
S-RS
SuC
S.GR
S-DD2
S-DD3
S-DD4
S-DIL
S-DIZ
C-R1
C-R2
C-R3
C-A4
C-A5
I-R1
I.R2
I-R3
I-A4
1-A5
A-L2
A-ALM
N-Flow
Figure 2:
Cost of bias mitigation techniques on the variance of bias metrics after normalization
(Levene's tests). Blue denotes decrease in variance (good), orange increase in variance (bad), and grey not statistically significant (NS).
variance. Interestingly, mitigation technique S-DD4 (RBA [008] debiasing) successfully reduces all of the seven bias metrics without sacrificing model accuracy or introducing more variance.
DL researchers and practitioners might overlook the cost of fairness variance when optimizing a model toward particular fairness goal if they unaware of the variance caused by DL software [71]
Finding 4: Out of 154 experiments with debiasing algorithms, 103 experiments show the applied algorithms reduce the bias values. In a majority (68.0%, or 70) of these 103 experiments, the debiasing algorithms have negative impact the model (36 increase the variance of bias value, 54 decrease the accuracy, and 15 increase the variance of accuracy, not exclusively).
Literature survey
We conduct literature survey from the top ML conferences to evaluate the awareness of fairness variance of DL systems.
Paper selection criteria: We extract papers from the last five years (2016-2020) of the top ML (NcurIPS/NIPS, ICML, and AAAI), and computer vision (CVPR, ICCV, and ECCV) conferences. We also include FAceT and its predecessor FAT/ML workshop, interdisciplinary venue of ML and fairness.
We only consider papers that contain fairness-related keywords (fair, bias, discriminate, aware, balance, disparity, opportunity, audit, gender, stereotype, race, amplification, ethic) in their titles using the Snowball Stemmer [733] Then, read the abstract and the paper and keep relevant papers involving the fairness of a classification task. Out of 24, 186 papers, we extracted 274 papers studying fairness, including 32 dedicated to DL. systems' fairness.
Paper survey results: While still taking only small portion of accepted papers, the number of papers related to fairness has been increasing over the last five years. Researchers need to be aware of DL. models' fairness variance caused by DI. software implementations.
Finding 5: DL. fairness is gaining more attention in the research community since 2017. The number of papers related to DL fairness top MI. and computer vision conferences has increased 750% since 2017 (two papers in 2017, 17 papers in 2020).
To understand whether researchers aware of the fairness variance of DL training, manually check whether the papers on the fairness DL systems use multiple identical runs their evaluation.
Finding 6: Of 32 papers relevant to fairness of DL systems, only 34.4% (11 out of 32) of them multiple identical runs evaluate their proposed approaches.
A complete list of 32 papers surveyed in this section be found in Appendix D.3

---

5 Implications and suggestions
Research validity: When comparing the results of a proposed method (e.g., a new debiasing algorithm or a new DL network), it is necessary to run statistical tests to confirm any improvements.
Testing for fairness in online learning: When testing an online-lcarning software system for fairness -bsee advertisement targeting
(where models are continuously updated), such as systems 70 [OT,, multiple identical runs are necessary, and statistical test results should be reported to measure the variance of bias values caused by the DL software implementation.
Research reproducibility: Given the increasing number of papers on fairness in AT 6
32
48
49
63 65 79 [05], software engineering [TO
14 16 94], and other conferences 12
52 reproduci-
that improve the reproducibility of DL methods, researchers may want to develop such guidance for fairness, e.g., considering multiple bias metrics, as variance levels are different for different bias metrics. Proper documentation of the software and hardware development environment and configuration parameters would also facilitate the reproduction of research results. As shown in Section software implementation alone causes software fairness to vary by up to 12.6%, which is much higher than the 2.9% accuracy variance discovered by previous work 77].
bility becomes increasingly important
Following prior efforts 43 56
72
Balancing hidden costs of debiasing: Since variance is a hidden cost for debiasing algorithms and could harm the fairness performance of online learning algorithms (Section (.3). there is usually trade-off between model accuracy, fairness, and variance. It is important that DL researchers and practitioners understand debiasing algorithms' costs, especially variance, and strike a balance among them.
Deterministic DL software for model tuning and debugging: Building deterministic DL libraries could be a research topic to improve model tuning and the debugging of DL systems. For partial solution, one could use the deterministic mode (Appendix E.T when it provided by DL libraries.
Maximizing fairness through multiple runs: To take advantage of software variance, practitioners could train multiple times to select the fairest model. In this case, proper testing of the DL model is recommended to ensure the satisfaction of regulatory requirements, considering the pre-deployment and real-world data gap.
Selection of bias metrics: In most cases, a debiasing algorithm does not reduce bias on all targets (Section 3.2), and multiple fairness constraints could contradict each other [49 50). Therefore, developers and testers for DL systems should carefully select fairness requirements. Furthermore, optimizing one bias metric may not be enough as it might make the model less fair other metrics. DL researchers should consider the task and use the appropriate bias metric. In the strictest sense, one may need to multiple bias metrics ensure fairness across multiple dimensions.
Limitations and discussions
Limitation: It is known that gender bias exists in language embeddings [108]. We only study image classification tasks because they are a better fit for our scope of group fairness and bias metrics on image classification tasks are better defined. Also, only empirically obtain the upper bound of fairness variance and did not include any theoretical proof. We leave both limitations as future work.
All 27 techniques reproduced are implemented in PyTorch instead of other popular DL. frameworks such as TensorFlow and MXNet. We decide to reuse the implementation from the original authors as is error- Prror-prone to reimplement all the techniques in another framework. Since modern DL frameworks share many implementation concepts (c.g., parallelism and auto-tuning, see Appendix E.T) as well as underlying libraries such as cuDNN and CUDA, they are likely to suffer from the same variance problem.
Deterministic mode of DL frameworks: A deterministic mode has been introduced to modern DL frameworks such as Pyrch and TensorFlow to tackle the nondeterminism from software imple- mentations. However, at the time of writing, the deterministic mode cannot eliminate the fairness variance described in this paper (See Appendix [..2 for details). addition, as the survey in our prior work 7 shows, over 80% of the respondents are unaware or unsure about the nondeterminism

---

caused by software implementations. DL researchers and practitioners would not use the deterministic mode they are unaware of the fairness variance caused by DL software.
Negative social impact: With FIT runs to estimate the fairness variance of neural networks, our proposed approach could increase the power consumption and CO2 emissions of DL model training. However, such additional evaluation is required for experimental validity and fairness compliance. In other words, the negative social impact of not performing multiple training runs includes dispropor- tionally hurting certain races or genders. Our paper raises awareness of this issue and calls for new solutions to strike balance. We are working on theoretically quantifying the upper bound of fairness variance and cheaper ways to estimate/bound variance.
7 Related work
Anecdotal evidence in fairness variance: Agrawal al. apply debiasing on the random forest classifiers and test for fairness variance with different random seeds and data splits for grossvvliddttion They find that debiasing algorithms could increase the fairness variance introduced by algorithmic nondeterminism. Different from their work, we focus on DL models and variance caused by DL. software implementations, which cannot be removed by fixing random seeds.
Friedler et al. 30 investigate the stability debiasing algorithms by measuring the standard deviation of fairness metrics over different dataset splits. However, they overlook the nondeterminism from the DL software implementations and only perform one training run for each split.
Fairness studies of ML systems: The fairness ML and DL systems have gained attention and several studies pointed out fairness issues in datasets (e.g.. CelebA [8], GitHub [4], and StackOver- flow (61) and major ML systems such as language models [3 92 95), face recognition [89], and sentiment analysis [88). Such studies demonstrate that ML systems can be unfair and highlight the importance of fairness. Yet to the best of our knowledge, we are the first to study the impact of DL software implementation fairness.
103 have been proposed to measure biases of MI. systems. Most them can applied to DL systems too. We evaluate fairness using seven of the most popular metrics and show that they are all impacted by DL implementation variance. None of the existing metrics considers such variance in their definition.
Bias metrics: Many metrics 6 15
19
24
46 67
Debiasing algorithms:
To mitigate biases of DL systems, many debiasing algorithms or bias resistant
networks have been proposed and evaluated 110 15 16
45
47
[5]] [59 66 83 (98 99 [06). These
techniques are important improve the fairness of DL systems but their evaluation can affected by fairness variance introduced by DL software implementations. Our study shows that the impact of some debiasing algorithms is not consistent during different identical runs.
Conclusion
This work studies the fairness variance of DL systems caused by software implementations. We examine five tasks on three networks and four datasets using 22 mitigation techniques. We find up to 12.6% difference in bias values across FIT runs and 68.0% of the experiments reduce model biases with cost fairness variance, accuracy, or accuracy variance. Our paper survey shows that only 34.4% of the papers use multiple identical runs to evaluate the proposed approaches. We call for the awareness of the fairness variance of DL systems caused by software implementations.
Acknowledgments
The authors thank the anonymous reviewers for their constructive comments well as the area chair and the senior area chair for overseeing the review process. This work has been partially supported by NSF 2006688 and a J.P. Morgan Al Faculty Research Award. We also thank NSERC for funding support. Any opinions, findings, and conclusions this paper are those of the authors only and do not necessarily reflect the views of the sponsors.
10

---

References
[1] Tameem Adel, Isabel Valera, Zoubin Ghahramani, and Adrian Weller, 2019. One-network adversarial fairness. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 2412-2420.
[2] Ashrya Agrawal, Florian Pfisterer, Bernd Bischl, Jiahao Chen, Srijan Sood, Sameena Shah, Francois Buet- Golfouse, Bilal A Mateen, and Sebastian Vollmer. 2020. Debiasing classifiers is reality at variance with expectation? (2020), 11 pages. arXiv:201 1.00407
[3] Mohsan Alvi, Andrew Zisserman, and Christoffer Nellåker. 2019. Turning blind eye: Explicit removal of biases and variation from deep neural network embeddings. In Computer Vision (ECCV '18 Workshops). Springer, Cham, 556-572.
[4] Andrew I. Beam, Arjun K Manrai, and Marzyeh Ghassemi. 2020. Challenges to the repro- ducibility of machine learning models in health care. JAMA 323, 4 (2020), 305-306.
[5] Arthur G Bedeian and Kevin W Mossholder. 2000, On the use of the coefficient of variation as measure of diversity. Organ. Res. Methods 3, 3 (2000), 285-297.
[6] Omer Ben-Porat, Fedor Sandomirskiy, and Moshe Tennenholtz. 2021. Protecting the Protected Group: Cireumventing Harmful Fairness. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 5176-5184.
[7] Sebastian Benthall and Bruce D Haynes. 2019. Racial categories in machine learning. In Conference on Fairness, Accountability, and Transparency (FAT* '19). ACM, New York, 289 298.
[8] Alex Beutel, Jilin Chen, The Zhao, and Ed Chi. 2017. Data decisions and theoretical implications when adversarially learning fair representations. arXiv preprint (2017).
[9] Dan Biddle. 2006. Adverse impact and test validation: A practitioner's 's guide to valid and defensible employment testing (2 ed.). Routledge, London and New York.
[10] Sumon Biswas and Hridesh Rajan. 2020. Do the machine learning models crowd sourced platform exhibit bias? An empirical study on model fairness. In 28th Joint Meeting on European Software Engineering Conference and Symposium the Foundations of Software Engineering (ESEC/FSE 2020). ACM, New York, 642-653.
[11] Emily Black, Samuel Yeom, and Matt Fredrikson. 2020. Fliptest: fairness testing via optimal transport. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 111-121.
[12] Su Lin Blodgett and Brendan O'Connor. 2017. Racial disparity in natural language processing: A case study of social media american english. arXiv preprint arXiv:1707.0006: (2017).
[13] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems (NIPS '16 Vol. 29). Curran Associates, Red Hook, NY, 4349-4357.
[14] Yuriy Brun and Alexandra Meliou. 2018. Software fairness. In 26th Joint Meeting on European Software Engineering Conference and Symposian the Foundations of Software Engineering (ESEC/FSE 18). ACM, New York, 759.
[15] Toon Calders and Sicco Verwer. 2010. Three naive Bayes approaches for discrimination free classification. Data Min. Knowl. Disc. 21, 2 (2010), 277-292.
[16] Joymallya Chakraborty, Suvodeep Majumder, and Tim Menzies. 2021. Bias in Machine Learning Software: Why? How? What to Do?. In Proceedings of the 29th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Athens, Greece) (ESEC/FSE 202 Association for Computing Machinery, New York, NY, USA, 429-440. attps::/doi... org 10. 1145 34682644 3468537

---

[17] Joymallya Chakraborty, Suvodeep Majumder, Zhe Yu, and Tim Menzies. 2020. Fairway: a way to build fair ML software. In 28th Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/ESE '20). ACM, New York, 665.
[18] Jiahao Chen, Victor Storchan, and Eren Kurshan. 2021. Beyond Fairness Metrics: Roadblocks and Challenges for Ethical AT in Practice. arXiv preprint arXiv:2 06200000027770 (2021).
[19] Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data 5, (2017), 153-163.
[20] COCODataset, [n.d.]. Common Objects in Context Terms of Use. https://cocodatasees org/ #termsofuse Last accessed 2021-05-25.
[21] Elliot Creager, David Madras, Jörn-Henriz Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. 2019. Flexibly fair representation learning by disentanglement. In International Conference on Machine Learning. PMLR, 1436-1445.
[22] Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Srikumar. 2020. On measuring and mitigating biased inferences of word embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 7659-7666.
[23] PyTorch Developers. [n.d.].
use benncuuseedetermmistiee aal.oiith_ggoomhmmmm_mm
https
wttp://os//staaeee////aaaaeerrre lese_deterministiie itteeemmmiiiiiilllgoiiiiiiiiiiii thma. html htmlh use_ ruse_daterminissicccom
thms
[24] Mengnan Du, Fan Yang, Na Zou, and Xia Hu. 2020. Fairness in deep learning: A computational perspective. IEEE Intelligens Systems (2020), 7 pages.
[25] Sanghamitra Dutta, Dennis Wei, Hazar Yueksel, Pin-Yu Chen, Sijia Liu, and Kush Varshney, 2020. Is there a trade-off between fairness and accuracy? A perspective using mismatched hypothesis testing. In International Conference on Machine Learning (ICML. '20, Vol. 119). PMLR, 2803-2813.
[26] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In 3rd Innovations in Theoretical Computer Science Conference (ITCS 12). ACM, New York, 214-226.
[27] Executive Office of the President. 2016. Big data: A report algorithmic systems, opportunity, and civil rights.
[28] DS Faber and H Korn. 1991. Applicability of the coefficient of variation method for analyzing synaptic plasticity. Biophys. J. 60, (1991), 1288-1294.
[29] Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkata- subramanian. 2015. Certifying and removing disparate impact. In 2 2/st SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 15). ACM, New York, 259-268.
[30] Sorelle A Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary, Evan P Hamilton, and Derek Roth. 2019. A comparative study of fairness- cnhanninn interventions in machine learning. In Conference Fairness, Accountability, and Transparency. 329-338.
[31] Stephen Gillen, Christopher Jung, Michael Kearns, and Aaron Roth. 2018. Online learning with an unknown fairness metric. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2605-2614.
[32] Naman Goel, Alfonso Amayuelas, Amit Deshpande, and Amit Sharma. 2021.
The Importance of Modeling Data Missingness in Algorithmic Fairness: A Causal Perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 7564-7573
[33] Aditya Grover, Jiaming Song, Ashish Kapoor, Kenneth Tran, Alekh Agarwal, Eric J Horvitz, and Stefano Ermon. 2019. Bias Correction of Learned Generative Models using Likelihood- Free Importance Weighting. Advances in Neural Information Processing Systems 32 (2019).
12

---

[34] Teng Guo, Feng Xia, Shihao Zhen, Xiaomei Bai, Dongyu Zhang, Zitao Liu, and Jiliang Tang. 2020. Graduate employment prediction with bias. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 670-677.
[35] Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems (NIPS' 16, Vol. 29). Curran Associates, Red Hook, NY, 3315-3322
[36] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR '16). IEEE, New York, 770-778.
[37] Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. 2018. Women also snowboard: Overcoming bias in captioning models. Proceedings of the Euro- pean Conference on Computer Vision (ECCV). 771-787.
[38] Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. 2018. Women also Snowboard: Overcoming Bias in Captioning Models (Extended Abstract). arXiv: Ttt1:1807700511 .00517 [es.CV]
[39] Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. 2019. Improving fairness in machine learning systems: What do industry practitioners need?. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1-16.
[40] IBM. 2018. Pytorch-seqgsseallss..yy at F1 11660077997777999955446660000000 ht https: www/icco seqsee
[41] Nasif Imtiaz, Justin Middleton, Joymallya Chakraborty, Neill Robson, Gina Bai, and Emerson Murphy-Hill. 2019. Investigating the effects of gender bias on GitHub. In 4 st International Conference on Software Engineering (ICSE' 19). IEEE/ACM, New York, 700-711.
[42] Jörn-Henrik Jacobsen, Arnold W M Smeulders, and Edouard Oyallon. 2018. i-RevNet: Deep invertible networks. In 6th International Conference on Learning Representations (ICLR '18). 11 pages.
[43] Christoph Jansen, Jonas Annuscheit, Bruno Schilling, Klaus Strohmenger, Michael Witt, Felix Bartusch, Christian Herta, Peter Hufnagl, and Dagmar Krefting. 2020. Curious Containers: framework for computational reproducibility in life sciences with support for Deep Learning applications. Future Gener: Comp. Sy. 112 (2020), 209-227.
[44] Nathan Kallus and Angela Zhou. 2019. The fairness of risk scores beyond classification: Bipartite ranking and the xauc metric. Advances in neural information processing systems 32 (2019), 3438-3448.
[45] Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. 2012. Fairness-aware classifier with prejudice remover regularizer. In Joint European Conference on Machine Learn- ing and Knowledge Discovery in Databases (ECML PKDD '12). Springer, Berlin Heidelberg, 35-50.
[46] Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. 35th International Conference Machine Learning (ICML '18, Vol. 80). PMLR, Stockholm, 2564-2572.
[47] Thomas Kehrenberg, Myles Bartlett, Oliver Thomas, and Novi Quadrianto. 2020. Null- sampling for Interpretable and Fair Representations. In Computer Vision (ECCV '20). Springer, Cham, 565-580.
[48] Hyemi Kim, Seungjae Shin, JoonHo Jang, Kyungwoo Song, Weonyoung Joo, Wanmo Kang, and II-Chul Moon. 2021. Counterfactual Fairness with Disentangled Causal Effect Variational Autoencoder. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 8128- 8136.
13

---

[49] Joon Sik Kim, Jiahao Chen, and Ameet Talwalkar. 2020. FACT: A diagnostic for group fairness trade-offs. In 37th International Conference on Machine Learning (PMLR. Vol. 119). PMLR, 5264-5274.
[50] Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2017. Inherent trade-offs in the fair determination of risk scores. In 8th Innovations in Theoretical Computer Science Conference (ITCS (17) (LJPlcs, Vol. 67). Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, 43:1-23.
[51] Alex Krizhevsky. 2009. Learning multiple layers of features from tiny images. Master's thesis. University of Toronto.
[52] Eren Kurshan, Hongda Shen, and Jiahao Chen. 2020. Towards Self-Regulating Al: Challenges and Opportunities of AI Model Governance in Financial Services. In Proceedings of the First ACM International Conference on Al in Finance (New York, New York) (ICAIF '20). Association for Computing Machinery, New York, NY, USA, Article 49, 8 pages. https
/ //doi.
3422564
[53] Gopiram Roshan Lal, Sahin Cem Geyik, and Krishnaram Kenthapadi. 2020. Fairness-Aware Online Personalization. (2020), 10 pages. arXiv:2007.15277
[54] Peizhao Li, Han Zhao, and Hongfu Liu. 2020. Deep fair clustering for visual learning. In Conference on Computer Vision and Pattern Recognition (CVPR '20). IEEE/CVF, New York, 9070-9079,
[55] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context, In Computer Vision (ECCV 14). Springer, Cham, 740-755.
[56] Chao Liu, Cuiyun Gao, Xin Xia, David Lo, John Grundy, and Xiaohu Yang. 2020. On the Replicability and Reproducibility of Deep Learning in Software Engineering. (2020).
arXiv:2006.1423
14244
[57] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015. Deep Learning Face Attributes in the Wild. In International Conference Computer Vision (ICCV '15). IEEE, New York, 3730-3738.
[58] Steve Lohr. 2018. Facial recognition is accurate, if you're white guy. New York Times 9 (2018), 8.
[59] Vishnu Suresh Lokhande, Aditya Kumar Akash, Sathya N Ravi, and Vikas Singh. 2020. FairALM: Augmented Lagrangian Method for Training Fair Models with Little Regret. In Computer Vision (ECCV 20). Springer, Cham, 365-381.
[60] David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. 2018. Learning Adver- sarially Fair and Transferable Representations. In 35th International Conference on Machine Learning (ICML. '1, Vol. 80). PMLR, Stockholm, 3384-3393.
[61] Anna May, Johannes Wachs, and Anikó Hannák. 2019. Gender differences in participation and reward on Stack Overflow. Empir. Softw. Eng. 24, 4 (2019), 1997-2019.
[62] Daniel McDuff, Shuang Ma, Yale Song, and Ashish Kapoor. 2019. Characterizing bias in classifiers using generative models. Advances Neural Information Processing Systems (NeurIPS 19, Vol. 32). Curran Associates, Red Hook, NY, 12 pages.
[63] Ninareh Mehrabi, Muhammad Naveed, Fred Morstatter, and Aram Galstyan. 2021. Exacerbat- ing Algorithmic Bias through Fairness Attacks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 8930-8938.
[64] Aditya Krishna Menon and Robert C Williamson. 2018. cation. In 1st Conference on Fairness, Accountability and Transparency (FAT* 18, Vol. 81). PMLR, New York, 107-118.
The cost of fairness in binary classifi-
14

---

[65] Blossom Metevier, Stephen Giguere, Sarah Brockman, Ari Kobren, Yuriy Brun, Emma Brun- skill, and Philip S Thomas. 2019. Offline Contextual Bandits with High Probability Fairness Guarantees. Advances in Neural Information Processing Systems 32 (2019), 14922-14933.
[66] Shaobo Min, Hantao Yao, Hongtao Xie, Chaoqun Wang, Zheng-Jun Zha, and Yongdong Zhang. 2020. Domain-aware visual bias eliminating for generalized zero-shot learning. In Conference on Computer Vision and Puttern Recognition (CVPR 20).. IEFE/CVP, New York, 12664-12673.
[67] Debarghya Mukherjee, Mikhail Yurochkin, Moulinath Banerjee, and Yuekai Sun. 2020. Two simple ways to learn individual fairness metrics from data. In International Conference on Machine Learning. PMLR, 7097-7107.
[68] Razieh Nabi and Ilya Shpitser. 2018. Fair inference on outcomes. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.
[69] Jun Hyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. 2020. Learning from Failure: De-biasing Classifier from Biased Classifier. Advances in Neural Information Processing Systems (NeurIPS '20, Vol. 33). Curran Associates, Red Hook, NY, 20673-
[70] Claudia Perlich, Brian Dalessandro, Troy Raeder, Ori Stitelman, and Foster Provost. 2014. Machine learning for targeted display advertising: Transfer learning in action. Mach. Learn. 95, (2014), 127.
[71] Hung Viet Pham, Shangshu Qian, Jiannan Wang. Thibaud Lutellier, Jonathan Rosenthal, Lin Tan, Yaoliang Yu, and Nachiappan Nagappan. 2020. Problems and Opportunities Training Deep Learning Software Systems: An Analysis of Variance. In 35th International Conference on Automated Software Engineering (ASE 220). IEEE/ACM, New York, 771-783.
[72] Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché Buc, Emily Fox, and Hugo Larochelle. 2020. Improving reproducibility in machine learning research (a report from the NeurlPS 2019 Reproducibility Program), (2020),
22 pages. arXiv:2003. 12206
[73] Martin F Porter. 2001. Snowball: A language for stemming algorithms.
[74] Reid Pryzant, Richard Diehl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang. 2020. Automatically neutralizing subjective bias in text. In Proceedings of the aaai conference on artificial intelligence, Vol. 34. 480-489.
[75] Novi Quadrianto, Viktoriia Sharmanska, and Oliver Thomas. 2019. Discovering fair represen- tations the data domain. Conference on Computer Vision and Puttern Recognition (CVPR 19). IEEE/CVF, New York, 8227-8236.
[76] Inioluwa Deborah Raji, Timnit Gebru, Margaret Mitchell, Joy Buolamwini, Joonseok Lee, and Emily Denton. 2020. Saving face: Investigating the ethical concerns of facial recognition auditing. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 145-151.
[77] George F Reed, Freyja Lynn, and Bruce D Meade. 2002. Use of coefficient of variation in assessing variability of quantitative assays. Clin. Diagn, Lab. Immun. 9, 6 (2002), 1235-1239.
[78] Félix Renard, Soulaimane Guedria, Noel De Palma, and Nicolas Vuillerme. 2020. Variability and reproducibility in deep learning for medical image segmentation. Sei. Rep. 10, (2020), 1-16.
[79] Ashkan Rezaei, Anqi Liu, Omid Memarrast, and Brian Ziebart. 2021. Robust Fairness Under Covariate Shift. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 9419-9427.
[80] Chris Rockwell and David F Fouhey. 2020. Full-body awareness from partial observations. In Computer VisV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16. Springer, 522-539.
15

---

[81] Hee Jung Ryu, Hartwig Adam, and Margaret Mitchell. 2017. InclusiveFaceNet: Improving face attribute detection with race and gender diversity, Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/MI. '18). 6 pages. arXiv:1712.0019;
[82] Sivan Sabato and Elad Yom-Tov. 2020. Bounding the fairness and accuracy of classifiers from population statistics. In International Conference on Machine Learning. PMLR, 8316-8325.
[83] Mhd Hasan Sarhan, Nassir Navab, Abouzar Eslami, and Shadi Albarqouni. 2020. Fairness by learning orthogonal disentangled representations. In Computer Vision (ECCV '20). Springer, Cham, 746-761.
[84] Yash Savani, Colin White, and Naveen Sundar Govindarajulu, 2020. Intra-Processing Meth- ods for Debiasing Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS 200 Vol. 33). Curran Associates, Red Hook, NY, 2798-2810.
[85] Shlomo Sawilowsky. 2009. New effect size rules of thumb. J. Mod. Appl. Stat. Methods 8, (2009), 597-599.
[86] Oscar Schwartz. 2019. In 2016 Microsoft's Racist Chatbot Revealed the Dangers of Online Conversation. IEEE Spectrum 11 (2019).
[87] Procheta Sen and Debasis Ganguly. 2020. Towards Socially Responsible Al: Cognitive Bias- Aware Multi-OOjective Learning. In Proceedings of the AAAI Conference Artificial Intelli- gence, Vol. 34. 2685-2692.
[88] Judy Hanwen Shen, Lauren Fratamico, Iyad Rahwan, and Alexander M Rush. 2018. Dar- ling or babygirl? Investigating stylistic bias in sentiment analysis. In Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML '18).
[89] Richa Singh, Akshay Agarwal, Maneet Singh, Shruti Nagpal, and Mayank Vatsa. 2020. On the robustness of face recognition algorithms against attacks and bias. In Conference Artificial Intelligence (AAAI '20, Vol. 34). AAAI, Palo Alto, CA, 3358313589..
[90] Koustuv Sinha, Joelle Pineau, Jessica Forde, Rosemary Nan Ke, and Hugo Larochelle. 2020. NeurlPS 2019 reproducibility challenge. ReScience C 6, 2 (2020), 11.
[91] Jesper B Sørensen. 2002. The and misuse of the coefficient of variation in organizational demography research. Sociol, Method Res. 30, 4 (2002), 475-491.
[92] Yi Chern Tan and L Elisa Celis. 2019. Assessing social and intersectional biases in contextual- ized word representations. In Advances in Neural Information Processing Systems (NeurIPS '19, Vol. 32). Curran Associates, Red Hook, NY, 12 pages.
[93] Flickr Team. [n.d.]. Flickr Terms of Use. Last accessed 2021-05-25.
https: // //www. www. ciickr..om/crraativeccoomom
[94] Yuchi Tian, Ziyuan Zhong, Vicente Ordonez, Gail Kaiser, and Baishakhi Ray. 2020. Testing DNN image classifiers for confusion bias errors. 42nd International Conference on Software Engineering (ICSE '20). ACM/IEEE, New York, 1122-1134.
[95] Jesse Vig. Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems (NeurIPS '20, Vol. 33). Curran Associates, Red Hook, NY, 12388-12401
[96] Christina Wadsworth, Francesca Vera, and Chris Piech. 2018. Achieving fairness through adversarial learning: an application to recidivism prediction. arXiv preprint 1807.01807.000 1099 (2018).
[97] Mei Wang and Weihong Deng. 2020. Mitigating bias in face recognition using skewness-aware reinforcement learning. In Conference on Computer Vision and Pattern Recognition (CVPR '20). IEEE/CVF, New York, 0322-93311
16

---

[98] Tianlu Wang, Jieyu Zhao. Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. 2019. Balanced datasets are not enough: Estimating and mitigating gender bias deep image representations. In International Conference Computer Vision (ICCV 19). IEEE, New York, 5310-5319,
[99] Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. 2020. Towards fairness in visual recogmition: Effective strategies for bias mitigation. In Conference on Computer Vision and Pattern Recognition (CVPR 20).
IEEE/CVE, New York, 8919-8928.
[100] Michael Wick, Swetasudha Panda, and Jean-Baptiste Tristan. 2019. Unlocking fairness: a trade- off revisited. In Advances in Neural Information Processing Systems (NeurIPS '19, Vol. 32). Curran Associates, Red Hook, NY, 10 pages.
[101] Allison Woodruff, Sarah E Fox, Steven Rousso-Schindle and Jeffrey Warshaw. 2018. A qualitative exploration of perceptions of algorithmic fairness. In Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, 14 pages.
[102] Sirui Yao and Bert Huang. 2017. Beyond parity: fairness objectives for collaborative filtering. In Proceedings of the 31st International Conference on Neural Information Processing Systems. 2925-2934.
[103] Sirui Yao and Bert Huang. 2017. New fairness metries for recommendation that embrace differences. arXiv preprint arXiv: 09838 (2017).
[104] Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. 2016. Situation recognition: Visual se- mantic role labeling for image understanding, In Conference on Computer Vision and Pattern Recognition. IEEE, New York, 5534-5542.
[105] Mikhail Yurochkin and Yuekai Sun. 2021. SenSel: Sensitive Set Invariance for Enforcing Individual Fairness. In ICLR 2021: The Ninth International Conference on Learning Represen-
tations.
[106] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. 2017. Fairness Constraints: Mechanisms for Fair Classification. In International Conference on Artificial Intelligence and Statistics (AISTATS '17, Vol. 54). PMLR, Fort Lauderdale, 962-970.
[107] Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018. Mitigating unwanted biases with adversarial learning. In Conference on A1, Ethics, and Society (AIES 18). AAAI/ACM, New York, 335-340.
[108] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints. In Conference on Empirical Methods in Natural Language Processing (EMNLP '17). ACL, Stroudsburg, PA, 2979-2989.
17


Reviews:
[{"review_id": "WtVJMUJlHP", "reviewer": "Reviewer_T4tk", "summary": "The paper did a thorough analysis of the effect of software implementation (e.g., floating-point and parallelism) on bias measures. It has been previously shown that FIT runs produce different accuracies! The result is surprising and although I knew that the same FIT runs can produce different outcomes, I didn’t know that the outcome could be that different! And it is a good study for the fairness community to be aware of! ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper did a thorough analysis of the effect of software implementation (e.g., floating point and parallelism) on bias measures. It has been previously shown that FIT runs produce different accuracies! The result is surprising and although I knew that the same FIT runs can produce different outcomes, I didn’t know that the outcome could be that different! And it is a good study for the fairness community to be aware of! \n\nThe paper is very well written and empirical analysis is very thorough ranging from different biases, different architectures, etc.\n\nSome suggestions (minor concerns):\nI think it would be great if you report how much accuracies vary (from previous work, e.g., 51), and compare it with fairness measures! Are fairness measures more brittle and experience a wider range of change? \nI think showing how much these measures vary with (1) different train/test split (2) different random seeds, and (3) due to software implementation, would be great (even over a few architectures to just get the sense of different components in variations.  I mean if (1) leads to much higher variation than (3) then studying (3) may not be very interesting. \nYou emphasis a lot on the fact that increasing fairness measures will lead to decrease accuracy which is a very well-known fact, I think emphasizing on increase in the variance of accuracy or variance of bias measure is more interesting. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "ABIL1JfZ0IU", "reviewer": "Reviewer_fhvk", "summary": "This paper investigates the impact of so-called implementation-level nondeterminism of machine learning frameworks on state-of-the-art fairness methods. Unlike algorithmic nondeterminism, which is caused by, e.g., dropout or random shuffling and can be rendered deterministic by fixing the random seed, implementation-level nondeterminism is a consequence of parallelism and non-associativity of floating-point arithmetic. Thus, even fixed-seed training can lead to different results, and prior work has shown that the accuracy can vary by up to 2.9% across identical runs. This paper empirically demonstrates that implementation-level nondeterminism also impacts many fairness methods and can lead to variations of up to 12.6% in standard fairness metrics.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** The paper is the first to empirically show that implementation-level nondeterminism can negatively impact state-of-the-art fairness methods. These findings are different from those of prior work by Friedler et al. (2019), which this paper needs to cite, showing that fairness methods are sensitive to different training-test splits. The related work is generally cited adequately, but Finding 6 should indicate which 32 papers were considered.\n\n**Quality:** The submission is technically sound. The paper uses various statistical tests to compute the significance of the results. However, the paper could improve the motivation for using these specific tests. For example, multiple tests for effect sizes based on the difference between means exist, but the paper only uses Cohen’s d. Moreover, the paper should repeat the experiments with different fixed seeds to see how the results are affected.\n\n**Clarity:** The paper is generally clearly written and well organized. However, I think the causes of implementation-level nondeterminism should be moved from Appendix E to the introduction to provide a better overview of the problem setting. In addition, there are some mistakes, such as missing words (e.g., L087 most (83.8%) *researchers* are, L103 label *which* should, L105 even *though* the) and typos (e.g., L658 Google *Imgae* Search).\n\n**Significance:** The results are meaningful to the fairness community as they elucidate the need for multiple fixed-seed runs when evaluating the merits of a novel method. However, as the paper mentions in Appendix E, this problem is also easily solvable using, e.g., PyTorch’s deterministic mode. In fact, the paper should conduct the same experiments using the deterministic mode for comparison. For this reason, I do not think that the current contribution is relevant enough to warrant publication at a top-tier conference.\n\n**Questions**\n\n- How does fixed-seed variance compare to the variance induced by varying the random seed?\n- Did you run the experiments sequentially for roughly ten months, as indicated on L164, or does this represent the total running time?\n\n**References**\n\nSorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary. Evan P. Hamilton, and Derek Roth. A comparative study of fairness-enhancing interventions in machine learning. FAT 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper adequately addressed the limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical issues with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "2yUiElaTM9l", "reviewer": "Reviewer_YRQu", "summary": "In this paper, the authors investigate the impact of software implementation on the fairness variance in the training of DL models. They perform an empirical study of DL fairness with 27 techniques (22 mitigation techniques and five baselines) on four popular datasets (CelebA, MS-COCO, imSitu, and CIFAR-10S) and three DL networks (ResNet-18, ResNet-50, and NIFR), measured by seven popular bias metrics. Their results show that models considered fair for one specific run might not be fair in another fixed-seed identical training run. There are up to 12.6% fairness variance across identical training runs with identical seeds. Their literature survey shows that while fairness is gaining popularity in AI conferences, only 34.4% of the papers use multiple identical training runs to evaluate their approach.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\n. Target an important problem in deep learning.\n\n. Extensive experiments.\n\n# Weaknesses\n\n- Novelty is a bit weak\n- Limited scope of the literature survey\n\nThis paper describes an empirical study to quantify the impact of software implementation on the fairness and its variance of DL systems. It targets an important problem of Deep learning (DL) systems, which demand fairness. The paper finds that DL software implementations introduce fairness variance: identical DL training runs with a fixed seed produce different results. Such variance could make DL models and networks violate fairness compliance laws, resulting in negative social impact. The authors performed extensive experiments: they evaluated the fairness variance of 27 techniques using seven bias metrics (a total of 189 experiments). Statistical significance of our findings were also analysed.\n\nThe novelty of this work is a bit weak as the impact of software implementation on the variance of DL systems has already been described in some recent work.  \n\nFor the software implementations, the paper did not consider the impact of various deep learning frameworks, \nsuch as Tensorflow, Pytorch, MXNet, etc. Do different DL frameworks affect your findings? Furthermore, there are also many other dependent software packages such as Python, SciPy, etc.\n\nThe literature survey only covered 32 papers on DL systems’ fairness, from a few conferences (NeurIPS, ICML, AAAI, CVPR, ICCV, ECCV, FAccT). It is not clear if these papers are complete and representative enough.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a very empirical paper that documents an interesting and surprising fact: fixed-seed identical training runs (FIT runs) can have surprisingly different fairness characteristics. 

I think this is an important fact that deserves to be known widely in the community and will likely be used to justify a lot of follow up work. The paper also includes a systematic literature survey to argue that this is both a little known fact and one that can affect the validity of some existing results. 

That said, while the specific focus on fairness is novel, and I don't disagree with the authors that more people need to be aware of the variance that can be introduced by various sources of implementation-level non-determinism, I think what the paper claims as implications of these results are mostly things that people should know to do already. For example, the suggestion that people should use statistical tests to check the validity of any proposed improvements is a good one, but it is not one that follows from this research. Given that training algorithms are fundamentally stochastic, applying statistical tests to confirm claimed improvements should be a requirement even if implementation-level non-determinism were not an issue. 
Decision: Accept (Poster)