Are My Deep Learning Systems Fair? An Empirical Study of Fixed-Seed Training

1 Introduction . DL systems , which consist of DL models and DL software ( for example libraries such as TensorFlow ) [ 71 ] , are widely used ; and enrusing their fairness is of paramount importance [ 47 , 59 , 97 , 98 ] . Biases in DL systems have been identified as a major concern for civil rights by the U.S. government [ 27 ] . Yet , fairness testing for DL systems is still an open challenge . Although the DL community is aware of the nondeterminism caused by using different random seeds [ 72 , 90 ] , such algorithmic factors are not the sole source of DL nondeterminism . Even with the ∗Jiahao Chen has since moved to Parity Technologies . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . same seed , e.g. , the same initial weights , DL systems are nondeterministic , i.e. , multiple training runs with the same network configuration , the same dataset , and the same seed , referred to as fixed-seed identical training runs ( FIT runs ) , produce different models with different accuracies [ 71 ] . Such nondeterminism is caused by software implementation , e.g. , software parallelism and floating-point computation of DL software ( details in Appendix E.1 ) . Our previous work [ 71 ] shows that software implementation alone causes models ’ accuracies to vary by up to 2.9 % , and most ( 83.8 % ) of the DL researchers and practitioners surveyed are unaware or unsure about the nondeterminism caused by software implementation . The small portion that is aware of such implemental variance underestimates it . The survey participants are researchers and practitioners with DL experience from universities and AI companies including Google , NVIDIA , Microsoft , and Facebook . These results suggest that even with fixed-seed identical training runs , one may still need to report error bars or use statistical tests . Since the focus of the prior study [ 71 ] was on the variance of accuracy and training time only , it is unknown how DL software implementation affects the variance of fairness . Various debiasing algorithms [ 3 , 47 , 59 , 98 , 99 ] optimize DL models toward a specific fairness goal and it is unclear how software implementation affects these debiasing algorithms . In addition , unlike accuracy , which is a widely-accepted metric , dozens of fairness metrics exist [ 29 , 35 , 46 , 60 , 64 , 99 , 108 ] . It is unclear if software implementation causes different levels of variance for different fairness metrics . In this paper , we study the fairness variance caused by software implementation in training DL networks . Overall , we found that fairness variance is large—up to 12.6 % difference ( 27.3 % versus 39.9 % ) across 16 fixed-seed identical training runs . First , our results show that one training run may produce a fair model but another fixed-seed identical training run may generate an unfair one . For example , we reproduced previous work [ 47 ] that proposed a new network and a debiasing algorithm and we found that one FIT run generates a fair model ( i.e. , a model that does not discriminate based on gender ) based on the threshold ( 20 % ) for the normalized disparate impact ( DI ) metric [ 29 , 64 ] ( DI of 17.9 % ) . A DI of 0 % is the fairest , while a 100 % DI is the least fair . However , a second FIT run produces an unfair model ( DI of 20.4 % , above the 20 % threshold ) . This 20 % threshold for DI has been used in a U.S. legal case to determine a system ’ s fairness [ 9 ] , where fairness variance could have changed the outcome . Second , the variance of fairness imposes challenges for the valid evaluation of debiasing algorithms . For example , our experiments show that while a debiasing algorithm [ 107 ] reduces the bias amplification ( BA ) [ 99 , 108 ] to 6.1 % in one FIT run ( compared with a BA score of 7.8 % for its baseline ) , this technique increases the bias amplification instead of reducing it when we check the average of 16 FIT runs : an average BA score of 8.7 % compared to an average BA score of 7.4 % for the baseline . The Mann-Whitney U-test shows the results from the 16 runs are statistically significant . The observed variance across 16 FIT runs is solely caused by software implementation . Finally , many mitigation techniques increase fairness variance or hurt accuracy . With Mann-Whitney U-test and Levene ’ s test , our experiments show that a majority ( 68.0 % ) of the mitigation techniques reduce biases with a cost : they increase fairness variance , increase accuracy variance , or decrease model accuracy . Since existing debiasing algorithms [ 47 , 59 ] often do not measure the variance of biases , they could introduce risks . When such debiasing algorithms are applied to online learning , where DL models are trained continuously [ 53 ] , the increased variance from debiasing algorithms could cause the resulting models ’ fairness to miss compliance requirements or service-level agreements . For example , Microsoft Tay [ 86 ] became a `` racist chatbot '' only after a few hours of deployment . What ’ s worse , even with fairness auditing before deployment , which is still not a standard practice in the industry [ 39 ] , a model can be fair on pre-deployment data , but unfair on real-world data , due to the gap between training/test data and real-world data . The fairness variance puts the trained models at a greater risk of being unfair on real-world data . Theoretically , while debiasing techniques on average improve fairness on the training set , they increase the variance of fairness on the test set [ 2 ] . In practice , such gaps are the key reason that facial recognition systems [ 58 , 76 ] have a racial bias . In addition to quantifying fairness variance , we conduct a survey of research papers from top artificial intelligence ( AI ) conferences of recent years ( 2016–2020 ) to understand how researchers handle the fairness variance caused by DL software implementations . Only 34.4 % of the 32 papers evaluating or addressing DL fairness issues use multiple training runs in their evaluation . To the best of our knowledge , we are the first to study software implementations ’ impact on DL fairness variance . The contributions are : • A variance study of DL fairness of 27 techniques ( 22 mitigation techniques and five baselines ) . The experiments are performed on four popular datasets ( CelebA , MS-COCO , imSitu , and CIFAR-10S ) with three DL networks ( ResNet-18 , ResNet-50 , and NIFR [ 47 ] ) , measured by seven popular bias metrics ( Section 3 ) . – Finding 1 : Across 16 FIT runs , software implementation causes a maximum difference in bias metrics of up to 12.6 % in the Demographic Parity ( DP ) metric and up to 11.8 % in the DI metric . Yet , most ( 83.8 % ) researchers and practitioners are unaware or unsure about the DL nondeterminism caused by software implementation [ 71 ] . – Finding 2 : The effectiveness of the debiasing algorithms can be inaccurate with only one fixed-seed training run . – Finding 3 : Most of ( 15 out of 22 ) the mitigation techniques tested increase at least one of the bias metrics . – Finding 4 : In a majority ( 68.0 % , or 70 ) of 103 experiments , debiasing algorithms have a negative impact on the model . • A literature survey on the fairness research papers of top machine learning ( ML ) , ( NeurIPS/NIPS , ICML , AAAI , and FAccT ) , and computer vision ( CVPR , ICCV , and ECCV ) conferences ( Section 4 ) . • Implications and suggestions for DL fairness researchers and practitioners ( Section 5 ) , including one should use FIT runs ( setup details in Section 2.5 ) with statistical tests ( Appendix B.3 ) to evaluate debiasing algorithms , when nondeterminism from DL software is unavoidable . Data and code availability : Experiment data and artifact for reproducibility study are available in a public GitHub repository1 . 2 Experimental method . We investigate 27 techniques of DL fairness on four popular datasets with three networks by reproducing four previous works [ 47 , 59 , 98 , 99 ] that fit our scope . 2.1 Scope . We focus on the impact of DL software implementation on the variance of group fairness [ 26 ] in classification tasks , where two orthogonal labels are assigned to each instance in the dataset , a task label for training a classifier , and a protected label that should not be discriminated against ( e.g. , gender ) . The set of instances with the same protected label forms a protected group . Although the protected labels are not used in the classification task , the trained models could still be biased because the protected labels can be leaked into the prediction by proxy variables [ 3 , 59 , 98 , 108 ] . 2.2 Bias metrics . Various metrics exist to measure biases of ML algorithms . Yet , it is still an open problem to select these metrics as an optimization goal for real-world problems [ 49 ] . Therefore , we choose three fairness conditions and seven bias metrics derived from them . Fairness conditions include statistical parity [ 26 ] ( SP ) , predictive equality [ 19 ] ( PE ) , and equality of opportunity [ 35 ] ( EOp ) . Seven bias metrics include demographic parity [ 15 ] ( DP ) , normalized disparate impact [ 15 ] ( DI ) , statistical parity subgroup fairness [ 46 ] ( SPSF ) , false positive subgroup fairness [ 46 ] ( FPSF ) , equalized odds false positive [ 24 ] ( EOFP ) , equalized odds true positive [ 35 ] ( EOTP ) , and bias amplification [ 99 , 108 ] ( BA ) . For all bias metrics , 0 indicates the fairness condition is satisfied , i.e. , a completely fair model , and 1 indicates a completely unfair model . Details of these metrics are in Appendix A . 1https : //github.com/lin-tan/fairness-variance/ Per-class and overall bias : Bias calculated on each class is called per-class bias and the bias for each model is the overall bias . For example , there are ten per-class biases in models trained on CIFAR-10 ( one per class ) . The overall bias of the model is the average of all per-class biases . Multi- ( task ) label classification : We treat each task label as an independent binary classification , and the overall bias value of the model is the average across all labels . 2.3 Reproduced studies . We conduct a literature survey of published papers from the recent five years ( 2016–2020 ) of the top ML ( NeurIPS/NIPS , ICML , AAAI , and FAccT ) , and computer vision ( CVPR , ICCV , and ECCV ) conferences ( Section 4 ) . Out of the 32 papers relevant to the fairness of DL systems , only four [ 47 , 59 , 98 , 99 ] have a working replication package so that we can reproduce their results . These papers study the fairness of DL systems for the five tasks described in Table 1 ( expanded details in Appendix B.1 ) . Overall , we replicate 27 different techniques from four state-of-the-art papers on five classification tasks . In the next section , we introduce each technique and detail our experiments .

Summary: The paper did a thorough analysis of the effect of software implementation (e.g., floating-point and parallelism) on bias measures. It has been previously shown that FIT runs produce different accuracies! The result is surprising and although I knew that the same FIT runs can produce different outcomes, I didn’t know that the outcome could be that different! And it is a good study for the fairness community to be aware of!