Are My Deep Learning Systems Fair? An Empirical Study of Fixed-Seed Training

1 Introduction . DL systems , which consist of DL models and DL software ( for example libraries such as TensorFlow ) [ 71 ] , are widely used ; and enrusing their fairness is of paramount importance [ 47 , 59 , 97 , 98 ] . Biases in DL systems have been identified as a major concern for civil rights by the U.S. government [ 27 ] . Yet , fairness testing for DL systems is still an open challenge . Although the DL community is aware of the nondeterminism caused by using different random seeds [ 72 , 90 ] , such algorithmic factors are not the sole source of DL nondeterminism . Even with the ∗Jiahao Chen has since moved to Parity Technologies . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . same seed , e.g. , the same initial weights , DL systems are nondeterministic , i.e. , multiple training runs with the same network configuration , the same dataset , and the same seed , referred to as fixed-seed identical training runs ( FIT runs ) , produce different models with different accuracies [ 71 ] . Such nondeterminism is caused by software implementation , e.g. , software parallelism and floating-point computation of DL software ( details in Appendix E.1 ) . Our previous work [ 71 ] shows that software implementation alone causes models ’ accuracies to vary by up to 2.9 % , and most ( 83.8 % ) of the DL researchers and practitioners surveyed are unaware or unsure about the nondeterminism caused by software implementation . The small portion that is aware of such implemental variance underestimates it . The survey participants are researchers and practitioners with DL experience from universities and AI companies including Google , NVIDIA , Microsoft , and Facebook . These results suggest that even with fixed-seed identical training runs , one may still need to report error bars or use statistical tests . Since the focus of the prior study [ 71 ] was on the variance of accuracy and training time only , it is unknown how DL software implementation affects the variance of fairness . Various debiasing algorithms [ 3 , 47 , 59 , 98 , 99 ] optimize DL models toward a specific fairness goal and it is unclear how software implementation affects these debiasing algorithms . In addition , unlike accuracy , which is a widely-accepted metric , dozens of fairness metrics exist [ 29 , 35 , 46 , 60 , 64 , 99 , 108 ] . It is unclear if software implementation causes different levels of variance for different fairness metrics . In this paper , we study the fairness variance caused by software implementation in training DL networks . Overall , we found that fairness variance is large—up to 12.6 % difference ( 27.3 % versus 39.9 % ) across 16 fixed-seed identical training runs . First , our results show that one training run may produce a fair model but another fixed-seed identical training run may generate an unfair one . For example , we reproduced previous work [ 47 ] that proposed a new network and a debiasing algorithm and we found that one FIT run generates a fair model ( i.e. , a model that does not discriminate based on gender ) based on the threshold ( 20 % ) for the normalized disparate impact ( DI ) metric [ 29 , 64 ] ( DI of 17.9 % ) . A DI of 0 % is the fairest , while a 100 % DI is the least fair . However , a second FIT run produces an unfair model ( DI of 20.4 % , above the 20 % threshold ) . This 20 % threshold for DI has been used in a U.S. legal case to determine a system ’ s fairness [ 9 ] , where fairness variance could have changed the outcome . Second , the variance of fairness imposes challenges for the valid evaluation of debiasing algorithms . For example , our experiments show that while a debiasing algorithm [ 107 ] reduces the bias amplification ( BA ) [ 99 , 108 ] to 6.1 % in one FIT run ( compared with a BA score of 7.8 % for its baseline ) , this technique increases the bias amplification instead of reducing it when we check the average of 16 FIT runs : an average BA score of 8.7 % compared to an average BA score of 7.4 % for the baseline . The Mann-Whitney U-test shows the results from the 16 runs are statistically significant . The observed variance across 16 FIT runs is solely caused by software implementation . Finally , many mitigation techniques increase fairness variance or hurt accuracy . With Mann-Whitney U-test and Levene ’ s test , our experiments show that a majority ( 68.0 % ) of the mitigation techniques reduce biases with a cost : they increase fairness variance , increase accuracy variance , or decrease model accuracy . Since existing debiasing algorithms [ 47 , 59 ] often do not measure the variance of biases , they could introduce risks . When such debiasing algorithms are applied to online learning , where DL models are trained continuously [ 53 ] , the increased variance from debiasing algorithms could cause the resulting models ’ fairness to miss compliance requirements or service-level agreements . For example , Microsoft Tay [ 86 ] became a `` racist chatbot '' only after a few hours of deployment . What ’ s worse , even with fairness auditing before deployment , which is still not a standard practice in the industry [ 39 ] , a model can be fair on pre-deployment data , but unfair on real-world data , due to the gap between training/test data and real-world data . The fairness variance puts the trained models at a greater risk of being unfair on real-world data . Theoretically , while debiasing techniques on average improve fairness on the training set , they increase the variance of fairness on the test set [ 2 ] . In practice , such gaps are the key reason that facial recognition systems [ 58 , 76 ] have a racial bias . In addition to quantifying fairness variance , we conduct a survey of research papers from top artificial intelligence ( AI ) conferences of recent years ( 2016–2020 ) to understand how researchers handle the fairness variance caused by DL software implementations . Only 34.4 % of the 32 papers evaluating or addressing DL fairness issues use multiple training runs in their evaluation . To the best of our knowledge , we are the first to study software implementations ’ impact on DL fairness variance . The contributions are : • A variance study of DL fairness of 27 techniques ( 22 mitigation techniques and five baselines ) . The experiments are performed on four popular datasets ( CelebA , MS-COCO , imSitu , and CIFAR-10S ) with three DL networks ( ResNet-18 , ResNet-50 , and NIFR [ 47 ] ) , measured by seven popular bias metrics ( Section 3 ) . – Finding 1 : Across 16 FIT runs , software implementation causes a maximum difference in bias metrics of up to 12.6 % in the Demographic Parity ( DP ) metric and up to 11.8 % in the DI metric . Yet , most ( 83.8 % ) researchers and practitioners are unaware or unsure about the DL nondeterminism caused by software implementation [ 71 ] . – Finding 2 : The effectiveness of the debiasing algorithms can be inaccurate with only one fixed-seed training run . – Finding 3 : Most of ( 15 out of 22 ) the mitigation techniques tested increase at least one of the bias metrics . – Finding 4 : In a majority ( 68.0 % , or 70 ) of 103 experiments , debiasing algorithms have a negative impact on the model . • A literature survey on the fairness research papers of top machine learning ( ML ) , ( NeurIPS/NIPS , ICML , AAAI , and FAccT ) , and computer vision ( CVPR , ICCV , and ECCV ) conferences ( Section 4 ) . • Implications and suggestions for DL fairness researchers and practitioners ( Section 5 ) , including one should use FIT runs ( setup details in Section 2.5 ) with statistical tests ( Appendix B.3 ) to evaluate debiasing algorithms , when nondeterminism from DL software is unavoidable . Data and code availability : Experiment data and artifact for reproducibility study are available in a public GitHub repository1 . 2 Experimental method . We investigate 27 techniques of DL fairness on four popular datasets with three networks by reproducing four previous works [ 47 , 59 , 98 , 99 ] that fit our scope . 2.1 Scope . We focus on the impact of DL software implementation on the variance of group fairness [ 26 ] in classification tasks , where two orthogonal labels are assigned to each instance in the dataset , a task label for training a classifier , and a protected label that should not be discriminated against ( e.g. , gender ) . The set of instances with the same protected label forms a protected group . Although the protected labels are not used in the classification task , the trained models could still be biased because the protected labels can be leaked into the prediction by proxy variables [ 3 , 59 , 98 , 108 ] . 2.2 Bias metrics . Various metrics exist to measure biases of ML algorithms . Yet , it is still an open problem to select these metrics as an optimization goal for real-world problems [ 49 ] . Therefore , we choose three fairness conditions and seven bias metrics derived from them . Fairness conditions include statistical parity [ 26 ] ( SP ) , predictive equality [ 19 ] ( PE ) , and equality of opportunity [ 35 ] ( EOp ) . Seven bias metrics include demographic parity [ 15 ] ( DP ) , normalized disparate impact [ 15 ] ( DI ) , statistical parity subgroup fairness [ 46 ] ( SPSF ) , false positive subgroup fairness [ 46 ] ( FPSF ) , equalized odds false positive [ 24 ] ( EOFP ) , equalized odds true positive [ 35 ] ( EOTP ) , and bias amplification [ 99 , 108 ] ( BA ) . For all bias metrics , 0 indicates the fairness condition is satisfied , i.e. , a completely fair model , and 1 indicates a completely unfair model . Details of these metrics are in Appendix A . 1https : //github.com/lin-tan/fairness-variance/ Per-class and overall bias : Bias calculated on each class is called per-class bias and the bias for each model is the overall bias . For example , there are ten per-class biases in models trained on CIFAR-10 ( one per class ) . The overall bias of the model is the average of all per-class biases . Multi- ( task ) label classification : We treat each task label as an independent binary classification , and the overall bias value of the model is the average across all labels . 2.3 Reproduced studies . We conduct a literature survey of published papers from the recent five years ( 2016–2020 ) of the top ML ( NeurIPS/NIPS , ICML , AAAI , and FAccT ) , and computer vision ( CVPR , ICCV , and ECCV ) conferences ( Section 4 ) . Out of the 32 papers relevant to the fairness of DL systems , only four [ 47 , 59 , 98 , 99 ] have a working replication package so that we can reproduce their results . These papers study the fairness of DL systems for the five tasks described in Table 1 ( expanded details in Appendix B.1 ) . Overall , we replicate 27 different techniques from four state-of-the-art papers on five classification tasks . In the next section , we introduce each technique and detail our experiments .

Summary: In this paper, the authors investigate the impact of software implementation on the fairness variance in the training of DL models. They perform an empirical study of DL fairness with 27 techniques (22 mitigation techniques and five baselines) on four popular datasets (CelebA, MS-COCO, imSitu, and CIFAR-10S) and three DL networks (ResNet-18, ResNet-50, and NIFR), measured by seven popular bias metrics. Their results show that models considered fair for one specific run might not be fair in another fixed-seed identical training run. There are up to 12.6% fairness variance across identical training runs with identical seeds. Their literature survey shows that while fairness is gaining popularity in AI conferences, only 34.4% of the papers use multiple identical training runs to evaluate their approach.