Venue: NAACL 2021

# On the Impact of Random Seeds on the Fairness of Clinical Classifiers

Silvio Amir

Jan-Willem van de Meent and Byron C. Wallace

Northeastern University

{s.amir,j.vandemement,b.wallace}@northeastern.edu

###### Abstract

Recent work has shown that fine-tuning large networks is surprisingly sensitive to changes in random seed(s). We explore the implications of this phenomenon for model fairness across demographic groups in clinical prediction tasks over electronic health records (EHR) in MIMIC-III -- the standard dataset in clinical NLP research. Apparent subgroup performance varies substantially for seeds that yield similar overall performance, although there is no evidence of a trade-off between overall and subgroup performance. However, we also find that the small sample sizes inherent to looking at intersections of minority groups and somewhat rare conditions limit our ability to accurately estimate disparities. Further, we find that jointly optimizing for high overall performance and low disparities does not yield statistically significant improvements. Our results suggest that fairness work using MIMIC-III should carefully account for variations in apparent differences that may arise from stochasticity and small sample sizes.

+
Footnote †: _Corresponding of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3808–3823

June 6-11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Fine-tuning pre-trained transformers Vaswani et al. (2017) such as BERT Devlin et al. (2019) has become the dominant paradigm in NLP, owing to their performance across a range of downstream tasks. Clinical NLP -- in which we often aim to make predictions on the basis of notes in electronic health records (EHRs) -- is no exception Alsentzer et al. (2019). However, fine-tuning large networks is a stochastic process. Performance can vary considerably as a function of hyperparameter choice, and many parameter sets can yield the same validation accuracy (i.e., the model is not identifiable), and more generally the problem is _underspecified_D'Amour et al. (2020). Recent work has demonstrated that the choice of random seeds alone can have dramatic impact on model performance in NLP and beyond, even when all other hyper-parameters are kept fixed Phang et al. (2018); Dodge et al. (2020); D'Amour et al. (2020).

In this work, we explore the intersection of randomness and fairness in the context of clinical NLP. Fairness is a particularly acute concern in clinical predictive tasks, given the potential of such models to influence treatment decisions. This has motivated work investigating biases in predictive models trained over EHR Zhang et al. (2020); Chen et al. (2018, 2019, 2020a); Pfohl et al. (2020); Chen et al. (2020); Tripathi et al. (2020).

We investigate the impact of random seeds on the fairness of fine-tuned classifiers with respect to demographic characteristics such as gender and ethnicity. There are many definitions of algorithmic fairness which formalize different desired properties Mehrabi et al. (2019). Following prior work, here we adopt a simple measure: The _mean differences in model performance across demographic subgroups_Chen et al. (2019). We find that, on the popular MIMIC-III dataset Johnson et al. (2016), seeds with comparable validation performance can

Figure 1: Differences (\(\Delta\)s) with respect to overall performance as a function of random seeds for demographic subgroups on the _Shock_ phenotype classification task. Points represent results from pairs of seeds with similar (\(\leq\)0.01 difference in AUC) validation performance to the best seeds.



give rise to large variations in disparities across demographic subgroups (Figure 1).

## 2 Data and Methods

We investigate the variability of overall model performance and fairness across random seeds for a set of clinical prediction tasks derived from the Multiparameter Intelligence Monitoring in Intensive Care (MIMIC-III) set of Electronic Health Records (EHRs; Johnson et al., 2016). For each task, we train a classifier on top of the contextualized representations of a BERT Devlin et al. (2019) model pretrained over EHR data Alsentzer et al. (2019).

Following recent work exploring randomness and fine-tuning, we consider the seeds used to shuffle the training data and to initialize the model parameters independently Dodge et al. (2020). Specifically, we generate \(K=1000\) pairs of shuffling and initialization seeds by sampling from a uniform distribution \({\cal U}(0,10000)\). For each seed pair, we measure the overall performance as well as the performance for each demographic subgroup in terms of the Area Under the ROC Curve (AUC).

### Mimic-Iii

MIMIC-III is a database of deidentified EHR comprising over 40k patients admitted to the intensive care unit of the Beth Israel Deaconess Medical Center between 2001 and 2012 Johnson et al. (2016). It comprises structured variables including vital sign measurements, lab test results, and medications. It also contains clinical notes (e.g., doctor and nursing notes, radiology reports, and discharge summaries), which are the focus of our analysis.

MIMIC-III contains demographic information, including potentially sensitive attributes such as ethnicity/race, sex, spoken language, religion, and insurance status (which may be seen as a proxy for socioeconomic status Chen et al. (2019)). We are interested in the interaction between randomness and fairness in clinical predictions. Following recent prior work Zhang et al. (2020) on the latter, we focus our analyses on two benchmark tasks proposed by Harutyunyan et al. (2019):

In-hospital Mortality (IHM)Predict risk of inhomspital mortality based on the first 48 hours of an ICU stay.

Phenotype Classification (PC)Classify which of 25 acute or chronic conditions (e.g., acute cerebrovascular disease, chronic kidney disease) are present in a given patient ICU stay record. Similar to Zhang et al. (2020), we treat each condition as an independent binary classification task. Table A.1 in the Appendix enumerates the full set of conditions and their respective prevalences.

We extracted training and test datasets for these tasks using the same pre-processing pipeline as Zhang et al. (2020).1 We kept the same data splits and reserved 20% of the training data as validation set per task. For each patient, we collected their clinical notes, as well as their gender and race/ethnicity (as recorded in the EHR). The notes were filtered according to the categories _Nurse_, _Physician_ and _Nursing/Other_ to avoid notes of poor semantic quality, as suggested by Zhang et al. (2020). Patients without relevant clinical notes were discarded, resulting in \(11384\text{/}2591\) and \(22033\text{/}4919\) training/test examples for the IHM and PC tasks, respectively. It should be noted that these datasets are highly imbalanced both in terms of labels and demographic distribution with \(55\%\)_Male_, \(85\%\)_White_, \(9\%\)_Black_, \(3\%\)_Asian_ and \(3\%\)_Hispanic_ patients. Table 1 shows the distribution of sample sizes across subgroups for each benchmark.

Footnote 1: [https://github.com/MLforHealth/HurtfulWords](https://github.com/MLforHealth/HurtfulWords)

### Fine-tuned Classifiers

We define text classifiers for clinical tasks that map clinical notes corresponding to individual patients to binary labels. We extract contextualized embeddings from notes using a pretrained Transformer encoder and then map these to outputs (predictions) via a linear layer. Transformers are feedforward networks and require fixed-length inputs. To handle longer sequences, we adopt an approach used in prior works Huang et al. (2019); Zhang et al. (2020). Given an input sequence, we: (1) Extract \(N\) subsequences with sizes equal to that expected by the Transformer input layer; (2) Make individu 

predictions on the basis of each subsequence, and; (3) Then aggregate them into a final prediction.

More formally, an encoder \(\phi\) operates over inputs of size \(E\) with \(H\)-dimensional hidden layers. Given a patient's clinical notes \({\cal X}\), we extract a set of \(N\) subsequences of length \(E\),

\[x=\{\{w_{1}^{1},\ldots,w_{E}^{1}\},\ldots,\{w_{1}^{N},\ldots,w_{E}^{N}\}\}\subseteq{ \cal X}\]

We construct a matrix \({\bf Z}\in\mathbb{R}^{H\times N}\) such that the \(n\)th column represents subsequence \(x_{n}\), \({\bf Z}_{[:,{\bf n}]}=\phi(x_{n})=\sum_{j}{\bf z}_{j}^{x_{n}}\) where \({\bf z}_{j}^{x_{n}}\in\mathbb{R}^{H}\) is the embedding produced by the last hidden layer of the encoder for token \(j\) in the context of \(x_{n}\). We then use a linear layer followed by a sigmoid activation to produce a prediction vector \(\tilde{\mathbf{Y}}\), encoding the class conditional probabilities for each subsequence. This vector is then used to calculate the final probability as

\[P(Y=1|\tilde{\mathbf{Y}})=\frac{\tilde{\mathbf{Y}}_{\text{max}}+\tilde{ \mathbf{Y}}_{\text{mean}}N/c}{1+N/c}, \tag{1}\]

where \(c\) is a scaling factor, which we set to \(c=2\), following Huang et al. (2019).

We implement classifiers with PyTorch using the Transformer encoders from the huggingface2 library Wolf et al. (2019). We initialize models to weights from ClinicalBERT Huang et al. (2019), which was trained over scientific literature and clinical notes. We train classifiers on the most recent \(N=10\) subsequences of \(E=512\) tokens from the notes associated with each patient. We train using the ADAM optimizer Kingma and Ba (2014) for \(500\) epochs with early stopping. We set the learning rate to \(\alpha=0.01\), which we found to have the best validation performance on average across all tasks.

Footnote 2: [https://huggingface.co/](https://huggingface.co/)

## 3 Results

We compare the overall performance with the performance for each subgroup as a function of random seeds. Figure 2 shows the overall performance (left) along with the gap between the best and worst observed subgroup AUCs (right), across tasks. We observe a large variance in both the overall performance and the gap. The former observation corroborates previous findings Dodge et al. (2020).

To quantify how random seeds affect individual subgroups, we measure the the absolute differences (\(\Delta\)s) between overall and subgroup performances. We then evaluate whether there are correlations between overall performance and subgroup \(\Delta\)s. Figures 3 and 4 present the results for the _Shock_ phenotype classification task -- one of the tasks with largest disparities observed in prior work Zhang et al. (2020). Similar trends were found for the remaining tasks, and we report all results in the Appendix (Figures A.2-A.3 and A.4-A.6).

Figure 3 shows that the performance of all subgroups varies significantly across random seeds and that variances are higher for minority groups. Larger variations in minority subgroups are to be expected, as any empirical estimate will have a variance that is inversely proportional to the sample size of a group. In Figure 4, we observe that there seem to be two distinct clusters of seeds: One corresponding to high performing models (right of plots), and another to suboptimal models.3 While the best performing models tend to have a lower variance of subgroup performance, there is otherwise no clear relationship between overall and subgroup performance. Indeed, we find that many models with similar overall performance correspond to widely different subgroup \(\Delta\)s, particularly for the minority groups.

Footnote 3: Dodge et al. (2020) also found that some seeds performed consistently well across all the evaluated tasks, while others always performed poorly.

To explore the implications of this phenomenon, we simulate a grid search over all the random seeds on the validation set. We select the best seed along with all other seeds with similar performance (i.e., within a difference of \(\epsilon=0.01\) absolute AUC). Figure 1 shows the test set subgroup performance \(\Delta\)s, for the best validation seeds, in the _Shock_ phenotype classification task (see Figures A.7-A.8 for the other tasks). Figure 5 summarizes the overall performance (left) along with the subgroup performance gap (right) across tasks.

We can see that selecting seeds on the basis of overall performance helps to reduce the subgroup performance gap (compare the right subplots in Figures 2 and 5). However, the top performing models show disparities with respect to both gender and ethnicity, suggesting that these models maximize performance for some groups at the expense of others. Moreover, we find _multiple seeds with similar levels of validation performance that correspond to very different subgroup \(\Delta s\)_.

Since we have not encoded any model selection preferences into the pipeline this variance may reflect a form of _underspecification_. Can we define 

criteria that explicitly accounts for subgroup performance? We could then ask whether it is possible to maximize both fairness and overall performance with respect to random seeds. We repeated the grid search experiments with simple criteria that incorporate some notion of subgroup performance, such as selecting the seeds that: (a) maximize subgroup macro-average performance; (b) minimize the average subgroup \(\Delta\); and (c) maximize the overall performance minus the average subgroup \(\Delta\). To account for the effect the sample sizes on the apparent subgroup performance, we directly compare subgroup \(\Delta\)s for each random seed on the validation set and the test set. We find that correlations between validation set and test set fairness are either non-existent or very weak in most tasks.

These findings imply that the same pipeline may produce models with similar validation performance but very different levels of apparent 'fairness' as a result of varying the random seed alone. However, the fact that training-set and validation-set fairness are not reliable indicators of test-set fairness suggests that variance due to small subset sizes may be significant.

This is in some sense not surprising, given the combination of pronounced class imbalance and small subgroup samples in this data (see Tables 1 and A.1). To confirm this, we repeat the experiments on a subset of the test data containing the same number of examples (equal to the smallest subgroup) for _all_ groups, including majority groups. Evaluating all subgroups using small samples yields similarly high variances in performance \(\Delta\) (Figure 6 and Appendix Figure A.1), which con 

firms that the sample size is a significant factor in the variation of apparent model performance across random seeds.

These findings suggest that work investigating the fairness of fine-tuned classifiers should be careful to account for: a) model variability due the choice of random seeds; and b) variance in performance estimates due to small sample sizes. See Appendix Section B for an illustrative example. These observations are relevant for research using MIMIC-III, and for any corpora with similar properties, namely the combination of class imbalance and comparatively small subgroup sizes, which is likely to be present in EHR data where conditions are relatively rare and one is interested in fairness to minority groups (which are smaller by definition).

## 4 Conclusions

We have investigated the impact of random seeds on the fairness of fine-tuned pre-trained models for clinical tasks. Specifically, we measured gaps in performance across gender and racial subgroups as a function of the choice of random seeds for data shuffling and parameter initialization. In line with prior work, we found that classifiers trained on MIMIC-III data are often biased with respect to demographic subgroups. The contribution of this work is the empirical confirmation that choice of random seed alone significantly affects the apparent bias: Seeds that yield comparable performance in aggregate on the validation data correspond to very different performances on subgroups in test data. Our analyses corroborate Dodge et al. (2020)'s findings on the importance of carefully chosen random seeds, but also suggest that an equal amount of attention should be payed to the impact of these choices on model fairness.

However, interpretation of these results is complicated by sample size effects. While MIMIC-III is in itself a large dataset, it also exhibits significant imbalance, both in terms of subgroups of patients and the prevalence of medical conditions. These imbalances compound when considering subsets of patients in the context of specific prediction tasks, which often leads to small sample sizes for minority subgroups. While we observed higher apparent variances for demographic minorities, our results also suggest that these variances can in large part be explained by the smaller sample sizes. Indeed, we found the variances in subgroup performance to be inversely proportional to the size of the subgroup.

Figure 5: Variation of model performance for seeds with validation performance similar to the best seeds, for all tasks. _Left:_ Overall performance. _Right:_ Gap between best and worst subgroup.

Figure 6: Differences (\(\Delta\)s) estimated from a balanced subset of the test data with equal sample sizes for all demographic subgroups. As in Figure 3, we show deviations relative to the overall AUC for the _Shock_ phenotype classification task.

 



## Ethical Considerations

Fairness has rightly been an issue of increasing concern within the NLP community. This issue is particularly important in clinical NLP, given the potential that such models may ultimately have on patient health. We have investigated the degree to which different subgroup performances may be observed even fixing the (aggregate) validation data performance; we find wide variances across subgroups. That said, this work also highlights inherent limitations of using MIMIC-III (the standard dataset for clinical NLP) to evaluate the fairness of models, given the relatively small samples of patients that belong to demographic groups of interest. We hope these contributions encourage continued research into fairness in the context of clinical NLP.

## Acknowledgements

We would like to thank Darius Irani for his contribution in replicating the experiments from Zhang et al. (2020). This material is based upon work supported in part by the National Science Foundation under Grant No. 1901117.

## References

* A. Alsentzer, J. R. Murphy, W. Boag, W. Weng, D. Jin, T. Naumann, and M. McDermott (2019)Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323. Cited by: SS1.
* I. Chen, F. D Johansson, and D. Sontag (2018)Why is my classifier discriminatory? In Advances in Neural Information Processing Systems, pp. 3539-3550. External Links: Link Cited by: SS1.
* I. Y. Chen, E. Pierson, S. Rose, S. Joshi, K. Ferryman, and M. Ghassemi (2020a)Ethical machine learning in health. arXiv preprint arXiv:2009.10576. Cited by: SS1.
* I. Y. Chen, P. Szolovits, and M. Ghassemi (2019)Can ai help reduce disparities in general medical and mental health care?. AMA journal of ethics21 (2), pp. 167-179. Cited by: SS1.
* J. Chen, I. Berlot-Attwell, X. Wang, S. Hossain, and F. Rudzicz (2020b)Analyzing text specific vs blackbox fairness algorithms in multimodal clinical nlp. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, Vol. 301-312, Cited by: SS1.
* A. D'Amour, K. Heller, D. Moldovan, B. Atlapanahi, A. Beutel, C. Chen, J. Deaton, J. Eisenstein, M. D. Hoffman, et al. (2020)Underspecification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:2011.03395. Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186. Cited by: SS1.
* J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith (2020)Fine-tuning pretrained language models: weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305. Cited by: SS1.
* H. Harutyunyan, H. Khachatrian, D. C. Kale, G. V. Steeg, and A. Galstyan (2019)Multitask learning and benchmarking with clinical time series data. Scientific data6 (1), pp. 1-18. Cited by: SS1.
* K. Huang, J. Altosaar, and R. Ranganath (2019)Clinicalbert: modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342. Cited by: SS1.
* A. E. Johnson, T. J. Pollard, L. Shen, H. L. Li-Wei, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. Celi, and R. G. Mark (2016)Mimic-iii, a freely accessible critical care database. Scientific data3 (1), pp. 1-9. Cited by: SS1.
* D. P. Kingma and J. Ba (2014)Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: SS1.
* N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan (2019)A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635. Cited by: SS1.
* S. R. Pfohl, A. Forcyciarz, and N. H. Shah (2020)An empirical characterization of fair machine learning for clinical risk prediction. arXiv preprint arXiv:2007.10306. Cited by: SS1.
* J. Phang, T. Fevry, and S. R. Bowman (2018)Sentence encoders on tilts: supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. Cited by: SS1.
* S. Tripathi, B. A. Fritz, M. Abdelhack, M. S. Avidan, Y. Chen, and C. R. King (2020)(un)fairness in post-operative complication prediction models. arXiv preprint arXiv:2011.02036. Cited by: SS1.
* A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017)Attention is all

you need. In _Advances in neural information processing systems_, pages 5998-6008.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. _ArXiv_, pages arXiv-1910.
* Zhang et al. (2020) Haoran Zhang, Amy X Lu, Mohamed Abdalla, Matthew McDermott, and Marzyeh Ghassemi. 2020. Hurful words: quantifying biases in clinical contextual word embeddings. In _Proceedings of the ACM Conference on Health, Inference, and Learning_, pages 110-120.

 

Clinical Prediction Tasks and Results

Table A.1 shows all the clinical prediction tasks and the respective prevalence. The following plots present the results for the Phenotyping classification tasks. Figures A.2 and A.3 show the performance \(\Delta\)s for each subgroup and the overall performance, as a function of the random seeds. Figures A.4 to A.6 show the overall performance against the \(\Delta\) for each subgroup. Figures A.7 and A.8 show the subgroup performance \(\Delta\)s for pairs of seeds with validation performance similar to that of the best seeds.

## Appendix B Fine-tuning Experiments

To illustrate the impact that random seeds can have on measures of algorithmic fairness, we replicate the experiments concerning the fairness of fine-tuned clinical classifiers reported in Zhang et al. (2020)4. The study compares 3 measures of multigroup fairness (recall gap, parity gap and specificity gap) with respect to protected attributes such as the gender, language, ethnicity and insurance status. The experiments were conducted over 57 downstream classification tasks: including the IHM and PC tasks we considered in this paper, 3 additional tasks derived from logical ORs on subsets of the PC tasks, and a variation of PC using only the first note. Table 4 in Zhang et al. (2020) reports the number of tasks with statistically significant gaps and the percentage of significant tasks which favor a subgroup.

Footnote 4: see Section 5.2 and Tables 1 and 4.

We use the same experimental setup and implementation5 to replicate the experiments for IHM and PC using only the first note (29/57 tasks). We repeat each experiment with 100 different random seeds (using the same seed to shuffle the data and initialize parameters) and compute the mean and standard deviation of each measurement across seeds (Table B.1). We find that in general the number of tasks with significant differences is roughly half of those reported by Zhang et al. (2020), which was expected since we considered half of the tasks. However, we also observe that changes in a single random seed can affect the disparities across protected groups, both in terms of the number and the magnitude of the gaps. We see that there can be variations of up to two tasks with significant gaps and differences of up \(31\%\) in the percentage of tasks favoring a specific group.

Figure A.1: Correlations between overall performance and subgroup performance on the _Shock_ phenotype classification task, evaluated on a subset of the test data with equal sample sizes for all demographic subgroups.

 

Figure A.2: Differences relative to overall performance as a function of random seeds for each subgroup



Figure A.3: Differences relative to overall performance as a function of random seeds for each subgroup



* [26] M. A. Bader, J. S. Heng, and D. R. C. Smith. The design of the multi-objective optimization problem in a distributed environment. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1038-1049, 2015.
* [27] P. L. Chen, Y. Zhang, and X. Wang. An efficient algorithm for solving convex problems with applications to nonlinear programming. _Journal of Machine Learning Research_, 10:1-10, 2015.
* [28] G. E. Hinton, I. Goodfellow, and J. Pouget. Generative adversarial nets. In _Advances in neural information processing systems_, volume 1, pages 1038-1049, 2016.
* [29] K. He, Z. Li, and Q. Weinberger. Deep learning with deep convolutional networks. In _NIPS_, pages 1399-1405, 2016.
* [30] T. Hochreiter and J. Schmidhuber. Accelerating deep learning by reducing memory consumption. In _Advances in Neural Information Processing Systems (ICMLSP)_, pages 1-10, 2006.
* [31] W. Huang, X. Liu, and X. Zhu. Deep residual learning for image recognition. In _IEEE International Conference on Acoustics and Speech Technology_, pages 1-10, 2015.

[MISSING_PAGE_POST]

 

Figure A.5: Correlations between overall performance and subgroup performance \(\Delta\)



* [25] M. J. Bader, A. K. Sridharan, and R. C. Sutton, "A new approach to the problem of uncertainty estimation in a multi-objective optimization framework," _IEEE Transactions on Signal Processing Systems_**10**(4):637-689, 2000.
* [26] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [27] T. L. Nguyen, Y. Li, and H. Zhang, "Learning deep neural networks with applications to image recognition," _arXiv preprint arXiv:1512.03081_**, 2015.
* [28] G. Vinyals, I. Sutskever, and E. Rabinovich, "Deep learning with convolutions," in _Advances in Neural Information Processing Systems_, vol. 10, no. 1, p. 103002, 2016.
* [29] N. Srivastava, A. Shazeer, and A. Zisserman, "Dropout normalization for deep feedforward nets," in _Neural Networks & Learning Research_**2**(4):1053-1068, 2016.
* [30] F. Chollet, X. He, X. Liu, and W. Teh, "Batch normalization by jointly optimizing deep network models," in _NIPS Workshop on Machine Learning and Knowledge Discovery (MLML)_, pp. 1-10, 2015.

[MISSING_PAGE_POST]

 

Figure A.7: Differences relative to overall performance as a function of random seeds for each subgroup. Each point represents a run for a pair of seeds with validation performance similar to that of the best seeds.

 

Figure A.8: Differences relative to overall performance as a function of random seeds for each subgroup. Each point represents a run for a pair of seeds with validation performance similar to that of the best seeds.

 



\begin{table}
\begin{tabular}{c|r|c|c|c}  & **Recall Gap** & **Parity Gap** & **Specificity Gap** \\ \hline
**Gender** & Male vs. Female (\% Tasks Favoring Male) & \(3.0\pm 1.2~{}(67.8\pm 25.2\%)\) & \(11.2\pm 2.0~{}(39.0\pm 8.1\%)\) & \(9.5\pm 2.0~{}(76.8\pm 10.3\%)\) \\ \hline
**Language** & English vs. Other (\% Tasks Favoring Other) & \(3.1\pm 1.4~{}(50.3\pm 28.9\%)\) & \(6.9\pm 1.9~{}(5.4\pm 7.0\%)\) & \(4.2\pm 1.5~{}(88.9\pm 14.8\%)\) \\ \hline
**Ethnicity** & White vs. Other (\% Tasks Favouring White) & \(2.5\pm 1.5~{}(93.0\pm 22.0\%)\) & \(7.3\pm 1.7~{}(92.6\pm 10.7\%)\) & \(4.9\pm 1.5~{}(11.1\pm 12.1\%)\) \\  & Black vs. Other (\% Tasks Favouring Black) & \(3.8\pm 1.5~{}(37.9\pm 21.2\%)\) & \(6.6\pm 1.7~{}(65.6\pm 15.0\%)\) & \(3.6\pm 1.4~{}(40.8\pm 22.4\%)\) \\  & Hispanic vs. Other (\% Tasks Favouring Hispanic) & \(5.1\pm 1.6~{}(8.4\pm 10.9\%)\) & \(7.6\pm 1.8~{}(0.0\pm 0.0\%)\) & \(9.3\pm 1.8~{}(99.1\pm 8.9\%)\) \\  & Asian vs. Other (\% Tasks Favoring Asian) & \(6.1\pm 1.5~{}(53.6\pm 15.8\%)\) & \(2.3\pm 1.3~{}(77.1\pm 31.0\%)\) & \(3.6\pm 1.5~{}(54.3\pm 25.0\%)\) \\  & Other vs. Other (\% Tasks Favouring Order) & \(10.0\pm 1.7~{}(6.0\pm 4.9\%)\) & \(2.6\pm 1.1~{}(1.0\pm 5.0\%)\) & \(4.1\pm 1.2~{}(94.5\pm 12.5\%)\) \\ \hline
**Insurance** & Medicare vs. Other (\% Tasks Favoring Medicare) & \(15.0\pm 2.0~{}(93.8\pm 9.5\%)\) & \(25.7\pm 2.5~{}(92.2\pm 8.6\%)\) & \(23.9\pm 2.6~{}(2.9\pm 2.6\%)\) \\  & Private vs. Other (\% Tasks Favoring Private) & \(7.1\pm 1.4~{}(10.5\pm 9.0\%)\) & \(19.5\pm 2.2~{}(4.2\pm 3.3\%)\) & \(19.7\pm 2.4~{}(95.5\pm 9.0\%)\) \\  & Medicaid vs. Other (\% Tasks Favouring Medicaid) & \(9.0\pm 1.7~{}(8.7\pm 7.8\%)\) & \(17.2\pm 2.1~{}(12.0\pm 3.3\%)\) & \(15.0\pm 2.1~{}(92.8\pm 9.3\%)\) \\ \end{tabular}
\end{table}
Table B.1: We replicated Zhang et al. (2020) analysis of multi-group fairness performance gaps for fine-tuned classifiers across gender, language, ethnicity, and insurance status. We evaluated 28 (out of 57) tasks and repeated the experiments with 100 different random seeds. We measured the average and standard deviation of the number of tasks with statistically significant differences, and the percentage of significant tasks which favor a subgroup.