MEDFAIR: Benchmarking Fairness for Medical Imaging
Authors: Yongshuo Zong
Yongxin Yang
Timothy Hospedales
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Fairness
Bias Mitigation
Medical Imaging
Benchmark

Abstract: A multitude of work has shown that machine learning-based medical diagnosis systems can be biased against certain subgroups of people. This has motivated a growing number of bias mitigation algorithms that aim to address fairness issues in machine learning. However, it is difficult to compare their effectiveness in medical imaging for two reasons. First, there is little consensus on the criteria to assess fairness. Second, existing bias mitigation algorithms are developed under different settings, e.g., datasets, model selection strategies, backbones, and fairness metrics, making a direct comparison and evaluation based on existing results impossible. In this work, we introduce MEDFAIR, a framework to benchmark the fairness of machine learning models for medical imaging. MEDFAIR covers eleven algorithms from various categories, ten datasets from different imaging modalities, and three model selection criteria. Through extensive experiments, we find that the under-studied issue of model selection criterion can have a significant impact on fairness outcomes; while in contrast, state-of-the-art bias mitigation algorithms do not significantly improve fairness outcomes over empirical risk minimization (ERM) in both in-distribution and out-of-distribution settings. We evaluate fairness from various perspectives and make recommendations for different medical application scenarios that require different ethical principles. Our framework provides a reproducible and easy-to-use entry point for the development and evaluation of future bias mitigation algorithms in deep learning. Code is available at https://github.com/ys-zong/MEDFAIR.

Published as a conference paper at ICLR 2023
MEDFAIR: BENCHMARKING FAIRNESS FOR MEDICAL IMAGING
Yongshuo Zong', Yongxin Yang', Timothy Hospedales
1 School of Informatics, University of Edinburgh, 2 Samsung AI Centre, Cambridge {y (yongshuo. zong, yongxin. yang, t hoospedaless | @ed aetedaac... ac. uk
ABSTRACT
A multitude of work has shown that machine learning-based medical diagnosis systems can be biased against certain subgroups of people. This has motivated growing number of bias mitigation algorithms that aim to address faimess is- sues machine learning. However, is difficult to compare their effectiveness medical imaging for two reasons. First, there is little consensus the criteria assess fairness. Second, existing bias mitigation algorithms are developed under different settings, e-g., datasets, model selection strategies, backbones, and fair- ness metrics, making a direct comparison and evaluation based on existing results impossible. In this work, we introduce MEDFAIR, a framework to benchmark the fairness of machine learning models for medical imaging. MEDFAIR cov- ers eleven algorithms from various categories, ten datasets from different imaging modalities, and three model selection criteria. Through extensive experiments, we find that the under-studied issue of model selection criterion can have a signifi- cant impact on fairness outcomes; while in contrast, of-he-oft bias mitiga- tion algorithms do not significantly improve fairness outcomes over empirical risk minimization (ERM) in both in-distribution and evaluate fairness from various perspectives and make recommendations for dif- ferent medical application scenarios that require different ethical principles. Our framework provides a reproducible and easy-to-use entry point for the develop- ment and evaluation of future bias mitigation algorithms in deep learning. Code is available at https: / /ggtttb...././///g////gg com/ ys zong/ Bong/MEDD /M///MEDFARR
settings. We
INTRODUCTION
Machine learning-enabled automatic diagnosis with medical imaging is becoming vital part of the current healthcare system (Lee et al., 2017). However, machine learning (ML) models have been found to demonstrate systematic bias toward certain groups of people defined by race, gender, age, and even the health insurance type with worse performance (Obermeyer et al., 2019; Larrazabal al., 2021). The bias also exists in models
al., 2020; Spencer et al., 2013; Seyyed-Kalantari trained from different types of medical data, such as chest X-rays (Seyyed-Kalantal Kalantai et al., 2020), CT scans (Zhou al., 2021), skin dermatology images (Kinyanjui al., 2020), etc. A biased decision-making system is socially and ethically detrimental, especially in life-changing scenarios such as healthcare. This has motivated a growing body of work understand bias and pursue fairness in the areas of machine learning and computer vision (Mehrabi et al., 2021; Louppe et al., 2017: Tartaglione et al., 2021; Wang al., 2020).
Informally, given an observation input x (e.g., skin dermatology image), a sensitive attribute s (e.g., male female), and a target y (e.g., benign or malignant), the goal of diagnosis model to learn a meaningful mapping from x to y. However, MI models may amplify the biases and con- founding factors that already exist in the training data related to sensitive attribute s. For example, data imbalance (e.g., over 90% individuals from UK Biobank (Sudlow et al., 2015) originate from European ancestries), attribute-class imbalance (e.g., in age-related macular degeneration (AMD) datasets, subgroups of older people contain more pathology examples than that of younger people (Farsiu et al., 2014)), label noise (e.g., Zhang et al. (2022) find that label noises in CheXpert dataset (Irvin et al., 2019) is much higher in some subgroups than the others), etc. Bias mitigation algorithms

---

Published as a conference paper at ICLR 2023
Algorithm
Dataset
Subgroup Rebalancing
Diverse Imaging Modalities
Domainn-ndeeendent
Sensitive Attributes
Adversarial Training
Different Sources Blas
Disentanglement
Diverse Sizes
Domain Generalization
Paired OOD
MEDFAIR
Evaluation
Model Selection
In-distribution
Performance-
Out-of-distriuttion
Distance to Optimal (DTO)
Statistical Tests
Minimax Pareto Optimal
Figure 1: Components of MEDFAIR benchmark.
therefore aim to help diagnosis algorithms learn predictive models that are robust to confounding factors related to sensitive attribute s (Mchrabi et al., 2021).
Given the importance of ensuring fairness in medical applications and the special characteristics of medical data, we argue that a systematic and rigorous benchmark is needed to evaluate the bias mitigation algorithms for medical imaging. However, straightforward comparison of algorithmic fairness for medical imaging is difficult, as there is no consensus on a single metric for fairness of medical imaging models. Group fairness (Dwork et al., 2012; Verma & Rubin, 2018 8) is a popular and intuitive definition adopted by many debiasing algorithms, which optimises for equal perfor- mance among subgroups. However, this can lead to trade-off of increasing fairness by decreasing the performance of the advantaged group, reducing overall utility substantially. Doing so may vi- olate the ethical principles of beneficence and non-maleficence (Beauchamp, 2003), especially for some medical applications where all subgroups need to be protected. There are also other fairness definitions, including individual fairness (Dwork al., 2012), minimax fairesss (Diana al., 2021), counterfactual fairness (Kusner et al., 2017), etc. It is thus important to consider which definition should be used for evaluations.
In addition to the use of differing evaluation metrics, different experimental designs used by existing studies prevent direct comparisons between algorithms based on the existing literature. Most obvi- ously, each study tends to different datasets to evaluate their debiasing algorithms, preventing direct comparisons of results. Furthermore, many bias mitigation studies focus on evaluating tabular data with low-capacity models (Madras et al., 2018 8; Zhao et al., 2019; Diana et al., 2021), and recent analysis has shown that their conclusions do not generalise to high-capacity deep networks used for the analysis of image data (Zietlow al., 2022). A crucial but less obvious issue is the choice of model selection strategy for hyperparameter search and early stopping. Individual bias mitigation studies are divergent or vague their model selection criteria. leading inconsistent comparisons even if the same datasets are used. Finally, given the effort required to collect and annotate medi- cal imaging data, models are usually deployed in different domain than the domain used for data collection. (E.g., data collected at hospital A is used to train model deployed at hospital B). While the maintenance of prediction quality across datasets has been well studied, is unclear fairness achieved within one dataset (in-distribution) holds under dataset shift (out-of-distributtin))
In order to address these challenges, we provide the first comprehensive fairness benchmark for medical imaging MEDFAIR. We conduct extensive experiments across eleven algorithms, ten datasets, four sensitive attributes, and three model selection strategies to assess bias mitigation algo- rithms in both in-distribution and -iito---istributionn settings. We report multiple evaluation metrics and conduct rigorous statistical tests to find whether any of the algorithms is significantly better. Having trained over 7,000 models using 6,800 GPU-hours, we have the following observations:
Bias widely exists ERM models trained different modalities, which is reflected in the predictive performance gap between different subgroups for multiple metrics.
2

---

Published as a conference paper at ICLR 2023
Model selection strategies can play important role in improving the worst-case perfor- mance. Algorithms should be compared under the same model selection criteria.
The state-of-the-arr methods do not outperform the ERM with statistical significance in settings.
both in-distribution and
These results show the importance of large benchmark suite such as MEDFAIR to evaluate progress in the field and to guide practical decisions about the selection of bias mitigation algo- rithms for deployment. MEDFAIR is released as reproducible and easy-to-use codebase that all experiments in this study can be run with a single command. Detailed documentation is provided in order to allow researchers to extend and evaluate the faimess of their algorithms and datasets, and we will also actively maintain the codebase to incorporate more algorithms, datasets, model selection strategies, etc. We hope codebase can accelerate the development of bias mitigation algorithms and guide the deployment of ML models in clinical scenarios.
2 FAIRNESS IN MEDICINE
2.1 PROBLEM FORMULATION
We focus on evaluating the fairness of binary classification of medical images. Given an image, we predict its diagnosis label in a way that is not confounded by any sensitive attributes (age, sex, race, etc.) so that the trained model is fair and not biased towards a certain subgroup of people.
Formally, Let D E {D:} be set of domains, where / is the total number of domains. A domain can represent a dataset collected from a particular imaging modality, hospital, population, etc. Consider domain D ()) to be distribution where we have input sample E Rdd over input space X, the corresponding binary label E (0, } over label space V and sensitive attributes -1 with m classes over sensitive space S. We train model h H to output the prediction 9 E {0. 1}, ie., h X' + y. where H. is the hypothesis class of models. Note that for each dataset D., there may exist several sensitive attributes the same time, e.g., there are metadata of both patients' age and sex. We only consider one sensitive attribute at one time.
E {0, 1, ...,, ,..,
In-distribution Given a domain D,, assume the input samples X,, their labels Y,, and the sensi- tive attributes S; are identically and independently distributed (iid) from joint probability distribu- tion P, (X,,, Y,, St). We define the evaluation where the training and testing on the same domain D, to be in-distribution, i.e., the training and testing set are from the same distribution. We train models for each combination of algorithms x datasets x sensitive attributes.
In clinical scenarios, due to the lack of training data, it is common to deploy a model trained in the original dataset to new nospitals/population that have different data distri- butions. We define the training on one domain D; and testing on the other unseen domain D; to be out-of-disttibution settings, where Dj and D, may have different distribution P; Y,, S:) and (X Y,, Sy). In this case, assume domains D; and D, must have the same input space (e.g., X-ray imaging), diagnosis labels, and sensitive attributes, but differ in their joint distributions due to collection from different locations different imaging protocols. We evaluate if bias mitigation al- gorithms are robust to distribution shift by directly using the model selected from the in-distribution setting of the domain D; test on the domain Dj-.
2.2 FAIRNESS DEFINITION IN MEDICINE
Here consider most salient fairness definitions for healthcare, i.e., group faimess and Max- Min faimess. We argue that one should focus on different fairness definitions depending on the specific clinical application.
Group Fairness Metrics based on group fairness usually aim to achieve parity of predictive per- formance across protected subgroups. For resource allocation problems that be considered zero-sum game due to the limited resources, e.g., prioritising which patients should be sent to a limited number of intensive care units (ICUs), it is important consider group fairness reduce the disparity among different subgroups (related discussions in Hellman (2008); Barocas & Selbst
Our framework can be easily extended non-binary classification.
3

---

Published as a conference paper at ICLR 2023
(2016)). We measure the performance gap in diagnosis AUC between the advantaged and disad- vantaged subgroups as an indicator of group faimess. This is in line with the "separability" criteria (Chen et al., 2021; Dwork et al., 2012) that algorithm scores should be conditionally independent of the sensitive attribute given the diagnostic label (i.e., Y T. S|Y), which is also adopted by (Gardner et al., 2019; Fong et al., 2021), I). On the other hand, Zietlow et al. (2022) find that for high-capacity models in computer vision, this is typically achieved by worsening the performance of the advan- taged group rather than improving the disadvantaged group, phenomenon termed as leveling down in philosophy that has incurred numerous criticisms (Christiano & Braynen, 2008; Brown, 2003; Doran, 2001). Worse, practical implementations often lead to worsening the performance of both subgroups (Zietlow al., 2022), making it pareto inefficient and comprehensively violating benefi- cence and mollfeeefieence principles (Beauchamp, 2003). Thus, argue that group fairness alone is not sufficient to analyse the trade-off between fairness and utility.
Max-Min Fairness It is another definition of fairness (Lahoti et al., 2020) following Rawlsian max-min fairness principle (Rawls, 2001), which is also studied as minimax group faimess (Diana et al., 2021) or minimax Pareto fairness (Martinez al., 2020). Here, instead of seeking to equalize the error rates among subgroups, it treats the model that reduces the worst-case error rates as the fairer one. may be more appropriate definition than group fairness for some medical applications such as diagnosis, as it better satisfies the beneficence and non-maleficence principles (Beauchamp, 2003; Chen al., 2018; Ustun et al., 2019), i.e., do the best and do no harm. Formally, for a model h in the hypothesis class H. denote (h) to be utility function for subgroup s. A model ht is considered be Max-Min Fair it maximizes (Max-) the utility of the worst-case (Min) group:
h argmax min (h). NEW
sES
In practice, it is hard to quantify the maximum optimal utility, and therefore we treat a model h2 to be fairer than the other model h4 if
min U. (b2 > min U. (he).
sES
We measure both group fairness and Max-Min faimess to give comprehensive evaluation for fairness in medical applications.
3 MEDFAIR
We implement a reproducible and easy-to-use codebase MEDFAIR to benchmark faimess in ma- chine learning algorithms for medical imaging. In our benchmark, we conduct large-scale experi- ments in ten datasets, eleven algorithms, up to three sensitive attributes for each dataset, and three model selection criteria, where all the experiments can be run with single command. We provide source code and detailed documentation, allowing other researchers to reproduce the results and incorporate other datasets and algorithms easily.
3.1 DATASETS
Ten datasets are included in MEDFAIR: CheXpert (Irvin et al., 2019). MIMIC-CX (Johnson al., 2019), PAPILA (Kovalyk et al., 2022), HAM10000 (Tschandl et al., 2018). Fitzpatrick177 17kk (Groh (Afshar et al., 2021), OCT (Farsiu al.,
et al., 2021), OL3I1 (Chaves et al., 2021), COVID- 2014), ADNI 1.5T, and ADNI 3T (Petersen et al., 2010), to evaluate the algorithms comprehensively, which are all publicly available to ensure the reproducibility. We consider five important aspects during dataset selection:
Imaging modalities. We select datasets covering various 2D and 3D imaging modalities, includ- ing X-ray, fundus photography. computed tomography (CT), magnetic resonance imaging (MRD, spectral domain optical coherence tomography (SD-OCT), and skin dermatology images.
Potential sources of bias. We involve datasets that may introduce bias from different sources, including label noise, data/class imbalance, spurious correlation, etc. Note that each dataset may contain more than one source of bias.
Sensitive attributes. The selected datasets contain attributes that are commonly treated sensi- tively and may be biased in clinical practice, including age, sex, race, and skin type.
4

---

Published as a conference paper at ICLR 2023
Detailed statistics of the datasets. "# images/scans" listed here the actual numbers used in this study after removing those missing sensitive attributes. For potential bias, LN, CI. DI, and SC represent label noise, class imbalance, data imbalance, and spurious correlation, respectively.
Table 1:
Dataset
Modality
# Images
Sensitive Attribute
Bias Sources
CheXpert
Chest X-ray (2D)
222,793
Age, Sex, Race
LN, CI, DI
MIMIC-CXR
Chest X-ray (2D)
370,955
Age, Sex, Race
LN, CI, DI
PAPILA
Fundus Image (2D)
420
Age, Sex
DI. C1
HAM10000
Skin Dermatology (2D)
9,948
Age, Sex
DI, CI
Fitzpatrick17k
Skin Dermatology (2D)
16,012
Skin type
LN. DI, CI
OL
Heart CT (2D)
8139
Age, Sex
DI, CI, SC
COVID-CT-MD
Lung CT (3D)
308
Age. Sex
DI, CI
OCT
SD-OCT (3D)
Age
DI, CI
ADNI 1.5T
Brain MRI (3D)
550
Age. Sex
SC
ADNI 3T
Brain MRI (3D)
110
Age. Sex
Size of datasets. As the sizes of medical imaging datasets are often limited by privacy issues, etc., it is important inspect whether the fairness algorithms effective with different sizes of datasets. The dataset sizes range from 420 ~ 370, 955 for 2D images and 110 ~ 550 for 3D scans.
Out-of-distribution evaluation. We include two pairs of datasets with the same modality but collected from different locations or different imaging protocols for out-of-din of-distribution evaluations. Specifically, we choose two 2D chest X-ray datasets i.e. CheXpert and MIMIC-CRR, and two 3D> brain MRI datasets ie., ADNI 1.5T and ADNI 3T.
Table lists the basic datasets information, and more detailed statistics are provided in Appendix B.
3.2
ALGGORITHMS
MEDFAIR incorporates 11 algorithms across 5 categories (related work in Appendix A): subgroup rebalancing, domain-independence, adversarial training, disentanglement, and domain generaliza-
tion. We carefully re-implement the following algorithms based on the original official codes:
Baseline
-Empirical Risk Minimization (ERM) (Vapník, 1999) minimizes the average error across the dataset without considering the sensitive attributes.
Subgroup Rebalancing
Resampling method upsampled the minority groups so that all of the subgroups appear dur- ing training with equal chances.
Domain-inddependence
Domain independent N-way classifier (DomainInd) (Wang et al., 2020) trains separate clas- sifiers for different subgroups with shared encoder.
Adversarial Training
Learning Adversarially Fair and Transferable Representations (LAFTR) (Madras et al., 2018) di-bi the representation by minimizing the ability recognize sensitive attributes. CConditional learning of Fair representation (CFair) (Zhao et al., 2019) tries enforce the balanced error rate and conditional alignment of representations during training.
Learning Not to Learn (LNL.) (Kim et al., 2019) unlearns the bias information iteratively by minimizing the mutual information between feature representation and bias.
Disentanglement
Entangle and Disentangle (EnD) (Tartaglione et al., 2021) disentangles confounders by in- serting an "information bottleneck", while still passing the useful information.
OOtthgonnal Disentangled Representations (ODR) (Sarhan et al., 2020) disentangles the use- ful and sensitive representations by enforcing orthogonality constraints for independence.
5

---

Published as a conference paper at ICLR 2023
Domain Generalization (DG)
Group Distributionally Robust Optimization (GroupDRO) (Sagawa et al., 2019) minimizes the worst-case training loss with increased regularization.
Stochastic Weight Averaging Densely (SWAD) (Cha et al., 2021), sfate-of-the method in DG, aims to find a robust flat minima by a dense stochastic weight sampling strategy.
Sharpness- Minimization (SAM) (Foret et al., 2020) seeks parameters that lie in neigh- borhoods having uniformly low loss during optimization.
The hyper-parameter tuning strategy is described Appendix B.2.2.
3.3
MODEL SELECTION
The trade-off between faimnss and utility has been widely noted (Kleinberg et al., 2016; Zhang et al., 2022), making objective nature of optimising for potentially conflicting fairness and utility. Previous work differs greatly in model selection. Some use conventional utility-based selection strategies, e.g., overall val- idation loss, while others have no explicit specification. We provide a summary of model selection strategies across the literature in Table A1. To investigate the influence of model selection strategies on the final performance, we study three prevalent selection strategies in MEDFAIR.
selection criteria particularly difficult to define given the multi-
Overall Performance-based Selection This is of the most basic and common strategies for model selection. picks the model that has the smallest loss value or highest accuracy/AUC across the validation of all sub-populations. However, this strategy tends to select the model with better performance in the majority group to achieve the best overall performance, leading to a potentially large performance gap among subgroups, which is illustrated in the red pentagon the right side of Figure 2 (note that it is not necessarily Pareto optimal).
Minimax Pareto Selection The concept of Pareto optimality was proposed by Mas-Colell al. (1995) and utilized in fair machine learn- ing to study the trade-off among subgroup ac- curacies (Martinez et al., 2020). Intuitively, for a model on the Pareto front, no group can achieve better performance without hurting the performance of other groups. In other words, defines the set of best achievable trade-offs among subgroups (without introducing unnec- essary harm). Based on this definition, we se- lect the model that lies the Pareto front and achieves the best worst-case AUC (the red star in the middle top of Figure 2). We present formal definition of minimax Pareto selection in Appendix B.4.
0.74
topia
0.72
AUC AUC Croup-1
0.70
Non Pareto optimal
0.64
Pareto optimel
Performene--bees Selection
0.62
DTO- PrO-based Selection
Minimax
0.74
0.76
0.78
0.80
0.82
0.84
AUC Group-O
Figure 2: Illustration of three different model se- lection strategies. Each data point represents dif- ferent hyper-parameter combination for one algo- rithm, where the red points are the models lying on the Pareto front.
DTO-based Selection Distance to optimal (DTO) (Han et al., 2021) is calculated by the normalized Fuclidean distance between the performance of the current model and the optimal utopia point. Here, we construct the utopia point by taking the maximum AUC value of each subgroup among all models. The DTO strategy selects the model that has the smallest distance to the utopia point (the red hexagon Figure 2).
3.4 EVALUATION AND IMPLEMENTATION
We apply the bias mitigation algorithms to medical image classification tasks and evaluate fairness based the performance of different subgroups (sensitive attributes). The sensitive attributes regarded as available during training (if needed). We consider two settings to evaluate fairness for medical imaging, i.e., in-distribution and nut-of-distribution
Evaluation Metries We use the area under the receiver operating characteristic curve (AUC) as the major metric, which is commonly used metric for medical binary classification. We evaluate

---

Published as a conference paper at ICLR 2023
the algorithms from three aspects: (1) utility: overall AUC across all subgroups: (2) group fair- ness: AUC gap between the subgroups that have maximum AUC and minimum AUC; (3) Max-Min faireess: AUC of the worst-case group. Besides, we also report the values of binary cross entropy (BCE), expected calibration error (ECE), false positive rate (FPR), false negative rate (FNR), and true positive rate (TPR) at 80% true negative rate (TNR) of each subgroup, as well the Equalized Odd (EgOdd). We provide detailed explanations of these metrics in Appendix B.3.
Statistical Tests Prior work has empirically evaluated bias mitigation algorithms and occasionally claimed that some algorithm works well based on results from a couple of datasets. We note that to make a stronger conclusion that would be more useful to practitioners, e.g., 'algorithm A works bet- ter than B for medical imaging' (i.e., A is better general, rather than better for dataset C' specifically), one needs to evaluate performance across several datasets and perform significance tests that check for consistently good performance that can not be explained by overfitting to single dataset. This is where the MEDFAIR benchmark suite comes in. To rigorously compare the relative performance of different algorithms, we perform the Friedman test (Friedman, 1937) followed by Nemenyi post-hoc test (Nemenyi. 1963) for both settings to identify if any of the algorithms is significantly better than the others, following the authoritative guide of Demšar (2006). We first calculate the relative ranks among all algorithms on each dataset and sensitive attribute separately, and then take the average ranks for the Nemenyi test if significance is detected by Friedman test. We consider p-value lower than 0.05 to be statistically significant. The testing results are visualized by Critical Difference (CD) diagrams (Demšar, 2006). In CD diagrams, methods that are connected by a horizontal line are in the same group, meaning they are not significantly different given the p-value, and methods that are in different groups (not connected by the same line) have statistically significant difference.
Implementation Details We adopt 2D and 3D ResNet-18 backbone (He et al., 2016; Hara et al., 2018) for 2D and 3D datasets, respectively. The light backbone is used to avoid overfitting as there are datasets with small sizes, and also to remain consistent with the backbone used in the original literature (Kim et al., 2019; Wang et al., 2020; Tartaglione et al., 2021; Sarhan al., 2020). Binary cross entropy loss is used as the major objective. To ensure the stability of randomness, for each experiment, we report the mean values and the standard deviations for three separate runs with three randomly selected seeds. Further implementation details for all datasets and algorithms can be found in Appendix B.
4 RESULTS
4.1 BIAS WIDELY EXISTS IN ML MODELS TRAINED IN DIFFERENT MODALITIES AND TASKS
Firstly, we train ERM on different datasets and sensitive attributes, and select models using the regular overall performance- hased strategy. For each dataset and sensitive attribute, we calculate the maximum and minimum AUC and underdiagnosis rate among subgroups, where we use FNR for the malignant label and FPR for "No Finding" label as the underdiagnosis rate. As shown in Figure 3. most points are to the side of the equality line, showing that the performance gap widely exists. This confirms a problem that has been widely discussed (Seyyed-Kalan) until now, has never been systematically quantified for deep learning across a comprehensive variety of modalities, diagnosis tasks, and sensitive attributes.
et al., 2021) but,
4.2 MODEL SELECTION SIGNIFICANTLY INFLUENCES WORST-CASE GROUP PERFORMANCE
We study the impact of model selection strategies on ERM using our three metrics of interest: The AUC of the worst-case group. the AUC gap, and overall AUC with ERM. We first conduct hyper- parameter sweep for ERM while training on all the datasets, and then compute the metrics and the relative ranks of the three model selection strategies. The results, including statistical significance tests are summarised in Figure 4, and the raw data in Table A8. Each sub-plot corresponds to a met- ric of interest (worst-case AUC, AUC Gap, Overall AUC), and the average rank of each selection strategy (Pareto. DTO, Overall) is shown a line. Selection strategies not connected by the bold bars have significantly different performances. The results show that for the worst-case AUC metric (left), the Pareto-optimal model selection strategy has the highest average rank of around 1.5, which is statistically significantly better than the overall AUC model selection strategy's 's average rank of around 2.5. Meanwhile, in terms of the overall AUC metric (right) the Pareto selection strategy is
7

---

Published as a conference paper at ICLR 2023
100
ADNI-Age
0.8
95
COVID CT MDD Age
Max Underdlagnosis Rate
0.7
COVID CT MD-Sex
90
Age
0.6
CheXpert-Sex
AUC
85
CheXpert
1tppatrick1kk-siin...
HAM10000-Sex
80
0.4
HAM10000 Age
MIMIC CXR-Age
0.3
CXR-Race
MIMIC_ CXR-Sex
0.2
OCT-Age
OL3
OL.3-Sex
60
70
80
90
100
0.0
0.1
0.2
0.3
0.4
0.5
PAPILA Sex
Min
Min Underdiagnosis Rate
Figure 3: The AUC (left) and underdiagnosis rates (right) for the advantaged and disadvantaged subgroups across each dataset and sensitive attribute, when training with ERM. Most points are off the blue equality line, showing that bias widely exists in conventional ERM-trained models.
Partto Optimal
Overwt ABC
Optimal DTD
Overall
Overall AUC DTO
Optimal
Worst-case AUC
AUC Gap
Overall AUC
Figure 4: Influence of model selection strategies on ERM, illustrated CD diagrams. The higher the rank of the AUC Gap, the smaller the gap.
not significantly worse than the overall model selection strategy. Thus, even without any explicit bias mitigation algorithm, max-min fairness can be significantly improved simply by adopting the corre- sponding model selection strategy in place of the standard overall strategy and this intervention need not impose a significant cost to the overall AUC.
4.3
No METHOD OUTPERFORMS ERM WITH STATISTICAL SIGNIFICANCE
(a)
2
3033
2011
3031
GroupDRO EnD LAPTI
Resempling
Domainind
SWAD
ERN
CPair
GroupORO
LNL
SAM
CDR
Worst-case AUC
AUC
Overall AUC
(b)
SWAD
GroupORO
SWM4D
Enb
Domainind
GrupORO
SAM
ODR
SAM
GroupRRo
Resempingg
Resamping
Chwir
LAFTR
UNL
Domahiin
EnD
Demanind
Worst-case AUC
Gap
Overall AUC
Figure 5: Performance of bias mitigation algorithms summarised across all datasets as average rank CD diagrams. (a) in-distribution, (b) dutto--distrbuutinn. SWAD is the highest ranked method for worst- and overall-AUC metrics, but it is still not significantly better than ERM.
We next ask whether any the purpose- designd bias mitigation algorithms is significantly better than ERM, and which algorithm is best overall? To answer these questions, evaluate the perfor-
8

---

Published as a conference paper at ICLR 2023
mance of all methods using the Pareto model selection strategy. We report the Nemenyi post-hoe test results on worst-group AUC, AUC gap, and overall AUC in Figure for in-distribution (top row) and cut-of-distributio (bottom row) settings with raw data in Tables A9 and A10. For in-distribution, while there are some significant performance differences, no method outperforms ERM significantly for any metric: ERM is always in the highest-rank group of algorithms without significant differ- ences. The conclusion is the same for the out-of-distribution testing, and some methods that rank higher than ERM in the in-distribution setting perform worse than ERM when deployed to an unseen domain, suggesting that preserving fairness across domain- shift is challenging.
It is worth noting there are some methods that consistently perform better, e.g. SWAD ranks a clear first for the worst-case and overall AUC for both settings, and thus could be a promising method for promoting faireess. However, from a statistical significance point of view, SWAD is still not significantly better despite the fact that we use a much larger sample size (number of datasets) than most of the previous fairness studies. This shows that many studies do not use enough number of datasets to justify their desired claims. Our benchmark suite provides the largest collection of medical imaging datasets for fairness to date, and thus provides the best platform for future research to evaluate the efficacy of any method works in rigorous statistical way.
5 DISCUSSION
Source of bias There are multiple confounding effects that can lead to bias, rather than any single casy-to-isolate factor. As summarised in Table and discussed further in Appendix A.4 and E, these include both measurable and unmeasurable factors spanning imbalance in subgroup size, imbalance in subgroup disease prevalence, difference in imaging protocols/iim//location spurious correlations, the intrinsic difference in difficulty of diagnosis for different subgroups, unintentional bias from the human labellers, etc. It is difficult or even impossible to disentangle all of these factors, making algorithms that specifically optimise for one particular factor to succeed.
Failure of the bias mitigation algorithms Although most bias mitigation algorithms are not con- sistently effective across benchmark suite, we certainly not trying to disparage them. It is understandable because some are not originally designed for medical imaging, which contains char- acteristics distinct from those of natural images tabular data, and more work may be necessary to design medical imaging specific solutions. More fundamentally, different algorithms may succeed if addressing solely the specific confounding factors for which they are designed to compensate, but fail when presented with other confounders or mixture of multiple confounders. For example, resampling specifically targets data imbalance, while disentanglement focuses more on removing spurious correlations. But real datasets may also simultaneously contain other potential sources of bias such as label noise. This may explain why SWAD is the most consistently high-ranked algo- rithm, as it optimises general notion of robustness without any specific assumption on confounders or sensitive attributes, and thus may be more broadly beneficial to different confounding factors.
Relation of domain generalization and fairness The aim of domain generalization (DG) algo- rithms is to maintain stable performance on unseen uob-poplattion, while the fairness prroiess-rrmttingg al- gorithms try to ensure that no known cobb-ooulatiins are poorly treated. Despite this difference, they share the eventual goal - being robust to changes in distribution across different sub- pobo-oouulaions.. As shown in section some domain generalization methods, such SWAD, consistently improve the performance of all subgroups, and thus overall utility. However, we also notice that they may also enlarge the performance gap among subgroups. It introduces a question of whether systemat- ically better algorithm (i.e., improving Max-Min fairness) is fairer if it increases the disparity (i.e., not satisfying group fairness)? This question goes beyond machine learning and depends on appli- cation scenarios. We suggest that relevant differentiator may be between diagnosis and zero-sum resource allocation problems where Max-Min and group fairness could be prioritised respectively.
Are the evaluations enough for now? Although we have tried our best to include a diverse set of algorithms and datasets in our benchmark, it is certainly not exhaustive. There are methods to promote fairness from other perspectives, e,g. self-supervised learning may be more robust (Liu et al., 2021; Azizi et al., 2022). Also, datasets from other medical data modalities (e.g., cardiology, digital pathology) should be added. Beyond image classification, other important tasks in medical imaging, such as segmentation, regression, and detection, are underexplored. We will keep our codebase alive and actively incorporate more algorithms, datasets, and even other tasks in the future.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We report the data preprocessing in Appendix B.1.2, hyper-parameterr space in Appendix B.2.2. All of the datasets we use are publicly available, and we provide the download links in Table A6. Source code and documentation are available at https: / zong. get.ggiithhi witihha.b.... 10/MED/AMMR....... Running all the experiments required 0.77 NVIDIA A100-SXM-80GB GPU years.
ACKNOWLEDGMENT
Yongshuo Zong is supported by the United Kingdom Research and Innovation (grant EP/S02431X/1),, UKRI Centre for Doctoral Training in Biomedical Al at the University of Edin- burgh, School of Informatics. For the purpose of open access, the author has applied creative commons attribution (CC BY) licence any author accepted manuscript version arising. We thank Dr. Maria Valdés Hernández for the discussion of data preprocessing. We acknowledge the data resources providers Appendix F.
REFERENCES
Parnian Afshar, Shahin Heidarian, Nastaran Enshaei, Farnoosh Naderkhani, Moezedin Javad Rafiee, Anastasia Oikonomou, Faranak Babaki Fard, Kaveh Samimi, Konstantinos N Plataniotis, and Arash Mohammadi. Covid-ct-md, covid-19 computed tomography scan dataset applicable in machine learning and deep learning. Scientific Data, 8(1):1-8, 2021.
Shekoofeh Azizi, Laura Culp, Jan Freyberg, Basil Mustafa, Sebastien Baur, Simon Kornblith, Ting Chen, Patricia WilWillsmms, S Sara Mahdavi, Ellery Wulczyn, et al. Robust and efficient medical imaging with self-supervision. arXiv preprint arXiv:2205.0972. 2022.
Solon Barocas and Andrew D Selbst. Big data's 'ata's disparate impact. Calif. L. Rev., 104:671, 2016.
Tom L Beauchamp. Methods and principles in biomedical ethics. Journal of Medical ethics, 29(5): 269-2744 2003.
Rachel KE Bellamy, Kuntal Dey, Michael Hind, Samuel Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilović. et al. Ai fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias. IBM Journal of Research and Development, 63 2019.
Aharon Ben-Tal, Laurent Ghaoui, and Arkadi Nemirovski. Robust optimization,, volume 28.
Princeton university press, 2009.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https:///ww wardb. com/. Software available from wandh.com.
Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathleen Walker. Fairleam: A toolkit for assessing and improving fairness ai. Microsoft, Tech. Rep. MSR-7R-2020-322 2020.
Campbell Brown. Giving up levelling down. Economics & Philosophy, 19())111-134, 2003.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp- 6299-6308. 2017.
Alessandro Castelnovo, Riccardo Crupi, Greta Greco, Daniele Regoli, Iariaa Giuseppina Penco, and Andrea Claudio Cosentini. A clarification of the nuances in the fairness metrics landscape. Scientific Reports, 12(1):1-21, 2022.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Infor- mation Processing Systems, 34:22405-22411 2021.
10

---

Published as a conference paper at ICLR 2023
Shubham Chaudhary, Sadbhawna Sadbhawna, Vinit Jakhetiya, Badri N Subudhi, Ujjwal Baid, and Sharath Chandra Guntuku. Detecting covid-19 19 and community acquired pneumonia using chest et scan images with deep learning. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 8583-8587. IEEE, 2021.
Juan M Zambrano Chaves, Akshay S Chaudhari, Andrew L Wentland, Arjun D Desai, Imon Baner- jee, Robert Boutin, David Maron, Fatima Rodriguez, Alexander T Sandhu, R Brooke Jeffrey, et al. Opportunistic assessment of ischemic heart disease risk using abdominopelvic computed tomography and medical record data: a multimodal explainable artificial intelligence approach. medRxiv, 2021.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Irene Chen, Fredrik D Johansson, and David Sontag. Why is my classifier discriminatory? Advances in neural information processing systems, 31, 2018.
Richard J Chen, Tiffany Y Chen, Jana Lipkova, Judy J Wang, Drew FK Williamson, Ming Y Lu, Sharifa Sahai, and Faisal Mahmood. Algorithm fairness in ai for medicine and healthcare. arXiv preprint arXiv:21 11.0100000000 2021.
Thomas Christiano and Will Braynen. Inequality, injustice and levelling down. Ratio, 21(4):392- 420, 2008
Elliot Creager, David Madras, Jörn-Herrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. In International conference on machine learning, pp. 1435-1445. PMLR, 2019.
Elliot Creager, Jörn-Hentt Jacobsen, and Richard Zemel. Exchanging lessons between algorithmic fairness and domain generalization. 2020.
Janez Demsar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research, 7:1-30, 2006.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Emily Diana, Wesley Gill, Michael Kearn,, Krishnaram Kenthapadi, and Aaron Roth. Minimax group fairness: Algorithms and experiments. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 66-76, 2021.
Brett Doran. Reconsidering the levelling-dwwn objection against egalitarianism. Utilitas, 13(1): 65-85, 2001.
John Duchi, Peter Glynn, and Hongseok Namkoong. Statistics of robust optimization: generalized empirical likelihood approach. arXiv preprint arXiv: 1610. 2016.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations theoretical computer science confer- ence, pp. 214-226, 2012.
Sina Farsiu, Stephanie Chiu, Rachelle O'Coneell, Francisco A Folgar, Eric Yuan, Joseph A Izatt, Cynthia A Toth, Ree-Related Eye Disease Study 2 Ancillary Spectral Domain Optical Coherence Tomography Study Group, et al. Quantitative classification eyes with and without intermediate age-related macular degeneration using optical coherence tomography. Ophthalmology, 121(1): 162-172, 2014.
Hortense Fong, Vineet Kumar, Anay Mehrotra, and Nisheeth K Vishnoi. Fairness for auc via feature augmentation. arXiv preprint arxiv:2111./282.1 2021.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness- aware minimiza- tion for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
11

---

Published as a conference paper at ICLR 2023
Sorelle A Friedler, Carlos Scheidegger, Suresh Venkatasubraamanian, Sonam Choudhary, Evan P Hamilton, and Derek Roth. A comparative study of fairness- enhanncnn interventions in machine learning. In Proceedings of the conference on faimess, accountability, and transparency, pp- 329-338, 2019.
Milton Friedman. The use of ranks to avoid the assumption of normality implicit the analysis of variance. Journal of the american statistical association, 32(200):675-70) 1937.
Josh Gardner, Christopher Brooks, and Ryan Baker. Evaluating the faimess of predictive student models through slicing analysis. In Proceedings of the 9th international conference on learning analytics & knowledge, pp. 225-234, 2019.
Judy Wawira Gichoya, Imon Banerjee, Ananth Reddy Bhimireddy, John I. Burns, Leo Anthony Celi, Li-Ching Chen, Ramon Correa, Natalie Dullerud, Marzyeh Ghassemi, Shih-Cheng Huang, et al. Ai recognition of patient race in medical imaging: a modelling study. The Lancet Digital Health, 2022.
Ary I. Goldherger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph Mietus, George Moody, thung-Kang Peng, and Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101(23)=e21-e222) e220, 2000.
Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri. Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1820-1828 1828, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMIR, 2017.
Xudong Han, Timothy Baldwin, and Trevor Cohn. Balancing out bias: Achieving fairness through balanced training. arXiv, 2021.
Xudong Han, Aili Shen, Yitong Li, Lea Frermann, Timothy Baldwin, and Trevor Cohn. fair- lib: A unified framework for assessing and improving classification fairness. arXiv preprint arXiv:2205.01887, 2022.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6546 66556,5552222 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Deborah Hellman. When is discrimination wrong? Harvard University Press, 2008
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1): 1-2, 1997.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group-accuracyy In Conference on Causal Learning and Reasoning, pp. 336-351. PMLR, 2022.
Jeremy A. Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ileus, Chris Chute, Hen- rik Marklund, Behzad Haghgoo, Robyn L. Ball, Katie Shpanskaya, Jayne Seekins, David An- drew Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, C. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and A. Ng. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. AAAI. 2019.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. In Amir Globerson and Ricardo Silva (eds.), Proceedings of the Shirty-Fourth Conference Uncertainty in Artificial In- telligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pp. 876-885, AUAI Press, 2018. URL http: / /aaai.aa...a.. www.oooig///..oo. 8 proceddi ngs wwwp/s/ppppppsppp/ppp 313 .df.. http:///aaaaaaaaa///////
12

---

Published as a conference paper at ICLR 2023
Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv. PhysioNet. Available online at: https://pyysionet. orgdommeentnimiccv/..ffccc accesssed August 23, 2021), 2020.
Alistair EW Johnson, Tom Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth Berkowitz, and Steven Horng, Mimic-cxr-jpg, large publicly available database of labeled chest radiographs. arXiv preprint urXiv:1901.c 0900220 2019.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: /60.00069 2016.
Sajad Khodadadian, AmirEmad Ghassami, and Negar Kiyavash. Impact of data processing on fairness in supervised learning. In 2021 IEEE International Symposium Information Theory (ISIT), pp. 2643-2648. IEEE, 2021.
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9012-9020, 2019.
Newton M Kinyanjui, Timothy Odonga, Celia Cintas, Noel CF Codella, Rameswar Panda, Prasanna Sattigeri, and Kush R Varshney. Fairness of classifiers across skin tones in dermatology. In International Conference Medical Image Computing and Computer-Assisted Intervention, pp- 320-329. Springer, 2020.
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-off's in the fair determi- nation of risk scores. arXiv preprint PXXiv:/609.058800 2016.
Oleksandr Kovalyk, Juan Morales-Sáncher Rafael Verdii-Monedero, Inmaculada Sellés-Navarro, Ana Palazón-Cabance and José-Luis Sancho-Gómer Papila: Dataset with fundus images and clinical data of both eyes of the same patient for glaucoma assessment. Scientific Data, 9(1): 1-12, 2022.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in neural information processing systems, 30, 2017.
Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Chi. Fairness without demographics through adversarially reweighted learning. Advances in neural information processing systems, 33:728 740, 2020.
Agostina Larrazabal, Nicolás Nieto, Victoria Peterson, Diego Milone, and Enzo Ferrante. Gen- der imbalance in medical imaging datasets produces biased classifiers for computer-aided diag- nosis. Proceedings of the National Academy of Sciences, 117(23): 2020.
Chelsea E Lee, Kacla Singleton, Melissa Wallin, and Victor Faundez. Rare genetic discases: nature's 's experiments human development. IScience, 231 123, 2020.
June-Goo Lee, Sanghoon Jun, Young- Won Cho. Hyunna Lee, Guk Bae Kim, Joon Beom Seo, and Namkug Kim. Deep learning in medical imaging: general overview. Korean journal of radiology, 18(4):570-584, 2017.
Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihycon Lee, and Jaegul Choo. Learning debiased representation via disentangled feature augmentation. Advances in Neural Information Processing Systems, 14,25123-25133 2021.
Hong Liu, Jeff HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. arXiv preprint arXiv:2110.05025, 2021.
Jianfang Liu. Tara Lichtenberg, Katherine Hoadley, Laila Poisson, Alexander Lazar, An- drew D Cherniack, Albert J Kovatich, Christopher Benz, Douglas A Levine, Adrian V Lee, et al. An integrated tega pan-cancer clinical data resource to drive high-quality survival outcome analytics. Cell, 470 2018.
13

---

Published as a conference paper at ICLR 2023
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the faimess of disentangled representations. Advances Neural Information Processing Systems, 32, 2019.
Gilles Louppe, Michael Kagan, and Kyle Cranmer. Learning to pivot with adversarial networks. Advances in neural information processing systems, 30, 2017.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. In International Conference on Machine Learning, pp. 3384 3393. PMLR, 2018.
Roman C Maron, Michael Weichenthal, Jochen Utikal, Achim Hekler, Carola Berking, Axel Hauschild, Alexander H Enk, Sehastian Haferkamp, Joachim Klode, Dirk Schadendorf, et al. Systematic outperformance 112 dermatologists in multiclass skin cancer image classification by convolutional neural networks. European Journal of Cancer, 119:57-65, 2019.
Natalia Martinez, Martin Bertran, and Guillermo Sapiro. Minimax pareto fairness: A multi objective perspective. International Conference on Machine Learning, pp. 6755-6764, PMIR, 2020.
Andreu Mas-Colell, Michael Dennis Whinston, Jerry R Green, et al. Microeconomic theory, vol- 1. Oxford university press New York, 1995.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1-35, 2021.
Kaisa Miettinen. Introduction to multiobjective optimization: Noninteractive approaches. In Multi- objective optimizaaiin, pp. 1-26. Springer, 2008
Peter Bjom Nemenyi. Distribution-free multiple comparisons. Princeton University, 1963.
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measur- ing calibration in deep learning. In CVPR Workshops, volume 2, 2019.
Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in algorithm used to manage the health of populations. Science, 366(6464):447-4555 2019.
Sungho Park, Jewook Lee, Pilhyeon Lee, Sunhee Hwang, Dohyung Kim, and Hyeran Byun. Fair contrastive learning for facial attribute classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10389- 10398, 2022.
R. C. Petersen, P. S. Aisen, L. A. Beckett, M. C. Donohue, A. C. Gamst, D. J. Harvey, C. R. Jack, W. Jagust, L. M. Shaw, A. W. Toga, J. Q Trojanowski, and M. W. Weiner. Alzheimer's disease neuroimaging initiative (adni). Neurology, 74(3):201-209, 2010. ISSN 0028-3878. doi: 10. 10.1212/WNL.0001 3111313311000 18ccb3255
Gcoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On fairness and calibration. Advances in neural information processing systems, 30, 2017.
John Rawls. Justice fairness: A restatement. Harvard University Press, 2001.
Charan Reddy, Deepak Sharma, Soroush Mehri, Adriana Romero-Soriang Samira Shabanian, and Sina Honari. Benchmarking bias mitigation algorithms representation learning through fair- ness metries. In Thirty- fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Amelic Royer and Christoph H Lampert. Classifier adaptation at prediction time. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1401-1409, 2015.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint arXiv: /91 1.08311, 2019.
Mhd Hasan Sarhan, Nassir Navab, Abouzar Eslami, and Shadi Albarqouni. Faimess by learning orthogonal disentangled representations. In European Conference on Computer Vision, pp. 746- 761. Springer, 2020.
14

---

Published as a conference paper at ICLR 2023
Laleh Seyyed- Kalantar,,, Guanxiong Liu, Matthew McDermott, Irene Y Chen, and Marzyeh Ghas- semi. Chexclusion: Fairness gaps in deep chest x-ray classifiers. In BIOCOMPUTING 2021:
proceedings of the Pacific sympostum, pp. 233-243. World Scientific, 2020.
Laleh Seyyed-Kalantaie Haoran Zhang, Matthew McDermott, Irene Y Chen, and Marzyeh Ghas- semi. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature medicine, 27 27(12)2116-2182, 2021.
Christine S Spencer, Darrell J Gaskin, and Eric T Roberts. The quality of care delivered to patients within the same hospital varies by insurance type. Health Affairs, 32(10))1331-7733 2013.
Cathie Sudlow, John Gallacher, Naomi Allen, Valerie Beral, Paul Burton, John Danesh, Paul Downey, Paul Elliott, Jane Green, Martin Landray, al. Uk biobank: open access resource for identifying the causes a wide range of complex diseases of middle and old age. PLOS medicine, le 2015.
Enzo Tartaglione, Carlo Alberto Barbano, and Marco Grangetto. End: Entangling and disentangling deep representations for bias correction. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 13508-13517, 2021.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The haml 0000 dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1): 2018.
Berk Ustun, Yang Liu, and David Parkes. Fairness without harm: Decoupled classifiers with pref- erence guarantees. In International Conference on Machine Learning, pp. 6373-6382. PMLR, 2019.
Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions on neural net- works, 10(5):988-999, 1999.
Sahil Verma and Julia Rubin. Faimness definitions explained. In 2018 ieeelacm international work- shop on software fairness (fairware). pp. 1-7. IEEE, 2018.
Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 9919-8928, 2020.
David Wen, Saad M Khan, Antonio Ji Xu, Hussein Ibrahim, Luke Smith, Jose Caballero, Luis Zepeda, Carlos de Blas Perez, Alastair K Denniston, Xiaoxuan Liu, al. Characteristics of publicly available skin cancer image datasets: systematic review. The Lancet Digital Health, 2021.
Julia K Winkler, Christine Fink, Ferdinand Toberer, Alexander Enk, Teresa Deinlein, Rainer Hlffmann-Wellenhee Lue Thomas, Aimilios Lallas, Andreas Blum, Wilhelm Stolz, et al. As- sociation between surgical skin markings in dermoscopic images and diagnostic performance of deep learning convolutional neural network for melanoma recognition. JAMA dermatology. 155 (10):1135-114) 2019.
Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. Advances in neural information processing systems, 30, 2017.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adver- sarial learning. In Proceedings of the 2018 AAAI/ACM Conference AI, Ethics, and Society, pp- 335-340, 2018.
Haoran Zhang, Natalic Dullerud, Karsten Roth, Lauren Lakden-Rayner, Stephen Pfohl, and Marzyeh Ghassemi. Improving the fairness of chest x-ray classifiers. In Conference Health, Inference, and Learning, pp. 204-233. PMLR, 2022.
Han Zhao, Amanda Coston, Tameem Adel, and Geoffrey Gordon.
Conditional learning of fair representations. In International Conference Learning Representations, 2019.
15

---

Published as a conference paper at ICLR 2023
Yuyin Zhou, Shih- Shih-Cheng Huang, Jason Alan Fries, Alaa Youssef, Timothy J. Amrhein, Marcello Chang, Imon Banerjee, Daniel L. Rubin, Lei Xing, Nigam H. Shah, and Matthew Lungren. Radfusion: Benchmarking performance and fairness for multimodal pulmonary embolism detec- tion from et and ehr. ArXiv, abs/21 11.1 1665, 2021.
Dominik Zietlow, Michael Lohaus, Guha Balakrishnan, Matthäus Kleindessner, Francesco Lo- catello, Bernhard Schölkopf, and Chris Russell. Leveling down in computer vision: Pareto ineffi- ciencies in fair deep classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10410-10421, 2022.
16

---

Published as a conference paper at ICLR 2023
A RELATED WORK
In Appendix A, We present a broad review of the model selection strategies adopted by existing lit- erature, current bias mitigation algorithms, existing fairness benchmarks, and domain generalization methods and their relationship to fairness.
A.1 MODEL SELECTION STRATEGY IN FAIRNESS
Considering the trade-off between fairness and the utility, how select the appropriate model among nyper--arameeles remains an important problem. We broadly review and summarize the model se- lection strategies of recent work in Table AI. N/A means they do explicitly specify their model selection strategies in their papers (although they may have implemented it in their open-source code). can be seen that the model selection strategies differ greatly among existing literature, mak- ing a direct comparison infeasible. Hence, we investigate the influence of model selection strategies our study.
Table A1: Summary of model selection strategies of fairess- aware methods and benchmarks.
Reference
Paper Type
Selection Strategy
Dataset Type
Model Type
Locatello al. (2019)
Benchmark
N/A
Images
Deep
Reddy al. (2021)
Benchmark
Report best test results for each metric
Tabular / Images
Shallow
Zhang al. (2022)
Benchmark
Best validation worst-group AUC
Images
Deep
Han et al. (2022)
Benehmark
Best validation loss / DTO
NLP
Deep
Friedler et al. (2019)
Benchmark
N/A
Tabular
Shallow
Wang et al. (2020)
Method
Best validation weighted mAP
Images
Deep
Madras et al. (2018)
Method
Report faimess/accuracy trade-off'
Tabular
Shallow
Zhao et al. (2019)
Method
Report fairess/accerrcy trade-off'
Tabular
Shallow
Creager et al. (2019)
Method
Report faimess/accuracy trade-off' N/A
Tabular/Images
Shallow/Deep
Kim al. (2019)
Method
Images
Deep
Tartaglione et al. (2021)
Method
Optimized on validation set
Images
Deep
Sarhan et (2020)
Method
Best validation loss
Images
Deep
(2022)
Method
N/A
Images
Deep
Martinez al. (2020)
Method
Best validation worst-group risk
Images/Tabulaa
Shallow/Deep
Idrissi al. (2022)
Method
Best validation group accuracy
Images
Deep
Lahoti (2020)
Method
Best validation overall AUC
Tabular
Shallow
Lee (2021)
Method
N/A
Images
Deep
A.2 BIAS MITIGATION ALGORITHMS
According to the stage when bias mitigation methods are introduced, they can be mainly classi- fied into three categories, namely pre-processing, mn-rrocessing, and post-processing. The pre- processing methods aim to curate the data in the dataset to remove the potential bias before learning a model (Khodadadian al., 2021), while the post-processing methods seek to adjust the predic- tions given trained model according to the sensitive attributes (Pleiss et al., 2017). In this paper, we focus on benchmarking in-processing methods, which aim to mitigate the bias during the training of the models. Below, we review four popular categories of bias mitigation algorithms.
Subgroup Rebalancing For imbalanced data, synthetic minority oversampling technique (SMOTE) (Chawla et al., 2002) is classic resampling method that over-samples the minority class and under-samples the majority class. Recent work found that simply using data balancing can effectively improve the worst-case group accuracy (Idrissi et al., 2022).
Jomain-Independence Wang et al. (2020) develop a domain indeeneendeeendennt training strategy that applies different classification heads to different subgroups. (Royer & Lampert, 2015) propose
17

---

Published as a conference paper at ICLR 2023
classifier adaption strategies in the prediction time to reduce the error rates when the distribution of testing domain is different.
Adversarial Learning There are two major categories of adversarial learning: (1) it plays min- imax game that the classification head tries to achieve the best classification performance while minimizing the ability of the discriminator to predict the sensitive attributes (Zhang et al., 2018; Kim et al., 2019). After training, the sensitive attributes in the representation are expected to be indistinct. (2) it enforces the fairness constraints (e.g., group faimess) the representation during trianing, and the representation is used for downstream tasks later (Xie et al., 2017; Madras et al., 2018; Zhao et al., 2019).
Disentanglement Disentanglement methods (Tartaglione et al., 2021; Sarhan et al., 2020; Creager et al., 2019; Lee et al., 2021) isolates independent factors of variation into different and independent components of a representation vector. In other words, these methods disentangle the sensitive attributes and task-meaningful information in the representation level. Then, the classification tasks can be conducted on representation containing only task-specific representation so that the sensitive attributes and other unobserved factors will not make an impact.
In our benchmark, we select typical in-processing algorithms from these categories to provide a comprehensive comparison.
A.3 FAIRNESS BENCHMARKS
There are efforts in general machine learning communities to benchmark the performance of the fairness-aware algorithms, inspecting their effectiveness different aspects. AIF360 (Bellamy et al., 2019) implements wide range of fairness metrics and debiasing algorithms, which is available in both Python and R language. Fairleam (Bird et al., 2020) also provides debiasing algorithms and fairmess metrics to evaluate ML models. Friedler al. (2019) benchmark series of aawree machine learning techniques. However, they only study the traditional machine learning models. For deep learning methods, Reddy al. (2021) benchmark algorithms from the perspective of represen- tation learning on tabular and synthetic datasets, and find they can successfully remove spurious correlation. Locatello et al. (2019) specifically benchmark the disentanglement fair representation learning methods on 3D shape datasets, suggesting disentanglement can be a useful property to encourage fairness.
In the area of computational medicine, machine learning models have been found to demonstrate a systematic bias toward wide range of attributes, such as race, gender, age, and even the health in- surance type (Obermeyer et al., 2019; Larrazabal et al., 2020; Spencer et al., 2013; Seyyed-Kalantari Kalantari et al., 2021). The bias also exists in different types of medical data, such as chest X-ray (Seyyed- Kalantari et al., 2020), CT scans (Zhou et al., 2021), skin dermatology (Kinyanjui et al., 2020), health record (Obermeyer et al., 2019), etc.
The most relevant work to ours is Zhang al. (2022), which compares series of algorithms chest X-ray images, and finds no method outperforms simple data balancing. However, it is unclear whether the conclusion can be generalized to other medical imaging modalities, and whether the selection of methods is comprehensive. In contrast, we benchmark a wider range of algorithms on different data modalities, study the ultimately more significant issue of model selection, and provide further analysis of the cause of the bias and the explanation of the effective algorithms. To the best of our knowledge, we are the first provide comprehensive benchmark for wide range of algorithms and datasets, and comprehensive analysis of different model selection criteria.
A.4 SOURCE OF BIAS
Data imbalance across subgroups is one of the most common sources of bias for medical imaging, as many biomedical datasets lack demographic diversity. For example, many datasets are devel- oped with individuals originating from European ancestries, such as UK Biobank and The Cancer Genome Atlas (TCGA) (Sudlow et al., 2015; Liu al., 2018). Also, more data samples in total will be from older people the dataset is collected to study specific disease that occurs more in older people, egg. dataset for Age-related macular degeneration (AMD) (Farsiu al., 2014). Overall,
18

---

Published as a conference paper at ICLR 2023
when there are more samples in one subgroup than the other, it can be expected that the machine learning model may have different prediction accuracy for different subgroups.
Class imbalance occur along with the data imbalance, where one subgroup has more samples of some specific classes while the other subgroup has more samples from other classes. For example, in the AMD dataset, subgroups of older people contain more pathology examples than subgroups of younger people. Also, the problem of class imbalance happens for rare diseases, which is generally due to genetic mutations that occur in very limited number of people (Lee et al., 2020), e.g., 10 in 1,000,000, So, the class imbalance is severe and there would never be enough data (even worldwide) for a balanced representation in the training set.
Spurious correlation can be learned by machine learning models undesirably from imaging de- vices. For example, a model can learn to classify skin diseases by looking at the markings placed by dermatologists near the lesions, instead of really learning the diseases (Winkler et al., 2019). It is also related to the data imbalance and class imbalance because the spurious correlations may oceur more frequently in subgroup with smaller number of examples by simply remembering all the data points, i.e., overfitting. Moreover, class imbalance itself may lead to a spurious correlation. For example, the model may try to use age-related features to predict pathology in dataset where most of the older patients are unhealthy.
Label noise can also be source of bias for medical imaging datasets. As the labeling of medical imaging datasets is labor-intensive and time-consuming, some large-scale datasets are labeled by automatic tools (Irvin et al., 2019; Johnson et al., 2019), which may not be precisely accurate and thus introduce noises. Zhang et al (Zhang et al., 2022) recruit a board-certified radiologist to relabel subset of the chest X-ray dataset CheXpert, and find that the label noises are much higher in some subgroups than the others.
Inherent characteristies of the data of certain subgroups can lead different performance for different subgroups even if the dataset is balanced, i.e., the tasks in some subgroups are inherently difficult even for humans. For example, in skin dermatology images, the lesions are usually more difficult to recognize for darker skin than that for light skin due to the low contrast (Wen et al., 2021). Thus, even with balanced datasets, a trained ML model can still give lower accuracy to patients with darker skin. Considering this, other measures beyond algorithms should be adopted to promote fairness, such as collecting more representative samples and improving imaging devices.
In summary, the bias is usually not from single source, and different sources of bias also usually correlate with each other, leading to the unfairness of the machine learning model.
A.5 DOMAIN GENERALIZATION AND FAIRNESS
Domain generalization (DG) algorithms aim to maintain good performance on unseen sub- population, while fairness- phomess-promoting algorithms try to ensure that no known sub-population is poorly treated. Generally, though differing in detail, their eventual goal is the same - being robust to distribution changes across different sub-populations, which is also discussed in (Creager et al., 2020). Hence, in this work, also explore fairness from the perspective of domain generalization.
One line of work for DG is to treat it as a robust optimization problem (Ben- Tal et al., 2009), where the goal is to try to minimize the worst-case loss for subgroups of the training set. Duchi et al. (2016) propose to minimize the worst-case loss for constructed distributional uncertainty sets with Distributionally Robust Optimization (DRO). GroupDRO (Sagawa et al., 2019) extends this idea by adding increased regularization to overparameterized networks and achieves good worst-case performance.
Another line of work focuses on finding flat minima in the loss landscape during optimization. As flat minima in the loss landscape considered to be able to generalize better to different domains (Hochreiter & Schmidhuber, 1997), methods have been proposed to optimize for flatness (Izmailov et al., 2018; Cha et al., 2021; Keskar et al., 2016; Foret et al., 2020). Stochastic weight averag- ing (SWA) (Izmailov et al., 2018) finds flat minima by averaging model weights during parameters updating every K epoch. Stochastic weight averaging densely (SWAD) adopts a similar weight en- semble strategy SWA by sampling weights densely, i.e., for every iteration. Also, SWAD searches the start and end iterations for averaging by considering the validation loss to avoid overfitting. Sharpness-aware minimization (SAM) (Foret et al., 2020) finds flat minima by seeking parameters
19

---

Published as a conference paper at ICLR 2023
that lie in neighborhoods having uniformly low loss. We select three popular methods of them in our benchmark GroupDRO, SAM, and SWAD.
20

---

Published as a conference paper at ICLR 2023
B IMPLEMENTATION AND EVALUATION DETAILS
B. DATA
B.1.1 DATASET
We summarize the statistics of the subgroups and class labels in Table A2 to A5. The numbers with percentages out of the brackets are the percentage of the appearing prevalence, and the numbers in brackets are the percentage of being unhealthy (class label). The used datasets are all publicly available, but we cannot directly include the downloading links in our benchmark. Thus we provide access links Table provides.
Table A2: The statistics of the subgroups and class labels. The numbers with percentages out of the brackets are the percentage of the appearing prevalence, and the numbers the brackets are the percentage of being unhealthy (class label).
CheXpert
MIMIC-CXR
HAM10000
# Images
222,793
370,955
9,948
# Patients
64,428
222,793
Unknown
Male
59.34% 120)
52.16% (62.64%)
54.28% (16.81%)
Female
40.65% (89.78%)
47.83% (57.11%) 1%)
45.72% (11.65%)
White
56.39% (90.02%)
60.56% (65.2%)
Unknown
non-White
43.61% (89.92%)
39.44% (52.01%)
Unknown
Age 0-20
0.8% (78.62%)
0.75% (25.37%)
2.42% (0.41%)
Age 20-40
16.05% (79.56%)
13.3% (36.66%)
45.01% (5.71%)
Age 40-60
31.07% (87.6%)
15.2% (53.46%)
6.98% (9.51%)
Age 60-80
39.01 % (93.0%)
31.13% (67.24%)
29.2% (23.92%)
Age 80+
13.08% (96.3%)
39.63% (76.65%)
16.39% (32. 13%)
Table A3: The statistics the subgroups and class labels. The numbers with percentages out of the brackets are the percentage of the appearing prevalence, and the numbers in the brackets are the percentage being unhealthy (class label). Age group 0 and age group range from 0-60 and 60+ except for the OCT dataset, whose age groups from 55-75 and 75+.
PAPILA
OCT
ADNI
ADNI3T
# Images
420
384
550
182
# Patients
210
269
417
80
Male
34.76% (23.97%)
Unknown
52.55% (44.98%)
34.62% (42.86%)
Female
65.24% (18.98%)
Unknown
47.45% (43.30%)
65.38% (39.50%)
Age Group 0
42.38% (8.43%)
40.89% (52.23%)
49.82% (44.16%)
46.70% (45.36%)
Age Group
57.62% (29.75%)
59.11% (14.54%)
50.18% (44.20%)
53.30% (35.29%)
B.1.2
DATA PREPROCESSING
Data splitting for experiments: unless otherwise specified, randomly split the whole dataset into cruining/validationnessii sets with a proportion of 80/10/10 for 2D datasets and 70/10/20 for 3D datasets.
21

---

Published as a conference paper at ICLR 2023
Table A4: The statisties of the Fitzpatrick dataset.
Skin Type
II
IV
V
% Images
18.40%
30.03%
20.66%
17.37%
9.57%
3.97%
Malignant
15.37%
15.43%
13.78%
10.82%
10.82%
9.61%
Table A5: The statistics of the subgroups and class labels. The numbers with percentages of the brackets are the percentage of the appearing prevalence, and the numbers in the brackets are the percentage of being unhealthy (class label). Age group and age group range from 0-60 and 60+.
COVID-CT-MD
OL31
Images
305
8139
Patients
305
8139
Male
60.00% (59.02%)
40.46% (5.53%)
Female
40.00% (50.00%)
59.54% (3.57%)
Age Group
73.11%
67.91% (2.26%%
Age Group
26. 26.89%(52.444)) 32.09% (8.81%)
CheXpert We first incorporate ethnicity labels (Gichoya et al., 2022) and the original data (Links in Table A6), dropping those images without sensitive attribute labels. The "No Finding" label is used for training and testing. We use all of the available frontal and lateral images, and images of the same patient do not share across vain//vlliatioon/eest split.
The race data is available via MIMIC-IV (Johnson et al., 2020) dataset, which is also deposited the PhysioNet database (Goldberger et al., 2000). We merge to the original MIMIC-CXR metadata based "subject ID". Other preprocessing steps similar to that of CheXpert.
MIMIC-CXR
PAPILA We exclude the "suspect" label class and images with labels of glaucomatous and non-glaucomatous for binary classification tasks. The dataset contains right-eye and left-eye images of the same patient. We split the trann/valiattonntesst in proportion of 70/10/20, and images of the same patient do not share across the train/validation/eest split.
HAM10000 We split 7 diagnostic labels into binary labels, i.e., benign and malignant follow- ing Maron et al. (2019). Benign contains basal cell carcinoma (bcc), benign keratosis-like lesions (solar lentigines seborrheic keratoses and lichen-plande like keratoses, bkl), dermatofibroma (df), melanocytic nevi (nv), and vascular lesions (angiomas, angiokeratomas, pyogenic granulomas and hemorrhage, vasc). malignant contains Actinic keratoses and intraepithelial carcinoma / Bowen's disease (akiec), and melanoma (mel). We discard images whose sensitive attributes are not recorded, resulting in 9948 images in total.
Fitzpatrick17k We split the three partition labels into binary labels i.e., benign and malignant. We treat "non-neoplastic" and "benign" the benign label, and use "malignant" the malignant label. Fitzpatrick skin type labels are used as sensitive attributes.
OL31 Opportunistic L3 computed tomography slices for Ischemic heart disease risk assessment (OL3I) dataset provides 8, 8,139 axial computed tomography (CT) slices at the third lumbar vertebrae (L3) level of individuals. We design the task predict whether the individual would be diagnosed with ischemic heart disease one year after the scan according to the labels provided (i.e. prognosis). Sex and age treated sensitive attributes.
22

---

Published as a conference paper at ICLR 2023
Table A6: Access to the datasets.
Dataset
Access
Original data: http:p/s//saaffcoooooooooooooooo
chexpert/
CheXpert
Demographic
data:
MIMIC-CXR
PAPILA
HAM10000
OCT
https:
OL331
Fitzpatrick17k
http://gtthuu.ccm/aatttp///ttttt
COVID-CT-MD
httper//dallor////ooo/////ffsfs2
ADNI-1.5T
ADNI-3T
OCT
The task is predict if the patients have age-related macular degeneration (AMD). The dataset only contains people whose ages are over 55. Thus, we treat age binary attribute splitting from 75 years old, i.e., group-O: 55-75, group-1: 75+. Scans are resized to 224 x 224 x 100, and random cropping of the size of 192 x 192 x 96 is used for training.
COVID-CT-MD We design the task to predict whether the patients are infected by COVID-19 19, L... COVID-19 in one class, Community Acquired Pncumonia (CAP) and Normal in the other class. We resize each slice resolution of 256 x 256, and then take the central 80 slices for training, as the crucial signal associated with infection COVID and CAP is often present in the middle slices (Chaudhary al., 2021). random cropping of size of 224 224 80 used for training. We split train/validation/tsst in a proportion of 70/10/20.
We design the task to predict if patients have Alzheimer's disease (AD). MRI scans from ADNI have been preprocessed. We resize the height and width seans of both 1.5T and 3T to the same size of 224 x 224 x 144 to reduce the variance for cross-domain testing. Random cropping of the size of 196 x 196 128 is used for training.
ADNI-L5TTANDI-3T
B.2 TRAINING
B.2.1 IMPLEMENTATION DETAILS
The experiments are conducted on a Scientific Linux release version 7.9 with one NVIDIA A100- SXM-80GB GPU. We trained over 7,000 models using ~ 0.77 GPU year. The implementation is based on Python 3.9 and Py 1.10. We adapt the source code released by the original authors to framework. We use ResNet-18 for 2D images and 3D ResNet-18 18 for 3D images the backbone network for all experiments except otherwise specified.
For 2D datasets, resize the images to the size of 256 x 256, and apply standard data augmentation strategies, i.e., random cropping to 224 x 224, random horizontal flipping, and random rotation of up 15 degrees. The backbone network is initialized using ImageNet (Deng al., 2009) pretrained weights and images are normalized with the ImageNet mean and standard deviation values. For 3D datasets, we resize 3D images according the original imaging characteristics described in B.1.2.
23

---

Published as a conference paper at ICLR 2023
The backbone network is initialized using kinetics (Carreira & Zisserman, 2017) pretrained weights and images are normalized with the kinetics mean and standard deviation values. Dataset-specific preprocessing and papee--rraeeters can be found below.
B.2.2 PARMEEP
To achieve the optimal performance of each algorithm for fair comparisons, we perform a Bayesian hyper-parameter optimization search for each algorithm and each combination datasets and sensi- tive attributes using a machine learning platform Weights & Bias (Biewald, 2020). We use the batch size 1024 and 8 for 2D and 3D images respectively. SGD optimizer is used for all methods and we apply early stopping if the validation worst-case AUC does not improve for 5 epochs. The follow- space is searched (20 runs for each method per dataset x sensitive attribute), means the value range, and means the discrete values:
ing
where []
ERM/ResamplingDDomnnnnnd Learning rate Ir [le-3, 1c-5).
LAFTR Learning rate Ir E [1e-3, le-4]. Adversarial coefficients [0.01,
CFair Learning rate [r E [1e-3, 16-4
4]. Adversarial coefficients E [0.01,5].
LNL Learning rate Ir E [1c-3, 1c 4 Adversarial coefficients 7) E [0.01, 5].
EnD Learning rate Ir E [1e-3, 1c 4 Entangling term coefficients a E [0.01, 5]. Disentangling
term coefficients 8 50.
ODR Learning rate (r E [1e-3, le- 4]. Entropy Weight coefficients λÆ E [0.01, 5]. Orthogonal- Disentangled loss coefficients ÅOD E [0.01,5], 5). KL divergence loss coefficients RoD [0.01. Entropy Gamma E [0.1,5] 5j.
GroupDRO Learning rate E [le-3, le 1e-4). Group adjustments 17 E [0.01, 5). Weight decay La E 1, le 1,
SWAD Learning rate Ir [le [1e-3, le - 4]. Starting epoch F., E {3,5, 7, 9). Tolerance epochs E, E 7,99, Tolerance ratio T, E (0.01,033).
SAM Learning rate Ir E 11e-1,1e-40 4]. Neighborhood size rho E [0.01,5].
Empirically, find the best learning rate for all methods is around 1c 4 except SAM, which requires a higher learning rate at around [le-1, 1, 1e-3- depending on the datasets.
B.2.3 METHODS
We summarize the benchmarked methods in Table A7. We show their categories, whether they can accept multiple sensitive attributes for training, and whether they require the information of sensitive attributes for training and testing.
B.3 METRICS
We explain the metrics used in this study in detail below:
AUC Area under the receiver operating characteristic curve (AUROC) is a standard metric to measure the performance of binary classification tasks, whose value is not affected by the imbalance of class labels. We use the name AUC in our text for simplicity. We measure the average AUC and the AUC of each subgroup. We pay particular attention to the AUC gap and the worst-case AUC to evaluate group fairness and max-min faimeess.
ECE Expected calibration eπror (ECE) (Guo et al., 2017; Nixon et al., 2019) is an indicator of group sufficiency (Castelnovo et al., 2022). A high ECE value may result different optimal best decision threshold.
BCE
Binary eross entropy (BCE) is the objective function optimize for the classification task.
24

---

Published as a conference paper at ICLR 2023
Table A7: A list of methods used in the benchmark. SA short for Sensitive Attributes, and N represent Yes and No, respectively.
Method
Category
Multiple SA
SA for Training
SA for Testing
ERM (Vapnik, 1999)
Baseline
Y
N
N
Subgroup
Resampling (Idrissi al.,
Y
Y
N
Rebalancing
2022)
DomainInd (Wang et al., Domain- Independence
Y
Y
N
2020)
LAFTR (Madras al., 2018)
Adversarial
N
Y
N
CFair (Zhao et al., 2019)
Adversarial
N
Y
N
LNL (Kim al., 2019)
Adversarial
Y
Y
N
EnD (Tartaglione al.,
Disentanglement
Y
Y
N
2021)
ODR (Sarhan et al., 2020)
Disentanglement
Y
Y
N
Domain
GroupDRO (Sagawa et al., 2019)
Y
Y
N
Generalization
Domain
SWAD (Cha al., 2021)
Y
N
N
Generalization
Domain
SAM (Foret et al., 2020)
Y
N
N
Generalization
TPR, FPR, FNR, FPR Define true positive (TP) and true negative (TN) values that are tually positive (negative) and correctly predicted positive (negative). Define false positive (FP) and false negative (FN) values that are actually negative (positive) but wrongly predicted positive (negative). Then, we define true positive rate (TPR), true negative rate (TNR), false positive rate (FPR), and false negative rate (FNR) as
TP 1 FNR
TPR TP+FN FN
(3)
TN
TNR= TN FP = =1-
(4)
We report the overall FPR, FNR, and their values for each subgroup. The threshold is selected based the minimum FI score following (Seyyed- Kalantarr et al., 2021). We also report the value of TPR at 80% TNR, which indicates the true positive rate of a given desirable true negative rate.
EqOdd Equalized Odds is a widely used group fairness metric that the true positive and false positive rates should be equalized across subgroups. Denote the input, label, and sensitive attribute as æ, the prediction and the output probability as ÿ.P. Following (Reddy et al., 2021), we define Equality of Opportunity w.r.t. y and i.e., EqOppO and Eq
EqOppO 1 lp(ğ 1ly 0, group-O) p(ū 1/y 0, group-
(5)
EqOppl = 1 lp(ū 1 = 1, group-O) p(ù = 1/y 1, group-1) 1)).
(6)
Then, we have EqOdd = 0.5 (EqOppO EqOpp1).
B.4 MINIMAX PARETO SELECTION
Following (Martinez et al., 2020) and the dominance definition (Miettinen, 2008), give formal definition of the Pareto optimal regarding AUC.
Dominant vector A vector E I is said to dominate E R* if E, > ti Vi 1,..., k and Fj 1, > ly (namely strictly inequality least component), denoted as L
25

---

Published as a conference paper at ICLR 2023
Dominant Classifier Given set of group- -ppucffcc metries function T(h), Lee AUC in our case, model h is said to dominate h² if T(h') T(h") ) denote h h" Liikewise, model h' ž h if T(h)) = T(h").
Pareto Optimality Given set of models H, and a set of group-specific metrics function T(h), Pareto front model is Ps.H {h H Dh E Hhh > h1 ( e > h wh H}. We call a model a Pareto optimal solution iff & E PA.I-
Finally, model h" is minimax Pareto fair classifier if it maximizes the worst-group AUC among all Pareto front models. For example, in Figure 2, each data point represents a different hyper- parameter combination for one algorithm, where the red points are models lying on the Pareto front. As we can see from the figure, the Pareto optimal points cannot improve the AUC of group 0 without hurting the AUC of group 1, and vice versa, indicating the best trade-off between different groups. The worst-case group is group 1, as the best AUC achieved for group is lower than the best AUC achieved for group 0. In this case, we select the model that achieves the best AUC of group (red star point) the disadvantaged group, to make the selection as fair as possible.
26

---

Published as a conference paper at ICLR 2023
C ADDITIONAL RESULTS
Table A8 are the results of ERM under different model selection strategies behind Figure 4. Table A9 presents in-distribution results of the maximum and minimum AUC, and the gap between them for all datasets, as well as their average ranks. Table A10 presents out-of-distriutiion results. The highest maximum and minimum AUC values, and the smallest value of the performance gap are in bold.
We also report the BCE, ECE, TPR @80 TNR, FPR, FNR and EqOdd (for binary attributes) of each subgroup for complete evaluation in the in-distribution setting in Table A1l to A13, and out-of-distribution setting Table A14 and A15.
Table A8: Performance of different model selection strategies of ERM. The numbers separated by slashes are the mean values and ranks of overall AUC, the worst-case AUC, and the AUC gap, respectively. Mean values are reported.
Dataset/Attr.
Overall Performance--ased
DTO
Minimax Pareto Optimal
ADNI 1,5T Age
91.83 / 87,53 4,70
91.55 /91.37 1.43
91.20 / 92.38 /1.04
ADNI /
82.77 / 76.97 / 18.72
81.57 / 75.33/ 22.73
81.36/ 78.52 24.90
COVID-CT-MD/ Age
73.89 / 55.003 8.73
22.51/60.00 7.99
71.93 / 65.00 00 8,91
COVID-CT-MD/ Sex
73.41 / 69.48 / 0.11
74.26 73.30 /0.17
74.49 / 75.34 0.09
CheXpert Age
89.50 / 78.01 / 10.80
89.62 / 77.92/ 10.92
88.09 78.36 10.33
CheXpert Sex
89.01 / 88.8 12/112/0 0.13
88.43 88.01 /0.08
88,90 / 88.70 0.02
CheXpert Race
88.64 / 87.53 0,20
88.23 / 88.07 / /0,29
87.92 / 87.84 0.16
Fitzpatrick 17k Skin Type
90.38 / /91377 3.72
90.71 90.42 4.005
91.51 / 90.67 3.70
HAM10000 / Sex
84.15 / 84.07 3.52
85.74 84.21 / 2.99
85.20 / 83.12 3.31
HAM10000/ / Age
89.23 / 76.77 16.73
88.67 / 78.42 15.82
90.00 / 77.53 14.72
MIMIC-CXR Age
86.13 / 80.64 / 6.19
85.98 81.97 / 5.78
86.40 / 81.0 5,32
MIMIC-CXR Race
87.04 / 85.44 0.73
87.52 / 85.02 /1.26
86.26 / 85.52 / 0.85
MIMIC-CXR / Sex
87.10 85.40 2.37
86.39 85.49 / 1.32
86.45 / 85.62 / 1,41
OCT/ Age
98.37 / 98.25 0.74
97.88 / 991.2// / 5,88
95.92 / 91 44.4/ 7.31
PAPILA/ / Age
66.27 55 88 25,34
64.07 / 56.31 / 23.48
64,98/ 53. 24,90
PAPILA / Sex
80.97 73. /
80,04 / 74.39 12.49
81.36 / 78.52 3.02
OL31 / Sex
74.25 / .1 99.27
74.01 68.55 / 99.13
73.87 68.93 8.57
O1 / Age
69.13/ 57.94 99.27
67.99 / 58.22/ 8.93
66.58 / 58.43 8.48
Avg. Rank
1.56 2.53 2.229
1.19/1933 2.05
2.29 / 1.49 / 1.65
27

---

Published as a conference paper at ICLR 2023
Table A9: Results of the in-distribution evaluation.
Dataset
Metrics
KRM
Resample
Domainled
LAFTR
CFair
L,NL.
Eni
GroupDRO
SWAD
SAM
Overall
85.20
$6.94
36.63
36.48
86.24
86.58
$6.92
86.36
87.03
91.44
39.90
Sex
Min.
83.12
85.04
84.93
$6.00
83.56
86.03
$4.96
83.54
85.92
91.14
59.55
BAM
Gap
3.31
3.09
2.79
0.48
5.12
0.93
3.29
4.59
1.50
0.44
0.73
Overall
90.23
Age
Min.
77.53
82.73
78.23
80.0
82.95
81.18
71.21
79.56
80.00
86.66
80.54
Gap
14.72
9.68
13.95
11.59
9.31
11.43
21.
13.61
13.01
5.72
12.77
Overall
Age
Min.
78.36
61.16
59.06
83,46
82.32
54.61
59.39
87.83
79.91
84.54
81.22
CheXpert
Gup
10.33
18.88
21,86
4.99
5.69
24.30
21.25
4.56
6.13
4,79
5.75
Overall
88.09
87.72
$8.13
$7.77
87.97
86.87
96.54
87.98
86.27
88,79
$8.09
85.01
86.24
Gap
0.02
0.48
0.03
0.20
0.13
0.20
0.50
0.01
0.06
0.13
0.15
Overall
87.92
87.76
$8.01
88.01
87.77
86.65
86.84
87.82
86.30
88.75
87.75
Race
Min.
87.84
87.66
87.88
$7.98
87.73
86.57
86.82
87.80
85.99
88.65
87.64
Gap
0.16
0.16
0.03
Overall
86.40
85.53
86.22
86.05
85.69
84.13
85.12
96.12
83.49
87.08
82.80
Age
Min.
81.06
70.97
70.87
81.11
81.05
66,60
73.18
70.78
78.62
82.15
77.97
Gap
5.16
7.76
Overall
86.45
86.21
86.31
86.28
86.36
85.18
85.97
86.42
85.89
87.05
83.65
MIMIC
Sex
Min.
85.62
85.31
85.53
85.36
85.55
81,22
85.35
65.64
84.96
86,23
$2.69
Gap
1.41
1.53
1.35
1.60
1,70
1.19
1.34
1.61
1.39
1.75
Race
Min.
85.52
85.31
85,52
85.67
85.53
84.27
84.95
85.58
84.90
86.38
$2.54
Gap
0.85
0.88
0.82
0.75
0.70
0.87
0.53
0.69
0.80
0.88
0.12
Overall
95.92
98.35
98.83
99.16
98,
97.93
99.71
99.35
98.02
94,88
97.44
Age
Min.
91.44
96.26
96.79
97.50
96.97
96.08
99.11
98.93
97.60
91.9%
96.79
Gap
7.31
3.01
3.21
2,50
2.93
3.71
0.89
1.00
0.26
3.64
0.71
Overall
91.20
92.51
84.89
84.47
94,07
87.05
77.61
88.97
83.95
85,98
66.54
Age
92.38
ADNI
Gap
1.04
3.65
0.56
4.76
8.19
4.81
14.10
9.52
17.21
7,28
7.42
Overall
81.36
92.17
$4,81
$4.39
81.82
66.75
$4.76
92.26
92.05
91.33
66.54
Min.
78.52
75.93
78.97
69.663
84.82
H5.56
90.74
88.15
Gap
10.38
9.58
Overall
64.98
74.07
61,50
77.78
77.10
69. 36
75. 75.42
75.87
75.65
82.27
72.62
PAPILA
Age
Min.
53.22
56.02
47,34
59.94
60.78
56.58
54,
56.58
58.26
66.39
$6.30
Gap
34.95
31.93
36.41
Overall
78.40
67.45
67.23
74,86
66.78
82.10
S0.70
77.33
79.69
76.66
72.95
Sex
Min.
78.04
61.58
63.51
70.31
61.98
81.76
78.12
76.56
79.17
66.67
64.58
Gap
0.08
6.01
5.87
6.55
7.82
1.05
4.23
1.87
1.22
16,08
13.07
COVID CTD Fitel
91.51
Skin Type
Min.
90.67
88.33
86.53
86.51
86.
86.64
80.29
85.20
89.97
90.94
87.93
Gap
3.70
6.24
7.05
8.00
6.74
6.88
11.46
6.69
5.71
6.75
5.06
71.93
Age
Min.
65.00
62.78
57.78
56.67
59
65.56
61
63.89
66.67
71.67
68.53
Gap
8.91
7.27
13.17
5.03
3.36
3.25
3.34
11.20
10.97
6.66
1.47
Overall
74.19
71.50
72.23
73,99
71.14
69.41
67.39
73.48
67.82
76.44
71.50
72.91
67.52
70.90
Gap
0.09
13.03
1.45
8.39
1.52
5.25
8.75
3.48
3.64
7.99
6.88
Overall
66.
67.76
58.96
62.12
68. 74
64.43
$2.43
67.
62.63
72.73
$2.03
Age
Min.
58.43
$2.58
$4.24
55.97
51.02
58.51
55.41
57.85
46.65
Gup
8.48
24.30
4.79
12,00
27. 04.04
8.12
10.36
16.75
1.87
11.91
15.59
Overall
73.87
74.47
$8.98
66.27
62.61
70.17
51.38
67.54
69.06
71.78
61.15
Sex
Min.
68.93
68.
$6.07
61.70
56.82
65.
47.31
64.37
61.36
64,23
58.22
Gap
10.29
Aug. Overall Rank
4.63
5.52
6.72
5.25
6.22
7,80
8.33
4.74
6.32
2,80
7.67
Avg. Rank
4.79
2.79
Avg. Gap. Rank
3.64
7.49
6.48
5.58
6.38
3.64
5.35
5.71
5.20
6.41
6.35
28

---

Published as a conference paper at ICLR 2023
Table A10: Results of the out-of-distribution evaluation. In the dataset column, the dataset in the first row is the training domain, and the second row is the testing domain.
Dataset
Attr:
Metrics
ERM
Resamppe
DomainInd
LAFTR
CFair
LNL
EnD
ODR
GroupDRO
SWAD
SAM
Overall
80.71
79.85
83.10
82.44
79.87
79.75
82.26
Ago
Min.
76.50
75.09
72.43
76.06
76.11
74.83
74.05
74.28
70.35
76.66
75.91
CheXpert
MIMIC-CR
Gap
5.15
7.01
7.13
6.25
5.24
5.97
5.02
8.67
7.00
6.01
Overall
82.73
82.92
82.86
82.51
82.73
81.96
83,91
Sex
Min.
81.63
81.97
81.93
81.49
81.84
81.25
81.84
82.44
80.83
83.04
82.73
Gap
1.67
1.65
1.79
Overall
82.50
82.39
83.30
82.63
81.68
81 $1,46
82.33
82.91
81.72
83.50
83.25
Race
Min.
81.40
81.49
82,50
81.76
80.98
80.60
81 60
82.20
80,46
82.55
82.43
Gap
Overall
85.87
$4.10
$6.06
85.17
84.60
84.71
83. 68
85.33
85.10
86.80
83.43
Age
Min.
78.84
77.29
78.70
76.89
76.84
77.58
79.27
77.77
76.61
79.18
77.97
MIMIC CXR CheXper
Gap
7.68
9.95
9.49
Overall
86.26
85.67
85.42
86.04
85.77
85.47
85.44
85.77
85.26
86.95
$1.89
Sex
Min.
85.72
85.34
85.09
85.56
85.49
85.16
84.96
85.25
84,51
86.56
84.29
Gap
0.69
0.89
1,26
1.01
Overall
85.91
85.64
85.69
85.92
84.99
84, 86
82.94
84.62
85.90
86.87
83.78
Min.
Gap
0.13
0.16
0.02
0.29
0.67
0.16
1.82
0.29
0.12
0.20
0.34
Overall
89.52
87,46
84.55
79.89
78.04
89.30
76.06
86
80.19
88,
$2.92
ADNI ST
ADNI 3T
Age
Min.
Gap
0.18
0.52
6.22
2.44
9.27
6.76
3.78
0.52
0.91
1.31
11.63
Overall
84.33
86.36
83.05
79.40
83.75
62.05
86.
89.91
86.45
86.47
52.92
77.63
Gap
0.24
4.03
0.18
2.73
1.92
0.24
1.05
1.35
0.41
1,46
8.92
Avg. Overall Rank
3.81
5,88
5.75
5.38
7.0
7.62
8.5
5.06
7.62
1.38
8.0
Avg. Min. Rank
3.88
6.38
7.19
7.88
6.81
5.12
8.25
1.75
7.25
Avg. Gap. Rank
5.06
4.75
8.19
4.19
4.81
7.88
29

---

Published as a conference paper at ICLR 2023
Table A11: Results of other metrics for HAM10000, CheXpert, and ADNI 1.5T dataset
Dutneet
|
Atte
Metrios
Group
ERM
Resemple
Deceiitted
LAFTR
CPair
LNL.
EaD
ODR
GropDRO
SWAD
SAM
Cpp.1
0.
0.2600.00
0.2640
0.404002
0.37-004
0:540000
0:3740.01
0.51-000
0.1-0000
0.2000:1
0:3340:00
0:3340.01
BCB
Cuy.1
0.004000
0. (000000
03ken0)
50556004
0344000
0336201
010:621
0302000
00si001
00
00550.01
Sex
Ogg
73.05433
76 64
77 7144
75:23:1
78.51409
TS00-43
74
15.51441
70.59417
TNR
Eeg
41.33484
71 m1-55
71.9-74
66:9179
75714255
0037-50
2.71-13
24
8700435
51.1043
PAR
1ap
1ap.
7.34
9.8445
15
11924427
1085635
1036.15
531115
7.41359
123645
892142
1204444
108677
162165
756121
10
596125
TA1125
52141K
K78637
5.292.1
FNR
Oap.1
$2.56873
44.23215.3
$2.561335
44,2327.
55.9721.11
12,49199
45.328128
S249232
31
$2
99,9429
57.5270
3149239
44.7278
W11264-
9181U1
45,3043
946200
HAMDN00
Dep.1
0.398003
0.
0.100101
0 025
Oip.1
0.228001
01663223
0 3
02110012
0 025
6212n01
Ckp.2
04320114
64188/12
1.
0 6
6.62n66
B422000
Cap.1
0.250001
004
0 0322202
0 02--022
o0isn0)
BCE
Cip.1
b0
0.366
0. 1eei000
2034000
0.5e500 0506000
Lap.
0.200005
0.15ane
91.15470
5100-12
5925153
1NR
Exp
24 1
2
Gip.
75.22+22
0 P.lell
77 77.0050
71,55105
75,47229
Lep.
4.3511
$32219
T41229
103022
4546L
545416
Upp.1
5.71808
584220
5200877
1002832
17.5655
24
FNN
127
Crp.
75.33872
35.68123
25 2
7038226
5
5417272
S7.31e4
2408168
Crp
0:37-01
0554000
0554000
0.3540.00
Eep
0:2-001
035-011
0224000
ne15000
026-120
07tung)
07
0140110
0714000
0154200
B 07
075-000
026-120
0754400
6122000
01330000
Leg.
01220012
0100002
100000
tpp
0020011
Comaanto
Olp.
061-010
000-00
0066600
6346600
0006501
010-06
0 03
6dianto
Ckp.
0010000
0000001
000-000
0024000
0:500m
n 0000000
0 000-00 000-000
0 0
60
D0100
4g0
1150000
1NK
Dep
1637231
6520415
12 12
73 7553-16
910
T-48-25
2 2UI
301441
6L70330
Cip.11
21.35222
2502211
1305118
114420
1S
1
1151216
122983
7
129111
5.331.7
R13822
10
592201
Clelppol
Oip.1
1666855
3.9480
2333455
12
34.5566
5,45220
S15624
35355133
61822
2
153339
21 31
6
1722013
1444el
17.7811
1
10.12335
25 0
1
[50004
26.802
2301550
2
[68721
18.6600
5
34
60:17633
17.78842
Th 100811
1
3
40.5655
7809415
5651536
1
53.46199
BCE
6334000
035-000
03340000
0354505
022-000
025-088
100-001
0314000
BCB
tip.1
0010011
00040000
0001000
0002000
001311
10080311
B1002000
Six.
Erp
81 110001
$105-0
T843406
N111
510143
K32946
1NK
Ckp.1
$1.3500.
$1.621.
$1.35a1
77.993.1
2696207
$155.0
5135503
FFR
Lap.
Lip.
8/580.7
765111
841406
2.8803
9 1000s
7000
VASULY
9.22112
118329
744627
KINAHH
1NK
Oip.1
4091822
43.3923
35.11113
56 1.
5:01:20
3500026
4369892
4L35214
56 363381.9
3694
4058230
ByDak
36.54005
07.5% 4
4300
3
97
590850
96.1693
Oep.1
02230100
0122000
0208000
022
0.388001
0338003
O 220000
6222110
Eep
127400
128-000
00346100
m3460m
0344000
07-0000
0-4-006
67740
BCE
Ogp.0
Oig.1
001:010
001:000
0314000
0.006000
0046000
000:0000
00lan01
001:000
00
0.020..0
000.0.01
000:200
040-001
010-001
0 0305004
00
0015000
66150m
Raaoe
(ep.10
7933407
1NR
K24041
14454
5185-02
02.21410
44-04
FPR
Lip.11
K38805
8 812411
1031837
591212
5 1AU11
3
93741.
K35003
192813
FNR
Exp
1pp.1
37.6462.7
18840
41.441K
4154225
35.5500
3615646
160914
M
3721227
7
37.4600
9633803
9 3000
98.13804
1M01803
BCB
Eip.1
113140111
0366009
02241 11
19
0314200
1161011
0-8-4-30
025-003
035001
DC1
Ckp.1
Upg.
8222118
0148014
0224200
0144000
0 0.26110L
12
133660044
1440001
S16a001
6172001
159.0500
Ogp.1
44.5a113
72 32
6501255
65.7711.
6052247
R0.3
35 35.7
8699274
35772222
1NR
0ig.1
12.22892
56.00125
60672319
92.00139
5010E5-1
47
22133
20.72112
661122234
33.392258
2252248
1887283
Oig.1
10634145
23.334137
14 1433143
15.57413
11.114130
EL EL437
111142
35.3373
55 5666633
ADM
INR
300000.0
26578113
20m000
1
1
1400550
1338111
0
13332115
91,34443
87 19424
91 99669
$1.99776
8747438
8191-9.1
61,50-29
BCE
Exp
Ear.
14
04400000
1534615
1424611
03001115
04558114
0.350005
DASAIIN T
BCE
0212000
0.300010
0.208004
0,302077
0.348900
0.19a16
11518001
Epp
HazaisA
15-11155
$1511A9
35-211122
$2.041455
16 634383
26201337
7367411.8
24611253
7187475
BUCOM137
FPR
Cep.1
0lp.1
1122311.1
2634672
19834319
165922107
14812110
9.22844
22.222105
34.1204
40/42221
12:13211
22
(4812170
19522251
18.22a12
15612722
66672222
45-12324
INR
Gg.
0pp.1
31.412477
13316153
30.00113
3n5544
40486105
23.3
23
33022100
16574208
9.52
33.38253
21-3:124
14266711
607258
143te124
16072113
3.3458
$5.79396
$4.0033
5865.1
91.49114
93.21359
13,546233
16.11133
$1.72544
5563274
7677812A
30

---

Published as a conference paper at ICLR 2023
Table A12: Results of other metrics for MIMIC-CXR, OCT, and Fitzpatrick17k dataset
Uataer
Altr.
Matric
Group
FIM
Reample
LAFTR
Chair
LNL
Ira
OHR
Gropphio
SWAD
SAM
Cmp
1:044001
IL1540022
(.644D09
0-860000
0-
0.2LI131
04410001
152a0115
Cm
0/448000
04440100
1004005
GANARC
032.1101
04640011
Cm
04778110
1000001
0.47A001
10228115
Cap.
0.348030
0164001
Ctp.1
0.1520:0
0.442/02
0.042001
0
BCE
01023101
00350001
Cip.3
0436n01 01
01
0.005.0
00
02
00360001
03588L11 11
Cop
081-8000
6.04001
08--001
0.8-601
00640
015-6
00340
78.33e2
T9
7791224
3451233
7163
763628
(6.6666
TPR:850
Cip
1
73 11t
T4 136
4
13/9433
7
TNR
3
03333003
3383
6LASEE5
SURALI
Cap.
9
39
40
4211830
35,14223
FPR
Cip.2
36.75213
11.2
30.00
206562.
A
21.55a1
53,02a1
21 2188
2112212
250552
2001219
5
Cap.
15.00115
16.7
167 Tiec
262222
146722
18.362.77
S0eE1 5
133767
50A113
S
MIMIC
543-02
12
100M00
013H16
34:74-10
128111 6
Cap.4
41.311177
39.34227
4244899
3
34018110
4100222
8025139
4238434
4157230
RCC
0A34R00
0511002
0470000
0260000
0440000
Cap.1
0/020111
0002001
01002200
01142001
00
0022000
001
TFR:850
Gyy
16.8800
2
7611502
3
25
5
TNR
Cip
7833
7831606
77
75
17
3
79:3300
72.5509
1911
7564-1
21 TOH1
26.0408
236699
265900
1696-1
2576-15
2
2604406
240141
2171464
2157667
2515406
ENR
Cep
20377110
13.9541.4
2005a14
15.12007
1932662
143900
19096122
14MADA
19098607
1751117
1587116
18.29113
1831119
1093861
3002A1
5453002
1826007
16.40022
15.021133
9541119
99053066
15.221
Cmp
0A30R00
B5400
0600000
WARMALIV 19
Cmp
09600100
0.460000
047
Cep
0.00900
01G2001
02
BARBLIN
Cip.1
0/0250011
00000000
0.010000
04004001
01020001
00020000
TFR:300
Cip
16.19994
7650684
35.2
3
TNR
Cip
769928
76.13915
17.0156
7..0605
74.770
963348 9
75.8490
118350
Crp
11.14-13
5230-15
5165-14
5041-30
533440
504540.5
FNR
Cim.
21:2281-
212401
1260003
2
11B1114
2121AR0
12.00113
2214217
14.381.0
1301005
K
1225804
X
40:1440
59:57403
50 907962
W159-077
91 55-05
9064403
9017405
5045402
50
SCE
Cip
0.110110
80000101
0110000
0.22001
0173111
02
Cap.
0.000100
01112001
00720012
0.112001
0133500
00721
Cip
00030002
OCT
TFR.850
Ciy
$7.004.0
8044239
8344457
1711
5
40.14560
51.5
7643451
57.1
Cip
3000a00
3000600
2000600
43.914414
100000.
41435575
5005651
1911
Cip
Cyp.
5.88550
104418
10441K
588659
0.0050
020406
39225-3
186-14
538859
104-14
5.885.
10401
000910
708458
302234
5
195834
1.663
1
ENR
1212252
6000000
15
666755
000.5.0
18:5.1
18
33565
3 335
11:55.1
1515e53
667758
1.37.58
1000
1311252
1000
20000100
1515552
Tap0001
41.5742.3
Crp
37.45551
35 18.66133
165 168557
10:45
15:3.0
30072233
56.4e1
31
Cim
31,003533
41.92413.
119108201
2811116
34.21224
1219819
8CE
Cip
24.0036
21.9222
19.5246
210024.
55.2511
21.5224
1234815
25,809
Cyp
20/76470
25 25985165
5441214
1802251
1551119
22714102
Cip.1
23.21a717
31.204142
18.339004
2011454
5404213
15.44
2007118
1539443
19S1a10.1
Cpp
6.325.6
1742317
6.9613
455e13
S3
31
39:910
178626
628e1
9.6655
Cip
5.782255
11,72200
1558457
75542.
357210
53k.10
25
535000
63811.
591513
BCB
Cip.2
4.
43500
185129
T
7:30AS
4TOYT
LOURS
24102
FlsgeiskITL
Cim
Crp.
94212511
5533235
508944
7755611
68811
9015364
5545225
589144
B63524
SH6L9
Skm
Cip.
42.79224
SU/9226
SLATB4
typ
Cap.
3404120
H4
9191aL&
Cip.:
Cip.4
06.122911
$1.76255
16
9
1
S6.
682325
$50520
$159551
1
T102at25
639923
T5.4240
518an8
251410
E51.29
651a16
7:5T0077
Cip
539911 5
500655
6AS-30
113055
533a1
Crp
70992.
178-11
188
609113
60cch
581220
708422
1
Ci
1.51119
260677
1
334814
4,11839
Cap:
426452
104600
635442
S33824
6J5AL6
3162215
2535230
54
Cpp
13.202.3 5
1133566
417741
54 1
307
9147219
3637545
26.44255
24
$2.724
Cyp
30675466
2210835
2500
20/7266
3057200
2405
1867885
34,77538
Cpp
$6.7258
463555
53
6567755
34.1140
319
2000000
3333333
Gip
51.381144
41.67144
3630665
33.33433
30 0000555
33
58.385322
41.67520
51.32144
25.06250
41.57a14A
31

---

Published as a conference paper at ICLR 2023
Table A13: Results of other metrics for COVID-CT-MD, PAPILA, and OL3I dataset.
Metrio
Groep
Receuple
Dansinded
LAPIR
CFair
GroppRO
Cap
0:76a1:22
0.7380185
0.7121044
0.7zat1c
1.2128000
LIMLILZO
Cip.1
077201115
0.7780.16
1.465.33
0/741115
(.974019)
(174000)
LINGLL
1.240UM
BCB
Cip.1
Otp.3
0.26a1.99
0.3
0.228019
0.34g000
0.368009
0.085005
0 0.5521055
0.
0.33
0.2389:05
0.3780:02
0.382004
03312084
0.516033
0.5680.00
0.5480044
0.000.12
0.66n60
0.2021005
0.308114
B4160116
Age
TPR
Cpp
48:91687
46.66331
34.46177
55.5
53
11
5211-497
50443276
50.3500.5
TNR
Oip
18.0768.
37.564139
37 339.
24 3100139
32,0:139
37.601100
29 33333
50 1093
45.33624
3644696
Ctp.1
47/02208
74.608110
1401527.5
39336873
65.474130
52 32
50.96733
44.4
25.7581277
COVID
FNR
Cip
4000=20000
01,55400
13.37411.5
7.35225
10 15:25
0.00000
1301-151
N607666
33.33-1155
5,80450
36.7711.5
14 6.66
2667-115
11 11:050
33.33-1155
5.5675
1169112
80.95412.7
11.84511
799511
7177-199
7699436
06.74108
BCE
Cip
0618116
0.9680.0
LI8494)
10
0.918112
Crp
0:240115
0
020000
0.3300 m
0354210
0254011
0.3140N5
0334001
Six
Cap.1
$1.551111
15.368577
12.411111
305187
S2 12
41
57.331122
4671843
INK
Cip.1
A1J05123
63/114501
6.865
41,922002
4200644
S1386660
48638108
53.318153
FPR
Cpp.1
Cpp
30.66127
44,44890
61.11221.0
63.498120
53092153
3000000
5314
22-12
58225773
47.2299
56.0022.0
6546673
77.7644
5633623
$258625
36.111175
41.77120
55.55277
44.4489.0
1NR
Cip
2564644
7:6650
10 1024-4
33.07-133
1242644
513444
25 3300-304
178244
N051589
2051:160
ByDM
19.1266
$6.8450
80117554
9078652
9382203
15.18122
0.31-d
m.
05350m
0 13
030-113
0.3840
0 036802
0
437266
Cep
07460.55
0.366001
1366815
0764011
17
0.5760 18
06670006
170003
19053 60
1.2500 23
BCB
0.2500102
0.17800
0.36801
0.110000
0318602
0 0.2ah
01
0.3080 05
041600l
0245503
0 0
05
0.35an
0.3
Age
Cep
10-100
TNR
13.77647
50433
1667-79
1931451
1524-54
M-99-10
1981679
FRH
Carp
6:1516
4716-155
105-55
2084556
18751272
SUMIN
218.24
37,25414
4.7422
2.236222
RAPIAA
FNR
Cip..
Cap.
$1.312289
23.31852
5000820
19.06332
33
2457214.3
169722K
15.12269
25.57200
38.571143
502033011
16812189
30336833
19,888
51.88229
1980282
16812249
3133383
19.06882
44914182
642211
$443120
ST44466.1
S01122
48754175
Cap.1
06424112
10880.19
0574804
0 5554005
03721112
LIGHLIT
06420220
3614261
0-420005
Ciy
01198018
0.30001
0338004
0
02550
03382044
O 15602
0 020005
0255006
Cip.1
02450.00
0.3680.0
0.199001
0.778006
0.33
036e203
0
TTR.8 50
Crp
1764-310
4500041:2
11 11:669
1300-44
1031-179
25
25 67-777
17 38-1-25D
TNR
Cep
464766
15-044
17 16-60
3411-175
19 17--157
4500-108
FPR
20:33336
31,37834
25.496122
20
41186310
23.1429.
4583:72
41065307
55002380
18518143
1665272
2157790
855e5.3
3137290
23.33102
31.3386
22.9277
3
41.8859
FNR
13.314115
35.314114
7503-00
7500400
41.54444
7000-300
1335-331
1535-115
5505-0.0
1531-115
02.45622
$199353
76 101--111
BCE
Crp
Cep
0:0440:8
06144
0:004001
0
0154000
0334000
0 10-001
031-001
0134012
0.15401
0664000
0 0.W-008
010-001
0 039-033
0554011
0 0104001
0 67144
BCE
02680.11
0000011
01068077
041400
Aac
Cap.
4137210.1
6111642
27176
5022221-
40
SLIMERS
2624411
26.3645
2310g1011
27.33316
Cpp
15006911
6
21 2LSSA
$210125
Cip
3733-20.
30666756
33006163
34976123
1790114
12.55400
18-100
37.46220
11.1589
IN8
Ogp.3
Cip.1
63872173
61.12314
53.52277
50000220
105923133
<5442356
50 3922012
S151-160
72.72:13
65412221
T899859
S0 5
9242569
(3 GUREIR
56536300
5151269
69441175
02122266
75.794722
66612220
90.33667
5441e52
90.7557
50:50
M4
83.43:244
$1445115
98.11
Crp
0.3141
1
85340
0
O 028-005
015-065
0.144
021-014
018-110
0174400
BCE
Cip
Op
01446 14
11
01346011
0.3460.4
6.968117
0 00ich00
004-000
0.1180
0.999
O 11101
O OmA602
00000110
0035001
0045001
00
Crp
4007449
42.14138
23434113
1596-121
1615661
16
3107467
TNR
Cpp
5111-100
550041355
51 16-181
17.19458
18104K1
S18100
1525411D
3189475
FPR
1182107
17,541377
54,88334
5
19722146
10906510
190824
8081265
Cap.1
51605224
46658225
61618153
5333458
705083-7
31672310
00002000
41.0229
6533276
13.33558
9134652
90.12246
75 69444
94.9155
38.35887
91.38646
96.70220
90119559
VL.7643
32

---

Published as a conference paper at ICLR 2023
Table A14: Results of other metrics in out-of-distribution setting. In the dataset column, the dataset in the first row is the training domain, and the second row is the testing domain.
Datasel
Attr.
Metrica
Group
FRM
Recample
Demuinled
LAFTR
CFair
LNI.
FnD
ODR
GrouptiRO
SWAD
SAM
Orp.
0/64-0:01
0:0140.00
1:3040.31
0.56-000
05
0.244600
10540:17
0.8440:14
07000.000
083-4007
Grp.1
0:30m0.:0
0.9080:06
2.0760.30
0.718803
0.2an.14
0.74g/
0.9720.17
0.8089.10
0.726.00
0.032
0.72866
BCE
Grp.2
0.7260.01
0.80e508
2.168023
0.595005
0.750000
0.775305
0715300
Grp.3
0/3480066
2.3
05965005
0.
0.5en00
0.7.5003
670e5002
Gm
0.6.0012
Gp 0:7860:01
0.40000
05
0 0.0000
0.5540077
03354005
0 07W44000
0:56-001
Gp
9:7940:00
1540.00
0-434001
0.796005
05:4000
0 03
04068.07
0.354004
0:34-000
0:792000
Grp-2
0.26e0000
03368000
0.465002
0246007
0.2660002
021
0:540011
0.2220003
0.3en000
0.1550
0.270000
Dp
0:1000:00
0.116662
0:2110m
0.100002
0150010
013:0.0
0:110005
0.151001
01410.05
0.21 (20-0000
0.571801
Gr..
73 0
M66.0
7205440
53.001.
11:20
26-22430
1355-300
Gm
TNR
Grp.3
50.51200
54 14..0
50.77330
62.563.
6063850
57.93550
65 12
6761220
60.25a10
5151820
61.365D
0505550
56.3425.0
6108:5.0
56.3211
715:111
65 63.97A10
$6,99110
63.5200
Grp.
58.4721.0
$4.00220
$1.3125.0
$6.42200
$5.20220
S4.11110
$4.8222
50994110
18:20
$7.52110
$5
Grp.
15
52.75e10
11.5011
SE57650
000055
17.00-10
FPR
Gp 38.0-10
41 16620 D
4151450
ST 57.8630
TW WIC-DD
35.3841 0
40.0711
Org- 24-2221.0
25-22220
3021330
1-0022
25.0721.
26 365020
34.91500
23 3P220
23.94800
25.5-10
Grp.4
129921.0
18 0
22 -See
25.244500
14.775.0
13.2200
Chelppr
XINICCAR
Gm
2:3:200
11:11:7
43S110
456100
4.1120
4
2.0000
Gp-
4,00000
590030
581e50
1199:20
1
5,00000
4.2-10
PSR
Gg.
10
1090110
13/541
1155c20
12/1220
10 1ORRERR0
1595220
1132230
12/421.
1200-40
Gm
29 16220
2700120
222322
25 P
23.8220
31 31M222
Omp.4
45.304/0
4250
BCE
Gm
Gip.1
0.7560008
0.700:044
DAT100K
06600000
0346000
0320007
064
0
0.711.114
0
0030001
0:00001
0002000
0
0.13:004
0.200.00
0.212006
0,3,0000
0.200011
Gip.1
02550.00
0.218006
02256004
0.645004
0.34000
0.238000
0.239.00
02
0.302000
02442002
Scn
Grp.
5835110
50-00:20
6625111
68,66120
65-0120
638TAED
59 00.0000
1041110
71:1100
1110.000
Grp.1
T0.6610
71,35e1.
71,35+10
06.76220
72.9500
72.3
FPR
Gip.1
Gr. 25.80210
31.9121.0
25.3333
30.662.
2357210
51,5410
32
20.4220
3804250
36.22200
23652220
33.96330
32/02110
23.4121.0
23 3333711
3183620
26.48
30,20110
28.06220
31.0220
FNR
Omp 16.45110
16.961
13,0450
1635-10
13255400
16:36430
3055-30
15,9e100
17:45220
16:0
16:0-20
Eq0ld
96.0011.0
97.00e0
96.4+110
56.2=10
9130010
94.57240
96 14
9600911.0
55 0
96.0500
5642-00
Omp 06ka0.02
09588.12
05020002
0.02004
0652801
Grp 07746.044
0.774006
03
0.744004
010-0005
07540001
07140000
0.340006
015-0005
092-000
07110001
ECE
Gip.
Gr.
0:440000
0.200000
0.190000
0.460.05
0.230002
0.2760003
03
0.1+001
021
05560
0.200001
0.200002
021
0.200000
055-0000
0.0e301
03550001
TPR050
Grp.
58
18.22
65.7230
6939830
60.0211.
62113110
65/60220
68.6522
68.81210
96 7672
70.99110
TSR
Op
56
91-10
463664.
10.4400
46:47-10
65 61:95130
6604/50
A69611.0
88 15-11
69 49-0-00
Grp.
27.07230
2907733
37.55430
2409e1
Grp
36 0M110
8365.0
3633630
35
0
16,30200
PNR
Gpp.1
Grg.0
15.1111
2611210
22:0022
1218100
24.70230
11.60110
34.87110
15 19.1.
1581110
23.0220
22,47220
12.30400
25.6GLD
21.92210
12.49110
2391:110
15:36100
25.17220
1549.10
Higradd
17.80110
19.35610
18.35410
18 104000
89 89/000
67 97-1-10
9897400
1805-00
Gry
0.4ke0003
06360.10
0:000011
05
054elus
047200L
0.489902
550en000
0372016
Gm.
0.3010:11
0464002
0:200005
6421605
041:000
0-4mmm
0.960001
0.392002
0:3000000
60718111
Orp.2
0:2900001
0,30+0:0
0.45-003
030m000
03540005
0.46000
0256600
0:5-0000
0.718224
Op
0.356000
0.3160.00
0.256001
0336000
02250003
0.
02240003
0 01366001
0:4-000
36
Grp. 0:15e0033
0.788005
0116600L
0,1853044
0220005
0.11000
01122002
01540001
Om.
0:1100:2
0.121662
0.106000
0150000
0.140002
01100000
0.10060
0 0121000
0.22.000
Gp. 0:120:01
0.160000
0006500
015
0.176001
0136000
0.355000
040e0111
Gg.
9:1560.01
0.200000
00100000
0.14
015
0.
0
0155cl
0.39e5
Gp..0
70.6685.0
7065540
11.001.
7045500
5634450
71.52500
59.25300
T0 50
TNR
Gm:
Gp:
Gip.1
78
19 1.0
74 145622 2.0
1049110
79.0010
1111
50 28:2
Th R55822
15 17
72 22S0TE
7030010
14.5811
5042111
50.0422
6111110
79.7722.
11.40110
88 29:20
19220
72:311000
6419 20
11 11.0 10
12.68L1
Grp.1
63.55330
63.1622
0392220
50.01
66.07440
63,0110
Grp.
22 12.202.0
25. 1600
19,3950
2533220
3305500
4407270
16535LD
18.75550
23 S
19.55a2.0
34.622110
Gip-1
167121.0
1581225
2009820
14.922.0
1698220
1682220
17.77210
21,9220
Gp
11
1
2
1091-10
130010
1741110
18-20
11:87410 i
20
Cmp
101620
MIMIC CAR CheApest
4.261.0
30
7
7,0022
6,251.0
3/65+1.0
6T
Orp. 166725.0
28. 19570
2533555
25
3556220
11:10
27.22150
005627
23.33230
Orp.1
02550
25 1129220
2549240
2.04550
20.91110
13555400
36 3434
57 01-00
443575
12 00:05
5136670
10 13.3550
538--0
Gp
7630-20
38 15400
7131/10
66 6090000
81 56100
73.47440
01 41.11450
1030-60
Grp. 3.300000
0314000
0334000
0394000
033+000
0,50-002
0300000
6.200004
0312000
03230.0
0.000000
0.292002
0.55000
ECE
Gmp
Orp.1
01210001
8:210/00
01110000
0:410000
00510m
0051000
030000
0.111004
0122000
0115000
014100
03400:0
01110005
0.16000
0.11101
0.000033
5 11100001
11000001
0.26 0000
0.26 000
6521906
1990660
75.83800
79.5921.0
17,3300
TNR
0ip.1
769011
7631a00
75.77100
77.001.
7696220
76.9850
7637LL
76.5022.0
76/75800
74.92-00
Grp.
924310
9.5110
1038110
9.9119
1061110
1035122
10.11100
11:211
S8111
12.100
Grp.1
80te10
9,0050
9.461
1065640
7
9/991.
1033000
PNR
Gip.1
Gg.
42.0a40
43.60550
42.5580
42.622
44.5633.0
40.32110
4157260
41.9911
$100220
41.5550
40.895500
3874220
4632540
42.2051.0
39.6220
30 78220
43552300
40.4223
41.511
37.9020
Eq0ad
WK4221.0
96.441.
97.4020
05:27330
97.3310
BCE
Gip.
Gm.
0.3200008
0111000
0.710001
0.300001
02510m
0.25000
0.
0.11000
0.31200
030-000
0.534000
033
0.3G
0.51000
022100
0.25000
01110.11
0.2-0000
0.582000
1.69
1.091132
0.110002
0054000
0.1380002
0112110
0:130::0
0.22000
05442007
Gip.1
0:2200.0
0.118000
0.08800
0.138002
0.113002
0.1300.00
0.8929022
01350001
0.202000 00
05422007
Rase
TPR050
Gg..
7%
11.111.
79.34e00
7461-20
25.59420
2424220
26.113.
30.55000
13
TNR
Gip.1
77.85610
77.4622
14.24
74.87410
7383220
73.99440
Meal
30.31600
13Me10
FPR
Grp.
Op
Billa1
94501.0
10:0an0
9,31810
5.16110
K.le1.0
9/00220
11.06a10
11.25 11:20110 11
11. 11.7141.
11.35820
911211
9:001
9.7123.
$9413.0
10 47410
8.821
4.411008
3.961100
1556-00
FSR
Grp.
43
0
44.266..0
(0-60
36 01-20
11440
45 15.1144
42-50
(1106110 0
1697-11
Egold
90 15
99.39430
00.000100
09.25+310
9% 43
98.656100
50 32.301
09.01.000
5k 1.
33

---

Published as a conference paper at ICLR 2023
Table A15: Results of other metrics in out-of-distribution setting. In the dataset column, the dataset in the first row is the training domain, and the second row the testing domain.
Metris
Group
Domsinte
CFair
GropDRO
Cips
0/7200
1.358331
I 1iSA1S1
Sesattlle
202400.0
n858om
0.
0:3aan12
1022000
ICE
Cip
Cpp
0.23a1 10
0.165000
03380
0.368005
0.148804
0.2080055
0 20e0004
0 0
5.1
0.150006
0:319000
0.3460
03265100
0358203
Ne
0
0.
0.135001
0.5500111
Crp
$8.39450
10
10
51:34-000
6110-250
75.15400
31514100
TNR
Cip
$1.804600
8331470
80 8031-70
75-31
68 66-44160
90 10440
6000100
90-10
6863-15
5470-13D
76114170
18.3099
10.1143111
77
19.111999
41941220
17:00710
20166120
43001260
11.91221
ADNI
ADNI 11
Crp
15:07480
W1111100
13-116
7500-158
747341711
2161-170
62545
837447111
16.77800
$3.304511
WIBLISO
7144410
$6.75330
BCE
06351114
0.500010
1008948
0 0.702010
0.589
0.545001
LIGEA4
05388.00
0.9120066
07335005
Cip.1
0
05
Sex
Cap.
5976250
50,7511411
54.991500
5L.7111.
6405223
104441900
22221.
75.223150
2639890
Ctp.1
712231110
15.55
64/92100
404111900
9L.0550
73.1764
17,55550
FRR
Cip
Op. 15.38110
26.51
16:3141100
34.4650
54.1422.0
30 30.0090
42.2211
31.3e14.
3358c140
26792100
10.0033.
59.2143500
16672240
10 T0
1491.50
5151110
21 11000660
1533650
11.3558.
56.5550
1N8
Cep
1013417
8.041100
200042501
31.004110
1451-10
135660
1:26-6.0
31:95111
23.33410
70 17.0
14754900
11-80
11204110
1915470
13.33-15
35.7347.
K.546.11
0.24600
34

---

Published as a conference paper at ICLR 2023
D INCORPORATE NEW DATASETS AND ALGORITHMS IN MEDFAIR
We implement MEDFAIR using the Py'Torch framework. We show example pseudo codes to demon- strate how incorporate new datasets and algorithms. Detailed documentation can be found https:: /ss zong. github. htt.hh................ //MEDEAIR/
D.1 ADDING NEW DATASETS
We implement a base dataset class BaseDataset, and a new dataset can added by creating new file and inheriting it.
from catasets, BaseDatsstt import BaseDataset class DatasetX (Baseaaaasta))
def init (self, metadata, path_ tac__tto_mmmeeeeee sens _ans_nare, sens_ cens_classes, transform)
super (Datasetx, inil__iii_______ (motadata, scns_name, sens_classes, transform)
def _etitem (self, idx)
item self metadata. ilec [idx]
ing = Image. open
[idx])
apply image transtorm/augmentatiii
ing telf.transformiime (img)
label torch. FlohheFootteensr ( (intt (item ('binarylabel"] 1)1)
convert sensitive attributes to numerical values
sensitive self. sesf sens name, self. _ens_classes,
item)
return img, label, sensitive, idx
D.2 ADDING NEW ALGORITHM
We implement base algorithm class BaseNet, which contains basic configuration and regular train- mng/aaiidation/estting loop. A new algorithm can be added by inheriting it and rewriting the training loop, loss, etc. if needed.
For example, SAM (Foret et al., 2020) algorithm can be added by re-implementing the training loop.
class SAM
def _init_ (self, opt):
super (SAM, self) initt
def _train loader):
gelff.etwwrk.trriinn for i, (images, targets, sensitive_attr, index) in
enumerate (loader)
enable bawtle_rnninngggee (sel networkk
outputs, features self.network (images)
loss self selff_criterrio (outputss, targets)
loss. () backkard
self.optiimzer.ffisse step (zero grad-Trra self.scheduler.ce
disable_rumiig_sstttssssss (sel. (self.network)
outputs, features self.network (images)
self sclf._ (cutputs, targets) mman () bbackward
(zero_grad-True)
self self.schednler.sstee/
35

---

Published as a conference paper at ICLR 2023
E ANALYSIS OF SOURCE OF BIAS
studies to analyze the source of bias. As shown in Table A2, we can observe severe data and class imbalance as the direct sources of bias for both datasets. Thus, we utilize resampling strategies explicitly mitigate the imbalance. We use three types of resampling to upsample the minority subgroup, class, or both subgroup and class so that all groups appear with equal chances during training, i.c. subgroup resampling, class resampling, and subgroup and class resampling, respectively.
We take two datasets HAM10000 and MIMIC-CXR
For the HAM10000 dataset, take the cartesian product of age and sex subgroups to construct 8 intersectional subgroups, whose statistics are shown in Table A16 (excluding age 0-20 as it has few samples). The results of the ERM, resampling strategies, and the rest-performing SWAD method are shown in Table A17. The worst-performing subgroup is "40-60 Male", which neither contains the least number images nor not the most class-imbalanceed subgroup. Also, it can be seen that for intersectional subgroups, different resampling strategies produce similar performance as the ERM and sometimes even worse, where all of them are worse than SWAD. In other words, there are other sources of bias beyond the observable data/class imbalance, which explains why resampling methods do not lead to better performance. For example, the skin type can be a potential source of bias (not recorded by metadata) it is dermatology dataset, and lesions darker skin are intrinsically more difficult to diagnose than that on lighter skin. Overall, they are difficult or impossible disentangle given the metadata as the sensitive attributes of different datasets can be correlated in different ways. Therefore, methods specifically optimizing for one particular factor are usually less effective.
Table A16: The statistics of the intersectional subgroup HAM10000 dataset. M. represents Male and F. represents female.
Subgroup 20-40 M.
40-60 M.
60-80
80+
20-40
40-60
60-80
80+
% Images
7.27%
9.52%
23.09%
23.05%
18.97%
10.96%
5.16%
1.99%
Malignant
4.53%
6.60%
10.93%
8.09%
25.75%
20.77%
31.14%
34.72%
Table A17: The performance of different methods HAM10000 dataset.
Attr.
Metric
ERM
Subgroup-R
Class-R
Subgroup class-R
SWAD
Overall
87.24
88.03
87.89
88.76
91.46
Sex
Min.
86.48
86.23
87.42
88.11
91.14
Gap
1.59
4.43
1.13
1.28
0.54
Overall
90.00
89.72
87.38
88.60
89.61
Age
Min.
77.53
82.73
79.41
85,02
86.66
Gap
14.72
9.68
8.85
5.52
5.72
Overall 88.13
88,03
88.78
87.75
89.32
Intersection
Min. 78.60
69.33
79.59
75.12
81.25
Gap 17.23
24.02
17.29
18.43
15.02
Following similar procedure, we further analyze the intersectional subgroup of MIMIC-CXR dataset, where we obtain 20 subgroups by taking the cartesian product of the age, sex, and race subgroups. As it be from Table A18, the number of images and the value of labels ("No Finding") vary greatly among different subgroups. The worst- porst-performing subgroup is the "60-80 non-White Female". which is imbalanced in data and class, but also does not contain the least num- ber of images nor not the most class-imbalanced subgroup.
From the results of Table A19, do observe slight performance increase of the worst-case sub- group after resampling the minority subgroup. class, or both, while there are also decreases in the overall performance. A possible reason is that the label noises in some subgroups more severe
36

---

Published as a conference paper at ICLR 2023
than the other subgroups identified by Zhang et al. (2022), and the upsampled subgroups may contain more noisy labels to worsen the overall performance.
To summarize, most of the medical imaging datasets contain multiple sources of bias instead of only one, where they are correlated different ways and is difficult or impossible to fully disentangle for separate analyses. The mixture of multiple confounders makes the algorithms that specifically optimize for one particular factor (e.g. data imbalance) fail to succeed, i.e. do not outperforms ERM. This, some extent, explains why the domain generalization method SWAD is the most consistently high-ranked algorithm as it does not have a specific assumption of the source of bias. Increasing the general notion of robustness may be beneficial various kinds of confounders.
Table A18: The statistics of the intersectional subgroup MIMIC-CXL dataset.
Subgroup
% Images
% "No Finding"
0-20 non-White Female
0.12%
57.48%
20-40 non-White Female
0.22%
72.18%
40-60 non-White Female
0.11% %
81.27%
60-80 non- non-White Female
0.3%
80.72%
80+ non- White Female
2.50%
50.69%
0-20 White Female
4.01 %
62.38%
20-40 White Female
2.58%
59.98% %
40-60 White Female
4.22%
73.8%
60-80 White Female
10.18%
40.51 %
80+ White Female
6.75%
47.58%
0-20 non-W Whine Male
7.03%
44.91 %
20-40 non-White Male
7.17%
55.74%
40-60 non-White Male
15.27%
29.67%
60-80 non-White Male
6.12%
34.17%
80+ White Male
11.35%
32.43%
0-20 White Male
6.88%
38.9%
20-40 White Male
5.41%
22.04%
40-60 White Male
1.60%
21.54%
60-80 White Male
6.03%
23.64%
80+ White Male
2.17%
27.14%
Table A19: The performance of different methods on MIMIC-CRR dataset.
Attr.
Metric
ERM
Subgroup-R
Class-R
Subsroup.class-R
SWAD
Overall
86.26
86.07
86.20
86.33
87.10
Race
Min.
85,52
85.31
85.46
85.56
86.38
Gap
0.85
0.88
0.86
0.95
0.88
Overall
86.45
86.21
86.37
86.24
87.05
Sex
Min.
85,62
85.31
85.45
85.39
86.23
Gap
1.41
1.53
1.60
1.46
1.39
Overall
86.40
85.53
86.02
85.43
87.08
Age
Min.
81.06
70.97
74.69
73.24
82.15
Gap
5.32
6.99
5.87
6.08
5.10
Overall
86.26
85,06
86.25
85.37
86.35
Intersection
Min.
72.43
74.31
73.31
74.18
75.96
Gap
18.92
11.59
16.06
15.45
14.78
37

---

Published as a conference paper at ICLR 2023
F ACKNOWLEDGEMENT TO DATA RESOURCES
For the usage of ADNI dataset:
Data used in preparation of this article were obtained from the Alzheimer's Disease Neuroimag- ing Initiative (ADNI) database (adni.loni.usc.. udd.eu....... As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not par- ticipate in analysis or writing of this report. A complete listing of ADNI investigators can be found at: http://adni loni. usc. wwd//wpcooteeteeeeeeeeeeeemmmn ADNT_ wecckeetteggemeet___.... pdf
hww.waadss/wwwooooooooo htcpaaahhooooooeeeee to to__pply/
The ADNI was launched in 2003 as a public-private partnership, led by Principal Investigator Michael W. Weiner.MD. The primary goal of ADNI has been to test whether serial magnetic reso- nance imaging (MRI), positron emission tomography (PET), other biological markers, and clinical and neuropsychologica assessment can be combined to measure the progression of mild cognitive impairment (MCT) and early Alzheimer's disease (AD). For up-to-date information, see www.adni- info.org.
Data collection and sharing for this project was funded by the Alzheimer's 's Disease Neuroimaging Initiative (ADNI) (National Institutes of Health Grant U0I AG024904) and DOD ADNI (Depart- ment of Defense award number W81XWH- 12-2-0012). ADNI is funded by the National Institute on Aging, the National Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following: Abb Alzheimer's s Association; Alzheimer's Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company: CereSpir, Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company; Eurolmmun; F. Hoffmann--ae Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio; GE Healthcare; IXICO LidssJanssen Alzheimer Immunotherapy Research & Development, LLC.; Johnson & John- son Pharmaceutical Research & Development LLC.;; Lumosity; Lundbeck; Merck Co., Inc.:Meso Scale Diagnostics, LLC.; NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transi- tion Therapeutics. The Canadian Institutes of Health Research is providing funds to support ADNI clinical sites in Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health (www.fnih.org Research and Education, and the study is coordinated by the Alzheimer's 's Therapeutic Research In- stitute the University of Southern California. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of Southern Californa.
The grantee organization is the Northern California Institute for
38


Reviews:
[{"review_id": "esOkB1HMQp", "reviewer": "Reviewer_sgSV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose a framework, named MEDFAIR, to benchmark the fairness of machine learning models for medical imaging. Through extensive experiments, the authors find that the under-studied issue of model selection criterion can have a significant impact on fairness outcomes. The recommendations are made for different medical application scenarios.", "summary_of_the_review": "As an empirical evaluation paper, this paper has conducted a lot of experiments and obtained some valuable conclusions, but the writing needs to be improved.", "strength_and_weaknesses": "The topic of this paper is significant and valuable. But there are many issues.\n\n1. Page 1 (bottom), \"label noise (e.g., Zhang et al. (2022) find that label noise in...\".\n2. Figure 1, why is this figure not mentioned in the text?\n3. Page 2 (bottom), \"(E.g., data collected at hospital A is used to train a model deployed at hospital B). \" I have never seen it written like this before.\n4. Page 2 (bottom) \"Bias widely exists in ERM models... \" and Page 3 (top) \"...the empirical risk minimization (ERM)\nwith statistical significance...\",  why does the latter ERM have the full name and not the preceding one?\n5. Page 3 (middle), \"...e.g., there are data of patients’ age and sex. \", statements are not concise enough.\n6. Page 3 (bottom), \"Our framework can be easily extended to non-binary classification.\" There is no explanation in the full text or even in the supplementary material on how to extend to non-dichotomous classification.\n7. Page 3 (middle), \"We train models for each combination of algorithms × datasets × sensitive attributes.\" What role does the model selection in Section 3.3 play?\n8. Section 3.2 lacks the reasons for choosing these algorithms.\n\n\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K_MCV-ehwP7", "reviewer": "Reviewer_aeZu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces MEDFAIR, a framework to benchmark fairness in machine learning models for medical imaging. It provides a reproducible environment for developing and evaluating bias mitigation algorithms for deep learning applied to medical imaging. The authors use it to evaluate fairness in multiple scenarios and make recommendations for different medical cases.", "summary_of_the_review": "The framework contribution for fairness in medical imaging is relevant and an active concern in the field. Experiments are extensive, detailed, and possibly reproducible. I am overall quite happy with this contribution as it may help other researchers in the field, all code is available and it is possible to add new algorithms, metrics, and datasets into the framework.", "strength_and_weaknesses": "*Strengths\n- The background on fairness research was deeply evaluated, not focusing only on medical papers but also on other fairness work applied to computer vision, which were included in the proposed framework.\n- The proposed framework has the potential to standardize comparison among different debiasing strategies in medical imaging\n- Aside from the contribution of the framework, the experiments and discussion highlight several key points and challenges for fairness awareness. \n- The appendix provides an extensive and explained report of the experiments, metrics, hyperparameters, as well as instructions for adding new algorithms to the proposed framework.\n\n*Weaknesses\n- The proposed framework is still missing several fairness strategies such as the self-supervised ones, as well as datasets from other medical data modalities, and also does not include segmentation, which is an essential task in medical imaging.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FWkXzskNZ36", "reviewer": "Reviewer_zqn8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors focus on benchmarking the fairness issue in the field of medical images, taking different algorithms, datasets and models into consideration. They first choose 9 different datasets, covering the different categories of medical images, and then implement 11 different algorithms that aim to mitigate the bias/fairness issue in machine learning systems, and finally conduct model selection for evaluation. The main contributions and conclusions are:\n1. By extensive experiments, they show that the bias issue widely exist in the area of medical images;\n2. Model selection introduces significant difference in the performance for different subgroups;\n3. Current bias mitigation algorithms fail to address the fairness issue in medical images", "summary_of_the_review": "This paper provides a benchmarking of fairness in medical image systems, with thorough experiments and significant results, I recommend this paper to be accepted.", "strength_and_weaknesses": "The main strength of this paper:\n1. The problem setup is interesting and of significant importance. Fairness issue in medical image systems can be harmful when put into practical use;\n2. The experiments are extensive and well-designed. The authors choose different datasets and algorithms to finish the benchmarking, which makes it more reliable;\n3. The paper is well-written and easy to follow;\n4. The results are significant and the limitations are clarified. In the results shown in the paper, there does exist bias issue in various tasks related to medical images and cannot be ignored.\n\nDoes not have to be weakness, but some concerns are:\n1. For now this paper only considers one sensitive attribute at a time, I believe it's important to also understand the correlation within the sensitive attributes, since in most datasets that's also imbalanced. Therefore some future work may also include a multi-factor analysis, for example adding a regression;\n2. Maybe I am missing it -- what is the diagnose label used in the experiments reported in the main body? Were other label/tasks also considered?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-9-mqd74eB", "reviewer": "Reviewer_q4yd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper performs an analysis of fairness training methods on multiple medical imaging tasks. They conclude that no method is significantly better than basic ERM training.\n", "summary_of_the_review": "The paper is well written and clearly discusses fairness but does not provide significant insight into the fairness issues of medical datasets. For this reason I do not recommend acceptance.\n", "strength_and_weaknesses": "The paper is well written and articulates existing methods of fairness very well. \n\nDetails about the experiments are not clear. Such as how the age was divided into subgroups and how many subgroups exist for each category are not mentioned. Knowing the number of subgroups where the min and the max performance are calculated would help to interpret the significance of the spread in Figure 3.\n\nThe main weakness is that there is not much of an actionable conclusion. The experiments contain runs of models on many datasets without a deep focus on any one of them, not focusing on the specifics of each one. The breakdown of biased variables is very limited to sex and age which doesn't really expose subtle issues. \n\nI find the paper spends more time discussing fairness concepts than studying it on medical data. The conclusions drawn from the experiments on medical data seem limited and not detailed enough to reflect deeply on the specifics of each domain.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%