Demographically-Informed Prediction Discrepancy Index: Early Warnings of Demographic Biases for Unlabeled Populations
Authors: Lucas Mansilla
Estanislao Claucich
Rodrigo Echeveste
Diego H Milone
Enzo Ferrante
Venue: Accepted by TMLR ()

Abstract: An ever-growing body of work has shown that machine learning systems can be systematically biased against certain sub-populations defined by attributes like race or gender. Data imbalance and under-representation of certain populations in the training datasets have been identified as potential causes behind this phenomenon. However, understanding whether data imbalance with respect to a specific demographic group may result in biases for a given task and model class is not simple. An approach to answering this question is to perform controlled experiments, where several models are trained with different imbalance ratios and then their performance is evaluated on the target population. However, in the absence of ground-truth annotations at deployment for an unseen population, most fairness metrics cannot be computed. In this work, we explore an alternative method to study potential bias issues based on the output discrepancy of pools of models trained on different demographic groups. Models within a pool are otherwise identical in terms of architecture, hyper-parameters, and training scheme. Our hypothesis is that the output consistency between models may serve as a proxy to anticipate biases concerning demographic groups. In other words, if models tailored to different demographic groups produce inconsistent predictions, then biases are more prone to appear in the task under analysis. We formulate the Demographically-Informed Prediction Discrepancy Index (DIPDI) and validate our hypothesis in numerical experiments using both synthetic and real-world datasets. Our work sheds light on the relationship between model output discrepancy and demographic biases and provides a means to anticipate potential bias issues in the absence of ground-truth annotations. Indeed, we show how DIPDI could provide early warnings about potential demographic biases when deploying machine learning models on new and unlabeled populations that exhibit demographic shifts.

Published in Transactions on Machine Learning Research (02/2024)
Demographicall--Iformee Prediction Discrepancy Index: Early Warnings of Demographic Biases for Unlabeled Popu- lations
Lucas Mansilla Estanislao Claucich Rodrígo Echeveste
Imonsill@ssinc unl. edu. ar eclaucich unl. edu. recheveste unl. edu.ar dmillone &sinc. unl. edu.ar ar eferrante @sinc. unl. edu. or
Diego H. Milone
Enzo Ferrante
Reseorch Institute for Signals, Systems and Computational Intelligence sine(t) Universidad Nacional del Litoral CONICET, Santa Fe, Argentina
Reviewed on OpenReview: https: openreview. net/ forum? id= d=8W6IDyF2g0
Abstract
An ever-growing body of work has shown that machine learning systems can be system- atically biased against certain sub-populations defined by attributes like race or gender. Data imbalance and under-representation of certain populations in the training datasets have been identified as potential causes behind this phenomenon. However, understanding whether data imbalance with respect specific demographic group may result biases for given task and model class not simple. An approach to answering this question is to perform controlled experiments, where several models trained with different imbal- ance ratios and then their performance is evaluated the target population. However, in the absence of ground-truth annotations at deployment for an unseen population, most fairness metrics cannot be computed. In this work, we explore an alternative method to study potential bias issues based on the output discrepancy of pools models trained on different demographic groups. Models within pool are otherwise identical terms of architecture, hyper-parameters, and training scheme. Our hypothesis is that the output consistency between models may serve as proxy anticipate biases concerning demo- graphic groups. In other words, if models tailored to different demographic groups produce inconsistent predictions, then biases are more prone to appear in the task under analy- Prediction Discrepancy Index (DIPDI)
sis. We formulate the Denoographicall--nnooree and validate our hypothesis in numerical experiments using both synthetic and real-world datasets. Our work sheds light on the relationship between model output discrepancy and demographic biases and provides means to anticipate potential bias issues in the absence of ground-truth annotations. Indeed, show how DIPDI could provide early warnings about potential demographic biases when deploying machine learning models new and unlabeled populations that exhibit demographic shifts.
1 Introduction
Machine learning (ML) models are susceptible to exhibiting biases against certain subpopulations defined in terms of sensitive demographic characteristics such as gender, age, or race. Examples of such biases can be found in variety fields, including predictive policing (Angwin et al. 2016), facial analysis [uuolamwin]
& Gebru 2018), and healtheare (Chen al.) 2019 Ricci Lara al. 2022 Factors that contribute
biased models may include the data used for training and evaluation, as well decisions made during the development process (uurehh & Guttag 2019). As MT. applications in the real world become increasingly

---

Published in Transactions on Machine Learning Research (02/2024)
widespread, it is important to evaluate models to ensure that they are not only accurate but also produce fair and ethical results.
In particular, under-representation of certain demographic groups has been identified as one of the main causes of bias when developing predictive systems. Although many types of biases exist and can be mea- sured using different metrics, here we are mostly concerned about disparities usually studied in the literature of group fairness (e.g. as measured by the gap in accuracy between dif- ferent demographic groups for classification systems, or the gap in mean absolute error per demographic group for regression problems). For example, gender imbalance in X-ray medical imaging datasets has been shown to have significant impact on the performance of assisted diagnosis systems for thoracic discases based on convolutional neural networks &Larrazabal et al., 2020). as measured by the gap in area under the receiver operating curve (AUC-ROC) for male and female individuals. Another example is given by under-representation of ethnic groups, which has also been found to influence model performance for cardiac image segmentation (Puyol-Antón et al., 2021). as measured by the differences in the Dice coefficient be- tween different, groups. However, in other lasks, such data imbalance has not been associated with unequal performance. In Petersen et al. (0022) for example, the authors found that in the case of Alzheimer's disease prediction from brain magnetic resonance imagos (MRT), gender imbalance in the training datasct did not lead to clear pattern of improved model performance for the majority group. A similar phenomenon was observed in Kinyanjui et al. (2020), ; where the authors studied under-representation of skin color when ana- lyzing dermoscopic images for skin cancer detection, and did not observe such disparities. This observation was then challenged by Grrh et al. (2021), ) which found disparities in performance arising from training a neural network on only subset of skin types. In all, it is not always fact that data imbalance will result in biased automated systems. To complicate matters further, even when the presence of biases can be assessed during the development of an automated tool, these properties may not transfer under distribution shifts (Schrouff et al.) 2022), for instance, once the model is deployed. This is a problem for fairness metrics which require ground-truth annotations, which are expensive to obtain and may not be available before deploy- ment. Given these issues, a valid question that one may then ask is: can we anticipate whether models will exhibit. biases with respect to data imbalance in terms of particular protected attribute the absence of ground-truth annotations?
predictive performance
Typical approaches to identify biases in ML models involve subgroup analysis and controlled experiments where both demographic and target labels are available (Tarrazabal et al. 2020 Buolamwini & Gebru 2018: Glocker et al.) 2021). Model performance across demographic groups is commonly evaluated employing one or more motrics (Corbett-Davies & Goel 2018) with the implicit assumption that the presence or absence of biases during development will be representative of the behaviour of these models when applied to previously unseen data at deployment. Recent findings regarding how fairness properties transfer across distribution shifts in real-world healthcare applications due to changes in geographic location or population demographics, warn us about the risks of this assumption (Schrooff et al.] 2022.. A system that did not exhibit strong biases in the source population may begin to do so when the target population changes. This is particularly concerning in applications like healthcare, where collecting expert annotations on large datascts can be costly and lime-consuning PRicci Lara el al., 2022), meaning that fairness metrics requiring labels may not be computed, with the result of biases going unnoticed. In this context, developing methods that can be used without the need for ground truth in the target population becomes highly relevant. In this paper, we are interested in exploring ways to anticipate polential bias issues that may arise in the context of given task for novel unlabeled target population. We do by proxy: using an index that we call Demoogapticalll-Inoooee
Prediction Discrepancy Index (DIPDI), which can be computed in the absence of ground truth annotations. We provide an analytical derivation demonstrating the relation between DIPDI and performance gaps, and show in numerical experiments using both synthetic and real-world datasets that this index is indeed indicative of bias proneness, providing an early warning for potential fairness issues in these settings.
2
Related work
The implicit assumption that model assessment during development is representative of its behaviour at deployment is not unique to fairness studies. Indeed, anticipating whether model will systematically fail
2

---

Published in Transactions on Machine Learning Research (02/2024)
not when ground-truth annotations are not available is current topic of interest in the field, and one way to tackle this issue is to look at predictive uncertainty (Gal et al. 2016). Intuitively, if a well-calibrated model systematically makes highly uncertain predictions for certain individuals, then chances are that these predictions will have a higher failure rate for those individuals. In this context, recent studies have analyzed the relation bet ween fairness and uncertainty, postulating that uncertainty estimates can be used to obtain fairer models, improve docision-making, and build trust in automated systems (Bhatt et al. (021]) For example, Lu et al. (2021) analyzed how alternative uncertainty estimation methods can be used to evaluate subgroup disparities in mammography image analysis, while Stone et al. (2022) leveraged epistemic uncertainty estimates to mitigate minority group biases during training. The work of DDusenberyy et al. (2020) discusses the role of model uncertainty in predictive models for Electronic Health Record (EHR), and shows how it can change across different patient subgroups, in terms of ethnicity, gender and age, considering Bayesian and deep ensemble approaches for uncertainty estimation. Even though in this work we do not directly rely on the notion of uncertainty, our study is highly influenced by this idca, as it explores the use of output discrepancy for set of models as a way of anticipating bias issues. This notion is closely related to measure of uncertainty for ensemble methods (Lakshminarayanai) 2016 Larrazabal et al. 2021). Another important concept in our study is that et al. 2017 Pividori et al.
ensemble variance, usually employed
of consistency (Wang et al. 2020,, defined as the ability of a set of multiple trained learners to reproduce an output for the same input. According to this concept, model outputs are analyzed irrespective of whether they are correct or incorrect, and as such, it does not require ground-truth annotations to be computed. This idea will be central to our study, as we explore how changes in consistency for pools of models trained on the same or different demographic groups will correlate with potential biases that may emerge in given task.
Contributions: Here we present a methodology to understand whether biases with respect to given demographic attribute are prone to arise in new unlabeled dataset. We do so by analyzing the output consistency of a pool of models, where each model is trained on separate demographic groups, but is otherwise identical in terms of architecture, based on the following hypothesis: if models specialized in different. demographic groups produce discrepant predictions for the same test data, then the task under analysis is prone to be biased against that demographic attribute. Note that throughout this manuscript, we consider that a task is prone to be biased with respect to given demographic altribute when we observe systematic performance gaps for models trained on different demographic groups characterized by such attribute.
and training scheme. We introduce a new index, DIPDI,
We validate our hypothesis using synthetic and real-world datasets, focusing on regression and classification tasks: age regression from face photos and X-ray images, classification of younger vs older celebrities in face images, as well as hair color classification. We use four real-world datasets and consider different cases of demographic imbalance in the training data. Our results indicate that DIPDI can be used to anticipate potential bias issues in the absence of ground truth labels, and confirm the association between output discrepancy and bias proneness. We also assess the behaviour of DIPDI for unseen populations with different types of distribution shifts, showing how it can be used to mcasure bias proneness in dynamic contexts. Moreover, since our metric does not require expert annotations to be computed, it could help to anticipate bias issues in real-world scenarios and give early warnings when deploying machine learning models on new, unlabeled populations.
3 Demographically-Iformed Prediction Discrepancy Index (DIPDI)
3.1 Quantifying output discrepancy within and between demoggapphiaallyynnfrmeed sets of models
Given two sets of predictive models A = [A1. A2) and B = (B1, B2), we are interested in analyzing how the output discrepancy of models within the same set compares to the output discrepancy of models coming from different sets, when they are evaluated on samples from an unlabeled dataset D. Here A(xx) : X - Y is a predictive model (e.g. a regression or classification model), where X2 E ID S X can be images or other types of data for subject k, and the output of A(xx) is a label y Y which could be real number for regression problems as well categorical label or soft probability estimate for classification problems.
3

---

Published in Transactions on Machine Learning Research (02/2024)
We then define average output discrepancy function (M1 M2), that takes as imput two models M1 and M2. and returns a number representing how different their outputs are on average when evaluated on all samples from D. We measure the discrepancy between two models using a discrepancy function d(-y)) so that the average output discrepancy is defined as
(xM,(xx), Nb, M2) = DDI
Σ XFED
(1)
For example, in regression problems the discrepancy function d( could be the absolute the quadratic error, while in classification problems it could be the Jensen-Shannon divergence between the output dis- tributions for models Mr.My In other words, the average output discrepancy is the mean discrepancy d(M1, M2) between the predicted values of models M1 and M2 for all subjects in the dataset. Tt returns a number closer to 0 when the outputs of the two models for every data sample are similar, and higher if they tend to differ. Since we are interested in analyzing the output discrepancy for models within and between sets, we consider the following ratio an indicator of relative output discrepancy:
P (A, B) log [N B.), B2) Na A2)Nn B2)
(2)
This inter-model prediction discrepancy will be close to when the output discrepancy for models within the same set (numerator) is similar to that of models coming from different sets (denominator), and it will be greater than 0 when the discrepancy for models coming from different sets is greater than that of models coming from the same set. When applied models trained on different demographic groups, we refer to this diverse set of models a deenographically--nooneee Prediction Discrepancy Index (DIPDI). This will be the case, for example, when models in A are trained on male individuals while models in B are trained on female individuals.
pool and Φp becomes our Demogruphically Infoomed
Note that in our current analysis, we have focused on model sets of size for simplicity (e.g. both and IE have two elements). However, is important to note that this concept can be extended to larger sets. The generalization involves considering combinations of pairs of models both within each set and between different sets. For extension of DIPDI to handle groups with more than two models, please refer to Appendix A.1
3.2
DIPDI as a proxy for anticipating bias issues
Our goal is to anticipate whether biases may arisc with respect to a particular protected attributc a in a novel dataset before annotated labels become available. Here we provide an example where the task al hand age regression and the protected attribute a indicates the gender of the individual, which for simplicity we take male (a M) or female (a F). We create two sets of models (age regressors): A, where models A, are trained only on male individuals, i.e. M; and B, where models B, are trained only on female individuals, i.e. a F. We say that this constitutes demographiealyy-nnforo trained on individuals from a particular demographic group characterized by the protected attribute a. Let us also have a fixed dataset that will be used as the novel target population where potential biases would want to be flagged. DD is a balanced dataset according to the protected attribute a (but unlabeled with respect output class, ic. without the reference age). In our example, this mcans that ID composed of 50% male and 50% female individuals.
pool of models, as each of them was
Our hypothesis is that for larger values of (A, BB), computed for pool of models comprising sets A and B. biases are more likely to emerge. In other words, we hypothesize that inconsistencies between the output discrepancy of models trained on highly imbalanced datasets with respect to the protected altribute a will tend to co-occur with potential bias issues. To confirm our hypothesis, we first look for biases with respect to a using ground-truth annotations in the target population (following a strategy similar to Larrazaba] lt al. (2020)), by computing performance gaps in terms of absolute error (using the ground-truth of each sub-population). Then we caleulate DIPDI, which does not require ground-truth labels for D, and verify if
4

---

Published in Transactions on Machine Learning Research (02/2024)
it produces results that are in line with the conclusions we drew when using the annotations. As sanity check, we incorporate control experiments where we break the assumption that sets A and IB are trained on different demographic groups (e.g. by training models in A and B using balanced data formed by 50% male and 50% female individuals), and show that in these cases DIPDI returns values close to 0. Moreover, the theoretical derivation included Appendix A.2 provides analytic expressions for the relationship bet ween performance gap and output discrepancies captured by DIPDI, assuming one-dimensional Gaussian soft outputs for classification models.
4 Experimental validation
We start by verifying the behaviour of DIPDI under controlled conditions using synthotic data (Section Then, we perform a set of experiments to evaluate the proneness to gender bias of tasks such as age estimation (from face and X-ray images) and younger vs. older classification of celebrities (from face images). Our focus is particularly in Larrazabal et al. (2020). We show that DIPDI anticipates potential biases against the minority group when training data is highly imbalanced in gender representation (Section 4.3 To this end, we employ ground-truth annotations for the target population to compute performance gaps for models trained with different imbalance ratios, in the different subgroups. Then, we proceed to compute DIPDI (which does not require ground-truth labels) in the target population. We show that bias gaps tend to occur for larger DIPDI values (Section 4.4p. We conclude the study by showing how DIPDI serve to anticipate potential bias issues at deployment in populations with distribution shifts, when target annotations are not yet available (Section 4.5
4.1
scenarios where specific subgroup is underrepresented,
discussed
4.1 DIPDI on synthetic data
We start by verifying the behaviour of the proposed DIPDI using synthetic data. The purpose is to show, using a simple example, how sensitive DIPDI when measuring bias proneness. To this end, we generate synthetic dataset composed of samples s (x,,)), where x E R² are bi-dimensional feature vectors coming from two bi-variate normal distributions Na=0(11o (0,0), Eg = I) and Na=1 = (1, 1), En T). These distributions simulate different demographic groups characterized by a protected attribute female) or = 0 (e.g. male). We then generale the corresponding labels Y/ E IR for each data sample to simulate a regression problem (e.g. age regression). We do this by assigning yi to be equal to the first feature dimension of the sample, lincarly interpolated to ensure it is within and a 100 years, with an added uniform random noise of 10 years, simulating an age estimation scenario (see Figure 7 in the Appendix B.I. We then train support vector regression (SVR) models on these samples, perform inference, and subsequently compute DIPDI based on the actual predictions produced by these models.
(e.g.
Importantly, to ensure that redee-eeprestnttttoo of a certain group will bias age regression models to exhibit better performance in the majority group, we sample each distribution by varying the proportion of male and female samples in training. The range of gender imbalance cases spans from 100-0 (100% male) to 0-100 (100% female), with increments 10%. We generated 10,000 training samples that were partitioned into 10 folds for statistical purposes, and an additional 1,000 samples for testing that were held constant across all experiments.
The results of the synthetic experiment are presented in Figure At the top (Figure la the mean absolute error (MAE) is shown for models trained with different imbalance ratios evaluated males, females, and the whole population. We can see performance disparitics across demographic groups, which highlights the impact of gender inbalance on predicted accuracy. The corresponding DIPDI values are shown in Figure [b] The observed results are consistent with our hypothesis that DIPDI helps to anticipate bias proneness. In Appendix [B22 we include results for an even simpler synthetic experiment, where instcad of simulating the dataset and training SVR models, we directly simulate the model outputs and show how DIPDI reacts in highly controlled conditions.
5

---

Published in Transactions on Machine Learning Research (02/2024)
Testing on Female
35
All
MAM
25
15-
10
100.0
90-10
20
70-30
60
50-50
40-60
30-70
20
10:90
0-100
Training Distribution (Male-Female)
(a)
12
10
6-
DIPDI
0-
-2-
100-0
A: 90-10
A:80-20 8::00-00
70-30 60-40
50-50
A: 40-60
A:30-77
A:20-80
A:10-90
0-100
0-100
50-50
60-40
8:90-10
100-0
Groups and Training Distribution (Male-Female)
Figure (a) Mean absolute error (MAE) of SVR models trained on synthetic datascts of age estimation with varying gender inbalance ratios. (b) DIPDI values for different group compositions, highlighting the impact of gender imbalance in prediction consistency, Statistical significance with respect to mean equal to 0 was measured by Wilcoxon test (ns: non-significant, *; p-value < 0.05, **, p-value < 0.01, ***, p-value < 0.001).

---

Published in Transactions on Machine Learning Research (02/2024)
4.2 DIPDI real scenarios: datasets and experimental setup
We conduct experiments on the task of age regression using convolutional neural networks (CNNs), employing
three public databases: ChestX-ray14 (Wang et al., 2017 UTKFace (Zhang et al. 2017 and IMDB-WIKI Rotth et al. 2018). We also evaluate DIPDI for classification problems, considering a binary classification (younger vs older celebrities) using the CelebA dataset (Liu et al., 2015). All experiments were performed using PyTTrrch (Paszke et al. 2017 and executed on an NVIDIA Titan X G
ChestX-ray14 dataset. The ChestX-ray14 dataset contains 112,120 high-resolution frontal-view radio- graphs of 30,805 unique patients with age and gender labels. Each image is annotated with up to 14 different chest disease labels extracted from radiology reports (not used in our study), age and gender. We use the ChestX-rayl4 dataset lo perform subgroup analysis in terms of gender and to evaluate DIPDI in models trained to perform age estimation from radiological images. We use here for gender the binary labels reported in the dataset, i.c. male and female. To avoid having two images of the same patient in train and test, we randomly selected one image per patient, resulting in a total of 28,350 images which include healthy and pathological cases. This database was divided into 10 folds using stratified cross-valideee validation strategy, where each fold is balanced by gender. For each cross validation instance, one fold is used to evaluate the model and the remaining 9 folds are used to train the model, which are further sub-divided into training (90%) and validation (10%) subsets for hyper-parameler luning and model selection.
For all experiments on ChestX-ray14 we used a DenseNet-121 (uuag et al. 2017) pretrained on ImageNet (Russakovsky et al., 2015). The last layer of the network was replaced with an adaptive pooling layer, followed by a single-output, neuron layer to predict age. The models were trained for 50 epochs using the Adam optimizer (Kingma & Ba. 2014) with default parameters and the mean absolute error (MAE) loss function.
FaKe dataset. The UTKFace dataset is collection of over 20,000 facial images spanning ages from 0 to 116, annotated for age, gender, and ethnicity. It cxhibits diverse variations in pose, facial expression, lighting, occlusion, and resolution. Images were filtered to include ages from 10 to 100 and followed the same training settings as applied in the case of ChestX-ray14. This dataset is utilized for subgroup analysis and assessing DIPDI in age estimation models.
We employed a VGG-16 architecture (Simonyan & Zisserman, 2014 pretrained on ImageNet, with the final layer replaced by adaptive pooling and single-output neuron layer for age prediction.
IMDB-WIKI dataset. The IMDB-WIKI dataset consists of 523,051 face images of 20,284 celebrities collected from IMDB and Wikipedia with age and gender labels. Age is estimated from the date of birth and the year when the photo was taken. The IMBD-WIKT dataset is used to perform subgroup analysis and to evaluate DIPDI for models trained to perform age estimation from facial images.
We used a VGG-19 architecture (Simonyan & Zisserman 2014] pre-trained on ImageNet. We added single- output neuron layer with ReLU activation and finc-tuned the last four layers. The models were trained with MAE loss for 10 epochs using the Adam optimizer with default parameters.
CelebA dataset. The CelebFaces Attributes (CelebA) dataset (Liu et al. 2015) is a large-scale repository comprising over 200,000 celebrity images, cach annotated with 40 attributes. This dataset covers diverse facial poses and background variations. In our study, we choose the 'young' attribute as the target label (thus classifying younger vs older individuals) for prediction, and balance both gender and target to mitigate potential spurious correlations.
To conduct subgroup analysis and evaluate DIPDI in classification models, we employed ResNet-50 archi- tecture (He et al.] 2016) pretrained on ImageNet, replacing the final layer with a two-output neuron layer for binary classification. The training process followed the same protocols as ChestX-ray and UTKFace, employing the cross-entropy loss.
'Qur code publicly available
7

---

Published in Transactions on Machine Learning Research (02/2024)
4.3 Assessing the impact of gender imbalance in supervised setting
Age estimation from X-ray images. We analyze the impact of gender imbalance in age estimation from radiological images by performing a supervised subgroup evaluation. The aim is to understand the age estimation task is prone to be biased with respect to gender a certain subgroup is under-represented. We will then see if the proposed DIPDI can predict such behaviour without ground-trute annotations. We train models with different degrees of gender imbalance and then examine their performance separately in male and female subgroups. We consider five cases of gender imbalance in training: 100-0, 75-25, 50-50, 25-75, and 0-100. Importantly, male and female subgroups in the test population are always equal in size. This means that every model is evaluated on equal footing.
The MAE for ChestX-ray14 is shown in Figure 2a The results show that imbalance with respect to the protected attribute leads to a significant difference in performance across subgroups, confirming that this problem is prone to be biased with respect to gender if there is under-repreeentation in the training dataset. For example, when testing on female subjects, models trained only on male (100-0) data have higher MAE than models trained on female images. The same happens when testing on female individuals: models trained only on female data (0-100) significantly outperform those trained on male data. Moreover, the differences between male and female subgroups are less significant when the training data is less imbalanced. These results are consistent with previous observations reported by Larrazabal et al. (2020) in the context of disease prediction from X-ray images. Appendix B.3 contains supplementary results for ChestX-ray14 including additional statistics and metries.
Age estimation from face images. We also perform similar analysis to study the impact of gender imbalance in age estimation from facial images. For UTKFace, we followed the same procedure as for ChestX- ray14. Figure 2c shows the MAE results for models trained with varying degrees of gender imbalance and tested on male and female subgroups over 10 folds. These results demonstrate a significant performance disparity across subgroups resulting from an imbalance in the protected attribute.
In the case of WIKI, we train an ensemble of 5 models with different degrees of malc-fomale imbalance and then evaluate their performance separately in male and female subgroups. We perform 20-fold cross- validation with a 60/20/20 ratio for the training, validation, and test sets.
The MAE is shown in Figure 2e for these experiments. We observe that the models perform better in the subgroup (either male or female) that is most represented in training, while their performance deteriorates in the other subgroup.
Younger vs older classification from face images. We explore the impact of gender imbalance when classifying celebrity photos using the CelebA dataset as either younger vs older individuals. Figure presentss the accuracy results by subgroup over 10 folds for models trained with varying gender imbalance ratios. The results reveal large gaps in performance for different demographic subgroups in highly imbalanced cases, while the gaps reduce for more balanced cases. We include supplementary results for CelebA in Appendix B.4 which include confusion matrices and a fairness metric (Equality of Opportunity).
4.4 Estimating bias proneness without ground-truth via DIPDI
In the previous section we confirmed that age estimation and classification of younger vs older individuals are tasks prone to be biased with respect gender, by computing error gaps between subgroups using ground-truth annotations. Now, we want to study if it is possible to measure such bias proneness in a given population without ground-truth labels using DIPDI. We are interested in analyzing if output discrepancy for demography-aware model sets can be used as proxy for anticipating potential fairness problems in specific ML lasks. To this end, we compute DIPDI in the same settings where we explicitly evaluated biases in the previous section. Note that we choose different discrepancy functions d(..)) (from Equation [ for regression and classification problems. For regression problems, we choose to be the absolute error between the model predictions. For classification, as we output soft probabilities, we adopt the Jensen-Shannon divergence between the output distributions.
8

---

Published in Transactions on Machine Learning Research (02/2024)
ChestX ray14
ChestX-ray14
10-
6.5
Testing
Maie
6.0
f5
4.5
A: 100-0
75-25
A:50-50
2:25-75
0-100
100.0
75-25
50.50
25-75
0.100
8: 0:100
B: 25-25-5
B-50-50
8:75:25
B 10000
Troining Distribution (Mall-Fommale)
Groups end Treining Distribution (Male-Female))
(a)
(b)
UTKFace
Testing
14-
Male Female
12-
MMF
02-
0.
25-75
0-100
100-0
75-25
50-50
25-75
0-100
Training Distribution (Male-Fomale)
Groups and Training Distribution (Male-Fermal
(c)
(d)
IM DB-WIKI
IMDR-WIKI
Testing on Female
9.0
8.5
1.00
0.75
DWVDI
0.50
0.25
7.0
.0.50
A: 100-0
A: 75-25
A:50-50
A: 25-75
0-100
100
75-25
50:50
25-75
0:100
25-75
50550
Training Distribution (Male-Famale)
Groups and Training Distribution (Male-feralel
(e)
(f) CelebA
0.90
2.0
0.85
1.-
0.80
Accurary
1.0
birn
0.75
0.-
0.70
Testing on Male
0.-
0.60
100.0
75-252
A-50-50
A:25-75
0-100
100.0
75-25
50-50
25-75
0-100
8: 0:100
8: 85255-75
B: 50:30
8:75-25
B 100.0
Training Distribution (Mole-Female)
Groups end Treining Distribution (Male-Female)
(g)
(h)
Figure Results for age estimation (rows 1-3) and younger vs older classification (row 4) across models trained with different gender imbalance ratios. Model performance, measured through MAE and Accuracy, shows wider gaps in highly imbalanced cases. It effectively captured by DIPDI with higher values for imbalance, and close-to-zero values for balanced cases,

---

Published in Transactions on Machine Learning Research (02/2024)
In all datasets, we consider different scenarios of gender imbalance for the set of models A and B to be evaluated. Five comparisons are made: 100-0 vs 0-100, 75-25 vs 25-75, 50-50 vs 50-50, 25-75 vs 75-25, and 0-100 vs 100-0. To control for finite-size sampling variability, we split the training data into four random disjoint partitions, so that no data is shared between models even when they are trained for the same demographic sub-group (i.e. even though A₁ and A2 are trained on subjects from the same demographic group; the exact individuals are not the same). Then DIPDI is computed on the held-out test set (i.c. the unlabeled data D), which is balanced by gender. Additional results for DIPDI are included in Appendix [.5
The plots in the right column of S DIPDI for the age regression and younger older classification tasks. Note that, in all scenarios, the index values are very close to 0 when comparing sets of models trained in the same population, but higher than O when comparing models from different populations, in line with the absence or presence of biases as a function of data imbalance shown in the corresponding left column. Taken together these results demonstrate the co-occurrence between higher DIPDI and bias proneness: models coming from the same demographic population, produce more consistent outputs when evaluated on larget population. This output stability is clearly evidenced by index values close to 0 for 50-50 distributions. In contrast, the index returns significantly higher values when it comes to models trained with different demographic subgroups, where biases are in turn prone to appear, as shown in our previous supervised analysis (Section 4.3 Importantly, note that no labels were required in the target population D when computing DIPDI.
4.5 Anticipating potential demographic biases in distribution shift scenarios with DIPDI
We have highlighted in the previous section the role of DIPDI in identifying potential demographic biases in populations that lack ground-truth annotations. Now, we turn our attention to a new challenge: demon- strating how DIPDI can deal with domain shift scenarios even when ground-truth data is unavailable. Prior research has identified the vulnerability of fairness properties of machine learning models when deployed on datasets differing from those used during model development (Schrouff et al. 2022). In this context, we leverage DIPDI as an unsupervised alternative to traditional fairness metrics for understanding bias prone- ness in populations under different types of distribution shifts. Here we focus on two cases: covariate shift, where the conditional distribution of the input features (e.g. pixel intensities) changes between source and target population; and label shift, where the conditional distribution of labels change between source and larget (e.g. different prevalence for disease, or different age distributions in our age regression problem). As discussed in Schrouff et al. (2022), such shifts as well as other types of changes in data distribution, may result in failures of fairness transfer across distribution shifts. In other words, models that were not biased in a source distribution may start to exhibit biases in the target distribution. To this end, we explore two different scenarios.
4.5.1 Label shift: age distribution experiment
This experiment involves a target population that is always balanced by gender, and we introduce label shifts by altering the age distribution within one gender group (either male or female, but not both). Specifically, we increase the proportion of individuals with ages exceeding a predefined limit (set at 45 in our experiments). while maintaining the age distribution within the non-shifted group. For each shift scenario considered, we calculate both the DIPDI and the MAE gap (AMAE) between male and female models, when tested separately on male and female subsets. For the male subset, we calculate the AMAE by subtracting the MAE of femalc-trained model from that of a malc-trained model. Similarly, for the femalc subsct, we subtract the MAE of male-trained model from that of a tranne model. Note that the caleulation of AMAE requires access to ground truth annotations, whereas DIPDI does not.
Figure g presents the mean and standard deviation of DIPDI and AMAE for age shift ratios ranging from 50% to 90% (a shift ratio of 90%, for example, implies 90% of the subpopulation is under 45, and 10% is at or above 45). Our aim here is to understand such label shift results in a task that is more prone to be biased with respect to the demographic groups under analysis. In principle, there could be three possibilities when compared with the original distribution: the problem is more, equally or less prone to be biased. Note that when the shift, affects the male group (Fig. Baj, DIPDI tends to slightly increase, and corresponding
10

---

Published in Transactions on Machine Learning Research (02/2024)
slightly increasing gap is observed for both male and female test groups. On the other hand, when varying the proportion of females younger than 45 years old in the unseen population, we observe that the AMAF between models trained on male and female individuals stays constant for males (red curve in Fig, 3b) right panel), but decreases for fomale subjects (blue curve), reaching a lovel of bias proneness equivalent to the one observed for males (at 90%, where the blue and red curves intersect). In other words, decreasing the age of the female population changes the bias proneness of that group (as mcasured by analyzing the AMAEs), making it more fair as it reaches levels of bias proneness equivalent in both populations. As expected, the DIPDI index follows exactly the same tendency (is reduced as bias proneness is reduced), confirming hypothesis. This different behavior observed when introducing label shifts in the test male and female groups may be rooted in the different ways in which the features of each group interact with age. In fact, for this dataset it is well known that the baseline performance is different for both groups. This could explain the greater susceptibility of a group to changes in age distribution.
These experiments underscore the effectiveness of DIPDI in a label shift scenario, particularly when ground- truth annotations are unavailable. An increase in DIPDI during deployment, compared to the development phase, can be interpreted as an indicator of intensifying bias proneness within one both demographic groups, whereas a decrease in DIPDI implics a potential reduction in bias within these groups.
4.5.2 Covariate shift: color distribution experiment
In this experiment, we evaluate a classification model trained for a new task: distinguishing between blond and non-blond celebrities in the CelebA dataset. While the model is trained on the original color images, we induce a covariate shift in the target population by transforming color test images (RGB) to grayscale. We guarantee larget and gender balance in both training and test splits to avoid spurious correlations in our experiment, executing 10 runs with different random seeds.
Figure g presents DIPDI values and the accuracy gap (AAcc) between models trained on male and female subjects, lested on the whole larget population with covariate shift (grayscale images) and without covariate shift (RGB images). We note that evaluating bias proneness of the original models in the shifted distribution using the ground-truth annotations shows decrcased bias overall. Notably, the AAce between models trained on male and female subjects becomes more similar for both groups for grayscale images compared RGB, The DIPDI index consistently captures this behavior, resulting in decrease when computed in grayscale images.
5 Discussion
In this work, we tackle the issuc of anticipating potential demographic biases at deployment in the absence of ground truth annotations. Typical methods designed to assess fairness require access to such annotations, which may be available at training time but not when deploying models for previously unseen data. A prototypical example of this would be model trained on public dataset which will then be applied to local population for which we do not have the corresponding annotations. Recent work has highlighted how inntiibution shifts may affect fairness (Schrouff et al. 2022), resulting in potential risk. While an explicit fairness metric may not be computed, we argue that we can employ the discrepancy of output predictions from pools of models trained on different demographic groups as a proxy to provide an carly warning about potential demographic biases. We propose a concrete solution in terms of an index, DIPDI, whose value indeed provides a mcasure for the proneness towards biased solutions.
Intuitively, we can think about output discrepancies in a set of models as notion of uncertainty, which in ensemble models is usually estimated the variance in the predictions of components of the ensemble [Lakshminarayanan et al., 2017). If all models in an ensemble agree on prediction, the uncertainty for this sample is likely low. Conversely, if there is a high variance in the predictions across the models, this indicates higher uncertainty. When we evaluate the discrepancy in the predictions of models trained for a particular demographic group, we could interpret them as models within an ensemble, and conscquently the discrepancy in their predictions for given subject could be seen as the uncertainty of the ensemble. In that sense, our index quantifies the relative uncertainty estimated when using 'ensembles' of models trained with
11

---

Published in Transactions on Machine Learning Research (02/2024)
UTKFace: Shift Test Data (only Males)
Testing on
1.2
1.0
DIPDI
MAE
3-
0.6
2-
0.4
1-
0.0
80
Age=045 (%)
Age=045 (%)
UTKFace: Shift on Test Data (only Females)
1.4
Testing on
Male
1.2
Female
DIPDI
0.8
1-
0.2
50
60
70
80
90
50
60
70
80
90
Agex445 (%)
Ages<45 (%)
(b)
Figure 3: Mcan and standard deviation DIPDI (left) and MAE gap (right) under label shift sconarios for male (a) and female (b) groups separately on UTKFace. Label shift is induced by modilying the age distribution of individuals under 45 years at increasing ratios for male and female test groups independently. Note that in both cases of shift, the DIPDI tends to follow the behaviour of the difference curve, showing an increase with increasing biases and decrease with decreasing biases.
12

---

Published in Transactions on Machine Learning Research (02/2024)
CelebA; Shift on Test Data
Testing
0.14
Male
0.10
DIPDI
0.06
-1.0
0.02
Grayscale
Grayscale
Test Data
Test Data
Figure 4: DIPDI values (left) and accuracy gap (right) corresponding to covariate shift (RGB to grayscale) in CelebA for blond classification. Note that accuracy gaps become more similar images transform from RGB to grayscale, supported by a decrease in DIPDI.
data from different demographic groups (numerator) and from the same demographic group (denominator). If both are similar (ratio equal to 1), then we get DIPDI value close to (log ratio 1) indicaling that the problem shows no early signs of potential bias with respect to the analyzed demographic values. However, higher discropancics (uncertainty) for models from different demographic groups will lead to DIPDI values significantly larger than 0, indicating bias proneness for the task under analysis. In particular, an increase in DIPDI from model development (training) to deployment could be interpreted as red flag, triggering further detailed assessment. We showed that DIPDI can also be used to understand how fairness transfers across distributions, validating assumption in scenarios involving label and covariate distribution shifts. Other types of distribution shifts (Quinonero-Cande Candela et al. 2008), and even compound shifts discussed in Schrouff et al. 2022)) could also be considered, but would require further validation.
We note that while we have expressed DIPDI here as a global population average, the same reasoning could in principle be applied to population subsets defined by the intersection of multiple demographic traits (i.e. intersectional fairness), or even on subject-by-subeer basis, closer to the definition of individual fairness. Such predictive discrepancies captured by DIPDI could serve to flag subjects or sub-groups al higher risk of suffering biases, constituting another avenue of research to explore in future work. Regarding counterfactual fairness approaches, DIPDI shares some resemblance it involves a form of hypothetical scenario analysis. However, DIPDI's approach is more empirical, focusing on the discrepancies in predictions of actual models trained on different populations, while counterfactual fairness measures often involve more complex causal modeling and assumptions. Finally, relation to approaches based on fairness through unawareness that simply exclude protected attributes from the model, DIPDI actively measures the impact of these attributes by training separate models different demographie groups. Overall, we believe that DIPDI offers fresh perspective in the fairness literature, focusing on the unsupervised setting, which is not commonly discussed in this field, and may spark new discussions towards developing novel unsupervised bias discovery methods to anticipate bias issues in the absence of ground truth.
Acknowledgments
This work was supported by Argentina's National Scientific and Technical Rescarch Council (CONICET), which covered the salaries of R.E., D.H.M. and E.F., as well as the fellowships of L.M. and E.C. The authors gratefully acknowledge NVIDIA Corporation for provinding GPU computing, the support of Universidad Nacional del Litoral (Grants MATD-PPCC-02221110000000
90620190100145LII, Agencia Nacional de Promo- ción de la Investigación, el Desarrollo Tecnológico y Innovación (Grants PICT 2018-3907, PRH 2017-0003, PRH 2022-00002) and the Google Award for Inclusion Research (AIR) Program.
13

---

Published in Transactions on Machine Learning Research (02/2024)
References
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. In Ethics of Dota and Analytics, pp. 254-264. Auerbach Publications, 2016.
Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q Vera Liao, prasanna Sattigeri, Riccardo Fogliato, Gabrielle Melangon, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, et al. Uncertainty as form of trans- parency: Measuring, communicating, and using uncertainty. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 401-413, 2021.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pp. 77-91. PMLR, 2018.
Irene Y Chen, Peter Sxolovits, and Marzyeh Ghassemi. Can ai help reduce disparities in general medical and mental health care? AMA journal of ethics, 21(2):167-179, 2019.
Sam Corbett-Davies and Sharad Goel. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv preprint ar arXiv:1808 180:080000003, 2018.
Michael W Dusenberry, Dustin Tran, Edward Choi, Jonas Kemp, Jeremy Nixon, Ghassen Jerfel, Katherine Heller, and Andrew M Dai. Analyzing the role of model uncertainty for electronic health records. In Proceedings of the ACM Conference on Health, Inference, and Learning, pp. 204 213, 2020.
Yarin Gal et al. Uncertainty in deep learning. 2016.
Ben Glocker, Charles Jones, Melanie Bernhardt, and Stefan Winzeck. Algorithmic encoding of protected characteristics in image-based models for discasc detection. arXiv preprint arxix:1100 10. 1475,, 2021.
Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri. Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 1820-1828, 2021.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, 770-778, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional networks. In Proccedings of the IEEE conference on computer vision and pattern recognition, pp. 4700 4708, 2017.
Diederik P Kingma and Jimmy Ba. ar 6980, 2014.
Adam: A method for stochastic optimization. arXio preprinl
Newton M Kinyanjui, Timothy Odonga, Celia Cintas, Noel CF Codella, Rameswar Panda, Prasanna Sat- tigeri, and Kush R Varshney. Fairness of classifiers across skin tones in dermatology. In Medical Image Computing and Computer Assisted Intervention-MICC Peru, October 4 2020, Proceedings, Part VI, pp. 320 329. Springer, 2020.
2020: 23rd International Conference, Lima,
Balaji Lakshninarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Agostina Larrazabal, Nicolás Nicto, Victoria Peterson, Diego H Milone, and Enzo Ferrante. Gender imbalance in medical imaging datasets produces biased classifiers for computer-aie aided diagnosis. Proceedings of the National Academy of Sciences, 125(22) 2020.
14

---

Published in Transactions on Machine Learning Research (02/2024)
Agostina J Larrazabal, César Martinez, Jose Dolz, and Enzo Ferrante. Orthogonal ensemble networks for biomedical image segmentation. In Medical Image Computing and Computer Assisted Intervenlion- MICCAI 2021: 24th International Conference, Strasboury, Prance, September 27- October 1, 2021, Pro- ceedings, Part III 24, pp. 594-603. Springer, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pp. 3730-3738, 2015.
Charles Lu, Andreanne Lemay, Katharina Hoebel, and Jayashree Kalpathy- Salpathy-Cramer. Cramer. Evaluating subgroup disparity using epistemic uncertainty mammography. arXiv preprint
2021.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Eike Petersen, Aasa Feragen, Maria Luise da Costa Zemsch, Anders Henriksen, Oskar Eiler Wiese Chris- tensen, Melanie Ganz, and Alzheimer's Disease Neuroimaging Initiative. Feature robustness and sex differences in medical imaging: A case study in mri-based alzheimer's disease detection. In Medical Image Computing and Computer Assisted Interwention-MICC September 18 22, 2022, Proceedings, Part I, pp. 88 98. Springer, 2022.
2022: 25th International Conference, Singapore,
M. Pividori, G. Stegmayer, and D.H. Milone. Diversity control for improving the analysis of consensus clustering, Information Sciences, 361:120 134, 2016.
Esther Putól-Antón, Bram Ruijsink, Stefan K Piechnik, Stefan Neubauer, Steffen E Petersen, Reza Razavi, and Andrew P King. Fairness in cardiac mr image analysis: an investigation bias due to data imbalance deep learning based segmentation. In International Conference on Medical Image Computing and Computer-Assistar Intervention, pp. 413 423. Springer, 2021.
J. Quinoncro-Candlla, M. Sugiyama, A Schwaighofer, and N Lawrence. Dataset shift in machine learning. In The MIT Press, 2008.
María Agustina Ricei Lara, Rodrigo Echeveste, and Enzo Ferrante. Addressing fairness in artificial intelli- gence for medical imaging, nature communications, 13(1):1-6, 2022.
Rasmus Rothe, Radu Timofte, and Luc Van Gool. Deep expectation of real and apparent age from a single image without facial landmarks. International Journal of Computer Vision, 126(2):1444 157, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 252, 2015.
Jessica Schrouff, Natalie Harris, Oluwasanmi Koyejo, Ibrahim Alabdulmohsin, Eva Schnider, Krista Opsahl- Ong, Alex Brown, Subhrajit Roy, Diana Mincu, Christina Chen, et al. Maintaining fairness across distri- bution shift: do we have viable solutions for real-world applications? arXiv preprint arXiv:2202.0103), 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image rocognition. arXiv preprint Xio: 1556, 2014.
Rebecca S Stone, Nishant Ravikumar, Andrew J Bulpitt, and David C Hogg. Epistemic seterrainntyweigheee loss for visual bias mitigation. In Proceedings of the IEEE/ /CF Conference on Computer Vision and Pattern Recognition, pp. 2898 2905, 2022.
Harini Suresh and John V Guttag, A framework for understanding unintended consequences of machine learning. arXiv preprint 1rXi::1901.1000 2:8, 2019.
Lijing Wang, Dipanjan Ghosh, Maria Gonzalez Diaz, Ahmed Farahat, Mahbubul Alam, Chetan Gupta, Jiangzhuo Chen, and Madhav Marathe. Wisdom of the ensemble: Improving consistency of deep learning models. Advances in Neural Information Processing Systems, 33:19750-19977 19761, 2020.
15

---

Published in Transactions on Machine Learning Research (02/2024)
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx- ray8: Hospital-soale chest x-ray database and benchmarks. on weakly-supervised classification and local- ization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097-2106, 2017.
Zhifei Zhang, Yang Song, and Hairong Qi. Age gnggrossion/regrossiiinn by conditional adversarial autoencoder. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, pp. 5810 -8188, 2017.
16

---

Published in Transactions on Machine Learning Research (02/2024)
A DIPDI: General formulation and theoretical analysis
A.1 General formulation of DIPDI
In the main manuscript (Section 3.1 we introduced DIPDI for pools of 2 models for simplicity. However, having more models in each pool could help to reduce noise in the estimation. Thus, here we introduce a more general formulation for pools A = {A,,..., Am and {Bi,... Bm of m models each, with m denoting an even integer. The generalized formulation can be expressod
IG B,) Im (D Am/2+ ND Bm/24z)
@D B)
log
(3)
log2 m
To illustrate, let us exemplify the case when m =
BI) (2) B3) Ba) (B),
t (A, B) = log
(4)
AG Bz) No No B2, Ba)
By rearranging factors, we obtain
(N Ni) Ba)
Bo) B1) NN No B4)
() log
+ log
(5)
2
(Ai, A2)
A2) Nr. B3)
1 pp B) (Pr((A), As), (Bi, B3}) + (1A2, Ad), [B2, B,)))
(6)
In other words, our generalized formulation enables computing DIPDI for sets of models, by considering it the average of pairwise DIPDIs for different subsets of size 2.
To better understand the definition of the products in Eq. 3 we can imagine an x m matrix where rows and columns represent models in groups and B. respectively. For the products in the numerator, we simply take the main diagonal the matrix. For the products in the denominator, we focus on the upper diagonal of m/2 elements. A visual representation of this idea presented in Table illustrating the scenario for the specific case where 4.
B
A
3
4
1
2
4
2
3
A
1
A
B
4
2
2
/
4
2
/
3
3
3
4
4
4
(a) Inter-set pairs
(b) Intra-set pairs within
(c) Intra-set pairs within
Table 1: Illustration of product definitions for DIPDI when m 4. The / denotes the presence of a pair of models in factors ND(:)) of Eq. 5
A.2 Theoretical analysis of the relationship between DIPDI and performance gap
In order to generate an intuition behind why DIPDI may anticipate biases, we will resort to a theorctical analysis in a simplified scenario. We will work with a binary classification problem, where the soft scores of the models will be taken to one dimensional and assumed normally distributed. We call XA and XB the distributions of outputs for models from set and respectively. For simplicity, we will work with the outputs corresponding the positive target class, which we take to correspond values on the left of the decision boundary, but the problem is symmetrical and the same derivation can be replicated for the
17

---

Published in Transactions on Machine Learning Research (02/2024)
negative class. In Fig. illustrate this scenario with two model sets (in blue and green) that produce different error rates, given by their corresponding shaded areas on the other side of the decision boundary. Since XA and XB are normally distributed, we characterize them by their respective means (Ha and μа) and standard deviations (oa and 03). The error gap between these sets models for a given decision boundary
a)
b)
1.0
decision boundary
X XA
RX0
0.5
2
C)
d)
2.5
0.5
2.0
0.4
pap(A
1.5
0.3
1.0
0.2
0.1
0.5
0.0
5 3 2
3
4
07
04
Figure 5: a) Sketch of the distributions of soft outputs for two sets of models & (in green) and B (in blue). The respective error rates correspond to the fraction of outputs beyond the decision boundary (dashed line), indicated shaded regions of the same color. b) Scatter plot of the error gap vs. DIPDI for multiple combinations of distribution parameters μi and on, for fixed μı and σ c) & d) Surface plots for the error gap and DIPDI, respectively, for the same parameter configurations as in panel b).
can then be written by integration as
gap(A.B) = 1 N (2: dx - [ N (z)) dæ
(7)
DIPDI is a measure of the mean discrepancy between outputs of two model sets, relative the mean discrepancies of outputs within sets. Without loss of generality, we can take the squared error the discrepancy function to simplify DIPDI in this analytical interpretation. Since we assume the model outputs are normally distributed, the mean squared distance between outputs is given by
[d2a = 122B + AB.
with and
(8)
HAB = 14 - u4
(9)
2 = O + +02 AB
(10)
For points from within the same distribution, we have in turn
8 [dGA = 20 hh
(11)
E [dd
20
(12)
18

---

PB. Without loss of generality we take da fixed and let оB vary (sce Published in Transactions on Machine Learning Research (02/2024)
So that in the limit of considering large mumber of pairs of models, and for No VEld(,,-)], the DIPDI becomes
(FA- - ma3)2 0² + o
(A, B) log
(13)
20x on
Note that DIPDI is then sensitive to both a difference in the mcan and in the variance between the model predictions from two sets. In Fig. D--d, we have kept the parameters of fixed (Ax = 0, σR = 1) while varying those of B. The discrepancy grows if the distributions have different means, and also if the variances are different. Intuitively, if the mean is closer or further away from the decision boundary, the error rate will change. In that case the error gap is due to systematic shift in the predictions. In turn, if the variances are different, then the reliability of the two model sets are different, and DIPDI can also sense that. Indeed we observe that higher DIPDI values correspond to higher error gaps Fig. D.. In what follows we provide analytic expressions for the relationship between performance gap and output discrepancies when either the means or the variances of both sets are different.
A.2.1 DIPDI and unreliability
We first study the case where Na Fig. Ga). In this case we have
E [[222]
+ og MAB
+ AB
0
(14)
0 o
Φ (A, B)
log
(15)
2
Solving for o3, we can compute the gap as
1.00 N 10: v 16%
o dr S, N (x:pus dx
gap(A,B) B)
(16)
We from this equation that, long μs b so that the mean of X3 is on the correct side of the boundary, the gap is monotonically increasing function of the mean discrepancy. Indeed, see how the gap increases with DIPDI in Fig. do.
A.2.2 DIPDI and systematic errors
We then study the case where σд og. Again, without loss of generality we take pa fixed and let μa vary (Fig. Ge). In this case we have
E [dAB]
MBB
03
μB
20
(17)
P 20
4 (A, B) = log
(18)
20
Solving for μB, we can compute the gap as
gap(A,B) = 1.-0 N (2: 8 [d]22) 202, dr 1 (x: ow) dr
(19)
Once again, we see from this equation that the gap is monotonically increasing function of the mean discrepancy, and tight correlation between DIPDI and gap is present (see Fig. G1).
19

---

Published in Transactions on Machine Learning Research (02/2024)
a)
b)
decision boundary
0.3
Xa
errors Xs
gapla, ().8)
0.2
errors XA
0.1
0.0
0.0
0.5
c)
d)
1.0
decision boundary
Xg
errors Xs
gap(4, [9]
errors XA
0.5
0.0
x
Figure 6: a) c) Sketch of the distributions of soft outputs for two sets models & (in green) and B (in blue). The respective error rales correspond to the fraction of outputs beyond the decision boundary (dashed line), indicated variance, while in the models have the same variance but different mean. b) & d) Error gap DIPDI for models with either different variance (b) or different means (d).
shaded regions of the same color. In a) the models have the same
but different
B Experimental validation: Additional results
B.1 DIPDI on synthetic data: simulating distributions
Figure 7 shows examples of datasets simulated for the synthetic experiment in Section 4.1 of the main manuscript. These datasets are generated by varying the proportion of male and female samples training, from 100% to 0% males with step of 10% between each sampling. Then, each data point of the distribution is interpolated to be between and 100 years, adding random noise of 10 years to simulate real cases of age regression.
B.2 DIPDI on synthetic data: simulating model outputs
In this section, we verify the behaviour of DIPDI under controlled conditions using synthetic data by simu- lating model predictions. We simulate the predictions of two sets of models A (Ai, A2) and B (Bi, Ba} when evaluated samples from synthetic dataset ID and then systematically evaluate DIPDI in scenarios with different levels of disagreement between A and B. The model discrepancy is here simulated by the addition of a stochastic value of varying size (disagreement level) to the output predictions (Figure Sa
We consider the task of age estimation, so the outputs of models in A and B are assumed to represent predicted ages. We start with fixed sample drawn from a uniform distribution of ages between 30 and 80, representing the ground-truth ages, yk E Y. We simulate synthetic predictions for the models in and Bh by perturbing W with Gaussian noise sampled from distributions ng N((0, and na oz)
20

---

Published in Transactions on Machine Learning Research (02/2024)
Males
2.0
2.0
Parraks
1.0
0.5
0.5
0.0
0.0
0.0
0.55
0.5-
0.5-
-1.0
-1.0
-1.0
-1.0-0.5 0.0
0.5
1.0
1.5
2.0
-1.0-0.5
0.0
0.5
1.0
155
2.0
-1.0-0.5
0.0
0.5
1.0
15
2.0
Figure 7: Examples of datasets generaled for the synthetic experiment in Section 4.1
0.00
0.25
3.20
3.00
0.000
100
100
(a)
Fold
Fold
Fold 3
Fold
Fold
10
1.5
1.0
DiPDI
0.5
0.0
1 2
910
2
910
1
910
1 2
910
(b)
Figure 8: (a) Synthetic data construction. Examples of predicted ages simulated for models in A and B, for increasing levels of prediction disparities (left to right). (b) DIPDI on synthetic data. Both σд and range from to 10. Each fold represents a new run of the experiment with different random seeds. The DIPDI index computed by averaging 3 simulations for each a
Thus, for a fictitious data sample with ground-truth label yk E y, the synthetic model predictions are A((x²) = t ng and (2(x,) = yk + ng. Varying the standard deviations allows us create scenarios where the predicted ages for the analysis groups are or less similar, and then analyze the behaviour of DIPDI under different discrepancy ratios (see Figure Sal
DIPDI values for different discrepancy scenarios are displayed in Figure 8b considering N = 1000 and & and dR values in the range [1-10]. Note that when the oulputs of and B are similarly perturbed (as shown on the diagonal of each image), then & is close to O. However, when perturbations are sampled from wider Gaussian in one set than the other (as shown outside the diagonal of cach image), & tends to be higher than 0. This confirms the desired behaviour for our index: when intra-set predictions are more consistent than inter-set predictions, the index returns larger values.
21

---

Published in Transactions on Machine Learning Research (02/2024)
B.3 Subgroup analysis for age estimation
In this section, we present additional results in age estimation for ChestX-ray14 and UTKFaae datasets. These results complement the insights discussed in Section 4.3 of the main manuscript.
Tables and present results for ChestX-ray14 and UTee, reporting the mean absolute error (MAF)
values for male and female subgroups under different scenarios gender imbalance the training data.
Training (Male-Female)
Testing on Male
Testing on Female
100-0
4.633 (0.125)
5.823 (0.301)
75-25
4.679 (0.189)
4.888 (0.109)
50-50
4.795 (0.149)
4.717 (0.149)
25-75
4.802 (0.153)
4.565 (0.105)
0-100
5.265 (0.189)
4.527 (0.109)
Table 2: Mean absolute error (MAE) (mean std) for age estimation on ChestX-ray14 across subgroups (male, female) for models trained with different gender imbalance ratios.
Training (Male-Female)
Testing on Male
Testing on Female
100-0
6.170 (0.207)
9.006 (0.442)
75-25
6.301 (0.277)
6.275 (0.368)
50-50
6.530 (0.208)
6.004 (0.319)
25-75
6.785 (0.284)
5.780 (0.326)
0-100
7.468 (0.418)
5.719 (0.427)
Table 3: Mean absolute error (MAE) (mean = std) for age estimation UTKFaae across subgroups (male, female) for models trained with different gender imbalance ratios.
Figure 9 shows the cunulative score (CS) values for ChestX-ray14 and UTKFace for male and female subgroups under different gender imbalance scenarios in the training data. The CS quantifies the proportion of test samples (N) for which the absolute error e falls below a specified threshold of years. This calculation is defined as follows:
Negn
CS(n)
where Nesn represents the number of test images for which the absolute age error is less than or equal to the corresponding threshold value.
B.4 Subgroup analysis for younger vs older classification
In this section, present additional results regarding the classification of celebrities from the CelebA dataset in younger vs older. These results complement the insights discussed in Section 4.3 of the main manuscript.
Figure 10 shows normalized confusion matrices computed by subgroups (male, female) aggregating all folds for models trained with different gender imbalance ratios. Table g presents the Equality of Opportunity (EOD) metric (Hardt et al. 2016). These values, computed as the absolute difference over folds, reveal more unfair performance on younger vs older classification for models trained on highly imbalanced datasets, what is consistent with the behaviour of DIPDI shown in the results from the main manuscript (Figure 2
22

---

Published in Transactions on Machine Learning Research (02/2024)
ChestX-ray14
Testing on Male
Testing on Female
1.0
1.0
Training
100-0
0.8
0.8
Curulaativ Score (CS)
0-100
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
(a)
UTKFace
Testing on Male
Testing Female
1.0
Training
100-0
Curruaative Score (CS)
0.8
0.8
0.100
50-50
0.6
0.6
0.4
0.2
0.2
0
3
12
15
0
3
6
12
15
(b)
Figure 9: Cumulative scores (CS) for age estimation on ChestX-ray11 (a) and UTKFace (b) by subgroups (male, female) for models trained with different gender imbalance ratios. Age threshold n spans from 0 to 15 years.
Training (Male-Female)
EOD
100-0
0.096 (0.043)
75-25
0.030 (0.021)
50-50
0.034 (0.025)
25-75
0.052 (0.043)
0-100
0.363 (0.071)
Table 4: Equality of Opportunity (EOD) (mean + std) for younger older classification CelebA for models trained with different gender imbalance ratios.
B.5 DIPDI for age estimation and younger vs older classification
Table presents additional results for DIPDI on age estimation and younger vs older classification tasks as discussed in Section of the main mamuscript. We observe that DIPDI produces values larger than 0 when training data highly imbalanced in gender attributes indicating a greater propensity to bias.
23

---

Published in Transactions on Machine Learning Research (02/2024)
Testing on Male
100-0
75-25
50:50
25-75
0-100
6-22
0.10
138
2:82
0.17
-28
0.02
0.19
188
0.22
2:43
0.00
8.10
00
24
C4
0:83
0.51
32
22
62
12
02
(a)
Testing Female
100-0
75-25
0-100
CA2
128
0.19
0.22
2.15
0.03
00
-66
666
5.05
0.94
-2
0.15
0.82
0:55
0.89
82
0.36
2:84
0.35
0:45
-10
-2
-00
-00
-00
Wedindd
Rndicted
Redicted
Redicted
Fiskteed
(b)
Figure 10: Normalized confusion matricos computed on male (a) and female (b) subgroups for models trained with different gender imbalance ratios.
A. B
ChestX-ray
UTKFaae
IMDB-WII
CelebA
100-0, 0-100
0.576 (0.103)
0.918 (0.241)
0.799 (0.271)
1.447 (0.247)
75-25, 25-75
0.027 (0.040)
0.126 (0.063)
0.273 (0.278)
0.065 (0.167)
50-50, 50-50
-0.005 (0.065)
0.010 (0.074)
-0.106 (0.224)
0.014 (0.213)
25-75, 75-25
0.012 (0.073)
0.080 (0.050)
0.558 (0.196)
0.177 (0.163)
0-100, 100-0
0.530 (0.114)
0.883 (0.159)
0.993 (0.346)
1.501 (0.280)
Table 5: DIPDI (mean + std) for age estimation (ChestX-ray14, UTKFace and IMDB- WIKI) and younger vs older classification (CelebA). Groups and B consist of two models trained with different gender imbalance ratios. Test data gender balanced.
24


Reviews:
[{"review_id": "fvLg443Inx", "reviewer": "Reviewer_AMxc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper addresses the issue of anticipating potential demographic biases at deployment in the absence of ground truth annotations.\n\n- The proposed Demographically-Informed Prediction Discrepancy Index (DIPDI) provides a measure for the proneness towards biased solutions and can serve as an early warning about potential demographic biases.\n\n- The paper demonstrates the applicability of DIPDI in understanding how fairness transfers across distributions, particularly in scenarios involving population shifts.\n\nWeaknesses:\n- This paper lacks the theoretical results on the relationship between the index Demographically-Informed Prediction Discrepancy Index (DIPDI) and demographic biases.\n\n- Could DIPDI also be used as the potential bias measurement for other existing statistical fairness notions? Besides, is there a comparation between DIPDI with existing fairness assessment methods?\n\n- DIPDI is limited as a global population average. This paper does not explore its application to population subsets defined by multiple demographic traits or on a subject-by-subject basis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This research introduces the Demographically-Informed Prediction Discrepancy Index (DIPDI) as a method to anticipate potential demographic biases in machine learning models when deploying on new and unlabelled populations.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses above.", "broader_impact_concerns": "A Broader Impact Statement on the ethical implications is required in this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MH6T9KGjS3", "reviewer": "Reviewer_iDM1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The survey of past literature is comprehensive.\n2. The authors provide abundant experiments with not only synthetic but also three real-world datasets.\nFor weaknesses, please refer to “Requested Changes.”", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Data imbalance on sub-groups in a dataset may or may not lead to unequal performance. \nThis paper proposes a method to detect biases related to a demographic attribute in unlabeled datasets. The proposed approach involves analyzing output consistency among models trained on different demographic groups. Using a novel index, DIPDI, the authors assess whether models exhibit inconsistent predictions for the same data, indicating potential bias against a specific demographic attribute. Validation on synthetic and real-world datasets, specifically age estimation from face photos and X-ray images, demonstrates DIPDI's ability to anticipate bias without ground truth labels. The metric is effective in assessing bias in dynamic contexts with demographic shifts and provides early warnings in real-world scenarios without requiring expert annotations.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It is recommended that the authors provide visualization or simple numerical examples in Section 3 to illustrate the intuition of the proposed metric DIPDI, despite that the definition is simple. E.g., concrete examples for functions in set \\mathbb{A} and \\mathbb{B}. Why there are only two models? Is DIPDI able to be extended to many models in the sets?\n2. The definition of DIPDI is formed for regression problems, it is recommended that the authors provide an according definition of DIPDI for classification tasks, and compare this metric with existing fairness metrics in order to add value to this work. \n3. Why choosing 50-50, 0-100 for the models in Table 1? Why not changing from 0 to 100 with a difference of 10, i.e., 10 different settings?\n4. Can the authors add a simple theoretical analysis between the relationships of distribution shift and DIPDI using some simple examples? \n5. The authors claim that the DIPDI can find potential bias. Can the authors validate it with a dataset that has ground-truth annotation to check the usefulness of this new metric? \n6. How do we related the definition of DIPDI with counter-factual desperate impacts?", "broader_impact_concerns": "Part of the wordings in the final section “Discussion” could be added as a paragraph for the broader impacts.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZuGGj6gczp", "reviewer": "Reviewer_EvUh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a compelling idea which I believe has the potential to draw on interesting connections across the fairness, robustness and distribution shift literature. I believe that the intuition behind this approach is sensible and could potentially provide useful test-time fairness signals, particularly in the case of shifted distributions. However, I’m not sure that the theoretical/conceptual work in this paper goes far enough to establish this idea in a fully-fleshed-out way, and I didn’t find that any of the experiments fully supported the claims made about the effectiveness of DIPDI.\n\nSome feedback:\nThe main criticism I have around this paper is with the experiments. There are 3 experimental sections and I don’t necessarily find any of them particularly compelling as support for the claims made around DIPDI. I’ll go through these one at a time:\n- Synthetic data - 4.1: I find these experiments unconvincing since they don’t really represent an application of the method; in particular, they skip a really key step which is the training of the models themselves. As currently set up it just seems like a demonstration of calculating the metric, which is not so helpful in my eyes. I think a good synthetic experiment which the full method (train a pool of models and compare outputs) is applied would be much more helpful.\n- Real data IID - 4.2-4.4: I think these experiments are mostly fine but I found them a little underwhelming for 2 reasons. First of all, they are only conducted on 100-0 and 50-50 data imbalance; given how important data imbalance is to the paper I would have expected a slower gradation (show me 90-10, 80-20, etc.). This would be very helpful to show how sensitive the metric is. Secondly, I was disappointed to see that potential confounds were removed from the ChestX-ray14 set - handling these types of confounds is necessary in order for fairness metrics to perform well on real data. The authors should use the data as-is, or need to justify this choice in my opinion.\n- Domain shift - 4.3: This is an interesting experiment and I’m glad the authors explored a shifted scenario. However, I’m confused by the main results shown in Figure 3. I’m not sure why these plots don’t look more similar between a and b. Specifically, it seems odd to me that the test=male subgroup doesn’t exhibit a similar diagonal pattern on the right hand side of 3a as the test=female subgroup does on the right hand side of 3b. This could use some more explanation as to why these behave differently and also as to the impact of this on the DIPDI metric. I also wouldn’t mind seeing more different experiments in distribution shift scenarios, possibly exploring different types - the claim that this metric is helpful under shift is intriguing but as of right now the experiments only address an extremely narrow version of that claim (attribute-specific label shift, with caveats).\n- I think this paper misses out on some potentially strong and clarifying impact by not discussing the connection to distribution shift. In particular, I’d be curious to know in what cases we might expect this method to work (what are its limitations) and how that interacts with the nature of the distribution shift. The authors reference the Schrouff et al paper which I believe contains an example of this type of discussion. Tracing out the limitations here would be very helpful - as is, I’m left in the dark about where I might expect this method to work\n- The claims around “bias” are unnecessarily broad - “bias” can entail many things and can imply many different metrics, however in this paper the authors are only concerned with disparities in accuracy. This is fine from a substance perspective - however I think that this should be mentioned prominently in the introduction and throughout, given the exclusive theoretical and empirical focus on that notion of bias.\n- It’s unclear to me why the experiments are all age prediction - I don’t necessarily have a problem with it but I think given how general the proposed method is, it might behoove the authors to demonstrate its effectiveness on some other tasks. This would be even stronger in a non-image modality, but not required\n- I think it would be interesting to see this metric defined in more generality - for instance, it seems like it could be made much stronger by extending to pools of > 2 models on each attribute. The authors don’t discuss this and as such I think the metric is more brittle than it needs to be\n\n\n\n\n\nSmaller notes:\n- Unclear exactly what a “new target population” is - is it a reweighting of existing sensitive groups? Reweighting along some other attribute? A totally new sensitive group? A different context?\n- Should draw more connections to other literature like distribution shift and underspecification\n- Citation format: missing brackets around many citations - look at \\citep in LateX rather than \\citet\n- In 3.1, is D a dataset or an input space? Some notation confusion. \n- In 3.1 why does the range of A have to be positive? \n- I’m not sure I totally understand the discussion/analogy to uncertainty in the Discussion section", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "EDIT: I have a couple of notes remaining in response but pending those being cleared up, I'm updated my review here to \"Yes\" on \"Claims and Evidence\", since the authors have updated their experimental section appropriately.\n\nIn this paper, the authors present a method for predicting when biases, in the form of disparities in accuracy, will arise at test time given unlabelled data. The method involves a metric, DIPDI, comparing the test-time difference in prediction between models that are trained solely on data from one sensitive group. The between group difference is compared to a within group difference, as determined by a sample of two models. Experimentally, the authors run their method on synthetic and real data and analyze the results, arguing that DIPDI successfully predicts disparities, even under particular kinds of distribution shift.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My requests here are substantial enough that I don’t think I can qualify any as sufficient for securing my recommendation for acceptance. However, the proposed adjustments would include:\n\nMost critical: Deepening and broadening of each of the 3 experimental sections as discussed above (synthetic: make the setup closer to the actual method, real data: more extensive exploration and analysis)\nAlso critical: clarification in the paper framing of what “bias” means throughout, narrowing scope to the specific metric of consideration\nModerate: explore the method in more conceptual depth, such as by drawing connections to distribution shift and by extending to larger ensembles than n=2\nModerate: extend the scope of experiments to outside only age prediction, or outside only images\n\nI do think this idea is quite interesting and would enjoy seeing a more thorough treatment of it both conceptually and empirically.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This study examines how machine learning systems can exhibit biases against certain demographic groups due to data imbalance and under-representation in training datasets. The authors propose an approach to identify potential biases using output discrepancy among models trained on different demographic groups. The proposed method DIPDI measures the consistency of model predictions without resorting to the labels in the new target population. Through numerical experiments with synthetic and real-world datasets, they demonstrate how DIPDI can help anticipate demographic biases in machine learning models deployed on new and unlabeled populations.

The reviewers agree that the paper solves an important problem in fair machine learning, the proposed method is well motivated and reasonable. The reviewers also suggested adding more theoretical justifications and experiments to illustrate the effectiveness of the method.  After revision, two reviewers are satisfied with the new version. Reviewer AMxc still has some concerns regarding the theoretical analysis and empirical evidence. However, both myself and the other two reviewers believe that the revised version now includes more comprehensive theoretical and empirical analysis, meeting the criteria for publication. Overall, I think the paper proposes a new method and verifies its effectiveness and the presentation is clear. I thus recommend for acceptance.
Decision: Accept (poster)