Subgroups Matter for Robust Bias Mitigation
Authors: Anissa Alloula
Charles Jones
Ben Glocker
Bartlomiej Papiez
Venue: ICML 2025 poster (2025)
Keywords: bias mitigation
robustness
spurious correlations
generalisation
fairness
Primary area: social_aspects->fairness
TL;DR: Our work highlights the importance of careful subgroup definition in bias mitigation and suggest it as a alternative lever for improving the robustness and fairness of machine learning models.

Abstract: Despite the constant development of new bias mitigation methods for machine learning, no method consistently succeeds, and a fundamental question remains unanswered: when and why do bias mitigation techniques fail? In this paper, we hypothesise that a key factor may be the often-overlooked but crucial step shared by many bias mitigation methods: the definition of subgroups. To investigate this, we conduct a comprehensive evaluation of state-of-the-art bias mitigation methods across multiple vision and language classification tasks, systematically varying subgroup definitions, including coarse, fine-grained, intersectional, and noisy subgroups. Our findings reveal that subgroup choice significantly impacts performance, with certain groupings paradoxically leading to worse outcomes than no mitigation at all. They suggest that observing a disparity between a set of subgroups is not a sufficient reason to use those subgroups for mitigation. Through theoretical analysis, we explain these phenomena and uncover a counter-intuitive insight that, in some cases, improving fairness with respect to a particular set of subgroups is best achieved by using a different set of subgroups for mitigation. Our work highlights the importance of careful subgroup definition in bias mitigation and presents it as an alternative lever for improving the robustness and fairness of machine learning models.

Subgroups Matter for Robust Bias Mitigation
Anissa Alloula Charles Jones? 2
Ben Glocker 2
Bartiomiej W. Papiež
Abstract
by bias in training data such as spurious correlations (SC), inder-representation of certain subgroups, or shifts in the presentation of the target Y (Jones et al., 2024). Bias miti- gation methods aim to address these issues by training more robust models which are less susceptible to these biases, thereby improving generalisation. These methods generally adapt model training to improve the performance of some disadvantaged subgroups within the training data.
Despite the constant development of new bias mit- igation methods for machine learning, no method consistently succeeds, and a fundamental question remains unanswered: when and why do bias miti- gation techniques fail? In this paper, we hypothe- sise that key factor may be the often-overlooked but crucial step shared by many bias mitigation methods: the definition of subgroups. To investi- gate this, we conduct a comprehensive evaluation of state-of-the-an bias mitigation methods across multiple vision and language classification tasks, systematically varying subgroup definitions, in- cluding coarse, fine-grained, intersectional, and noisy subgroups. Our results reveal that subgroup choice significantly impacts performance, with certain groupings paradoxically leading to worse outcomes than no mitigation at all. Our find- ings suggest that observing a disparity between a set of subgroups is not a sufficient reason to those subgroups for mitigation. Through theoret- ical analysis, we explain these phenomena and uncover a counter-intuitive insight that, in some cases, improving fairness with respect to a par- ticular set of subgroups is best achieved by using different set of subgroups for mitigation. Our work highlights the importance of careful sub- group definition in bias mitigation and presents it as an alternative lever for improving the robust- ness and fairness of machine learning models.
Despite the number of bias mitigation methods which have been proposed, benchmarks are increasingly reporting that the performance of these methods is inconsistent when tested in new settings, and that they often fail to surpass the empirical risk minimisation (ERM) baseline (Zong et al., 2023; Zietlow al., 2022: Chen et al., 2023; Shrestha et al., 2022; Alloula et al., 2024). Some efforts have been made to begin to elucidate the conditions under which certain bias mitigation methods might be valid, such as the work of Jones et al. (2025) in fair representation learning and Schrouff et al. (2024) in data balancing. However, the choice of an appro- priate mitigation method is only one aspect of the problem. To successfully mitigate bias, we must also select which subgroups we wish apply the methods on- question which very little work has explicitly addressed.
Indeed, most bias mitigation methods rely on some form of grouping to first identify disadvantaged subgroups within the training data and then to implement group-based strate- gies aimed at improving generalisation or fairness. This can be as simple as observing a disparity in model performance between men and women, and trying fix this by rebal- ancing the training data such that both subgroups are more uniformly distributed (Weng et al., 2023), or noticing that model performs poorly on data coming from a specific type of scanner, and applying a robust learning strategy such as adversarial training to prevent learning of scanner-specific but mask-irrelevant information (Ganin & Lempitsky, 2015). To date, the literature has predominantly focused on simple and coarse subgroups, for example blonde hair and non- blonde hair in the CelebA dataset (Liu et al., 2015b) or waterbirds and landbirds in the waterbird dataset (Sagawa* et al., 2020). In medical applications, the subgroups are often "white" or "non-white", "men" or "women", or some coarse binning of age (Ricci Lara et al., 2022). Subgroup choice often motivated by two factors: a) practical con- straints, for instance only having annotations for common
1. Introduction
A significant barrier to the wider deployment of machine learning (ML) models is their tendency to fail when tested on distributions that differ from their training data. One particularly concerning manifestation of this issue is perfor- mance degradation for population subgroups, often caused
University of Oxford, UK IImperial College London, UK. Cor- respondence to: Anissa Alloula altiiss..llolaaaaal wwt.aa/wwt.tc...........
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Subgroups Matter for Robust Bias Mitigation
attributes, and b) ethical or societal goals to achieve faimess with respect specific subgroups. However these sub- groups may poorly capture the underlying cause of model underperformance, thus obscuring critical information for bias mitigation methods.
et al. (2012), offers a theoretically elegant approach by eval- uating fairness at the individual level, its practical challenges have limited its adoption. As a result, group-based analy- ses remain the dominant paradigm. More recently, efforts have been made to move beyond traditional binary cate- gories (e.g., men/women or white/non-whitee to identify disparities that such coarse classifications may obscure. For instance, Kearns et al. (2018) and Buolamwini & Gebru (2018) illustrate how failing to account for intersections of attributes can entirely hide performance disparities. Xu et al. (2024) discuss the difficulty of identifying intersec- tional bias when there are number of attributes at play, and propose a generative approach to discover high-bias inter- sections amongst many possibilities. Similarly, Movva et al. (2023) demonstrate that in clinical risk prediction models, variation in performance within commonly-used ethnic groups often exceeds the variation between these coarse groups, advocating for the use of much more precise cat- egories to describe ethnicity. These works highlight how heterogenous subgroups on which unfairness is observed can be, yet most of them do not consider how this translates to conducting bias mitigation.
In this work, we aim to better understand whether we can optimise this crucial step of subgroup definition in the same way that new bias mitigation methods are optimised. We in- vestigate the role of subgroup definition on the performance of these methods, and whether poor subgroup definition might explain why these methods often fail. We construct a setting of bias inspired by a real-world chest radiograph example (Olesen et al., 2024) where there is a spurious cor- relation during training but which absent during testing, and which is present in different proportions across sub- groups, resulting in disparities in model performance. We consider realistic ways in which subgroups might be gen- erated based on relevant attributes, and explore how they impact the performance of four state-of-the-art bias miti- gation methods in four semi- semi-synthetic vision and language datasets. We identify key patterns in the performance of these methods across groupings. Certain groupings lead to large improvement over the ERM baseline, while oth- ers substantially lower performance relative to the baseline. We propose that the effectiveness of a given subgrouping strategy is linked to its ability to recover the unbiased test distribution. We summarise the key contributions of this work as follows:
2.2. Subgroup definition for bias mitigation
A smaller body of work has considered how this problem of subgroup definition extends to bias mitigation. For example, Awasthi al. (2020), Wang et al. (2020a), and Stromberg et al. (2024) explore how noise in subgroup annotations may impact post-processing, distributionally robust optimi- sation (DRO), and last-layer-rtrrininng respectively. They find that fairness is not guaranteed under some perturbations of the true attributes. Ghosal & Li (2023) also extend group DRO (gDRO) with probabilistic subgroup labels if there uncertainty with respect to the subgroup annotations. Li et al. (2023); Kim et al. (2024) show the difficulty of mit- igating bias when there are multiple spurious correlations and annotated subgroups. Perhaps the work most closely related to ours is Zhou et al. (2021), which considers a set- ting where a spurious correlation is causing bias, and show that gDRO fails in an example where the subgroups do not directly account for the spurious correlation. They attribute this failure the inability to upweight bias-conflicting sam- ples effectiiely, their constructed "imperfect" groups also include spuriously correlated samples though their sub- group construction appears somewhat unrealistic. All of these works point to a sensitivity of bias mitigation methods to the subgroups defined, however they each only explore one possible flaw in subgroup annotations and restrict their scope a single bias mitigation method.
We show that the groupings used for bias mitigation strongly affect how well each method works, and pro- vide insights on optimal grouping strategies.
We argue that observing a disparity in model perfor- mance across set of subgroups does not justify using those subgroups for mitigation, and may in fact, make matters worse.
We provide a possible explanation for the differences in subgroup effectiveness based on the minimum KL divergence between the subgroup- weighhed biased dis- tribution and the unbiased test distribution.
We challenge the conventional assumption that the best way to obtain "fairness" with respect to a specific set of subgroups is always achieved by using those same subgroups for bias mitigation.
2. Related work
2. Bias identification
2.3. Bias mitigation without subgroups
Research on bias detection has increasingly focused on refin- ing subgroup definitions to capture complex patterns of un- fairness. While individual fairness, as introduced by Dwork
While the above works explore some possible failures in subgroup definition for common bias mitigation methods,
2

---

Subgroups Matter for Robust Bias Mitigation
others have developed new methods altogether which do not require subgroups to be defined in the traditional way. For instance, Kearns et al. (2018) and Hebert-Johnson et al. (2018) propose algorithms which aim to achieve faimess across all identifiable or richly structured subgroup classes. Bias discovery methods bypass the question of pre-defining subgroups altogether, and are useful in settings where sub- group annotations are missing. Some methods decouple bias discovery from bias mitigation by first "discovering" biases through inferring subgroup annotations with an ex- ternal model (Han & Zou, 2024; Marani et al., 2024) or by clustering points based on their feature representations (Krishnakumar et al., 2021), and subsequently performing bias mitigation. Other methods forgo explicit subgroup iden- tification and instead rely on some form of regularisation or upweighting of misclassified samples (Ahn et al., 2022: Park et al., 2024; Liu et al., 2021).
information may be unavailable at inference time.
Group distributionally robust optimisation (gDRO) (Sagawa" et al., 2020) reweights samples during loss calcu- lation. The objective is to minimise the worst-case expected loss across each subgroup Pg. (Equation (2)). In practice, an online optimisation algorithm is used to assign higher weight high-loss subgroups in each batch.
0ADRO = arg min max ((o.a))Prraing |e(0: (z. (e(0 {
aE0 gEG
(2)
Resampling (Idrissi et al., 2022) also relies on reweighting but achieves it by adjusting the sampling probability of each subgroup Pa at the batch level so that each batch is balanced across subgroups. The loss for resampling is equivalent to:
We restrict our main analysis to established mitigation meth- ods which use subgroup annotations because (a), they often serve as the upper bound for mitigation methods (Zhou et al., 2021; Pezeshki et al., 2021; Bayasi et al., 2024), (b), many nominally label-free methods still require subgroup anno- tations for validation or hyperparameter tuning (Pezeshki et al., 2024), and (c), label-free methods frequently infer subgroups, making it still essential to understand their role.
Oresampling == arg min AE
B(x.y))PPrasn
[0(0; (z, v))]-
k
9 0
Domain Independent (DomainInd) learning adjusts the model architecture by replacing the single classifier head with separate classifier heads, each corresponding a subgroup, such that although each sample is passed through the same encoder, the decoder can be fine-tuned to each subgroup (Wang et al., 2020b). At inference, the classifier head with the largest activation makes the final prediction.
3. Background
3.1. Overview of bias mitigation methods
3. EMPIRICAL RISK MINIMISATION
Conditional learning of fair representations (CFair) aims to learn fair and robust representations of the target label independent of any subgroup information (Zhao et al., 2020). This is achieved by aligning conditional representa- tions of samples from different subgroups.
Traditional deep learning models conduct empirical risk min- imisation (ERM), where, for given inputs x E X and labels y E 3' from a distribution P, the objective is to find model 0 E O that minimises the expected loss Ep[e(0) (1, v))] However, in practice, only a subset of P, denoted Ptrains is available, so the loss over all samples in Ptrain is min- imised:
1.3. SUBGROUP DEFINITION ACROSS THESE METHODS
In this work, we distinguish between two categories of bias mitigation methods: reweighting (gDRO and resampling), and model-based methods (DomainInd and CFair). This is because of differences in how subgroups are generally defined. While in model-based methods subgroups are de- fined solely based on attribute(s) A. reweighting methods can additionally define subgroups over A x y. This is be- cause the former methods perform best if each subgroup contains both positive and negative samples. We expand on this distinction and its implications in Appendix A.2.
ÔERM := arg min ((r.y))Proau [e(0; (I, ())].
(1)
8E0
3.1.2. COMMON BIAS MITIGATION METHODS
Minimising the average loss over Ptrain often results in poor generalisation and poor performance on minority sub- groups in the data. To address these shortcomings, bias mitigation methods have been proposed as alternatives to ERM. In this work, we focus on 4 commonly used bias mit- igation methods which have demonstrated -t-te-tf- results in certain tasks and represent the broad spectrum of existing methods. We define & disjoint subgroups P, indexed by g = [1,...,k) which partition Ptrain- We as- sume each training sample is annotated with its subgroup label g, thus giving the tuples (x. y. g); however, subgroup
3.2. Problem setting
We frame the task according to the faimesss paradigm de- scribed in Jones et al. (2025), whereby the objective is to generalise from a biased training distribution to an unbiased
3

---

Subgroups Matter for Robust Bias Mitigation
Biased train/val distribution
Unbiased test distribution
X5
S
X
Selection
A
Legend:
X
Y
P Prain
S-1
Prest
Figure 1. Causal graphs representing interactions between Y. A. and S variables in the training data and in the unbiased test data. Conditioning selection in the training data results in spurious correlations between Y, A, and S, Coloured bars also illustrate the proportions of Y, A, and S combinations in both settings.
testing distribution. Dataset bias can take many forms (as discussed in Jones et al. (2024)), but here, we focus on bias arising from spurious correlations between certain attributes of the data and the class label which disappear the un- biased deployment setting. We define two binary attributes A and S whose information is encoded in X in the form of latent features X and Xs respectively. Features relating to the true class Y are represented as Xy. In the unbiased test setting, attribute-related information is independent of Y, such that P(Y)= = P(Y A) = P(Y S). However, Prrain, mechanisms like data selection may lead to the violation of this independence. Causal graphs illustrating this scenario shown in Figure 1.
samples satisfying this correlation.
We represent each distribution as probability vectors TR with each element corresponding to the probability of sam- pling one (Y, A) subgroup. The unbiased distribution is
defined be uniform, i.e., P unbiassd [2 5..- while
Puzin [44 0.05 0.8 0.2 0.05 0.95
0.2 0.31
Fur-
ther details are presented in Table and Figure 1.
Table 1. Probability distributions across Y, A, and S in the biased train and validation dataset and the unbiased test set.
Probability distributions
Puni
P Pastiesed
In particular, in this study, we consider an example of spurious correlation between A and Y. Furthermore, we consider that the spuriously correlated samples are unevenly distributed across the second attribute S. This is inspired from real world medical imaging example from Olesen al. (2024). Specifically, they describe a chest X-ray di- agnosis model which shows better performance in one sex subgroup (denoted here by S). Previous attempts reduce this disparity, such as resampling of the data across S, and manipulation of sex-specific regions of the X-rays, were ineffective, leading to the hypothesis that sex-specific dif- ferences were not causing the disparity (Weng et al., 2023). Subsequent work revealed that men and women presented different proportions of chest drains and ECG wires, which the model used spurious correlations to predict disease, and that, balancing the test data with respect to these arte- facts (corresponding to A in this work) resulted in equal performance across sexes. To mimic this example, we use 0 con-
P(Y = 1) P(A=1)
0.5
0.5
0.5
0.5
00) PPV
P(Y =1]=====1===]]]
0.875
0.5
P(Y 0.A P
0)
0.95
0.5
P(Y 0 8= P 1,4
0.8
0.5
4. Experiments
4.1. Subgroup generation
To understand how different subgroup definitions affect the generalisation performance of bias mitigation methods, construct multiple sets of subgroups based A, S, and Y and train each model with them. Our goal is to simu- late realistic scenarios, for instance where one may only have access to certain variables or to noisy subgroup an- notations, or when deciding between the use of coarse or fine-grained subgroups (c.g., a level of ethnicity categorisa- continuous variable like age). We
a semi-synthetic setting where the subgroup tains 95% spuriously correlated samples, while the subgroup S contains smaller, though still substantial, proportion of spuriously correlated samples (80%). With this setup, there would be substantial difference in a model's per- formance across subgroups if it only correctly classified
tion, or discretising denote subgroups constructed the intersection of multiple variables a x as (a, b).
For basad methods, these include: 1) subgroups based on single variable: A, Y, and $ 2) subgroups based on the intersection of two or three variables: (A, Y), (8, Y), and

---

Subgroups Matter for Robust Bias Mitigation
(Y,S, A) 3) SC/no-SC subgroups where we group the two bias-aligned (A,Y) subgroups together and the two bias- conflicting (A, Y) subgroups together 4) subgroups based on the random splitting of existing subgroups: (A, Y)s, (S, Y)s which split each (A,Y) and (S, Y) subgroup in two such that there are 8 subgroups in total 5) 4 completely random subgroups. For model-based methods, we do not include subgroupings based on Y and also add A,. S1 (A and 5 subgroups randomly split in two respectively), and (A,S) subgroups. Finally, we explore 6) the impact of noise in subgroup annotations by injecting noise (in the form of mislabelling) into 1- - 50% of the (A,Y) and A subgroup annotations for data- and model-based methods respectively. This noise does not affect the class labels Y. For Civil comments, in addition to the synthetic granular subgroups, we directly explore the impact of granularity on real subgroups, the dataset contains subgroup information of multiple hierarchy levels.
gender, and S any mention of religion. Additionally, as this dataset contains multiple levels of subgroup annotations, we do real experiment on the impact of subgroup granularity (instead of simply randomly splitting subgroups in two), We compare mitigation on the A groups to mitigation on granular A groups (e.g., any mention of males, any mention of another gender, and no mention of any gender), and likewise for S (e.g., any mention of the Christian religion specifically, any mention of a non-Christian religion, and no mention of religion).
4.3. Implementation details
We implement and train models with each of the gDRO, resampling, DomainInd, and CFair bias mitigation methods. We apply each method to each of our generated subgroups and average the results over three random seeds. We repeat this process for the four datasets, comparing performance of the bias mitigation methods with the baseline ERM method. In total, we train 306 models, with 440 NVIDIA A100 hours of compute. The training strategy, hyperparameters, architectures etc. are the same across all models, detailed in Table A7, except for necessary adjustments to apply each bias mitigation method. The code is available here.
In total, we consider 15 subgroup combinations for reweighting-basee methods and 12 for model- meth- ods. We illustrate these subgroups in Figure A6. This comprehensive set of subgroup definitions allows us sys- tematically investigate the impact of subgroup choices on the effectiveness of bias mitigation methods.
We report the mean and standard deviation of the aggregate area under receiver operating characteristic curve (AUC) on the unbiased test set (following the fairness paradigm described in Jones et al. (2025)), alongside worst-group accuracy and accuracy gap across subgroups. We select these measures for their directness and simplicity compared to other fairness criteria. We do not vary subgroup definition for evaluation and only evaluate accuracy with respect to S and subgroups, as we are simply interested in how subgroup definition affects the mitigation process.
4.2. Datasets and tasks
We evaluate performance in image classification tasks in four datasets which we construct to satisfy the distributions specified in Figure 1. We summarise all details in Table A5.
We adapt the MNIST dataset (Lecun et al., 1998) by bina- rising the classification task into predicting whether a digit even or odd (Y)) We add additional attributes by modi- fying the image background colour (A) black or white, and colouring the foreground (S) as red green. This controlled setting allows for clear evaluation of subgroup influences in simple task.
5. Results and discussion
5.1. ERM performance drops on the unbiased test set
To explore a more challenging and realistic setting, we re- peat the experiments with chest X-ray images from the Che dataset (CXP) (Irvin et al., 2019). The task is classification of the presence of pleural effusion (Y). S is the sex of the patient and A the presence of pacemaker, us- ing annotations provided in (Anthony & Kamnitsas, 2023).
The baseline model, trained with no bias mitigation, shows a sharp drop in performance when tested on the unbiased test set, with a decrease of 0.10 to 0.25 in AUC for all datasets (Table 2). This drop occurs because the model has learned to rely on the spurious attribute as a proxy for Y, and this correlation is absent in the unbiased test set. Moreover, the ERM model exhibits disparities in performance. This is particularly pronounced for the S subgroups, both on the biased validation set and on the unbiased test set. A standard approach would therefore have been to apply mitigation to the S or (S, Y) subgroups. In the following sections, we explore whether various bias mitigation methods, used with different groupings, can improve test set performance and reduce these disparities.
We explore another real vision dataset commonly used in fair ML research, CelebA (Liu et al., 2015a). The task is binary classification of whether the individual has blonde hair (Y') with additional attributes A, perceived gender, and S, whether the individual is smiling.
Finally, we explore whether our findings extend to the text modality through the use of the Civil comments dataset, also commonly used in fair ML (Borkan et al., 2019). The target Y is toxicity prediction, with A being any mention of
5

---

Subgroups Matter for Robust Bias Mitigation
Table The baseline model performance shows sharp decrease the unbiased test for all datasets. Suhgroup-wis accuracy also reveals large disparities with respect to S in both datasets. Mean and standard deviation across three random seeds are shown.
MNIST
CXP
Civil comments
CelebA
Accuracy
Val
Test
Test
Val
Test
Val
Test
Overall
0.943 0.012
0.698
0.054
0.898
0.007
0.003
0.726
0.024
0.954
0.003
0.865
0.007
Min
0.936 0.025
0.694
0.003
0.869
0.013
0.020
0.878
0.006
0.724
0.037
0,952 1 0.002
0.838
0.019
Gap (4)
0.014 0.026
0.008
0.112
0.058
0.014
0.220
0.028
0.016
0.013
0.004 1 0.047
0.005 1 0.007
0.055 1 0.025
Min (8)
0.917 + 0.024
0.599- + 0.061
0.841 + 0.010
0.633 + 0.020
0.837
0.002
0.726 + 0.029
0.940 + 0.004
0.861 + 0.002
Gap (5)
0.052 + 0.025
194 = 0.080
0.116
0.015
0.052 0.026
0.095
0.009
0.006 + 0.047
0.029 + 0.008
0.009 + 0.014
Subgroups which yield worse results than ERM
baseline
0.00
Subgroups which improve upon ERM
NN/T SDRO
NNUS Recsmp Ing
CXP 9DRO
CXP Resampling
esA g0MO
esA Resamping
Commenttts sDRd
ComCmnentts Sesempling
YSA
AT
Rancom
$Y_$
Subgroup
Figure 2. Performance on theP the unbiased in gDO and resampling is highly dependent on the subgroups used across all four datasets. Bars represent overall change AUC relative to the ERM baseline, with error bars indicating the standard deviation across random seeds.
5.2. Groupings used for mitigation strongly impact bias mitigation performance
away from relying on the spurious attribute A. For model- based methods, a similar pattern is evident (Figure B7); A subgroups generally present the best performance (analo- gous to (A, YY) in data-based methods). For instance, with A subgroups in DomainInd, performance is increased by 0.07 in the CXP dataset. On the other hand, S subgroups consistently harm performance, decreasing it by up to 0. 14 for CFair in MNIST.
We find that test set performance is highly dependent on the subgroups used for mitigation, with some subgroups boost- ing overall performance by more than 0.07, while others reduce it by up 0.08 (Figures 2 and A7). We first note that the four datasets reveal similar patterns across group- ings for reweighting based bias mitigation methods and model-based methods, suggesting that there are universal trends relating to the bias setting which generalise outside of the specific dataset, task, and mitigation method. For reweighting bbased methods, subgroups based on (A, i.e., (A,Y) (A, s),,,, SY, A), (A,Y) with small levels of noise, and SC/no-SC subgroups, enable the model to focus on the (A, Y) pairs without the spurious correlation during training. Therefore the model learns to predict Y independently of A, leading to better generalisation perfor- mance. Conversely, subgroups which do not take (A, Y) information into account tend result in worse performance than the baseline model (for instance (.SSYY.(S. Y) YY. A, and Random subgroups), they fail to guide the model
Applying bias mitigation with certain groupings can lead to worse outcomes than ERM. For instance, S subgroups are detrimental to generalisation performance 10 out of 16 experiments, and have no effect the remaining 4 (Fig- ures 2, A7). They are detrimental despite the fact that there is substantial disparity in performance between them (Ta- ble 2). This observation suggests that in the absence of information about the underlying mechanism causing bias, it may be better to refrain from using mitigation methods altogether, even for simple approaches like data balancing. We believe that this finding may partly explain the failures of bias mitigation methods reported in recent studies (Zong
6

---

Subgroups Matter for Robust Bias Mitigation
al., 2023; Zietlow et al., 2022: Chen et al., 2023; Shrestha et al., 2022), where inadequate subgroup definitions may have caused pounter-productive bias mitigation.
2.12
Increasing subgroup granularity has no significant effect on performance. For example, there is little difference be- tween results for (A, Y) vs (A, Y)s and (S, Y) vs (S, Y)s (Figure 2). This is most likely because the bias mitigation methods tested here are designed to maintain stability if practice, this suggests that dividing subgroups into finer subgroups is unlikely harm performance, and may even be beneficial if one is unsure which specific attributes may be responsible for underper- formance. This insight could be relevant in settings where there are multiple levels of granularity to choose from (such as continuous data e.g., age, or multi-level categories e.g. ethnicity). We validate this with the real coarse and granular subgroups in Civil comments, and indeed find no signifi- cant difference in unbiased generalisation for resampling and g gDRO, as shown in Figure 3.
with
-6.0
MAST: 2060
127 uERO
there are many subgroups.
6AACC
Foraa Acsorccii
Figure 4. Noise in (A, Y) subgroup labels leads to a degradation in AUC for gDRO and resampling. Each dot represents mean performance on the unbiased test set for specific grouping, with error bars indicating the standard deviation across 3 random seeds.
using completely random subgroups generally worsens per- formance relative to ERM. This is most likely because the methods themselves are sub-optimal relative to ERM on the same distribution. This suggests that when one has no idea if the attributes for which they have annotations are related to possible biases, mitigation should not be conducted.
ABC 1o rbbooedtio
0.2
5.3. The ability to recover the unbiased distribution is key to mitigation success
We next aim to explain the observed variation in perfor- mance across different groupings. Inspired by (Zhou et al., 2021), we explore whether it is possible to recover the unbi- ased distribution by weighting the chosen subgroups. Our hypothesis is that the closer the weighted Perains which we denote Pu is to the test distribution P Panbiased, the better the model will perform on Punbiosed- This reflects the broader consensus in the generalisation literature that aligning train and test distributions (c.g., through methods like data balancing) can reduce generalisation error in non- i.i.d. settings (Mansour et al., 2009; Dong et al., 2024; Wang et al., 2023). For example, Ben-David et al. (Ben-David et al., 2010) show that for any hypothesis h, assuming the same labelling function in both source and target domains, the test error is bounded as follows:
0.5
AT WKgDOoigenarr Al Aroamniing (panzer) gon Indigtort
SY roaumppinng frolig ani
Figure 3. Performance on Pundiased similar for the coarse and granular subgroups in Civil.oomments (mention of gender vs specific male/female mention (A) and mention of any religion vs specific mention of Christianity (S)). Error hars show standard deviation across random seeds.
Mitigation methods are relatively robust to noise. As shown in Figure 4, performance of gDRO and resampling degrades when noise in the (A, Y) annotations exceeds 10%. However, even under these conditions, these methods still outperform the baseline model, indicating that they are rela- tively robust to annotation noise affecting minority of sub- group annotations. This aligns with findings from Awasthi et al. (2020) and (Stromberg et al., 2024) who explore the impact of noise in post-pros post-processin and last-layer retraining respectively. A similar trend is observed for DomainInd in the CXP dataset, although the effect is less clear for other model-based experiments, where performance frequently falls below the bascline across all noise levels (Figure A8).
eT (h) S eTT (h) + Drv Piv(PPuste (Ptrain Pusa)
where Div(..) denotes the total variation divergence.
We therefore choose to measure the divergence between P and Punbiused for each subgrouping see whether this is predictor of performance the unbiased distri- bution. We model the divergence as a Kullback-Leible divergence following prior work which give upper bounds (Aminian et al., 2024; Masiha et al., 2021; Wu et al., 2024; Nguyen et al., 2022) and lower bounds (Masiha et al., 2021) for expected generalisation error. In our case, Pinkser's
Random subgroups are generally detrimental. Across the four datasets and mitigation methods we also find that
7

---

Subgroups Matter for Robust Bias Mitigation
inequality implies that test error can be bounded as:
and gDRO respectively (Figures 5, F10). This aligns with our hypothesis that the extent which the unbiased distribu- tion can be restored during training significantly influences generalisation performance. Of course, other factors may still impact performance, such as inherent differences in task difficulty across subgroups (Petersen et al., 2023). We also note that we are only able to do this analysis by as- suming that any divergence between distributions is fully attributable to differences in P(Y,S, A) and because we have full knowledge of how these distributions change at test time. These assumptions would rarely be validated in practical settings. Despite this, our results suggest that as- sessing whether an unbiased distribution can be recovered provides a useful starting point for defining subgroups in bias mitigation. Thus, defining subgroups based on the cause of generalisation error may be more effective than simply defining subgroups based on observed disparities.
eTTunbiased (h) - erT (h) + V KL Puntiased)
Since all our models reach a similarly low train error, we posit that the differences in upper bound are largely driven by the divergence between both distributions'. We explore whether the divergences achieved for each subgrouping cor- relate with generalisation error.
We assume that the difference between both distributions is attributable to differences in probabilities of sampling each (Y, S, A) subgroup. Thus we represent each Pirain as proba- bility vectors in R., as defined in Section 3.2. This yields 0.527.
an initial KL divergence (L I| Punbiased) We next explore whether, for each possible subgrouping, by either resampling or gDRO, this divergence can be duced. For resampling, each subgroup in the chosen sub- grouping is resampled such that they are uniformly dis- tributed. For instance, resampling across (Y, S. A) would give Pu...n = [s. :.....;.. d.. For gDRO, the distribution of the subgroups is learned during training by determining what weights apply to each subgroup. We determine the theoretically optimal weights w by solving a convex optimisation problem:
These divergences also explain previous observations, such as the similarity of results when subgroup granularity in- creases (KL divergence is also unchanged), robustness to noise (KI. divergences show little change), and the similar- ity between results for gDRO and resampling (also similar divergences). Moreover, it is interesting to note that in- corporating S into the (A, Y) groups (to get (Y, (A)) is the most optimal grouping, as we observe empirically for MNIST (Figure 2). Although S is not involved in the (A, Y) spurious correlation and not a cause for poor generalisation performance, simply reweighting the four (A, Y) groups induces an unwanted correlation between S and Y. since the (A, Y) groups are imbalanced with respect to S (Table 1).
min KL irain WEAM
P Punbiisee))
where A denotes the probability simplex over the selected k subgroups. We present the divergences obtained for all groupings in Table 3 with full explanations and calculations provided in the Appendix E.
5.4. Subgroup choice also impacts disparities
Table Minimum KL(Peain cain P achievable by reweight- ing subgroups with gDRO and resampling.
In addition to overall generalisation performance, we ex- plore the impact of subgroup choice on disparities, specifi- cally between S subgroups (Table 4 and D9). Fairness re- sults largely align with overall generalisation performance. We make the surprising observation that the best results in terms of disparities for a given grouping are not necessar- ily achieved by using that grouping in the bias mitigation process. For example, (A, Y) subgroups show far better worst-group-accuracy and smaller accuracy gaps across S subgroups than when the S attribute is used for grouping (e.g.. SY, SYs) for both datasets and methods. The mini- mum accuracy is up to 0.21 higher and an accuracy gap up to 10 times smaller. This is because the (A, Y) spurious corre- lation is causing the disparity in performance across S, and it can only be unlearned with certain groupings. This finding contradicts the general assumption that in order to improve faimess with respect to certain subgroup, this subgroup should be used in the bias mitigation process (Mehrabi et al., 2021). To clarify, we do not advocate using different sub- groups for evaluation, but rather that alternative subgroup definitions in mitigation may better improve faimess.
KL divergence to Puntiased gDRO
Grouping
Resampling
0.527
0.527
0.527
0.527
0.527
0.527
AY
0.113
0.113
SY
0.527
0.527
YSA
0,000
0,000
SCIno-SC SC
0.113
0.113
AYs
0.113
0.113
SYs
0.527
0.527
Random
0.527
0.527
Noisy AYo.11
0.113
0.113
Noisy_Ai AYo..
0.113
0.114
Noisy_AYo.n AYo.1
0.113
0.116
Noisy AYo.25
0.114
0.131
Doisy_AYo.co 0.118
0.189
We observe a high correlation (between 0.73 and 0.97) be- tween the minimum achievable divergence and the perfor- mance of each model across the four datasets for resampling
We would ideally like to directly estimate generalisation under distribution shift (instead of just having an upper bound), hut this would require very strong assumptions (Chuang et al., 2020).
8

---

Subgroups Matter for Robust Bias Mitigation
CelebA gDRO
CelebA Resampling
-0.825, p 0.001
= -0.930,, 0.001
AUC on Punb-aoed
0.96
0.95
0. 0.94
0.93
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
KL divergence Punteesed
divergence Panzlesed
Figure 5. The ability to recover the unbiased test distribution (P (mbbiasedd is a significant predictor of overall generalisation performance on P Punbiased for gDRO and resampling. Each dot represents AUC the unbiased test for specific grouping, with error bars indicating the standard deviation across 3 random seeds. Pearson's 's correlation coefficients and associated p-values are also shown.
by comparing our results on the full 60K MNIST dataset to downsampled 7K version. We find largely similar trends, with an example shown in Figure G12.
Table 4. Worst group accuracy and accuracy gap across S groups, averaged gDRO and resampling. While some groupings lead improved faimess relative the baseline (green), others detrimental (red). Most groupings involving S have the opposite of their intended effect and decrease fairness with respect to S. We report mean and standard deviation across three random seeds.
5.6. Limitations and future work
This work is limited to a specific setting of bias i.e. spu- rious correlations, but many other types exist, including bias caused by differences in the manifestation of across subgroups, differences annotation of across subgroups, and under-representaaiin of certain subgroups. We hope that research provides solid foundation for further exploration into how subgroups should be defined for bias mitigation across a wide range of real bias settings. Another limitation is that, in practice, outside of (semi-)synthetic scenarios like ours, one often lacks extra annotations on other attributes, and not perfectly aware of what might be causing underperformance. While we advocate for data collectors gather much metadata possible to enable precise analyses, we recognise that this is not always fea- sible. Therefore, we encourage practitioners to carefully analyse their models' errors and thoughtfully investigate potential causes of underperformance before implementing bias mitigation strategies for specific subgroups.
Grouping
min Acc.
Acc. gap
baseline
0.705 + 0.025
0.064 1 0.045
A
0.696 + 0.019
0.074 1 0.034
0.696 + 0.022
0.077 0.037
0.694 + 0.024
0.072 + 0.039
AY
0.779 + 0.02
0.032
SY
0.699 = 0.016
0.026
YSA
0.792 1: 0.013
0.02 0.022
SC/o-SSC
0.772 +: 0.02
0.04 + 0.03
AYs
0.778 + 0.022
0.041 0.032
SYs
0.7 + 0.021
0.072 + 0.032
Random
0.693 = 0.02
0.075 + 0.035
Aoo.5_AYo.50
0.741 0.016
0.056 0.029
5.5. Subgroup choice is similarly impactful in other settings
Finally, conduct various additional experiments verify that our results hold in less restrictive settings. We im- plement tust-train-twice (JTT), method which does not require subgroup labels at training, but still requires some for model selection, and find that, again, its success is de- pendent on subgroup choice (Appendix C: Table C8 and Figure C9). We also repeat the MNIST experiments in setting where the spurious correlation is weaker such that overall there are 77.5% spuriously correlated samples (in- stead of 87.5%). We find that while all results are higher overall, the same trends still appear, as shown in Figure GI1. We also explore whether the relatively small size of the datasets we use (as we are constrained by the availability of each (Y, S, A) combination) might impact our findings
6. Conclusions
To our knowledge, this is the first work to specifically and comprehensively consider how subgroup definition can im- pact existing bias mitigation methods. We demonstrate the extent to which certain subgroup definitions can "make or break" bias mitigation methods, and provide an explanation as to why this occurs. We urge practitioners carefully consider possible causes of bias rather than indiscriminately applying bias mitigation techniques to any underperforming group. Our work enables more consistent and effective bias mitigation in real-world applications.

---

Subgroups Matter for Robust Bias Mitigation
Acknowledgements
ized odds postprocessing under imperfect group informa- tion. In Proceedings of the Twenty Third International Conference Artificial Intelligence and Statistics, pp.
A.A. is supported by the EPSRC grant number EP/S024093/1, the Centre for Doctoral Training SABS: R3, University of Oxford, and by GE Healthcare. C.J. is supported by Microsoft Research, EPSRC, and The Alan Turing Institute through Microsoft PhD scholarship and a Turing PhD enrichment award. B.G. received support from the Royal Academy of Engineering as part of his Khe- iron/RAEng Research Chair. The computational aspects of this research were supported by the Wellcome Trust Core Award Grant Number 203141/2/16/Z and the NIHR Ox- ford BRC. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the De- partment of Health. The authors would also like to thank the reviewers of this paper whose comments contributed to substantially improving the work.
1770-1780. PMLR, June 2020.
Bayasi, N., Fayyad, J., Bissoto, A., Hamarnch, G., and Garbi, R. BiasPruner: Debiased Continual Learning for Medical Image Classification. In MICCAI, 2024.
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., theory of learning from different domains. Machine Learning, 79:151-175, 2010. URL httt:::/www.sppinn.... muntent//$gg22066555555
Pereira, F., and Vaughan,
com/
Borkan, D., Dixon, I.., Sorensen, J., Thain, N., and Vasserman, L. Nuanced metrics for measuring unin- tended bias with real data for text classification. CoRR, Nts/1903.04561, 2019. URL httt:://taxii.....o 0ttp//903.04555
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work. none which we feel must be specifically highlighted here.
Buolamwini. J. and Gebru, Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classifica- tion. In Proceedings of the 1st Conference Fairness, Accountability and Transparency, pp. 77-91. PMLR, Jan- uary 2.
References
Ahn, S., Kim, S., and Yun, S.-Y. Mitigating dataset bias by using per-sample gradient. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Appli- cations, 2022. URL http:::///peeeevieeeee n
Chen, Z., Zhang, J. M., Sarro, F,, and Harman, M. A comprehensive empirical study of bias mitigation meth- ods for machine learning classifiers. ACM Trans. Softw. Eng. Methodol,, 32(4), May 2023. ISSN 1049-33IX. doi: 10.1145/3583561. URL https: / /ww. org/10. 1145/3583561.
Alloula, A., Mustafa, R., McGowan, D. R., and Papiež, B. W. On biases in a uk biobank- based retinal image classifica- Puyol-Antón, E., Zamzmi, G., Feragen, A., King, A. P., Cheplygina, V. Banz--enjammnsen, M., Ferrante, E., Glocker, B., Petersen, E., Baxter, J. S. H., Rekik, I., and Eagleson, R. (eds.), Ethics and Fairness in Medical Imaging, pp. 140-150, Cham, 2024. Springer Nature Switzerland. ISBN 978-3-031-72787-4.
Chuang, C.-Y., Torralba, A., and Jegelka, S. Estimating gen- eralization under distribution shifts via coominn-nnvarinn representations. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020.
tion model.
Deng. J., Dong, W., Socher, R., Li, L.JJ., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vi- sion and Pattern Recognition, pp. 248-255, 2009. doi: 10.1109/CVPR.200
Aminian, G., Masiha, S., Toni, L., and Rodrigues, M. D. Learning algorithm generalization error bounds via auxil- iary distributions, 2024. htw.https://aaee org/ abs/ / 2210 00483.
Anthony, H. and Kamnitsas, K. On the use of mahalanobis distance for out-of-distribution detection with neural net- works for medical imaging. In Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, pp. 136-146. Springer Nature Switzerland, 2023, doi: 10. 00.10077/978-3-31444333 14. URL /tatttss://doet org/10 1007820978----8 3- -331-40 031-44336 11433366-144
Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for lan- guage understanding. CoRR, abs/1810.040000 2018. URL http://azxzs /rg/ abs 1810. 04805.
Dong, Y., Gong, T., Chen, H., Song, S., Zhang, W., and Li, C. How does distribution matching help domain generalization: An information-theoreet analysis, 2024. URL htLhttts:// peitts://arxiv..
Awasthi, P., Kleindessner, M., and Morgenstern, Equal-
httt:p::/arxxv..rrrrab///// wwt./oog/////ooooooooe 09745.
10

---

Subgroups Matter for Robust Bias Mitigation
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS '2, pp. 214-226, New York, NY, USA, January 2012. Association for Computing Machinery. ISBN 978- 1-4503-1115-1. doi: 10.1 10.1144/2990336.200000 200000
Sandberg, J. K., Jones, R., Larson, D. B., Langlotz, C. P., Patel, B. N., Lungren, M. P., and Ng, A. Y. Chexpert: large chest radiograph dataset with un- certainty labels and expert comparison. In Proceed- ings of the Thirty-Third AAAI Conference on Artifi- cial Intelligence and Thirty-First Innovative Applica- tions of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artifi- cial Intelligence, Press, 2019. ISBN 978-1-577355-809-1 doi: 10.1609/ aaai. mmai.v33i01..0 033310013311990 URL. http::::/dooooooo/o. 1 1609/ aaai wtt.aaai..a.a..... ww3...............3.3....... 3301590.
Ganin, Y. and Lempitsky, V. Unsupervised domain adapta- tion by backpropagation. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Ma- chine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1180-1189, Lille, France, 07- 09 Jul 2015. PMLR. URL hitps: wttpa/ttss//ppceeeeeeee httpp//roeeenn..... mlr press / v37 gani m15 html.
1AAT119/ATT111/AAA
AAAI
Izmailov, P., Kirichenko, P., Gruver, N., and Wilson, A. G. On feature learning in the presence of spurious correla- tions. In Proceedings of the 36th International Confer- ence on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088
Ghosal, S. S. and Li, Y. Distributionally Robust Opti- mization with Probabilistic Group. Proceedings of the AAAI Conference on Artificial Intelligence, 37(10):11809 11817, June 2023. ISSN 2374-3468. doi: 10. 10.1609/aa.. v37910.26394.
Jain, S., Hamidieh, K., Georgiev, K., Ilyas, A., Ghassemi, M., and Madry, A. Improving subgroup robustness via data selection. Globerson, A., Mackey, L.,, Belgrave, D.,, Fan, A., Paquet. U., Tomezak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, vol- Curran Associates, Inc., 2024.
Han, Y. and Zou, D. Improving group robustness on spuri- ous correlation requires preciser group inference. In Forty- first International Conference on Machine Learning, 2024. URL https: / openrevi wwt. forum? id Kycvyygg.....
ume 37, pp. 11490-94511.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Jones, C., Castro, D. C., De Sousa Ribeiro, F,, Oktay, O., McCradden, M., and Glocker, B. A causal perspective on dataset bias in machine learning for medical imaging. Na- ture Machine Intelligence, 6(2):138-146, February 2024. ISSN 2522-5839. doi: 000108//42225--02-00777779
Johnson, U.. Kim, M.. Reingold, O., and Roth- blum, G. Multicalibration: Calibration for the (Compuationally-idennfffff
Jones, C., Ribeiro, F. d. S., Roschewitz, M., Castro, D. C.,, and Glocker, B. Rethinking fair representation learn- ing for surformance-senssinn International Conference on Learning Representations, 2025. URL https:/ /wwp//ppewweeweeeee/oeooeoo net/ forum? id=pB2ntPrdrI.
masses. In Dy, and Krause, A. (eds.), Proceedings of the 35th In- ternational Conference on Machine Learning, vol- ume 80 of Proceedings of Machine Learning Re- search, pp. 1939-1948. PMLR, 10-15 Jul 2018. URL https::p///// httttps://proceedingss mlr, www..rrresss......m... press/v80/ bebert-jjhnsooe Ba html.
tasks. In The Thirteenth
Kearns, M., Neel, S., Roth, A., and Wu, Z. S. Prevent- ing Fairness Gerrymandering: Auditing and Learning for Subgroup Faimess. In Proceedings of the 35th Interna- tional Conference on Machine Learning, pp. PMLR, July 2018.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261-2261 2017. doi: 10.1 2VPR.2017.243.
Kim, N., Kang, J., Ahn, S., Ok, J., and Kwak, S. Im- proving Robustness to Multiple Spurious Correlations by Multi-Objective Optimization. In Proceedings of the s1 International Conference on Machine Learning, pp. 24040-24058. PMLR, July 2024.
Idrissi, B. Y., Arjovsky, M., Pezeshki, M., and Lopez-Paz, D. Simple data balancing achieves competitive worst- group-aceuracy. In Proceedings of the First Conference on Causal Learning and Reasoning, pp. 336-351. PMLR, June 2022.
Kirichenko, P., Izmailov, P., and Wilson, A. G. Last layer re- re-training is sufficient for robustness to spurious corre- lations. In ICLR, 2023. URL https: ptttspss///rxiii /tt./iaivv..//// orgg/ abs/2204 029
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Sh- panskaya, K., Seekins, J., Mong, D. A., Halabi, S.,

---

Subgroups Matter for Robust Bias Mitigation
Krishnakumar, A., Prabhu, V., Sudhakar, s., and Hoffman, J. Udis: Unsupervised discov- ery of bias in deep visual recognition mod- els. In BMVC, Pp. 143, 2021. URL https: www.:bbmc22221vii. assets/papers/e pdf.
Masiha, M. S., Gohari, A., Yassaee, M. H., and Aref,
M. R. Learning under distribution mismatch and model misspecification. CoRR, abs/2102.05695, 2021. URL
https///arsiv.orr//aaaa
conference..com
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., and
Galstyan, A. A survey on bias and fairness in machine learning. ACM Comput. Surv. 54(6), July 2021. ISSN 0360-0300. doi: 10.1145/3457607. URL https:// doi 2tt..o/./...11//4557777..
Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 86(1 (6(11):222 11:2278--22 23244, November 1998. ISSN 2558-2256.. doi: 10.1109/5.72679..
Movva, R., Shanmugam, D., Hou, K., Pathak, P., Guttag, J., Garg, N., and Pierson, E. Coarse race data conceals disparities in clinical risk score performance. In Ma- chine Learning for Healthcare Conference, pp. 443-472. PMLR, 2023.
Li, Z., Evtimov, I., Gordo, A., Hazirbas, C., Hassner, T., Ferrer, C. C., Xu, C., and Ibrahim, M. A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 20071 20071-20082, June 2023. 2023.01922.
Nguyen, A. T., Tran, T., Gal, Y.. Torr. P. H. S., and Baydin, A. G. KI guided domain adaptation, 2022. URL https: / //arxiv. org. haagg//a////// 210.. 07780.
doi: 10.1 109/CVP52222..
Liu, E. Z., Haghgoo, B., Chen, A. S., Raghunathan, A., Koh, W., Sagawa, S., Liang, P., and Finn, C. Just train twice: Improving group robustness without training group information. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6781-6792. PMLR, 18-24 Jul 2021. URL https: /pproceeiinggssseeeeeeeeeee...se www.mmrr.sss..ee press/ v139/1iu210 html.
Olesen, V., Weng, N., Feragen, A., and Petersen, E. Slic- ing Through Bias: Explaining Performance Gaps in Medical Image Analysis Using Slice Discovery Meth- ods, pp. 3-13. Springer Nature Switzerland, Octo- ber 2024. ISBN 9783031727870.
doi: 10.1007/ 978-3-031--27887-011 URL http: /ttp//xxooo////////////oo woi.ooooooooo orr// 10. N0.1007/978
72787-77777-0-
Park, G. Y., Jung, C., Lee, S., Ye, J. C., and Lee, S. W. Self-supervised debiasing using low rank regulariza- tion. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. doi: 10448550/ARXI.22100055
Liu, Z. Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of Interna- tional Conference on Computer Vision (ICCV), December 2015a.
Petersen, E., aPETERSEN20231 S. H., and Feragen, A. The path toward equal performance in medical machine learning. Pat- terns, 4(7):100790, 2023. ISSN 2666-3899. doi: http::/oo...g///00000//ppe..e..2 URL science/ itteene//rrtaae///////6682222229
Melanie Ganz,
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learn- ing face attributes in the wild. In Proceedings of the 2015 IEEE International Conference on Computer Vi- sion (ICCV), ICCV 11, pp. 3730-3738, USA, 2015b. IEEE Computer Society. ISBN 9781467383912. doi: 10. 10.1109/1CCV22015..2. URL httL:::///do...... /ttp:/oo..oor//o//
Pezeshki, M., Kaba, S.-O., S.-O.,,eiii,,,,, Y. Courville, Precup, D., and Lajoie, G. Gradient Starvation: A Learning Proclivity in Neural Networks, November 2021.
Mansour, Y., Mohri, M., and Rostamizadeh, A. Domain adaptation: Learning bounds and algorithms. CORR, abs/0900.33430, 2009. URL http: http:://axxivorg/ abs/0902.0m 3430.
Pezeshki, M., Bouchacourt, D., Ibrahim, M., Ballas, N., Vincent, P., and Lopez-Paz, D. Discovering environ- ments with XRM. In ICML, 2024. URL https: //openreview. net/ forummrmmmm I...IIIIIIIIIIoo..
Marani, B.-E,, Hanini, M., Malayarukil, N., Christodoulidis, S., Vakalopoulou, M., and Ferrante, E. ViG-Bias: Visually Grounded Bias Discovery and Miligation, pp. 414-429, Springer Nature Switzerland, Novem- ber 2024. ISBN 9783031732027.
Ricci Lara, M. A., Echeveste, R., and Ferrante, E. Address- ing fairness in artificial intelligence for medical imaging. Nature Communications, 13(1):4581, August 2022. ISSN 2041-1723. doi: 10.1038/s4 467-022-321866
doi: 10.1007/ URL http: /xx.dd.. 0rg/10.1007/978-3-000
778-3-031-73202--224
12

---

Subgroups Matter for Robust Bias Mitigation
Sagawa", S., Koh*, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks. In In- ternational Conference on Learning Representations, 2020. URL http:::/ httprreviiineee//orreeeeeeee id=ryxGuJrFvS.
Wu, X., Manton, J. H., Aickelin, U., and Zhu, J. On the gen- eralization for transfer learning: An information-thhorriic analysis, 2024. URL https:/ papps//aarae /ttp:a/axvvvvv//// orgg abs/ 2207.05377.
Xu, G., Chen, Q., Ling, C., Wang, B., and Shui, C. Inter- sectional Unfairness Discovery. In Proceedings of the 41st International Conference on Machine Learning, pp. 54888-5499 PMLR, July 2024.
Schrouff, J., Bellot, A., Rannen-Triki A., Malek, A., Albu- querque, I., Gretton, A., D' AAmour, A. N., and Chiappa. S. Mind the graph when balancing data for fairness or robust- ness. The eightt Annual Conference on Neural Information Processing Systems, 2024. URL. https: / //openreview net /ott/fooe forum? 10R222221151..
Zhao, H., Coston, A., Adel, T., and Gordon, G. J. Conditional learning of fair representations. In In- ternational Conference Learning Representations, 2020. URL https: /oppnrevieeeeeeeeeeeeeeeeeeeeeee net /orum??rm @d-Hkek1ONFPA
Shrestha, R., Kafle, K., and Kanan, C. An Investigation of Critical Issues in Bias Mitigation Techniques. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2523,2523, Waikoloa, HI, USA, January 2022. IEEE. ISBN 978-1-660 978-1-66540-915-5. doi: RAL100/WACVS1448.22220000
Zhou, C., Ma, X., Michel, and Neubig, G. Examining and Combating Spurious Features under Distribution Shift. In Proceedings of the 38th International Conference on Machine Learning, pp. 12857-12867. PMLR, July 2021.
Stromberg, N., Ayyagari, R., Welfert, M., Koyejo, S., Nock, R., and Sankar, L. For robust worst-group accuracy, ig- nore group annotations. Transactions on Machine Learn- URL https:
Zietlow, D., Lohaus, M., Balakrishnan, G., Klein- dessner, M., Locatello, F. Scholkopf, B., and Russell, C. Leveling down in computer vi- sion: Pareto inefficiencies in fair deep classifiers. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10400-10411, 2022. URL https:://ap semant cschola.......... org/ [eg//ooppuaDD:24319923
ing Research, 2024. ISSN 2835-8856. / /ppeereviie net / forum? dew.:mmmmdd1111111711111
Wang, S., Guo, W., Narasimhan, H., Cotter, A., Gupta, M., and Jordan, M. Robust Optimization for Fairness with Noisy Protected Groups. In Advances in Neural Information Processing Systems, volume 33, pp. 5190- 5203. Curran Associates, Inc., 2020a.
Zong, Y., Yang, Y., and Hospedales, Medfair: Bench- marking faimesss for medical imaging. In International Conference on Learning Representations (ICLR), 2023.
Wang, X., Saxon, M., Li, J., Zhang, H., Zhang, K., and Wang, W. Y. Causal balaneing for domain generalization. In ICLR, 2023. URL https: /tt:://aaxiii wawii.iiivvoo.o org/ abs/ 2206. 05263.
Wang, 7., Qinami, K., Karakozis, C.,, Genova, K., Nair, P., Hata, K., and Russakovsky, Towards Fairness in Effective Strategies for Bias Mitiga- tion. In 2020 JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 8916-8925, Seattle, WA, USA, June 2020b. IEEE. ISBN 978-1-72817-166-5. doi: 10. WWWp:CCCVPR222022200000000
Visual Recognition:
Weng, N., Bigdeli, S., Petersen, E., and Feragen, A. Are sex- based physiological differences the cause of gender bias for chest x-ray diagnosis? In Clinical Image-Basee Proce- dures, Fairness of Al Medical Imaging, and Ethical and Philosophical Issues in Medical Imaging: 12th Interna- tional Workshop, CLIP 2023 1st International Workshop, FAIMI 2023 and 2nd International Workshop. EPIMI 2023 Vancouver, BC, Canada, October 8 and October 12, 2023 Proceedings, 142-152, Berlin, Heidelberg, 2023. Springer- Verlag ISBN 978-3 978-3-031-4933 3-031-45248-2. doi: 10.1007/9788.3-031-4449999099 URL https: org/10.1007/978-3-00
13

---

Subgroups Matter for Robust Bias Mitigation
This appendix provides additional details and experiments that support the main text. It is structured follows:
A Supplementary experimental details the datasets used, subgroups constructed, and model implementations.
B Mitigation results for more bias mitigation methods: DomainInd and CFair.
C Mitigation without subgroup labels (Just Train Twice).
D Supplementary results for gDRO and resampling.
E Explaining results through the divergence between P and P uunhissed-
F Correlation between KL divergence to the unbiased distribution and unbiased generalisation across all four datasets.
G Ablations on strength of SC and size of dataset.
A. Supplementary experimental details
A.1. Dataset details
Table 5. Details on the datasets used mitigation experiments.
Dataset
MNIST
CheXPert
CelebA
Civil comments
Even/odd digit
Pleural effusion
Blonde hair
Toxicity
A
Background colour Foreground colour
Presence of a pacemaker Perceived gender Smiling 12500
Gender
Sex
Religion
Dataset size
60000
3225
8900
We downsample some of the datasets from their original size because we are constrained by the availability each (Y, 8, A) combination. For example, for CheXPert, pacemaker annotations only available for 4862 images, and we have to further downsample the dataset to make it balanced with respect to disease (Y) and sex (S).
A.2. Subgroup construction
Logend:
S
P train
S-1
Summary subgroups constructed.
Subgrouping
n. subgroups
gDRO and resumpling DomainInd
CFair
S
2
/
2
/
x
SC
/
Noisy A
2
x
(A, Y)
(S, Y)
/
(A, ()
/
X
Noisy AY
4
x
/
X
S4
x
X
nolee
noldo
Random
AYa
8
/
X
SYa
x /
(Y)
X
Figure 6. Subgroup construction for our experiments.
14

---

Subgroups Matter for Robust Bias Mitigation
We show the subgroups used for each method and visualisation of some example subgroups in Figure 6. For model-based methods (DomainInd and CFair), we do not use Y to construct subgroups because for these methods to work best, each subgroup should contain both positive and negative classes. This is because methods like DomainInd and CFair learn representations for each subgroup separately. DomainInd trains a separate classifier for each subgroup, so it would not make sense to train separate classification head for positive and negative classes. Similarly, CFair seeks to align subgroup representations, so it would not make sense to align representations of one subgroup containing only positive images to another subgroup containing only negative images, as this would defeat the point of training discriminative classifier. On the other hand, for reweighting based methods, including the Y in the subgroups helps to balance the final reweighted dataset with respect to class, and therefore improves results, especially in case where the spurious correlation involves the class Y. This explains why we find that the subgroups which work well for DomainInd and CFair (e.g., A) are just merged version of the ones which work well for gDRO and resampling (e.g., (A, Y)). To the best of our knowledge, no papers have explicitly discussed this distinction despite its practical importance.
A.3. Implementation details
Table 7. Implementation details for all models.
Training strategy
MNNST
ColebA
Civilromment
2-byer
21 OMang 2017
ResNer50 2016)
BEITCClaciher (incased)
Pre-training
Note
ImageNet (Deng et all, 2009)
ImageNer (Deng 2009)
Bookcorpus, Wikipedia (Gnglish)
Imoage size
3x28s28
3x209229
Angmratation
Flip, mitaioon, Cunsion ble
Flip, mistion, oolor affine transfommatonn cxp
Flip. retties, color jaser, affine tran/formation, crep
Nore
Optimisrr
Adum
Adam
Adam
Admm
Loss
Donary emo--entmpy
Binrry crow--entrpy
Binary crosp-enropp
Bintry emes-enmuyy
Leurning rale
0.001
0,0005
0.001
0,00005
Louringg scheduler
SteplR (y - and 10)
Supl.R R my-m 0.1 y- 10)
Supl. - 0.1 and y= 10)
SupplR (yy m 0.1 and 10)
Wright docay
0.0001
0.0001
0.0001
0.0001
Max spechs
50
100 stpping after 10)
10 jearly stopping after $)
10 (early stopping afte
We conducted hyperparameter tuning on the baseline ERM model within the ranges below, and selected the model with the highest validation AUC. The choice of backbones based their strong performance in previous similar work (Irvin et al., 2019; Jain et al., 2024; Izmailov et al., 2022; Kirichenko et al., 2023; Idrissi et al., 2022).
Backbones for vision models: ResNer18, ResNet50, DenseNet121 (not for MNIST images)
Batch size: 32, 64, 128, 256, 512
Learning rate: [le-5:le-]]
Weight decay: [le-5:le-4]
We also specify additional hyperparameters for the mitigation methods: a step size of 0.01 and a size adjustment factor of I was used for gDRO, and μ coefficient of 0.1 was used for the adversarial loss of CFair, following the MEDFAIR implementation (Zong et al., 2023).
15

---

Subgroups Matter for Robust Bias Mitigation
B. Results for more mitigation methods: DomainInd and CFair
Overall, CFair and DomainInd show less improvement on the unbiased test set than reweighting based-methods (Figure 7), Despite this, we still observe similar trends as for the reweighting based methods, such as S-related groups being clearly detrimental performance. Learning independent models for A groups also boosts performance for DomainInd in CXP, CelebA, and Commmens, and while it does not significantly change performance relative to the baseline in MNIST, it is still higher than for any other grouping. Moreover, shown in Figure 8, DomainInd appears sensitive to even low levels of noise, while CFair's 's performance is less degraded by noise in the A subgroup labels (except for in MNIST, where both methods are ineffective, as performance stays close to the baseline for all subgroupings).
00:0
-000
040
-120
Figure 7. Relative performance Punbinsed for different groupings in DomainInd and CFair across all four datasets. Similar trends gDRO and resampling can be seen, where subgroupings constructed around A generally improve performance as they prevent the SC from being learnt, while other subgroups are generally detrimental. Error bars indicate the standard deviation across random seeds.
PNST
CSP
Ohe
1350
6.3-
0.50
6.4
0.00
0.75
Cher
(0)
db
comments
0330
1.3%
E12-
2.4
Figure 8. Effect noise in A subgroup labels on AUC for DomainInd and CFair. Each dot represents mean performance on the unbiased test set for specific grouping, with error bars indicating the standard deviation across 3 random seeds.
16

---

Subgroups Matter for Robust Bias Mitigation
C. Subgroup discovery methods
We implement Just Train Twice (JTT) as proposed by Liu et al. (2021). JTT consists a two-stage process, where first a standard ERM model is trained for several epochs, and then second model that upweights the training examples that the first model misclassified trained. Although it does not require subgroup labels for training, to select a final model to use (e.g., based the -pT-scccific hyperparameters), subgroup labels do need be used.
As shown in Table 8 and Figure 9, we find that with validation subgroup labels to guide model and hyperparameter selection JTT performs mostly on par with our other methods, however, performance is again highly dependent on the choice of subgroups. When no subgroup annotations are used (i.e. model selection is done by overall validation accuracy), the method does not improve over ERM (except for on MNIST where JTT works remarkably effectively, most likely due the simplicity of the task).
Table Just train twice generalisation performance unbiased test across the four datasets is highly variable depending the validation set subgroups used for model/hyper-paramede selection. We colour the experiments which improve over the baseline (no mitigation) in green and those do in red. We report AUC and standard deviation three random seeds.
Subgroup
MNIST
CXP
CelebA
civil comments
Baseline
0.792 0.057
0.740 + 0.002
0.943 + 0.002
0.805 0.017
SY
0.89 = 0.002
0.791 = 0.009
0.005
= 0.005
0.786 + 0.028
AY
0.925 + 0.019
0.791 0.009
0.943
0.831 0.004
0.89 = 0.002
0.695 0.018
0.948
0.786 0.011
AY_8
0.919 0.012
0.701
0.943 0.012
0.812 - 0.021
S
0.89 = 0.002
0.009
0.948 + 0.003
0.003
0.786 + 0.028
SY.8
0.89 0.002
0.734
+
0.786 0.028
Y
0
0.734
0.947 0.005
0.776 0.019
Noisy AY 0.5
0.919 0.012
0.705 + 0.042
0 0.004
0.831 + 0.004
Random
0.925 = 0.019
0.734 + 0.035
0.948 + 0.003
0.786 0.028
SC/no-SC
0.936 0.009
0.791 = 0.009
0.831 0.004
YSA
0.922 0.006
0.682 = 0.034
0.943 = 0.012
0.831 - 0.004
No val subgroups
0.925 # 0.019
0.734 + 0.035
0.948 + 0.003
0.786 + 0.028
Figure Performance on the unbiased set gDRO resampling and JTT highly dependent on the subgroups used. Bars represent overall change in AUC relative to the ERM baseline, with error bars indicating the standard deviation across random seeds.
17

---

Subgroups Matter for Robust Bias Mitigation
D. Supplementary results for resampling and gDRO
Table 9. Worst group accuracy and accuracy gap across 5 groups for gDRO and resampling. Some groupings improve faimess relative to baseline. while others decrease worst-
0.0 GIDS
0.022 SQUI
20.00
0.001
5001
0.012 0008
= 0006
0.022 0005
0.00 =002
0,005 205:0000 0.07
0005S 2052000 ARR
0:01 samm
0:0 200s
0,055 a0i0s
0021 aniss
Recmpling
Arc.
0.001
0.0
0.00
0:01
0.02
0.01
0.006 0.0
0009
0016
0.7
0015
0:01
0:33
6 0016
6.74 001
0.86 0622
0 d 0022
6TH 1004
069 0028
0.85 0
6PH 10011
Chill cel.covaall
0.72
0,723
0,80
0.85
0.01
0
6.74
0.86
(L0 (MMN
0UIS 0004
0.011 4 6006
0 30M
DUSS 085-2000 011
0.07 4 0045
0007 A 4004
0.0S 0008-1000 10/2
DUILS 1004
0 100
BU0S ROS5 6000
(007) RIMS
QUIS GOOS
4 6
(UM GOW
0,000
B0RO
0726 0/01
0.16 0.00
0.009
0.7210002 0.001
0.34 0001
0.72 +000
000003
0517=0000
0.21
=0015
0.705
00000
00000
0.255
0.38 0011
0.788 0000
0000
0001Z
0.9
mid
0.79
0724
0.21
0217
0.21
0.755
0.255
0.33
0.70
0.09
0.79
6:79
group accuracy and increase the gop bet the best- and worst worst-grool group. Mean and standard deviation across three random seeds are shown.
0.0%9 0.006
0:09
0:35
0000
0009
0
#800
0
6004400000
8:00
0 83
801
0
000
B60 0000
0/228
0,:99
0.08
0/09
0,233
00.3
0,27
0,024
0017
0.14
0013
00W
0015
0014
Resampog
0102
1003
0X7] 0002
0. 0
0,00 1001
0,00 0.00 GAIE
0,922 6 AAISS
0,508 1 100L
QURIS AIMI
0 1AS78-000 2001
0.597 GOIS
ONN ROM
0,57 2
min
0.00
0,002
0.M
0.564
Cdcba
0.004
0.005
0Z00Z
0000
0.004 0003
2009
0.09 0000
0.02 =001
0007 5 0004
00:3=00 0001
0006 0000
0A2I HOMIS
000 ZUI
F00
0085 HAIN
Arc.
0.099
0.01
0.08
0.088
0.01
0.044
0:092
0.09
0002
0.03
0077
8DRO
mis Arc.
0001
0.998
(9 0001
0.03 190320022 #022
0.63 0003200000
1000
20226
0.355 0153
(U00) 0.00 20000
10000
1000
(L096 20000
2000
(199 ION
(.06)
0357
0.898
(.00
0,099
0.897
0.89
(MD1 0405
1 0.02
0.05 0002
0:02
009-40000
010 0002
0.010005
0.033
0045 0000
OUTS 0.09
QUOI 0U0L 0404
0U02 0.N
000 00010.01 0O1
QUON 0621
(UNO GON
QU02 10020000 00M4
0L0E OUT
(IS 1006
Resmpling
Acc.
0100
0,099
000
0633 0:000
01024 0:000
0 00:03
00000
00:05
0611 1-0.00
0.82 0,012
0.5 TUASE
0654 10013
0.SE 0.00
OOS 1OOI
0G21 2O00
0658 SOOR
0873 1000
min
0,55
0429
0.05
0.83
0654
CXp
(126
003
0102
LU22
0,002 (1030
0.02 002
0.044 0029
0 ROIS
0.09 0022
0,607 &IOI GL0SS 1000
0.071 1U09
0,06 0002
0.88 AUNS 0,05 10D
0
0.053
0.05
0.053
0.077
0.00
0,033
mid MARO
10000-000 AVC
0.50550000
0129
0.08
0007
0
0,025
0106
0.60
0.N AURS
0 QUIS
0,02 AOE
0 P.W:A00 AZ2
0AW AMD
0.81 1 0I04
0.03
0.645
0.00
0,22
0,075
0,60
0.68
0.02
19 6..VV
118 008
1185=000 00855
00
=0/
0.87 1.7=000 00
0185 4185=0011 2041
0.99 A199 #08
0193 1033=0U FO4ST
0084 103T
0148 208
1699 2002
0143 FQUIT
LISS A185 20
0082 2055
NOG 008
(195=000
Resamping mammpinng
0.85
0155
(195
(.99 0.060
0.063 50.09
+0,55
50.023
$009
59025
0.06
1001
SOOLS
OL7 2004
07244
0.578
0131
0.515
0.578
(U35
4U39
0.06
03844
AND
MNIST
mis
0.113
22:0000000
0.04
0.022 0.03
0.39 +
0000
0
0027
0.22
0.31
2002
0002
0 0G2
061 008
QU42 4M200000 OGSS
Q11 GON
0.95 2 0037
0616 0.8
0414 0ON
Azc.
0,20
0.22
061
0,31
0,09
0035
0641
Arc.
(1999 (0
01574 10/33
0.22 2
0
015 0001
012 0:00
0.575 0
0.56 10/05
20019
1271=200
20mS
0288 1001
0D1
20118
BO0S 10/2
1L57 2012
0250 10/44
UNT OORO
01577
0.5
02544
mis
Noley MaieyAYoo. AYo
Nuio AYos
Nois AYos
None AYas
Subgmapp
Bateine
Rundom
SC
AY
Noley
AY'
SY,
YSA
18

---

Subgroups Matter for Robust Bias Mitigation
E. KL divergence between Ptr train and P
unbiased
E.1. Method overview
Our objective is to measure the minimum KL divergence which can achieved to Punbiased by partitioning Purin into subgroups and re-weighting these subgroups.
We define each distribution probability vectors in IR with each element corresponding the probability of sampling
one (Y, 8, A) subgroup. The unbiased distribution is defined to be uniform, i.e., Punbiased 8 ..... { 0.2
while Pirin
10.55
0.05
0.8
0.05
0.96
0.2
J.
T
1
I
Initially, KT((P (Pminn || P Punhiused)
R 0.527. Our aim is to see whether different subgroupings can reduce this divergence.
Let 9 = .G....., G be a partition the 8 atomic subgroups into k disjoint groups. For a set of weights w [w.,..., w/] E A over these groups, we define new weighted distribution train R follows:
Perain Li]
P [i] Wi
for j E Gi.
LIEG.
Pirai
Let the atomic subgroup indices correspond to S, A) combinations order [0, 0,,,, 4,5, 6, 7]. For the subgroups we constructed, we therefore have:
Y: (4,5,677)
A: (H0,2,4.00,,
S: (10,1,4,,),
(A,Y): (1.22),(1.3) (5.7))
(S,Y): ([0,1),
(Y) (0), (1), (2), (4), (6)
SCC: 2,0, (,. (1,3,4,6)
Random: (,0,1,2,3,4,5,6,,,)
AYg: [[0,2)
SY;: ((0.1) (0. 1), (2, (2,3), (4,5), (4.5), (6. (6.7),(6))))) (6,7)}
For the Noisy AY subgroups, have the same construction the (A,Y) subgroups, except that b fraction of the (A,Y) subgroup annotations are misannotated following the original Perain distribution, while the remaining (1 b) are consistent with the original (4, Y) subgroups.
Figure provides an illustration of some of these subgroups.
E.2. Resampling
For resampling, these weights are uniformly distributed such that w = [L/k, 1/k] E A
We can therefore calculate
Pw for each grouping keeping the relative proportions of the (Y,5, combinations constant within a group. For
example, for Y. the two groups have probabilities which sum to Pc; Pc2 0.95
0.us
+ 0.8
D2 = }. so the
4
0.26 0.06
0.8
0.2
2.8
2.5
02
0.3
relative proportions within the two subgroups are Pa, Po,
Po, 1. By multiplying these
Po, Pa. Pa, Paz
Paz
probabilities by w = 12 21, we get Pm = Purun. so (L
train |l
Puntiised) z 0.527.
proceed this way for all subgroups, and obtain the divergences detailed in Table 3.
19

---

Subgroups Matter for Robust Bias Mitigation
E.3. gDRO
For gDRO, the groups are the same but the weights are learned during training. Therefore we determine the weights which could in theory be achieved to give the lowest KL divergence ?. To do this, reframe the problem convex optimisation problem where minimize the following objective:
min EA*
KL (Pimin train
Punbiased)
subject to Wy 0, w 1.
For each subgrouping, we calculate the relative probabilities within a subgroup (as for resampling) and then use scipy. . opt imi ze. minimize and scipy. wwp...pppiaa...................p entr to determine the optimal weight vector subject to the constraints above.
For example, for (A, Y) subgroups, Pc, PG. 0.95
and Pc PGs 0.05
0.25 so the relative
proportions within the two subgroups are Pa. Po, Pe.
Pa. Pa. J. Minimisation gives
[L 4: 1: d. yielding Pur.in [0. (3.136, 0.050, 0.114, 0.200, 0.050, 0.136, 0.200, 0.11 and (L. Punbiased) 0.113.
Complete results for all groupings are presented in Table 3. We often find that these weights correspond those used for resampling.
"We note that these weights may not necessarily be attained in practice by all gDRO models because they are not specifically trained with this objective (although in our setting, minimising KL divergence to the unbiased test set should be a reasonable proxy for minimising worst-group loss). Also, stochasticity in training, optimisation challenges, and inherent difficulties in the task across subgroups may also affect the attainment of this optimum. Despite this, we believe doing this calculation still provides important indication of the potential effectiveness a chosen subgrouping.
20

---

Subgroups Matter for Robust Bias Mitigation
F. Correlation between KL divergence and unbiased generalisation
MNIITT
MNIST Resampling
1.000
0.001
-0.73.
0.95
AUC Peeasoo
0.50
0.925
0.000
0.875
0.850
CXP gDRO
CXP Resampling
0.825
R 0.734, < 0.01
0.876, 0.001
0.800
AUC Pushpea
0.75
0.750
0.725
0.700
0.75
0650
CelebA gDRO
CelebA Reesamp ing
0.57
0.001
-0.930,
AVC on Portion
0.98
0.95
CiviiComments gDRO
CivlCComments Resampling
90
R -0.942, 0.001
R -0.857, 0.001
0.88
6.84
AUC on
0,80
0.76
01
0.2 divergence
o'a divergence Portuned
0.2
Purtnsse
Figure 10. Test AUC is highly correlated with the minimum achievable KL divergence between Pi and Punbissed across all four datasets for gDRO and resampling in CXP. Each dot represents mean performance the unbiased set for a specific grouping, with error bars indicating the standard deviation across 3 random seeds.
21

---

Subgroups Matter for Robust Bias Mitigation
G. Various ablations
G.I. MNIST results with a weaker spurious correlation
To verify that our results still hold in settings where the spurious correlation is weaker, we re-generate the MNIST dataset in the exact same way, except that P(Y 0, = 0 = 0) P(Y 1, = 1 S = 0) 0.85 and P(Y = A = 0 8 = 0) P(Y = A, - - 0.70, such that overall there are 77.5% spuriously correlated samples, instead of 87.5%. We repeat the same experiments and find that, while all results are higher overall, the same trends still appear. Notably, we identify significant correlation between the minimum achievable KL divergence to P Penbiased and the overall performance on P Punbiased, as shown in Figure GI 1. This suggests that subgroup choice is an important factor in less extreme settings of bias as well.
MNIST with weaker SC gDRO
MNIST with weaker SC Resampling
R - -0.967. 0.001
R -0.987, p = 0.001
0.92
AUC on Punbared
0.91
0.89
0.88
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
KL divergence Punsissed
KL divergence Pancissed
Figure 11. Relationship between AUC and the minimum achievable distance to the unbiased test distribution (P ((ambiosedd) for gDRO and resampling in MNIST with weaker spurious correlation. Each dot represents mean performance on the unbiased test set for a specific grouping, with error bars indicating the standard deviation across 3 random seeds.
G.2. MNIST experiments with smaller dataset
ST eDRO
MNIST Resampling
0 801, p O 003
= -0 973. 003
0.315
0.030
0,975
AUC
0.300
0.050
ST gORD
MNIST Resampling
0.825
0.969, p 0.001
0.987, p 001
0.800
0.830
0.825
AuC
0.715
0.750
075
KL divergence
divergence to
Figure 12. Relationship between AUC and the minimum achievable KL divergence to the unbiased test distribution (P (Pundiased) for gDRO and resampling in MNIST with a downsampled dataset. Trends appear similar across both dataset sizes, suggesting that the results on the other three datasets would hold had we been able use larger subgroup-annotater dataset. Each dot represents mean performance on the unbiased test set for a specific grouping. with error bars indicating the standard deviation across 3 random seeds.
22


Reviews:
[{"review_id": "TZ1tSz8Soy", "reviewer": "Reviewer_86pJ", "summary": "The paper investigates how subgroup definition impacts the effectiveness of bias mitigation methods in machine learning, hypothesizing that inconsistent success stems from this often-overlooked factor. Through experiments on semi-synthetic image classification tasks with varied subgroup definitions (coarse, fine-grained, intersectional, and noisy), the authors show that subgroup choice significantly influences outcomes, sometimes worsening fairness. Theoretical analysis reveals that the best subgrouping for bias mitigation is not always the one directly aligned with fairness objectives. Key contributions include introducing a novel setting with spurious correlations, demonstrating subgroup-dependent performance patterns, and providing theoretical insights into optimal grouping strategies, challenging conventional fairness assumptions.\n\n## update after rebuttal\n\nI believe the authors have put significant effort into the rebuttal, providing many new figures and tables in their shared folder. If these are integrated into the final camera-ready version, I believe the paper will be above the acceptance threshold and will make a strong contribution with meaningful results to the field. Therefore, I have raised my score accordingly.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review and helpful suggestions. We summarise the >200 supplementary experiments and edits we have made to address them below. All extra results are shared in this [folder](https://rb.gy/xebqmp). We hope this will give the reviewer confidence to raise their score.\n\n**1. More experiments on diverse datasets** and additional practical examples or a clearer link to real-world applications would further solidify the significance\n\nWe agree with the reviewer's suggestion and have **expanded our analysis to two more datasets** with different modalities and complexity: civil_comments (text) and CelebA (natural images). We find that results closely align across the four datasets. They show similar patterns on the impact of subgroups on mitigation effectiveness with performance strongly correlating to the ability to restore the unbiased distribution. Please see details on the updated setup in Tab A3 and key results in Figs 2 and 5. We believe this strengthens the papers’ claims.\n\nWe have clarified the link to real-world applications by further discussing practical examples like differences in the proportion of chest drains causing sex disparities and by including real results on the effect of granularity of subgroups in civil_comments as discussed in response 9 to Reviewer 6xGC.\n\n**2. Lacks comparisons** with key existing methods, many of these do not rely on group labels\n\nWe thank the reviewer for their comment and relevant references. We have added them to our related work section, and have in particular expanded our discussion on methods which do not require subgroup labels as they do provide important context for our work.\n\nFor our experiments, we focus on a select few established and simple methods as a foundation since many newer approaches (like DFR or AFR) share core principles with resampling and GroupDRO. A recent benchmark also showed no significant performance differences between mitigation techniques [1].\n\nWe initially restricted our experiments to methods with subgroup labels because (a) it is important to understand how subgroups affect mitigation as even methods which do not rely on labels often infer these subgroups (as Reviewer 6xGC noted), and (b) many methods which do not require subgroup labels often actually require some labels on the validation data (e.g. for hyperparameter selection), as discussed in [2], and hence subgroup definition remains an important question, and finally (c), methods with labels are often the upper bound [3-5], though recent exceptions exist [6,7].\n\nHowever, we do agree they could provide an interesting comparison, and so conducted additional experiments on Just Train Twice (JTT) [8], which does not use subgroup labels for training (but requires some for model selection). As shown in Fig C1 and Tab C1, we find that with validation subgroup labels to guide model and hyperparameter selection JTT performs mostly on par with our other methods, however, **performance is again highly dependent on the choice of subgroups**. When no subgroup annotations are used (i.e. model selection is done by overall validation accuracy), the method does not improve over ERM (except for on MNIST where JTT works remarkably effectively, most likely due to the simplicity of the task). We have devoted a new section in our Appendix to these results.\n\n**3. Overlap between first and fourth key contributions**\n\nThe second contribution demonstrates broadly how subgroup choice impacts mitigation effectiveness, while the fourth contribution highlights a specific counterintuitive finding: that achieving fairness between two particular groups may require using different groups for mitigation rather than those groups themselves. We've clarified this distinction in the manuscript, as we believe this specific insight represents an important and non-obvious result.\n\n**4. Structure of theoretical section**\n\nWe thank the reviewer for their comment and have modified the manuscript to improve the clarity of this section by including key intuition, examples, and restructuring A5 and A6. We believe this should significantly help readers better understand the mechanics of our findings.\n\n**5. While AUC is a valid metric, it is not widely used in this literature**. Including WGA would provide better insights\n\nWe include WGA in Tables 2 and 4. For the other results, we use overall AUC for consistency with [1]. Moreover, since we frame the objective as generalisation to an unbiased test set (as in [9]), overall AUC is a useful measure of generalisation performance. AUC is also threshold-independent, which helps give confidence that our results are not simply a side-effect of poor/biased threshold selection.\n\n[1]Zong et al, ICLR (2023)\n\n[2]Pezeshki et al, ICML (2024)\n\n[3]Taghanaki et al, NeurIPS (2022)\n\n[4]Bayasi et al, MICCAI (2024)\n\n[5]Han et al, ICML (2024)\n\n[6]Liu et al, ICLR (2023)\n\n[7]Pezeshki et al, NeurIPS (2021)\n\n[8]Liu et al, ICML (2021)\n\n[9]Jones et al, ICLR (2025)", "claims_and_evidence": "1. **Overlap between first and fourth key contributions**: The claim that subgroup choice impacts disparities (first contribution) appears closely related to the finding that the best way to achieve fairness for a subgroup is not necessarily using it in bias mitigation (fourth contribution). Since both findings seem to be discussed in Section 5.4, the distinction between them is unclear. The authors should clarify how these two contributions differ or consolidate them if they are essentially the same.\n2. **Empirical support for optimal grouping strategies (second contribution)**: While the authors suggest an optimal grouping strategy (A, Y, S), their evidence is primarily based on Colored MNIST, a synthetic dataset. This raises concerns about generalizability to real-world settings. More experiments on diverse datasets would strengthen the claim.\n3. **Theoretical explanation (third contribution)**: The theoretical analysis is currently relegated to the appendix without clear integration into the main text. A more structured discussion in the main sections, with problem formulation and intuition, would improve clarity and support the claim more convincingly. Better section organization in A.5. and A.6. would also make the theoretical insights more accessible.\n\nOverall, while the findings are interesting, clarifying overlapping claims, expanding empirical validation, and improving theoretical discussion would strengthen the paper’s contributions.", "methods_and_evaluation_criteria": "The proposed evaluation framework is well-structured and relevant for analyzing the impact of subgroup choice on bias mitigation. The subgroup generation strategy is comprehensive, covering various realistic scenarios, including noisy annotations, coarse vs. fine-grained groupings, and intersectional subgroups. The use of Colored MNIST (synthetic) and CXP (real-world chest X-ray dataset) ensures both controlled and practical evaluations. However, some areas could be improved:\n1. Evaluation Metrics: While AUC is a valid metric, it is not widely used in this literature. Including Worst-Group Accuracy (WGA) would provide better insights, making results more comparable to prior work.\n2. Baseline Comparisons: The paper lacks comparisons with key existing methods, such as Whac-a-Mole (which accounts for both known and unknown spurious correlations) and approaches like DFR (Kirichenko et al.), AFR (Qiu et al.), EVaLS (Ghaznavi et al.), and SELF (LaBonte et al.). Many of these do not rely on group labels for debiasing or model selection, making them highly relevant benchmarks. Their inclusion would clarify the relative effectiveness of subgroup selection in bias mitigation.\n3. Generalizability & Real-World Validation: While the paper includes a real-world dataset (CXP), additional benchmarks from other domains would strengthen the findings. This is particularly important given the reliance on Colored MNIST for some subgrouping insights, which may not always translate to real-world settings.\n\nOverall, the methodology is well-motivated, but incorporating WGA, broader baselines, and additional real-world benchmarks would enhance the study’s impact and comparability.", "theoretical_claims": "Yes I have checked the theoretical claims and proofs but it was not so easy to follow and understand for me.", "experimental_designs_or_analyses": "Yes, the experimental design and analyses are generally valid. The subgroup generation strategy is comprehensive, and the evaluation setup is well-structured. Theoretical insights align with empirical findings, supporting the study’s conclusions. However, adding key baselines and alternative metrics like Worst-Group Accuracy (WGA) would further enhance validity.", "relation_to_broader_scientific_literature": "The paper builds on existing work in fairness and bias mitigation by challenging the assumption that the best way to improve fairness for a specific subgroup is to use that subgroup in mitigation. This aligns with prior findings in Whac-a-Mole (Sagawa et al.), which showed that bias mitigation can shift bias rather than eliminate it. It also relates to DFR (Kirichenko et al.), AFR (Qiu et al.), and SELF (LaBonte et al.), which explore debiasing without explicit group labels, suggesting alternative ways to approach fairness. Additionally, the paper’s theoretical analysis on subgroup weighting connects to prior work like EVaLS (Ghaznavi et al.), which examines model selection in biased settings. By emphasizing subgroup choice, the paper provides a novel perspective on why bias mitigation often fails, complementing and extending these prior studies.", "essential_references_not_discussed": "Yes, several relevant works are missing that would provide important context for the paper’s key contributions:\n1. Whac-a-Mole (Sagawa et al.) – This work examines spurious correlations and the unintended consequences of bias mitigation, particularly in cases where debiasing one issue exacerbates another. It directly relates to the paper’s argument that subgroup choice can lead to worse outcomes.\n2. DFR (Kirichenko et al.), AFR (Qiu et al.), EVaLS (Ghaznavi et al.), and SELF (LaBonte et al.) – These methods focus on fairness interventions that do not rely on explicit subgroup labels for debiasing or model selection. Since the paper critiques subgroup definition as a bottleneck, these works provide alternative solutions that should be acknowledged.\n\nCiting and discussing these works would provide a more comprehensive view of the challenges in bias mitigation and situate the paper’s findings within the broader literature.", "other_strengths_and_weaknesses": "One of the key strengths of the paper is its novel approach to identifying optimal subgroups for bias mitigation, as well as its thorough benchmarking across various subgroup combinations. This is a meaningful contribution that can help guide future work on improving fairness in machine learning models. The systematic evaluation and detailed subgroup generation strategy are well-executed, providing valuable insights into how subgroup choices can affect model performance.\n\nHowever, the paper could benefit from more evidence or case studies demonstrating how this optimal subgroup identification can be applied in a real-world pipeline. While the theoretical analysis and experimental results are strong, additional practical examples or a clearer link to real-world applications would further solidify the significance and applicability of the approach.", "other_comments_or_suggestions": "L222 “drops the on unbiased test set” → “drops on the unbiased test set”", "questions_for_authors": "1. Clarity of First Key Contribution: Could you please clarify the distinction between the first and fourth key contributions?\n2. Empirical Results for Optimal Grouping Strategies: Can you provide more empirical results or case studies demonstrating how the optimal subgroup strategies perform in real-world datasets, beyond the synthetic or semi-synthetic examples (e.g., coloured MNIST)?\n3. Theoretical Insights: The theoretical discussion in the appendix feels somewhat disconnected from the main text. Could you summarize the key theoretical insights more explicitly in the main body of the paper to help readers better understand the underlying mechanics of your findings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Y9ATGZdou", "reviewer": "Reviewer_mP4g", "summary": "The authors investigate the impact of group definitions on the performance of bias mitigation methods using semi-synthetic experiments on binary classification of images. Specifically, the authors introduce a spurious correlation into the training datasets by selecting examples based on two attributes and a label and then apply subgroup-based bias mitigation methods (gDRO, resampling, DomainInd, and CFair) using different subgroup definitions. Their evaluation reveals that subgroup definitions play a crucial role in model performance: the subgroup should capture the spurious correlation in the data to perform well. The authors quantify this finding using the minimum distance metric evaluated between the train dataset divided by subgroups and the unbiased test dataset. This metric seems to correlate well with bias mitigation performance, potentially revealing theoretical mechanisms behind the impact of subgroup definitions.\n\n## update after rebuttal\n\nThe authors addressed all my questions. I will keep my evaluation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments, and we are glad to hear they appreciate the paper and the originality of our findings. We respond to the comments raised in the review below, and refer to additional results, which are all presented in this [folder](https://drive.google.com/drive/folders/1N5Fpf8VK41awBIYireIoaAjQES_i8xtV?usp=sharing).\n\n**1. Observing the correlation between the minimum distance metric and performance should be positioned as a potential mechanism**\n\nWe agree with the reviewer’s comment and have changed the wording of this in the relevant sections, for instance in the introduction: “We provide a potential explanation for the differences observed based on the KL divergence between the optimal group-weighted biased distribution and the unbiased test distribution.”\n\n**2. Evaluating models on the dataset with \"reversed\" spurious correlation** could be interesting\n\nWe are grateful for this suggestion, and performed the suggested evaluation to further probe the generalisation abilities of our different models. We find that for most datasets, bias mitigation methods, and subgroups, the performance suffers a small decrease on the dataset where the SC is “reversed” compared to the balanced unbiased test dataset (average of $-0.01$), as shown in Tab D1. We also see very similar trends across subgroups to the unbiased test dataset, as shown in Fig D1.\n\n**3. How do you explain the success of subgroups based on $A$ paired with model-based methods** (given that these subgroups perform poorly with reweighting-based methods)?\n\nWe thank the reviewer for this question and agree that it is an important point to discuss. We touched on this briefly in the main text (“For model- based methods, a similar pattern is evident, with the added requirement that the subgroups contain both positive and negative classes”), but agree that this point deserves a more thorough and clear explanation.\n\nWe came to the conclusion that to “map” a subgrouping from a reweighting-based method to a model-based method, the $Y$ component should be removed (e.g. each subgroup should contain both positive and negative classes). This is because methods like DomainInd and CFair learn representations for each subgroup separately. DomainInd trains a separate classifier for each subgroup, so it would not make sense to train a separate classification head for positive and negative classes. Similarly, CFair seeks to align subgroup representations, so it would not make sense to align representations of one subgroup containing only positive images to another subgroup containing only negative images, as this would defeat the point of training a discriminative classifier. On the other hand, for data-reweighting based methods, including the $Y$ in the subgroups helps to balance the final reweighted dataset with respect to class, and therefore improves results, especially in our case where the spurious correlation involves the class $Y$.\n\nThis explains why we find that the subgroups which work well for DomainInd and CFair (e.g. $A$) are just a merged version of the ones which work well for gDRO and resampling (e.g. $AY$). To the best of our knowledge, **no papers have explicitly discussed this distinction despite its practical importance**. We have now elaborated on this in the main body of the paper and included a full discussion in the Appendix.\n\n**4. The paper should more clearly state that it focuses on worst group accuracy** and does not consider different wide-spread fairness criteria and methods that enforce fairness constraints\n\nWe agree that this is an important point and have modified the manuscript to state this more clearly. In addition to the original sentence in the problem setup section “we frame the task according to the fairness paradigm described in Jones et al. (2025), whereby the objective is to generalise from a biased training distribution to an unbiased testing distribution”, we have added an extra sentence in the experimental setup to explicitly state that we are evaluating models based on overall performance on the unbiased dataset and worst-group performance. We selected these measures for their directness and simplicity compared to other fairness criteria.\n\n**5. The paper should clearly state that it only analyzes one specific model of spurious correlation**\n\nWe have clarified this in the problem setup section. \n\n**6. Could you argue why the results for your model of spurious transfer to more sophisticated scenarios?**\n\nWe believe the reviewer is asking if our results transfer to more complex datasets/tasks. We think that in more sophisticated scenarios there may be more complex SCs and causes for bias, so trends may vary on *which* specific subgroup is best, but we think our key point, that **subgroup definition impacts mitigation effectiveness**, holds across scenarios.\n\n\nWe hope we have addressed the reviewer’s questions, but please let us know if there are additional points we could further clarify.", "claims_and_evidence": "I mostly have comments on positioning and writing.\n1. Observing the correlation between the minimum distance metric and performance does not constitute a theoretical analysis. I think this finding should be positioned as a potential mechanism or explanation.\n2. I think the paper should more clearly state that it focuses on the worst group accuracy and does not consider different wide-spread fairness criteria (e.g., demographic parity or equalized odds) and methods that enforce fairness constraints.\n3. Additionally, while the authors acknowledge that they focus on spurious correlations in Section 5.6, I believe the paper should also clearly state that it only analyzes one specific model of spurious correlation.", "methods_and_evaluation_criteria": "The datasets and models seem reasonable. However, evaluating models on the dataset with \"reversed\" spurious correlation could be interesting: it would show whether the models leaned generalizable correlations or simply balanced accuracy across groups.", "theoretical_claims": "I have briefly looked at calculations in Appendix A.6.", "experimental_designs_or_analyses": "I think the chosen bias mitigation methods (gDRO, resampling, DomainInd, and CFair) are reasonable. Many more recent modifications to these methods exist. However, the analysis of more established versions may be suitable for the exploratory study.", "relation_to_broader_scientific_literature": "The paper investigates an important question for group-based bias mitigation methods. These methods are fairly popular in the literature on fairness, robustness, and OOD generalization.", "essential_references_not_discussed": "I do not have additional suggestions for the references.", "other_strengths_and_weaknesses": "The paper is well-written, the findings seem original.", "other_comments_or_suggestions": "I do not have any suggestions.", "questions_for_authors": "1. How do you explain the success of subgroups based on $A$ paired with model-based methods (given that these subgroups perform poorly with reweighting-based methods)?\n2. Could you argue why the results for your model of spurious transfer to more sophisticated scenarios?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "likZrvlfkM", "reviewer": "Reviewer_6xGC", "summary": "This paper studies how the definition of subgroups affects the efficacy of bias mitigation techniques for spurious correlations. A causal graph approach is introduced to formalize correlations between classes, attributes, and subgroups, which are then manipulated to study AUC with respect to an ERM baseline on two semi-synthetic image datasets. The results show that subgroups have a large impact on bias mitigation, including leading to worse outcomes than ERM, and that increasing granularity has no significant effect on performance. Finally, the paper proposes the distance from an unbiased distribution achieved by resampling as an effective generalization measure.\n\n## Update after rebuttal\nAlthough I initially recommended rejection, the authors provided a very detailed and comprehensive rebuttal which addressed all of my critiques. Most importantly, the paper is substantially improved with the addition of more challenging datasets across a variety of domains, as well as a reduction in theoretical overclaims and situation of the theory within established generalization results. Therefore, I now recommend a weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and comprehensive review. We agree with the comments and feel they have helped us substantially improve the paper. We have made clarifications to the manuscript and run >200 additional experiments. We include details of our changes below and share all results in this [folder](https://rb.gy/xebqmp). We hope this will give the reviewer confidence to raise their score.\n\n**1. Lack of challenging datasets across domains**\n\nWe agree that expanding the scope of evaluation would strengthen the results. To address this, **we have expanded our evaluation to include two additional datasets**: civil_comments (text), and CelebA (natural images). Encouragingly, these datasets show consistent trends and help to bolster the generalisability of our conclusions. Please see details on the updated setup in Tab A3 and key results in Figs 2 and 5.\n\n**2. Downsampling** is often superior to resampling. Is resampling’s poor performance due to group definitions or its suboptimality?\n\nWe should clarify we do not think the takeaway should be that resampling performs poorly, but that it performs *variably* depending on the subgroups used. In Fig 2, we observe that it performs quite well for some subgroups (e.g. it increases AUC by >0.10 relative to ERM on MNIST with $YAS$ groups and by >0.04 on CXP with $AY_8$ groups).\n\nHowever, we agree that given the literature it would be interesting to see whether similar results hold for downsampling. **We implement downsampling** as described in [1] and find that results are closely aligned for CXP, CelebA, and civil_comments (Fig C2). Downsampling actually underperforms on MNIST, so we maintain resampling for its consistency.\n\n**3. Granularity findings across two versions of civil_comments**\n\nWe agree this would be a very interesting case study. As we were unable to find this in the literature, we did this experiment by splitting the attribute $A$ (any mention of gender) into two more subgroups (mentions male/does not) and likewise for $S$ (religion) into Christian/non-Christian. We found that performance was very similar across the granular and coarse groupings, in line with our results on synthetic granular groups (Fig 3). We are grateful for this suggestion and think this is a nice real-world and practical insight to strengthen our paper.\n\n**4. Downsampling CheXPert dataset**\n\nThis is a fair point, however, we note that pacemaker annotations are only available for 4862 images, and we have to further downsample the dataset to make it balanced with respect to disease ($Y$) and sex ($S$). Despite the size, we found that by starting with a pre-trained model we were able to get good convergence behaviour.\n\nIn addition, we modified our setup to use the full 60k MNIST images, and, to better understand scaling behaviour with respect to dataset size, we compared the performance of the model trained on the full dataset to the one trained on the downsampled 6k variant. As expected, models trained on the larger variant performed better on average, however trends across subgroups were very similar (Figs D2,3). We thus believe our results are not an artifact of the small dataset size on CheXPert (further supported by our observation of the same trends on CelebA and civil_comments).\n\n**5. Overclaimed theoretical section**\n\nWe have reworded the appropriate sections (please see reply 1 to Reviewer mP4g).\n\n**6. Choice of distance metric**\n\nAfter further consideration we changed the metric to KL divergence. We still see a strong correlation between the divergence of the weighted biased distribution to the target distribution and unbiased generalisation (Fig 5). We note that KL divergence and MAE produce quite similar results, in particular in subgroup ordering (Tab 2).\n\n**7. Limited novelty of Fig 3 in comparison to [2]**\n\nWe acknowledge those experiments share similarities with [2], and we appreciate how their findings align with ours, reinforcing both conclusions. We now mention [2] in our manuscript. However, our evaluations differ in scope, as we consider a broader set of methods, including GroupDRO, CFair, and DomainInd, and focus on full fine-tuning rather than last-layer retraining.\n\n**8. Discuss references which study definitions of subgroups across spurious features**\n\nWe appreciate this suggestion and have added a discussion about this type of study in our related work. Their findings complement ours by exploring how subgroup choice impacts mitigation in a setting where there are multiple SCs, while we focus on a simpler setting where there is only one SC but multiple additional variables (e.g. possible mitigation targets).\n\n**9. Minor issue with the paper’s assertion l.121**\n\nWe have adjusted our statement accordingly and incorporated more citations. We further elaborate in response 2 to Reviewer 86pJ.\n\n**10. Minor edits**\n\nWe thank the reviewer for noting these and have edited the manuscript.\n\n[1]Idrissi et al, CLeaR (2022)\n\n[2]Stromberg et al, TMLR (2024)", "claims_and_evidence": "Overall, the presented evidence supports the paper’s empirical claims on the two datasets studied. My main concerns are a limited scope of evaluation (discussed in Methods and Evaluation Criteria) and overclaimed theoretical results (discussed in Theoretical Claims).", "methods_and_evaluation_criteria": "1. My main concern regarding evaluation is the lack of challenging datasets across a variety of evaluation domains. This paper focuses on semi-synthetic versions of the MNIST and CheXPert datasets. The simplicity of MNIST lends itself to more of a sanity check than a benchmark. Moreover, while CheXPert is a good real-world dataset, the paper downsamples it to only 2330 training images (originally 200K+). Overall, the evaluation in this paper is below the standard in spurious correlations literature of at least 3-4 datasets across both vision and language tasks (e.g., [2, 3, 4, 7]), making it difficult to assess the generality of its conclusions. \n\n2. This paper provides results on resampling, but does not mention the downsampling method (wherein data from larger groups is removed prior to training) proposed by [1] (cited by the authors). Downsampling is often superior to resampling, as resampling WGA may collapse over training [1, 5, 6]. In particular, it’s not clear whether the poor performance of resampling in Figure 2 is due to group definitions or the suboptimality of resampling as a technique.\n\n[1] Idrissi et al. Simple data balancing achieves competitive worst-group-accuracy. CLeaR 2022.\n\n[2] Izmailov et al. On Feature Learning in the Presence of Spurious Correlations. NeurIPS 2022.\n\n[3] Kirichenko et al. Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations. ICLR 2023.\n\n[4] Koh et al. WILDS: A Benchmark of in-the-Wild Distribution Shifts. ICML 2021.\n\n[5] LaBonte et al. The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlations. NeurIPS 2024.\n\n[6] Stromberg et al. Robustness to Subpopulation Shift with Domain Label Noise via Regularized Annotation of Domains. TMLR 2024.\n\n[7] Zhou et al. Examining and Combating Spurious Features under Distribution Shift. ICML 2021.", "theoretical_claims": "1. The statement from the introduction that this paper “[provides] a theoretical explanation for the differences observed based on the minimum distance between the group-weighted biased distribution and the unbiased test distribution” is overclaimed. While Pearson coefficients are provided, no theoretical result is proven which connects the proposed distance metric to model performance (i.e., a generalization bound).\n\n2. The proposed distance metric is insufficiently justified independently of [1]. It makes intuitive sense -- the lower the distance metric, the better the “optimal” resampling will be. However, the distance from the resampling distribution to $\\mathcal{P}_{unbiased}$ is chosen to be the mean absolute error without justification. Why not mean squared error, the absolute or squared error on the worst group only, or a probability divergence such as KL divergence?\n\n[1] Zhou et al. Examining and Combating Spurious Features under Distribution Shift. ICML 2021.", "experimental_designs_or_analyses": "The experimental design, particularly with respect to the subgroup generation step, is clearly explained and well-justified. The additional specifications and results in the Appendix are welcomed.", "relation_to_broader_scientific_literature": "Multiple other works have proposed rethinking the definitions of spurious correlations (e.g., [5]) as well as the definitions of subgroups [2, 3]. However, to my knowledge this is the only paper to evaluate subgroup definitions in the context of the causal learning literature (e.g., [1, 4]).\n\n[1] Jones et al. Rethinking fair representation learning for performance-sensitive tasks. ICLR 2025.\n\n[2] Kim et al. Improving Robustness to Multiple Spurious Correlations by Multi-Objective Optimization. ICML 2024.\n\n[3] Li et al. A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others. CVPR 2023.\n\n[4] Schrouff et al. Mind the graph when balancing data for fairness or robustness. NeurIPS 2024.\n\n[5] Yang et al. Change is Hard: A Closer Look at Subpopulation Shift. ICML 2023.", "essential_references_not_discussed": "1. Figure 3 of this paper studies performance of gDRO and resampling under noise in subgroup labels. A recent paper which should be discussed in this section is [3], which provides evidence of degradation of the WGA of resampling (called upweighting by [3]) under noise in the subgroup labels (called domain labels by [3]). Overall, the novelty of the experiments in Figure 3 is somewhat limited in comparison to [3].\n\n2. It would also be beneficial to discuss references which study definitions of subgroups across spurious features. For example, [1] examine a dataset with multiple spurious correlations and show that defining subgroups with respect to bias type (e.g., age or gender) has a large impact on robustness method performance. Moreover, both [1] and [2] show that mitigating spurious correlation with respect to one subgroup definition may actually exacerbate bias with respect to a different definition.\n\n[1] Kim et al. Improving Robustness to Multiple Spurious Correlations by Multi-Objective Optimization. ICML 2024.\n\n[2] Li et al. A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others. CVPR 2023.\n\n[3] Stromberg et al. Robustness to Subpopulation Shift with Domain Label Noise via Regularized Annotation of Domains. TMLR 2024.", "other_strengths_and_weaknesses": "One strength of the paper is its comprehensiveness in investigating as many as 15 different types of subpopulation groupings. The granularity and random groupings are particularly interesting.\n\nOne weakness (as noted by the authors in the Limitations section) is that the paper focuses on the influence of subgroup definition within bias mitigation methods that utilize group annotations, which is a somewhat unrealistic setting. Nevertheless, since understanding methods that utilize group annotations is likely a prerequisite for understanding methods that infer subgroups, the study is still worthwhile.", "other_comments_or_suggestions": "I have a minor issue with the paper’s assertion that “approaches [which do not utilize group annotations] often fail to consistently outperform traditional subgroup-based methods and are not\nwidely adopted” (line 121-122). The references cited in this paragraph are somewhat outdated, and the statement minimizes the progress made by more recent methods. For example, [1, 5, 9] are competitive with gDRO [8] and [4, 7] are competitive with DFR [2]. As far as adoption of such methods, at least [6] (cited in this paper) is considered widely adopted, though nowadays far from state-of-the-art.\n\nRegarding the granularity discussion in Section 5.2, an interesting case study may be the CivilComments dataset [3]. There are two versions of this dataset in the literature: one with four groups, where the identity categories (male, female, LGBT, black, white, Christian, Muslim, or other religion) are collapsed into a single spurious feature, and one which uses the un-collapsed identity categories. It would be interesting to see if the granularity findings hold across these two versions of CivilComments, and whether results in the literature are consistent.\n\nBelow, I’ve included a list of typos or sentences where more clarification is needed. Also, the plots are pixelated and sometimes hard to read; it would be helpful to increase the DPI.\n\n1. Line 88: Subgroup typo.\n\n2. Line 135: $\\mathbb{E}\\_P$ should be $\\mathbb{E}_{\\mathcal{P}}$?\n\n3. Equation 2: $\\mathcal{G}$ is not defined; I assume it is the set of groups.\n\n4. Equation 3: The average over $m$ is meaningless here as one is already taking the expectation over $\\mathcal{P}_{train}$; the index $i$ is unused within the sum.\n\n5. Equation 3: $P_{train_G}$ should be $P_{train_g}$?\n\n6. Lines 205-206: =0 and =1 should be in math mode.\n\n7. Section 5.3 and 5.5: \\mid should be used for conditional probabilities.\n\n[1] Han et al. Improving Group Robustness on Spurious Correlation Requires Preciser Group Inference. ICML 2024.\n\n[2] Kirichenko et al. Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations. ICLR 2023.\n\n[3] Koh et al. WILDS: A Benchmark of in-the-Wild Distribution Shifts. ICML 2021.\n\n[4] LaBonte et al. Towards Last-layer Retraining for Group Robustness with Fewer Annotations. NeurIPS 2023.\n\n[5] Liu et al. Avoiding spurious correlations via logit correction. ICLR 2023.\n\n[6] Liu et al. Just train twice: Improving group robustness without training group information. ICML 2021.\n\n[7] Qiu et al. Simple and Fast Group Robustness by Automatic Feature Reweighting. ICML 2023.\n\n[8] Sagawa et al. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. ICLR 2020.\n\n[9] Zhang et al. Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations. ICML 2022.", "questions_for_authors": "As mentioned in the “Theoretical Claims” section, why was mean absolute error chosen for the minimum distance metric instead of mean squared error, a worst-group metric, or KL divergence?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper empirically shows that bias mitigation methods with a given set of subgroups don't necessarily work for other subgroups not included for the mitigation. This finding is not surprising because these mitigation approaches specifically aim to achieve fairness with respect to the given subgroups. It is not possible for any methods to guarantee that there will be no performance gap between any unknown arbitrary subgroups. Nevertheless, the reviewers felt that it is still worthwhile reporting this empirical result. 

Another critical issue of the paper was evaluation. The reviewers had concerns about insufficient evaluation even though the paper is purely empirical. The authors added a large amount of evaluation results from new experiments. These results should have been conducted and included in the original paper submission such that the reviewers can have full access to the content. Furthermore, it would be unfair to other authors who submitted (or couldn't submit) all the evidence before the submission due. 

Despite these concerns, AC acknowledges the unanimous support from the reviewers. If this paper is accepted, the authors should incorporate all the suggestions from the reviewers and the new results in the paper.
Decision: Accept (poster)