ICML 2025

Subgroups Matter for Robust Bias Mitigation
Anissa Alloula 1 Charles Jones 2 Ben Glocker 2 Bartłomiej W. Papie˙z 1
Abstract
Despite the constant development of new bias mit-
igation methods for machine learning, no method
consistently succeeds, and a fundamental question
remains unanswered: when and why do bias miti-
gation techniques fail? In this paper, we hypothe-
sise that a key factor may be the often-overlooked
but crucial step shared by many bias mitigation
methods: the definition of subgroups. To investi-
gate this, we conduct a comprehensive evaluation
of state-of-the-art bias mitigation methods across
multiple vision and language classification tasks,
systematically varying subgroup definitions, in-
cluding coarse, fine-grained, intersectional, and
noisy subgroups. Our results reveal that subgroup
choice significantly impacts performance, with
certain groupings paradoxically leading to worse
outcomes than no mitigation at all. Our find-
ings suggest that observing a disparity between a
set of subgroups is not a sufficient reason to use
those subgroups for mitigation. Through theoret-
ical analysis, we explain these phenomena and
uncover a counter-intuitive insight that, in some
cases, improving fairness with respect to a par-
ticular set of subgroups is best achieved by using
a different set of subgroups for mitigation. Our
work highlights the importance of careful sub-
group definition in bias mitigation and presents it
as an alternative lever for improving the robust-
ness and fairness of machine learning models.
1. Introduction
A significant barrier to the wider deployment of machine
learning (ML) models is their tendency to fail when tested
on distributions that differ from their training data. One
particularly concerning manifestation of this issue is perfor-
mance degradation for population subgroups, often caused
1University of Oxford, UK 2Imperial College London, UK. Cor-
respondence to: Anissa Alloula <anissa.alloula@dtc.ox.ac.uk>.
Proceedings of ...