Rethinking Fair Representation Learning for Performance-Sensitive Tasks
Authors: Charles Jones
Fabio De Sousa Ribeiro
Mélanie Roschewitz
Daniel C. Castro
Ben Glocker
Venue: ICLR 2025 Poster (2025)
Keywords: Fairness
Causality
Representation Learning
Distribution Shift
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We provide theoretical proofs and empirical evidence to reveal fundamental limitations of fair representation learning methods.

Abstract: We investigate the prominent class of fair representation learning methods for bias mitigation. Using causal reasoning to define and formalise different sources of dataset bias, we reveal important implicit assumptions inherent to these methods. We prove fundamental limitations on fair representation learning when evaluation data is drawn from the same distribution as training data and run experiments across a range of medical modalities to examine the performance of fair representation learning under distribution shifts. Our results explain apparent contradictions in the existing literature and reveal how rarely considered causal and statistical aspects of the underlying data affect the validity of fair representation learning. We raise doubts about current evaluation practices and the applicability of fair representation learning methods in performance-sensitive settings. We argue that fine-grained analysis of dataset biases should play a key role in the field moving forward.

Published as a conference paper at ICLR 2025
RETHINKING FAIR REPRESENTATION LEARNING FOR PERFORMANCE-SENSITVV TASKS
Charles Jones! Fabio De Sousa Ribeiro', Mélanie Roschewitz',
Daniel C. Castro² & Ben Glocker!.
'Department of Computing, Imperial College London, UK
Miicooooft Research Health Futures, Cambridge, UK
'Corresoondence: fcharles. jones b bo...occcee//mmmrraallll uk
ABSTRACT
We investigate the prominent class of fair representation learning methods for bias mitigation. Using causal reasoning to define and formalise different sources dataset bias, reveal important implicit assumptions inherent to these methods. We prove fundamental limitations on fair representation learning when evaluation data is drawn from the same distribution training data and run experiments across a range of medical modalities to examine the performance of fair representation learning under distribution shifts. Our results explain apparent contradictions in the existing literature and reveal how rarely considered causal and statistical aspects of the underlying data affect the validity of fair representation learning. We raise doubts about current evaluation practices and the applicability of fair representation learning methods in performance-seeee sensitive settings. We argue that fine- fine-grained analysis of dataset biases should play key role in the field moving forward.
INTRODUCTION
If wish deploy deep predictive models in high-stakes settings, such medical diagnosis, we must understand and mitigate performance disparities across population subgroups (Buolamwini & Gebru, 2018; -wyyed-Kllantar et al., 2021). Despite considerable effort in developing methods for debiasing representations of deep models, little progress has been made towards understanding the validity of such methods for real-world deployment. Proposed methods often achieve state-of-the-art results one benchmark, only to be beaten by conventional empirical risk minimisation (ERM; Vapnik, 1999) on more comprehensive evaluations (Zietlow et al., 2022; Zong et al., 2023). Further analyses have shown concerning 'levelling down' effect (Mittelstadt et al., 2023), indicating that today's ' group fairnsss methods may even cause harm if deployed in the real world.
One aspect behind the apparent failure of faimess methods is inconsistent approach to model evaluation. One prominent approach focuses on maximising subgroup performance for test data that independent and identically distributed (IID) to training data, effectively ignoring dataset bias and treating fairness as a learning problem (e.g. Zietlow et al., 2022; Dutt et al., 2023). A second approach assumes that training data includes known spurious correlations and seeks to generalise to an out-of- -ut-o--disttibttinn test set with the bias removed (e.g. Kim et al., 2019; Tartaglione et al., 2021). A third approach even ignores absolute performance entirely, aiming instead to enforce relative equality of properties such as predicted positive (Zemel et al., 2013) or true positive (Hardt et al., 2016) rates.
These three branches of research represent fundamentally different paradigms of fairness analysis; they make different ethical assumptions and require different methods, metrics, and benchmarks. Concerningly, however, much work leaves the distinction between these approaches implicit, and we often see methods from one paradigm employed (potentially inappropriately) in others. Specifically. we will consider the prominent class of fair representation learning methods (FRL; Zemel et al., 2013; Cerrato et al., 2024), which aim to remove sensitive information from learned representations. These methods were initially developed to enforce the demographic parity metric (i.e. enforcing an equal proportion of positive predictions in each group) but have since been applied in settings focusing maximising IID performance (e.g. Pfohl et al., 2021; Zhang et al., 2022), or overcoming distribution shifts (e.g. Kim al., 2019; Wang al., 2020), with mixed results.

---

Published as a conference paper at ICLR 2025
We apply tools from causal reasoning (Pearl, 2011) to clarify the distinctions between different paradigms in fairness analysis. We analyse implicit assumptions harming the validity of FRL methods when applied outside of the settings they were designed for, deriving theoretical results that explain apparent contradictions in the existing literature. Our results indicate that bias mitigation methods must be clearer about their assumptions and limitations, and we call the community to be explicit about what problems the proposed methods aim to solve. Our contributions are:
$2 We provide a unifying perspective on the fairness literature by organising relevant work into three parallel streams, each representing different methodological and evaluation paradigms.
$3 We define causal structures representing realistic scenarios of dataset bias and discuss how the bias mechanisms may affect the performance and fairness of predictive models.
$4 We prove fundamental limitations on the validity of FRL methods when applied in IID settings and propose two hypotheses for the validity of FRL under distribution shift.
$5 We support our theoretical results and hypotheses with a comprehensive set of real-world experiments and discuss the implications of our results for the field moving forward.
2 THREE PARADIGMS OF GROUP FAIRNESS ANALYSIS
We begin by introducing three distinct paradigms of group fairness analysis from the literature, detailing how FRL methods have been applied in each one. Note that we do not make claims about the legitimacy
appropriateness of each paradigm such decisions must be made with ethical knowledge of the application domain (Fazelpour et al., 2022: Mccradden et al., 2023). By organising the relevant literature into these three paradigms, aim clarify the consequences of (mis)applying FRI. methods outside of the problems they were initially developed for.
Enforcing group parity Some of the earliest and most influential research in fair machine learning focuses on enforcing equality of classifier properties across subgroups. This is the context in which Zemel al. (2013) introduced FRI., training strategy which prevents models from encoding sensitive information in their representations. In nigh-dimensional deep learning problems, FRI. is typically implemented through either adversarial training (Edwards & Storkey, 2016; Alvi et al., 2018) or by applying disentanglement techniques (Creager et al., 2019; Sarhan et al., 2020). Variants of FRL have been applied in both supervised and unsupervised (Louizos et al., 2017) settings to enforce demographic parity on downstream predictive tasks (Madras al., 2018). 3). In the supervised case, FRL may be class-conditional (Zhao et al., 2020), corresponding to the equalised odds criterion (Hardt al., 2016) instead. Beyond FRI, large body of work in this paradigm focuses understanding tradeoffs between group fairness metrics such as equal opportunity, calibration, and demographic parity (Kleinberg et al., 2016; Chouldechova, 2017; Kim al., 2020; Friedler al., 2021).
Maximising (subgroup-wise) IID performance A notable aspect of the group parity paradigm is that equality is often achieved by worsening performance for some (or all) groups ('levelling down': Wachter et al., 2021; Zietlow et al., 2022: Mittelstadt et al., 2023), which is likely unacceptable in domains, such as medical diagnosis (Petersen et al., 2023; Weng et al., 2024). In such fields, we have seen a shift from considering fairness as question of group parity goal of maximising performance for all groups (Martinez et al., 2020; Diana et al., 2021). Considerable effort has been made in applying FRL methods to this setting but with limited success. McNamara al. (2019), Zhao & Gordon (201)), and Zhao et al. (2022) derive various negative theoretical results for the performance of FRL on IID tasks, While these results have been known for some time, there seems to remain confusion on this point in the literature, and we have seen repeated attempts to apply FRL in IID settings. Empirically, Pfohl et al. (2021), Zhang et al. (2022), Zietlow et al. (2022), and Zong et al. (2023) benchmark various FRL methods under IID assumptions, finding that they consistently underperform compared to ERM or alternative bias mitigation techniques.
sefformancc-seensitee
Generalising to unbiased distributions In the IID setting, any bias present in the training must also appear in the test set. Thus, maximising test-time performance may be undesirable, as will likely encourage models to reflect whatever bias were initially trying to remove. The third paradigm of research thus views fairness as a problem of generalising from a biased training dataset to an unbiased deployment setting (Kim al., 2019; Wang al., 2020; Tartaglione al., 2021). In this context,
2

---

Published as a conference paper at ICLR 2025
fairness and distribution shift two sides of the same coin a fair model, by definition, seeks to maximise subgroup-wizd performance when generalising to an unbiased test set. This branch of work lends itself particularly well to causal analysis, which provides a unifying language for understanding shifts across groups and settings (Pearl & Bareinboim, 2011; Castro et al., 2020). Wachter et al. (2021), Anthis & Veitch (2024), and Jones al. (2024) connect distribution shifts assumed causal and ethical properties of the underlying data-generating process, relating causal notions of fairness (Kusner et al., 2017: Chiappa. 2019; Plečko & Bareinboim, 2024) existing work in group fairness and robustness. Singh et al. (2021) and Schrouff et al. (2022) further study properties of fair classifiers under specific distribution shifts, with Makar & D' Amour (2022) demonstrating that fairness and robustness may be in alignment under some assumptions on the causal structure of the data.
The group parity paradigm considers fairness as something that can be traded off for performance, whereas the latter two paradigms consider faimess as aligned with maximising subgroup-wing performance on a given test set (either IID or unbiased). For this reason, we will refer to the latter paradigms as performance-sensitiv were developed for the group parity paradigm) valid when applied performance-sensitivy settings?
In this work, we ask simple question: are FRL methods (which
CAUSAL STRUCTURES OF DATASET BIAS
We now take a moment to define what we mean when we say that a dataset is biased. We consider classification problems where we have access to a training dataset of inputs X, targets Y, and sensitive attributes A. The targets are a potentially noisy reflection of some unobserved underlying condition Z (i.e. Y = Z when there is no label noise). Taking a causal interpretation, let Etr be structural causal model (SCM) representing the generative processes in the training dataset. Similarly, Eie is the SCM of the test dataset on which we want to make predictions. We focus on the task of learning a probabilistic model approximating the conditional test distribution p (Y X).
Fair representation learning is predicated on the idea that inputs may encode sensitive subgroup information that may be spuriously correlated with the targets. To express this distinction between task-specific and sensitive information, we need a richer description of our input vector. Following Jiang & Veitch (2022), we consider X to be random vector which may be partitioned into two random variables: Xz, representing target-related features directly caused by Z; and XA, representing features related to the sensitive attribute, directly caused by A.
By construction, XA is predictive of the sensitive attribute A, so we say it encodes sensitive informa- tion. In high-dimensional problems, such as imaging, we may view {X2, 1x) } as high-level latent features that models may implicitly depend on when trained to predict Y from X. For instance, consider skin lesion classification task where self-reported race is the sensitive attribute. Here, Xz may be the pixels representing the lesion, whereas XA may correspond to skin pigmentation. Importantly, the amount of sensitive information encoded in the inputs may vary across application domains due to differences in the A + XA pathway. Jones et al. (2023) refer to the ease with which A may be predicted from XA A as subgroup separability, finding that performance degradation of ERM models under dataset bias is strongly affected by subgroup separability.
When the mapping from inputs to targets is consistent across groups, sensitive information is irrelevant for class prediction; we define such distributions as unbiased. In our skin lesion example, the dataset would be unbiased if exploiting skin pigmentation information does not help performance on the lesion classification task. We formalise this notion of dataset bias in Definition 3.1.
Definition 3.1 (Unbiased distribution). The distribution induced by structural causal model is unbiased if, given Xz, sensitive information XA provides no information relevant to predicting y'!:
Y 11 e XA X2
+
(V Xz) = pe (Y Xz, XA).
Applying the graphical d-separation criterion (Verma & Pearl, 1990). we may derive three fundamental mechanisms of dataset bias that may violate Definition 3.1, causing sensitive information to become spuriously correlated with the target (Jones et al., 2024). We illustrate the unbiased distribution in Figure la and highlight each of the three potential shortcuts in Figure (b-d).
|l represents statistical independence, see $A.1 for table of notation.
3

---

Published as a conference paper at ICLR 2025
XA
X
A
Xa
XA
A
X
0
9+
0-
Xz
Y
X2
Z
Xz
Y
Xz
Z
(a) Unbiased distribution
(b) Presentation disparities
(c) Prevalence
(d) Annotation disparities
disparities
Figure Causal structures of dataset bias in classification tasks. The input X is decomposed into latent features X2, X based on their causal relationships with the sensitive attribute A and (unobserved) underlying class Z. the unbiased setting (a), sensitive information irrelevant to predicting the target Y. This condition may be violated by (b) feature entanglement of A and Z, (c) differences in base rates across subgroups, or (d) differences in labelling policy across subgroups.
Figure 1b is a disparity in class presentation 3 {a, P(Xz 2, + P(Xz Z,a"). where the same features encode sensitive and class-specific information. This is in contrast to dis- parities in class prevalence 3 (a, aa): P(Z + P(z a²) illustrated in Figure Ic, where base rates shift across groups. Finally, Figure ld represents disparities in annotation (a, ) P(Y a, * P(Y Z, a²), where different groups are labelled with different policies. These structures represent realistic sources of bias, with Jones et al. (2024) discussing extensively how each may occur naturally in medical imaging scenarios. We provide further background and discussion, with brief worked example of each mechanism in Appendix $A.2.
We will assume in this paper that the disparities Figure 1 [(b-d} are spurious and should be mitigated. However, any of the mechanisms in Figure may constitute fair or unfair situations in the real world (Chiappa, 2019). For example, in medical imaging, disease prevalence and presentation may legitimately vary across populations (Mccradden et al., 2023) due known physiological mechanisms. In practice, a domain expert would need to determine the fairness of each situation.
RETHINKING FAIR REPRESENTATIONS
Motivated by our causal formulation of dataset bias, we take detailed look the limits of fair representations from the perspective of performance-sensittve faimesss paradigms. Let's begin by recalling from Zemel et al. (2013) that the stated aim of fair representation learning to
"lose any information that can identify whether the person belongs to the protected subgroup, while retaining much other information possible".
We will refer to the first part of this goal effectiveness is FRI. effective at removing sensitive information that would have been encoded by ERM? The second part will be called harmlessness does FRL avoid harming performance by retaining task-relevant information? We begin by proving that fair representations cannot both effective and harmless if test data is IID to training data.
Notably, our results follow from our causal setup in $3, showing how causal approach helps to clarify complex issues in bias and fairness. We do not presuppose any architecture or implementation for the classifiers. Nor do make assumptions about the functional mechanisms in the underlying SCM. We scrutinise the objective of learning fair representations through the lens of implied conditional independence relationships. By taking this approach, distribution being approximated, as opposed to the training dynamics of any specific model. We include discussion of assumptions and proofs for all Lemmas $A.3.
focus the underlying structure of the
4.1 FUTILITY IN THE IID PERFORMANCE PARADIGM
Preliminaries We consider models of the following form: feature extractor fo mapping inputs to representations R. and a classifier which maps representations to predictions. Both components typically implemented as (deep) neural networks. Fair representation learning imposes the train-time constraint that fair representations RF must be (marginally) independent of the sensitive attribute,
4

---

Published as a conference paper at ICLR 2025
denoted as RPRI. 1L etr A, leading to predictor satisfying demographic parity. We contrast this to the unconstrained ERM strategy (i.e. learning RERM). While fo is always a function of (i.e. the feature extractor takes the whole of X input), we will slightly abuse the notation fxX)) to indicate that the feature extractor is only non-constant w.r.t. some subset X of X.
Assumption 4.1. Unconstrained representations depend on input features X C X iff they form Markov blanket over Y at train-time:
HERM fo(X') = Y L Xx)
The Markov blanket contains all information sufficient to predict Y in an idealised (infinite-sample) setting (Peters et al., 2017, Chapter 6). We may view X as a sufficient statistic for predicting Y: hence Assumption 4. is closely related to the information bottleneck principle (Tishby et al., 2000), which stipulates that representations should be minimal and sufficient for predicting Y. Intuitively speaking, Assumption 4.1 states that a properly trained ERM model encodes relevant information in its representations whilst ignoring irrelevant information.
Lemma 4.2. Fair representations must depend on Xz only:
RELL
C.e
= RERL (X().)
Lemma 4.3. Unconstrained representations are fair iff the training distribution is unbiased:
RERM 11 Cxe
= Y Cue XA Xz-
We now take an information-theoretic perspective to define our two desiderata for fair representations: effectiveness (Definition 4.4), and harmlessness (Definition 4.5). While both properties are intuitive and desirable, we show how they each imply constraints on the training and testing distributions in Lemmas 4.6 and 4.7, respectively. By showing that these constraints are incompatible when the distributions coincide, we derive our futility result for IID settings (Proposition 4.8). We denote
I (::)) the mutual information between random variables in the distribution induced by
Definition 4.4 (Effectiveness). Fair representations effective if, at train-time, they do not encode sensitive information that unconstrained representations would encode:
I Sir (A; RERM)
E (A; RFRL) =
Definition 4.5 (Harmlessness), Fair representations are harmless if, test-time, they have equal in- formation relevant to predicting the targets as the input (i.e. they do not discard relevant information).
Iou (Y; RRFR) e XY;
(5)
Lemma 4.6. Effectiveness (E) implies bias at train-time:
E Y C.. X Xz.
(6)
Intuitively, Lemma 4.3 implies that unconstrained model will not encode sensitive information in its representations when trained on a dataset where that information is irrelevant for task prediction (i.c. unbiased according to Definition 3.1). However, effectiveness (Definition 4.4) requires that an unconstrained model does encode sensitive information its representations - else there would be no point removing it with FRI! Thus, Lemma 4.6 follows, stating that FRI. can only be effective if the training data is biased.
Lemma 4.7. Harmlessness (H) implies no bias at test-time:
H
Cvo XA Xz-
Lemma 4.7 states that enforcing demographically invariant representations must lead to a performance penalty when testing on a biased (according to Definition 3.1) dataset. This result is closely related to Zhao & Gordon (2019), who relate the performance penalty under prevalence disparities to the difference in base rates across groups. Our result in Lemma 4.7 does not attempt to derive any bounds on the performance penalty, but is more general. We show that there is performance penalty when deploying FRL on any dataset violating the unbiasedness condition in Definition 3.1, including (but not limited to) the causal structures in Figure 1 (b-d).
5

---

Published as a conference paper at ICLR 2025
Proposition 4.8 (Futility). Fair representations may not be effective (E) and harmless (H. if the train and test datasets are identically distributed:
EAH per + pe
Proof. Suppose, for the sake of contradiction, that we have II training and testing distri- butions per pe and that effectiveness and harmlessness are satisfied. Substituting Lemmas 4.6 and 4.7, we get that
EAH = (Y 1 XA X2) A (Y XA X2).
which is a contradiction.
We emphasise the importance of Proposition 4.8, given that performance-oriented IID benchmarks persist in the literature. Fair representation learning is futile for performance-senstnn IID tasks. The strategy carries implicit assumption that training data contains bias not present at test time. Intuitively, preventing model from using information can only worsen performance unless the predictive power given by that information is entirely spurious and expected disappear test time. Proposition 4.8 hence provides a theoretical explanation on why previous empirical studies benchmarking FRL methods in settings did find any consistent improvements over ERM methods (Pfohl et al., 2021; Zhang et al., 2022; Zietlow et al., 2022; Zong et al., 2023).
4.2 POTENTIAL VALIDITY IN THE DISTRIBUTION SHIFT PARADIGM
Proposition 4.8 demonstrates that FRL methods cannot be motivated by performance in HID settings. We now turn our attention to whether FRL may benefit performance under distribution shift. This setting is more interesting, and today, contradictory empirical results exist in the literature. For example, Kim et al. (2019) and Tartaglione et al. (2021) demonstrate successes of FRL on simple colOur-MNIST benchmarks. contrast, Wang et al. (2020) find that FRL methods fail on the more complex CIFAR-S benchmark. Such results seem to indicate that the underlying structure of the dataset and the shift may affect the validity of FRL methods in the distribution shift paradigm.
To minimise test-time risk under distribution shift, need some notion of what information is stable across domains and what information is unstable or spurious (Peters et al., 2016; Arjovsky et al., 2019). Revisiting Figure 1, notice that while the shorteut paths (red arrows) unstable across domains, the Z + Xz causal pathway is stable across all causal structures, and thus an encoder which depends only on Xz is necessary to transport from biased training setting an unbiased deployment setting (Jiang & Veitch, 2022; Makar & D' Amour, 2022). This is encouraging for FRL, as Lemma 4.2 demonstrates that depending on Xz only a necessary condition for fair representations.
Crucially, however, Lemma 4.2 is not a sufficient condition. There is no guarantee that enforcing RFRL _ Et A is sufficient to learn an encoder which can recover faithful representations of Xz in all cases. Indeed, there is evidence in the existing literature that the validity of enforcing invariant representations is dependent on the underlying causal structure of the problem, with Veitch et al. (2021) and Makar & D' Amour (2022) each proving results for the robustness of closely related methods under different causal structures of distribution shift.
From a representation learning perspective, proving the validity of FRI. would involve proving causal identifiability (Khemakhem et al., 2020) of the Xz feature, which is challenging in the general case (Hyvärinen et al., 2024). Additionally, even FRL cannot guarantee identifiability, FRL may still provide performance benefit over ERM, especially on datasets with strong bias or high subgroup separability. In such cases, ERM models are more likely to rely on the bias shortcut and may suffer extreme performance degradation (Jones al., 2023). Given these challenges with theoretical analysis, we focus instead on a simpler and weaker concept of validity: does FRL practically attain better performance than ERM? We propose two hypotheses, which will explore in $5.

---

Published as a conference paper at ICLR 2025
Hypothesis 4.9. FRL validity under distribution shift depends on the underlying causal structure of the bias present at train-time.
Hypothesis 4.10. FRL validity under distribution shift depends on the amount of sensitive information initially present the inputs (subgroup separability).
EXPERIMENTS AND RESULTS
We support our theoretical analysis with a large-scale set of experiments medical image data. We adapt the experimental setup from Jones et al. (2023), consisting five datasets across the modalities of chest X-ray (CheXpert, MIMIC; Irvin et al., 2019 9; Johnson et al., 2019), dermatoscopy (HAM10000, Fitzpatrick 17k: Tschandl et al., 2018; Groh et al., 2021; Groh et al., 2022), and fundus imaging (PAPILA: Kovalyk et al., 2022). Each dataset is associated with a binary disease classification task and binary sensitive attribute. Where datasets have multiple sensitive attributes available, they treated separately, giving eleven dataset-attribute combinations. We treat the unaltered datasets as unbiased and generate biased variants each dataset according to the mechanisms in Figure 1. In each bias mechanism, we inject bias into subgroup ('Group ) by either dropping samples, corrupting the image, or corrupting the label, whereas the other subgroup (' ('Group 0') is left uncorrupted. We provide details on each dataset, including the procedures generate each biased variant, $A.4.
Our experiments compare subgroup- wise accuracy of ERM against popular adversarial FRL method (Kim al., 2019) and we repeat our analysis with a class-conditional FRL method (Zhao al., 2020) in $A.6. Both methods are representative of stateoof-the-ae in FRI. For each dataset- combination, we train each method on each dataset variant over five random seeds for a total of 660 training runs. We evaluate performance by considering the percentage- accuracy between FRL and ERM (A Acc) each subgroup. For subgroup separability, we use measurements reported for each dataset by Jones et al. (2023)². Further hyperparameter, training, and model details can be found in SA.5.
difference in mean
5.1 VERIFYING FUTILITY IN THE IID PERFORMANCE PARADIGM (PROPOSITION 4.8)
Figure plots the performance gap between FRL and ERM in the IID case. The training and testing datasets are generated by randomly splitting the unbiased variant of each dataset. For all dataset- attribute combinations, Á Acc is negative or approximately zero for both subgroups, supporting the settings.
finding Proposition 4.8 that FRL can only maintain or worsen performance
0 Arc (pp)
-5
-10-
-5-
Increasing Subgroup Separability
-20-
HAM10000
PAPILA Age
CheXpert Age
CheXPert Raon
CheXpert Sex
Sex
DAPILA Sex
HAM10000 Age
Fitzpatrick17K Skin
MIMIC Age
MIMIC Rate
MIMIC Sex
Group
Group
FRM
FRE
Figure 2: Percentage-point accuracy gap for FRI. models compared to ERM models IID disease classification tasks (train/test unbiased). Positive Ace means FRI. outperforms ERM. Datasets are sorted by increasing subgroup separability on the x-axis.
Renchharrknn by (Zong et al., 2023) found statistically meaningful differences between methods. 'Jones et al. (2023) acquire subgroup separability measurements using test-time AUC of classifiers trained to predict the sensitive attribute. Since we the same model class ResNet18 (He al., 2016) these measurements also appropriate for experiments.
7

---

Published as a conference paper at ICLR 2025
Interestingly, the aataset-attribon combinations which suffered the most under FRL had the lowest subgroup separability, whereas the settings with better FRL performance had higher subgroup separa- bility. This seems to indicate that when inputs encode sensitive information more strongly, FRL is better at removing it without affecting the primary task. Conversely, when sensitive information is more difficult extract from the inputs, features relevant the primary task may be more tightly entangled with those relevant to predicting sensitive attributes. In this case, attempting to remove features predictive of the sensitive attribute may degrade primary task performance more.
5.2 TESTING POTENTIAL VALIDITY UNDER CAUSAL SHIFTS (HYPOTHESIS 4.9)
We now consider the performance FRL under distribution shift, testing Hypothesis 4.9 that FRL performance depends on the underlying causal structure of the shift. Figure 3 plots the performance gap between FRI. and ERM when trained on each bias mechanism and tested on an unbiased test set. We find that FRI. performs best relative to ERM under presentation disparities, where it can boost performance for Group (the disadvantaged group) in settings with high subgroup separability. In the other two bias mechanisms, FRL provides little benefit, providing evidence that the underlying causal structure of the bias matters for practical validity of FRL.
X
0.0
-2.5
Acc
64
5.0
Xz
Presentation disparities
XA
100
20
Xz
2
Prevalence disparities
30
X
Ace (p)
100
20
8< Xz
increasing S Subgrou Separabiityy
Annotation disparities
PAFILA
CheXpert Ago
CheXpert Raco
CheXpert Set
SOX
Ago
HEMM10000
"K
MIMIC Six
Six
Agu
Skin
Age
Race
Group
Group
RRM FRI
Figure 3: Percentage-point mean accuracy gap for FRI. models compared to ERM models when trained each mechanism of dataset bias (test is always unbiased). Positive A Acc indicates that FRI outperforms ERM on the unbiased test set.
5.3 TESTING POTENTIAL VALIDITY AS A FUNCTION OF SUBGROUP SEPARABILITY (HYPOTHESIS 4.1
Perhaps the most noticeable pattern in Figure is how the performance gap varies strongly with separability, supporting Hypothesis 4.10 that the practical validity of FRL depends on subgroup separabilityy Across all bias mechanisms, FRL did not offer any improvements for datasets with low subgroup separability, similar to what has been observed in the unbiased settings. We investigate this further in Figure 4, which aggregates the results from Figure 3 over all three bias mechanisms, using
8

---

Published as a conference paper at ICLR 2025
the subgroup separability AUC from Jones et al. (2023) as the x-axis. Our results indicate that there is clear correlation between subgroup separability and empirical validity of FRL.
(4) Group 0 T- 0.600, - 0.0099
(b) Group 1 -- 0.561. 0.0165
-10
A,oc0 (p9)
-20-
30
400
0.6
0.7
0.8
0.9
1.0
0.6
0.7
0.3
0.9
1.0
Subgroou Separability (AUC)
Subgroup Separablity (AUC)
CheXpert-Age
CheXpert-Sex Ettepatric11/:sSk
HAM10000-Age HAM1000B-Sex
MIMIC-Age MIMIC-Race
MIMIC-Sex PAPILA-Age
PAPILA-Sex
Chexpert-Race
Figure 4: Percentage-per point mean accuracy gap for FRL models compared to ERM models, aggregated over all bias mechanisms and plotted against subgroup separability AUC, as reported by Jones et al. (2023). Positive Ace indicates that FRL outperforms ERM on the unbiased test set. We use Kendall's statistic to test for monotonic association between A Acc and subgroup separability. y-axis error bars represent standard deviations of the aggregated Acc measurements. r-axis error bars represent standard deviations in subgroup separability measurements.
Figure 4 makes the dependence of FRI. validity on subgroup separability clear, demonstrating a statistically significant monotonic association between A Acc and subgroup separability. On dataset- attribute combinations with high subgroup separability, FRL improves performance relative to ERM for the disadvantaged group (Group 1) whilst maintaining performance for other the group. In settings with low separability, FRL substantially worsens performance for both groups.
6 DISCUSSION
By organising the related literature into three paradigms of fairness analysis in $2, our work helps to untangle confusion across previous work stemming from multiple conflicting evaluation paradigms and implicit assumptions about what is considered fair. Our causal treatment of dataset bias in $3 shines a light on how the structure of the underlying distribution is key to reasoning about fairness, directly motivating theoretical and empirical results in $4 and $5, We discuss three insights from our work and potential directions for the field.
FRL is not a useful fairness strategy for performance-sensitive IDD tasks Proposition 4.8 states that if we are to apply fair representation learning on IID benchmarks, we must implicitly drop one of the effectiveness or harmlessness criteria. Which criterion we lose depends on whether our data is biased or unbiased according to Definition 3.1.
On unbiased data, Lemma 4.6 shows that we must drop the effectiveness criterion, so FRL provides no fairness benefit over ERM, which would not encode sensitive information anyway (provided Assumption 4.1 holds, as discussed in $A.3). Furthermore, there is no reason for one implement FRL (or indeed any bias mitigation method) if they were confident that they had an unbiased dataset.
On biased data, Lemma 4.7 shows how we lose the harmlessness criterion and should expect overall test-time performance to degrade relative to ERM methods. One interpretation of this result is that group fairness metrics such as demographic parity are not aligned with minimax fairesss (Martinez et al., 2020) under dataset bias; thus. Lemma 4.7 may be seen as a general impossibility result. It is complementary to Pfohl et al. (2023), who investigate whether Bayes-optimal classifiers satisfy equalised odds under causal structures of dataset bias. Note that Lemma 4.7 does not provide

---

Published as a conference paper at ICLR 2025
bounds for the amount of performance degradation; these may be derived for narrower settings with assumptions on the bias mechanisms (Zhao & Gordon, 2019; Zhao et al., 2022).
In this light, recent results from real- world evaluations (c.g. Pfohl et al., 2021; Zhang et al., 2022; Zietlow et al., 2022: Zong et al., 2023), showing that FRL methods worsen performance for all groups, are unsurprising and may be viewed fairness-performan typically have some amount of pre-existing bias, and most evaluations are IID because the train/test sets are generated via random splitting. We should not expect FRL methods achieve otate-of-the-art performance in these cases, and we caution against enforcing invariant representations if evaluation and deployment settings are expected to be IID training. FRL methods should not be used 'blindly'.
tradeoffs. Keal-world datasets
Statistical and causal considerations affect the validity of FRI under distribution shift By taking a fine-grained approach, our work proposes - and provides empirical evidence for two statistical and causal factors that are rarely considered in fairness analysis (Hypotheses 4.9 & 4.10).
Our results in $5 demonstrate how the empirical validity of FRL the distribution shift paradigm depends on both the causal structure of the bias and the amount of sensitive information present to begin with (subgroup separability). We found that FRL methods could only improve performance on an unbiased test relative to ERM when trained on datasets with presentation disparities and high subgroup separability. When trained on other bias mechanisms or on data with lower subgroup separability, PRL consistently degraded performance relative to ERM. Particularly notable was the magnitude of the performance degradation as subgroup separability decreased.
We argue that further theoretical work understand the precise relationship between dataset bias, subgroup separability, and generalisation performance of ERM and FRI. under distribution shift will be a particularly productive area of study moving forward. We provide an extended discussion on connections to domain generalisation and potential directions for future work in $A.2.
Real-world evaluation of FRL remains challenging Finally, we emphasise that real-world evalua- tion of fairness methods under the distribution shift paradigm remains challenge. Proper evaluation of FRL under distribution shift requires training on a biased dataset and testing on an unbiased one, but it tough to find real-world data which satisfy these criteria; we rarely have full knowledge of the biases, and if we had access to an unbiased dataset, we could use it for training without needing FRI..
To overcome this obstacle, some work (e.g. Kim al., 2019; Tartaglione al., 2021) leverages synthetic data with known biases. Others (e.g. Wang et al., 2020, and our experiments in $5) take the alternative approach of injecting bias into real-world data. However, both approaches are unlikely to perfectly simulate the true complexity of real-world biases. Until we better understand the causal and statistical nature of real-world bias, proper evaluation of fairness methods will remain difficult. Other disciplines have a long history of using standardised research protocols and reporting guidelines (c.g. for clinical trials). It may be time to consider similar strategies for planning and assessing research advances on the frontiers of machine learning.
ACKNOWLEDGEMENTS
C.J. is supported by Microsoft Research, EPSRC, and The Alan Turing Institute through a Microsoft PhD scholarship and Turing PhD enrichment award. M.R. is supported by an Imperial College London President's ' PhD scholarship and Google PhD fellowship. B.G. received support from the Royal Academy of Engineering as part of his Kheiron/RAEng Research Chair. B.G. and F.R. acknowledge the support of the UKRI Al programme, and the Engineering and Physical Sciences Research Council, for CHAI- EPSRC Causality in Healthcare AI Hub (grant number FP/Y028856/1).
REFERENCES
Alvi, M., Zisserman, A., Nellaaker, C. (2018). "Turning a Blind Eye: Explicit Removal of Bi- ases and Variation from Deep Neural Network Embeddings". In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (cited on p. 2).
10

---

Published as a conference paper at ICLR 2025
Anthis, J. & Veitch, V. (2024). "Causal context connects counterfactual faimess robust prediction and group fairness". In: Advances in Neural Information Processing Systems 36 (cited on pp. 3, 16).
Arjovsky, M., Bottou, L., Gulrajani, I., & Lopez-Paz, D. (2019). "Invariant Risk Minimization". In arXiv preprint 1xXiv:/97..02550 (cited on p. 6).
Buolamwini, & Gebru, T. (2018). "Gender Shades: Intersectional Accuracy Disparities in Commer- cial Gender Classification". In: Proceedings of the Ist Conference Fairness, Accountability and Transparency, pp. 77-91 (cited on p. 1).
Castro, D. C., Walker, I., & Glocker, B. (2020). "Causality Matters in Medical Imaging." In: Nature Communications (cited on p. 3).
Cerrato, M., Köppel, M., Wolf, P., & Kramer, S. (2024), "I0 Years of Fair Representations: Challenges and Opportunities". In: arXiv preprint arXiv:2407.03882 (cited on p. 1).
Chiappa, S. (2019). "Path-specific counterfactual fairness". In: Proceedings of the AAAI conference on artificial intelligence. Vol. 33. 01. pp. 7801-7808 (cited on pp. 3, 4, 16).
Chouldechova, A. (2017). "Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments". In: Big Data 5.2, pp. 153-163 (cited on p. 2)
Creager, E., Madras, D., Jacobsen, J.-H., Weis, M., Swersky, K., Pitassi, T., & Zemel, R. (2019). "Flexibly fair representation learning by disentanglement" In: International conference on machine learning. PMLR, pp. 1436-1445 (cited p. 2).
Diana, E., Gill, W., Kearns, M., Kenthapadi, K., Roth, A. (2021). "Minimax Group Fairness: Algorithms and Experiments", In: Proceedings of the 2021 AAAT/ACM Conference on AI, Ethics,
and Society, pp. 66-76 (cited on p. 2).
Dutt, R., Bohdal, O., Tsaftaris, S. A., & Hospedales, T. (2023). "FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness Medical Image Analysis". In: The Twelfth International Conference Learning Representations (cited on p. 1).
Edwards, H. Storkey, A. (2016). "Censoring Representations with an Adversary". In: 4th Interna- tional Conference on Learning Representattonss pp. 1-14 (cited on p. 2).
Fazelpour, S., Lipton, Z. C., Danks, D. (2022). "Algorithmic Fairness and the Situated Dynamics of Justice", In: Canadian Journal of Philosophy 52.1, pp. 44-60 (cited on p. 2).
Federici, M., Tomioka, R., & Forré, P. (2021). "An Information-theorrtie Approach to Distribution Shifts". In: Advances in Neural Information Processing Systems. Vol. 34, pp. 17628-17641 (cited
on pp. 16, 17).
Friedler, S. A., Scheidegger, C., & Venkatasubramannin, S. (2021). "The (im) possibility of fairness: Different value systems require different mechanisms for fair decision making". In: Communica- fions of the ACM 64.4, pp. 136-143 (cited on p. 2).
Groh, M., Harris, C., Daneshjou, R., Badri, O., & Koochek, A. (2022). "Towards Transparency in Dermatology Image Datasets with Skin Tone Annotations by Experts, Crowds, and an Algorithm"", In: Proceedings of the ACM on Human- Computer Interaction 6.CSCW2, 521:1-521:26 (cited on p. 7).
Groh, M., Harris, C., Soenksen, L., Lau, F., Han, R., Kim, A., Koochek, A., & Badri, O. (2021). "Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology With the Fitzpatrick 17k Dataset". In: Proceedings of the IFFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1820- 1828 (cited p. 7).
Hardt, M., Price, E., Price, E., & Srebro, N. (2016). "Equality of Opportunity in Supervised Learning..
In: Advances in Neural Information Processing Systems. Vol. 29 (cited on pp. 1, 2).
He, K., Zhang, X., Ren, S., Sun, J. (2016). "Deep Residual Learning for Image Recognition". In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 770-778 (cited on pp. 7, 19).
11

---

Published as a conference paper at ICLR 2025
Hyvärinen, A., Khemakhem, I., & Monti, R. (2024). "Identifiability of latent- variable and structural- equation models: from linear to nonlinear". In: Annals of the Institute of Statistical Mathematics 76.1, pp. 1-33 (cited on p. 6).
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-lleus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., Seekins, J., Mong, D. A., Halabi, S. S., Sandberg, J. K., Jones, R., Larson, D. B., Langlotz, C. P., Patel, B. N., Lungren, M. P., & Ng, A. Y. (2019). "CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison". In: Proceedings of the AAAI Conference on Artificial Intelligence 33.01, pp. 590-597 (cited on 7).
Jiang, Y. & Veitch, V. (2022.. "Invariant and Transportable Representations for Anti-Causal Domain Shifts".. In: Advances in Neural Information Processing Systems 35, pp. 20782-20794 (cited on pp. 3, 6, 16).
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-y., Mark, R. G., & Horn. S. (2019). "MIMIC-CXR, de-Identified Publicly Available Database of Chest Radiographs with Fexe-Text Reports". In: Scientific Data 6.1, p. 317 (cited on p. 7).
Jones, C.,, Castro, D. C.,, De Sousa Ribeiro, F., Oktay, O., McCradden, M., & Glocker, B. (2024).
"A Causal Perspective on Dataset Bias in Machine Learning for Medical Imaging". In: Nature Machine Intellienne. pp. 1-9 (cited on pp. 3, 4, 15).
Jones, C., Roschewitz, M., & Glocker, B. (2023). "The Role of Subgroup Separability in Group-Fair Medical Image Classification". In: Medical Image Computing and Computer Assisted Intervention - -MICCAI 2023, pp. 179-188 (cited on pp. 3, 6, 7, 9, 18, 20).
Khemakhem, I., Kingma, D., Monti, R., & Hyvarinen, A. (2020). "Variational Autoencoders and Nonlinear ICA: A Unifying Framework". In: Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics. Vol. 108, pp. 2207-2217 (cited on p. 6).
Kim, B., Kim, H., Kim, K., Kim, S., & Kim, J. (2019). "Learning Not to Learn: Training Deep Neural Networks with Biased Data". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
(cited on pp. 1, 6, 7, 10).
Kim, J. S., Chen, J., & Talwalkar, A. (2020). "FACT: A Diagnostic for Group Fairness Trade-offs". In: Proceedings of the 37th International Conference on Machine Learning, pp. 5264-5274 (cited on p. 2).
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016). "Inherent Trade-Off's in the Fair Determina- tion of Risk Scores". In: arXiv:1609.05880 0580000000007708 (cited on p. 2).
Kovalyk, O., Morales-Sánchez, J., Verdúi-Monedero, R., Sellés-Navarro, I., Palazón-Cabanes, A., & Sancho-Gómez, J.-L. (2022). "PAPILA: Dataset with Fundus Images and Clinical Data of Both Eyes of the Same Patient for Glaucoma Assessment". In: Scientific Data 9.1, p. 291 (cited on 7).
Kusner, M. J., Loftus, J., Russell, C., Silva, R. (2017). "Counterfactual Fairness.. In: Advances in Neural Information Processing Systems. Vol. 30 (cited on 3).
Loshchilov, I. & Hutter, (2018). "Decoupled Weight Decay Regularization". In: International Conference on Learning Representations (cited on 19).
Louizos, C., Swersky, K., Li, Y., Welling, M., & Zemel, R. (2017). "The Variational Fair Autoencoder". In: arXiv: 15X 1.00830 (cited on p. 2).
Madras, D., Creager, E., Pitassi, T., & Zemel, R. (2018). "Learning Adversarially Fair and Transferable Representations" In: Proceedings of the 35th International Conference on Machine Learning. Vol. 80, pp. 3384-3393 (cited on p. 2).
Makar, M. & D' Amour, A. (2022). "Fairness and Robustness in Anti-Causal Prediction".. In: Transac- fions on Machine Learning Research (cited on pp. 3, 6, 16).
Martinez, N., Bertran, M., & Sapiro, G. (2020). "Minimax Pareto Faimess: A Multi Objective Perspective". In: Proceedings of the 37th International Conference on Machine Learning. Vol. 119, pp. 6755-6764 (cited on pp. 2, 9).
Mceradden, M., Odusi, O., Joshi, S., Akrout, I., Ndlovu, K., Glocker, B., Maicas, G., Liu, X., Mazwi, M., Garnett, T., Oakden-Rayner, L., Alfred, M., Sihlahla, I., Shafei, O., & Goldenberg, A.
12

---

Published as a conference paper at ICLR 2025
(2023). "What's Fair Is... Fair? Presenting JustEFAB, an Ethical Framework for Operationalizing Medical Ethics and Social Justice in the Integration of Clinical Machine Learning: JustEFAB". In: Proceedings of the 2023 ACM Conference Faimess, Accountability, and Transparency, pp. 1505-1519 (cited on pp. 2, 4).
McNamara, D., Ong. C. & Williamson, R. C. (2019). "Costs and Benefits of Fair Representation Learning". In: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 263- 270 (cited on p. 2).
Mittelstadt, B., Wachter, S., & Russell, C. (2023). "The Unfairness of Fair Machine Learning: Levelling down and Striet Egalitarianism by Default". In: SSRN:4331652 (cited pp. 1, 2).
Pearl, (2011). "Causality: Models, Reasoning, and Inference, Second Edition". Cambridge Univer- sity Press (cited p. 2).
Pearl, J. & Bareinboim, E. (2011). "Transportability of Causal and Statistical Relations: A Formal Approach". In: Proceedings of the Twenty-Fifth AAAI Conference Artificial Intelligence, pp. 247- 254 (cited on p. 3).
Peters, J.. Bühlmann, P.. & Meinshausen, N. (2016). "Causal Inference by Using Invariant Prediction:
Identification and Confidence Intervals". In: Journal of the Royal Statistical Society Series B: Statistical Methodology 78.5, pp. 947-1012 (cited p. 6).
Peters, J., Janzing. D., & Schölkopf, B. (2017). "Elements of Causal Inference: Foundations and Learning Algorithms". The MIT Press (cited p. 5).
Petersen, E., Ferrante, E., Ganz, M., & Feragen, A. (2023). "Are Demographically Invariant Models and Representations in Medical Imaging Fair?" In: arXiv:2305.01399 (cited on p. 2).
Pfohl, S. R., Foryciarz, A., & Shah, N. H. (2021). "An Empirical Characterization of Fair Machine Learning for Clinical Risk Prediction". In: Journal of Biomedical Informatics 113, p. 103621 (cited on pp. 2, 6, 10).
Pfohl, S. R., Harris, N., Nagpal, C., Madras, D., Mhasawade, V., Salaudeen, O. E., Heller, K. A., Koyejo, S., & D' D'Amour, A. N. (2023). "Understanding Subgroup Performance Differences of Fair Predictors Using Causal Models". In: NeurIPS 2023 Workshop Distribution Shifts: New
Frontiers with Foundation Models (cited on 9).
Plečko, D. & Bareinboim, E. (2024). "Causal fairesss analysis: causal toolkit for fair machine learning". In: Foundations and Trends@ in Machine Learning 17.3, pp. 304-589 (cited on p. 3).
Rosenblatt, L. & Witter, R. T. (2022). "Counterfactual Fairness Is Basically Demographic Parity". In: arXiv:2208.08 228.0033333 (cited on p. 16).
Sarhan, M. H., Navab, N.. Eslami, A., & Albarqouni, S. (2020). "Fairness by Learning Orthogonal Disentangled Represcntations", In: Computer Vision ECCV 2020, pp. 746-761 (cited on 2).
Schrouff. J., Harris, N., Koyejo, S., Alabdulmohsin. I. M., Schnider, E., Opsahl-Ong, K., Brown, A.. Roy, S., Mincu, D., Chen, C., Dieng, A., Liu, Y., Natarajan, V., Karthikesalingam, A., Heller, K. A., Chiappa, S., & D' Amour, A. (2022). "Diagnosing Failures of Fairness Transfer across Distribution Shift in Real- World Medical Settings". In: Advances in Neural Information Processing Systems 35, pp. 19304-19318 (cited on p. 3).
Kalantari, I.., Zhang, H., McDermott, M. B., Chen, I. Y., & Ghassemi, M. (2021). "Under- diagnosis Bias of Artificial Intelligence Algorithms Applied to Chest Radiographs in Under-Served Patient Populations". In: Nature Medicine 202/ 27:12 27.12, pp. 2176-2182 (cited on p. 1).
Silva, R. (2024). "Counterfactual Fairness Is Not Demographic Parity, and Other Observations". In: arXiv:2402.02666 (cited on p. 16).
Singh, H.. Singh, R., Mhasawade, V., & Chunara, R. (2021). "Faimess Violations and Mitigation under Covariate Shift", In: Proceedings of the 2021 ACM Conference on Fairness, Accountability; and Transparency, pp. 3-13 (cited on pp. 3, 16).
Tartaglione, E., Barbano, C. A., & Grangetto, M. (2021). "EnD: Entangling and Disentangling Deep Representations for Bias Correction". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13508-13517 (cited on pp. 1, 2, 6, 10).
13

---

Published as a conference paper at ICLR 2025
Tishby, N., Pereira, F. C., & Bialek, W. (2000). "The Information Bottleneck Method". In: arXiv:physicc/0004057 (cited on p. 5).
Tschandl, P., Rosendahl, C., & Kittler, H. (2018). "The HAM10000 Dataset, Large Collection of
Multi-Source Dermatoscopic Images of Common Pigmented Skin Lesions". In: Scientific Data 5.1, p. 180161 (cited on p. 7).
Vapnik, (1999). "An Overview of Statistical Learning Theory". In: IEEE Transactions on Neural Networks 10.5, pp.
(cited p. 1).
Veitch, V., D' Amour, A., Yadlowsky, S., & Eisenstein, (2021). "Counterfactual Invariance to Spurious Correlations in Text Classification" In: Advances in Neural Information Processing Systems. Vol. 34, pp. 16196-16208 (cited on pp. 6, 16).
Verma, T. & Pearl, J. (1990). "Causal Networks: Semantics and Expressiveness". In: Proceedings of the Fourth Annual Conference on Uncertainty in Artificial Intelligence, pp. 69-78 (cited on pp. 3, 15).
Wachter, S., Mittelstadt, B., & Russell, C. (2021). "Bias Preservation in Machine Learning: The Legality of Fairness Metrics under EU Non-Discrimination Law". In: West Virginia Law Review (cited on pp. 3).
Wang, Z., Qinami, K., Karakozis, I. C., Genova, K., Nair, P., Hata, K., & Russakovsky, O. (2020). "Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (cited on pp. 1, 2. 6, 10).
Weng, W.-H., Sellergen, A., Kiraly, A. P., D' Amour, A., Park. J., Pilgrim, R., Pfohl, S., Lau, C.,, Natarajan, V., Azizi, S., Karthikesalingam, A., Cole-Lewis, H., Matias, Y., Corrado, G. S., Webster, D. R., Shetty, S., Prabhakara, S., Eswaran, K., Celi, L. A. G., & Liu, Y. (2024). "An Intentional Approach to Managing Bias in General Purpose Embedding Models". In: The Lancet Digital
Health 6.2, 1026-e330 (cited on p. 2).
Zemel, R., Wu, Y., Swersky, K., Pitassi, T., & Dwork, C. "Leaming Fair Representations" In: Proceedings of the 30th International Conference on Machine Learning. Vol. 28. 3, pp. 325-333
(cited on pp. 1, 2, 4).
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016). "Understanding Deep Learning Requires Rethinking Generalization". In: International Conference on Learning Representations (cited on p. 18).
Zhang. H., Dullerud, N., Roth, K., Oakden-Raynen L., Pfohl, S., & Ghassemi, M. (2022). "Improving the Fairness of Chest X-ray Classifiers". In: Proceedings of the Conference on Health, Inference. and Learning, pp. 204-233 (cited on pp. 1, 2, 6, 10).
Zhao, H.. Coston, A., Adel, T., & Gordon, G. J. (2020). "Conditional Learning of Fair Representa- tions". In: International Conference on Learning Representations (cited on pp. 2, 7, 18).
Zhao, H., Dan, C., Aragam, B., Jaakkola, T. S., Gordon, J., & Ravikumar, (2022). "Fundamental Limits and Tradeoffs in Invariant Representation Learning". In: Journal of Machine Learning Research 23.340. pp. 1-49 (cited on pp. 2, 10).
Zhao, H. & Gordon, G. (2019). "Inherent Tradeoffs in Learning Fair Representations" In: Advances in Neural Information Processing Systems. Vol. 32 (cited on pp. 2, 5, 10).
Zietlow, D., Lohaus, M., Balakrishnan, G., Kleindessner, M., Locatello, F., Schölkopf, B., & Russell, C. (2022). "Leveling Down in Computer Vision: Pareto Inefficiencies in Fair Deep Classifiers". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10410- 10421 (cited on pp. 1, 6, 10).
Zong, Y., Yang, Y., & Hospedales, T. (2023). "MEDFAIR: Benchmarking Fairness for Medical Imaging". In: The Eleventh International Conference on Learning Representations (cited on pp. 2, 6, 7, 10).
14

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1
ACRONYMS AND NOTATION
ITD
independent and identically distributed. empirical risk minimisation.
ERM
SCM
structural causal model.
FRL
fair representation learning.
X
random variable.
X
random vector.
scalar realisation of random variable X.
vector realisation of random vector X.
structural causal model.
pe
probability distribution induced by C.
X x
X, Y are statistically independent in the distribution induced by C.
(9(XYY)
mutual information between X, Y in the distribution induced by e.
A.2 EXTENDED DISCUSSION
We provide an extended discussion, adding depth to areas that some readers particularly those interested in building this work may find interesting. Some elements of this section adapted from conversations during the review process. We thank the anonymous reviewers for spurring us to think about these topics.
WORKED EXAMPLES OF BIAS MECHANISMS
To further illustrate the bias mechanisms in $3, consider medical example where we wish to classify some disease from chest X-ray images X, with biological sex as a sensitive attribute:
A prevalence disparity may take the form of a shift in the marginal distribution of Y across groups. For example, there may be greater proportion of positive males in the dataset than positive females due to differences, historical disparities in healthcare, etc.
combination of physiological differences, demographic
A presentation disparity may occur if there is a shift in the generative process P(X Y); for example, one group may be systematically diagnosed later in their disease progression, leading the same condition appearing more severe with different pathological features.
An annotation disparity when there is shift in the diagnostic mapping P(Y Z). This may occur if different groups annotated with different policies, e.g. due to historical healthcare disparities or diagnosis practices at different hospitals.
Each of these mechanisms would cause ERM-trained classifier rely sensitive information when trained to predict disease from chest X-rays. Importantly, for all cases, a domain expert would need to examine the causes of each disparity. If the association is deemed spurious unfair (as we assume throughout this paper), then the disparity should be mitigated. If this association is not spurious, then it may contain potentially useful information that a predictive model should leverage.
ON THE CHOICE OF CAUSAL STRUCTURES
The bias mechanisms in Figure I are derived by applying the d-separation criterion (Verma & Pearl, 1990) to find the simplest fundamental graphs which violate Definition 3.1. They are based on the causal structures proposed by Jones et al. (2024), who also provide practical examples justifying their applicability to real-world settings. Notably, when there is no possibility label noise (i.c. Y = Z), these structures collapse to the familiar anticausal setup (i.e. X + Z Y becomes X Y).
15

---

Published as a conference paper at ICLR 2025
These structures may thus be seen generalisations of previously studied anticausal bias mechanisms, such as those from Singh et al. (2021) and Makar & D' Ammour (2022), and so results that are valid for setup should be valid for the anticausal case in general. Similar structures have also been applied in the robustness and distribution shift literature (Veitch et al., 2021; Jiang & Veitch, 2022).
We also note that, while use the structures in Figure 1 in our setup and to motivate the biases in $5, our futility result in Proposition 4.8 is more general. Proposition 4.8 relies only on Definition 3.1 and the causal decomposition of X into {X A, Xz). We thus emphasise that the class of problems for which FRL is futile is much larger than the causal structures explicitly enumerated in $3, See Figure for two further relevant examples.
Y
X
A
A
Z
Xz Z
XAA X XAAZ
Xz
Figure 5: Two further examples of bias mechanisms for which Proposition 4.8 applies to. Left is causal structure (i.e. X- Y). where different groups with the same Xz features are annotated differently. Right includes an interaction feature XA^Z, acting as a collider for A and Z. Any model that implicitly conditions on the XA^Z feature will see a spurious correlation between and Z.
CONNECTIONS TO COUNTERFACTUA FAIRNESS
Our work focuses statistical notions of performance and fairness, these most commonly evaluated by the community and are explicitly targeted by FRI. methods. This is closely related to causal notions of fairness such as counterfactual fairness (Rosenblatt & Witter, 2022; Anthis & Veitch, 2024), however, there are some subtleties to this connection (Silva, 2024). In many cases but notably, not all (Silva, 2024) - counterfactual fairness implies demographic parity, and in such situations, Proposition 4.8 also applies to counterfactually fair FRL predictors. Similarly, conditional FRL enforces equal opportunity. which can be implied by extensions of counterfactual fairness such as path speec--peiific counterfactual fairness (Chiappa, 2019), allowing similar results to be derived.
CONNECTIONS TO DOMAIN GENERALISATION
On one level, FRI and domain generalisatioo//aaatatii techniques (especially methods based on adversarial training and disentanglementt share many similarities. Often, the problem setup in both of these fields is very similar, with fairness using a 'sensitive attribute' and domain generalisation using a 'domain' or 'environment' variable. This similarity gives us hope that our results may be insightful in fields beyond fairness, however, we do not consider such claims within the scope of this paper.
key point to note that in our formulation, there are two simultaneous shifts: disparity across groups (e.g. prevalence, presentation, annotation disparities, as enumerated in Figure 1) and potential shift across train/test domains (i.e. training on biased data and testing on unbiased data in the distribution shift paradigm). We can thus relate our problem to the traditional distribution shift setup by extending the framework of Federici et al. (2021) for instance.
To illustrate this, consider the joint distribution over training and testing datasets by using the binary indicator variable to distinguish between them. We can now decompose the shift across groups and domains like so:
IX,Y, = ((;;) 1(Y;T) X) + 1(A; X,Y).
(9)
In this formulation, the LHS term represents the overall distribution shift, and the terms on the RHS
represent covariate shift, label shift, and attribute shift, respectively. If the selection yields informa- tion about the joint distribution then the training and test distributions are IID, i.c. [(X, Y, A; T) = O. Different factorisations of this joint mutual information imply different data-generating processes
16

---

Published as a conference paper at ICLR 2025
and correspond to the various shifts shown in Figure 1. Furthermore, different selection effects can be represented by the functional relation between (X, A)) and the selection variable T.
aetribute-based selection effeet could be represented by the causal mechanism == f(A, N), where N is an exogenous noise variable. There are various other combinations possible, including multivariate selection effects T f(X, Y, A, N), ones consisting only of exogenous noise = (NN,, which would represent the unbiased setting.
For instance,
Federici et al. (2021) derive some practical upper bounds on the (latent) concept shift quantity which, under some reasonable assumptions, are guaranteed to minimise concept shift. In our view, deriving practical bounds for other types of distribution shifts of the sort studied in the present work and beyond constitutes fertile ground for future research.
A.3 PROOFS AND DISCUSSION OF ASSUMPTIONS
Lemma 4.2. Fair representations must depend on X2 only:
RERL 11 Cu.
A = R fx(x2).
Proof. The result follows from the causal decomposition $3, where XA H A by definition. Now, let X (X2, X. and RFRI. (X()::
RERL = -
= X
= RFRL = fo(X2)
Lemma 4.3. Unconstrained representations are fair iff the training distribution is unbiased:
RERM e.r - Y 1 X
Xz.
Proof. This result is straightforward consequence of our causal decomposition in $3, combined with Assumption 4.1. Let X {Xz, Xa), and X X. s.t. Y ll tu manipulation, we get that
XXX* X". Through simple
1 c. XA Xz
X (Xz), X {XA}: RERM = fe(Xz) (Assumption 4.1); RERM et A.
Lemma 4.6. Effectiveness (E) implies bias at train-time:
E - Y Car XA Xz-
(6)
Proof. This result follows from Definition 4.4 and Lemma 4.3. Begin by noticing that Equation (4) implies the following independence statement:
er (A; RERM >
RERM y Eu. A.
Since this is the logical negation of the LHS of Equation (3), it follows that the RHS must also be negated when effectiveness is satisfied due to the logical equivalence of the sides ( ). Thus, effectiveness implies bias at train time.
Lemma 4.7. Harmlessness (H) implies no bias at test-time:
H -= Euo X Xz.
Proof. This result follows from Definition 4.5 and Lemma 4.2. Starting from the definition of harmlessness, decompose the RHS expression using the chain rule of mutual information:
H. e (Y: = Sie (Y; X2,
+ ee (Y; RFLL Cie (Y; X2) + E (Y;XAl Xz).
From Lemma 4.2, recall that RPRL = fo(X2). Now we may apply the data processing inequality e (Y; fo(X))) < I ( (Y; xz) and nonnegativity of mutual information to see that unbiased test set is necessary (but not sufficient) for barmlessness:
- Je XA Xz) = Y Eu XA Xz-
17

---

Published as a conference paper at ICLR 2025
What if Assumption 4.1 is violated? Assumption 4.1 is needed define what information properly trained ERM model relies on and is used in the proofs of Lemmas 4.3 and 4.6. If we reject Assumption 4.1, get the (rather unintuitive) result that ERM model may rely sensitive information even when trained on an unbiased dataset where such information provides no predictive power. In practice, this may occur if the model is underfit has insufficient training data. In this case, the FRL strategy may have some use for unbiased IID settings. By constraining the solution space, may be possible for FRL improve sample efficiency during training, analogous a regulariser or inductive prior. It is unclear, however, whether this scenario is particularly relevant with today's 's practice of capaciiy models trained convergence on large datasets (Zhang al., 2016).
Applying Assumption 4.1, we derive that [e (A; REERb) =0 > Y Il L XA X2. that is, unconstrained representations encode no sen- sitive information if and only if the training data is unbiased. In this case, we define FRL as (trivially) ineffective since it cannot provide any fairness benefit information anyway. This case is unlikely to be particularly common since it is unclear why any researcher would apply FRI. methods to dataset that they are confident is unbiased.
Why the strict inequality in Definition 4.4?
ERM, which would not encode sensitive
A.4 DATASET DETAILS
The datasets were all preprocessed and split using the same procedure as (Jones et al., 2023), who also report summary statistics. For each dataset, the disease prediction task was constructed by binning all available disease labels (e.g. pneumonia, glaucoma) into the positive class. Other labels (e.g. no- no-finding) were binned into the negative class. Binary subgroup labels for 'Group and 'Group were constructed according to the following criteria:
When the sensitive attribute is sex: 'Male' -Group 0', 'Female' 'Group 1'.
For race: "White" = *Group 0', 'Non-White' (all other labels) 'Group 1'.
For age: < 60 = *Group 0', 2 60 'Group 1'.
For skin type (Fitzpatrick scale): I-lll = 'Group 0', IV-VI 'Group 1'.
To generate the biased variants of each dataset, we implemented the following procedure:
Presentation disparities: 50% of positive individuals in 'Group 1' have the image corrupted by reducing sharpness*.
Prevalence disparities: 50% of positive individuals in 'Group 1' are dropped from the dataset.
Prevalence disparities: 50% of positive individuals in 'Group are mislabeled as negative.
A.5 TRAINING DETAILS
Training consisted of two phases: an initial hyperparameter tuning phase, followed by final sweep with fixed hyperparameters (the latter phase generated the results reported in $5). In the tuning sweep, the methods were trained and evaluated across all datasets. The final hyperparameters were selected by considering combinations for which training successfully converged across all datasets and achieved the best performance. When selecting adversarial coefficients for the two FRI. methods, we ensured that the accuracy of the adversarial prediction head did not exceed the approximate prevalence of the subgroups. This was to prevent the selection of hyperparameter values that would result in the adversary being ignored. The final hyperparameters used are reported in Table 1.
A.6 CONDITIONAL FRL RESULTS
We include the results of our extended experiments using a conditional FRI. implementation (Zhao al., 2020). Notice that the results demonstrate the same trends as the main results in $5.
'torchvi c1 adjust sha rprees implementation, with sharpness factor 0.5.
18

---

Published as a conference paper at ICLR 2025
Table 1: Hyperparameters used across all runs in $5.
Config
Value
Architecture
ResNet18 (He et al., 2016)
Optimiser
AdamW (Loshchilov Hutter, 2018) (Ir: le 4, B1: 0.9, 82: 0.999)
Adversarial coefficients
(Marginal FRE: 1.0, Conditional FRE: 0.05}
Schedule
Constant
Max Epochs
50
Early Stopping
(Monitor: worst group AUC, Patience: 5 epochs) RandomResizedCrop. RandomRotation(159) 256 (32 for PAPILA)
Augmentation
Batch Size
A Ace (pp)
-10-
-20
Increasing Subgroup Separability
HAM10000 Sex
PAPILA Age
CheXpart Age
CheXpert Race
CheXpert Sex
PAPILA Sex
HAA110000 Age
Fizzpatrick17k 7K
MIMIC Age
MIMIC Race
MIMIC Sex
Skin
Group
Group
ERM FRL
Figure 6: Percentage-poin mean accuracy gap for conditional FRL models compared to ERM models on IID disease classification tasks (train/test unbiased). Positive A Acc means FRL outperforms ERM.
X
Acc (P9)
10+
Xz
Presentation disparities
155
X,A
Acc (pp)
10-
20+
X
Prevalence disparities
30-
XA
-+
Acc (pg)
10-
15-
-20
Xz
-25-
Incrasinn Subgroup Separbbility
Annotation disparities
-30-
HAM10000
CheXpert
ChaXpert
ChaXpart
EAPILA
HAMI 0000
Fizpatrkkk11K
MINIC
MIMIC
MIMIC
Age
Age
Figure 7: Percentage-point mean accuracy gap for conditional models compared models when trained on each mechanism of dataset bias (test set is always unbiased). Positive A Acc indicates that FRL outperforms ERM the unbiased test set.
19

---

Published as a conference paper at ICLR 2025
(a) Group 0 0.673, 0.0031
(b) Group 0.600, 0.0099
10-
4 Acc (pp)
220
-50
d.a
1.0
0.9 Subgroup Separability (AUC)
0.7
Subgroup Separabiliy (AUC)
CheXpert-Age
CheXpert-Sex
HAM10000-Age HAMIOD0D-SCC Siex
Chexpart-Race
/
MIMIC- Race
PAe
ERM = FRL
Figure 8: Percentage-point mean accuracy gap for conditional FRL models compared to ERM models, aggregated over all bias mechanisms and plotted against subgroup separability AUC, as reported by Jones al. (2023). Positive A Acc indicates that FRL outperforms ERM the unbiased test set. We use Kendall's 7 statistic to test for monotonic association between A Acc and subgroup separability.
20


Reviews:
[{"review_id": "7A4gswKCim", "reviewer": "Reviewer_svct", "summary": "This paper tackles an important challenge in explainable and reliable AI by examining fair representation learning methods focused on achieving group fairness. The authors critically explore the limitations of existing approaches. They present three main fairness paradigm: enforcing group parity, maximizing subgroup IID performance, and generalizing to unbiased distributions. Their key theoretical insight is that achieving group fairness cannot be both effective and harmless (relative to ERM) within the IID setting. Furthermore, the paper suggests that the success of fairness learning methods depends heavily on shifts in test data distribution, hypothesizing that performance is also influenced by the underlying causal structure and the amount of sensitive information available in the training dataset.", "questions": "1. This paper mainly talks about group fairness, what about counterfactual fairness? Is the counterfactual fairness also impossible in the iid setting?\n\n2. This work shows that distribution shift is closely related to group fairness, and there are some results in the area of domain generalization / adaptation. Can you elaborate some results in that area? Is there any method closely related to the group fairness?\n\n3. How does different types of distribution shift (class imbalance, covariate shift, concept shift ...) in the dataset related to the three types of disparities?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper is self-contained and easy to follow, and the notation is clear. It is a joyful experience to read a manuscript like this. The theory is simple yet insightful. The experiment supports the theoretical as well.", "weaknesses": "1. One major concern is the lack of results in the distributed shift setting. While the negative result in the IID is sound, it would be good to have some results for out-of-distribution, or connects the out-of-distribution fairness to some existed theoretical analysis results in the area of  distribuiton shift.\n\n2. Another concern is that the trade-off between fairness and accuracy has been explored by previous methods (Zhao et al., 2022). Why and how this paper is different than previous methods on analyzing the trade-off between fairness and accuracy, \n\nI would like to raise my score if these concerns are addressed.\n\n[1] Zhao, H., & Gordon, G. J. (2022). Inherent tradeoffs in learning fair representations. Journal of Machine Learning Research, 23(57), 1-26.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7PtC1Dmnf3", "reviewer": "Reviewer_ydWf", "summary": "This paper makes the argument that fair representation learning (FRL) is not a useful approach in the class of cases where model accuracy (or analogous metrics) is key. They provide an overview of several paradigms of fair ML, and highlighting which are “performance-sensitive”. They then apply a causal framework to discuss the situations in which FRL might be helpful, using this to claim that FRL is not helpful to achieve performance-sensitive objectives when test data is IID. They show experiments to support this, in particular highlighting that even when test data is IID, the causal structure of the problem matters.", "questions": "- The characterization in Fig 1 is interesting but a couple contextual questions - it’s not clear to me as a reader if this is a) a complete characterization of the types of causal structures that can produce this phenomenon, or b) if this is a novel contribution on the part of this paper, or something pre-existing in the literature", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Note: I think I may have reviewed a previous version of this paper at another conference (also double blind, no anonymity issues). I have not gone back to look at my past comments so this review is with fresh eyes, and I believe that a number of improvements have been made to this draft. However, apologies if there are overlapping comments anywhere\n- Really enjoyed the framing in Sec 2 around group parity, IID performance, and generalization to unbiased OOD, I think this sets up the paper quite well and gives a nice overview of the relevant parts of the fairness literature\n- does a nice job of synthetizing causal reasoning with the FRL literature. In particular, the experiments around FRL effectiveness and causal structure are quite interesting\n- in general I think the main point made here is correct and correctly-scoped, and useful for a specific segment of the community interested in these things", "weaknesses": "- I’m a little confused by the X_Z and X_A framework, what about features that are impacted by both Z and A? It’s not clear if this framework allows for that overlap\n- It would be good at some point to discuss whether any of this applies in an anti-causal setup - the “futility” framing is quite strong and yet I feel as though the large anti-causal class of problems are not really touched on here\n- Quibble with Def 4.5 - technically, I would phrase this as “have an equal amount of information” rather than “do not discard relevant information”. The question of whether information is discarded I think gets more into implementation and I’m not sure how “discarding information” is defined technically.\n- Quibble with Def 4.4/Lemma 4.6 - I find something a little odd here which I think is around the corner case where ERM representations have no sensitive information. I think the important part of Def 4.4 (the “effectiveness”) of FRL is the equality to 0 - this is whether they are truly “fair” in this definition. However, Lemma 4.6 relies heavily on the first part of the def’n (having strictly less information) - if the ERM reprs have 0 sensitive information then trivially all FRL will violate effectiveness, but this seems wrong as that is not a property of the FRL method, but rather a property of the ERM representations which are given. Anyways, this is not a massive issue with the general point but I do think there needs to be some more careful wording with how the intuition is formalized here (e.g. maybe the I(ERM, A)=0 case needs to be separated out as “trivial”)\n- The results in Fig 2 on subgroup separability go in the opposite direction of what I would expect - the authors note this and give some thoughts on why this might be. However, it suggests something troubling - that at separability=0.5 (i.e. no information at all about A, trivial case), the fair representation methods will yield low-accuracy representations, when in fact this should be the easiest case (just return the representations as given). This suggests to me that the implemented methods may not be working as intended. It would be good to have a supplementary experiment showing that the methods, are in fact, working as intended - otherwise the empirical results are a little harder to parse due to the chance of experimental failure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rghBTqVMfq", "reviewer": "Reviewer_nneh", "summary": "The paper defines causal structures representing realistic scenarios of dataset bias and discuss how the bias mechanisms may affect the performance and fairness of predictive models. The authors then prove fundamental limitations on fair representation learning when evaluation data is drawn from the same distribution as training data.", "questions": "1. The same as weakness 1, is there more comparison between the causal structures proposed by the authors and other literature?\n\n2. Are there more experiments from another dataset in performance-sensitive areas other than just medical imaging?\n\n3. Typo: in the contribution part (page 2), shouldn't the orderings begin with 1?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality and novelty: the authors propose a novel causal framework to categorize the bias in the data.\n\n2. Supportive theoretical results: the authors prove the limitations of the FRL in IID settings.\n\n3. Clear structure: the writing and organization is clear, making the paper easy to follow.", "weaknesses": "1. The causality-based notion in fair ML literature has been well discussed, and there have been numerous causal structures proposed. Can the authors provide a comparative analysis with one or two to further prove the novelty?\n\n2. Limited experiments: it seems the experiments only focus on medical imaging scenarios. The results would be more convincing if datasets from other performance-sensitive areas are considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xaXm448TBY", "reviewer": "Reviewer_E1KU", "summary": "This paper summarizes the field of fair representation learning into three main categories and then illustrates from a causal perspective that the underlying problem that people have been trying to solve in fair representation learning is in fact somewhat ill-defined. The paper argues that if the test and training distributions are the same, fair representation learning is \"futile\". In addition, the paper also argues that if there is a distribution shift at test time then there is hope as well as when the features are somewhat separable.\nThe paper establishes theoretical foundations and well as experimentally illustrates the theoretical findings.", "questions": "see above for the questions", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper puts into perspective the whole fair representation learning field, using causal language.\n- The paper critically establishes a key problem in the field and how researchers benchmark their own methods.\n- The paper also shows experimentally that their theoretical findings have practical consequences which is very much appreciated. In particular, they show that if there is indeed a test=train time distribution, fair representation learning does not help much. and vice versa.", "weaknesses": "- The paper requires somewhat of a graphical model/ causal background which might be less beginner-friendly\n- The paper is very dense and requires multiple passes to fully understand the paper. I would recommend adding further intuitions to the paper.\n- Lastly, i might have missed this. Do the authors have any comments on the relations between FRL to standard fairness learning schemes that are not representation-based? I would be curious to hear of these problems persist necessarily.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)