Mind the Graph When Balancing Data for Fairness or Robustness
Authors: Jessica Schrouff
Alexis Bellot
Amal Rannen-Triki
Alan Malek
Isabela Albuquerque
Arthur Gretton
Alexander Nicholas D'Amour
Silvia Chiappa
Venue: NeurIPS 2024 poster (2024)
Keywords: Fairness
Robustness
Data balancing
Causality
Fine-tuning
Primary area: fairness
TL;DR: We use a causal framework to investigate the failure modes of data balancing for fairness or robustness, its conditions for success, and its interaction with other mitigation strategies.

Abstract: Failures of fairness or robustness in machine learning predictive settings can be due to undesired dependencies between covariates, outcomes and auxiliary factors of variation. A common strategy to mitigate these failures is data balancing, which attempts to remove those undesired dependencies. In this work, we define conditions on the training distribution for data balancing to lead to fair or robust models. Our results display that in many cases, the balanced distribution does not correspond to selectively removing the undesired dependencies in a causal graph of the task, leading to multiple failure modes and even interference with other mitigation techniques such as regularization. Overall, our results highlight the importance of taking the causal graph into account before performing data balancing.

Mind the Graph When Balancing Data for Fairness or Robustness
Jessica Schrouff Google DeepMind schrouff@gogge..cmm
Alexis Bellot Google DeepMind
Amal Rannen-Triki Google DeepMind
Alan Malek Google DeepMind
Isabela Albuquerque Google DeepMind
Arthur Gretton Google DeepMind Gatsby, UCL
Alexander D' D'Amour Google DeepMind
Silvia Chiappa Google DeepMind
Abstract
Failures of fairness or robustness machine learning predictive settings can be due to undesired dependencies between covariates, outcomes and auxiliary factors variation. A common strategy to mitigate these failures data balancing, which attempts to remove those undesired dependencies. In this work, we define conditions on the training distribution for data balancing to lead to fair or robust models. Our results display that, in many cases, the balanced distribution does not correspond to selectively removing the undesired dependencies in a causal graph of the task, leading to multiple failure modes and even interference with other mitigation techniques such as regularization. Overall, our results highlight the importance of taking the causal graph into account before performing data balancing.
Introduction
When training prediction models, practitioners often desire that the model's outputs display safety properties in addition to high performance, such as being fair across demographic subgroups [29]5] 511 or being robust to distribution shifts [e.g. 19 199. These objectives can be difficult to attain if there are undesired dependencies between covariates X, labels Y, and auxiliary factors of variation Z, such confounding factors or hidden stratification 26 27]. A commonly referenced example is that of an animal classification task from wildlife pictures [e.g. 644] the model might identify patterns in the background of the images that indicative of the type of animal (e.g. the presence of snow for polar bears or grass for cows), which might lead to the model failing to recognize the same animal when it is another background. When the auxiliary factors relate to demographic attributes, the deployment of such models can have societal implications, e.g. patients not being assigned medical resources due to factors related race (54).
Multiple mitigation strategies have been proposed to remove undesired dependencies pre-, in- or post-processing. Amongst them, balancing the training data is typically considered straightforward approach and has been used or researched various settings [e.g. 37 60 g
B3]
40 2] This
approach modifies the training distribution. indicated with P' Y. Z), into new, balanced distri- bution (which we refer to as Q(X, Y, Z)) that aims to approximate an 'idealized' training distribution in which the undesired dependencies distribution to attain different fairness or robustness criteria. A popular approach to construct a balanced distribution is by balancing classes (resp. groups), leading to a uniform distribution over Y (resp. 2). While successful for addressing failures of robustness [e.g. [33] or of fairness due to under-repreeentation of certain groups [e.g. (75. this approach does not induce independence between
absent 48 14 77.. Models then trained this balanced
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Y and Z. To approximate independence, 'joint' balancing on (Y, 2) is often performed [e.g. 48 g Joint balancing can be implemented by matching the numbers of samples in all (y. 2) groups (only feasible when Y and Z have small, discrete domains) via subsampling the majority groups [e.g. 81 upsampling the minority groups le.g. [3), resampling the data with weights proportional to P') (Z)/P*(Y, 2), or reweighting the loss (9. Our work focuses on joint balancing given its suitability to mitigate a marginal dependence between Y and Z =ا While the choice of the method for jointly balancing can impact the results [1] 65 33 these methods can be all seen modifying p as described in Definition [+]
Definition 1.1 (Jointly balanced distribution). We say that the distribution Q(X, Y, Z) is jointly balanced version of P'(X,Y, 2) if Q(X, Y, Z) (x, Y, (*))))((2)
P'(Y.)
In some cases, data balancing has proven to be effective mitigation strategy for undesired dependen- cies, performing on-par with other, more complex mitigation techniques [33]. Recently, data balancing has also shown promises for mitigation during fine-tuning partial retraining [4 44 49 79] 75), which is relevant to the settings of training large-scale models and with large amounts of data. Never- theless, data balancing has also displayed failure modes in which the obtained models were not fair, robust or optimal 776 48 58 2). These failure modes have not been thoroughly characterized and can be difficult to predict. Furthermore, the impact of data balancing on other mitigation strategies has not been studied extensively.
Given data balancing's popularity as baseline mitigation strategy for undesired dependencies, we aim to formalize some of its promises and pitfalls. Our analysis relies on a causal graphical framework, which allows investigating the impact of data balancing in different data generating processes. Our contributions can be summarized as follows: (1) we display failure modes of data balancing in semi- synthetic tasks and highlight how predicting these failures be challenging; (2) we introduce conditions for data balancing to attain invariance to undesired dependencies as defined by fairness robustness criteria; (3) we prove that data balancing does not correspond to 'removing' undesired dependencies from a causal perspective, and can negatively impact fairness or robustness criteria when combined with regularization strategies; and (4) we illustrate how our findings can be used to distinguish between failure modes and identify next steps.
2 Preliminaries
Let X, Y, 7 be discrete random variables with X E X' corresponding to a of covariates (e.g- tabular, images or text). Y E y to an outcome to be predicted, and Z E Z to an auxiliary factor of variation, such as a sensitive attribute or the type of background of an image, that displays statistical dependence with Y. We assume access to data sampled from distribution P P'(X, Y, 2), where p the true data-generating distribution. We consider family of models F E X + y that will be trained on data from P'(X, Y, Z) to minimize the risk Rp (f) == Ex X, X.Y) where & loss function. We define f E F to be the optimal model, i.e. one where the risk attains the minimum on Pt. We assume that | [X] = (X), which occurs, for example, if f is the square loss or cross-entropy loss.
Definition 2.1 (Optimality). A prediction model f E F is optimal w.r.t argming'eF Rp (f').
ff
2.1 Desired criteria on a model's predictions
Due to undesired independencies, while a model may be optimal on Pt, it might not be optimal on another distribution of interest P'(X, Y, Z) (e.g. in deployment). and/or might display disparities across subsets of the data (e.g. ('(X Y Z = =)) (2). To mitigate this issue, multiple safety criteria have been defined the fields of fairness and robusiness.
Fairness criteria can be defined in terms of the dependence between the model's output fX) and the auxiliary factor of variation Z. We consider established fairness criteria [5] (l). including demographic parity lf(() IL Z, (23,, equalized odds [f(X) IL Z Y. 29] and predictive parity (Y 11 Z fXX.. 24 Beyond fairness of fX), we also consider fairness of intermediate representations (xX), e.g. o(X) IL Z [I], for their usage in downstream tasks.
Fairness:
We briefly discuss group class data balancing in Appendix A.1

---

Robustness: In this field, the focus is typically on finding models fe parameterized by 0 E θ that provide the lowest risk across a family of target distributions P. For instance, the 'worst group performance" criterion aims to select parameters such that the performance on a 'worst' distribution P' is optimized, i.e. 8" mingee m Ry Rf)( 66 20) P can be defined so that each distribution P' represents specific subpopulation [64]. to minimize the loss in each subgroup, or aiming for an invariance of Hp across subgroups [risk trisk-invariane, 48).
Definition 2.2 -niiiiiivariancee. A prediction model f is iisk-invariant w.r.t. a family of distributions P if (f( = (f) VP, P' P.
If a model is optimal on Ps and risk-invariant w.r.t. P. it is also optimal w.r.t. P. The choice of P is context-specific and reflects some domain knowledge about shifts that are likely to arise in a given application. For instance, a plausible family of target distributions could imply a shift in the dependence between Y and Z, also known as correlation shift [62], and be expressed as P = P (X, Y. Z) = P Y, (' (X)
Alternatively, we can define P
using a causal framework (see Section 2.2 when the data generation process is known 48
We acknowledge that selecting amongst those criteria context context-eependent and do not advocate for specific choice. We call prediction model f invariant undesired dependencies, denoted with f E Fine, if it satisfies one of such criteria. For brevity, we focus on risk- iiskiinvariance in the main text and consider faimess criteria in Appendix. Obtaining invariant model can be performed in different ways, with data balancing being a popular approach.
2.2 Causal framework to analyse data balancing
To understand the effects of data balancing, we need to investigate its impact on the distribution Pt. A causal formalization is useful for studying how distributions change under different interventions. To analyse the implications of data balancing, we use the framework of causal Bayesian networks
A Bayesian network (55] 56 15 42) is a pair (g, Pt), in which is a directed acyclic graph whose nodes XI...,, X= represent random variables and in which p is a joint distribution over the nodes. The absence of edges in G implies a set of statistical independence assumptions satisfied by pt that can be expressed by the factorization p'(X' ........, = -I12_, p' ('Xx" Ia(("))), where pa(X") denote the parents of Xd, namely the nodes with an edge into X (we say that Pa fuctorizes according to 9). A CBN is Bayesian network in which an edge expresses causal influence, so that a) are direct causes of X A directed path between X' and X in a CBN is also called a causal path. A non-directed path, also called non-causal path, expresses statistical dependence of non-causal nature. We refer to the statistical dependence between X' and X that arises only due to the presence of non-causal paths as purely spurious. In our setting X Inspired by prior work 74 covariates X.
(CBNs) [e.g. 771 13 52
74
25 48
X X Y U Z U U where U are unobserved variables. we make a decomposition assumption on the form of the
70
77).
Assumption 2.3 (Decomposition of X). The covariates X can be decomposed into three unobserved random variables has causal paths to/from Y.2) 2) X does not have causal paths to/from Y but has causal paths to/from Z, 3) XYAZ has causal paths to/from both Y and Z, representing entangled signals, and 4) X is measurable w.r.t. oXx 7. L X XYxz), the joint o-algebra. In particular, there exists a function g such that X = g( X XYnz) almost everywhere and pt ( P' ('(x(X,, X XYAZ), Y, Z, U).
and Xy such that: 1) X does not have causal paths to/from Z but
Y, Z, U) =
In the animal classification example, X would correspond to the animal pixels, X to the back- ground pixels (e.g. snowy or grassy landscape), and XY to characteristics of the animal that depend on its environment (c.g. color of the fur pixels in bears). Intuitively, we want to build a prediction model ff that only depends on the animal pixels. While the decomposition may be readily available when a causal graph of the application is available and the data is tabular, we typically do not have direct access to the different functions of X and these need to be isolated algorithmically.
Following Schölkopf et al. (66), we consider both the case in which X U XYAZ are direct causes of the label Y (causal task) e.g. estimating the helpfulness of a text review, and the case in which Y a direct cause of X U XYAz (anti-causal task) as in object detection tasks in computer vision. Figures I b) display examples of anti-causal and causal tasks with a purely spurious dependence
3

---

Graph
Data Balancing
Regularization
Next steps
X
(8)
riikkinnvariant / /optimal
f(X) IL Z|Y
N.A.
/ riisk-invariant / /optimal
Xy
X
(b)
/ risk-invariant
(XX L Z /risk-invariant /optimal
Use regularization, without prior balancing (as per Section 5)
Xoptimal
Xy
Xy
(c)
U2
Xrisk-invariant Xoptimal
(X) IL Z|Y Xrisk-invariant Xoptimal
Refer to Kaur al., 2023; Alabdulmohsin al., 2024 which address the cases with multiple auxiliary factors.
U3
U1
2
Y
X
Xrisk-invariant
f(X) Z Z
Use regularization
Xoptimal
/risk-invariant /optimal
XYAZ
Table I: Examples of causal Bayesian networks with undesired dependencies between Y and Z displayed by red edges. Light gray indicates unobserved variables. Xy = ( in (a-b) and there no entanglement between Y and Z via X. In (c), expand the system to include V U and its influence For cach Causal Bayesian Network considered, display when data balancing leads to risk-invarian and/or optimal model. We compare these with regularization following Veitch et al. [74] and suggest next steps.
X, which is given by Xv.
between Y and Z. It is important to note that statistical relationships between the different variables and functions of X are determined by the graph: for instance, in Figure Ta) X IL Y, while in Figure bb) X Z.
Based on a CBN of the task and Assumption 2.3 we characterize undesired dependencies as the presence of undesired paths between Z and which we indicate through red edges (Fig- ure []. Based on this depiction of undesired dependencies, we can define the family of target distributions P such that black edges are preserved, but those in red may lead to changes in the distribution. For the anti-causal task in Figure a)), we can hence write P = (Y,
(' (Z Y) (X (X ])} in which P' (Z Y) represents any distribution but all other causal mechanisms are fixed 48 which corresponds correlation shift.
3 Can we predict when data balancing fails?
As reported previously, data balancing can display failure modes, e.g. due to the presence of other confounders 76 2), finite sampling effects (48) or a dependence between and Z when conditioning on X (Y JL Z X) 58). However, this list is non-exhaustive and, to the best of our knowledge, there no unifying study of those failure modes or of how they could be mitigated. In this section, we perform joint data balancing different tasks illustrate that successes and failures of this approach can be difficult to predict (see Table For details of the experiments, see Appendix D
Let's first consider semi-synthetic examples generated from the graphs in Figure [a.b). i.e. an anti-causal and causal task with a purely spurious correlation. We aim to obtain a risk- invariant and optimal model these tasks by training on the jointly balanced distribution Q.
Anti-causal task: number detection in MNIST, Inspired by Brown et [8, we modify MNIST images 45 17 by adding factor of variation Z such that the top of the image is replaced by red noise for Z and blue noise for = 1 (Figure I]. We sample dataset in which the factor of variation and label are dependent (P' (P'() = 0 Z 0) 0.95, P'(Y = 1 0) = 0.10, called 'confounded" data), a jointly balanced dataset, and dataset from a distribution po in which the undesired dependency is absent (P" 0 Y) 0.5). We train convolutional networks to predict whether the number in an image smaller larger than 5, assessing the models on their training distribution and on p0

---

Table 2: Model performance on semi-synthetic data, for the tasks in Figure 1. 'Acc' refers to accuracy, 'Worst Grp' worst group accuracy, 'Encoding' confounder encoding as measured by transfer learning and 'qqu. Odds' refers to equalized odds between subgroups. t (resp. 4) means the higher (resp. lower), the better.
Graph
Dataset
p
po
Acc. (T)
Acc. (1)
Woest Grp (1)
Eacodiig (- 0.5)
Equ. Oddds (J)
(a)
95/10
0.937 4: 0.002
0.717 4 0.027
0.380 4: 0.062
0.99% 4: 0.004
0.530 4: 0.015
Figure MNIST
I:
(a)
Balanced
0.871 1: 0.008
0.850
0.006
0.836 0.075
0.486 0.005
0.018 0.008
data
(b)
Class bal.
0.635 + 0.015
0.558
= 0.013
0,092
= 0.013
0.0%0 = 0.113
0.512 = 0.058
samples.
Jointly bal.
0.574 + 0.016
0.583 + 0.017
0.390
+ 0.014
0.545 + 0.037
0.000 + 0.046
(c) With V
0.769 1: 0.001
0.647
0.023
0.555
0.031
0.665 1 0.134
0.094
1 0.035
(d)
Entangled
0.903 +: 0.011
0.672 - 0.004
0,000 2 0.001
0.881 = 0.223
0.554 +: 0.028
Models trained with confounded data (95/10) display biased outputs (Table 2 with low worst group performance and high equalized odds. Performance pd is also lower compared to that on ps (0.937 + 0.002), showing that these models are not risk-invariant w.r.t. P. Models trained from balanced data obtain high overall performance and worst group accuracy, as well as low equalized odds. In addition, we were not able to decode Z from the model representation ((X) at the penultimate layer, suggesting that the model has not learned Xy.
helpfulness of reviews with Amazon reviews [53]. Inspired by Veitch et al. 74 we refer to the causal task of predicting the helpfulness rating of an Amazon review (thumbs up or down, Y) from its text (X). We add synthetic factor of variation Z such that words like 'the' or 'my" are replaced by 'thexxxx' and 'myxxxx' (Z- = 0) or 'theyyyy' and 'myyyyy' = 1). We train a BERT 34 model on class-balanced version of the data for reference (due to high class imbalance), and compare to a model trained on jointly balanced data, both evaluated on their training distribution and on distribution p6 with no association.
Causal task:
In this case, jointly balancing improves fairness and risk- iis-iinvriiacce, with the model's performance on the training distribution (acc.: 0.574 + 0.016) being similar to that on po (Table This however comes at high performance cost when compared to the class balanced model's performance on p (ace: 0.658 + 0.015). Therefore, data balancing might not lead to optimality for this causal task.
Using the same framework, we can replicate the failure modes due to another confounder described in Wang et al. 776), Alabdulmohsin et al. 22 as well as that from Puli et al. [58).
Anti-causal task with another factor of variation V. It is common for multiple auxiliary factors to influence the data generating process, and they tend to correlate with each other [c.g.21] 21]. To emulate this case, we introduce more unobserved variables U2, U3 as well as a factor of variation V which affects the data through Xy (Figure [().. We modify the MNIST data generation to include Xy depicted by green cross the top left or top right of the image and jointly balance the data on (Y, Z) before training the model. We evaluate the obtained predictor on a distribution where V and Z are not correlated with Y and observe (Table large gap between worst group accuracy and overall performance, as well as non-null equalized odds. These results suggest that the model is not fair or robust.
Anti-causal task with entangled data. We map the work in Puli et al. (58] to our decomposition of X and propose the example graph in Figure d()) where XY represents an entangled function of X. To match this data generating process, the color of the noise in MNIST samples is defined by OR(Y, Z) and the evaluation distribution is the disentangled pa with no dependence between Y and
Z. Once again, the obtained model is not fair, robust or optimal (Table 2). Appendix A.2 discusses this case further.
Motivated by these examples of both success and failures, we define conditions for the success of data balancing, and highlight when the cases above fail to meet these conditions.
4 Conditions for data balancing to produce an invariant and optimal model
In this section, we introduce sufficient condition on the data generative process and a necessary condition on the trained model that, taken together, lead to a risk-invariant and optimal prediction model after training on Q (proofs in Appendix B.I. In Appendix B.2 we derive similar conditions
5

---

for fairnsss criteria. Throughout the rest of the paper, we use an subscript to indicate under which of ps or Q statistical independence holds, e.g. Y pt to indicate PY Z) = PY)
We consider the criterion of invariane (Definition 2.2) under correlation shift, i.e. P (P'(X,Y,X) PX P) (Z)Y))))))))) According to decomposition of X, the risk- minimizing function f(X)= := E X] should only be function of X and not of X or Xy To achieve this result with data balancing, we build on a prior result by Makar et al. [48], which shows that a model trained statistic for Y, i.e. no other part of X influences! Y.
balanced distribution only depends on X 2 if X represents a sufficient
Definition 4.1. (Sufficient Statistic) We say that X a sufficient statistic for Y in E EQ X / (note that X & is a function of X).
Definition 4.1 implies that the risk-minimizing function f for Q does not vary with X Xy However, this condition is not sufficient on its own to ensure that f is risk-invariant w.r.t. P, as X or Y may have non-causal relationships with Z. To ensure optimality and risk-invariance w.r.t. P. we derive the sufficient condition in Proposition 4.2
Proposition 4.2. If Z Le Z Y and x is sufficient statistic Y in Q, then the risk-minimizer S(X) == Eo Y X] is risk-invariant and optimal w.r.l. P.
The conditions of Proposition 4.2 concern Q. However, it would be of interest to express them in p it is possible to observe all covariates (e.g. in the case of tabular data). Based on our expression for Q, we can derive sufficient conditions on Pt, expressed in Corollary 4.3 Let's denote (X+, XYaz}
by R.
Corollary 4.3. I pp 1 and X pr Z ZIY, then the risk-minimizer S(X) Eo X] is risk-invariant and optimal w.r.l. P.
In general, we can expect that anti-causal tasks with purely spurious correlations will satisfy these conditions, as per their definition. However, this would not be the case for most causal tasks as X + p. Z Y. This result in line with our findings in Section as the MNIST data generated from the graph in Figure ([) validates Corollary 4.3 but the Amazon reviews data generated from Fib does not.
It may be less obvious, but the conditions for a sufficient statistic are not met in Figures Tc,d) as Xv A yY x Z in the case of another factor of variation V, and Xy H pt (Y, x2) Z in the case of entangled data. We hence see that when a causal graph of the application is available, Corollary 4.3 can provide indicators on when data balancing might succeed or fail, with the caveat that it is not a necessary condition.
While Proposition
and Corollary 4 provide conditions the data generating process, prior work [c.g. 10 31 has demonstrated that the learning strategy also influences the model's fairness and robustness characteristics.
Proposition 4.4. Let f E F be some fitted model and e > O. Assume that, for all P', P" E P, we have Ep Y (((X Ep [Y| X=1 S G. Then f is e-risk invariant, meaning that
Rr( Ry (f)se. Se.
sup P',P"EP
Proposition 4.4 states that the learned function ff needs to be nearly optimal over P. This statement, while straightforward, implies that (i) f(X) needs to preserve all the information about the expec- tation of Y in X and that (ii) f(X) changes with X or XY only marginally. Let's rewrite f(X) = h((()). where h is 'simple' function and o(X) is a model representation. This case could correspond to the last layers of a neural network or when learning model based on representation ((X) (e.g. embeddings, transfer learning). Based on Proposition 4, of X) must be disentangled in the sense that the simple function h eliminates any dependence on X or XYAZ. For example, is a linear function, it must be possible to linearly project out all dependence on X and XY We note that such representation can be obtained even if the data is entangled, e.g. by dropping modes of variation during training. Unlike other strategies this property on its own and disentangled representation would be necessary. This condition hence
58 data balaneing cannot enforce

---

1.2 X2
10
1.0
Entangled
00
0.0
0.0
0.
2.5
0.4
04
044
02
02
0.2
02
0.
00
Figure 2: Accuracy across different values of the MMD hyper-parameter for models trained on balanced data and evaluated on their respective training distribution (dashed) and pd (solid line) averaged across replicates. We consider anti-causal tasks: (left) purely spurious case, (middle) when another confounder V is present, and (right) the entangled dataset. Worst group performance on 1d is displayed in red. Markers display individual replicates.
suggests another failure mode of data balancing when the conditions on the data are validated, but the representation is of low quality. We believe this failure mode displayed in Kirichenko et al. [4]] as the success of their data balancing mitigation only holds when using models pre-trained on large datasets.
In this section, we have identified conditions for data balancing to be successful. In the next section, we go one step further to understand how data balancing impacts the data generating process, and how it interacts with other mitigation strategies for undesired dependencies, focusing on regularization.
5 Impact of data balancing on the CBN
Joint data balancing assumed to remove statistical dependence between Y and Z while keeping other relationships in the CBN of the task unaffected [e.g. 48 77 14]. This could be interpreted as 'dropping' edges in the undesired paths in G. e.g. removing the influence of U on Y and/or Z in Figure Ia), leading to new graph go. While this interpretation is correct for joint balancing in the case of Figure[[a), a.), (ropositionn5.Jbeelli
(proof in Appendix () shows that it can be erroneous general: the distribution Q underlying the balanced data might not factorize according to G" and therefore might not obey the statistical dependence relationships implied by Go. Therefore, balancing data to make Z and Y statistically independent, i.e. selecting samples in proportion to (*)) (Y) (Y)/P'(2) Y), not equivalent to generating data from a distribution that factorises according to g in general. This factorization is important because downstream distributions P' (X, Y, Z) are often assumed to follow this factorization; in fact, this assumption underlies a number recommendations for applying regularization methodologies such as in (74].
Proposition 5.1. Let (g, Pt) be the CBN underlying the data, where G contains an undesired path between Z and Y, and let go be modification of G in which the undesired path has been removed. The distribution Q obtained by jointly balancing the data need not factorize according to go.
Proposition shows that statistical (in)dependencies that we assumed would remain fixed (i.e. the black edges on the graph) can be modified by the process of joint balancing. As consequence. further interventions on Q (e.g. the addition of regularizer) should not be motivated by Go, and we show below that combining data balancing with other mitigation strategies can lead unexpected results.
5.1 Data balancing can hinder regularization and vice-versa
When confronted with failure mode, it is reasonable to ask whether additional fairness or robustness regularizer on the training loss might be beneficial. Based on Proposition [5.]] we see that this question might have diffferent answer if we are in ps or in Q. Below, we consider each failure mode and ask whether performing an additional regularization motivated by the literature would mitigate the undesired dependencies in Q. The results summarized in Table [ with suggested next steps. In Appendix C.1.2. we discuss when balancing with regularization is sufficient for different fairness criteria.
Anti-causal task. In the case of an anti-causal task with dependence between and (Figures (a,,,d)). Veitch et al. (74) recommend to impose an independence between f(X) and Z conditioned
7

---

(a) Trained on pt
(b) Trained on Q
(c) MMD=16
Cofiradde
Figure 3: Accuracy across different values of the confounder strength (i.e. different P' P), for each value of MMD regularization considered (displayed by the color gradient). (a) Models trained on P', (b) Models trained on Q. Results are averaged across seeds for clarity. Notice the different y-scales. (c) Displays the mcan and standard deviation across seeds for MMD=16.
on Y. If we consider both the purely spurious correlation and the entangled case, we see that regularization and data balancing would have the same effects of blocking Py dependence between (Y.x*) and ( X We demonstrate that X2 IL. 2 Y in both and Q (see Appendix C.1p. p. and this regularization is sensible under both distributions. This means that performing the regularization provides the sufficient conditions for a risk-invariant model, whether or not joint data balancing performed. In theory, data balancing not needed but is also not harmful. In the case of an added confounder, we have that Xv depends on both Y and Z due to non-causal paths through V. Therefore, imposing that f(X) O Z Y might lead results whereby the model only depends on V or is trivial (e.g. predicts a constant) as the regularization encourages the removal of any dependence on Z, which related to Y via Xv. This behavior would be observed in both Ps and Q. but data balancing on its own might be less detrimental than regularization in terms of predictive power even though it does not resolve all undesired dependencies. In this case, regularization hinders data balancing.
Based on the balanced data from Section 3 we add conditional Maximum Mean Discrepancy [MMD, (8) to encourage f(X) Me 2 2 during training, varying the strength of this regularizer via a hyper-parameter. In the case of the purely spurious statistical dependence between Y and Z (Figure Ta)). there is little variation between the metrics across MMD strengths, and the model is fair and robust (Figure leeft)) In the entangled case (Figure right)), the model's performance on Q and pd are close for medium values of the hyper- parameter (before MMD overpowers the training) and worst group performance improves markedly. This result suggests that, with the added regularizer, f onlly varies with X²). ). Performing the same regularization in the presence of another confounder (Figure 2 middle)) leads to plateau in performance on Q, but low performance on po and chance-level worst group performance. In this case, we posit that the model relies exclusively on Xy for its predictions, and the regularizer is detrimental compared to data balancing on its own (MMD=0 on the plot).
Causal task. Finally, let us consider the causal task in Figure [b). In similar case, Veitch et al. [74] suggests a regularizer such that f(X) pp Z, which would encourage the model f to vary only with X as X Il pt Z. However, data balancing induces a dependence between X and Z, as expressed below:
Exp,y X 22
Q( 2) = Expxyy P' (2
Σ P' ZYY)
The RHS cannot be simplified further because 2 Ar Z Y, because Y is a collider under Pt. Thus, the left hand side is function of Z in general (see Appendix C. for further details and a numerical simulation). In this case, regularizing to enforce s(X) Q Z would destroy information in X whereas the same regularization under p would have enabled f(X) to use all of the information in X2- Therefore, data balancing may hinder regularization.
We illustrate this result on the Amazon reviews dataset from Section 3 by imposing a marginal MMD regularization f(X) 11 Z during training and evaluating risk-invariance across multiple P' P. When training on Pt, we observe that the regularization allows to 'flatten' the curve, such that from
8

---

1.
Table 3: VGG model performance on CelebA, when trained on p
Q. with ImageNet tre-training ('Pre-trained") Q. with MMD
0.6
('MMD') on Pe with segularizer=5. All models evaluated Q.
Model
Acc. (1)
Worst Grp (1)
Encoding (- 0.5)
Equ. Odds (1)
29.99 99 6 P.
ERM on p
0.791 0.037
0.314 0.083
0.888 + 0.015
0:243 0.036
MMD
ERM on 4
0.839 = 0.022
0.674 4: 0.088
0.709 4: 0.066
0.125 4: 0.022
Fre-trained on Q
0.006
0.726 0.037
0.7 0:033
00111000:00
MMD on p
0.813 = 0.036
0.146 + 0.172
0.630 + 0.010
0.001 0.002
Figure 4: Model performance on test sets sampled from P (dotted) and Q (dashed). The model trained P' with regularization f(X) Y.
medium high values of MMD regularization, the model is risk-invariant (Figure a)). On the jointly balanced data, medium values of the regularization degrade risk-invariance (see green curves on Figure bb). Overall, model performance is also lower for the models trained on Q compared to models trained on p across test sets from P E P, at similar levels of regularization (see Figure 3 for MMD=16). This result displays that X / is a sufficient statistic for Y in Q.
Case study: distinguishing between failure modes in CelebA
In this section, we show that when Y and Z are available training time, can try to distinguish between failure modes of data balancing by using our different observations, even in the absence of a full causal graph. We illustrate this using the benchmark task of detecting blond hair in pictures of celebrities in the CelebA [46] dataset. This label has strong correlation with perceived gender: half of the non-males have blond hair, while only 7% of males do. We consider balanced, subsampled dataset (train: = 4, 096, test/valid: = 400) and the original, confounded dataset. We train a VGG (68 and four Vision Transformer [ViT. 18 architectures, with number of parameters ranging from 17 to 690 millions.
We observe that, while training with balanced data leads higher worst group accuracy and lower equalized odds scores than training with the historical data (Table B an important gap remains between the overall and worst group performances. These results show that data balancing leads to improvements in downstream fairness and robustness metrics, but does not provide risk-invariant or fair model on its own. Therefore, it likely that one of the conditions for data balancing to be sufficient is not fulfilled and understanding which condition is violated can guide our selection of another technique.
Distinguishing between failure modes. We first assume that the task is anti-causal. We then aim to understand whether there is another confounder, the data is entangled, or the representation is entangled (Proposition 4.4). As per Kirichenko et al. [41), we first attempt to improve our representation by pre-training the VGG with ImageNet (16). While
observe an increase in performance with pre-training, there is no clear decrease in equalized odds. This result suggests that the failure may lie elsewhere. We then train models with MMD on pt, with the expectation that we would observe a plateau for entangled data when the model learns f(X²), or stark decrease worst group performance in the presence of another confounder. While there is no major pattern of correlation between Y and another attribute in the balanced data (see Appendix E.2.2). small effects might combine, or there might be other, unobserved attributes that influence Y. For medium value of the regularization hyper-paramedte worst group performance. This result suggests an effect of another confounder and next steps can include methods such as Alabdulmohsin et al. [2]. which controls for all (observed) auxiliary factors of variation.
the model displays plateau in performance and poor
7 Related works
Balanced data as mitigation for invariant models. Our results extend those of Makar et 48 which considered a single causal graph. Wang et al. (76) displayed that balancing data did not lead to

---

reduction in bias amplification. The authors posit that this failure of balanced data to correct for spurious signals due to unobserved confounding factors which is confirmed Alabdulmohsin et al.
[]. Rolf et al. 63 investigated upsampling by relying on a scaling law per group, focusing on the
question of fairness vs performance trade-off [22]. Focusing on causal NI.P settings, Joshi et al. [36]
investigated causal and non-causal features, concluding that data balancing does not help in all cases.
Closer to our work is that of Puli et al. [58], in which the authors showed that having Y LL Z does not imply that Y Oo Z| X and the model can learn signals related to Z. Puli et al. [58] propose a method to learn a representation r such that Y IL Z rx)). Our work provides a framework to understand these different failure modes and proposes strategies to distinguish between them. While we focus on pre- pre-processing mitigation with a fixed distribution Q(X, Y. Z), another line of work considers dynamic resampling in- processing [e.g. 35 [2]. As the resampling converges towards a fixed distribution YY) we would expect failure modes in the presence of entangled data or of another confounder. Nevertheless, the variation in P')) at the early stages of training might be beneficial, e.g. by disentangling the representation. We leave this investigation for future work.
Causal feature selection. Some works have used causal framing to select features such that f(X) has robustness and/or fairness properties [e.g. 47 69 25 67 38), . Similarly, our work defines independence conditions on covariates to obtain an optimal, invariant model, and can be used to select features. Two major distinctions between feature selection works and ours reside in the fact that we consider the case in which we do not observe X explicitly and that investigate the impact of data balancing.
8 Discussion
In this work, we uncover important results to guide the use of data balancing for mitigating undesired dependencies between covariates, outcomes and auxiliary factors of variation. We first show (Section 3) that joint data balancing might not achieve the desired fairness or robustness criteria, and that the failures may seem difficult to predict. Motivated by these results, we introduce conditions under which data balancing leads robust or fair model (Sections 4 B.2). Importantly, we show that data balancing is not equivalent to "dropping an edge' in the causal graph and can lead to distributions that do not factorize according to the desired graph (Section [ This can have downstream consequences if further mitigation strategies are motivated by the causal graph and highlights why regularization and data balancing might not go 'hand in hand'. This last result shows that data balancing should not be performed as 'default', and mitigation strategies should be based on the causal graph of the application. Finally, even in the absence of a causal graph, our findings may help to pinpoint which condition(s) are not fulfilled, and guide further mitigation (Section 6).
Limitations. The conditions defined in Section for invarianee depend on the expression of P as a correlation shift [48 62). Other expressions or shifts are likely to lead to other conditions. In our experiments, we have mostly subsampled datasets to obtain balanced distributions. We would expect similar results for other joint balancing methods. Variations are, however, possible due to the finite-set set nature of the computations (48, e.g. with reweighting displaying more variance [ potentially under-performin aimed to provide upper bounds for the effectiveness of data balancing, we did not use additional training strategies for mitigation beyond regularization. We believe that our causal framework can be a useful tool to analyze other pre- or in-processing methods that enforce independence between variables in the data generating process [e.g. [ 58). On the other hand, our framework might not be suited to analyze the effects of other mitigation strategies, c.g. hyper-paramette optimization [57]. We discuss the broader societal impacts of our work in Appendix E.2.2
in overparametrized settings [1I 65]. We also note that, while we
Future work. This work considered variety of causal graphs in order to provide general insights rather than task-specific conditions. However, investigating specific graphs could enable to leverage further strategies including other balancing techniques [e.g. 38 [7].. We believe that our causal framing could then be a useful resource to analyze the effect of these strategies on downstream fairness and robustness criteria. Finally, we illustrate our propositions with binary classification tasks and confounders. While our reasoning applies to more complex settings, there might be further considerations to account for when generalizing beyond binary variables, especially with respect to estimation.
10

---

Acknowledgments and Disclosure of Funding
We thank Virginia Aglietti for feedback on this work and Victor Veitch for sharing experimental code for the Amazon reviews experiments. This work was funded by Google DeepMind.
References
[1] Alabdulmohsin, and Lučié, M. A near-optimal algorithm for debiasing trained machine learning models. Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.).
Advances in Neural Information Processing Systems, 2021. URL https///apperreveeeete/oonmn (d=HSTBQNFPKSJ
[2] Alabdulmohsin, I,, Wang, X., Steiner, A., Goyal, P., D' A'mour A., and Zhai, X. CLIP the bias: How useful is balancing data in multimodal learning? In International Conference Learning Representations, 2024.
[3] Anthis, J. R. and Veitch, V. Causal context connects counterfactual fairness to robust prediction and group fairness. In Advances in Neural Information Processing Systems, volume 37, 2023. URL. htt.://pp//oopeeeeeeeeeee/////o/ wttp://wwwwe/ffoommmiiiii http://orrrmmmdddaamiiiiiieppp
[4] Arjovsky, M., Bottou, L., Gulrajani, L., and Lazzz-Paz, D. Invariant risk minimization, 2019.
Preprint 1907.02893. URL L http:://aaxiv.org//aaaaa 1907.02893
[5] Barocas, S., Hardt, M., and Narayanan, A. Fairness and Machine Learning: Limitations and Opportunities. MIT Press, 2023.
[6] Ben-Tal A., den Hertog, D., De Waegenaere, A., Melenberg, B., and Rennen, G. Robust solutions of optimization problems affected by uncertain probabilities. Manage. Sci., 59(2): 341-357, 2013.
[7] Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. JAX: composable transformations of Python+NumPy programs, 2018. URL http://gittub.com/gooogg/ea
[8] Brown, A., Tomasev, N., Freyberg, J., Liu, Y., Karthikesalingam, A., and Schrouff, J. Detecting shortcut learning for fair medical AI using shortcut testing. Nat. Commun., 14(1):4314, 2023.
[9] Byrd, J. and Lipton, Z. What is the effect of importance weighting in deep learning? In Chaudhuri, K. and Salakhutdinov, (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 872-881. PMLR, 2019.
[10] Carlini, N. and Wagner, Towards evaluating the robustness of neural networks. 2017 IEEE Symposium Security and Privacy (SP), pp. 39-57. IEEE, 2017.
[11] Celis, E., Keswani, V., Straszak, D., Deshpande, A., Kathuria, T.,, and Vishnoi, N. Fair and diverse DPP-based data summarization. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 716-725. PMLR, 2018 URL htttss:////proooeednng mlr. vrssprress/80//celiss html
[12] Chen, X., Fan, W., Chen, J., Liu, H., Liu, Z., Zhang, Z., and Li, Q. Fairly adaptive negative sampling for recommendations. In Proceedings of the ACM Web Conference 2023, WWW '23, pp. 3723-3733, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 978 1450394166. doi: 1/444/3543357.35835555 URL https://bb.oog//0111445555333333
3583355
[13] Chiappa, S. Path-Specific counterfactual fairness. AAAI, 3300)))7800117788 2019.
[14] Compton, R., Zhang, L., Puli, A., and Ranganath, R. When more less: Incorporating additional datasets can hurt performance by introducing spurious correlations, 2023. Preprint 044311
2308.04431. URL http://arxxiv.or//bb//222
[15] Cowell, G., Dawid, A. P., Lauritzen, S., and Spiegelhalter, D. J. Probabilistic Networks and Expert Systems, Exaci Computational Methods for Bayesian Networks. Springer-Verlag. 2007.
[16] Deng, J., Dong, W., Socher, R., Li, L.JJ., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255. IEEE, 2009.

---

[17] Deng, L. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine. 29(6):141-142, 2012.
[18] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth
16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021. URL httpps:/ppenreeiiww.ct//ooooonit YicbbaaaNTY
[19] Drenkow, N., Sani, N., Shpitser, I., and Unberath, M. systematic review of robustness in deep learning for computer vision: Mind the gap?, 2021. Preprint 2112.00639, URL http://aaxii.orr//bbb/112..
[20] Duchi, J., Glynn, P., and Namkoong, H. Statistics of robust optimization: A generalized empirical likelihood approach, 2016. Preprint 1610.03425. URL. http://arxiv.or//bb///100
03425
[21] Duffy, G., Clarke, S. L.., Christensen, M. He, B., Yuan, N., Cheng, S., and Ouyang, D. Confounders mediate Al prediction of demographics in medical imaging. NPJ Digit Med, 5(1): 188, 2022.
[22] Dutta, S., Wei, D., Yueksel, H., Chen, P.-Y., Liu, S., and Varshney, K. Is there a rrade-off between fairness and accuracy? A perspective using mismatched hypothesis testing. In III, H. D. and Singh, (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 199 of Proceedings of Machine Learning Research, pp. 2803-2813. PMLR, 2020. URL https//prooceedings wwwp:eeedinng......eee.... mlr.sssss/vvl 19/dutta20.hhtm
[23] Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS '12, pp. 214-226, New York, NY, USA, 2012. Association for Computing Machinery. ISBN URL https::/doi..rr//100/ http://do...rg/1...11//222222222 145/2090236
9781450311151. doi: 10.1 00.1145/2002236.2090222
2090255
[24] Flores, A. W., Bechtel, K., and Lowenkamp. C. T. False positives, false negatives, and false analyses: A rejoinder to "machine bias: There's s software used across the country to predict future criminals. and it's biased against blacks.". . Fed. Probat., 80(2), 2016.
[25] Galhotra, S., Shanmugam, K., Sattigeri, P., and Varshney, K. R. Causal feature selection for algorithmic fairness. In Proceedings of the 2022 International Conference on Management of Data, SIGMOD $22. pp. 276-285, New York, NY, USA, 2022. Association for Computing doi: 10.1
Machinery. ISBN 9781450392495.
/ttp:stts:///ooo.orr
1145/351422 12111351211111111 3
[26] Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, A., and Brendel, W. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accu- racy and robustness. In International Conference on Learning Representations, 2019. URL http://wwww.eeetooummdddd--eaeee
[27] Gichoya, J. W., Banerjee, I., Bhimireddy, A. R., Burns, J. L.,, Celi, L. A., Chen, L.-C., Correa, R., Dullerud, N., Ghassemi, M., Huang, S.-C., Kuo, P.-C., Lungren, M. P., Palmer, L. J., Price, Kalantari, L., Trivedi, H., Wang, R., Zaiman, Z., and Zhang, H. Al recognition of patient race in medical imaging: a modelling study. Lancet Digit Health, 4()) 2022.
J., Purkayastha, S., Pyrros, A. T., Oakden-Rayner, L., Okechukwu, C., Seyyed-Kalamtari
[28] Gretton, A., Borgwardt, K. M., Rasch, M. J., and Scholkopf, B. A kernel Two-Sample test. J.
Mach. Learn. Res., 13(25):723-773 2012.
[29] Hardt, M., Price, E.,, Price, E., and Srebro, N. Equality opportunity in supervised learning. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Infor- mation Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https://proceeddnng neurips.ccpppperfflsssspppp
[30] Harris, C. R., Millman, K. J., van der Walt, J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E.,, Taylor, J., Berg, S., Smith, N. J., Kern, R., Pieus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., Del Río, J. F., Wiebe, M., Peterson, P.,
P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H. Gohlke, C., and Oliphant, T. E. Array programming with NumPy. Nature, 585(7825):357-36 2020.
[31] Hooker, S., Moorosi, N., Clark, G., Bengio, S., and Denton, Characterising bias in compressed models, 2020. Preprint 2010.03058. URL http:///arrv..oog//////000000000
12

---

[32] Hunter, J. D. Matplotlib: A 2D graphics environment. Comput. Sci. Eng., 9(3):90 95, 2007.
[33] Idrissi, B. Y., Arjovsky, M., Pezeshki, M., and Lopez-Paz, D. Simple data balancing achieves competitive worst-group-accurage
In Schölkopf, B., Uhler, C., and Zhang. K. (eds.), Proceedings of the First Conference on Causal Learning and Reasoning, volume 177 of Proceedings of Machine Learning Research, pp. 336-351, PMLR. 2022. URL wttp://ddnngg.m...rrsss///////ee
[34] Devlin, M.-W. Chang, K. L. and Toutanova, K. Bert: Pre-training of deep bidirectional
transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), volume 1, pp. 2, 2019.
[35] Jiang, H. and Nachum, O. Identifying and correcting label bias in machine learning. In Chiappa, S. and Calandra, R. (eds.), Proceedings of the Twenty Third International Conference Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 702-712. PMLR, 2020. URL httpss//prooceedings.. www.press.v./////. 108/jann22a.htmm
[36] Joshi, N., Pan, X., and He. H. Are all spurious features in natural language alike? an analysis through a causal lens. In Empirical Methods in Natural Language Processing (EMNLP), 2022.
[37] Kamiran, F. and Calders, T. Data preprocessing techniques for classification without discrimi- nation, Knowl, Inf. Syxt, 33(1):1 1-33, 2012.
[38] Kaur, J. N., Kiciman, E., and Sharma, A. Modeling the data Kata-generating process is necessary for out-of-distriiutiin generalization. In The Eleventh International Conference on Learning Representations, 2023. URL
[39] Kehrenberg, T., Chen, Z., and Quadrianto, N. Tuning faimess by balancing target labels. Front Artif Intell, 3:33, 2020.
[40] Kim, D., Park, S., Hwang, S., and Byun, H. Fair classification by loss balancing via fairness- aware batch sampling. Neurocomputiing 518:231-241, 2023.
[41] Kirichenko, P., Izmailov, P., and Wilson, A. G. Last layer re-training is sufficient for robustness
to spurious correlations. In The Eleventh International Conference on Learning Representations, http::///wwneeeeeeee//////////// 2023. L
[42] Koller, D. and Friedman, N. Probabilistic Graphical Models: Principles and Techniques. MIT Press, 2009.
[43] Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Pereira, F., Burges, C., Bottou, L., and Wein- berger, K. (eds.), Advances in Neural Information Processing Systems. volume 25. Cur- ran Associates, Inc., 2012. URL Paper.pdf (at.2222b99966677666646224222522
[44] LaBonte, T., Muthukumar, V., and Kumar, A. Towards last-layer retraining for group robustness with fewer annotations. In Thirty-seventh seventh Conference on Neural Information Processing Systems, 2023. URL http:///wwnweoeeeeee//////o/m//a
[45] Lecun, Y., Bottou, L., Bengio. Y., and Haffner, P. Bradient-based learning applied to document recognition. Proc. IEEE, (6(11):2278 22224, 1998.
[46] Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In 2015 IEEE International Conference on Computer Vision (ICCV). IEEE, 2015.
[47] Magliacane, S., van Ommen, T., Claassen, T., Bongers, S., Versteeg, P., and Mooij. J. M.
Domain adaptation by using causal inference to predict invariant conditional distributions. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[48] Makar, M., Packer, B., Moldovan, D., Blalock, D.,, Halpern, Y., and D' Amour, A. Causally motivated shortcut removal using auxiliary labels. In Camps- Valls, G., Ruiz, F. J. R., and Valera, I. (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 739-766. PMLR, 2022. URL http:ss/pproceeengg..iiit I I/m
[49] Mao, Y., Deng, Z., Yao, H., Ye, T., Kawaguchi, K., and Zou, J. Last-layer fairness fine- tuning is simple and effective for neural networks, 2023. Preprint 2304.03935. URL http
13

---

[50] McKinney, W. Data structures for statistical computing in python. In Proceedings of the 9th Python in Science Conference. SciPy, 2010.
[51] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., and Galstyan, A. A survey on bias and fairness in machine learning. ACM Comput. Surv., 54(6):1-35, 2021.
[52] Mooij, J. M., Magliacane, S., and Claassen, T. Joint causal inference from multiple contexts. J. Mach. Learn. Res., 21 (99)): 1-108, 2020.
[53] Ni, J., Li, J., and McAuley, J. Justifying recommendations using distandy-labeled reviews and fine-grained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference natural language processing (EMNL.P-JCCNLP), pp. 188-197, 2019.
[54] Obermeyer, Z.., Powers, B., Vogeli, C., and Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464):447-453,, 2019.
[55] Pearl, J. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann Publishers Inc., 1988.
[56] Pearl, Causality: Models, Reasoning, and Inference. Cambridge University Press, 2000.
[57] Perrone, V., Donini, M., Zafar, M. B., Schmucker, R., Kenthapadi, K., and Archambeau, C. Fair bayesian optimization. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 854 854-863, 2021.
[58] Puli, A. M., Zhang, L. H., Oermann, E. K., and Ranganath, R. Out-of-distributin generalization in the presence of nuisance-induced spurious correlations. In International Conference on Learning Representations, 2022. URL http:://pwweweoeeeeeeee/ouunadaa
[59] Quinonero-Candela, J., Sugiyama, M., Schwaighofer, A., and Lawrence, N. D. (eds.). Dataset shift in machine learning. Neural Information Processing series. MIT Press, London, England, 2022.
[60] Rančić, S., Radovanović, S., and Delibašić, B. Investigating oversampling techniques for fair machine learning models. Decision Support Systems XI: Decision Support Systems, Analytics and Technologies in Response to Global Crisis Management, pp. 110-123. Springer International Publishing, 2021.
[61] Roh, Y., Lee, K., Whang, S. E., and Suh, C. Fairbatch: Batch selection for model fairness. In International Conference on Learning Representations, 2021. URL https://openreview.ue "orumm"id=NNnpaa.. YNnpaAKeCix
[62] Roh, Y., Lee, K., Whang, S. E., and Suh, C. Improving fair training under correlation shifts. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 29179-29209. PMLR, 2023. URL https://prooeeedings.s mln press/v202/ro223aahmm
[63] Rolf, E., Worledge, T. T., Recht, B., and Jordan, M. Representation matters: Assess- ing the importance of subgroup allocations in training data. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9040-9051, PMLR, 2021. URL https:/pproceedings.tm mir, press///s/smr..reer/ss......ee.. httpss//w11/rofff/ la.hmml
[64] Sagawa* S., Koh*, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL https:// httpr//iwwweetffoommmddddyororro
[65] Sagawa, S., Raghunathan, A., Koh, P. W., and Liang, P. An investigation of why over- parameterization exacerbates spurious correlations. In III, H. D. and Singh, A. (eds.), P Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8346-8356. PMLR, 2020. URL. https http:g........eee/////aaaaaaaaaa
[66] Schölkopf. B., Janzing, D.., Peters, J.. Sgouritsa, E.,, Zhang, K., and Mooij. J. On causal and anticausal learning. In International Conference on Machine Learning, pp. 459-466. 2012.
[67] Schrouff, J., Harris, N., Koyejo, S., Alabdulmohsin, I. M., Schnider, E.,, Opsahl-Ong. K., Brown, A., Roy, S., Mincu, D... Chen, C., Dieng, A., Liu, Y., Natarajan, V., Karthikesalingam, A., Heller,
14

---

K. A., Chiappa, S., and D' Amour, A. Diagmosing failures of fairness transfer across distribution shift in real-world medical settings. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D.. Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 19304-19318. Curran Associates, Inc., 2022.
[68] Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
[69] Singh, H., Singh. R., Mhasawade, V., and Chumara, R. Fairness violations and mitigation under covariate shift. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 3-13. Association for Computing Machinery, New York, NY, USA, 2021.
[70] Sreekumar, G. and Boddeti, V. Spurious correlations and where to find them, 2023. Preprint 2308. 1103.. URL. http://aaxxivorr//bb/// 11003
[71] Subbaswamy, A. and Saria, S. Counterfactual normalization: Proactively addressing dataset shift using causal mechanisms. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pp. 947-957, Association For Uncertainty in Artificial Intelligence (AUAI), 2018.
[72] Sun, Q., Murphy, K., Ebrahimi, S., and D' Amour, A. Beyond invariance: Test-time label-shift adaptation for distributions with "spurious" correlations, 2023. Preprint 2211.15646.. URL http://axxivoor/abss//2/
[73] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jegou, H. Training data-efficient image transformers & distillation through attention. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10347-10357. PMLR, 2021.
[74] Veitch, V., D' Amour, A., Yadlowsky, S., and Eisenstein, J. Counterfactual invariance to spurious correlations in text classification. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL
https//openneviie.nnefffrrmmm BideeeonnniddddoKoooinii
[75] Wang, A. and Russakovsky, O. Overwriting pretrained bias with finetuning data. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 3934-3945 IEEE, 2023. doi: 10.1009/CCV51000.2223300 003666. URL https://doi.orr//
1109/CCVV1000.0023...
[76] Wang, T., Zhao, J., Yatskar, M., Chang, K., and Ordonez, V. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In 2019 IEEF/CV International Conference on Computer Vision (ICCV), pp. 5309-53188 Los Alami- tos, CA, USA, 2019. IEEE Computer Society. doi: 10. 10.1109//CC.222190000444 URL
https:/dbbiieeccomptttrcccc
[77] Wu, S., Yuksekgonul, M., Zhang. L., and Zou, J. Discover and cure: concept-aware mitigation of spurious correlation. In Proceedings of the 40th International Conference on Machine Learning, 23. JMLR.org, 2023.
[78] Yan, S., Kao, H.-T., and Ferrara, E. Fair class balancing: Enhancing model fairness without observing sensitive attributes. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM '20, pp. 1715-1724, New York, NY, USA, 2020. Association for Computing Machinery.
[79] Yang. Y., Nushi, B., Palangi, H., and Mirzasoleiman, B. Mitigating spurious correlations in multi-modal models during fine-tuning In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 39365-39379. PMLR, 2023. URL L hbtps://rooceding...li 202////022222222223333
[80] Yang, Y., Zhang, H., Gichoya, J. W., Katabi, D., and Ghassemi, M. The limits of fair medical imaging ai in the wild, 2023, Preprint 10083. URL
[81] Zemel, R., Wu, Y., Swersky. K., Pitassi, T., and Dwork, C. Learning fair representations. In Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th International Conference Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 325-333, Atlanta, Georgia, USA, 2013. PMLR. hww.h//ss/ppocccdnggg.tppees///p
15

---

A Failure modes of data balancing
A.1 Failure mode: Balancing on one variable can increase bias
It is common to consider balancing on classes or groups as it requires fewer labels than joint balancing. However, without further intervention, class or group balancing on its own does not provide an invariant model when Y and Z are marginally dependent [e.g. 44]. In Figure [a). this means that X LL oZ Y,, invalidating Prop Below, we formalize the observation in Yan et al. (78) that balancing on one variable might affect the representation of the other, and provide bounds on the impact of this strategy.
Formalization and proof. We formalize this issue in Proposition A. for the binary case with a binary attribute.
Proposition A.1. Consider data balancing of Y; the marginal of Z will be farther from uniform than the marginal of Z before balancing if
Pt 1))
sgn
= sgn (B[ZlY 0] - E[Z = =1).
P'(Y=1)
Intuitively, if the biases of Y and Z are in the same (resp. opposite) direction, then this condition is satisfied if Z has a negative (resp. positive) correlation with Y. For example, if we have P'
1) = 4. E| = 1] = 1 and B[Z|Y = 0] = then E| = before balancing but E[Z] = 3 after
balancing.
Proof of Proposition 1.7 We assume that Y and Z, representing the label and confounder, are both binary. We will data-balance on Y. Let Z S denote the distribution of Z after data balancing. To characterize when the distribution of Z | S is farther from uniform than the distribution of Z, we will first derive
= 1] -) + P' =0) (0) (B|Z)Y= = -1 2)
E[Z]- 2 = P'(Y = 1) B
and
-| 1 (
1
-/2
Y =
B Y = 0] - 2
2 2
2
2
Now, taking the difference, we have
E|Z - 1
-
1
p'(Y==1) - ) E|Z = 11 1// + P (P((X=0)
= =0 - [E|z Y = 0J - 5 E = +] (P'(Y = =0) -)
+
2
2
1
P'(Y =1) 1
))
(P'(Y=0)-
- 2) E|Z Y =0
= S|
2
+
2)
2
= E[Z|S]
1
+
(P'(Y-1))
(P'(Y = =1 1)- - 2 (B Y = 1| E Y = 0))
2
We can derive some sufficient conditions for bias increase, which occurs when B|Z - 1 2
E Z
1 We proceed by cases. If E [ 1 > 0, then
E[Z|S|
2/2 = B|Z
1
(11(Y=1)-
P'(Y = 1) 1 Y = - Y =])
2
1
+ (P'(Y= 1) 1 (B|Z = 1] - E Y = 0])
2
so E S] =
11) -
-P)(==))))) (E Y = -1]- Y = 0)) Thus, the bias = [ Y = 0) > 0.
gets worse if =) 2)
Similar reasoning shows that if E|2] - 1 <0 then
B|Z S] - _ = || 2/ - (P'(Y = 1) - - [L| Y = -| Y = 0))
16

---

10.
Same direction
beportance aopert
10
Reverse direc-
tion
Figure 5: Proportions of Y = 0, (grey bars) and Z = 0, (purple bars) before (left) and after (right) balancing the data on Y.
and we can conclude that the bias is worsened if (P'(Y- = - (E|Z = - - = 0)) < 0. Taking both statements together, obtain the statement of the proposition.
For example, we have Pi(Y 1) = B[Z = 1] = and E| = 0 3. then E 2
but E[Z S] - 2 = ;: despite Z starting as unbiased, the data balancing induces bias of 6-
There are few implications of this derivation. First, we obtain an easy upper bound for the worsening of the bias of Z caused by data balancing: taking absolute values of both sizes and using the triangle inequality on the right yields
2 2
1 -
BiZ|s|
1 2
+ P =
1
E =1 - B = 0]1,
Bringing the second term over the left hand side and applying the same logic produces
1 2
1 2 P'( = 1) - E = 1| Y =0]l -)
Z S]
- EiZ
and combining both terms shows that the difference in bias of 2 and S is bounded by
2 E 1 - E ss - _ s P'(Y = 1) E Y = 1] 1EZ2 Y = =0]l-
Simulation. We present simple simulation to illustrate our reasoning: U ~ (.0 is common cause to Z and Y. More specifically, the continuous distributions of and both have the form U + c, with ~ (O(005 0.02). We then binarize Y by thresholding at 0. This creates an imbalance the marginal of Y. such that random sample 5,000 examples has 68% of positive labels. We then want to vary the marginal of Z, which also requires affecting their correlation. To this end, we vary the threshold for binarizing Z. This leads us to 2 main cases: for thresholds above (i.e. Y's threshold), the marginal of Z is imbalanced in the same direction as that of Y. For thresholds smaller than O., we obtain the opposite, i.c. if = 1 is over-represented, = 1 is under-represented.
We illustrate these cases in Figure 5 We observe that when the marginals are similar, balancing Y brings Z closer to uniform distribution (top row). However, the marginal distribution of Z becomes more imbalanced after balancing on Y if the two distributions are reversed (bottom row). When the correlation small, there is little change in the marginal of Z when balancing on Y, which is expected.
For completeness, we perform 200 simulations with different thresholdings for Z and present the results in Figure 6
A.2 Failure mode: entangled signals
In the case where includes non-trivial intersection information XYAZ, data balancing will in general be insufficient to ensure that there is no association bias. This is because a risk-minimizing predictor f(X) will condition on Xyaz, and the distribution of these intersection features is influ- enced by Z.
17

---

1.2
Drsportenn trdie
02
Scheampied
5.0
commatiio
co.oomaation
Figure 6: Distribution (' = 1) before (blue) and after (orange) balancing data according Y. for different values of the binarization threshold of Z which translates into different correlation coefficients between Y and Z. Left: similar direction of under-representation. Right: opposite direction.
Specifically, will give where Y is marginally independent of Z and there is no uncontrolled confounding, but ([(() 2 0J + ([s( Z = 1].
Suppose we have the following data generating process (DGP):
P'(Y = 1) = 0.5
Pt = 1) = 0.5
P = 1,2 = 1) = PY = 1) = i... Yz
p'X-= = p i OR z
O.W.
Note that in this case the entirety of would be classified intersection information Xy In this setup, the Bayes-optimal probabilities for classification, f(X), are given by:
pd = (Y
p p-0.5 0.5
= (Q)==P(YY-1/X>
P'(X=1)
0.75p + 0.25g
and
(1- P Y P=
(1 - (1-p) -0.5 1 (0.75p + 0.25g)
P
Note that when condition Z 0, 1, the expectation of f(X) is different whenever (1) # i.e, whenever the distribution of X actually depends on the function of Y and Z, and (2) f(1) * f(0).
ie., there is some information in X to predict Y:
Fif(X) 2 = 1] = F(F[S(x) x, 2 = 1]]= pf() + p)S(0)
( Z = E[E[f(X) X, 0]] -0..5p + (l5g + (0.5(1 p) + 0.5(1 - -))
In the simple case where p and g 0 (i.e., X Y OR Z deterministically), we get
f(x) P' = x) {
2/3 if X = if = 0. 2/3 if 1/3 if Z 0.
E[f(X) 2
B Conditions for data balancing to lead to an invariant and optimal model
We first investigate the case of a risk-invarianl model w.r.t P, and then discuss faimeess criteria.
18

---

B.1 Risk-invariant, optimal model
In this section we provide proofs for Section
Recall that P = (P)(X,Y,Z) = (X (x,,))))XXxxx,Y,
and that assume data balancing distribution Q(X, Z) E P of the form Q(X, Y.Z) 2) P*( Y, 2) (2) Also recall that we define X to be sufficient statistic for Y in Q if Eo X] Eql YY [
X 1 Z|Y and X2 is sufficient statistic for Y in Q, then the risk-minimizer f(X) == Eq! X] is risk-invariant and optimal w.r.l. P.
Proposition 4.2
Proof. Let P' be an arbitrary distribution in P. We have
P( 1
Σ
(
(,2)P'(ZIY)
① Σ Q (.Z))((ZI) Z Q(x y),
where (1) holds as P Q E P and by the independence assumption. As P'(Y) = Q(Y) we obtain
P' Y) Q( Y). As X is a sufficient statistic for Y in Q S(X) = EolY x] =
Eq Y ) that is f(X) (and therefore the loss {(f: (lf;X,Y) remains constant for different values of xy_ XYaz, giving
Exj.y [((;; X, Y)) Ex
The same reasoning be repeated for p" P. obtaining ExYYPP/((fi.)) [" Y),, which proves that ff is risk-invariant w.r.t. P.
As f miny Ex,y [[(;; X, ()] and Ex,y Ex (ilf;X,YV Y))
ox VP' E P. obtain f m Ex [e(f' X, Y)),, VP' E P. which implies that is optimal w.r.t. P. O
Corollary
4.3
Let R = (x), Xynz). R 1p (X), Y Z and X 1p Z Z Y, then
)(x)= = Eq Y X is risk-invarie invariant and optimal w.r.. P.
Proof. We have
X,
QY (2
Ezy Q(R,XX,
1 Σz 1
Ezy P
2
Exy PC (1
Ez (x
P(R))P((X) YY)
Er ((X+(T)P(())
= |
where (1) holds by the definition of the balanced distribution Q and (2) holds by the independence assumptions. This derivation shows that Y 1l Q R X and therefore that X 2 is sufficient statistic for Y in pt. We are the same conditions as Proposition 4.2 which implies that f is risk-invariant and optimal w.r.t. P.
Let f E F be some fitted model and e > Assume that, for all P', E P, we have Ep Y ((X Ep x= S 5. Then c-risk invariant, meaning that
Proposition 4.4
Rv (f) Rp (f) S.
sup pr,P"EP
19

---

Proof.
By definition, f is risk-invarian w.r.t. P and optimal. By the triangle inequality, we can then
write
[R (f) Rf)
1 Rp (f) (f)( (f*) Rp (p) 1 Rp (f) Rp (f) Rp (f*) - Rp (f)
_ +
2 2
B.2 Conditions for data balancing to lead to a fair model
In this section, we focus fairness definitions and provide sufficient conditions for data balancing to lead to a fair model. The results we describe do not address the case where X is not accessible directly.
Proposition B.1 (Demographic parity). X Q 7 X Lr YY; that is balancing successfully induces independence between X & and Z if X and Z are independent given Y in the original data distribution.
Proof. First, note that, under the assumed conditional independences,
Q()
Σ
Xς,Xyn2
P' Xy,X
E
((X))))
where the first equality is by applying the definition of Q and the last line follows from the assumed invariance. We can then derive
Q(
Σxxy Q(X2,
(X
Σxxt P' P'XX2 (YP(()))
P (2
EP((X') ())
= P
This equality implies that Q(X Z) does not depend Z, verifying the conditional independence claim.
Remark B.2, We would not expect X and to be independent in Q if X and Z are not independent given Y in p In the previous proof, we derived
Q(
Σ p' (X (,
Except for some corner cases, we would expect that pt (X / Z, Y) would not vary with Z Q( (Z) does not.
Proposition B.3 (Predictive parity). Y 0Q Z X if X 1pp Z Z|;; that is, data balancing
successfully induces independence between Y and Z given X if X 7 and Z are independent given Y in the original data distribution.
20

---

Proof. Let X2 pt Z Y.. The following derivation demonstrates the claim,
Q YZ)
QYIX2,22
2x
2 (X (X2))
Ey P (YZ))
2
( ()P((Y))((Z)
Ey P (
= (()
where (1) holds by the definition of data balancing on the joint, (2) holds by the assumption of conditional independence. Therefore, the l.s. is not a function Z which establishes conditional independence.
Remark B.4. We would not expect Y and Z to be independent given X in Q unless X and Z are independent given Y pt. From the previous proof, we wrote
( Y, (Y) Q(Y 1 2) Σy P (Y))))())PP())
so generally we would expect Q(Y X22, Z) to depend on Z. which implies Y AQ Z |x2, whenever (Xx ZY) depends on Z (i.e. when X Ar Y)).
Proposition B.5 (Equalized odds). X LL Y if X 1pp Y ; that is data balancing does not disturb independence between X 7 and Z given Y if X and are independent given Y the original data distribution Pt.
pt Y. Note that in this case we just need to show that data balancing does not disturb the conditional independence X2 p1 Z Y present in the original data (we already had equalized odds in original data). The following derivation demonstrates the claim,
Proof. Let 2
Q(X2,Y,
Q( (,Y
Σx
(
P' Y, ())()))
Ex (' (Y22))()))()
P'XX (YX)))2))
2
Ex | (Y)P)))
(X))))))
where (1) holds by the definition of data balancing and (2) holds by the assumption of conditional independence. Therefore, the l.h.s is not function of which establishes conditional independence.
Remark B.6. Similar the demographic and predictive parity cases, we would expect that, in most cases when X Hi Z Y holds, it is because X IL Z Y.
C Impact of data balancing on the CBN
Recall that we assumed that discrete, but all the results are easily extended for continuous Z.
Y + X
Let (G, Pty be the CBN underlying the data, where G contains an undesired path between and Y, and lei G be modification of in which the undesired path has been removed. The distribution Q obtained
Proposition 5.1
Z
21

---

by joint balancing the data to make Z and Y statistically independent. i.e. Q(Y, Z) = PP Y. (Z) might not factorize according to
9o.
Proof. Example 1: Causal task with causal and non-causal paths. Consider 9 =1 X
Y,Z + U + Y), for unobserved U. We have
Q(XY,2 - Σx Q(Y (2 - Q(X,Y,Z)
P ())(2))) Ey P (2X))
((XIZ,Y))'0
where the r.h.s is function of Z in general X is not independent Y given Z in Pt, If Q were 9 = 22 + X + }, then Y IL Z| X in Q. To show the claim it suffices therefore to construct a distribution P' such that X is not independent of Y given Z.
Example 2: Causal task with non-causal path. Consider G = {X + + Y). We have that,
Qx,(X,Y,2) Q(X = Σrx Q(X,, = E, 2)P(2))(() = EP(X)X22)P(()
The r.h.s is a function of Z in general as is not independent of Z given Y in a distribution pt consistent with G. Therefore, one may not interpret the mutilated graph G = [[X- + Y} as a correct representation of the conditional independences implied by the balanced distribution Q.
Example 3: Causal task with causal path. Consider G = 1z + - We have that, Q(),Y,)) = P'(X Y. Z) (2) Q( = Σr Q(XYV Σx
Pt (Y
(2)))) Er P(XIZ,Y)P(0
The r.h.s is function of Z in general as X is not independent of Z given Y in pt. Therefore, one may not interpret the mutilated graph go = (2, X - Y} as a correct representation of the conditional independences implied by the balanced distribution Q.
Example 4: Anti-causal task. Consider G = (Y - , + U + +, W + X}. We have that,
Σr.w Q(X,Y,Z,W) EYw W Q(X 2) = Σx.,w Q(X,Y.Z,W) Σvx. P. W (.2))
P(((XXX2))P(M))
The r.h.s is function of Z in general as X is not independent of Z given Y in a distribution p consistent with G. Therefore, one may not interpret the mutilated graph g' = (Y + X, Z - W X} as a correct representation of the conditional independences implied by the balanced distribution Q.
C.1 Regularization and data balancing don't always go hand in hand
C.1.1 Risk-invariance
We first consider the graph in Figure I d and show that X IL Z Y in both Q, which justifies its use in addition data balancing, although there might not be benefit of using both techniques simultaneously (in theory).
Proposition C.I. Consider the graph G in Figure [d). Then X 1L Z Y in both the training data distribution P' (consistent with 9) and the distribution after balancing, namely Q.
Proof. X IL Z Y holds in the training data distribution pt by d-separation. For the conditional independence in Q, consider the following derivation.
Exraz P* XYAZ (2)))
Q( 1Y,
Exvre Pt P'XXX (2Y))
P' Z,Y) = - - in)
22

---

import numpy up import scipy
Number of samples. 10000
# Canerate binary data with sinple data generating nodel 2 >> Y <- X
te(ap.randor (eizee()i
- ie(ap.random sormal > 0.3)
7 1+(x
0.6.np. randon = 1+(u 0.5*mp.. random..... normal (size=n) > 0.1)
(size=n) 0.5)
Marginal of
p_2 = up. apr.araa (Inp. (2a-(2--1) (Itp..ean for 1 in z1)
Marginal y. = ap.
for i in y)) aprarray (emp...oo(((==11))<)))
y.
for 1, in 2ip(2,y)1)
Resampling probabilitios
indep.prote probe - * p.2y indep_probe ap. wwp.euuiiddeeppp..... (indep.probs)
Ra-sanple according computed probabilities
indeces randon. choice size-n, replace-True, p-indep.probs) z_bal, z_bal, y_bal =[indeces), x[indeces], y[indeces]
it Check that Y and are independent contingency pantangeey_.aall_.bl.. = acipy stats. wow.ingencc...oorsttttaaaaaaeeee Implement chi squared test.
statistic, pvalue,
scipy stats www..contingeeccyccooooooooocc (ont bay__y)
Check whether x and indopendent tewsingnney_tabl_bbll. statistic, pvalue,
acipy atats wtppy.atts..com contingecynnnnyyc...oo.o.o coattiagncy.ccoottatabbaaaa
scipy state chii_ contingecne ott..eee......aaaaacoooooooooooo (ontinnocccotttttttttteteaaa.a
Figure 7: Python code to assess impact of balancing in numerical simulation graph Figure bb).
The r.h. a function of Z and therefore X IL Z holds in Q.
However, when considering the graph in Figure (b), we introduce dependence between X and 2, which can be easily checked by the simulation Figure 7 in which we consider the simplified graph Z Y X. While we are able obtain the marginal dependence between and Z (x2 0.34), we introduce a dependence between X and Z (x² < 0.0001).
C.1.2 When does data-balancing together with regularization lead to fair models?
This section gives several results to analyze the combination of data balancing implemented to generate independence between outcomes Y and sensitive attributes Z and regularization in two variants, First, regularizing to learn representations W = o(X,) / such that W LT Z Y; and second regularizing to learn representations W = o(X²) such that W HQ Z. We write X 1Lpt Y to state that X and are independent in distribution pt.
Regularization such that ) IL Y
Proposition C.2 (Demographic parity). Balancing and regularization such that W x(X²) and W Le Z Y is sufficient for demographic parity, i.e. W 1l. Z.
Proof.
Q(W Z)
O(W ZYY)Q(Y Z) |QWW|Y) Q(W),
where (1) holds by the assumption of balancing in which Z 2 and regularization W Z Y..
D
23

---

Proposition C.3 (Predictive parity). Balancing and regularization such that W = o(X2) and W Le is sufficient for predictive parity, i.e. d Z W.
Proof.
Q(ZIY,W) -
where both equalities hold by the assumption of balancing in which Z ود Y and regularization W Q Y..
Proposition C.4 (Equalized odds). Balancing and regularization such that W ((X²) and W d Y is sufficient for equalized odds, i.e. W 1L Y..
Proof. Regularization induces W LQ Z and equalized odds is satisfied by design.
Remark:
Note that balancing and regularization together are not always necessary, for example the section above shows that balancing on its own can successful in some cases.
Regularization such that o(x²) IL Z.
Proposition C.5 (Demographic parity). Balancing and regalarization such that W = x(X2) and W L Z is sufficient for demographic parity, i.e. W L Z.
Proof. Regularization induces W d Z and so demographic parity satisfied by design.
Proposition C.6 (Predictive parity). Balancing and regularization such that W = o(X²) and W L Z not sufficient for predictive parity, i.e. Y 11 Z W does not hold.
Proof. We give counter- Let A, B, C be three independent variables with values in {0, 1}. C}. Let Q be a probability distribution over
Let X = { = BJ.Y Y = - = (X2, Y. Z). In particular, we could imagine Q to be generated after balancing and regularization since W جا Z and Y مe Z. However, conditioned on X Y and Z determine each other and so predictive parity does not hold Q.
Proposition C.7 (Equalized odds). Balancing and regularization such that W ((X²) x and W LQ Z is not sufficient for equalized odds, i.e. W 1Q Z Y does not hold.
Proof. The counter-example above applies.
D Experiments
D.1
Datasets
This work uses the MNIST 45 17
Amazon reviews [53]
ImageNet [16 https::/immae- org. and CelebA [46] http::/mmlaa.iie.../mm www.eccc....ckeee...e Croeeeeeets/CCllbb
html datasets, which are all openly accessible and can be used for research purposes.
MNIST semi-synthetic data: For simplicity, we binarize the digit recognition task to label Y Y {0, according to whether the number in the image is < or 2 5 such that matches the ground truth with probability 0.98. The top of the image is replaced by noise coloured in red for 7 = 0 and blue for Z (see Figure T}. We can relate the confounder and the label such that 95% (resp. 5%) of images with Y have red (resp. blue) noise pattern, while 10% (resp. 90%) of the images with Y = have a red (resp. blue) pattern, corresponding to our original distribution p In this distribution, the marginal distributions of Y and 2 are (close to) uniform. We sample 30, 000 samples from pt, as well as dataset jointly balanced on Y and Z (Q, n = 30, 000). We also sample test data based on a ground truth مu generated with (" 0JY) 0.5 (n 000). Finally, we generate X dataset that contains white instead of colored noise.
MNIST semi-synthetic data with added confounder: We add V and Xv to our data generating process where Xv is green cross either the left or right of the image, with fixed vertical

---

position. The horizontal position of the cross is given by V and V is correlated with Y (P*(V-
O = 0) = 0.2, P*(V = 0|Y 1) = 0.9). We generate a confounded dataset (95/10) as previously, which we balance jointly on Y and Z. We then train 5 replicates of the same architecture, and test our model on Q, as well as on the ground truth pa where (V = 0/Y 0) = (V 0
1) = = 7 = Y = 0) = po [ = o = 0.5.
MNIST semi-synthetic data, entangled: We define the color of the noise based on an OR(Y, 2). We define Q by generating samples with Q(Z = Ojy 0) Q(Z = 0|Y 1) = 0.5, while po is represented by the disentangled test dataset described above.
Amazon reviews with confounder: We refer to Veitch et al. [74] and define a causal task based on Amazon reviews for the clothing category which predicts whether the review was found to be helpful (i.e. obtained 'thumbs up" votes) or not based on the review's ' text. We generate a random variable U as the unobserved confounder, and define Y as the binary helpfulness label, randomly flipping the label based on U (association: p=0.4). This leads to reviews with Y = O being more associated with À * U + (1- X) U2. where U2 is another random variable distributed uniformly and X is a parameter that controls the relationship between U and Z, and by transitivity, between Z and Y. In pt, À is selected be 0.8, leading to correlation of 0.35 between Y and Z. To create X we add perturbations to the text based on the value of Z that wouldn't 't (in theory) affect Y. We select the words { /and, the, you, my, they) and add a suffix 'xxxx' (resp. "yyyy') ) when Z = 0 (resp. Z = 1). Finally, Y is imbalanced, with only 5% of the dataset with Y = 1. We hence re-balance the classes before the modelling. This operation is also performed by the joint balancing.
U We define 2 as Z
D.2 Metric definitions and operationalization
Our work focuses on statistical group fairness criteria criteria on the model's predictions.
5 These can be translated independence
Definition D.1 (Demographic parity). A predictor s(X) is said to satisfy demographic parity w.r.t. sensitive attribute Z and distribution pa if f(X) 11 pt Z.
Definition D.2 (Predictive parity). A predictor f(X) trained to predict outcome Y is said to satisfy predictive parity w.r.t. sensitive attribute 2 and distribution pt Y ll pt fX).
Definition D.3 (Fqualized odds). A predictor f(X) trained to predict an outcome Y is said to satisfy equalized odds w.r.t. sensitive attribute 7 and distribution pt if f(X) 11 pa Z|Y.
In our experiments, we estimate equalized odds as in Alabdulmohsin & Lučić 1. For this metric, the lower, the better.
EO = 0.5 max [x( 7 2, Y = 0] min Ex[[(x] 2= Y, = =0] #E2
=E2
+ 0.5 max #EZ
2 =, [x
min [x[ 7 =, =1].
sE2
In terms of robustness metrics, we evaluate a simplified version of iisk-invariance by computing model performance on test set sampled from Pt, and contrasting this result with the model's performance on a test set sampled from p8 (when known), or from Q. We also estimate worst-group
performance (64]
WG = min [x,x(1[f(X) vl | 2']
s'eZ
An invariant model that is optimal would hence display high performance on both pt and pa/Q, as well as high worst-group accuracy.
Metries like risk-invariance or equalized odds provide insights on the model's outputs, but do not probe the model's 's representation. As we are interested in large-scale models that might be further fine- tuned, it is important to understand whether the model's representation is invariant on P. Defining a representation as ((X), we can write f(X) h(o(X)) in which we assume the representation to be fixed (i.e. frozen model weights) and is a learnable function. In Zemel et al. [8], the authors define a fair representation w.r.t. binary Z as demographic parity on the representation:
Ex ((X) = EXEXz= (()))=/6((X)))) V2, Z,
25

---

where x²== corresponds to the samples with Z = 2. This is equivalent to assessing the 'encoding' of Z in 6(X), by training a linear layer h ((X) + Z [27 8. Chance level performance of h(((X) would then suggest that the representation is independent of Z. In the present work, we estimate the encoding of Z using pD or Q such that assessing the encoding of Z equivalent to assessing the encoding of Z. Models that encode less of the auxiliary factor Z have been shown to reach more "global' optimum compared to models that encode the signal more strongly [independently of whether invariant predictions are obtained [80]
D.3 Model architectures
We consider multiple architectures in this work, with attempt to cover different model sizes and characteristics.
Small convolutional network, similar in spirit to AlexNet 43 It includes 5 convolution
blocks with kernel sizes (4, 3, 2, 2, 2, 2) and output channels (3, 6, 9, 12. 12, 9), with max
pooling after each convolution, as well as two dense layers with Relu non-linearity before the output head.
VGG network [68] with square kernels of size 3, output channels of dimensions (64, 64, 128, 128, 128, 256, 256, 256, 512, 512, 512) and strides (1, 1, 1. 1, 1, 1, 2, 1, 1).
Vision Transformers [8] of different sizes: ViT-micro (17M parameters), ViT-Tiny (44M), ViT-S (174M) and ViT-B (690M), with the Tiny sizes and up taken from [73].
For text data, we use the BERT architecture, as defined in TensorFlow Hub.
We use stochastic gradient descent optimizer with Nesterov momentum of 0.9 for all models.
D.3.1 Hyper-parameter searches
We include a hyper-parameter search over the learning rate (5 values in log-scale between 9e and 0.1) coupled with a batch size search between sizes of 128, 256 and 512 examples. In terms of regularization, the small convolutional network include dropout the dense layers (search on 0.1, 0.2, 0.3), while VGG includes batch normalization in the dense layers (as per their original implementations). We impose an 1.2-regularization of 1e during training for all architectures.
We note that hyper-parameters did not seem to make a difference on the MNIST results. For VGG, there was a larger variation, as well as a larger variance across multiple seeds.
When performing MMD conditional regularization, we vary the strength the regularizer in [0.0, 0.1, 0.2, 0.5, 1., 2., 3., 4., 5., 6., 7., 7., 10.]. with 5 replicates for each value. To minimize computational expenses, we fix the learning rate to 0.001, dropout rate to 0.1 and batch size to 64 (for downsampled datasets) or 256.
D.4 Assets, code and resources
We use the BERT model cart_en_uncsedd L- 1- 17-1-A-182212--- from TensorFlow Hub. All other models are trained from scratch in our code infrastructure written in Python and JAX (7). The results are then
eo__uncased_L-12_HHOO0
analyzed with Python and the numpy (30]. matplotlib [32 https://matplotiib.org and pandas (50 huts://pandas. httpaaandaas.ddaaoorr..... pydata.org/) packages. For the small convolutional networks, training was performed with GPUs (V100) and evaluation used GPU per model instance. BERT used 2 Tensor Processing
Units (TPUs) for training and 1 TPU for evaluation. For all other models, we used 4 Tensor Processing Units for training and I TPU or GPU (P100) for evaluation. We note that, apart from ViT-B and
BERT, all experiments could be run on CPU.
E
Results
E.1
Failure modes of data balancing with MNIST
Other confounder We notice that correlation between V and Z in Q is decreased (p = -0.16) compared to p' (p -0660) but is not null. In addition, we observe that the model relies on V (accuracy on Q: 0.769 0.008, on po; 0.647 + 0.023). As a consequence, models trained on Q display bias w.r.t Z (see equalized odds and worst group performance).
26

---

10 1.0
1.0
VGG VITT
0.8
EdU od0d
0.6
0.4
96
0.0
0.0
Figure 8: Model encoding, accuracy, worst group accuracy and equalized odds for the VGG architecture, and different sizes of VIT (m: micro, Tí: tiny, B) when trained with balanced CelebA data. Each dot is model replicate, while the dashed line represents average across replicates.
Cormelction wtt why
0.2
0.0
-0.-
Oriinal Balanced
-0.4
5.9 Arched Exctireed
Blge, Liectr Eler
Bushy Cholet
Aboultt couiii Coreen
Goneeaatio with
0.0
Original
Bar
Brchelt Shardon
Blund
Dor Eyriinn geoey Busty, Eo
Figure 9: Pearson correlation between each attribute and (left), Z (right) in sample of the original data (leal), compared to balanced sample (blue) of training data.
Entangled signals
During training, the model reaches 0.903 + 0.011 accuracy Q, but only 0.672 + 0.004 accuracy on po_ Worst-group accuracy is low and equalized odds high, displaying a failure mode of data balancing.
E.2 Celeb-A
E.2.1
Model performance
Model encoding and performance across different model sizes is displayed in Figure We show that all models trained on the subsampled data display an encoding of the auxiliary factor Z.
E.2.2
Distinguishing between failure modes
Correlation patterns in balanced data We plot the Pearson correlation between and all other available attributes (39 in CelebA) in Figure [ (left), and similarly for Z (right). We note that the correlation that increases most when balancing the data is between and the 'black hair' label. As this label has low correlation with Z, this does not seem problematic. We also observe smaller changes in attributes related hair ('bushy-eyebrows', 'bald') and accessories ('wearing-hat'..
Broader impact
Our work investigates a common mitigation strategy for failures of fairness or robustness machine learning predictive settings. We aim to clearly highlight when data balancing is promising, and when fails, hence advancing the field of trustworthy machine learning. As with most papers addressing fairness questions, we acknowledge that our mathematical formulations of faimesss criteria might not correspond to desired societal impact, e.g. terms of equity. Specific considerations for our work include the of the CelebA [66] dataset, and in particular the 'is-male' binary label provided.
27

---

We acknowledge that a binary characterization of gender is not representative and can be harmful. In addition, would be desirable to have self-reported instead of perceived gender. Our work considers cases for which auxiliary factors of variation Z are observed train, test or fine-tuning time. This is a limitation of our investigation, as our insights might not be available when Z unobserved. This exemplified by the more difficult case of distinguishing between failure modes without a P in the classification CelebA images.
28

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We present an analysis work that investigates when joint data balancing might fail succeed, and how to probe different failure modes. Our contributions are highlighted at the end of the introduction and all claims match the theory and experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include a separate section in our discussion, and further provide a Broader Impact section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "L.imiations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
29

---

Answer:
[Yes]
Justification:
All proofs are provided the Appendix and cross-referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all details in the Appendix, including hyper-parameter search. It is important to note that the experiments illustrate the theory and do not provide contributions
per
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which
5. Open access to data and code
30

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Our work uses open access datasets to illustrate baseline method for mitigat- ing undesired dependencies. There is no specific code contribution in our experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (httpss//nips.ccppbblcc
guides/CoaeSubbisssinoodeeyyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https: for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The data splits, and more specifically the evaluation distributions, are central work, and are detailed in the main text. Other experimental details provided in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Ys]]
Justification: All experiments were run from seeds and results are reported in terms of their average and standard deviation, while plots report each model trained.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
31

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for caleulating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative emror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Ye]]
Justification: We provide these details in the Appendix. We note that the architecture or specific model used is not relevant to our message.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://meerppscccooulll/hhhhhss
Answer: [Yes]
Justification: We do not include banned datasets and mention limitations of our data and methods clearly.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have included an additional section at the end of our paper to discuss Broader Impact of work.
Guidelines:
32

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the authors of the datasets used and provide details the infrastructure and terms of use in Appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
33

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paaperswiteode.. https/ittthood..cocataaatttttt has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not include crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not include crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
34

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "MwJ6BOOX0I", "reviewer": "Reviewer_YT39", "summary": "The paper studies the role of data balancing e.g. on sensitive attributes or class labels on obtaining fair or robust models. It identifies various causal graphs and corresponding independence conditions under which data balancing is expected to succeed (or may fail) to provide recommendations on when to use (or not to rely on) it. The main contributions are in deriving sufficient conditions for data balancing to yield invariance under correlation shifts and relating the conditions to observations made in prior work through extensive experiments.", "questions": "Please clarify which results are necessary, sufficient, and both.\n\nMinor comments which do not require a response from the authors:\n\nAlthough not necessary, please consider citing the original works for risk invariance under correlation shift for the claim in line 194, e.g. from Rojas-Carulla 2018 Invariant Models for Causal Transfer Learning https://arxiv.org/abs/1507.05333 or Peters et al. 2016 Causal inference using invariant prediction: identification and confidence intervals https://arxiv.org/abs/1501.01332.\nConsider relating the findings to more recent work such as Kaur et al. 2023 Modeling the Data-Generating Process is Necessary for Out-of-Distribution Generalization https://arxiv.org/abs/2206.07837.\n\nPlease specify whether it is feasible and how to check if the sufficient conditions hold.\n\nConsider commenting on data augmentation that might achieve the same conditions as data balancing in some scenarios.\n\nI would suggest adding a footnote that the term sufficient statistic differs slightly from its traditional definition e.g. in line 198 it is more appropriate to say that E[Y|X_Z^\\perp] is a sufficient statistic for the outcome risk function E[Y|X], not for Y.\n\n---\nAfter the response\n\nThanks for responding to the my concerns. I do not have any more questions and feel that the work is significant in that it presents a better understanding of data balancing (a commonly-used method in fairness or robustness problems). I have raised my score to 7 as a result.", "limitations": "Limitations are acknowledged in good amount of detail.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Data balancing is a simple pre-processing method that one may occasionally resort to in practice to achieve fairness or robustness. Explaining when it may not work or when it works is significant.\n2. Presentation and visualizations are clear, gives the required background, and explains the results in context of prior work. Jointly treating robustness and fairness in the exposition is a good way to generalize the results across the two problems. I like the organization of the results in successive sections 3-6 which are natural questions one may ask on data balancing.\n3. The work discusses the implications when causal graph is not completely specified, features are learned, and tests the hypotheses on real, high-dimensional data (in contrast to causal ML work which typically only tests on tabular data). Experiments are carefully constructed.", "weaknesses": "1. The writing can be clarified in places. For instance, I did not find a result which gives both necessary and sufficient condition contrary to the claim in Section 4 - please correct me if wrong.\n2. (minor) Results can be summarized more concisely in a figure/table that shows the causal graphs or dependence structures where data balancing is fine and recommendations (e.g. on regularizers) for the other scenarios.\n3. (minor) Some more recent works could be discussed. See comments below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments on our work. We respond to the comments below:\n\n1. **Necessary and sufficient conditions**: Thank you for this comment. Our results show that Propositions 4.2 and 4.3 are sufficient, while Proposition 4.4 is necessary. Based on the comment from Reviewer AEL7, we redefine Proposition 4.4 to make it more general, and rework the proof to highlight its necessity. We also rephrase Section 4 to avoid any confusion. \n\nSection 4: “In this section, we introduce *a sufficient condition on the data generative process and a necessary condition on the model representation* that, taken together, lead to a risk-invariant and optimal prediction model after training on $Q$.”\n\nLine 218: “We hence see that when a causal graph of the application is available, Corollary 4.3 can provide indicators on when data balancing might succeed or fail, *with the caveat that Corollary 4.3 is not a necessary condition*.”\n\nLine 220: “While Proposition 4.2 and its corollary provide conditions on the data generating process, prior work has demonstrated that the learning strategy also influences the model's fairness and robustness characteristics. In Proposition 4.2, we assume that the optimal risk-minimizer $f(X):=E_Q [ Y \\mid X]=E_Q [ Y \\mid X^{\\perp}_Z]$ can be found during training a machine learning model $\\hat f(X)$. Therefore, $\\hat f(X)$ should be of a function class that can represent $E_Q [ Y \\mid X^{\\perp}_Z]$. Let's consider the special case where $\\hat f(X)$ has the form $h(\\phi(X))$, in which $h$ is a ``simple'' function of $\\phi(X)$. This case would correspond to e.g. the last layers of a neural network or when learning a model based on a representation $\\phi(X)$ (e.g. embeddings, transfer learning). We have the following necessary conditions on $h(\\phi(X))$:\n\n**Proposition 4.4.**\n For $\\hat f(X) = h(\\phi(X))$ to be optimal and risk-invariant w.r.t. $\\mathcal{P}$, we require that (i) $h(\\phi(X))$ is able to represent $E_{Q}[Y \\mid X_Z^\\perp]$, and (ii) $h(\\phi(X)) = E_Q[Y \\mid X_Z^\\perp]$ such that $h(\\phi(X))$ is only a function of $X_Z^\\perp$.\n\nIn Proposition 4.4, we require that (i) $h(\\phi(X))$ preserves all the information about the expectation of $Y$ that is in $X_Z^\\perp$, and that $h(\\phi(X))$ only changes with $X_Z^\\perp$ and not with $X_Y^\\perp$ or $X_{Y \\wedge Z}$. Given our assumptions on $h$ and $\\phi$, $\\phi(X)$ must be disentangled in the sense that the simple function $h$ eliminates any dependence on $X_Y^\\perp$ or $X_{Y \\wedge Z}$. For example, if $h$ is a linear function, it must be possible to linearly project out all dependence on $X_Y^\\perp$ and $X _{Y \\wedge Z}$.\nWe note that such a representation can be obtained even if the data is entangled, e.g. by dropping modes of variation during training. [...]”\n\n\nAppendix B.1.\n\nProof:\nRemember that we assume that $\\hat f(X)$ takes the form $h(\\phi(X))$. If $h(\\phi(X))$ cannot represent $E_{Q}[Y \\mid X_Z^\\perp]$, it is straightforward that $\\hat f(X)$ cannot be optimal. Similarly, for $\\hat f(X)$ to be risk-invariant w.r.t. $\\mathcal{P}$, we need that $h(\\phi(X)) = E_Q[Y | X] = E_Q[Y | X_Z^\\perp]$ (sufficient statistics). As the right hand side varies only with $X_Z^\\perp$, $h(\\phi(X))$ can only vary with $X_Z^\\perp$ and cannot depend on $X_Y^\\perp$ or $X_{Y \\wedge Z}$.   \n\n\n2. **Summary of results**: Thank you for this suggestion. Based on this comment and those of other reviewers, we now provide such a table, depicted in the attached pdf.\n\n\n3. **Recent work**: Thank you for the additional references and other minor suggestions. We have incorporated them in our revision. With regards to Kaur et al., 2023, we note that our Figure 1(c) is a realization of the canonical graph presented in their Figure 2(a). We will add this work to our related works section and include it in possible mitigation strategies when considering multiple attributes. We however note that this work focuses on an anti-causal graph in which $X_c$ (unobserved) is the only cause of $Y$ (hence a sufficient statistics) and studies multiple mitigation strategies. On the other hand, we focus on data balancing and attempt to consider multiple data generative processes.\n\n\nWe hope these clarifications alleviate your concerns and that you will consider amending your score to support the publication of this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VRTZ1jv8jk", "reviewer": "Reviewer_hyP8", "summary": "The paper focuses on the topic of data balancing for fairness and robustness and uses a causal graph as a tool to analyze the effects of data balancing. In the paper, the paper tries to show both the positive impacts and potential pitfalls of data balancing. For the fairness aspect, the paper focuses on the independence between groups and labels in the data. For the robustness aspect, the paper focuses on the lowest risk across a family of target distributions. The paper uses several synthetic and benchmark datasets to support their theoretical analyses on data balancing.", "questions": "The questions are included in the weakness section.", "limitations": "The paper has reasonable limitation, future work, and impact section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* S1. The paper empirically shows both the positive and negative impacts of data balancing, which possibly provides some insights into both the fairness and robustness field.\n* S2. The paper also suggests some causal graph-based analyses to understand the effects of data balancing.", "weaknesses": "* W1. This paper examines both fairness and robustness, yet the relationship between these two aspects or the reason the paper focuses on these two remains somewhat obscure. The paper could benefit from giving a more in-depth discussion or intuition on why fairness and robustness are susceptible to similar issues. Potential discussions may include 1) how similar analysis can apply to these two different objectives and 2) why the paper focuses on fairness and robustness among various safety criteria.\n\n* W2. While the paper employs synthetic and benchmark datasets to illustrate various failure cases, a more concrete real-world motivating examples would further highlight the importance of addressing these failure cases in both fair and robust training.\n\n* Minor suggestion: It would be better to use full name of CBN in Figure 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and suggestions. We respond to each point below:\n\n**W1: fairness and robustness.** Our work focuses on undesired dependencies between $Y$ and $Z$ and on the use of data balancing to mitigate any bias that might result from these dependencies. This focus directly maps to fairness criteria, as well as to robustness to distribution shift in which the additional factor of variation represents the environment. In fact, fairness and robustness criteria can be equivalent in certain cases, as described in [Makar and D’Amour, 2023](http://arxiv.org/abs/2209.09423). Amongst fairness and robustness, we consider multiple, widely used criteria. To the best of our knowledge, other safety criteria do not clearly map to this framing of undesired dependencies (e.g. privacy, accountability, recourse, interpretability, alignment) and do not refer to data balancing. We however note that some terms defined in the Ethics community can be evaluated with fairness and robustness metrics, such as representational harms (equivalent to demographic parity) or stereotypes (e.g. robustness to correlation shifts or equalized odds).\n\nWe provide further discussion on the selection of fairness and robustness criteria in the preliminaries:\n\nSection 2.1.: *Due to undesired independencies*, a model may be optimal on $P$ but perform poorly on another distribution of interest $P’(X, Y, Z)$ (e.g. in deployment), and/or might display disparities across subsets of the data (e.g. $P(X, Y |Z = 0)$). *To identify such behavior, various safety criteria have been proposed, with criteria in the fields of *fairness* and *robustness to domain shift* being specifically designed to capture changes in model outputs across distributions*.\n\n**W2: real-world motivating example.** In this work, we focus on simple cases to identify the successes and pitfalls of data balancing. We feel that our synthetic setup illustrates those best, due to their simplicity and the availability of the ground truth causal graph. To further illustrate failure modes in simple cases, we are adding a semi-synthetic version of Amazon reviews, which represents a causal case similar to that of Figure 1(b). Using this dataset, we show that data balancing overall improves fairness and robustness metrics, but does not lead to an optimal model. We further show that performing data balancing hinders the effectiveness of regularization as it leads to models with lower performance across multiple distributions, and introduces variance in the risk for higher values of the regularizing hyper-parameter.\n\nSection 3:\n\n**Causal task: helpfulness of reviews with Amazon reviews (Ni et al., 2019)**. Inspired by Veitch et al. (2021), we refer to the causal task of predicting the helpfulness rating of an Amazon review (thumbs up or down, $Y$) from its text ($X$). We add a synthetic factor of variation $Z$ such that words like 'the' or 'my' are replaced by 'thexxxx' and 'myxxxx' ($Z=0$) or 'theyyyy' and 'myyyyy' ($Z=1$). We train a BERT (Devlin et al., 2019) model on a class-balanced version of the data for reference (due to high class imbalance), and compare to a model trained on jointly balanced data, both evaluated on their training distribution and on a distribution $P^0$ with no association.\n\nIn this case, jointly balancing improves fairness and risk-invariance, with the model's performance on the training distribution (acc.: $0.574\\pm0.016$) being similar to that on $P^0$ (Table 1). This however comes at a high performance cost when compared to the class balanced model's performance on $P$ (acc: $0.658\\pm0.015$). Therefore, data balancing might not lead to optimality for this causal task.\n\nSection 5: We add the following empirical analysis to the discussion on the causal task of Figure 1(b). Please see the pdf added to the main rebuttal for the figure and table.\n\n“We illustrate this result on the Amazon reviews dataset from Section 3 by imposing a marginal MMD regularization $f(X) \\perp Z$ during training and evaluating risk-invariance across multiple $P' \\in \\mathcal{P}$. When training on $P$, we observe that the regularization allows to 'flatten' the curve, such that from medium to high values of MMD regularization, the model is risk-invariant (Figure 4). On the jointly balanced data, medium values of the regularization degrade risk-invariance (see green curves on Figure 4(b)). Overall, model performance is also lower for the models trained on $Q$ compared to models trained on $P$ across test sets from $P' \\in \\mathcal{P}$, at similar levels of regularization (see Figure 4(c) for MMD=16). This result displays that $X^\\perp_Z$ is not a sufficient statistic for $Y$ in $Q$.\n\n**Minor comment** Thank you, we will not use the acronym in the Figure (see table in pdf).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z5X8nMO0fD", "reviewer": "Reviewer_AEL7", "summary": "The paper analyses training risk invariant models using data balancing. The authors consider the cases in which data balancing might help obtain risk-invariant models and the cases in which data balancing does not achieve the desired effect. The paper also considers the effect of regularization for robust model training, and how that compares to data balancing.", "questions": "1. In line 161, does \"performance\" correspond to model accuracy? If so, how is the performance on $P^0$ lower than that on P? (where the latter has an overall accuracy of 0.717)\n2. How do you try to decode $Z$ from the representation $\\phi(X)$? This is an important piece of information that seems to be missing?\n3. In Table 1, what is the test distribution on which the accuracy is measured? What is P(Z=1) for example, and is the test distribution of (X, Y, Z) fixed across the different tasks in Table 1? It seems that these test distributions differ across the different tasks. If so, the comparison of accuracy does not seem to be fair. \n4. There is no motivation behind the choice of the 4 cases in Figure 1. These seem somewhat arbitrary. Why were these chosen specifically? Is there a systematic way of considering the different cases?\n5. Does the model f need to be a neural network necessarily? If so, this should be made clear.\n6. What do \"penulatimate representation\" and \"intermediate representation\" mean? Are these referring to the hidden layers of the model? \n7. I found the paragraph starting on line 270 to be quite vague. For example, the authors talk about \"regularization\" without explicitly defining the regularization term.\n\n> \"If we consider both the purely spurious correlation and the entangled case, we see that regularization and data balancing would have the same effects of blocking any dependence...\"\n\nI am not sure I completely understand this statement either. The regularization is a methodology which would potentially modify the distribution of f(X), but the distribution of data (X, Y, Z) would still remain unchanged, so how does regularization change the DAG of data?\n\nIn figure 3 and table 2, the accuracy on the original data distribution P should also be logged. In fact, in real-world settings, the practitioners would care about the accuracy on the original data distribution (not Q or P^0).", "limitations": "The authors have addressed the limitations of their work in the final section.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The results presented are sound, and the problem considered in the paper is of practical relevance. I liked that the authors tried to analyse the problem on a simple semi-synthetic setup first, before moving onto the real-world data. This provides a nice test-bed to test and motivate the methodology presented.", "weaknesses": "1. The paper only considers risk invariance w.r.t. correlation shift. Can this be generalised to other forms of distribution shifts?\n2. The paper only considers the question of joint balancing of Y, Z. As the authors show in Proposition 5.1, joint balancing will not effectively remove the undesirable paths in G in all cases. Can the conditions be generalised to other forms of balancing?\n3. Obtaining a DAG for a real-world setup involves the decomposition of $X$ (and positing causal relationships between these components). This seems like a non-trivial task in general. The authors don't touch upon the practical aspects of obtaining the DAG in the real world. \n4. The paper does not provide a systematic methodology for determining whether data balancing is a good idea for a given real-world dataset. The experiment seems to be using a trial-and-error method to establish the failure modes, but this seems highly impractical. \n5. Overall, I think the presentation can be improved. Certain parts of the paper are a bit vague. (See the questions below for more details.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and questions. We reply to each below, although the format of the rebuttal doesn’t allow us to provide the proposed text amendments. We prioritized answering all comments but would be happy to provide these suggested changes during the discussion. We hope that this alleviates your concerns and would appreciate a revision of our score if it does.\n\n**W1: risk-invariance** In this work, we discuss risk-invariance, optimality and multiple fairness criteria. We select these criteria as they are best suited to capture the effects of undesired dependencies between $Y$ and $Z$ on the model’s outputs. In particular, risk-invariance to correlation shift corresponds to the settings of “spurious correlations” discussed in the field of robustness (e.g. Sagawa et al., 2020). Using a similar framework, one could define conditions for other risk-invariance criteria. This is mentioned in our discussion (line 372). It is however unlikely that joint data balancing on $Y$ and $Z$ would provide invariant models if there is a covariate shift as the balancing does not affect $X$ directly. We will add this comment to the discussion.\n\n**W2: other balancing schemes** Our work focuses on joint balancing as it is commonly used to address undesired dependencies between $Y$ and $Z$. We briefly discuss class and group balancing in the Appendix, but they do not lead to an independence between $Y$ and $Z$ and are not included in further analyses.  It is unlikely that unique conditions can be derived across multiple balancing schemes, especially as these balancing schemes are typically defined based on specific causal graphs (e.g. Kaur et al., 2023, Sun et al., 2023). For instance, Kaur et al. (2023) show that no unique regularization strategy or set of independences can be imposed to obtain an invariant model in an anti-causal case with multiple attributes. We will amend line 383 which discusses this.\n\n**W3: decomposition of $X$** In Veitch et al. (2021), the authors display a causal and an anti-causal graph that decomposes $X$ into its subcomponents. They show that, under certain assumptions, this decomposition always exists. Therefore, when a causal graph of the application exists, the decomposition of $X$ can be performed based on the definitions of the subcomponents of $X$. Similarly, the relationships between these components can be defined by an upper-bound, i.e. only assuming the independences based on the definitions. In our work, we assume specific (in)dependencies between the subcomponents of $X$ to provide an upper-bound on the effectiveness of data balancing. We will add text under Assumption 2.3 to clarify this. We however agree that the decomposition increases complexity when the number of variables increases. In this case, grouping variables can ease the task, as in Kaur et al., 2023.\n\n**W4: application in real-world** In section 6, we posit that we have a partial causal graph, i.e. we make the assumption of an anti-causal task. While we hope to be in the case of Figure 1(a), our results suggest that the graph is more complex. To understand which graph might represent our data generative process, we consider how regularization interplays with potential failures and use these analyses to highlight which graph is most likely. This process does not perform “trial-error” but rather tests specific hypotheses and aims at identifying the most suitable mitigation strategy when the graph is not fully specified. We will make this clearer and reframe Section 6.\n\n**Q1, Q3** All results in Table 1 are assessed on $P^0$. Performance on $P$ is mentioned in the text. We will clarify in the text and table caption. $P^0$ is the same for Figures 1(a) and (d), but includes $V$ for 1(c) as otherwise it would be out-of-distribution due to the absence of $X_V$. We note that we do not compare results with each other, but rather assess whether each case leads to an invariant and optimal model.\n\n**Q2** To assess encoding, we perform transfer learning by fixing the representation $\\phi$ and training a new linear layer $h$ that predicts the factor of variation $Z$. All experimental details are in Appendix D, including metrics and their operationalization (D.2). We will make this clear and move materials if possible.\n\n**Q4** These graphs were selected as they represent different cases studied in the literature, albeit sometimes simplified to provide the upper bound for data balancing. This will be added. Beyond the illustration of our results, the specific selection of these graphs does not affect our results as we aim to be graph-independent (e.g. section 4, Proposition 5.1).\n\n**Q5** The model does not require to be a neural network, although the regularization scheme might differ for different architectures.\n\n**Q6** Yes, this will be clarified. We will also add that this condition should be respected by any architecture, although we only formalize it for a specific form of $f(X)$. Please see our response to **W1** from Reviewer YT39 for the full rewrite.\n\n**Q7** We will explicitly relate the recommended “independence between f(X) and Z conditioned on Y” to a regularization term (implemented in experiments with MMD).\n\n**Q8** The regularization does not affect the DAG but would approximate a distribution generated from a DAG in which these paths are blocked. We will clarify in the text.\n\n**Q9** We add the performance on $P$ in tables (see pdf). We note that practitioners who desire fairness and/or robustness criteria evaluate their model on multiple distributions, and that $P^0$ or $Q$ might be more suitable choices as discussed in Dutta et al., 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CIb6P8r42a", "reviewer": "Reviewer_rHnX", "summary": "This paper theoretically studies the applicability of data balancing in achieving fairness or robustness. The paper shows that data balancing may fail depending on the data generation mechanism. The paper introduces conditions for data balancing to produce invariant and optimal models. The paper also shows that data balancing can negatively impact fairness or robustness when combined with regularization strategies.", "questions": "In Assumption 2.3, does “a function of X” mean “a subset of X”?\n\nCan the proposed method be applied to other types of distribution shift, like prior shift or concept shift?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality\nThe paper extends the previous results on data balancing as mitigation for invariant models by considering different failure models and proposes strategies to distinguish between them.\n\nQualify\nThe paper offers some valuable insights into data balancing techniques, especially the failure modes of data balancing.", "weaknesses": "This theoretical work on data balancing for fairness and robustness in machine learning presents some valuable insights, but falls short in several key areas. The paper's main contribution lies in demonstrating cases where data balancing techniques fail to achieve fairness. However, it does not effectively synthesize these findings into clear principles or rules, lacking a clear, overarching takeaway message. A more effective approach might have been to distill the various cases and propositions into a set of guiding principles or a unified framework for approaching fairness using data balancing.\n\nThe authors present several propositions in Section 4, summarizing some of their results. While these propositions offer a formal structure to the paper, the conclusions they provide are relatively intuitive and do not significantly advance the field's understanding of data balancing.\n\nAnother limitation of the paper is its focus on negative results. The authors primarily demonstrate scenarios where data balancing techniques are ineffective (\"CANNOT results\"), without offering constructive solutions to these identified problems. This approach, while valuable for highlighting potential pitfalls, reduces the paper's overall significance and applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comment and evaluation of our work. We respond to each comment below and provide a new table and figure in the general rebuttal. We hope that these changes provide a more balanced and guiding framing of data balancing for fairness and robustness and that you will consider revising your score if they do.\n\n**W1: lack of guidance** Based on this comment and on the suggestion from Reviewer YT39, we now provide a table including each graph, whether data balancing provides an invariant and optimal model, whether regularization does (and which type), and next steps if they don’t.\n\nWe note that section 4 provides conditions that are not dependent on the causal graph of the application and that Proposition 5.1 also holds irrespective of the graph.\n\n**W2: intuitive results** We partially agree with the reviewer that some results might seem intuitive to researchers familiar with the field. We would however argue that most practitioners are not aware of the limitations of data balancing, and that Proposition 5.1 and its consequences on the interaction between data balancing and other mitigation strategies are novel.\n\nTo better highlight interesting cases, we slightly modify Figure 1(b) to refer to a purely spurious dependence between $Y$ and $Z$ in the causal case. While it might be intuitive for purely spurious cases to be addressed with joint data balancing, we show that for a causal case this is not a valid assumption. We also add experiments with Amazon reviews to illustrate this result.\n\nSection 3:\n\n**Causal task: helpfulness of reviews with Amazon reviews (Ni et al., 2019)**. Inspired by Veitch et al. (2021), we refer to the causal task of predicting the helpfulness rating of an Amazon review (thumbs up or down, $Y$) from its text ($X$). We add a synthetic factor of variation $Z$ such that words like ‘the' or ‘my' are replaced by ‘thexxxx' and ‘myxxxx' ($Z=0$) or ‘theyyyy' and ‘myyyyy' ($Z=1$). We train a BERT (Devlin et al., 2019) model on a class-balanced version of the data for reference (due to high class imbalance), and compare to a model trained on jointly balanced data, both evaluated on their training distribution and on a distribution $P^0$ with no association.\n\nIn this case, jointly balancing improves fairness and risk-invariance, with the model's performance on the training distribution (acc.: $0.574\\pm0.016$) being similar to that on $P^0$ (Table 1). This however comes at a high performance cost when compared to the class balanced model's performance on $P$ (acc: $0.658\\pm0.015$). Therefore, data balancing might not lead to optimality for this causal task.”\n\nSection 5: We add the following empirical analysis to the discussion on the causal task of Figure 1(b). Please see the pdf added to the main rebuttal for the figure and table.\n\n“We illustrate this result on the Amazon reviews dataset from Section 3 by imposing a marginal MMD regularization $f(X) \\perp Z$ during training and evaluating risk-invariance across multiple $P' \\in \\mathcal{P}$. When training on $P$, we observe that the regularization allows to 'flatten' the curve, such that from medium to high values of MMD regularization, the model is risk-invariant (Figure 4). On the jointly balanced data, medium values of the regularization degrade risk-invariance (see green curves on Figure 4(b)). Overall, model performance is also lower for the models trained on $Q$ compared to models trained on $P$ across test sets from $P' \\in \\mathcal{P}$, at similar levels of regularization (see Figure 4(c) for MMD=16). This result displays that $X^\\perp_Z$ is not a sufficient statistic for $Y$ in $Q$.”\n\n**W3: focus on negative results** Our work does provide success cases, including conditions for these. We also discuss cases in which balancing helps without leading to an invariant model. We hope that with the addition of the guidance table (answer to W1) and the results from Amazon reviews showing that the model is invariant but not optimal, the reviewer will agree that we provide a nuanced and helpful analysis of data balancing for fairness and robustness.\n\n**Q1** A function of $X$ can be a subset of $X$ but can also express a latent variable that does not directly select features, e.g. in an image. We will clarify.\n\n**Q2** In this work, we discuss risk-invariance, optimality and multiple fairness criteria. We select these criteria as they are best suited to capture the effects of undesired dependencies between $Y$ and $Z$ on the model’s outputs. In particular, risk-invariance to correlation shift corresponds to the settings of “spurious correlations” discussed in the field of robustness (e.g. Sagawa et al., 2020). Using a similar framework, one could define conditions for other risk-invariance criteria. This is mentioned in our discussion (line 372). It is however unlikely that joint data balancing on $Y$ and $Z$ would provide invariant models if there is a covariate shift as the balancing does not affect $X$ directly. For concept or prior shifts, the same methodology could be applied, although having a canonical causal graph would help. We will add this comment to the discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their constructive comments and suggestions. We have answered each point in detail, but wanted to highlight some of the important changes here for visibility:
- We reframed our findings in a comprehensive table to guide the reader for next steps (see pdf).
- We have added a semi-synthetic dataset based on Amazon reviews to highlight how causal cases can behave unexpectedly (see pdf).
- We have clarified the language throughout the paper and reworked Proposition 4.4 to highlight its necessity, as well as its framing in the context of neural networks.

Our work provides a framework to investigate a popular formulation of data balancing, leading to the detection of failure cases as well as success cases. We discuss both the positive and negative aspects, and aim for generality rather than focusing on a single graph. We complement our theoretical insights with experiments on 3 datasets, in text and vision applications, using multiple architectures.
We hope these changes and our response address most comments and are looking forward to further discussions during the next week.

Meta-review: The paper provides a critical take on data balancing for fairness. The main contribution is in identifying the conditions under which data balancing will/will not work using a causal graph formalism. Reviewers appreciated the technical depth and clarity in presentation. While some of the results may appear obvious in hindsight, the paper does a good job of providing a formal framework and connecting to existing empirical results. Based on the rebuttal responses, I suggest the authors to add the real-world experiment and a discussion of limitations wrt. other kinds of shifts (e.g., covariate shift), and provide guidance on use of data balancing.
Decision: Accept (poster)