The World is Changing: Improving Fair Training under Correlation Shifts
Authors: Yuji Roh
Kangwook Lee
Steven Euijong Whang
Changho Suh
Venue: Submitted to ICLR 2023 (2023)
Keywords: trustworthy AI
fairness
correlation shifts

Abstract: Model fairness is an essential element for Trustworthy AI. While many techniques for model fairness have been proposed, most of them assume that the training and deployment data distributions are identical, which is often not true in practice. In particular, when the bias between labels and sensitive groups changes, the fairness of the trained model is directly influenced and can worsen. We make two contributions for solving this problem. First, we analytically show that existing in-processing fair algorithms have fundamental limits in accuracy and fairness. We introduce the notion of correlation shifts, which can explicitly capture the change of the above bias. Second, we propose a novel pre-processing step that samples the input data to reduce correlation shifts and thus enables the in-processing approaches to overcome their limitations. We formulate an optimization problem for adjusting the data ratio among labels and sensitive groups to reflect the shifted correlation. A key advantage of our approach lies in decoupling the roles of pre-processing and in-processing approaches: correlation adjustment via pre-processing and unfairness mitigation on the processed data via in-processing. Experiments show that our framework effectively improves existing in-processing fair algorithms w.r.t. accuracy and fairness, both on synthetic and real datasets.

Under review as a conference paper at ICLR 2023
THE WORLD IS CHANGING: IMPROVING FAIR TRAINING UNDER CORRELATION SHIFTS
Anonymous authors Paper under double-blind review
ABSTRACT
Model fairesss is an essential element for Trustworthy AL. While many techniques for model fairness have been proposed, most of them assume that the training and deployment data distributions are identical, which is often true in practice. In particular, when the bias between labels and sensitive groups changes, the group fairness of the trained model is directly influenced and can worsen. We make two contributions for solving this problem. First, we analytically show that existing in- in-processing fair algorithms have fundamental limits in accuracy and group fairness. We introduce the notion of correlation shifts, which can explicitly capture the change of the above bias. Second, we propose novel pre-pree preprrccsssigg step that samples the input data to reduce correlation shifts and thus enables the in- processing approaches to overcome their limitations. We formulate an optimization problem for adjusting the data ratio among labels and sensitive groups reflect the shifted correlation. A key advantage of our approach lies in decoupling the roles of pre-processing and in-processing approaches: correlation adjustment via pre-processing and unfairness mitigation on the processed data via in-processing. Experiments show that our framework effectively improves existing in-processing fair algorithms w.r.t. accuracy and fairness, both on synthetic and real datasets.
INTRODUCTION
Model fairness is becoming indispensable in many artificial intelligence (AI) applications to prevent discrimination against specific groups such as gender, race, age (Feldman et al.) 2015 Hardt jet al., 2016) or individuals(Dwork et al.] 2002a).. In this work, we focus on group fairness, and there three prominent group fairness approaches: pre-processing, where training data is debiased; in-processing, where model training is tailored for fairness; and post-processing, where the trained more related works discussed in Sec. 6
model's 's output is modified to satisfy fairness
While fairness in- pr-oceeessing approaches are commonly used to mitigate unfairness, most of them make the limiting assumption that the training and deployment data distributions are the same (Zafar et al. 2017a| Zhang et al. 2018 Roh et al.] (222). However, the two distributions are usually different, especially in terms of data biases (Wick al. 2019 Maity al. (02T). ). For example, recent work shows that the bias amounts likely differ between previously collected data and recently collected data (Ding al. 2021). Moreover, when the data bias changes, the fairness and accuracy of the trained model are now unpredictable at deployment, as the above assumption is broken.
In this work, we introduce the notion of correlation shifts between the label y and group attribute in the data to systematically address the data bias changes. Although several works have been recently proposed to investigate fair training on different types of distribution shifts, including covariate and concept shifts (Singh et al., 2021: Mishler & Dalmasso 2022). they usually do not explicitly consider bias changes between y and 7. comparison, our correlation shifts enables us to theoretically analyze how exactly data bias changes affect fair training see how correlation shift compares with other types of distribution shifts in Sec. (6]
For fair training under correlation shifts, we first 1) analyze the fundamental accuracy and fair- ness limits of in- proccesss approaches with the fixed distribution assumption using the notion of correlation in the data and then 2) design novel pre-processing step to boost the performances of in processing approaches under the correlation shifts. We show that existing in processinn fair algorithms are indeed limited by the training distribution and may perform poorly the deployment

---

Under review as a conference paper at ICLR 2023
Correlation between y 6
Training
Correfation (ie, blast decreases
Deployment
Training
High correlation
Achievable performance: low
Neg. label
Pos. label
① Pre processssinng
Group
Group
Correlation
2
Fair model training (in-processing)
Low correlation
A fair clessifier w.r.t DP fairness from the training data
The same classifler now unfair
Deployment
Achievable performance High
(b) A toy example for illustrating the impact of correlation shifts (i.c., bias changes) on the trained classifier.
(a) A high-level workflow under correlation shifts.
Figure I: The central axis in the left figure represents the correlation between the label y and sensitive group attribute z. The correlation of training data is usually higher than that of deployment data. In Sec. 3 we show that the correlation determines the achievable performance of fair training. Thus, we first run our pre- propessessing and then apply existing fair algorithms on the processed data to address the correlation shift and improve the performances of fair training. Not addressing the correlation shift may result in reduced performance as shown in the right figure- - see details in Sec. 2
distribution. In particular, a high (y, z)-correlation results in a poor accuracy-fairness tradeoff for any fair training. Therefore, most in- un-preeessing fair algorithms assume identical training and deployment distributions, there is no guarantee their performances on the training data carry over to the deployment data. Based on the theoretical analysis, we propose pre-processing step for reducing the shifted correlation by taking samples of (y, z)-classes. Using a possible range of the shifted correlations, we solve an optimization problem that finds the new data ratio among (y, z)-cassses to adjust the correlation for the shift, which gives in- pr-prceessing approaches a better opportunity to perform well. The new data is then used as the input of any fair algorithm.
A key advantage of our framework is the decoupling of processin and in in-processing, for unfair- ness miligation where the pre-processing adjusts the correlation while the in-processing performs the rest of the unfairness mitigation, as described in Figure Ia We note that our pre-processing aims to boost the performances of in-processing approaches based on our theoretical analysis, whereas exist- ing pre-processing approaches for fairness simply remove the bias in the data and are not designed to explicitly benefit the in processing approaches- see Sec. [.1 for details. Our framework thus takes the best of both worlds of pre- and in-processings where (1) pre-processing solves the data problems, and (2) in-processing performs its best on the improved data. Our framework is not only useful for improving the fairness of single metric, but can also be extended to support multiple metrics.
In our experiments, we verify our theoretical results and demonstrate how our framework outperforms state-of-the-art pre-processing and in- in-processing baselines. Experiments on both synthetic and real-world datasets (COMPAS (Angwin et al.) (016) and AdultCensus (Kohavi 1996)) show that our framework effectively improves the accuracy and fairness performances of the state-of-the-ee in-processing approaches (Zatar et al.) 2017a| Zhang et al. 2018 Roh et al. 202I) under correlation shifts. Also, our framework performs better than two-step baselines that first run an existing pre- processing approach (Kamiran & Calders 2011) and then an in- in-processing approach. We also show that our framework is still beneficial when we do not know the exact range of the shifted correlations.
Summary of Contributions (1) We introduce the notion of correlation shifts, which is important to connect the data bias changes and behaviors of fair training. (2) Using the notion of correlation, we theoretically show that existing in pn-processin fair algorithms are limited by the training distribution and may perform poorly on the deployment distribution. (3) We propose novel pre-processing step to boost the performances of fair in-processing approaches. (4) We demonstrate that our framework effectively improves the performances of the state-of-the-arr fair algorithms under correlation shifts.
2 LIMITATIONS OF FAIR TRAINING WITH FIXED DISTRIBUTION ASSUMPTION
Most in in-processing approaches for fairness assume that the training and deployment distributions are the same, which means that they assume the same level of bias as well. However, data bias may shift over time as confirmed by recent studies (Wick et al., 2019 Maity et al. 2021: Ding et al., 202T). which means that the deployment data may actually have different bias than the training data.
In fair training, the data bias reflects the relationship between a label y and sensitive group attribute z. For example, if all positive labels are in the same group, the data be considered highly biased. Conversely, if the labels are randomly assigned to all groups, the data can be considered unbiased.
A bias change in the deployment data may have an adverse affect Figure [b] shows a toy example that illustrates how a fair classifier's 's performance is affected by a bias change during deployment. Here, the bias can be expressed via correlation, and we discuss their
trained model's performance.
2

---

Under review as a conference paper at ICLR 2023
relationship in Sec. B The training distribution is biased, where Group has more positive labels than Group 1. On the other hand, in the deployment distribution, the bias decreases as the positive labels equally distributed for each group. Suppose we train a fair classifier on the training data as shown on the left side where the DP fairness is perfect (i.e., Pr(ý=1) are the same for the groups), but the accuracy is not result. On the deployment data, the DP worsens while the accuracy still remains imperfect. The underlying problem is that the classifier was trained with a different bias in mind.
In the next section, we formalize the notion of bias change and provide a theoretical analysis and simulation results that explain the above observations with the following notation and fairness metrics.
Notation & Fairness Metrics Let 0 be the model weights, x E X be the input feature to the model, y E Y be the true label, and ý E Y be the predicted label where ý is a function of (x. 0). Let Z E 2 be sensitive group attribute, e.g., gender or race. We assume a binary setting (Y Z 00 1)). We focus on the prominent group fairness metrics, demographic parity (DP) (Feldman et al. 2013) and equalized odds (EO) (Hardt al. 2016), where DP achieved when the positive prediction rates are 1- Pr()=112=0) and EO is achieved when the label-wise
the same for the groups (i.e., accuracies are the same for the groups (i.e., Pr(s=yly=yy 1= Pr Pryy-lyy) 0))), Vy E {0, 1)).
FAIR TRAINING UNDER CORRELATION SHIFTS
To systematically study the effects of data bias changes, we first analyze the achievable performances of fair training via correlation between y and z that can be used to explicitly measure data bias regarding sensitive groups (Sec. 3.T) and discuss the limitations of fair in- in-processings when this correlation shifts (Sec. 3.2 We define correlation as follows:
(y, >)cooreelttion Data bias can be represented via the correlation between the label y and the sensitive group attribute z, where the correlation represents statistical relationship between two random variables. We thus define (y, z)-correlation using Pearson's 's correlation coefficient (Rodgers) Covly-x)
& Nicewander 1988 Pyz a(y)o(zj+ which is known to effectively capture biases in real-world scenarios. Here, Cou(.) (-) is the covariance, and (-) is the standard deviation. As we assume y and are binary, we can express Pyz as follows (Cohen & Cohen
1975):
Pr(y = 1,z= 1) Pr(y = 0,z 0) Pr(y = 1,z = 0)
Pyz
Prly = Pr = Pr(z = P)
3.1 ACHIEVABLE PERFORMANCE ANALYSIS VIA CORRELATION
We identify the fundamental accuracy and fairness limits of fair training based on the (y, 2)-correlation of the data. We first analyze the most common case of improving the fairness w.r.t. single fairness metric. We then extend our analysis to the more complicated case of improving the fairness w.r.t. multiple metrics, which is important for capturing various social contexts, but has been seldom studied in the literature. In both cases, we show that the (y, correelltin determines the achievable performance of fair training.
CASE 1 - Fair Training w.r.t. a Single Metrie When improving group fairness, fair training is known to face accuracy-fairme fairer. Recently, Menon & Williamson (2018) investigate that the accuracy-fairnes tradeoff w.r.. demographic parity is affected by how much and are aligned in the data. For example, if the y and z values are identical for all examples, achieving high fairness may require low accuracy. In contrast, if y and are randomly set, fairness and accuracy can be achieved together. The following proposition shows this previous work's 's result.
tradeoff, where the accuracy is sacrificed to make the model
Proposition (From Menon & Williamson (018)). (Informal) When model is trained w.r.t. demographic parity, a high alignment between y and z leads to a worse accuracy-fairness tradeoff.
For our purposes, we infer a similar relationship using (y, z)-correlation based on Proposition I The following lemma makes connection between the (y, 7)-correlation and the conditional probabilities of y given z under some conditions.
Lemma 2. If the marginal probabilities of y and (i.e., Pr(y v) and Pr(z 2)) remain the same, the (y, correllltnn Pyz is proportional to the difference between the conditional probabilities of y given different values (i.e., Pr(y = 1/2 1) Pr(y- 1|. = 0)).
3

---

Under review as a conference paper at ICLR 2023
The proof for Lemma can be found in Sec. A.1 can infer that the (y, correlation also determines the accuracy-fairees. tradeoff under certain conditions. According to the previous work (Menon & Williamson 2018), alignment is defined as how many examples in each group have a specific label. We can thus measure alignment as Pr(y- = 5/7 1) Pr(y= = v'Iz = 0) where y * y and y, v {0, 1}. Then, we can convert this term into the difference between the conditional probabilities of y given different z values, which used Lemma - see details in Sec. A.2 As a result, we derive the following corollary, which shows that the (y, zorrrelllaionn determines the achievable eecuracy-fairnes tradeoff.
By applying Lemma 2 Proposition I
Corollary 3. When a model is trained w.r.t. demographic parity, and the marginal probabilities y and z remain the same, the achievable accuracy-fairness tradeoff of the model is determined by the (y: corrrelltt,... The higher the correlation, the worse the uccuracy-fairner tradeoff.
Remark 4. We can relax the assumption that marginal probabilities are fixed in Corollary When the marginal probabilities of y and z change up to "ly and "íz. respectively, Corollary can be generalized as follows: the achievable accuracy-fairess tradeoff is determined by Py.z 1/. where Pyz
the (y, coorreltonl and ni E lV /Prly=1)>>> -; V (
Thus,
the higher the Рy,z 7), the worse the accuracy faimmes tradeoff. In our framework, we actually allow the marginal probabilities y y and z. change up to T and L see details in Sec. 4.1
Simulation We now confirm our theoret-
Unfwiruew (DP & BD)
ical observations via simulation see de-
Untatrosse (DP)
0.2
tails on the setting in Sec. [.1 The left plot in Figure 2 shows the accuracy-unfairnes, performances of classifiers on two synthetic datasets with low and high correlations when using DP for measuring fairness (the plot on the right will be explained later). We measure unfairness where lower value indicates better fairness (i.e., perfectly fair when the value is 0) see the exact metries in Sec. 5 We generate various synthetic classifiers on the two datasets show the full range of possible model performances. The blue dots (red crosses) are the classi- fiers on the dataset with low (high) corre- lation. As result, low correlation results in better accuracy-fairness tradeoffs (i.e.. close to the bottom right). We will discuss the black stars and squares in Sec. 3.2
0.
Low Core.
Curz.
1.
0.7
0.8
0.9
Accuracy
Accuracy
Figure 2: Simulation results of accuracy-unfairess (left: DP and right: DP & EO) performances of various classi- fiers on two synthetic datasets that have low and high (y. z)-correlations. Each blue dot (red cross) indicates single classifier's 's performance on the low (high) correlation data. The lower the unfairness value, the better. For each dataset, we generate enough classifiers to show the full range of pos- sible performances. As a result, low correlation enables classifiers to attain better accuracy-fairness tradeoff's (i.e., close to the bottom right), regardless of supporting single or multiple fairness metrics. Also, the optimal classitiers trained on high correlation data (black stars) have suboptimal performances the low correlation data (black squares).
Remark 5. We note that Corollary ddoes not necessarily apply for equalized odds (EO), which is achieved when the accuracies conditioned on the true label are the same for the groups. In theory, perfect classifier can achieve perfect fairness W.E.t. EO, so the (y. corvellltnn does not determine the limits of the accuracy- fairness tradeoff w.r.t EO. However, classifiers are not perfect in practice, and we empirically observe that higher (y, z)-correlation leads to a worse accuracy-fairnes tradeoff' w.r.t. EO as well- see the empirical results in Sec. B.2
CASE 2 Fair Training w.r.t. Multiple Metries Beyond the single metric scenario, we now extend our analysis to support multiple fairness metrics together. Although supporting multiple metrics is necessary address fairness under various social contexts, most fairness works do not address this problem. A major challenge is that group fairness metrics are known to be mutually exclusive, which means that these metries cannot be perfectly satisfied together, unless the data is completely unbiased (Barocas al.) 2019: Kleinberg et al., (017)). Interestingly, our (y. z) -ocrrrelationn provides an opportunity to support multiple fairness metrics together as much as possible in a principled fashion. We focus supporting two metrics and leave the support of more metrics as future work.
We first analyze when a model can improve both demographic parity (DP) and equalized odds (EO) to get hints on the relationship between the (y, -ooorellatii and improving fairness w.r.. two metrics. Here, we focus on the e-fairness of both metrics, which is a relaxed version of perfect fairesss (Barocas et al. 2019), where e indicates the unfairness level of the model.
4

---

Under review as a conference paper at ICLR 2023
For example, can define e- c-DP Pr(y = 1/2 Pr(ý- 1/z 0)| and E-EO as [Pr(ý = y = y, 2 = =) Pr(ý = u/y = y. = 0) S E. Note that E E [0. 1 and a lower E means higher fairesss. The following proposition shows when a model can achieve both c-DP and E-EO.
Proposition 6 (E-DP & E-EO). Let model achieve both E-DP and E-EO. Then, the fol- - y/z=zyz= - Pr(y= = g/2 = z')l Pr(f = y = y, = 2) U, (0.
lowing inequality holds: Pr(y =
Pr(g = uly V 2)1 S 28, y # y',
The proof for Proposition 6 can be found in Sec. A.3 Here, the achievable fairnsss level is affected by the data bias, which is represented by the difference between the conditional probabilities of y given (i.e., Pr(y = y/z =) Pr(y = y/z z²))) For example, if the data is highly biased, a model cannot achieve high fairness (i.e., low E) unless the second term on the left side (i.e., Pr(y vly v Pry y/y y. =)I) decreases. However, the second term is related to the accuracy, where lowering it to zero causes the model to make random predictions or predictions that are all the same. Thus, Lemma Znd Proposition give the following corollary.
Corollary 7. When a model is trained w.r.t both DP and EO, and the marginal probabilities y and z do not change, the achievable fairness of the model is determined by the (y, z)-correlation if the accuracy does not change. The higher the correlation, the lower the achievable fairness.
Similar to the single-metin metric case, we confirm the theoretical result of supporting multiple metrics via the simulation in the right plot in Figure. 2 Again, we observe that low correlation enables classifiers to attain better accuracy-fainen fairness tradeoffs. We also leave discussions for improving fairness w.r.t. another prominent metric called predictive parity (PP) (Berk et al. 202I) together with DP and EO in Secs. A.4 (e-PP and e-DP) and A.5 (s-EO and e-PP).
3.2 EFFECTS OF CORRELATION SHIFTS
As the performance ranges of fair training are different according to the data correlation, a fair classifier learned on some training data does not necessarily have optimal performance on deployment data with different correlation. We say there is a correlation shift between two data distributions D1 and D2 when 10 -pe * O, where P indicates the (y zycorrellaio of D, Here, the correlation shift reflects the bias change between data distributions.
We revisit our previous simulation in Figure 2 that helps to understand how the correlation shifts affect fair training. We consider a correlation shift from training distribution with high correlation to deployment distribution with low correlation. We start from few optimal classifiers trained on the nigh-correlated data, which are denoted as black stars. These classifiers unfortunately have subop- timal performances the low-correlation deployment data (black squares). Hence, in in-processing approaches that assume the same training and deployment data distributions lose the chance of achieving better faimess and accuracy in the deployment distribution under correlation shifts.
Based on our theoretical and simulation results, we design a pre- pre-processing approach using (y. z)- correlation a tuning knob to give existing in-processing approaches better opportunity to achieve maximum accuracy and fairness performances in the following section.
4 FRAMEWORK
To improve the performance of fair training in the presence of correlation shifts, we propose a novel fair training framework that consists of two main parts: applying pre-processing approach for reflecting the shifted correlation and utilizing any existing in-processing algorithms for fair training on top of the improved data. We first design optimization problem for finding new data distribution that follows the shifted correlation by adjusting the (y, z)-class ratios. Then, we explain our overall training process and extensions support similar distributions. We note that our framework implicitly assumes that the distribution of input feature x does not change drastically see details Sec. A.6
4.1 OPTIMIZATION
We design an optimization that finds the best (y, z)-class ratios given a (y, z)-correlation Pyz by using Lemma 2 which shows that Pyz is proportional to the conditional probability difference under some assumptions. Note that the conditional probability can be written using the class weights. Let the original data ratio of each (y=y, =lz)sslass be Wy=zzz=y Let the new data ratio be w/ "y= which
5

---

Under review as a conference paper at ICLR 2023
is required to satisfy the shifted correlation of the deployment data. Note that Evy.euy=yzze 1. Also, let be the correlation constant that is the difference between the conditional probabilities of y given z in the deployment data. Let [a. 3] be the range of the correlation constant с (i.c., E 8)) The range may be known in advance or constructed by using distribution estimation techniques (Huang et al.) 2006] Zhang et al., 2013) for computing the shifted correlation and generating a range using its confidence interval. We use relaxed version of the assumption in Lemma 2 where the marginal probabilities of and z can change by up to "yy and Y, respectively, as discussed in Remark [ With these conditions, we set up problem whose goal is to minimize the squared differences between the original and new data ratios to help reduce the information loss, similar other pre-processing approaches (Zemel et al. 2013 Quadrianto al. 2019):
min
Wyy=
W
" y-1-1200 1,z-0
s.t. a y-1==1 Wy=02=1 Wy-1,z=0 + uy=0
1,2-1
3B,
1(1/=1==1 + w/ =/= - Premin (y = 1)] S ty: ((05= + =/==,==1)
Priaii(( 1)1 $ Te.
Σ Wymp 00 wy=u 1 Ay E (0, 1).2 10 Vig
where Primin (y 1 ++ 1y=1.z=0 and Priain (z = 1) = 1,== Note that c). the first constraint can be + uy--0,z1)-- 1/- (u/ 1, + 25- =
when we know the exact shifted correlation value (i.e., a = rewritten as 1
The above optimization is a non-convex quadratically constrained quadratic problem (non-convex QCQP), where the objective is quadratic, and the first constraint is non-convex quadratic. However, the non-convex QCQP is known be hard to solve (d' Aspremont & Boyd 2003). Thus, we apply the semidefinite (SDP) relaxation, which is one of the convex relaxations known to give reasonable lower bound of the optimal value of the original non-convex QCQP (Park & Boyd 2017):
min Tr(XP)) + 90 X,
S.t. Tr(PPa) 2 0, Tr(XP3) S O.
X
105 - Primin = 1)1 S ?/5 10 x- Primin (z = 1)1 S % 9 x
1, 05 S Vi,
xT
0 0
where Tr(-) is the trace, X xIT x
32
83
x4 T
=
[[w[,1
w/,n
w/,1
w T
qu 2 [11,1
W1.0 W0,1
0 T
92 [1
0
1
0
93= [1
1
0
0
94
[1
1
0
(1-0)/27
0
8/2
8)/2
Pb diag (1), Pa
0/2
1//2 0
and P3
8/2
(-1-9)/2
0
2)/
a/2
8)/2
8/2
(1-a)/2
a/2
0
(1-8)/2
9/22
Details on the conversion are Sec. A.7 we can solve it using convex optimization solvers (e.g., CVXPY (Diamond & Boyd 2016)).
Since the above SDP relaxation problem now convex,
4.2 OVERALL TRAINING
Algorithm 1: Fair Training under Correlation Shifts
Input: training data 1), original ratio Vly.2
We present the overall process for fair training under correlation shifts in Algorithm [ The algorithm includes two main parts: pre- pre-proeessing for reflecting the shifted correlation and in-rrocessing for fair training on top of the improved data. We first find the new (y, z)-class ratio Wy. based on the SDP relaxation of our optimization, which can be solved using convex optimization solvers (c.g..
correlation range (a, f., thresholds +y
w/ and 22 inoprccessing algorithm f sepscllee(wyya. a, 9, Ty. %2)
d2 + at uig-az= /Wygp
Vij E EI EYxz
d (deli= (Optional) d=
MinDistChange(2 D, Wy w/y2))
CVXPY). We then calculate the sample-wise weights to ensure that the sample weight sum in each (y=y, z=2)-
Draw new data D' from D via weighted
sampling w.r.t. d
class is Wy=y.z== n, where n is the total number of samples in the original training data. Within each (y=y.
-nintiial model parameters
for each epoch do: Update / based on f on D'
-l2sss all samples have the same weight. We then Output:0
draw new data D' from the original training data via weighted sampling according to the sample-wise weights. Finally, we train a model using an in-processing fair algorithm f on D'.
In addition, using an optional step MiinDisCChange, we can address the scenario where the pre-processing should minimally change the original training data, which is sometimes preferred in other applications (Kamiran & Calders 201 Details on this extension are in Sec. A.8

---

Under review as a conference paper at ICLR 2023
5 EXPERIMENTS
We perform experiments to evaluate our framework. We use logistic regression in all experiments. We evaluate the models on separate test datasets and repeat all experiments with five different random seeds. We use CVXPY
convex optimization solver. More detailed settings are in Sec. B.3
Fairness metries We focus on two prominent group fairness metrics: demographic parity
(DP) (Feldman et al., 2015) and equalized odds (EO) (Hardt al.) (2016). We measure
the faimess disparities (i.e.. unfairness) among sensitive groups as follows: DP disparity max Pr(y- = 1|z = -- = 1)] and EO disparity = max Pr(y = z = y) Pr(ý- = v/y 1)|. When measuring the unfaimness w.r.t. both DP and EO, we take the maximum of both disparities (i.e., max(DP disp., EO disp.)). Note that lower disparity means better faimesss.
Datasets We use a total of three datasets for training: one synthetic dataset and two real-world benchmark datasets. We generate the synthetic dataset using a method similar to Zafar et al. 20)7a). The synthetic training dataset has 2,000 samples and consists of two non-sensitive attributes (x1, x2), one sensitive attribute z, and one label attribute y see details in Sec. B.3 We also utilize two real datasets: ProPublica COMPAS (Angwin et al. 2016) consists of 5,278 samples, and its labels indicate recidivism; AdultCensus (Kohavi 1996) has 43,131 samples, and its labels indicate a person's 's annual income. We use gender as the sensitive attribute.
To construct test data representing the deployment distribution with shifted correlation (Ges), three methods: (1) re-sampling data within each (y, z)-class in the original test data, (2) modifying the Z values while fixing the x and y distributions in the original test data (see details in Sec. B. 3.3)
and (3) utilizing a newly-collected data, where we train on the AdultCensus dataset (Kohavi 1996). but test on recent version of this dataset called ACSIncome (Ding et al., (02T).
Baselines We compare our framework with three types of baselines: (1) vanilla (non-fair) training using logistic regression, (2) in- in-processing-on -nly training, and (3) two-step training that first runs an existing pre-processing algorithm and then an in-processing algorithm. For in- pr-oeessiiinnonly training, we use the following three approaches: Faimness Constraints (FC) (Zafar et al.) 2017a b), which adds an unfairness penalty term to the loss function; Adversarial Debiasing (AD) (Zhang et al., 2018), which adversarially trains a classifier with a fairness discriminator; and FairBatch (FB) (Roh) et al. 202I), which adaptively adjusts batch ratios among groups to improve fairness. When run in- propcesssinng approaches for multiple fairness metrics, we naturally extend each approach by combining the fairness constraints for different metrics- details in Sec. [B.3] For two-step training, we use a pre-processing algorithm called Reweighing (RW) (Kamiran & Calders) 2011) to debias the training data before running the above in-processing algorithms, where RW balances the data amounts across groups. We thus run the in- pr-processing algorithms on the less- biased data by RW.
Hyperparameters We consider three scenarios of knowing the range of the correlation constant e in the test data. As discussed in Sec. 4.1] one can infer this range using distribution estimation techniques (Huang et al. 2006)] Zhang et al. 2013). which gives a confidence interval that can be used as the correlation constant range [] (1) In Secs. 5.1 and 5.2 we assume that the range indicates the exact Cres value (i.e., a=3= (2) In Sec. B. 10 we assume that the range is Geaa + x%% (3) In Sec. [5.3] we assume that the range is incorrect. We set both 7y and 7/2 to 0.1, which is a larger value than the actual marginal probability changes in the test data. We also test for the values of 0.2 and 0.3, and the overall trends remain the same. The in-processings' hyperparameters are in Sec. B.3
5.1 ACCURACY AND FAIRNESS
We first compare the accuracy and fairness performances of our framework with baselines on the synthetic and COMPAS datasets in Table see many more results in Appendix B including the AdultCensus experiments (Sec. B.4), which show similar results. The test data has lower correlation than the training data. LR in the first row shows vanilla training without any fairness technique. Other baselines are clustered based on three in in-rrocessing approaches: FC, AD, and FB. For each in- in-processing approach X, applying our pre-processing (denoted as Ours + X) generally shows better fairness while achieving comparable or even better accuracies, either when supporting only DP or both DP and EO. The in-processing-onle baselines mostly show worse fairness and accuracies compared to applying our approach, because the in in-processing-onle baselines are trained with different biases from test data distribution in mind. The baselines of applying RW before in processss generally do not achieve high faimess compared to ours. The reason is that existing pre-processing approaches like RW simply mitigate data bias as much as possible, which is not always beneficial for the in processing. In comparison, our approach takes more principled approach by adjusting the bias according to
7

---

Under review as a conference paper at ICLR 2023
Table 1: Performances on the synthetic and COMPAS test datasets w.r.t. a single metric (DP) and multiple metrics (DP & EO). The test datasets are constructed via re-sampling from the original distribution. The correlation constant c of the test data is 50% of that of the training data. We compare our framework with three types of baselines: (1) non-fair training: LR; (2) in processing -nnly training: FC, AD, and FB; (3) two-step training: RW (Kamiran & Calders 201 1) + in processings. In the last row, we also show the performances of an in p--proeessing algorithm (FB) trained on the test distribution, which can be considered as the upper bounds.
Synthetic
COMPAS
Single (DP)
Multiple (DP & BO)
Single
Multiple (DP BO)
Method
Acc.
Unfair.
Acc.
Unfair
Acc.
Unfair
Acc.
Unfair.
LR
.865 1: .000
.173 1: .000
865 4: .000
.173 2: .000
.600
.000
.129 = .000
660 2: .00
.225 4: .000
PC 7afar ml. 2017a
.78 + 0111
.088 013
.83 + 0001
.015
000
656
.004
.150
001
654 + 000
.137
.034
RWTTC
848
0003
848
0004
.004
068
010
.124
031
Ours+KC
849 + 002
.834 +: .004
X51 + 0005
.060
+ .003
637
0008
.037
.021
652 + 420
.106 + .038
AD Zhang al 2018
762 1:
.116
.032
001
.821
1: 004
.088
003
.65
.003
.054
1: .015
.61
.005
.111 1: .036
845
006
847
000
.00
092
007
.145 1956
Ours+AD
814 1: 0011
.017 .006
.842
:: .009
.054 .008
.50 =: .003
.045
= .008
.664
000
.117 :: .039
FB (Rob al. 2021 RWFFB
821 +: 000
048
000
849
+ 001
[091
0035
647
0001
038
.013
650
0002
.187 + .099
859
.55
094
003
.197
Ours+FB
836 4: 0001 .003
.001
852 2: 00H
.058
.001
.648
004
.027
.001
.67
001
.130 :: .014
FB on tost dist. (upper bound)
.388 +: 002 .003 .002
.859 +: 0002 .088 0044
659 = 0011
.012 .008
664 #: 001 .055 + .014
the correlation shift with the purpose of improving the in- in-processing performance. As result, ours enables the proccesssing approaches to be closer to optimal performances (last row). In Sec. [B.5] we show that the pre-pro rre-proeessed data by our algorithm is more aligned with the true test distribution than the original training data in terms of (y. 2) corrrlllttnnnn and Wasserstein distance.
We observe similar results when using the two other test settings explained above. One is to modify the z values while fixing the x and y distributions using the synthetic dataset (Sec. [.6).. The other is to use two income datasets collected in the 1990s (i.e., AdultCensus) and 2010s (i.e., ACSIncome) for training and lesting, respectively, where they have different (y, cooorelltni values (Sec. B.7
In addition, we enrich the experimental results by 1) presenting accuracy and faimess trade-off curves of the in-processing-only baseline and our approach (Sec. [.S) and 2) showing the effects of the optional step in Algorithm | (Sec. B.9). which finds possibly-differen sample weights within each (y, zl-class to minimize the overall distribution change between the training and pre- pre-processed data.
5.2 VARYING THE CORRELATION OF THE TEST DATA
We compare the algorithm performances when varying the correlation of the test data. Figure 3a shows the accuracy and fairness performances of FB and Ours+FB. As the test data's correlation constant e varies from 10% to 70% of the training data's correlation constant, our pre-processing improves the accuracy and fairness of the in-processing algorithm. Interestingly, when the correlation of the test data differs significantly from the training data (i.e., close 10%), the in-processing-only baseline (FB) shows worse fairness. We suspect that the baseline is mitigating a different type of bias than that of the test data. On the other hand, our pre-proce pre-ersocssnn successfully enables the in-processing algorithm to achieve high fairness and accuracy (c.g.. for a 10% test corr., the unfairness decreases from 108 to 0.003). In addition, we vary the correlation from 110% to 150% in Sec. BB.1 where we show how our pre-processing enables in-processing to achieve high fairness.
5.3 HANDLING UNKNOWN CORRELATIONS
We also evaluate approach when the exact shifted correlation is unknown with two scenarios: 1) misspecifying the correlation in the algorithm and 2) giving a range of correlation shifts to the algorithm. /Scenario // We first consider when the shifted correlation is incorrectly specified. We set a- = 3 = Cspecified. where (specitied # Qust- Figure 3b shows the performances of FB and Ours+FB when the true correlation of the test data 60% correlation of the training data. Ours improves the in in-processing-ovee fairness when the specified correlation is higher than 30% Hence, our approach is still beneficial when the estimation error is within 10%. /Scenario 2) In Sec. B.10 we also run our approach with a range of correlation shifts, which can be inferred by distribution estimation techniques (Huang eet al. 2006; /Zhang et al. 2013). We expand the [a, 3] range to be [2%,st=x%% Gret +x%% instead of a constant value. As a result, there are two takeaways: 1) our framework successfully boosts the in proocessing-oll baseline performances when the [a, :9] range is reasonable, and 2) even if we do
baseline's 's accuracy for the entire range of considered correlations and improves
not have any information about the correlation shift, our framework performs at least as well as the
in-processing-only baselines.
8

---

Under review as a conference paper at ICLR 2023
FB
Oue 1PM
Accurary
2,800
OceedT
10%
Correlationn
0.00
Correlation
Corvelation (Specifled)
50%
Correlation (Specißed)
36%
(a) Varying the correlation of the test data.
(b) Varying the specified correlation in our algorithm.
Figure 3: Performances of FB and Ou on the synthetic data while (a) varying the correlation of the test data to have 10% to 70% correlation of the training data and (b) varying the specified correlation in our algorithm have 10% to 70% where the true correlation of the test data is 60%.
6 RELATED WORK
As model fairmess becomes essential for Trustworthy AI, various faimess techniques have been proposed to improve group fairness that do not discriminate specific demographics (Barocas et al.) 2019) Usually, the fairness techniques can be categorized into three prominent approaches: pre-, in- and post-processings - see their representative works in Sec. C.I Among the three categories, in-processing approaches are widely used for fair training due to their high fairness and accuracy performances, but most of them assume that the training and deployment distributions are the same.
Recently, there is an emerging focus on fair training under data distribution shifts. In Table 2 we summarize various
Table 2: Different distribution shifts.
Category
Type of Shifts
types of distribution shifts into the following two categories:
General distribution shifts
covariate shift
Pe(x]
label shift
Pr(y)
General distribution shifts (Singh et al. 2021 Rezaci et al. 2021; Chen et al., 2022; MMshhler& & Dalmasso 2022) fo- cus shifts involving the input feature (x) and label (y), which are widely studied in the traditional machine learning
concept shift
Pr(y (x)
Fairness
demographic shift
Pr(z)
specific
subpopalation shift
Pr(y. 2)
shifts
correlation shift (ours)
Pr(rly)
literature. Here the bias changes between label (y) and group (z) are not explicitly considered.
Fairness-specific shifts (Maity et al., 2021; An et al. 2022 Giguere et al., 2022) handle group (z) distribution changes, Z is especially correlated with fair training. We note that our work also falls into this category. A recent study (Maity et al. 2021 theoretically analyzes the behavior of fair training under change in bias called subpopulation shifts, where specific group has fewer positively- labeled examples during training time compared to deployment time. Another study (Giguere et al., 2022) designs a new test method serve fair model under another data distribution change called demographic shifts, where the subgroup distribution may change see an empirical comparison between this work and ours in Sec. B.12 A recent work (An et al. 2022) proposes a self-training- based transfer algorithm that requires specific model architecture (c.g., adversary network) to support data changes, including subpopulation shifts. In comparison, our contribution lies in 1) introducing the notion of correlation shifts, which is important for explaining with theoretical evidence the connection between the data bias changes and behaviors of fair training. 2) analyzing the fundamental performance limits of in-processing approaches in the presence of correlation shifts, and 3) proposing a pre- pre-prceessing step based on the theoretical analysis for assisting the existing fairness approaches. In addition, our framework is general and can support any model architecture and training procedure. We leave more detailed comparisons in Sec.
Another line of research is supporting robustness in fair training, including handling noisy groups (Clis et al. 2021 Wang et al.) (020) or poisoning attacks (Mehrabi et al. [021] [oolan et al.)
2020 Although this direction is not immediate focus, we do perform preliminary experiments in
Secs. B.13 and B.14 show some potential to support noisy group attributes or poisoning attack scenarios. We believe our method can be further extended with other robust training methods.
In addition to group fairness, we discuss other noteworthy fairness definitions including individual fairness (Dwork al. (2012a) and causality- fairness (Kilbertus et al., (017)) in Sec. C. also explain with a concrete example how our work connects to causality- -based fairness in Sec. C.3
We
7 CONCLUSION
We addressed the problem of model fairness in the presence of bias changes in the data. We first introduced the new notion of (y, z)-coreelation for capturing bias and analyzed the accuracy and fairness limitations of existing in-processing approaches in the presence of correlation shifts. We then proposed decoupling framework where pre-processing is used to adjust the correlation, and in- in-processing is used for unfairness mitigation. The pre-pree rreppoceessing step adjusts the data ratio among (y, z)-classes to reflect the shifted correlation and can optionally minimize the distribution change of the training data as well. Experiments showed how our pre- proceessssing enables existing in processing approaches to achieve high fairness and accuracy under correlation shifts and outperform baselines.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
We believe work can have positive societal impact by improving model fairness. In particular, we anticipate that our work can make existing fairness algorithms more applicable to real-world must carefully choose the right fairness
scenarios where data bias usually changes. However, metric to avoid unintended discrimination. To ensure privacy, we do not use any personal identifiers (e.g., name and date of birth). Also, we do not conduct experiments with human subjects.
REPRODUCIBILITY STATEMENT
To reproduce the experimental results, we describe details on experiments and implementation (e.g.. models, data preprocessing, hyperparameters, and devices) in Sec. 5 and Sec. B We also provide source codes for training and testing in the supplementary.
REFERENCES
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna M. Wallach. A reductions approach fair classification. In ICML. pp. 60-69, 2018.
Bang An, Zora Che, Mucong Ding, and Furong Huang. Transferring fairness under distribution shifts via fair consistency regularization. In NeurlPS, 2022.
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias: There's software used across the country to predict future criminals. And its biased against blacks. ProPublica, 2016.
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning. fairml- book.org, 2019. hatt::/wwae fai rm wwm.lboo..oo org
Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. Fairness in criminal justice risk assessments: The state of the art. Sociol. Methods Res., 50(1):3-44, 2021.
L. Elisa Celis, Lingxiao Huang, Vijay Keswani, and Nisheeth K. Vishnoi. Fair classification with noisy protected attributes: A framework with provable guarantees. ICML, 2021.
Yatong Chen, Reilly Raab, Jialu Wang, and Yang Liu. Fairness transferability subject to bounded distribution shift. In NeurIPS, 2022.
Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon. Fair generative modeling via weak supervision. In ICML,, 2020.
Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, I Luca Oneto, and Massimiliano Pontil. Lever- aging labeled and unlabeled data for consistent fair binary classification. In NeurlPS, pp. 12760- 12770. 2019.
Jacob Cohen and Patricia Cohen. Applied multiple regression/correlttion analysis for the behavioral sciences. 1975.
Andrew Cotter, Heinrich Jiang, and Karthik Sridharan. Two-player games for efficient non-convex constrained optimization. In ALT, pp. 300-332, 2019.
Steven Diamond and Stephen Boyd. CVXPY: A Python-embedee embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83)11-5,20 2016.
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), NeurlPS, 2021.
Flávio du Pin Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R. Varshney. Optimized pre-processing for discrimination prevention. In NeurIPS, pp. 3995 4004, 2017.
10

---

Under review as a conference paper at ICLR 2023
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In ITCS, pp. 214-226, 2012a. ISBN 978-1-4503-1115-1.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard S. Zemel. Fairness through awareness. In ITCS, pp. 216,226,22 2012b.
Alexandre d' Aspremont and Stephen Boyd. Relaxations and randomized methods for nonconvex qeqps. EE392o Class Notes, Stanford University, 1:1-16, 2003.
Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubra- manian. Certifying and removing disparate impact. In SIGKDD, pp. 259-268, 2015.
Stephen Giguere, Blossom Metevier, Yuriy Brun, Philip S. Thomas, Scott Niekum, and Bruno Castro da Silva. Fairness guarantees under demographic shift. In ICLR, 2022.
Clark R Givens and Rae Michael Shortt. A class of wasserstein metrics for probability distributions. Michigan Math. J., 31(2):231 1984.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In NeurIPS, pp. 3315-3323, 2016.
Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. NIPS, 19:601-608, 2006.
Heinrich Jiang and Ofir Nachum. Identifying and correcting label bias in machine learning. In AISTATS, pp. 702-712, 2020.
Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimi- nation. Knowl. Inf. Syst. 33(1):1-33, 2011.
Faisal Kamiran, Asim Karim, and Xiangliang Zhang. Decision theory for discrimination-avee classification. In IEEE ICDM, pp. 924 224-929, 2012.
Niki Kilbertus, Mateo Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Schölkopf. Avoiding discrimination through causal reasoning. In NeurIPS, pp. 656-666, 2017.
Jon M. Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determination of risk scores. In ITCS, pp. 43:1 43:1-43:23, 2017.
Ron Kohavi. Scaling up the accuracy of naive-hayes classifiers: A decision-tree hybrid. In SIGKDD, pp. 202-207, 1996.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual faimess. In NeurlPS, pp. 4066 4076. 2017.
Subha Maity, Debarghya Mukherjee, Mikhail Yurochkin, and Yuekai Sun. Does enforcing faireess mitigate biases caused by subpopulation shift? In NeurIPS, 2021.
Ninarch Mchrabi, Muhammad Naveed, Fred Morstatter, and Aram Galstyan. Exacerbating algorith- mie bias through fairness attacks. In AAAI, volume 35, pp. 8930-8938, 2021.
Aditya Krishna Menon and Robert C Williamson. The cost of faimess in binary classification. In FAT*, pp. 107-118, 2018.
Alan Mishler and Niccolè Dalmasso. Fair when trained, unfair when deployed: Observable fairness measures are unstable in performative prediction settings. ArXiv, 2022.
Arvind Narayanan. Translation tutorial: 21 fairness definitions and their politics. In ACM FAccT, volume 11 1170, 2018.
Jaehyun Park and Stephen Boyd. General heuristics for nonconvex quadratically constrained quadratic programming. ArXiv, 2017.
11

---

Under review as a conference paper at ICLR 2023
Gabriel Peyré and Marco Cuturi. Computational optimal transport: With applications to data science. Found. Trends Mach. Learn., (1(5-6):355-600) 2019.
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon M. Kleinberg, and Kilian Q. Weinberger. On fairness and calibration. In NeurIPS, pp. 5684-5693, 2017.
Novi Quadrianto, Viktoriaa Sharmanska, and Oliver Thomas. Discovering fair representations in the data domain. In CVPR, pp. 8227-8236, 2019.
Ashkan Rezaci, Angi Liu, Omid Memarrast, and Brian D. Ziebart. Robust faimeess under covariate shift. AAAI, pp. 1419-9427, 2021.
Joseph Lee Rodgers and W Alan Nicewander. Thirteen ways to look the correlation coefficient. The American Statistician, 42(1):59 (666,99666, 1988.
Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. FR-Train: A mutual information- based approach to fair and robust training. In JCML, pp. 8147-8157, 2020.
Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. FairBatch: Batch selection for model fairness. In ICLR, 2021.
Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss minimization. In ICML, pp. 5739-5748, 2019.
Harvineet Singh, Rina Singh, Vishwali Mhasawade, and Rumi Chunara. Fairess violations and mitigation under covariate shift. In FAccT, pp. 13, 2021.
David Solans, Battista Biggio, and Carlos Castillo. Poisoning attacks on algorithmic fairness. In ECML PKDD, pp. 162-177, 2020.
Philip S Thomas, Bruno Castro da Silva, Andrew G Barto, Stephen Giguere, Yuriy Brun, and Emma Brunskill. Preventing undesirable behavior intelligent machines. Science, 366(6468),999-1004,, 2019.
Serena Wang, Wenshuo Guo, Harikrishna Narasimhan, Andrew Cotter, Maya Gupta, and Michael Jordan. Robust optimization for fairness with noisy protected groups. In NeurlPS, pp. 5190-5203, 2020.
Michael Wick, swetasudha panda, and Jean-Baptiste Tristan. Unlocking fairness: trade-off revisited. In NeurIPS, 2019.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez RodrizgzZ and Krishna P. Gummadi. Faireess constraints: Mechanisms for fair classification. AISTATS, pp. 962-970, 2017a.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P. Gummadi. Fair- ness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In WWW, pp. 1171-1180, 2017b.
Richard S. Zemel, Yu Wu, Kevin Swersky, Toniann Pitassi, and Cynthia Dwork. Learning fair representations. In ICML, pp. 325-333, 2013.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In AIES, pp. 335-340, 2018.
Fuzhen Zhang. The Schur complement and its applications, volume 4. Springer Science & Business Media, 2006.
Junzhe Zhang and Elias Bareinboim. Fairness in decisiin-makingg the causal explanation formula. In AAAI, pp. 2037-2045, 2018.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. ICML, pp. 819-827, 2013.
Han Zhao, Amanda Coston, Tameem Adel, and Geoffrey J. Gordon. Conditional learning of fair representations. In ICLR, 2020.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX THEORY
A.1 PROOF FOR LEMMA
2
Continuing from Sec. 3.1 we provide proof for Lemma 2
Proof. We denote the Pearson's correlation coefficient between y and Pyz- By definition,
Pyz a(y)a(z)
Confy.z) For binary y and Z, we can rewrite Pyz as follows (Cohen & Cohen
1975
Cov(y,z) Pr(y = 1, = 1) Pr(y = - Prly = 1,2 0) Pr(y 0,2
pyr o(y)o(z)
VPr(y = 0) Pr(z
If the marginal probabilities of and (i.e., Pr(y = y) and Pr(z =)) remain the same,
Pyz
Pr(y 1, Pr(y = 0, 0)
Pr(y = z = 0) Pr(y = 0, 1) Pr(y 1, 1) Pr(y 0,z 0) Pr(y 1, 0) Pr(y 0, =
Pr(z= = 1)
Pr(z = 0)
Pr(z = 0)
Pr(z = 1)
Pr(y 1/z 1) Pr(y 0|z 0) Pr(y 1/z 0) Pr(y 0 1)
= Pr(y = 1|z = 1)(1 Pr(y = 12=-1zz00 = 0)) - Pr(y = 1|2 = 0) Pr(y = 1|2 = 1))
-Pr(y 1/z 1) - -Pr(y------1 1|z = 0) - Pr(y- - 1/2
1) Pr(y 1|z 0) Pr(y 1) 1/2 1)
PPr(y = 1/z = 1) -Pr(y- = 1|z =)) 0)-
Therefore, the marginal probabilities of y and z remain the same, the (y, z)-corellatio Pyz is proportional to Pr(y 1 1) Pr(y = 1|z 0). which is the difference between the conditional probabilities of given different z values.
A.2 THE ACCURACY-FAIRNESS, TRADEOFF WHEN IMPROVING FAIRNESS W.R.T. SINGLE METRIC
we show that the (y, cororllltt determines the accuracy-fairnes tradeoff under certain conditions by applying Lemma to Proposition] 1
Continuing from See. 3.1
The previous work (Menon & Williamson 2018) shows that a higher alignment between y and leads to a worse accuracy-fairnes, tradeoff w.r.t. demographic parity more details are described in Proposition
Menon & Williamson (2018). According to the previous work, alignment is defined as how many examples in each group have specific label. We can thus measure the alignment as Pr(y = 1|z = 1) + Pr(y = 0|z = 0). Then, we can rewrite the equation as follows:
alignment = Pr(y = 1/z = 1) + Pr(y = 0|2 = 0)
= =Pr(y 1/z = 1)+ + Pr(y 1/z 0))
= -14 + Pr(y = 1/2 = 1) Pr(y = 1/2 = 0).
By applying Lemma 2 to the above result, we observe that the alignment between y and z is proportional to ((y, -)-oorrelttion +1) when the marginal distributions of y and z remain the same. Therefore, when a model is trained w.r.t. demographic parity and the marginal probabilities of y and z do not change, higher (y, z)-correlation results in worse accuracy fairness tradeoff.
A.3 eDDP & EEO
Continuing from Sec. 3.1 we provide proof for Proposition 6
Proof. A model achieves both s-DP and e-EO when the two inequalities Prr( y/z 2)
Pr(ý = y/z = z²)l < E (i.e., e-DP) and Pr(y = yly = y. z = 2) Pr(y = yly = 8. = z')l S (i.e., e-HO) satisfied, where y a v'. y. E (0, and = * 2', (0, 1).
13

---

Under review as a conference paper at ICLR 2023
By combinnig the law of total probability and E-DP, we can get the following mecquality
/ / /
Pr(ý- = 1/2 = 2. / / 
P ())
- =E
- Se
/
= =) + Pr(ý = y/2 -  - ()
y|z -
- y/z
= =)
-
- =
- Pr(y = y|ا - =.
- z/) S.
(1)
Also, e-BO can be rewritten as follows:
y¹)) Pr(y P = 1/ y = =) /    - -Pr(y = y/z = y = y²) Pr(y = y/zz= = z - Pr(ý -
E + Pr(y= = y|z = y = v) s Pr(y = y/z = y, z = y) < + Pr( = 1/z = y = v). (2)
By substituting Eq. 2 to Eq.
1 we can get the following inequality:
(-e + Pr(y = y|z - =
- 2,y= = - y) Pr(y = y/z = z¹)
= z)
Prr(y-
< Prri = z/ =,,y = 5/) Pr(y = = =) + Pr(y = y/ z, - = y) Pr(y = =yz = z)
-Pig -Prl- = g/z = =z'y = 3¹) Pr(y= = =y'z== =²) - Pr(ý = =y/z = y = y) Pr(y = ylz = z
(3)
s (s + Pr(y = -y/== = -
= Pr(y = =y'z +/===)+(3) + (++ = y/z = y = =w) Pr(y = u/z =) 2)
- -Pr( -Pr(y = y/z u z = 5/) ))) Pr(y= y'
-
z= = z') - Pr(y = y/2 = y'y = y) Pr(y = y/z = z¹)
By subtracting Eq. 2 from Eq. 3
- (-E
+ Pr(y v/z = = Prly = -)= + (- Pr / y) Pr(y = y/z 2)
-
= - -Pr(Y = u/2 - 2' y v/) Pr(y = -/'zz= z') - Pr(y Pr(y = y/7 y|z = 2
9) Pr(y Pryy = y/z = z¹)
y)
00
S & + (E + Pr(y y/2 = / v\)) Pr(y = -/ + (s 2) ( =
y = y)) Pr(y = y|z= = 2)
Pr(y = y/z = 2'.y=  Pr(y = y iz = z Pr(y = y/ = v) Pr(y = y/z = 2')
By rearranging the terms, we get the followings:
-2c < (Pr(y = y/z = 2'.y = y²) - Pr(y- = y/2 = y.y = P) = s/2z = =) Pr(y = 1 = z()) S 2c
+ > 5 (Pr(8 = y/z = 2 =
= 5²) - Pr(S- = -/zz=y//s>
= = v)) - Pr(y = y/z= 2) - + Pr(y = y/z = =²)) S 2e.
Thus, we get the following inequality, which is in Proposition 6
IProy 3/2 2) Pr(y = w/2 z²)l Pr(s = ply y, =) Pr(O yly = 3/ z = z)1 - 2s.
A.4 E-PP & DD
Continuing from Sec. 3.1 we consider fair training w.r.t. both predictive parity (PP) and demographic parity (DP). We can define sPPP& & s-DP as follows:
-E - S Pr(y = ylğ = y, z = 0) - Pr(y = =ls = y < & (i.e., EPP
- 1)= < (i.e.. &
< Pr(g = u| = 0) - Pr(y = -
&DP)
Based on these defmitiions, we give a proposition for sPPP and DDP: Propooddtty 8 (c- & DD), Let a model achieve both PPPP and e-DP. Then, Then, the g rneguality holds:
Pr(y v. u/z
=) 2) - Pr(ý = y/z 2) + Pr(y = uا =,, 24) Se, x#2 2,2 E EZ, y V.W V.
z = 2) - Pr(y = y. 8 w|z
2 P = y = u.z=
Proof. From eDDP, & we get
- + Pr(ý = y = 1) < Pr(y z/2 0) S Pr(y y/z 1). 1)).
(4)
14

---

Under review as a conference paper at ICLR 2023
From E-PP, get
< Pr(f=v/z=0) 7r( Pr(ý=y,z=0)
Pr(z=0)
Pr(y=y.9=g,2=0) Pr(ý=y/z=1) Pr(ý=y,
z=1) Pr(j:))=))p(((-,,..)) pr(ý=v,7=1)
Pr(z=1) Pr(y=v,8=y,2-1 S..
(5)
By substituting Eq. 4 to Eq. 5 we can get the following inequality:
Pr(z = 0)
Prly=y.š= Pr(S v/2 1)
Pr(z ((1z=1) Pr(z=1)
: Prly=
(-e+ Pr(y- = 1)]
Pr(y y. = 0)
Pr(y=
Pr(j y. y,2= 1)
Pr(y v,2=
Pr(z 0)
Pr(y -r(y=,,=,,,z=0
Pr(z = 1)
Pr(y mry=va Pr(y 1,2= 1) (6)
SPr(S u/2 0) Pr(} y, 0)
Pr(s u/z 1) Pr(ý y. 1]
Pr(y= = v,2=
Pr(z 0) Prly= 0) Pr(ý = u/z 1) Pr(y = v,z=
Pr(z= 1)
Pr(y = v. = v,
< {s + Pr(s u|z= 1)} Pr(O = v,
Pr(y=1,z=0)
= v,z=1
By subtracting Eq. 5 from Eq. 6
Pr(z Prly=1,8=,,2=0 - -r(y-=/2=1) Pr(ý v,z 0)
Pr(z Pr(z=1) 11)
p. y
- Pr(3y)11111
Pr(y=
Pr(s 1,2=
Pr(y= v,z=1)
<0
Pr(z- = 0)
Pr(y = -. = v,2=
Pr(z =) 1)
Pr(y = n. v, 1)
S {e Pr(S=y/<==1
pr(j=y/z=1)
Pr(ý = =.z=0) 0)
Pr(9 = 9,2
Pr( = y, y,z=1)
Pr(ý = y.z 1)
By rearranging the terms, we get the following incquality:
- Pr(z 0) Pr(y = y. y.z= =0)
(Pr(ý = y = 0))²
< Pr(l = y/z = 1){ Pr(z Pryy ÿ 1,
Pr(z 0) Prly v y, 7
(Pr(ý = y, 2 = 1))²
(Pr(ý = -.z 0))²
Se Pr(z 0) Pr(y y. y, 0)
+
(Pr(ý = y. =
which is rewritten as follows:
- Pr(y = u/y = x +) = y/z=
Pr(ý 0)
< Pr((9 = 5/2= 1){ Pr(y = v/y = v, = 1) Pr(y = u/و = 0) =
Pr(ý = 1)
Pr(ý = 7 = 0)
& Pr(y = y/ = y,, = 0) Pr(ý = 00)
Pr(ý y/z = 0)
By multiplying
Pr(j=yl P
for all terms in the above inequality, we can get
Pr(ý 0) -
= 0) Pr(y = y y. 1) Pr(y = s/z = Pr(y = vly y,z 0)
S..
Pr(y b y, 2 Pr(S =/
(7)
Let A Pr(y = yl , + Pr(ý = y|z 0). Since -
Pr(ý = y/z = 0)
Pr(ý y/z 1) < E (i.c., DD)), we make another inequality from Eq. 7
(-c+ + Pr(y = y z = 1)) Pr(y = u/ = y,z 2 1) (e + Pr(y y z = 0)) Pr(y = u/y = y. z
A
Pr(ý- = y/z = 0) Pr(y = : y.z 1) - Pr(ý = y/z = 1) Pr = y = y.z 0)
(
A
+ Pr+Priý = 1
= = y.z = 1) - (- = P 3/و = y. 0
15

---

Under review as a conference paper at ICLR 2023
By subtracting Eq. 7 and Eq. 8
+ Pr(y = Po =) -++ = = P) 0)
-E+
A
0
(s+ -
1) (-8 Pr($- = = - y,
8 +
A
Since Pr(ý = y/ 2) P u/y y,z= 2) = -Pr(y = y. = 8/2 = =), we can rewritten the
above inequality as follows:
- =
= y. v/ - y, 0)} + Pr(y = v. - -
y/z 0)
-c+
A
00
v,z 0)) + = Pryy
P
v.
y/z = 0)
+
A
Now, let B= Pr(yy y,
1) B- Pr(y 5 = y,z 0) and C- Pr(y = v. = u/z 1)
Pr(y = y = =y) Then,
1 - A
(-c. Btt 1 (5. BBC.).)...
By arranging the terms,
B
C
< 0 0 (1+ B C
(1+
A A
1 +
-0
- 2
S.
A B
Therefore, conclude
Pr(( = y. y, Pr(y = y = 0) + Pr(y y y,z= Se. E.
g/z 0) Pr(y = n. g/z 1)
2 Pr(yy =
If we reverse the z values in the derivation, get the same formula with only the value changed in the above expression. As a result, we get the following inequality, which is in Proposition g
Pr(y y. u/z 2) - Pr(y y, g/z =1)l
2 P = u = y.z= 2) - Pr(y = y 2) + Pr(y = y/و y,z= x)) Se, 2 2 Z, VeY.
Therefore, to make e = 0. the numerator term Prr( = y, -)
y z'
yy= = the following is satisfied: (y, 9) 1 z, which implies 1 z and y 1 z. Here, y 1 z indicates that |Pr(y = 3/2 = z) Pr(y = y/z z²)l is zero. As result, perfectly satisfying PP and DP requires the data to be fully unbiased. We thus suspect that the achievable model fairness w.r.t. both PP and DP is affected by the (y, correeltin.
should be zero. When Prr( y, y|z = 2) Pr(y
A.5 E-EO & PP
Continuing from Sec. 3.1 we consider fair training w.r.t. both equalized odds (EO) and predictive parity (PP). We can define BOH & e-PP as follows:
< Pr(ý vly y, z 0) Pr(ý vly y,z 1) (i.e., E=EO)
- < Pr(y = v/ = y. z = 0) - Pr(y = yl = y,z = 1) < E (i.e., s-PP)
Based on these definitions, we give proposition for s-RO and e-PP:
16

---

Under review as a conference paper at ICLR 2023
Proposition 9 (s-EO & c-PP). Let a model achieve both E-EO and E-PP. Then, the following
inequality holds:
Pr(8=yly=y, 2=zz)
Pr(f=y)
PPr(y=y) - Pr(y-y/e=2) Pr(g-y|z=z*) | Se, 2 # 2, E E, y yeV.
Pr($-1,2=2) Pr(y seryyyy,2 ),
Proof. From s-EO, we get
-E -E+ Pr(ý yly = y. 1) - Pr(y sly = = 0) S = u/y y, 1). (9)
From E-PP, get
Pr(y = y.z 0)
Pr(y = -, 1,
- < Pr(y = yly y. 0) Pr(ý = y, - 0)
Pr($- y/y y, = 1)
S..
Pr(y = y.z =)
(10)
By substituting Eq. 9 to Eq. 10
we can get the following inequality:
Pr(y = y, 7. =
Pr(y = y 1)
(-c+ + Pr(y- = yly = y. z = 1))
Pr(s = uly = y, = 1)
Pr(ý = y. Z = 0)
Pr(ý = y.z = 1)
Pr(y = y. z = 0)
Pr(y = y. 1-=11 1)
S PPr(8 = -yly = y, = 0) Pr(O = y, 7 = 0)
Pr(ý = -yy = y. = = 1) Pr(ý = yy 7 ==1=1
(II)
Pr(y - y2 0)
Pr(y - -,2 = 1)
< (s + Pr(ý = y = y, z = 1))
-Pr(ý = yly = y, z == 1)
Pr(ý = y. z = 0)
Pr(ý = y, 2 = 1)
By subtracting Eq. 10 from Eq. 11
-e+ (-E Pr(y- vly y.z= 1)) Pr(y = y. z = 0)
Pr(y = y, 1) Pr(ý y.z 1)
-Pr( yly y.z= 1)
Pr(ý = y.z = 0)
<
S + (E Pr(ý = uly = y. z = 1)) Pr(y = y, z = 0)
Pr(y = y, y = 1)
- Pr(f = ply = y.z 1)
Pr(ý y.z 0)
Pr(ý y, 1)
By rearranging the terms, we get the following inequality:
- Pr(y = y. z = 0) + Pr(y = y. 7, = 0)
Pr(ý- = y, 2 = 0)
< Pr(s = yly = y, = 1)1 Pr(y- y, 0)
Pr(y y,z 1)
Pr(g = y, 7 = 0)
Pr(y = y, 7 = 1)
Se Pr(s- y, -z 0) 0)= + Pr(y ,, = 0)
Pr(y = y.z =02=0 0)
By multiplying -1(]= for all terms in the inequality, we can get
Pr(j=y,=0)
Pr(ý= - y. (()= Pr(y = v,z 0) Pr(y = v, 1)
- Pr(y= = y,z + Pr(y = y,z 0) Pr(y y.z 0) Pr(y y.z = 1)
S
which is rewritten as follows:
Pr(s y/y v,z
v,z = 1)
Pr(y = y,z y,z 0) Pr(ý = y,z= Pr(y = y. 1)} Se.
- Pr(ý=y.zz 0) + Pr(y = y, z - 0) (Prly = y,
Since Pr(y = ,, 0) = Pr(y- v) Pr(y- = y, 1) by the total probability law,
Pr(ý-yly-n,
Pr(s-y,z-0) Pr(ý=y,z=1) Pr(y= Se.
(Pr(y=y) Pr(y=
+ +
17

---

Under review as a conference paper at ICLR 2023
By arranging the terms,
Pr(ý y/y = 1,
Pr(y y, 0)
-ES Pr(ý- = y. 2 = 0) Prly - y, (Pr(y = y) Pr(y - y, 1) (1+
Pr(ý = y, = 1) S Pr(ý y)
Pr(ý
v.z=
7 E Pr(ý = y, 2 = 0) - Pr(y = - y,z=
(Pr(y y) Pr(y = v,z=
Pr(y = y.z = 1) SE.
Therefore, we can conclude
Pr(O = yly = 1)
Pr(j = y) = po)y-r(y- Pr(ý y, - 1) s.
P
Pr(ý = y. - 0) + Pr(y y.z-
If we reverse the z values in the derivation, we get the same formula with only the value changed in the above expression. As a result, we get the following inequality, which is in Proposition | Pr(ý=v) (Pr(y=y) - Pr(y=ul/=z") Pr(s=ulz=z() se, # 2, E Z. E W.
Pr(ý=-ly=w- 2=z)
Pr(y=v,z=2) + P Pr(y=v,2=2)
Therefore, ε when Pr(y = y) Pr(y ylz= and Pr(ý y) = y|z= (i.e..
y L. and y 1 z). Here, satisfying both y 1. and 1 z is the sufficient condition of perfectly satisfying EO and PP. Note that y 1L z implies that the data unbiased. We thus suspect that the achievable model fairness w.r.t. both EO and PP is affected by the (y, cooorrllato..
A.6 ASSUMPTION ON THE DISTRIBUTION OF INPUT FEATURE X
Continuing from Sec. clarify how we consider the input feature x: our framework (Sec. implicitly assumes the distribution of x does not change drastically, but we empirically observe that our framework performs well in the real-world scenario when the x distribution changes (Sec. 5
Our framework is designed to work best when the x distribution does not change, but does not strictly require this condition unlike other previous works on fairness-specific shifts (Maity et al.) 2021 Giguere et al. 2022). Specifically, Algorithm I Juses random sampling w.r.t. the new weights on each (y, 2)-csass, it is desirable that the distribution conditioned on each (y, z)-class stays the same. Of course, if the x distribution completely changes, the fairness or accuracy will degrade.
In this paper, we empirically show that our framework indeed performs well in real-world distribu- tion shift scenarios. For example, Table Section B.cclionB..ssa BB.7shows the accuracy and faimess performances when using the two income datasets collected in the 1990s (Kohavi, 1996) (i.e., AdultCensus dataset) and 2010s (Ding et al.) (2021) (i.e., ACSIncome dataset) for training and testing, respectively. Since the two datasets are collected separately in different periods, both the (y, 7)-correlation and the x distribution can be different between these datasets. Nevertheless, we observe that our framework successfully improves the accuracy and fairness performances of the in- processing-one baselincs.
A.7 SEMIDEFINITE RELAXATION
Continuing from Sec. 4.1
we provide details of the semidefinite relaxation in our optimization.
Recall our original optimization as follows:
min and
(Wy=y.z== uy-
Vy.3
W 1=12==1
w 11111==0
s.t.
S wy=1===1 + oy=0,z=1 w Wy=11z=0 + w/ u/=0,2=0
S B,
(10mm1.mm + w/ (,2m) - Promin (y = 1)] S Ty:
+ w w(-0)=))) - Pr (z = 1)] S Y/2
Σ S // = 1, < w/ wy=y,== S 1, Vy E (0,1), 2 E 10
Vy.z
18

---

Under review as a conference paper at ICLR 2023
where Py =1
and Pruain 1)
As the above optimization is a non-convex quadratically constrained quadratic problem (non-convex
QCQP), now apply the semidefinite relaxation (SDP relaxation) rewrite the above optimization using matrices:
(Park & Boyd 2017). We first
min x² Pox + %
x. Pax 0, x²' pœT 0,
1/2 x Primin (y = 1)1 S 7y,
|/3 x Primin (z = 1) S %/2
44 x
T
1, 0 Hi S Vi
where
32
33
xA
[w/,1
wi,0
W
0,0
40
-2 2
W1.0
W0,1
W0.0
92
[1
0
1
07 1
1
0
0
[1
1
-0/2
(1-a)/27
0
-8/2
82
Po diag(1) Pa =
-a/2
0
(-1-a)/2
and P3
-3/2
0
(-1-8)/2
(-1-a)/2
-2/2
0
(-1-8)/
0
-3/2
(1-a)//2
0
-a/2
(1-
9)2
0
8/2
As Px Tr((((xx*) we can get the following optimization:
min Tr(XPt) + 40 x
X,
s.t. Tr(PP,) 2 Tr Tr(XPp) S
4 Primin (y = 1)] S 7y,
1/3 x Pr (a 1) S %
q = 1, 0 H2 S 1 Vi,
X Zx
where Tr(-) the trace and X Tx.
In the above optimization, the last constraint is non-convex. Thus, we relax the non-convex constraint X xx into - xx= 2 0, which is convex, and then use a Schur complement (Zhang 2006 to
get the final SDP form:
min Tr(XPo) % X
s.t. Tr(P 2 0, Tr(XPb) <
1 Priain (y 1)1 S 7y,
1 σ Preain (a 1) S 7/2
d r = 1, 0 x, < Vi,
T
When we solve the above SDP relaxation problem using convex optimization solvers, we set
X x 1 Then, get the solution
the 5x5-matrix A the variable indicate
x4] T
32
33
by taking the first four elements of the last vector in the resulting matrix A.
A.8 OPTIONAL STEP FOR MINIMIZING OVERALL DISTRIBUTION CHANGE
Continuing from Sec. explain the details the optional step in Algorithm I We the
optional step (MinDi st Change) to find non-unin non-uniform data sample weights within each (y, classsss that minimizes the overall distribution change in terms of the Wasserstein distance (Givens & Shortt
19

---

Under review as a conference paper at ICLR 2023
1984), rather than using identical weights. In the optional step, first divide the examples in each (y=y, z=2) --class into two sets {y=y, z=2, t=0) and {y=y, Z=2, t=1] with equal numbers, where t specific feature that can be criterion for dividing the examples. For example, median of a non-sensitive attribute X1 and set t = for an example if the x₁ value of the example is otherwise. We then make candidate set of partial weights wy Wy==.z==' Note that we can
can find
lower than the median of X1. We
ey=y,==,.=1 on each (y=y, z=z, t=f)-class, where > wy y extend beyond the binary setting to find more detailed partial weights. For each partial weight candidate, we calculate the candidate sample-wise weights (dump) to ensure that the sample weight sum in each (y=y, z=z, t=t)-class Wy==,,==s,= n. where is the total number of samples in the original training data. Then, we draw new data D from the original training data D via weighted sampling according to dump- Then, we calculate the Wasserstein distance between and D via an optimal transport technique (Peyré & Cuturi (019). As result, the algorithm returns the final sample-wise weights (dmin) that result in the closest distribution from the original data.
Algorithm 2: MinDistCCange
Input: train data D, original ratio Wlyz, new ratio w'y,
In each (y=y, z=2)-class, divide the examples into two sets {y=y, z=2, ==} and {y=y, z=2, t=1] with equal numbers partials
2/m,
Wy=yzz= - []. Vy, =) E x
for each (y=y, class do for p in partials do
" ",=0_=2,=0 w/ wy==ym==" p
W yymg,-m,mm1
w/ v,0== (1 - p)
Append [w] ( -y- to Wy=gz==
candidates Wy=1,==1 yy=1 X Wy=0,z=1 Wy=0,z=0
minD + an initial large value
partial-weight in candidates
dj 4- //y=,,,=,,=//////// (Wy 0.5). Vj E (y,,,,), dump ii...
{0,1}
Draw data 1 from D via weighted sampling w.r.t. damp
wassD < Calculate the Wasserstein distance between D and D via optimal transport
if wassD < minD then
mmnn dmp
minD = wass
Output: d
B APPENDIX EXPERIMENTS
B. SIMULATION SETTINGS
Continuing from Sec. 3.1 we explain the details the simulation. The goal of the simulation is to confirm our theoretical observations by generating various synthetic classifiers that show the full range of possible model performances. To this end, we generate 1000 data samples, where each sample has y and z features. We first set and z to have specific correlation. We then generate various synthetic classifiers to have different predicted labels ý by varying the probability Pr(ý = 1) in each (y, zllcsas,, regardless of the other input features. Note that we do not train actual models (e.g., logistic regression, SVM). For each synthetic classifier, we measure the accuracy and fairness performances based on y, and the classifier's y. We repeat the above procedures while varying y and z to have different correlations. As a result, we get the simulation results in Figure by plotting all the classifier accuracy and fairness performances within each (y, -)roorellt
20

---

Under review as a conference paper at ICLR 2023
B.2 EMPIRICAL OBSERVATIONS FOR EQUALIZED ODDS
Continuing from Sec. 3.1 we perform an experiment to observe that higher (y, z)-correlation leads to worse accuracy-faiin fairness tradeoff w.r.t. equalized odds (EO) in practice. Figure shows the accuracy-unfairmess performances of the logistic regression model trained using FairBatch (Roh et al.) 202T) to improve EO two synthetic datasets with low and high (y, z)-correlations. As result, low correlation enables classifiers to attain better accuracy-fairnee tradcoffs (i.c., close to the bottom right) w.r.t. EO. This experiment shows that the (y, z)-correlation indeed affects the accuracy-fairnet tradeoff w.r.t EO in practice.
Unfairness (EO)
0.3
Low Corr.
High Corr.
0.2
0.1-
0.81
0.85
0.86
0.87
Accuracy
Figure 4: Accuracy-unnaaete unfairness performances of fair training with FairBatch (Roh et al. 2021 on two synthetic data with different (y, z) correlations. We measure faimeess w.r.t. equalized odds (EO),
B.3 OTHER EXPERIMENTAL SETTINGS
Continuing from Sec. 5 we provide more details on the experimental settings. The batch sizes of the synthetic, COMPAS, and AdultCensus datasets are 100, 200, and 2000, respectively. For the synthetic dataset, we use 2000 samples for the training dataset and 1000 samples for the test dataset. For the real datasets, we split the entire data into 4:1 for the training and test datasets. We set the learning rate to 0.0005. Our experiments are performed using PyTorch on a Linux server with Intel Xeon Silver 42IOR CPUs and NVIDIA Quadro RTX 8000 GPUs.
We generate the synthetic training dataset with 2,000 samples and consists of two non-sensitive attributes (x1, x2), one sensitive attribute z, and one label attribute y. Each sample (x1, x2,
is drawn from the following Gaussian distributions: (x1,xg)ly 0 (V[[22; -2]. [10, 1; 1, 1)) and (x1,x))b x2)ly = ~ (2; 2]. [5, 1; 5)). For the sensitive attribute z, we generate a biased
distribution: Pr(z 1) Pr((x) xyy 1)/ 1)/Pr((X(, xg)ly = 0) Pr((x), x2)ly = 1)] where
(x1, x6) = (x1 cos(TT/ 44 X2 sin क /)) sin(T/4) + X2 cOs(m/44).
When construct the synthetic test dataset by modifying the z values while fixing the and y distributions in the original test data, we change the k value in (x1,x)) = (x1 cos(TT/) X2 (iT()/)),, X1 sin(T/ (/kk/ X2 cos(T/k)).
For all in in-processing algorithms, we start from a candidate set and use cross-validation on the (pre- processed) training data choose the hyperparameters that result in the best fairness while having an accuracy that best aligns with other results.
To support multiple fairness metrics, we naturally extend each in-processing approach by combining the fairness constraints for different metrics via a tuning knob that adjusts the importance between each metric. Here, the fairness constraints are implemented differently in each algorithm. Fairness Constraints (FC) (Zafar et al.) (0177) adds each unfairness penalty term to the loss function. Thus, we extend FC by adding multiple penalty terms and adjust the importance between each penalty term by a tuning knob. Adversarial Debiasing (AD) (Zhang et al., (2088) utilizes a discriminator of each fairness metric for the adversarial training. Thus, we extend AD by adding multiple fairness discriminators and adjust the importance between each discriminator by adding tuning knob in the classifier's 's loss function. FairBatch (FB) (Roh et al., (022)) solves a bilevel optimization that has an objective for minimizing fairness disparity according to each faimesss metric. Thus, extend FB by minimizing the maximum of fairness disparities (e.g., max( Pdissp, EOdisp.) and adjust the importance between each disparity using tuning knoh.
21

---

Under review as a conference paper at ICLR 2023
B.4 ACCURACY AND FAIRNESS- ADULTCENSUS
Continuing from Sec. 5.1, we show the results on the AdultCensus dataset. Table shows the accuracy and fairness performances of the algorithms on the AdultCensus test dataset W.r.t. single metric (DP) and multiple metrics (DP& & EO). Other setting are identical to Table | results where our framework improves accuracy and faimess of the in-processing-one baselines and also shows better fairness than the two-step baselines using RW.
We observe consistent
Table 3: Performances on the AduliCensus test dataset. Other experimental settings are identical to those in Table
Single (DP)
Multiple (DP & EO)
Method
Acc.
Unfair.
Acc.
Unfair.
LR
.84 + .001
.074 + .002
.84 = .011
.074 + .002
FC
805 .013
.021
.004
807 = .013
.078 .017
RW+FC
.100 + .004
.020 + .006
.810 = .0088
.048 + .013
Ours+FC
.820 + .003
.005 + .002
.805 + .014
.033 + .013
AD
.777 + .022
.007
.004
.806 + .011
.052 .004
RW+AD
.088 + .010
.025 + .007
800 = .015
.035 + .003
Ours+AD
.803 + .009
.007 + .003
.805 + .009
.033 + .003
FB
825 .002
.017
.001
.86 = .001
.049 .001
RW+FB
.188 .008
.020
.005
.817 + 008
.071 .010
Ours+FB
.244 + .001
.008 + .003
.826 + 003
.037 + .012
B.5 HOW THE PRE-PROCESSED DATA ALIGNS WITH THE TRUE TEST DATA
Continuing from Sec. 5.1 we also show that the pre-processed data distribution (Drre) by our algorithm is more aligned with the true test distribution (Drea) compared to the original training distribution (Drain)) in terms of (y, z)-correlation and Wasserstein distance. We calculate the second- order Wasserstein distance via an optimal transport technique (Peyré & Cuturi 2019) on the synthetic data. We experiment on three degrees on shifts and make two observations in Table 4 1) the correlations of Dre and Dies are indeed more similar relative to Duain and 2) the Wasserstein distance between Diee and Drest is lower than between Dimin and Drest- Both observations confirm that the reweighed data aligns well with the test data.
As our method improves the alignment between Dpre and Drest the in-processing algorithms trained on Dre (i.e., ours + in-processing algorithms) show high accuracy and fairness performances on D as shown in Sec. 5
Table 4: Alignment between data distributions. We the synthetic data.
Level of conrelaionn shift Gest
Severe
Normal
No shift
Crnin (correlation in Dimin)
0.3591
0.3591
0.3591
Gpee (correlation in D D
0.0359
0.1796
0.3590
Crest (correlation Dess)
0.0360
0.1800
0.3591
Wass. dist. between Dumin and Drest 0.7966
0.6807
0,4399
Wass. dist. between Denin and Drest
0.6257
0.6022
0.4399
B.6 ACCURACY AND FAIRNESS OTHER TEST DATA CONSTRUCTION: SYNTHETIC DATA
Continuing from Sec. 5.1
we compare the algorithm performances when using different method to construct the test dataset of the synthetic-date experiment. Table shows the accuracy and fairness performances the synthetic dataset when constructing the test dataset by modifying z directly.
22

---

Under review as a conference paper at ICLR 2023
Table 5: Performances the synthetic test dataset. The test dataset constructed via modifying the values of the original distribution. Other settings are identical those in Table
Single (DP)
Multiple (DP & EO)
Method
Acc.
Unfair.
Acc.
Unfair.
LR
871 + .000
.138 + .000
.871 + .000
.138 + .000
FC
8055 1: .006
.040
.005
.800 + .001
.119 + .003
RW+FC
.847 + .005
.047 + .007
.852 = .005
.053 + .004
Ours+FC
.831 + .003
.005 + .004
.854 = .004
.052 + .002
AD
.792 .012
.048
.009
.855 = 0088
.133 + .006
RW+AD
8366 + _010
.042 + .008
.849 + .004
.054 + .004
Ours+AD
.820 + .006
.005 + .003
.847 + .022
.057 + .007
FB
804 0011
.051
.002
.831 + .001
.128 + .002
RW+FB
844 + .044
.036
+ .003
883 = 000
.057 + .011
Ours+FB
824 1 .002
.015 1 .002
.844 1 .003
.054 1 .003
We again generate the synthetic dataset using a method similar to Zafar et al. (2017a). The synthetic dataset consists of two non-sensitive attributes (x1, X2), sensitive attribute z, and one label at- tribute y. Each sample (X1, X2, y) is drawn from the following Gaussian distributions: (X1, x2)ly = ([ [5, 1; 5)). For the sensitive at- tribute Z, we generate a biased distribution: Pr(z= 1) = Pr( x2) (P(( x2)ly = 0) + Pr((x1, x2)ly 1)] where (x1, x6) = (X1 cOs(T/k)) X2 sin k), X1 sin( (k) +X2 COS(IT /o))) For
O 2[-2 2], [10, 1; 1J) and (x1, X2) =
the training dataset, we set & to 4. We then change the k value to generate the test dataset with 50% of the training data correlation.
As result, we observe consistent results where our framework improves the accuracy and fairness performances of the in pn--poocessingonne baselines and generally shows better fairness than the two-step baselines using RW.
B.7 ACCURACY AND FAIRNESS- OTHER TEST DATA CONSTRUCTION: REAL DATA
Continuing from Sec. 5. we compare the algorithm performances when using different method to construct the test dataset of the real-data experiment. Table 6hows the accuracy and fairness perfor- mances when using the two income datasets collected in the 1990s (Kohavi 1996) (i.e., AdultCensus dataset used in Sec. B.4b and 2010s (Ding et al. 202I for training and testing. respectively. Here, the 1990s and 2010s datasets have the different (y, 2) --correlation values 0.214 and 05100, respectively. In addition, the imput feature x distribution also slightly shifted, as shown in Figure 5 As a result, we observe consistent results in Table where our framework improves the accuracy and fairness performances of the in-processing-onj baselines.
2nd Principal Component
AdultCensus (1990s)
600
ACSIncome (2010s)
40
20
-20
-20.0
0.0
200
400
000
1st Principal Component
Figure 5: Principal component analysis (PCA) results on the input feature x of the two US income datasets: AdultCensus [Kohavi 1996) and ACSIncome (Dig et al., (02T). Here, the distributions of the principal components on input feature x are different in the two datasets.
23

---

Under review as a conference paper at ICLR 2023
Table 6: Model performances the ACSIncome data (Ding et al. 2021 collected in the 2010s. The
models are trained AdulICensus data (Kohavi 1996) collected the 1990s. Both datasets have labels that indicate each person's 's annual income.
Single (DP)
Method
Acc.
Unfair.
FC
.646 + .013 .104 + .015
Ours+FC
.652 + .015
.090 + .017
.637 + .031
.137
.037
Ours+AD
.644 + .024
.060 + .039
FB
.643 + .007
.128 + .021
Ours+FB
.670 + .004
.098 + .002
FB on test dist. (upper bound) .704 + .007 .044 .035
B.8 ACCURACY AND FAIRNESS TRADDOFF CURVES
provide the accuracy- fairness disparity (DP) trade-off curves of FairBatch (FB) and Ours+FB on the the synthetic data in Figure Our framework shows a better accuracy- fairness -eirrecy-fairness tradeoff compared to the in in-processing approach (FB). FB shows "inversed" curve, as it is trained with wrong biases in mind.
Continuing from Sec. 5.1
Unfairness (DP)
0.100
FB Ours+FB
0.075
0.050
0.025
0.000
0.82
0.83
0.84
0.85
Accuracy
Figure 6: Accuracy- fairness disparity trade-off curves of FB and Ours+FB on the synthetic data.
B.9 USING THE OPTIONAL STEP
Continuing from Sec. 5.1 we show the results of when using the optional step using MinDistChange in A Algorithm]] I This step finds possibly-diffcee data sample weights within each (y, z)-class to minimize the overall distribution change between the training and pre-processed data. Figure 7 shows the Wasserstein distances between the original training data and pre-processed either the basic or optional step, and the correlation constant ranges from 10% to 70% of the training data's correlation. As a result, the optional step generally reduces the Wasserstein distance between the training and pre-processed data distributions, especially for smaller e values. In addition, Table 7 shows the performances on the synthetic test dataset when our algorithm uses either the basic step (Ours) or the optional step (Ours+Optional), As a result, when using the new data from the optional step as an input of the in- un-processing approaches, the final classifier's s accuracy and fairness are not sacrificed much compared to using data from the basic step, while minimally changing the data distribution.
data when algorithm
Wass, Dist.
0.4
Basic
Optional
0.2
10%
30%
30%
70%
Correlation
Figure 7: Wasserstein distances between the original training data and pre-processed data when algorithm uses either the hasic step the optional step, while varying the target correlation have 10% 70% correlation the training data.
24

---

Under review as a conference paper at ICLR 2023
Table 7: Performances the synthetic test dataset. Other settings identical those in Table
Single (DP)
Method
Acc.
Unfair.
Ours+FC
.849 + 002 .034 + .004
Ours-OOtiioal-FCC
.830 + .003
.027 + .005
Ours+AD
.814 + .011
.017 + .006
Ours+Optional+AD
.808 + .007
.014 + .006
Ours+FB
8836 + 001
.003
.001
Ours+Optional-FB
.842 = .002 .010 + .004
B.10 SETTING THE CORRELATION RANGE TO CTEST + x%
Continuing from Sec. 5.2 we evaluate our approach when [a, [] |ces x%, Gest 2%). Figure shows the accuracy and fairness performances of FB and Ours+FB, while varying the correlation constant e of the test data. Here, we set the specified correlation range our algorithm [a. | [Crest 2%, Crest + x%),, where I E {10, 50, 100). When = 10%, we observe similar trends as Figure 3a where our framework improves the accuracy and fairness performances of the in tn-ppoocesinn-onn. baselines. When and fairness performances of our framework converge to the in- in-processing-nne 50%. the performances of our framework are in between those of when
100% (i.e., the worst-case setting of [a:, 38). the accuracy baselines. When 10% and 100%. baseline
There are two takeaways: 1) our framework successfully boosts the in in-processing-ont performances when the (a, /] range is reasonable, and 2) even if we do not have any information about the correlation shift, our framework performs at least as well as the in-processing-onte baselines.
Ourse-/BB, fur 1856
Unfairness (DP
0.100
0.86
6098
Accuracy
Oune-PB, fur
100%
0.075
FB
0.050
0.84
0.025
0.82
D.000
10%
30%
50%
70%
10%
30%
50%
70%
Correlation
Correlation
Figure 8: Performances of FB and Ours+FB on the synthetic data while varying the correlation of the test data Ciest have 10% 70% correlation of the training data. We set the specified correlation range in algorithm [a,,
CCest 2%, Cee + 2%), where æ E {10, 50, 100).
B.11 VARYING THE CORRELATION OF THE TEST DATA: MORE BIASES
Continuing from Sec. 5.2 vary the correlation of the test data to have biases than the training data. Figure Sareee/showw shows the accuracy and faimess performances of FB and Ours+FB, while varying the correlation constant of the test data up to 150%. When the correlation increases more than 100% of that of the training data, the in- un-processing-onff baseline (FB) cannot improve fairness because the training data does capture the bias level in the test data. Hence, the in-processing-onl baseline cannot be used in applications that require high fairness. On the other hand, our pre-pre pre-processing enables the inoorocssin approach to achieve the high fairness that may need, with some accuracy degradation.
B.12 EMPIRICAL COMPARISON WITH GIGUERE ET AL.. 2022
Continuing from Sec. 6 we add new baseline called Shifty(Giguere et al.) 2022), which focuses on the distribution shift most relevant to ours. Shifty first trains candidate models on the training data and selects only the models showing high fairness in the shifted deployment data. To give favorable condition to Shifty, we assume that Shifty knows the exact test distribution. Table shows the accuracy and faimess performances of the in pn-precsssing-on our framework w.r.t single metric (DP) and multiple metrics (DP & EO) in the synthetic and COMPAS datasets used in Table As result, both ours and Shifty improve the fairness of the
baseline FairBatch, Shifty, and
25

---

Under review as a conference paper at ICLR 2023
0.85
Unfairness (DP)
0.100
Accuracy
0.075
0.050
0.75
FB
0.025
PB
0.70
0.000
10%
50%
70%
90%
210%
1506
70%
110% 190%
Correlation
Correlation
Figure Performances of FB and Ours+FB on the synthetic data while varying the correlation of the test data to have 10% to 150% correlation of the training data.
Table 8: Performances on the synthetic test dataset. We compare in nn-processing-onne -nnliy baseline FairBatch (Roh
202T), Shifty (Giguere al. 2022,, and our framework. Other settings are identical those Table
Synthetic
COMPAS
Single (DP)
Multiple (DP & BO)
Single (DP)
Multiple (DP & EO)
Method
Acc.
Acc.
Unfair.
Unfair
Acc.
FB (Roh et al. 2021
821 + .000
.048
.000
849 + .001 .091 + .005
.67
.000
.038 .013
.600 + .022
.187 .019
FB Fsmmly
838
.001
.024
008
844
.033
.03
008
547
001
.029
001
.699
002
.162 002
OurseFB
836 + 001
.003 001
.52 = .004 .058 .011
.648
004
.027 .001
.657 .004 .130 .014
in prwppssiinn----eo only baseline, but ours shows better fairness than Shifty while achieving similar or higher accuracy. The reason is that Shifty is selecting the final model among the candidates that were already trained on the original training data, whereas ours trains a new model on the improved (pre-processed) data.
B.13 SUPPORTING NOISY GROUP ATTRIBUTES
Continuing from Sec. 6 evaluate the performance of method increasingly noisy data and observe that our method can potentially be extended to the noisy group scenario.
We perform new experiment, where 10-50% of the group information in the training data is randomly flipped. Table 9 shows the accuracy and fairness performances of the in-processing-onte baseline FairBatch (FB) and our framework (Ours+FB) w.r.t. demographic parity (DP). Here, FB cannot achieve high fairness performance (i.e., low DP disp.), as the group distribution in the training data is different from that in the data. In comparison, our framework enables FB to achieve relatively high fairness at the expense of some accuracy degradation.
We note that these results are preliminary and only show that our method has some potential to support noisy group attributes, and we believe there is plenty of room for improvement. In particular, we believe our method can be further extended with other robust training methods like (Shen & Sanghavi 2019)
Table Performances the synthetic test dataset when the group information the training data is randomly flipped. We train algorithms w.r.t. demographic parity (DP).
10% flipping
30% flipping
50% flipping
Unfair.
Unfair. (DP)
Unfair.
Method
Aec.
Acc.
Acc.
(DP)
(DP)
FB
0.818
0.087
0.864
0.165
0.883
0.250
Ours
FB
0.808
0.069
0.823
0.101
0.838
0.142
B.14 SUPPORTING POISONING ATTACK SCENARIO
Continuing from Sec. 6 we evaluate the performance of our method on poisoned data and observe that our method can potentially be extended to the poisoning attack scenario.
We perform a new experiment using one of the poisoning attack methods for fair training (Roh et al. 2020), where 10% of the training labels of specific group are flipped so maximize the accuracy
26

---

Under review as a conference paper at ICLR 2023
degradation. Table 10 shows the accuracy and faireess performances of the in in-processing -onlly baseline FairBatch (FB) and our framework w.r.t demographic parity (DP) on the poisoned data; also report FB the clean data, which can be considered as the upper-bound performance. Here, when achieving similar fairness, our framework improves the accuracy of FB on the poisoned data. We suspect that, while our pre-processing reduces the correlation shift between training and deployment data, the effect of the poisoning attack is also mitigated.
We note that these results are preliminary and only show that our method has some potential to support the poisoning attack scenario, and we believe there is plenty of room for improvement.
Table 10: Performances the synthetic test dataset when 10% of the training labels are poisoned (Roh et al.
2020
We train the algorithms w.r.f. demographic parity (DP).
Method
Acc.
Unfair. (DP)
FB
0.699
0.026
Ours
0.721
0.024
FB on the clean data (upper bound)
0.773
0.023
C APPENDIX MORE RELATED WORK
C.I TRADITIONAL MODEL FAIRNESS
As model fairness becomes essential for Trustworthy AI, various fairness definitions have been proposed to reflect legal and social requirements (Narayanan 2018). Among the definitions, we focus on group fairness, which aims to not discriminate specific groups There are three prominent group fairness metrics: demographic parity (Feldman et al.) 2015), equalized odds (Hardt et 2016), and predictive parity (Berk et al., 2021). To support the fairness metrics, various fairness techniques have been proposed, which can be categorized into three prominent approaches: (1) pre-processing
approaches (Kamiran & Calders 2011] Zemel et al. 2013) Feldman et al.) 2015 du Pin Calmon
al. 2017 Choi et al.] 2020 Jiang & Nachum 2020). which debias, reweight, generate training
data, (2) in- processsn approaches (Zafar et al. 2017a by Agarwal et al., 2018 Zhang et al.) 2018
Cotter al.
2019 Roh et al. [2020] 2021), which modily model training itself for fairness, and (3)
post-processing approaches (Kamiran et al. 2012; Hardt et al. 2016 Pleiss et al., 2017] Chzhen
al., 2019), which manipulate only the model outputs without changing the training inside. Among
the three categories, in in-processing approaches are widely used for unfaimeess mitigation, but most of them assume that the training and deployment data distributions are the same.
Another line of research is to support multiple fairness metrics (Thomas et al., 2019 Zhao et al. 2020p_ Thomas et al. (2019) proposes a fairness testing framework that can support multiple metrics. Zhao et al. (2020) shows that EO can be achieved while preserving the original DP. In comparison, we analyze when a model can achieve both e-DP and e-EO.
Beyond group fairness, there are other noteworthy fairness definitions including individual fair- ness (Dwork et al.] (012b),, which aims to give similar predictions to similar individuals, and causality-baase based fairness (Kilbertus et al.) [017;; Kusner et al.) 2017; Zhang & Bareinboim] 2018). which aims to improve fairness by understanding the causal relationship between attributes. Extending analysis and framework to these definitions is interesting future work.
C.2 FAIRNESS UNDER DATA DISTRIBUTION SHIFTS
Continuing from Sec. 6 we further compare work with the previous studies on data distribution shifts. The following paragraphs contain detailed discussions of the two categories of distribution shifts: general distribution shifts and thirness-speciffe shifts.
Among the general distribution shifts (i.e., covariate, label, and concept shifts in Table 2 the concept shift is the most relevant definition to the correlation shift. As shown in Table 2 of Sec. [ the correlation and concept shifts focus on Pr(zly) and Pr(y[x). respectively. As the concept shifts consider the distribution changes of the label (y) and input feature (x), this type of shift can implicitly
27

---

Under review as a conference paper at ICLR 2023
describe the correlation shifts when the input feature contains the group attribute (z). However, unique characteristic of the correlation shifts is to explicitly capture the bias changes between y and where Z is especially relevant to fair training. Thus, the notion of correlation shifts enables to analyze the behavior of fair training when the bias changes.
In addition, the key difference between the correlation shift and the other fairness-specifie shifts (subpopulation and demographic shifts) that the other shifts are defined under specific assumptions on the data distribution, which are not required in our correlation shift definition. Here are the assumptions in the subpopulation and demographic shifts:
The subpopulation shift (Maity et al., 2021) assumes that the loss expectations w.r.t. input feature X and label Y of training and deployment distributions are the same (i.e., Pizain ((((), Frest [((()) y) = z|. where E indicates the expectation). Thus, as discussed in the related work section, a major example of subpopulation shifts is when specific group has fewer positively- labeled examples during training time compared to deployment time, while the distributions the and y attributes remain the same.
Similarly, the demographic shift (Giguere et 2022) assumes that the joint probabilities of and on the training and deployment distributions are identical (i.e., Priain (x = y = y/z = 2) (x = z, y z/z 2)). This assumption is similar to the assumption of the subpopulation shift, but the difference is whether the loss values of the model are explicitly considered or not. The above assumptions are used the theoretical analyses of the previous works (Maity et al. 2021 Giguere et al. 2022). In comparison, are not limited by any additional assumptions, and we show that the achievable performances of fair training determined by the (y, 2)-correlation.
theoretical analyses in Sec. B of using correlation shifts
C.3 CONNECTION TO CAUSALITY- BASSDD FAIRNESS
Continuing from Sec. 6 we discuss how our work can be connected with causality- -based faimess. To this end, we give concrete example show the correlation shifts when the sensitive group attribute itself is confounder, one of the important roles of attributes in causality- based fairness.
We consider car insurance example, where the goal is to predict future accident rates from the past driving record: suppose that input feature x is the past driving record, label is the future accident rate, and sensitive attribute is the race.
Then, the structural causality graph between them would look like: z + (x, y) and x - y, i.e., z being a confounder (Figure IO).. The arrows from z and y could be due to (a) different socialization and driving behavior formation process that partially depends on racial identity and (b) another unobserved mediator such as where they live in, etc. In this example where is confounder, clear that z and are correlated.
We expect this confounding effect to change over time. This will make the sensitive attribute either stronger weaker confounding factor, which impacts the correlation between y and z.
Race
X
Past driving record
Future
accident rate
Figure 10: A causality graph for the insurance example in Sec.
C.3
28


Reviews:
[{"review_id": "o0QtgAtz7z", "reviewer": "Reviewer_gS56", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the challenges of in-processing fair algorithms with respect to accuracy and fairness in the setting of biased deployment data. A pre-processing step is proposed to mitigate this issue, along with an optimization approach.c", "summary_of_the_review": "I tend to marginally accept this paper given the importance of the problem and the said approach.", "strength_and_weaknesses": "Strengths: \nAn important problem is addressed here. It is valuable to understand how bias affects fairness properties. The paper is generally well-written.\n\nWeakness:\n\n1. The correlation between the sensitive attribute and the label could be more nuanced depending on the role of the sensitive attribute. For example, if the sensitive attribute is a confounder, a control variable, or even a mediator. In each case, the relation would affect the fairness properties differently. Adding some discussion around this would be helpful.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "R3kCDp_w18", "reviewer": "Reviewer_9fri", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the distributional shift problem in fair machine learning. The authors first propose a type of distributional shift called correlation shift, which studies the Pearson’s correlation coefficient between the label and the sensitive attribute. The connection between the correlation shift and the notorious accuracy-fairness tradeoff. Based on that, the authors propose a pre-processing strategy that adjusts the ratio of data to correct the shift. Experimental results demonstrate the efficacy of the proposed technique.", "summary_of_the_review": "The studied problem is important as it helps with the generalization of fair machine learning methods. However, I have concerns regarding its motivation and settings for analysis.", "strength_and_weaknesses": "Strengths:\n- Distributional shifts is indeed an important problem in machine learning.\n- The proposed technique is able to handle correlation shift in the given settings.\n\nConcerns/Questions:\n- When we consider distributional shifts, should we only consider the joint distribution of label y and sensitive attribute z? I am quite surprised that we can study it without any assumption on the distribution of input feature x. The model is essentially modeling $P(y|x, z, \\theta)$ (or $P(x,y,z|\\theta)$ depending on whether the model is discriminative or not). It seems likely that $P(y,z)$ and the marginal distributions are the same but the underlying input feature's distribution $P(x)$ is different, which might still cause bad performance on either accuracy/fairness.\n- Is there any relationship between the proposed correlation shift and the concept shift in many OOD literature? \n- I am not sure how to read the simulation results in section 3. Is each blue/red dot correspond to one classifier? How are the so-called synthetic classifier generated? And what are their architectures (logistic regression, SVM, MLP)?\n- As mentioned in several sections, there are existing works on other types of distributional shifts. I think some of them should be considered as important baseline methods. \n- There should be a dedicated paragraph/section to discuss the related work and how this work differs with them.\n- Minor comments: (1) fairness is a broad area which is not only group fairness, so it is better to specify that the work is on group fairness rather than just 'fairness'; (2) before using an abbreviation, its full name should be written out first to avoid confusion (e.g., AI -> artificial intelligence (AI)); (3) please carefully check typos or grammatical errors in the revised version.\n==== After rebuttal ====\nI think the concern on x's distribution remains, e.g., how should we define the 'no drastic change in the distribution of x' more formally? Would it be too strict to hold in real-world scenarios? Why (or why not)?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HzOOI_P5d37", "reviewer": "Reviewer_c8Rf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "While many techniques for model fairness have been proposed, the majority of them assume that the distributions of training and deployment data are identical, which is often not the case in practice. In particular, the bias between labels and sensitive groups changes, which may impair the performance of machine learning algorithms. To this end, this paper proposes a novel fair training framework that is divided into two parts: using a pre-processing approach to reflect the shifted correlation and using any existing in-processing algorithms for fair training on top of the improved data. Experimental results have validated the effectiveness of the proposed method under correlation shifts.\n", "summary_of_the_review": "This paper works on an timely problem, has a solid theoretical analysis and strong experimental results. I recommend acceptance of this paper.", "strength_and_weaknesses": "Advantages:\n- This article is well-written in general and addresses a timely, and seemingly overlooked, issue in fairness.\n- Solid theoretical and simulation results. \n- Strong mitigation performance.\n\nDrawbacks:\n- One minor concern is that the authors only consider scenarios in which we know the deployment data distributions. Will the proposed solution be useful despite our lack of knowledge about deployment data distributions?\n- Furthermore, it would be great if the authors could demonstrate the results using the fairness and accuracy trade-off curve rather than a single point on the curve. This could help to better demonstrate the effectiveness of the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "96VJN-pERO", "reviewer": "Reviewer_H3mp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper discusses the idea of enforcing fairness when training and testing/deployment distributions are not identical by making two contributions-- (i) the authors propose the notion of \"correlation shifts\" which analytically captures the aforementioned issue; (ii) the authors propose a pre-processing optimization approach that corrects for correlation shifts and existing in-processing approaches can then be used on the transformed data to ensure fairness. They conduct experiments on a number of baselines that show the efficacy of their approach and also propose dataset configurations with correlation shifts in testing.", "summary_of_the_review": "I recommend this paper for acceptance based on the novel ideas proposed, the extensive experimental evaluation, and theoretical analysis. To strengthen the work further, there are a few papers related to this work for which the authors could provide some more comparative details as part of the discussion section.\n\n__Edit__: After the discussion period with the AC and other reviewers, I have thought about some of the additional concerns raised. I am thus changing my score to borderline reject, but I do believe the work holds immense potential once it addresses these concerns. The main issue stems from the solution approach itself (I am summarizing from the AC/reviewer comments):\n- The authors assume that a correlation constant between the training and deployment data exists. Thus, problem is not being solved. Instead, the \"answer\" is already included in the approach to achieve fairness in the deployment phase. The challenge in addressing the data bias change is how to estimate the correlation change-- even if it is chosen to be provided as a range instead of a precise number. It is non-trivial to utilize approaches such as those of (Huang et al., 2006; Zhang et al., 2013) for estimating the correlation factor range as these are not designed to handle the shift between the label and the sensitivity attribute.\n- The experiments concerning unknown correlations, a large range of correlations, or the scenario when the knowledge of deployment data distribution is lacking, are conducted on a synthetic dataset, which might not coincide with real-world datasets.\n- After some more thought, I am inclined to agree with Reviewer 9fri's concerns regarding the input distribution, and the scenario when the input distribution itself changes drastically.", "strength_and_weaknesses": "- Strengths:\n1. The approach is a useful contribution to the field of fair ML since it is (a) well-motivated theoretically, (b) is a pre-processing step so advances in in-processing fair methods can still be utilized, and (c) the relaxed version of the pre-processing optimization problem is convex, ensuring \"optimality\" in approximating the true optimal.\n2. The authors conduct extensive experiments to evaluate their approach with many correlation-shifted versions of datasets (training-set as Adult dataset (1996) with test-set as ACS income (2021), for example) which I believe can also be used as benchmarks for future work in this field. I also appreciate the inclusion of empirical results when the correlation shift range $[\\alpha, \\beta]$ is not known exactly (Section B.9), or misspecified (Section 5.3). \n3. I believe the fact that the approach can work with multiple notions of fairness (such as Equalized Odds and Demographic Parity) is also a major strength.\n4. The simulation results and toy examples discussed throughout the paper increase readability and understanding quite a bit.\n\n- Weaknesses:\n1. While the authors mention many related works under \"Fairness-Specific Shifts\", I am not quite sure why some of these approaches cannot be applied to the correlation-shift problem covered in the paper. Surely, subpopulation shifts (Maity et al, 2021) seem to be a more specific and constrained version of within-group correlation shifts, is there a way to link these together theoretically? Similarly, what are the exact differences between demographic shifts (Giguere et al, 2022) and correlation shifts? It might be beneficial if the authors can provide more details explaining the above, and possible even some theoretical/empirical evidence on the same.\n2. Can the correlation shift study and pre-processing optimization approach studied in the paper have direct impact on the few existing works on poisoning attacks on fairness of classifiers [1-3]? As these attacks poison the training data, there is bound to be some correlation shift between the \"original\" test distribution (where the attacker would like the defender to fail) w.r.t the poisoned training distribution, and possible the pre-processing approach proposed can counteract the negative effects of these attacks. It would be interesting to see this discussed as well.\n\n\n- References:\n- [1] https://ojs.aaai.org/index.php/AAAI/article/view/17080\n- [2] https://link.springer.com/chapter/10.1007/978-3-030-67658-2_10\n- [3] https://link.springer.com/chapter/10.1007/978-3-031-00123-9_30", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject