Understanding and Mitigating Accuracy Disparity in Regression
Authors: Jianfeng Chi
Han Zhao
Geoff Gordon
Yuan Tian
Venue:  (2021)
Keywords: Algorithmic Fairness
Representation Learning

Abstract: With the widespread deployment of large-scale prediction systems in high-stakes domains, e.g., face recognition, criminal justice, etc., disparity on prediction accuracy between different demographic subgroups has called for fundamental understanding on the source of such disparity and algorithmic intervention to mitigate it. In this paper, we study the accuracy disparity problem in regression. To begin with, we first propose an error decomposition theorem, which decomposes the accuracy disparity into the distance between label populations and the distance between conditional representations, to help explain why such accuracy disparity appears in practice. Motivated by this error decomposition and the general idea of distribution alignment with statistical distances, we then propose an algorithm to reduce this disparity, and analyze its game-theoretic optima of the proposed objective function. We conduct experiments on four real-world datasets. The experimental results suggest that our proposed algorithms can effectively mitigate accuracy disparity while maintaining the predictive power of the regression models.

Under review as a conference paper at ICLR 2021
UNDERSTANDING AND MITIGATING ACCURACY DIS- PARITY IN REGRESSION
Anonymous authors Paper under double-blind review
ABSTRACT
With the widespread deployment of large-scale prediction systems in high-stakes domains, e.g., face recognition, criminal justice, etc., disparity on prediction accuracy between different demographic subgroups has called for fundamental understanding on the source of such disparity and algorithmic intervention to mitigate it. In this paper, we study the accuracy disparity problem in regression. To begin with, we first propose an error decomposition theorem, which decomposes the accuracy disparity into the distance between label populations and the distance between conditional representations, to help explain why such accuracy disparity appears in practice. Motivated by this error decomposition and the general idea of distribution alignment with statistical distances, we then propose an algorithm to reduce this disparity, and analyze its game-theoretic optima of the proposed objective function. We conduct experiments on four real-world datasets. The experimental results suggest that our proposed algorithms can effectively mitigate accuracy disparity while maintaining the predictive power of the regression models.
INTRODUCTION
Recent progress in machine learning has led to its widespread use in many high-stakes domains, such as criminal justice, healthcare, student loan approval, and hiring. Meanwhile, has also been widely observed that accuracy disparity could occur inadvertently under various scenarios in practice (Barocas & Selbst 2016). For example, errors are inclined to occur for individuals of certain underrepresented demographic groups (Kim) 2016). In other cases, [Buolamwini & Gebru (2018)
showed that notable accuracy disparity gaps exist across different racial and gender demographic
subgroups on several real-world image classification systems. Moreover, [Bagdasaryan et al. (2019)
found out that a differentially private model even enlarges such accuracy disparity gaps. Such accuracy disparity gaps across demographic subgroups not only raise concerns in high-stake applications but also can be utilized by malicious parties causing information leakage (Yaghini et al.) (019)).
Despite the ample needs of accuracy parity, most prior work limits its scope to studying the problem in binary classification settings (Hardt et al. 22166; Zafar et al. 2017b) Zhao et al.) [2099: Jiang et al. 2019). In a seminal work, Chen et al. 2018) analyzed the impact of data collection on accuracy disparity in general learning models. They provided a descriptive analysis of such parity gaps and advocated for collecting more training examples and introducing more predictive variables. While such a suggestion is feasible in applications where data collection and labeling is cheap, it is not applicable domains where is time-consuming, expensive, or even infeasible to collect more data, e.g., in autonomous driving, education, etc.
Our Contributions In this paper, we provide a prescriptive analysis of accuracy disparity and aim at providing algorithmic interventions to reduce the disparity gap between different demographic subgroups in the regression setting. To start with, we first formally characterize why accuracy disparity appears in regression problems by depicting the feasible region of the underlying group-wise errors. We also provide a lower bound on the joint error and a complementary upper bound on the error gap across groups. Based on these results, we illustrate why regression models aiming to minimize the global loss will inevitably lead to accuracy disparity if the input distributions or decision functions differ across groups (see Figure la).
We further propose an error decomposition theorem that decomposes the accuracy disparity into the distance between the label populations and the distance between conditional representations. To mitigate such disparities, we propose two algorithms to reduce accuracy disparity via joint distribution alignment with total variation distance and Wasserstein distance, respectively. Furthermore,

---

Under review as a conference paper at ICLR 2021
Erri,
Ery,
Erre,
External Information
2Ema Erro 0
Err, Reip
Transformed Representation
Input Data
&
A
Alkce
Bob
Err 2Emp
Erro.
X
Z
Ero,
Errs
(a) Geometric interpretation of accuracy disparity.
(b) Game-theoretic illustration of our algorithms.
Figure I: The left figure illustrates how accuracy disparity arises by minimizing the global squared loss. The right figure gives schematic illustration of the proposed algorithmic framework.
analyze the game-theoreen optima of the objective function and illustrate the principle of our algorithms from a game- came-theoretic perspective (see Figure [b). To corroborate the effectiveness of our proposed algorithms in reducing accuracy disparity, we conduct experiments on four real- world datasets. Experimental results suggest that our proposed algorithms help to mitigate accuracy disparity while maintaining the predictive power of the regression models. We believe our theoretical results contribute to the understanding of why accuracy disparity occurs in machine learning models, and the proposed algorithms provides an alternative for intervention real-world scenarios where accuracy parity is desired but collecting more data/features is time-consuming or infeasible.
2 PRELIMINARIES
Notation We use X' C Rd and y S R to denote the input and output space. We use and Y to denote random variables which take values in X' and V respectively. Lower case letters x and y denote the instantiation of X and Y. We use (X) to denote the Shannon entropy of random variable X, (X Y) to denote the conditional entropy of X given Y, and I(X; Y) denote the mutual information between X and Y. To simplify the presentation, we use A E {0, 1} as the sensitive other words,
attribute, e.g., gender, race, etc. Let H. be the hypothesis class of regression models. for h E H. h X y is predictor. Note that even if the predictor does not explicitly take the sensitive attribute as an input variable, the prediction can still be biased due the correlations with other input variables. In this work we study the stochastic setting where there is joint distribution D over X,Y Y and from which the data are sampled. For a E {0, 1} and y E R, we use D. to denote the conditional distribution of D given A and D to denote the conditional distribution of given Y = y. For an event E, DEE) denotes the probability of E under D. Given a feature transformation function g X + Z that maps instances from the input space X' to feature space 2, we define 93 D == D o g- to be the induced (pushforward) distribution of D under g, i.e., for any event F' 2, 9; D F') variable remains unchanged if it is positive or otherwise 0, i.e. (Y)+ equals to Y if the value of Y is positive or otherwise O.
D((= X g(xx) F')). We use (+) to indicate the value of
Given a joint distribution D, the error of a predictor h under is defined as Errp(h) := EDI(y 2( To make the notation more compact, we may drop the subscript D when it is clear from the context. Furthermore, we also use MSEp Y Y) to denote the mean squared loss between the predicted variable ÿ = h(X)) and the true label over the joint distribution D. Similarly, we also use CED(A Á) denote the cross-entropy loss between the predicted variable Ā and the true label A over the joint distribution D. Throughout the paper, we make the following standard assumption in regression problems:
Assumption 2.1. There exists M 0. such that for any hypothesis H. 3 h X + V. 11/]lo0 S M
and | < M.
Problem Setup We study the fair regression problem: the goal is to learn regressor that is fair in the sense that the errors of the regressor are approximately equal across the groups given by the sensitive attribute A. We assume that the sensitive attribute A is only available the learner during
2

---

Under review as a conference paper at ICLR 2021
the training phase and is not visible during the inference phase. We would like to point out that there are many other different and important definitions fairness Narayanan 2018 even in the sub-category of group fairmess, and our discussion is by means comprehensive. For example, two frequently used definitions of fairness in the literature are the so-called statistical parity (Dwork 2012) and equalized odds (Hardt al., 20T6). Nevertheless, throughout this paper we mainly focus accuracy parity as our fairness notion, due to the fact that machine learning systems have been shown exhibit substantial accuracy disparities between different demographic subgroups (Barocas & Selbst 2016 Kim 2016 Buolamwini & Gebru 2018). This observation has already brought huge public attention (e.g., see New York Times, The Verge, and Insurance Journal) and calls for machine learning systems that (at least approximately) satisfy accuracy parity. Formally, accuracy parity is defined as follows:
al.
Definition 2.1 (Accuracy Parity). Given joint distribution D, a predictor h satisfies accuracy parity EITD, (h) Errp, (h).
The violation of accuracy parity also known as disparate mistreatment (Zafar et al., 2017a). In practice the exact equality of on accuracy between two groups is often hard to ensure, we define error gap to measure how well the model satisfies accuracy parity:
Definition 2.2 (Error Gap). Given a joint distribution D, the error gap of hypothesis h is A (h)
[REr, (h) - Erro, (h))-
By definition, if model satisfies accuracy parity, A (h) will be zero. Next we introduce two distance metrics that will be used in theoretical analysis and algorithm design:
Total variation distance: measures the largest possible difference between the probabilities that the two probability distributions can assign to the same event E. We use dIv Q) to denote the total variation:
dP Q) sup (P(F) - 2(F))-
Wasserstein distance: the Wasserstein distance between two probability distributions is
1 fap sap- fdo
WiP,
sup DE(t-fIlES
where ||f is the Lipschitz semi-norm of a real-valued function of f and Ω is the sample space over which two probability distributions P and Q are defined. By the Mistorovicc-Rubinstern duality theorem (Villani 2008), we recover the primal form of the Wasserstein distance, defined as
TEI(P.O) / (
W1PP, )) := inf
where I(P, Q) denotes the collection of all couplings of P and Q, and X and Y denote the random variables with law P and Q respectively. Note that we use Lı distance for d(.,-) throughout the paper, but the extensions other distance, e.g,, Le distance, is straightforward.
MAIN RESULTS
In this section, we first characterize why accuracy disparity arises in regression models. More specifically, given a hypothesis h E H. we first describe the feasible region of ErrD, and ErrD, by proving a lower bound of joint errors and an upper bound of the error gap. Then, we give geometric interpretation to visualize the feasible region of ETTD, and Errp, and illustrate how error gap arises when learning a hypothesis that minimizes the global squared error. We further analyze the accuracy disparity by decomposing it into the distance between label populations and the distance between conditional representations. Motivated by the decomposition, we propose two algorithms reduce accuracy disparity, connect the game-theoretic optima of the objective functions in our algorithms with our theorems, and describe the practical implementations of the algorithms. Due to the space limit, we defer all the detailed proofs to the appendix.
3.1 BOUNDS ON CONDITIONAL ERRORS AND ACCURACY DISPARITY GAP
When we learn a predictor, the prediction function induces A .. where Ý is the predicted target variable given by hypothesis h. Hence for any distribution Do (Di) of X, the predictor also induces
3

---

Under review as a conference paper at ICLR 2021
distribution hyDu (h;D1) of Y. Recall that the Wasserstein distance is metric, hence the following chain of triangle inqqualities holds:
Wi Wi(Do (())) & (1o)) +
(())))))))))))+)
Intuitively, Wi (Do(Y), hy and Wi ( measure the distance between the true label distribution and the predicted one on = 0/1 cases, respectively. This distance is related to the prediction error of function h conditioned on A =
Lemma 3.1. Let y h(X) R, then for E [0, (..1).Wi(D.u))) (P.,))))) S v (h).
With the above results, can get the following theorem that characterizes the lower bound of joint error on different groups:
Theorem 3.1. Let y = h(X) E I. we have ErrD, (h) + Errp, (h) 2 ( (V1()(())) -
Wi (h,Do, h2 h2D
that if the difference between the label distribution across groups is large, then statistical disparity could potentially lead to a large joint error. Moreover, Theorem 3.1 3.1 could be extended to give lower bound on the joint error incurred by h well:
In Theorem 3.1
Corollary 3.1. Let V h(X) E R and D(A 0) E [0, 1], we have Err min(a, a)- (P)) - Wi (1y D:)
Next, we upper bound the error gap to gain more insights on accuracy disparity. For E {0, 1]. define the conditional variance Varp, [Y| EED.((= E. [ and it shows up as the irreducible error of predicting Y when we only use the knowledge of X. We also know that the optimal decision function conditioned on A under mean squared error to be ED, [Y]X]. The following theorem characterizes the upper bound of the enror gap between two groups:
Theorem 3.2. For any hypothesis 7 3 h X + D, if the Assumption 2.1 holds, then:
A Eer (h) 8M2 dTo (X)) + IDD [arro [Var [X] - ED, |arD, [Y]]] + 4M min [ED (Y - Ez, (Y]X)I), Ed, [[E,, (Y|X) ED, (Y[X)I)}
upper bounds the error gap across groups by three terms: the first term corresponds to the distance of input distribution across groups, the second term is the noise (variance) difference, and third term is the discrepancy of optimal decision functions across different groups. In an ideal and fair setting, where both distributions noiseless, and the optimal decision functions are insensitive to the group membership, then Theorem (ineorem3.2jimmg implies a sufficient condition to guarantee accuracy parity is to find group-invariant representation that minimize drv (Do(D) ), Di (X)).
Remark Theorem 3.2
Geometric Interpretation By Theorem 3.T and Theorem in Figure Ia we visually illustrate how accuracy disparity arises given data distribution and the learned hypothesis that aims to minimize the global squared error. In Figure la given the hypothesis class H. we use the line Errp, + Erry, = B to denote the lower bound in Theorem 3.1 and the two lines |rtt,, Errp, A denote the upper bound in Theorem 3.2 These three lines form feasible region (the green area) of Errp, and Errp, under the hypothesis class H. For any optimal hypothesis which is solely designed to minimize the overall eиror, the best the hypothesis h can do is to intersect with one of the two bottom vertices. For example, the hypotheses (the red dotted line and the blue dotted line) trying minimize overall error intersect with the two vertices of the region to achieve the smallest ErrD, -intercept (Errp, -intercept), due to the imbalance between these two groups. However, since these two vertices not on the diagonal of the feasible region, there is no guarantee that the hypothesis can satisfy accuracy parity (Errр, = Errp,), ), unless we can shrink the width of green area to zero.
Conditional Distribution Alignment Reduces Accuracy Parity In Theorem 3.2 we illustrate how accuracy disparity arises in regression models due to noise, distance between representations, and distance between decision functions. However, it is nearly impossible to collect noiseless data with group-invariant input distribution. Moreover, there is no guarantee that the upper bound will be lower if we learn the group-invariant representation that minimizes (Po(()))).))))) Di (X)) alone, since the learned representation could potentially increase the variance. In this regard, we prove a novel upper bound which is free from the above noise term motivate aligning conditional distributions to mitigate the error disparity across groups. To do so, we relate the error gap to the label distribution and the predicted distribution condition Y y
4

---

Under review as a conference paper at ICLR 2021
Theorem 3.3. If Assumption 2.1 holds, then for vh H. let Y h(X), the following inequality holds:
E (h) - dDV(()
+ 3M min(Ep, [ - ED; [],, Ep, D [P] Epy(P]II
Remark We see that the error gap is upper bounded by two terms: the distance between label distributions and the discrepancy between conditional predicted distributions across groups. Note that this is different from the decomposition we have in Theorem [.2 where the marginal distribution on X instead of Y. Given a dataset, the distance of label distributions a constant since the label distribution is fixed. For the second term, if we can minimize the discrepancy of the conditional predicted distribution across groups, we then have model that is free of accuracy disparity when the label distribution is well aligned.
3.2 ALGORITHM DESIGN
Inspired by Theorem 3 we can mitigate the error gap if we align the group distributions via minimizing the distance of the conditional distributions across groups. However, it is intractable to do so explicitly in regression problems since can take infinite values on IR Next we will present two algorithms to approximately solve the problem through adversarial representation learning.
Given a Markov chain X _ Z " Y, we are interested in learning group-invariant conditional representations so that the discrepancy between the induced conditional distributions Dj (Z g(X) and Dj (Z = (X)) is minimized. In this case, the second term of the upper bound in Theorem 3.3 is minimized. However, it in general not feasible since Y is a continuous random variable. Instead, propose to learn the representations of Z to minimize the discrepancy between the joint distributions Do = g(X), Y) and Dj (Z- = gXX), Y). Next, we will show the distances between conditional predicted distributions D (Z = (()) and D (Z- g(X)) are minimized when we minimize the
joint distributions Do (X), Y) and ( = gX). ) Y) in Theorem 3.4 and Theorem 3.5
To proceed, we first consider using the total variation distance to measure the distance between two distributions. In particular, we can choose to learn a binary discriminator f Z x Y - Å that achieves minimum binary classification error on discriminating between points sampled from two distributions. In practice, we use the cross-entropy loss as a convex surrogate loss. Formally, we are going to consider the following minimax game between g and f:
min max fEF 9
(
Next we show that for the above equation, the optimal feature transformation g corresponds to the one that induces invariant conditional feature distributions.
Theorem 3.4. Consider the minimax game in (T). The equilibrium (g", (²) ) of the game is attained
when 1). Z (X) is independent of A conditioned Y; 2). (Z, = D(A 1 Y Z).
Since the equilibrium the game Z is independent of A conditioned on Y, the optimal f" could also be equivalently written as f" = D(A = Y), i.e., the only useful information for the discriminator in the equilibrium is through the external information Y. In Theorem 3.4] the minimum cross-entropy loss that the discriminator (the equilibrium of the game) can achieve is ((A Z, Y) (see Proposition A. in Appendix A). By the basic property of conditional entropy, we have:
min CED(A II ((x(X), [ JEF
= (AA Z,Y) = H(A T(A; Y).
We know that H(A) Y) a constant given the data distribution. The maximization of g in (T] is equivalent to the minimization of minz=g(X) ((4; Z Y), and it follows that the optimal strategy for the transformation is the one that induces conditionally invariant features, e.g., ((A; Z Y) 0. Formally, we arrive the following minimax problem:
min max h.g JEF
MSEx(b()(X)), Y) X CEo(A (()(),))
In the above formulation, the first term corresponds to the minimization of prediction loss of the target task and the second term is the loss incurred by the adversary f. As whole, the minimax
5

---

Under review as a conference paper at ICLR 2021
optimization problem expresses trade-off (controlled by the hyper-parameee À 0) between accuracy and accuracy disparity through the representation learning function g.
Wasserstein Variant Similarly, if we choose to align joint distributions via minimizing Wasstertein distance, the following theorem holds. Theorem 3.5. Let 9² == arg ming Wi X) Pig(XX)) then ( = 9"( =
(Z = g" (X)) almost surely.
One notable advantage of using the Wasserstein distance instead of the TV distance is that, the Wasserstein distance is a continuous functional of both the feature map ! as well as the discriminator (Arjovsky al. 2017). Furthermore, if both g and are continuous functions of their corresponding model parameters, (which is the case for models we are going to use in experiments), the objective function will be continuous in both model parameters. This property of the Wasserstein distance makes it more favorable from an optimization perspective. Using the dual formulation, equivalently, we can learn a Lipschitz function f Z x Y + IR as a witness function:
min h. Zo mnn Z1 DL f1
()Ex()()(X)), Y) + (f( Y) +)
f(Z,, Y)
Interpretation To make our algorithms casier to follow, we provide a game- theoretic interpretation of our algorithms in Figure 1b Consider Alice (encoder) and Bob (discrimi- nator) participate a two-player game: upon receiving a set of inputs X, Alice applies a transformation to the inputs to generate the corresponding features Z and then send them to Bob. Besides the features sent by Alice, Bob also has access to the external information Y, which corresponds to the corresponding labels for the set of features sent by Alice. Once having both the features Z and the corresponding labels Y from external resources, Bob's goal is to guess the group membership A of each feature sent by Alice, and to maximize his correctness as much as possible. On the other hand, Alice's s goal is to compete with Bob, i.e., to find transformation to confuse Bob as much she can. Different from the traditional game without external information, here due to the external information Y Bob has access to, Alice cannot hope to fully fool Bob, since Bob can gain some insights about the group membership of features from the external label information. Nevertheless, Theorem 3.4 and Theorem 3.5 both state that when Bob uses binary discriminator or Wasstertein discriminator to learn A, the best Alice could do is to to learn a transformation so that the transformed representation Z is insensitive to the values of A conditioned any values of Y.
4 EXPERIMENTS
Inspired by our theoretical results that decompose accuracy disparity into the distance between label populations and the distance between conditional representations, we propose two algorithms to mitigate it. In this section, conduct experiments to evaluate the effectiveness of our proposed algorithms in reducing the accuracy disparity.
4.1 EXPERIMENTAL SETUP
Datasets We conduct experiments on four real-world benchmark datasets: the Adult dataset (Dua &
Graff 2017). COMPAS dataset (Dieterich et al. (2016). Law School dataset (Wightman & Ramsey 1998). and Communities and Crime dataset (Dua & Grafil 2017). All datasets contain binary
sensitive attributes (e.g., male/female, white/non-white) We refer readers to Appendix B for detailed descriptions of the datasets and the data pre-processing pipelines.
Methods We term the proposed algorithms CENET and WASSERSTEINNET for our two proposed algorithms respectively. For each dataset, we perform controlled experiments by fixing the regression neural network architecture to be the same. We train the regression nets via mean squared loss. Note that although the Adult dataset and COMPAS dataset are for binary classification tasks, we can still take them as regression tasks with two distinctive ordinal values. To the best of our knowledge, no previous study aims to minimize accuracy disparity in regression using representation learning. However, there are other similar fairness notions and mitigation techniques proposed for regression and we add them as our baselines: (1) Bounded group loss (BGL) (Agarwal et al. 2019), which asks for the prediction errors for any groups to remain below a pre-defined level e; (2) Coefficient of determination (CoD) (Komiyama et al., 2018), which asks for the coefficient of determination between the sensitive attributes and the predictions to remain below a dre-defined level e.

---

Under review as a conference paper at ICLR 2021
X
Wasserstenneet BOL
Wasserseentee
0.16
BOL
0.40
0.15
X
0.14
0.36
0.13
0.12
0.32
0.11
0.10
0.065
0.060
0.085
0,070
0.075
0.082
0,0000
0.0025
0.0050
0.0075
00100
0,0125
c.01
AEr
(a) Adult
(b) COMPAS
CENet
CENet
WassersteiiNtt
0.35
WasterttahNol
0.5
BGL
ColD
0.30
CoD
0.25
0.3
0.15
0.10
0.2
0.05
0.00
0.015
0.020
0.090
0.035
0.000
0.61
0.02
0.04
Agr
AE
(c) Crime
(d) Law
Figure 2: Overall results: R² regression scores and error gaps in different datasets. Our goal is to achieve high R2 scores with small error gap values (i.e., the points located in the upper-left corner).
Among all methods, we vary the trade-off parameter (i.e., À in CENET and WASSERSTEINNEY and in BGL and CoD) and report and the corresponding n2 scores and the error gap values. For each experiment, average the results for ten random seeds. We refer readers to Appendix for detailed parameter and hyper-paramend
settings in our experiments. We also defer the additional experimental results and analyses on how the trade-off parameters À and affects the performance of different algorithms to Appendix
4.2 RESULTS AND ANALYSES
The overall results are visualized in Figure 2 The following summarizes our observations and analyses: (1) Overall, trade-offs exist between the predictive power of the regressors and accuracy parity: for each method we test, the general trend is that with the decrease of the values of error gaps, the values of R² also decrease. The exception is CENET in the Adult dataset and Crime dataset since training CENET is unstable when is large and we will provide more details in Appendix C (2) Our proposed methods WASSERSTEINNET and CENET are effective in reducing the error gaps while keeping the H²2 scores relatively high in the Adult, COMPAS and Crime dataset. In the Law dataset, the error gaps decrease with high utility losses in our proposed methods; (3) Among our
!CoD cannot he implemented on the Adult dataset since the size of the Adult dataset is large and the QCQP optimization algorithm to solve CoD needs quadratic memory usage of the dataset size.
7

---

Under review as a conference paper at ICLR 2021
proposed methods, WASSERSTEIN achieves better accuracy and accuracy disparity trade-offs while CENET suffers significant accuracy loss and may fail to decrease the error gaps in the Adult and Crime dataset. The reason behind it is that the minimax optimization in the training of CENET could lead to an unstable training process under the presence of noisy approximation to the optimal discriminator (Arjovsky & Bottou 2017; Arjovsky et al., 2017); (4) Compared our proposed methods, BGL and CoD can also decrease error gaps to certain extent. This is because: (i) BGL aims keep errors remaining relatively low in each group, which helps reduce accuracy disparity; (ii) CoD aims to reduce the correlation between the sensitive attributes and the predictions (or the inputs) in the feature space, which might somehow reduce the dependency between the distributions of these two variables. In comparison, our proposed methods do better in mitigating the error gaps.
RELATED WORK
Algorithmic Fairness In the literature, two main notions of fairness, i.e., group fairness and individual fairness, has been widely studied (Dwork et al. [2012] Zemel et al. 2013, Feldman et al., 2015, Hardt et al. 2016 Zafar ct al. 2017b, Madras et al. 2019 Khani & Liang 2019). In particular, Chen et al. (2018) analyzed the impact of data collection on discrimination (e.g.. false positive rate, false negative rate, and zero-one loss) from the perspectives of bias- variance- decomposition, and they suggested collecting more training examples and collect additional variable to reduce discrimination. In comparison, our work precisely characterizes the disparate predictive accuracy in terms of the distance between label populations and the distance between conditional representation and propose algorithms to reduce accuracy disparity across groups in regression.
Fair Regression A series of work focus faimess under the regression problems (Calders et al.
2013; Johnson et al., al.]|2016 [2016; Berk et al. 22188) Komiyama et al. [2018] Chzhen et al. 2020: Bigot
2020 Zink & Rose] (2200) Mary al.] 2019 Narasimhan et al. 2020). To the best of our knowledge no previous study aimed to minimize accuracy disparity in regression from representation learning.
However, there are other similar fairness notions proposed for regression: [Aaawal al. (2019) proposed fair regression with bounded group loss (i.e., it asks that the prediction error for any protected group remain below some pre-defined level) and used exponentiated aneeniateddgradient approach to satisfy BGL:IKomiyana sensitive attributes between the predictions to some pre-defined level and used off the--f convex optimizer to solve the problem. In contrast, we source out the root of accuracy disparity through the lens of information theory and reducing via distributional alignment a minimax game.
et al. (2018) aimed to reduce the coefficient of determination between the
Fair Representation A line of work focus on building algorithmic fair decision making systems using adversarial techniques to learn fair representations (Edwards & Storkey| 2015 Beutel et al.. 2017,Adel et al.) 2019 Zhao et al. 2019). The main idea behind is to learn a good representation of the data so that the data owner can maximize the accuracy while removing the information related to the sensitive attribute. Madras et al. (2018) proposed generalized framework to learn adversarially fair and transferable representations and suggests using the label information in the adversary to learn equalized odds equal opportunity representations in the classification setting. Apart from adversarial representation, recent work also proposed to use distance metrics, e.g., the maximum mean discrepancy (Louizos et al.) 2013) and the Wasserstein distance (Jiang al. 2019) to remove group-related information. Compared to their work, we propose to align (conditional) distributions across groups reduce accuracy disparity using minimax optimization and analyze the game-theoretic optima in the minimax game in the regression setting.
CONCLUSION
In this paper, we theoretically and empirically study accuracy disparity in regression problems. Specifically, we prove an information -thoorettt lower bound on the joint error and a complementary upper bound on the error gap across groups to depict the feasible region of group-wise errors. Our theoretical results indicate that accuracy disparity occurs inevitably due to the label distributions differ across groups. To reduce such disparity, we further propose to achieve accuracy parity by learning conditional group-invariant representations using statistical distances. The game-theoretic optima of the objective functions in our proposed methods are achieved when the accuracy disparity is minimized. Our empirical results on four real-world datasets demonstrate that our proposed algorithms help to reduce accuracy disparity effectively. We believe our results take an important step towards better understanding accuracy disparity in machine learning models.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Tameem Adel, Isabel Valera, Zoubin Ghahramani, and Adrian Weller One-network adversarial fairness. In Thirty-Third AAAI Conference on Artificial Intelligence, 2019.
Alekh Agarwal, Miroslav Dudik, and Zhiwei Steven Wu. Fair regression: Quantitative definitions and reduction-based algorithms. In International Conference on Machine Learning, pp. 120-129, 2019.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arxiv e-prints, art. arXiv preprint arXiv: 1rX.::1701.4882,,, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 201-2
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate impact on model accuracy. In Advances in Neural Information Processing Systems, pp. 15453- 15462, 2019.
Solon Barocas and Andrew Selbst. Big data's disparate impact. Calif. Rev. 104:671, 2016.
Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. Faimess in criminal justice risk assessments: The state of the art. Sociological Methods Research, pp. 0049124118782553 2018.
Alex Beutel, Jilin Chen. Zhe Zhao, and Ed H Chi. Data decisions and theoretical implications when adversarially learning fair representations. arXiv preprint vrXiv:177.000075 2017.
Jérémie Bigot. Statistical data analysis in the wasserstein space. ESAIM: Proceedings and Surveys, 68:1-19, 2020.
Sarah Bird, Miro Dudík, Richard Edgar, Brandon Hom,, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathleen Walker. Fairlearn: A toolkit for as- sessing and improving fairness in AI. Technical Report MSR-TR-2020-32, Microsoft, May 2020.
cationn on/ for assessssssing--ae and -anggngg-aa-n-mmooooooo-----aeee
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities commercial gender classification. In Conference fairness, accountability and transparency, pp- 77-91, 2018.
Toon Calders, Asim Karim, Faisal Kamiran, Wasif Ali, and Xiangliang Zhang. Controlling attribute effect in linear regression. In 2013 IEEE 13th international conference data mining, pp. 71-80. IEEE, 2013.
Irene Chen, Fredrik D Johansson, and David Sontag. Why is my classifier discriminatory? In Advances in Neural Information Processing Systems, pp. 3539-3550, 2018
Exgenii Chzhen, Christophe Denis, Mohamed Hebiri, Luca Oneto, and Massimiliano Pontil. Fair regression with wasserstein barycenters. arXiv preprint arXiv:2006.07286, 2020.
Constantinos Daskalakis and Ioannis Panageas. The limit points of (optimistic) gradient descent in min-max optimization. In Advances in Neural Information Processing Systems, pp. 9236-9246.
2018.
William Dieterich, Christina Mendoza, and Tim Brennan. Compas risk scales: Demonstrating accuracy equity and predictive parity. Northpointe Inc, 2016.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL htth:/ttttthhhhiaeeeee
ics. uci. eddu/ml
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Faimess through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pp. 214-226, 2.12.

---

Under review as a conference paper at ICLR 2021
Harrison Edwards and Amos Storkey. Censoring representations with adversary. arXiv preprint arXiv:1511.0589, 2015.
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubra- manian. Certifying and removing disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 259-268. ACM, 2015.
Yaroslav Ganin, Bygeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adverssriil training of neural networks. The Journal of Machine Learning Research, 17(1):2096-0300, 2016.
Alison L Gibbs and Francis Edward Su. On choosing and bounding probability metrics. International statistical review, 70(3):419-4355 2002.
Moritz Hardt, Eric Price, Nati Srebro, al. Equality of opportunity in supervised learning. In Advances in neural information processing systems, pp. 3315-3323, 2016.
Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Jiang, and Silvia Chiappa. Wasserstein fair classification. arXiv preprint urXiv:I907. 12059, 2019.
Kory D Johnson, Dean Foster, and Robert A Stine. Impartial predictive modeling: Ensuring fairness in arbitrary models. arXiv preprint 160. 2016.
Fereshte Khani and Percy Liang. Noise induces loss discrepancy across groups for linear regression. arXiv preprint /01i::/911.0000 2019.
Pauline T Kim. Data-driven discrimination at work. Wm. & Mary L. Rev, 58:857, 2016.
Junpei Komiyama, Akiko Takeda, Junya Honda, and Hajime Shimao. Nonconvex optimization for regression with fairness constraints. In International conference on machine learning, pp. 2737-2746, 2018.
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The variational fair autoencoder. arXiv preprint arXiv: 1..00003, 2015.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. arXiv preprint arXiv: /802 002.06309, 2018.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Faimess through causal awareness: Learning causal latent-variable models for biased data. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 349-358, 2019.
Jérémic Mary, Clément Calauzènes, and Noureddine El Karoui. Faimess-aware learning for continu- ous attributes and treatments. In International Conference on Machine Learning, pp. 4382-4391, 2019.
Harikrishna Narasimhan, Andrew Cotter, Maya R Gupta, and Serena Wang. Pairwise fairness for 2020.
ranking and regression. In AAAI, pp.
Arvind Narayanan. Translation tutorial: 21 faimness definitions and their politics. In Proc. Conf. Fairness Accountability Transp., New York, USA, 2018.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Linda Wightman and Henry Ramsey. LSAC national longitudinal bar passage study. Law School Admission Council, 1998.
Mohammad Yaghini, Bogdan Kulynych, and Canmela Troncoso. Disparate vulnerability: On the unfaimess of privacy attacks against machine learning. arXiv preprint arXiv: 00389, 2019.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna Gummadi. Faireess beyond disparate treatment & disparate impact: Learning classification without disparate mistreat- ment. In Proceedings of the 26th International Conference on World Wide Web, pp. 1171-1180. International World Wide Web Conferences Steering Committee, 2017a.
10

---

Under review as a conference paper at ICLR 2021
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Faireess constraints: Mechanisms for fair classification. In Artificial Intelligence and Statistics, pp. 962-970, 2017b.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International Conference on Machine Learning, pp. 325-333, 2013.
Han Zhao, Amanda Coston, Tameem Adel, and Geoffrey J Gordon. Conditional learning of fair representations. arXiv preprint arXiv:1910.07163. 2019.
Anna Zink and Sherri Rose. Fair regression for health care spending, Biometrics, 76(3)9773-982. 2020.

---

Under review as a conference paper at ICLR 2021
APPENDIX
the appendix, we give proof's the theorems and claims our paper, the experimental details and more experimental results.
A MISSING PROOFS
Lemma 3.1. Let = h(X) E R, then for a E {0, Wi (i(,,(Y), S s VETDD. (h).
Proof. The prediction error conditioned on {0, 1) is
Errp, (h) B ((x) 9 = a|
1 h(X)IIA = a)
2 ((D.((),
inf
B[ h(x)))*
(D)))
Taking square root at both sides then completes the proof.
Theorem 3.1. Let y h(X) I, have ErrD, (h) ErrD, (h) (W((((())))
Wi(h:Do,h,Di)
Proof. Since W1 is distance metric, the result follows immediately the triangle inequality and Lemma 3.1
(M) & VErrD, (h) + W (4, + ErTD, (h).
Rearrange the equation above and by AM-GM inequality, have
((D)(()) (hyhPo hyD1) s / EErrD. (h) + v EErD, (h) S 2(ErrD. (h) ErrD, (h)). Taking square at both sides then completes the proof.
Corollary 3.1. Let γ = h(X) E R and D(A 0) E (0, 1], we have Errp(h) min(a, ((,
a} (P)) Proof.
The joint error is
(rrp(h)
a Errp, (h) (1 a) ErrD, (h)
min(a, 1 a) (Errn, (h) Errp, (h))
- a 1 (; 2
(Theorem 3.1
Lemma A.1. Assumption 2.1 holds, then the following inequality holds: [Ep,1(h(X))
Ep, [T] - Ep, [[h(X) - Ep, | < 8M (Do(X), (x))).
Proof. First, we know that |I(X) - ED, YY |X]] < 2M, Va (0, 1). since ||A|io s M and
[Y| S M. Now it suffices to bound:
[ED.((h(X) - ED, [r x (h,1(((()) ED, (r|]))
= =((((X) - Ez, |Y 11) dDo -dDi)l
< [I(X)- -ED,, [Y [X]] || IDDoo dDilli
(Hölder's inequality) (Assumption 2.1
4AM² ||dD. dD 1
8M² (Dv (X)).
Note that the last equation follows the definition of total variation distance.
12

---

Under review as a conference paper at ICLR 2021
Lemma A.2. If Assumption 2.1 holds, then the following inequality holds: [Ex,((((X) Ep. [P][) - Ep, [h(X) - Ep, X I 4M Ep. EE,, Ep, [Y]]]]
Proof.
[h(x)((() ED. [x]]] ED. E.. (h(() - ED, |X|I
-ED,,h? 2N(X)),, [V|]] R, - (²) 2 [X E2 [XI]I
2M Ep, [15D, [X|X] [r|]|| + 2M ED, 11b., [[a]]/EEE.__II]. [X] EDD |Y X]]]
(Assumption 2.
4M Ep, ED,, Y Ep,
Theorem 3.2. For any hypothesis H/ 9 h: X - if the Assumption 2.1 holds, then: A (h) < 8M2 drv (Do()) D1 (X)) + ED, Varr, [Y|]] - Ed, Varp, [X]]]
+ 4M (in ED, (Y|) Ez, (Y]]) ED, [l RD, (Y]X)]]}.
Proof. First, we show that for {0, 13.
ErrD, (h)
Ev. (() -
ED,,(h(X) ED, [Y Eo, [XX] Y)
ED.1(h(X) ED. [Y Ep. [( ED. (
-2ED, (()()-()X)----)))))))) -1, Y X (x|xT]
Ep,1(1(X) -ED, [Y|])) IYY ED. (M
Note that the last equation holds since
(b,(((( ED. DDD (X|X)) [x((1]
ED.(x) (x) - ED. (YIA))Y
ED. (XXIIXI
((xx - IE,, (x)) -E
((h(x) - ED. (B))) [X]]] -ED,(Y|X)) 0.
Next we bound the error gap:
EITDD (h) - Errp, (h)l
= -IDo [(((X) [Y x - RD, [(((X) ED, (
+ ED,I(Y Ex,, [Y)])) - ED, 1(Y - ED, (X 1
|ED.. ((t(X) EE, [Y| LB. [(((X) Ep, [x|x)))]] (Triangle inequality)
+ |EDD [Varaa [Y|X]] E,,, [Varp, [X]]]]]
Now it suffices to bound:
[ED,((((X) ED, [X ED, (((()) ED, Y x1)
=[Ex,[(h(X) ED, [ ED. En, [(((X) ED. YY xX))
+ EED (((X) - TE,,, [V] ED, (1(()) - TE,, YY xT)
< [ED,(((()) - IE,, (Y [x] - ED, E,,,(/,,) [(h(X) - ED. YY XI)
(Triangle incquality)
+ (At Ep, Y x1) - ED. [(((x) - Ep, [x)3].
Invoke Lemma A.1 and Lemma A.2 to bound the above terms:
( F.,, [.. [
Fp, E.p (((()) - E., [f]
- Ep, + - (X]]) - Ep, -b, (x(x))))
- < 4M Bttp DI (Lemma A.I & Lemma A.2
13

---

Under review as a conference paper at ICLR 2021
By symmetry, we also have:
[x))(((() PR, [.,,I.])]] [M]]]] ED. (x(()) ED, [r]]]
+ ED [h Ep, [ X - ED, (((()) - Ep, [x]]]
2 4.M Ep, [EDD x| ED, [r]]]]] +
(Dooo) (x)). (Lemma A.1
Lemma A.2
Combining the two inequalities above together, we have:
EDD (x(x) E,, ED. (M(X) ED, (Y xX)
+ EED. (h(X)) E,, [X|x]) ED, (h(X)) ED. Y x))
S (D)(),,
+ 4M min(En, [|E [Y]X] - ED, [V] Er, [|B,, [Y]] - ED, [*X]]]]]
Incorporating the two variance terms back to the above inequality then completes the proof.
Theorem 3.3. Assumption 2.1 holds, then for H. y = ((X),, the following inequality holds:
( < 8M
+ 3M min min Dg [ EDG Mil, ED, [g [P] EDy FP1l?
Proof. First, we show that for {0. 1):
ErTD, (h) = Ep. [(((X) Y) = ED, [62 - xx = (h?())
2Xx(X)] + ED..(Y)
Next, we bound the error gap:
EETDD (4) - Errp, (h))
-EEp,1h2 (X) 2Y + ED, ED, (XY()))))X(X))----) --) EE, [Y
I ED, (X) 2Y)(X)] - ED, [ (X)(X)I [ED, [Y ED, [Y (Triangle inequality)
For the second term,
casily prove that
- [Y [r = (Y dDo 1000-dD11 dDi)) < | dDill 111 S (Too())) DY))), where the second equation follows Hölder's inequality and the last equation follow the definition of total variation distance. Now it suffices to bound the remaining term:
Ep,lh² (xx 2Y)(X) ED, [x)(X) 2Yh(X)I
=I (x)(((())))) - 2y) dxoo(x v) - / ((x)(M(()) - 2y) dunr(x
S If ((x)((())))) 2y) dx (x)(()))))o)l)) - JI ((x)(((x)) 29) dpo((ly)dd)
(Triangle inequality)
+ +|ʃ (/)(((()))))))))))))))<()))))() - 2y) dei (x 1ʃ ((x)(((()))) 29) dpo((()))
We upper bound the first term:
duolr) - /f ((x)(h(x)) - (no))l)))))) der())
(()((((x) - 2y)
$ // [x(x)(((x) - 22)(() d1)( (x)())
$ / (dpo(s) - d(r(w)) / ((up)())] Mxx)) 2y| duo(xly)
S SM| Bo. 2/ = v) ( dpo(s) - dyr(v))
(Assumption 2.1)
S 3M2 / /lapo(0) -dpr(v)
(Assumption 2.1
< (
14

---

Under review as a conference paper at ICLR 2021
Note that the last equation follows the definition of total variation distance. For the second term, have:
lf (())(((x) - 2y) (x dui(t) - // h(x)()(xx) - 2y)
/] (x)(((((x)))) uun (x))) dup dur(v) (v) + Iʃ 2y (xx)(d) - dxu(xlu)) du (v)
(Triangle incquality)
3M D [Y] Ey [I]
(Assumption 2.1
To prove the last equation, we first that:
12 (x)) duo(xlv)) de) dur(v)
(xuph(()))(((((( - dxo(()))) dub (v)
M / 1ED,I((X) = - - - =1] (v)
(Assumption 2.1
M ED, [EEDy [ [P]]]
Similarly, we also have:
|| 2y (x) - (xl((( apr(o))
///////sppp)t//a (sup duo((x))) dunle))
/ʃ
2
S 2M / [BD.l(())1 = = BB. (xX)) = dpr(y) (Assumption 2.1
2M ED,1|Ep Y Ey [P]
By symmetry, we can also see that:
EE. (hool42(A (X))())))-))))))))(()))) 21 h(x)] Ep, [x²(() - 2Y [(X)]] 6.M (DN 3M BBy [8] - EDy [1]]].
Combine the above two equations yielding:
Ep, (h,,(h(()) - 2Y4(X) Ep, [x² - 2Y (X)l
< <6M (Y) (Y)) + 3M min (En, [E y - EDy [ Er, [E [ - E
Incorporating the terms back to the upper bound of the error gap then completes the proof.
Theorem 3.4. Consider the minimax game in (T). The equilibrium (g", f() the game is attained when 1). Z (X) is independent of A conditioned Y; 2). f (Z = D(A Y, Y,
Proof. To prove Theorem 3.4 we first give Proposition A.I
Proposition A.1. For any feature map g X Z, assume that F contains all the randomized binary x x A, then minjer fer CEp(A || f()(X), H(A | Z,Y).
classifiers and F
Proof. By the definition of cross-entropy loss, have:
CEp(A || f) = EEp [I( A +1 log((((())) YYM
-EE,, [1(AA 0) log(1 -14 Y))]
-EE,y [I(A =
=-Ez, [D(A
- Y) log (og()
= Ez,y (DxA 7,Y) [(22 HH,)
2 H(A) Z, Z,Y),
where (-I(- denotes the KL divergence between two distributions. From the above inequality, it also clear that the minimum value the cross cross-entrop entropy loss is achieved when fZZ, Y) equals the conditional probability D(A 1 7, Y), i.e., (2, D(A = 7 = g())
15

---

Under review as a conference paper at ICLR 2021
Proposition A.1
states that the minimum cross-entropy loss that the discriminator can achieve is
(A) Z. Y when f is the conditional distribution D = 1 Z = g(X), Y). By the basic property of conditional entropy, we have:
min CED(A FEJ
(A II f((()), (A Y,Y) H ((A Y).
Note that (A Y) is a constant given the distribution D, so the maximization of g is equivalent to the minimization of minz-g(X gXX) [AA; Z Y), and it follows that the optimal strategy for the transformation g is the one that induces conditionally invariant features, e.g., [AA; Z Y) 0. On the other hand, if 9" plays optimally, then the optimal response of the discriminator is given by
f²( = D(A Z = (x), = D Y
Theorem 3.5. Let g² == arg min, Wi ())) D) K)), then (Z = =-'(X)) =
DV (Z = 9" (X)) almost surely.
Proof. By the definition of Wasstertein distance, we have:
inf (1166(2 7EI(Po.D;) / d((zo: yo), 21) ((20 yo), (z1, 31)) / rET(Po.D1) pp // d((zo. yo). (z) y1 d (zo. 21 yo: v1) dy 8/1)
inf
inf rer(Pa, -erinn p/ /f ||zo 2i + 1y0 yi dz 21
yo: yn dr( v1)
inf Ter(Do.D;) /f lwo vil dr(vo. vn) dz(2o. y0.
inf (ETTD.(M).PIM)) / |1o vn d7 v1)
To finish the proof, next we prove the lower bound is achieved when Dj (Z = - =
g" (X)): it is easy to see Wi (D) (Z), Dj (Z)) ||zo - z ll dz ži you 91) when the conditional distributions are equal. In this case, when the Wasserstein distance is minimized, then Z conditionally independent of given Y.
B EXPERIMENTAL DETAILS
Adult The Adult dataset contains 48,842 examples for income prediction. The task is to predict whether the annual income of an individual is greater or less than 50K/year based on the attributes of the individual, such as education level, age, occupation, etc. our experiment, we use gender (binary) as the sensitive attribute. The target variable (income) is an ordinal binary variable: 0 if < 50K/year otherwise After data pre-processing, the dataset contains 30, 162/15,0600 training/test instances where the input dimension of each instance is 113. We show the data distributions for different demographic subgroups in Table [
To preprocess the dataset, we first filter out the data records that contain the missing values. We then remove the sensitive attribute from the input features and normalize the input features with its means and standard deviations. Note that we use one-hot encoding for the categorical attributes.
For our proposed methods, we use three-layer neural network with ReLU as the activation function of the hidden layers and the sigmoid function as the output function for the prediction task (we take the first two layers as the feature mapping). The number of neurons in the hidden layers is 60. We train the neural networks with the ADADELTA algorithm with the learning rate 0. and batch size of 512. The models are trained in 50 epochs. For the adversary networks in CENET and WASSERSTEINNET.
two-layer neural network with ReLU as the activation function. The number of neurons in the hidden layers of the adversary networks is 60. The adversary network in CENET also sigmoid function as the output function. The weight clipping norm in the adversary
16

---

Under review as a conference paper at ICLR 2021
network of WASSERSTEINNET is 0.005. We use the gradient reversal layer (Ganin et al. 2016) to implement the gradient descent ascent (GDA) algorithm for optimization of the minimax problem since it makes the training process more stable (Daskalakis & Panageas 2018). For the rest of the datasets we used in our experiments, we also use gradient reversal layer to implement our algorithms.
We use the Fairlearn toolkit (Bird et al., 2020) implement BGL: we use the exponentiated-gradien algorithm with the default setting as the mitigator and vary the upper bound E {0.07, 0.1, 0.2, 0.5) of the bounded group loss constraint. For each value of c, run ten random seeds and compute the means and standard deviations.
The COMPAS dataset 6,172 instances to predict whether criminal defendant will recidivate within two years or not. contains attribute such as age, race, etc. In our experiment, we use race (white or non-wo non-white) as the sensitive attribute and recidivism as the target variable. We split the dataset into training and test set with the ratio 7/3. We show the data distributions for different demographic subgroups in Table ?
COMPAS
For all methods, we use a two-layer neural network with ReLU as the activation function of the hidden layers and the sigmoid function as the output function for the prediction task (we take the first layer as the feature mapping). The number of neurons in the hidden layers is 60. We train the neural networks with the ADADELTA algorithm with the learning rate 1.0 and batch size of 512. The models are trained in 50 epochs. For the adversary networks in CENET and WASSERSTEINNET, use a two-layer neural network with ReLU as the activation function. The number of neurons in the hidden layers of the adversary networks is 10. The adversary network in CENET also use sigmoid function the output function. The weight clipping norm in the adversary network of WASSERSTEINNET is 0.05.
We use the Fairlearn toolkit implement BGL: we use the exponentiated gradiient algorithm with the default setting as the mitigator and vary the upper bound e E {0.1, 0.2, 0.3, 0.5) of the bounded group loss constraint. For each value of e, we run ten random seeds and compute the means and standard deviations.
As for CoD, follow the source implementation We use the same syper-parameter settings as (Komiyama et al. (018): We use the kernelized optimization with the random Fourier features and the RBF kernel (we vary hyper-parameter of the RBF kernel 7 {0.1, 1.0, 10, 100)) and report the best results with minimal MSE loss for each time we change the faimess budget e. We also vary e E {0.01, 0.1, 0.5, 1.0} and run ten random seeds and compute the means and standard deviations.
Table 1:
Data distribution of Y and A in Adult dataset.
Table 2:
Data distribution of Y and A in COMPAS dataset.
0
Y =
Y=0
=
= 0 A 13026
20988
9539
A = 0
1849
1 1148
1669
A
1514
1661
Communities and Crime The Communities and Crime dataset contains 1,994 examples of socio- economic, law enforcement, and crime data about communities in the United States. The task is to predict the number of violent crimes per 100K population. All attributes in the dataset have been curated and normalized to [0, 1]. In our experiment, we use race (binary) as the sensitive attribute: 1 if the population percentage of the white is greater or equal to 80% otherwise 0. After data pre-processing, the dataset contains 1,595/399 training/test instances where the input dimension of each instance is 96. We visualize the data distributions for different demographic subgroups in Figure (3b
To preprocess the dataset, we first remove the non- oon-rrddittive attributes and sensitive attributes from the input features. Note that all features the dataset have already been normalized [0, 1] so that we do not perform additional normalization to the features. We then replace the missing values with the mean values of the corresponding attributes.
For all methods, we use two-layer neural network with ReLU as the activation function of the hidden layers and the sigmoid function as the output function for the prediction task (we take the first
17

---

Under review as a conference paper at ICLR 2021
layer as the feature mapping). The number of neurons in the hidden layers is 50. We train the neural networks with the ADADELTA algorithm with the learning rate 0.1 and batch size of 256. The models are trained in 100 epochs. For the adversary networks in CENET and WASSERSTEINNET, we use a two-layer neural network with ReLU as the activation function. The number of neurons in the hidden layers of the adversary networks is 100. The adversary network in CENET also use sigmoid function as the output function. The weight clipping norm in the adversary network of WASSERSTEINNET is 0.002.
We use the Fairlearn toolkit to implement BGL: we use the exponeniateed-gradien algorithm with the default setting the mitigator and vary the upper bound (0.01, 0.02, 0.03, 0.05) of the bounded group loss constraint. For each value of 6, we run ten random seeds and compute the means and standard deviations.
As for CoD, we follow the same papam--rr settings as (Komiyama et al. 2018): We use the kemnelized optimization with the random Fourier features and the RBF kernel (we vary hyper- parameter of the RBF kernel E {0.1, 1.0, 10, 100}) and report the best results with minimal MSE loss for each time we change the fairness budget e. The hyper-parameter settings follow from (Komiyama et al. 2018). We also vary e E {0.01, 0.1, 0.5, 1.0] and run ten random seeds and compute the means and standard deviations.
350
700
AwO
A=1
300
600
examples
250
Examples
500
200
400
150
300
200
50
100
.050
.350
.250
.350
.600
.350
.6.9
.750
850
.950
.55
.150
259
330
.450
.550
.050
.350
.850
.352
(a) Law School Dataset
(b) Communities and Crime Dataset
Figure 3: Data distributions for different demographic subgroups in two datasets.
Law School The Law School dataset contains 1,823 records for law students who took the bar passage study for Law School Admission The features in the dataset include variables such as undergraduate GPA. LSAT score, full-time status, family income, gender, etc. In our experiment, use gender as the sensitive attribute and undergraduate GPA as the target variable. We split the dataset into training and test with the ratio 8/2. We show the data distributions for different demographic subgroups in Figure 3a
For all methods, we use two-layer neural network with ReLU the activation function of the hidden layers and the sigmoid function as the output function for the prediction task (we take the first layer as the feature mapping). The number of neurons in the hidden layers is 10. We train the neural networks with the ADADELTA algorithm with the learning rate 0.1 and a batch size of 256. The models are trained in 100 epochs. For the adversary networks in CENET and WASSERSTEINNET, we use a two-layer neural network with ReLU as the activation function. The number of neurons in the hidden layers of the adversary networks is 10. The adversary network in CENET also use sigmoid function as the output function. The weight clipping norm in the adversary network of WA is 0.2.
We use the Fairlearn toolkit to implement BGL:: we use the exponentiated-gradde default setting the mitigator and vary the upper bound (0.01, 0.02, 0.03, 0.05) of the bounded group loss constraint. For each value of we run ten random seeds and compute the means and standard deviations.
algorithm with the
We use the edited public version of the dataset which can be download here: https: http::://gittbb..oo com ai
18

---

Under review as a conference paper at ICLR 2021
As for CoD, follow the same the kernelized optimization with the random Fourier features and the RBF kernel (we vary hyper- parameter of the RBF kernel y E {0.1, 1.0, 10, 100}) and report the best results with minimal MSE loss for each time we change the fairness budget 6. The hyper-parameter settings follow from (Komiyama et al. 2018). We also vary e {0.01, 0.1, 0.5, 1.0) and random seeds and compute the means and standard deviations.
settings as (Komiyama al. 2018): We use
C ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSES
In this section, we provide additional experimental results and analyses.
C.I IMPACT OF FAIRNESS TRADE-OFF PARAMETERS
We present additional experimental results and analyses to gain more insights into how the fairness trade-off parameters (e.g., Ả and e) affect the performance of the model predictive performance and accuracy disparity in each methods.
Table R2 regression scores and error gaps when À changes in CENET and WASSERSTEINNET.
0.0
1.0
10
50
100
CENET 0,4419=00.0024 0.4179=00.0019
0.3908=0001.36 36 0.3410=0.0210
0.2813=0.0215
CENET 0.4419:+0.0024
WASSERSTEINNET
0,4388=+0.002.
0.4136=0.00032
0.3891=0.0063 0.3653=0.0125
Adult
0.0697.10.0004
0.0647110.0011
0.059610.0027
0.062110.0057 0.0678_20.0011
Agn
WASSERSTEINNET 0.0697:200004 0.0691=0.0006
0.0698=0.0011
0.0631=0.0022
0.0592=00.003
A
0.0
0.1
0.5
1.0
5.0
CENET
0..6312
0.161010.0119 0. 0.1542 0.0129
0.1515
0.0125
10.IST
WASSERSTEINNET 0.1631
10.0127 1664510.0136
0.1564 1 10.0225 0.147110.0051
0.1439
0.1439-0.0143 0..0143
COMPAS
CENET
2.088+0.000
K.007550.0MM
0.0067 0.0046
0.0039 0.0066
0.0063 0.0006
Err
WASSERSTEINNET 0.00881100044
0.008810.0045
0.0079-10.044
0.0070=0.0036
0.0072-10.0032
0.0
0.1
5.0
10
CENET
0.5435=:0.0077
0.529010.0107
0.1632:40.0572
0.1334-10.0720 O. 1692-0.. 1509
R²
Crime
CENET 0.5435 10.0077 0,5467 10,0063
WASSERSTHINNET
0,5472 1000065
0.5446
0,5319 0.0091
1000143
E0:OUU
0.0175100,00
0.023010:0027
0.0221
10.0079
0.0265-0:005T
AET
WASSERSTBINNET 0.0191 +0.0003
0.0194+0.0000
0.0191 11991+0.0000 0.0004
0.0180
+
0.0005
0.0173+0.00 0.0010
A
0.1
5.0
10
CENET 1.00944
0.0200
0.1059
0.046410.054
0.0235
0.0032
112
WASSERSTEINNET 0.1197=000314 0.1134+0.0339
0.0902++0.029
0.0316=0000476
0.0146=0.0552
Law
CENET
0.0102++0.0010 0.0101+0.0009
0.0090=0.0018 0.0070=0.0030 0.0066=0.0030
Agrr
WASSERSTEINNET 0.0102+10.0100 0.009810.0016 0.009010000019 0.0072+0.0025 0.0069=000027
Table 3 shows R² regression scores and error gaps when À changes in CENET and WASSERSTEIN- NET. We that the gap gradually decreases with the increase of the trade-off parameter À in most scenarios with small accuracy loss (except for CENET Adult dataset and Crime dataset when À is large), which demonstrates the overall effectiveness of our proposed algorithms. Plus, the increase of À generally leads to the instability of training processes with larger variances of both values of f22 and error gap. In contrast WASSERSTEINNET CENET outperforms in mitigating the accuracy disparity while achieving similar or better accuracy in COMPAS and Law dataset. In Adult and Crime dataset, when À is small, CENET also does better in reducing the error gap than WASSERSTHINNEL with similar accuracy loss. The results follow the fact that minimizing total variation distance between two continuous distributions ensures the minimization of Wasserstein distance (Gibbs & Sul 2002). However, when À increases, WASSERSTEINNET achieves better accuracy and performance disparity trade-off while CENET suffers significant accuracy loss and may fail to decrease the error gap. It is not surprising since the estimation of total variation in minimax optimization could lead to an unstable training process (Arjovsky & Bottou 2017 Arjovsky et al. 2017).
Table shows R² regression scores and error gaps when changes in BGL. We see that with the decrease of the trade-off parameter e, both the values of R22 and error gaps decrease. This is because when upper bound e of BGL small, the accuracy disparity is also mitigated. When is above/below certain threshold, the values of R² and error gaps then increase/decrease. It is also worth to note that the cooonnniited-ggadiiin approach to solve BGL does not introduce the randomness during optimization.
19

---

Under review as a conference paper at ICLR 2021
Table R regression scores and error gaps when changes in BGL.
0.07
0.1
0.2
0.5
Adult
R2- 1.500810.000
0.3508:10.00 0000
0.3696=100000
0.3696=10.0000
Am
0.0612:10.0000
0.0612::0.0000
0.0726==000000
0.0726=10.0000
0.1
0.2
0.3
0.5
COMPAS
0.1478:10.0000
0. 0.147810.0000
0. 0.150710.0000
0.1507:10.0000
Arr
0.0072:10.0000
0.0072+10.000
0.0066=0.0000
0.0086+10.0000
0.01
0.02
0.03
0.05
Crime
R
0.3922==0.0000
0.3922++0.0000
0.5380=0.0000
0.5380=0.0000
AET
0.0189=10.0000
0.0189=000000 0.0238=0.0000
0.0238#0.0000
0.01
0.02
0.03
0.05
Law
R2
0.1407:00.0000
0. 1.00000000000
O. 0.00000000000
0.1412:10.000
Am
0.0094+0000000
0.0004
0.0094=0.0000
0.0101=0.0000
Table 5: R2 regression scores and error gaps when changes in CoD.
e
0.01
0.1
0.5
1.0
COMPAS
R²
0.1033-0.0011 0.114410.0100
0.1146=0.0099
0.1146=0.0099
A
0..004 10.0000000055
0.0085 0.0060
1.0085+0.006/
0.01
0.1
0.5
1.0
Crime
R2
0.1262-10.0000
0.3284 10.0000
0.3603 - 0.0000
0.3603 0.0000
En
0.0312:10.0000 0.0307-10.0000
0.0343-0.0000
0.0343-100000
0.01
0.1
0.5
1.0
Law
R2
0.1262-F0.0000
0.3284 F0.0000
0.3606
0.0000
0.3603-100000
A Err
0.0312-+0.0000
0.0307+0.000
10343-0.0000
0.0343
Table 5 shows R² regression scores and error gaps when changes in CoD. We see that with the decrease of the trade-off parameter c, both the values of R² and error gaps decrease. It is worth to note that the the optimization of QCQP to solve CoD does not introduce the randomness, and the only randomness introduced in COMPAS dataset is because using the random Fourier features in prediction achieves the best performance in COMPAS dataset.
C.2 VISUALIZATION OF TRAINING PROCESSES
We visualize the training processes of proposed methods CENET and WASSERSTEINNET in the Adult dataset and COMPAS dataset in Figure 4 and Figure respectively. We also compare their training dynamics with the model performance that solely minimize the MSE loss (i.e., X 0) and we term it as No DEBIAS.
Debias
Debias
0.22
0.67
0.18
Emeer Gap
0.06
M55
0.18
0.14
6.12
0.03
0.10
0.02
10
20
30
40
50
10
20
30
40
50
Tpoch
Eporc
(a) MSE Loss
(b) Error Gap
Figure 4: Training visualization CENET, WASSERSTEINNET (A = 50) and No DEBIAS (A = 0) in the Adult dataset.
20

---

Under review as a conference paper at ICLR 2021
No Debias
No Debias
0.25
CENet
GENet
Wesserstaattet
Winuerrabeireee
0,230
BrerGap
NSE
0.225
0.007
0.036
0.005
0.210
0.094
0.205
10
30
50
10
Epoch
Epech
(a) MSE Loss
(b) Error Gap
Figure 5: Training visualization of CENET, WASSERSTHINNET (A 5) and No DEBIAS (A = 0) in the COMPAS dataset.
In Figure 4 and Figure we can see that as the training progress, the MSE losses in both datasets are decreasing and finally converge. However, the training dynamics of error gaps are much more complex even in the No DEBIAS case. Before convergence, the training dynamics of error gaps differs among different datasets. Our methods enforce the models to converge to the points where error gap are smaller while preserving the models' predictive performance. is also worth note that minimax optimization makes the training processes somehow unstable.
21


Reviews:
[{"review_id": "oHw6PMJPZ_h", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Figure 1 (left) is illustrative and insightful. I suggest authors include a simulated/semi-synthetic dataset (semi-synthetic versions can be created using the datasets that are already being used for experiments) and show clearly that their adversarial training procedure is likely to learn hypothesis within the feasible region that guarantees or is close to minimizing the disparity. I think the cost function and training procedure will have additional grounding with such analysis.\n\n2. Missing citation - https://arxiv.org/abs/1901.10566 and comparison in related work.\n\n3. \"Nevertheless, throughout this paper we mainly focus accuracy parity as our fairness notion, due to the fact that three widely used commercial face recognition systems have been shown to exhibit substantial accuracy disparities between different demographic subgroups (Buolamwini & Gebru, 2018). This observation has already brought huge public attention (e.g., see New York Times, The Verge, and Insurance Journal) and calls for commercial face recognition systems that (at least approximately) satisfy accuracy parity\". I have concerns about this claim in the paper. The takeaway of highlighted problems with commercial face recognition is not that that the methods will be ready for practical use by ensuring approximate accuracy parity. I suggest the authors remove this motivation as facial recognition as a system is fraught with many other societal challenges that cannot be resolved by ensuring parity of performance. \n\n4. Please be more elaborate in proofs in the appendix rather than having the reader fill in many gaps. \n\n5. What is the sensitivity of regularization to performance and how was the parameter selected?\n\n6. Minor - \ntypos in appendix:\n1.  \"follows immediately the triangle inequality and Lemma\" in proof of Thm 3.1\n2.  \"Now it is suffice to bound the term\" proof of Thm 3.3", "extra_scores": {}, "extra_text": {}}, {"review_id": "aCB34rueywO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper has two parts: (i) in the first part the authors provide an upper and a lower bound for error disparity among groups in the regression setup. (ii) In the second part, they propose an adversarial learning method to mitigate the disparate error among groups. Mainly they focus on reducing the error disparity by reducing the distance between the joint distribution of Z (the new learned representation) and Y between the two groups. The paper was finely written, and it was interesting. I will address my concerns regarding this paper as follows:\n\nFirst part: \n\nI did not understand the importance of Theorem 3.1 the lower bound is dependent on the classifier (thus is not inherent) and the authors only focused on the first term and did not talk about the second term.\n\nRelated work: There is some related work after Chen 2018 paper which focuses on understanding the source of error disparity among groups (e.g., https://proceedings.icml.cc/static/paper_files/icml/2020/1320-Paper.pdf which also points out the difference between the distribution of the groups or https://arxiv.org/pdf/1906.08386.pdf ). I think the authors should explain the other work on understanding the source of error disparity.\n\n\nSecond part:\n\nMy main concern regarding this part is that I feel like the authors did not place their work accurately among the related work for the fair representation part. In particular, why we cannot readily use Madres 2018 approach for the regression task. I think the authors should explain why the regression setup provides new challenges on fair learning in comparison to the classification setup. In the experiment section, for the binary datasets do we get the same result as this paper if we use the fair representation methods in classification? What is inherently different here. \n\n\nMinor suggestion: I got a bit confused about equation 3, you wrote that the signature of f is Z-> R but it seems that you are trying to minimize joint distribution. I think instead of the game-theoretic interpretation (which is kind of clear) if you can expand on this and why the optimization is easier is better. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ww1weCPtFwE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with a fair regression problem in which the accuracy disparity is employed as a fairness measure. The authors derived the upper and lower bounds on the difference of accuracy between groups to demonstrate that imbalance in the groups' sizes leads to accuracy disparity. Furthermore, they propose learning algorithms enabling us to mitigate the accuracy disparity, which is accomplished by minimizing the upper bound they derived. The empirical evaluations show that the present methods achieve a better trade-off between accuracy and fairness than some existing fair regression methods.\n\n\nThe strong points are as follows:\n- The authors tackle an important and interesting problem, disparate mistreatment in the regression problem, which is not much investigated so far.\n- The authors give some theoretical insight into the hypothesis that the imbalanced groups lead to unfairness by ignoring the minor group.\n\nThe weak points are as follows:\n- The theoretical results are somewhat weak to support the claim that the imbalance groups lead to the accuracy disparity.  \n- This paper lacks a comparison with some important existing methods that share the same concept for mitigating unfairness.\n- There is something suspicious in the experimental results.\n\nOverall, my recommendation is rejection because the theoretical and experimental results are somewhat weak to support the authors' claims. Also, I have concerns that the experimental results are something wrong.\n\nThe theoretical contribution the authors claim to make is to clarify the cause of the accuracy disparity. The cause here is the imbalance in groups, as discussed in the geometric interpretation paragraph on page 4.  Theorem 3.1 and Theorem 3.2 give insight into the authors' claim; however, they are slightly weak to support the statement that the group imbalance yields accuracy disparity. Because Theorem 3.1 and Theorem 3.2 only provide the upper and lower bounds, the feasible region can be an arbitrary shape in the green area in Fig. 1. The possibility remains that the actual feasible region is cone-like shape such that its apex is on the diagonal line. In this case, the minimization of the overall error can result in matched accuracy. Indeed, there is a counterexample of the claim. Suppose the hypothesis class consists of functions that output a constant, and the conditional variances $\\mathrm{Var}[Y|A]$ are equivalent for any $A$. Then, the optimal regressor, which minimizes the overall mean squared error, achieves the matched accuracy because $\\mathrm{Err}_{\\mathcal{D}_a}= \\mathrm{Var}[Y|A=a]$ in this case. I guess some assumptions on the hypothesis class and underlying distribution are necessary to prove the authors' claim. \n\nThe authors design the proposed algorithm to match the distribution over the representation conditioned on the true outcome and sensitive attribute. This requirement is equivalent to the equalized odds. There are several works for tackling the fair regression under the equalized odds constraint, including \n- J. Mary et al. Fairness-Aware Learning for Continuous Attributes and Treatments. In ICML'19.\n- H. Narasimhan et al. Pairwise Fairness for Ranking and Regression. In AAAI'20. \nSince these methods' design concept is equivalent to the presented one, the authors should clarify their method's merits compared with these existing methods.\n\nI'm very suspicious about the experimental results of BGL and CoD. We can find in the original papers of BGL and CoD that these methods can achieve $R^2$ from 0.32 to 0.64 in the crime dataset by choice of their hyperparameters. These values are calculated by my hand from the MSE shown in these papers using the relationship $R^2 = 1 - \\mathrm{MSE} / \\mathrm{Var}[Y]$. However, in the experimental result shown in Fig. 2 (c), the values of $R^2$ for BGL and CoD are at most 0.55. \n\nIn Section 4.2, the authors claim that Fig. 2 shows that trade-offs exist between accuracy and fairness. However, I cannot find such an inclination from Fig. 2. \n\nIt is unclear that the presented methods can achieve a better fairness-accuracy trade-off than the existing fair representation methods. The fair representation methods may work in the regression setting with small modifications, even if the original ones are designed for the classification setting. Hence, these methods can apply to this paper's setting.\n\n### Minor comments\n- Why are the Game-Theoretic Interpretation helpful? Why can we obtain new insights by interpreting the Eqs. (1) and (3) in a game-theoretic manner? I cannot find any new insights from the game-theoretic interpretation.   ", "extra_scores": {}, "extra_text": {}}, {"review_id": "AX-mIv40MlD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper theoretically and empirically studies accuracy disparity in regression problems. It proves an information-theoretic lower bound on the joint error and a complementary upper bound on the error gap across groups to depict the feasible region of group-wise errors. It further proposes to achieve accuracy parity theoretically and empirically by learning conditional group-invariant representations using statistical distances. \n\nOverall, I vote for accepting. The discovery between accuracy parity and the distribution gaps across groups is interesting.\n\nPros:\n    1. The paper provides a deeper understanding of the accuracy parity, which is interesting to me and motivates the proposed algorithms.\n\nCons:\n    1. The motivation for the studied problem is not very clear to me. I know the importance of both regression and accuracy parity. But why we want to consider the combination of these two notions? What is the main difference between considering classification with accuracy parity?\n    2. Page 4, Geometric Interpretation. Mention Theorem 3.3 by a mistake?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This papers considers the problem of accuracy disparity in regression for the case of binary sensitive attributes. It provides bounds for accuracy disparity and introduces two methods to enforce this criterion based on representation learning. 

The reviews are in agreement that the paper is generally clear and well written, but have different opinions regarding the significance of the contribution and the experimental section. I did read the paper with care myself and overall I do share the concerns raised by Reviewers 3 and 4 that the paper does not place itself accurately wrt to the current literature, both in the discussion and the experimental section. The response to the reviewers about methods that can achieve accuracy disparity for classification is not satisfactory, also considered that two of the analysed datasets are about binary classification tasks. Regarding the results on these datasets, it would be useful to report classification accuracy rather than (in addition to) R^2. 
The proposed methods do not seem to show a significant advantage versus the methods considered for comparison.

Minor comments:
The proposed methods are inspired by Theorem 3.2. However, is not enforcing accuracy disparity by minimising some distance between conditional distributions what the literature does? I cannot think of other meaningful ways to achieve this criterion. In fact, referring to this theorem, the authors says 'However, it is nearly impossible to collect noiseless data with group-invariant input distribution. Moreover, there is no guarantee that the upper bound will be lower if we learn the group-invariant representation that minimizes dTV(D0(X), D1(X)) alone, since the learned representation could potentially increase the variance. In this regard, we prove a novel upper bound which is free from the above noise term to motivate aligning conditional distributions to mitigate the error disparity'. But minimizing dTV(D0(X), D1(X)) might not be desirable if the dependence between the sensitive attribute A and the data is considered legitimate. The point of using conditional distributions is to allow that dependence to be retained. 





Decision: Reject