Pareto Optimality in No-Harm Fairness
Authors: Natalia Martinez
Martin Bertran
Guillermo Sapiro
Venue:  (2020)
Keywords: Fairness
Fairness in Machine Learning
No-Harm Fairness

Abstract: Common fairness definitions in machine learning focus on balancing various notions of disparity and utility. In this work we study fairness in the context of risk disparity among sub-populations. We introduce the framework of Pareto-optimal fairness, where the goal of reducing risk disparity gaps is secondary only to the principle of not doing unnecessary harm, a concept that is especially applicable to high-stakes domains such as healthcare. We provide analysis and methodology to obtain maximally-fair no-harm classifiers on finite datasets. We argue that even in domains where fairness at cost is required, no-harm fairness can prove to be the optimal first step. This same methodology can also be applied to any unbalanced classification task, where we want to dynamically equalize the misclassification risks across outcomes without degrading overall performance any more than strictly necessary. We test the proposed methodology on real case-studies of predicting income, ICU patient mortality, classifying skin lesions from images, and assessing credit risk, demonstrating how the proposed framework compares favorably to other traditional approaches.

Under review as a conference paper at ICLR 2020
PARETO OPTIMALITY IN NO-HARM FAIRNESS
Anonymous authors Paper under double-blind review
ABSTRACT
Common fairness definitions machine learning focus on balancing various no- tions of disparity and utility. In this work we study faimess in the context of risk disparity among sub-populations. We introduce the framework of Pareto-optimal fairness, where the goal of reducing risk disparity gaps is secondary only to the principle of not doing unnecessary harm, concept that is especially applicable to high-stakes domains such as healthcare. We provide analysis and methodology po-unnecessary-iee harm classifiers
obtain maximally-fai argue that even in domains where fairness at cost is required, no-unnecesary-har fairness can prove to be the optimal first step. This same methodology can also be applied to any unbalanced classification task, where we want to dynamically equalize the misclassification risks across outcomes without degrading overall per- formance any more than strictly necessary. We test the proposed methodology on real case-studies of predicting income, ICU patient mortality, classifying skin le- sions from images, and assessing credit risk, demonstrating how the proposed framework compares favorably to other traditional approaches.
finite datasets. We
INTRODUCTION
Machine learning algorithms play an important role in decision making in society. When these al- gorithms are used to make high-impact decisions such as hiring, credit-lending, predicting mortality for intensive care unit patients, or classifying benign/malign skin lesions, it is paramount guar- antee that these decisions are both accurate and unbiased with respect to sensitive attributes such as gender ethnicity. A model that is trained naively may not have these properties by default; see, for example BBrocas & Selbst (2016).
In these critical applications, it is desirable to impose some fairness criteria. Much of the fairness in machine learning literature attempts produce algorithms that satisfy Demographic Parity, which aims to make al algorithm's 's predictions independent of the sensitive populations (Louizos et al. (2015); Zemel et al. (2013); |Feldman et al. (2015)) ); or Equality of Odds or Equality of Opportunity, which aims to produce predictions that are independent of the sensitive attributes given the ground truth (Hardt et al. (2016); Woodworth et al. (2017). Notions of Individual Fairness have also been ad- vanced (Dwork et al. (2012); Joseph et al. (2016); Zemel et al. (2013). These notions of fairness can be appropriate in many scenarios, but in domains where quality of service is paramount, such as healthcare, we argue that necessary strive for models that are close to fair as possi- ble without introducing any unnecessary harm to any subgroup (Ustun et al. (2019). Even the overall fairness goal is potentially harmful, zero-gap classifier, pursuing first Pareto-fair classi- fier and later applying other harmful methodologies ensures that all possible non- harmful trade-offs covered before explicitly degrading performance, therefore minimal harm is introduced to the decision.
In this work we make use of the concept of Pareto optimality to measure and analyze discrimination (unfairness) in terms of the difference in predictive risks across sub-populations defined by our sensitive attributes, a fairness metric that has been explored in other recent works such as Calders & Verwer (2010): Dwork et al. (2012) Feldman et al. (2015): Chen et al. (2018;; Ustun et al. (2019). We examine the subset of models from our hypothesis class that have the best trade-offs between sub-population risks, and select from this set the one with the smallest risk disparity gap. This is direct contrast to common post-hoe correction methods like the ones proposed in Hardt et al. (2016); Woodworth et al. (2017). where noise is potentially added to the decisions of the best performing sub-population. While this latter type of approach diminishes the risk-disparity gap, does so by degrading performance on advantaged groups, the previously disadvantaged groups do not directly benefit from this treatment. Since our proposed methodology does not require test-time

---

Under review as a conference paper at ICLR 2020
access to sensitive attributes, and can be applied to any standard classification regression task, it can also be used to reduce risk disparity between outcomes, acting as an adaptive risk equalization loss compatible with unbalanced classification scenarios.
Main Contributions. We formalize the notion of no-harm risk fairnes using Pareto optimality (Mas-Colell et al. (1995)), state of resource allocations from which impossible reallocate without making one subgroup worse. We show that finding a Pareto-fair classifier is equivalent to finding model in our hypothesis class that belongs to the Pareto front (the set of all Pareto opti- mal allocations) with respect to the sub-population risks with the smallest possible risk disparity. This general notion is already amenable to non-binary sensitive attributes. We analyze the fairness performance trade-offs that can be expected from different approaches with an illustrative exam- ple. We provide a concrete algorithm that promotes fair solutions belonging to the Pareto front; this algorithm can be applied any standard classifier or regression task that can be trained using (Stochastic) Gradient Descent. We show that if the goal is to obtain a zero-gap classifier, first recov- ering the fairest Pareto optimal solution and then adding harmful post-hoc corrections ensures the lowest risk levels across all subgroups. We demonstrate how our methodology performs on synthetic and real tasks such as inferring income status in the Adult dataset Dua & Graff (2017a) irrespective of their ethnicity or gender, predicting ICU mortality rates in the MIMIC-II dataset from hospital notes Johnson et al. (2016), classifying skin lesions in the HAM10000 dataset
TaatasetTschanddee Tschandl et al. (018))
and assessing credit risk on the German Credit dataset Dua & Graff (2017b).
2 RELATED WORK
There is an extensive body of work on fairness in machine learning. Following Friedler et al. 2019%
we compare our methodology against the works Feldman et al. 5): Kamishima et al. (2012) Zafar et al. (017)) Our method shares conceptual similarities with Zafar et al. (2017); Woodworth et al. (2017) Agarwal et al. (2018), with differences on how we define our faimess objective and adapt it to work with standard neural networks. Although optimality is often discussed in the fairness literature, it is usually in the context of auilit-diserimiiation tradeoffs. To the best of our knowledge, this is the first work to discuss optimality with respect subgroup risks a unified classifier, distinction that disallows extreme performance degradation in the pursuit of fairness.
The work presented in Hashimoto et al. (2018) discusses decoupled classifiers as way of minimiz- ing group-risk disparity, but simultaneously cautions against this methodology when presented with insufficiently large datasets. The works of Chen et al. 20:8); Ustun et al. (2019) also empirically report the disadvantages of decoupled classifiers as a way to miligate risk disparity. Here we argue for the use of a single classifier because it allows transfer learning between diverse uub-populations. We do not need access to the sensitive attribute during test time, but in cases where this is possible, we instead choose to incorporate it as part of our observation features.
The work of Chen et al. (2018) uses the unified bias-variance decomposition advanced in Domin- gos) (2000) to identify that noise levels across different sub-populations may differ, making per- fect fairness parity impossible without explicitly degrading performance on one subclass. Their methodology attempts to bridge the disparity gap by collecting additional samples from high-risk sub-populations. Here we modify our classifier loss to bridge the disparity gap without inducing unnecessary harm, which could prove to be synergistic with their methodology.
PROBLEM STATEMENT
Consider we have access to dataset DD (10,, ye, a.));= containing n independent triplet samples drawn from a joint distribution (I,, Vi, a.) ~ P(X, Y. where x; E X' are our input features (c.g.. images, tabular data, etc.), y/ E y is our target variable, and a; E A indicates group membership or sensitive status (e.g., ethnicity, gender); our input features X may or may not explicitly contain A.
Let h E H be classifier from a compact hypothesis class H trained to infer from h : X + y: and loss function& &. : y y - R. We define the class-specific risk of classifier l on subgroup a as (h) = Ex ((())) Y)). The risk discrimination gap between two subgroups a, a' E A is
In this paper we use hhrmharm" to mean "no-unnecessary harm." in other words, the system doesn't 't degrade performance on any class unless is strictly necessary improve performance in disadvantaged class.
2

---

Under review as a conference paper at ICLR 2020
and we define the pairwise discrimination gap vector as F (h)) = a (.))a, Our goal is to obtain a classifier h E H that minimizes this gap without causing unnecessary harm to any particular group in A. To formalize this notion, we define: RK, noted as r', if rf 2 r/ Vi = 1,... h and 3j :r > r'J (i.e., strict inequality on at least one component).
measured as I "a (h) (a)
Definition 3.1. Dominant risk vector: A vector r' E RR is said to dominate
Definition 3.2. Dominant risk classifier: Classifier // is said to dominate h. noted as h > ht, if the Al
risks vector r' = (Ra(h)) o= dominates r = [Ra (h))
Definition 3.3. Pareto front: We define the Pareto front as A)H, = [h E H AM E H h K.) This means that there is no other classifier in that at least as good in all risks and strictly better in at least one of them. It is the set of classifiers such that improving one group's 's risk comes the cost of increasing other's.
The Pareto front defines the best achievable trade-offs between population risks Rg (h). This def- inition is already suited for classification and regression tasks where the sensitive attributes are categorical. Constraining the classifier to be in the Pareto front disallows laziness, there exists no other classifier in the hypothesis class 7. that is at least as good on all class-specific risks and strictly better in one of them. As shown in Chen et al. (2018): Domingos (2000), the risk can be decom- posed in bias, variance and noise for some loss functions, where the noise is the smallest achievable risk for infinitely large datasets Baayes-optmal risk). If the noise differs between sensitive groups, zero- aero-discrimination (perfect fairness) can only be achieved by introducing bias or variance.
Literature on fairness has focused on putting constraints on the norm of discrimination gaps (Zafar al. (207:: 2015); Creager et al. (2019); Woodworth et al.) (2017). We follow similar criteria in Defimition 3.4 and define the Pareto-fait fair classifier as the classifier in the Pareto front that minimizes [["((h]ll> so (the maximum risk discrimination gap). Note that one could alternatively choose to find the Pareto classifier that minimizes the maximum subgroup risk.
Definition 3.4. Pareto-fair classifier and Parcto-fair vector: A classifier h- is optimal Pareto- fair classifier if it minimizes the discrimination gap among all Pareto front classifiers, h"
arg min [["x(,)) The Pareto-fair vector r" E RIAI is defined as r" = { {Ra (h")) )(EAA (EP(H..A)
Even when perfect equality of risk is desirable, Pareto classifiers still serve as useful intermedi- aries. To this end, Lemma 3.1, shows that applying a mechanism for reaching equality of risk on a dominated classifier lt E HH leads to equal or worse risks than applying it to a Pareto classifier hip E P(H, A) that dominates h.
Lemma 3.1. h & P(H, A) + h E P(H, hp > h A Ra (hE < (h Va, with En an equal-risk classifier: R2 ER) = max R2d (h), Ya and heR R. (h a' E'EA
max R2 (hp).
a' EA
To exemplify these notions graphically, Figure 1 shows a scenario with binary sensitive attributes and binary output variable y where none of the Pareto front classifiers achieve equality of risk. Here the noise level differs between subgroups, and the Pareto-fair vector r' is not achieved by either a Naive classifier (minimizes expected global risk), or classifier where subgroups are re-sampled to appear with equal probability (rebalanced Naive classifier). Note that the amount of performance degradation required to enforce perfect fairness starting from the Naive classifier is higher than when starting from the Pareto-fair vector.
Our objective is to find the Pareto- classifier h" as in Definition 3.4 In particular, we will give X), Y). It can be shown regularity conditions on the hypothesis class H. and the loss function (())
that for a sufficiently rich class of hypothesis functions H and for risk functions R. (h) that are
convex with respect to h, the space of risk vectors is convex (see Geoffrion (1968); Koski (985);; Miettinen (2012)). Under these conditions, we can find an auxiliary loss function o : RIAA +
defined in terms of the subgroup risks (Ra, (h))aeA (denoted from here on as r E RIAI) that has a global minima on r".
To prove the existence of loss function & with the desired global minimum r., it is convenient to think of the convex set of risk vectors as the intersection of a convex Pareto set (defined by a convex Pareto front and all risks that are dominated by it), and an additional convex set Ω. We further require both this Pareto set and Ω to be smooth, so that we can apply standard tools from smooth convex optimization and prove that for every risk vector r' in the Pareto front there exists a function that has r' as its global optima. This is formalized in Lemma Definition 3.5. Pareto field: A function P Ω + R is a Pareto field over a convex set Ω C IR if P. E C¹ is continuously differentiable function such that v,P(r) > Ovr E Ω vi 1,....k. k.
3

---

Under review as a conference paper at ICLR 2020
Data distribution
Risk tradeoffs
Discrimination Risks tradeoffs
1.0
MYY
Ro
0.20
R
Equal Risk
Naive
0,08
Pareto-Fair
(
Rebalanced Naive Naive
Rebalanced
P(X (4)
0.4
Parsto-Fair
P[X]A 0
R
Po
hamm
Equal
0.10
0
R2 R2
2
X
0.18
0.20
Ro
0.24
0.14
0.16
0.18
P.
Figure 1: Example of achievable risk tradeoffs for a binary regression problem with two unbalanced {0,1) (Pa=1 Pa=o). Bottom left and top left figures show conditional distributions of observation variable Pa (x), and target variable p(u = | a) respectively. p(u = 1 Ir, a) are simple piece-wise constant functions with levels plaw and phigh. Middle figure shows the Pareto front defined by subgroup risks; we observe that noise levels differ across subgroups and perfect Pareto fairness is unattainable. Equality of Risk requires pure degradation of service for group = 1. Both Naive and Rebalanced Naive classifiers do not attain the best possible no-harm classifier in this case. The proposed Pareto-fair point is shown in green, with its corresponding optimal Equal risk classifier shown in purple. The utopic point (Rg ,I((Pt) can only be attained when the classifier also has access to the sensitive attribute (h(X, A)). Rightmost figure shows the trade-offs attainable between discrimination and mean risks, alternative perspective to the risk trade-off figure.
sensitive groups
Lemma 3.2. Let Q C R be convex set, and P 12 + R a Pareto field Then the set D) fr Q: P(r) = 0} is Pareto set in Q and the set D+ (r E Q P(r) > 0} the set of dominated points, i.e., D {r E Q 3r' E D r r>rr}
Lemma 3.3. Let Ω RR be convex set defined by Ω fr RK gr(r) > 0Ve e {1..... C).g continuously differentiable): let Q + R be convex Pareto field with cor- responding proper Pareto set D {r E Ω : P(r) = 0}. Let & E D and o(r. = En pu(r;; c)+ with f, where denote (x)+ max? 0). There exists set
of vectors P > O such that:
f arg min o(r, ( s.t. P(r) (r) 2 0YC Cl.
rER*
Where C is the number of constraints that characterized Ω. Proofs for all lemmas given in the supplementary material, Section A.1 Lemma 3.3 motivates our loss function to take the form
|Ai d(r: E ri+ 14; c) with |A| the number of sensitive groups. The challenge is to find μ² such that our fareto-fair vector r" minimizes o(r, 14*). Section 4 we provide an algorithm to approximately recover the Pareto-fair classifier h² by adaptively searching for μe and optimizing a classifier on @(.,u) using tools from standard Stochastic Gradient Descent.
4 OPTIMIZATION METHODS
Recall that we wish to recover the Pareto-fair classifier h+ within our hypothesis class. From Lemma 3.3 there exists a loss function of the form
|A|
((rip.c) = Σ ri + uilri - c2
i=1
p.:: = =
Ra (a(h) + (o(o - c) -h)
aEA
such that h" arg aig ((h; pr, c). Note that we can state the loss function directly on the risk KER vectors r, or implicitly on the classifier Since Ra(h) is differentiable with respect to h, @(h; pt, c) can be directly minimized using gradient descent on h for any choice of values μt, Let D
4

---

Under review as a conference paper at ICLR 2020
{z. y, a) implementation of the Pareto-fair framework is formalized in Algorithm where we specify how to update the penalty coefficients c.
i- DV (2. y, i-11
Ny be our training and validation datasets respectively. The proposed
Algorithm 1: ParetoFairOOiiimzzaionn
Given: ho, l, DTr DVaal Muy Tip, Tmmaz; 7 0, 1,6, E (0, B f t- 1, p t- 1, pecom t- O, Coount 6- 0, c 0, f- 00, he
while Ccoant S Amax and JAcount S. n2 do
Mexmn <- Heram + 1, Count <- Cosunt +
ho: r + AdapiixeOpiiizzetoo, l. c, D", 'D'i,, ny, Ir, B) // Optimize current loss
// Check that solution is Pareto efficient and reduces fairness gap II (h))l < I and r is not dominated by previous validation risks then
Cold c, min, r¹ - .. (r oot d' <- arg max, ra Peaue t-
h" t he, r + [=||[
(r ((x))
clse
+r [lr, [e pet ,,, 7 75: ho + h
end
Ma + (1+
end
// Exit loop due to excessive iterations or no improvement in fairress Return: hs
We regularly check that reductions in the faimess gap generalize to the validation set; addi- tionally check if the trade-offs are in the non-dominated solution set (i.e., we have not observed a universally better classifier during training). Algorithm 2 AdaptiveOptimizz,, shown in Section A.22p summarizes how we perform stochastic gradient descent steps with early stopping in-between μi, updates. Lemma A.5 (shown in supplementary material, Section A.I shows that this algorithm is when the minimization step for fixed μ., c performed exactly.
convergent for |
To conclude this section, we stress that the proposed framework is independent of the desired al- gorithm class H. and loss function E; these are kept from the original application. The Pareto-fair classifier uses the same inputs as the Naive classifier, with parameters that have been optimized towards Pareto fairness. Code will be made available.
5 EXPERIMENTS AND RESULTS
We applied the methodology described Section to learn a Pareto-fair classifier (classifier in the group-risk Pareto front with the smallest risk disparity). We first validate our methodology on synthetic data with known Pareto-fair classifiers. Observations are drawn from a Gaussian mixture model where each sensitive attribute is encoded by a corresponding Gaussian mode, and target attributes are binary. We demonstrate our methodology show how the risk disparity gaps are subsequently reduced. Where applicable, we compare our results against the methodologies proposed in Zafar et al. (2017); Kamishima et al. (2012): Hardt al. 2016); Feldman et al. (2015).
publicly available faimess datasets, and
5.1 SYNTHETIC DATA
We tested our approach on synthetic data where the observations are drawn from conditional Gaus- sian distributions I|A a N N(MA. 1). the target variable y conditional Bernoulli variable with distribution Y|X x. = a Ber (Ja (x)) with fa(x) p.oo 1 [xr S ca] p/iggh [[x >> cal). and [A| = 3. We used Brier Score (BS, refer Section A.44 as our loss function. Under these conditions, the Baycs-optimal classifier for each subgroup is h(x)) = fz( The sub- group risk function for any classifier h can be computed as R. (h)
Ex|A=a -Y- h(x))²]
(x))=o(fa(XX(1 12())) (1 fa (x()))((X)))
Network details are given in the supplemen- tary material, Section A.6 Figure 2 shows the analytically-derived Pareto-fair classifier, as well as the trade-off obtained by the proposed algorithm. We remark that the Pareto- classifier in this scenario cannot be achieved using linear logistic regression, something that can be seen from the functional form of the Bayes Optimal classifier derived in Section A.8 in supplementary mate- rial. From the center graph in Figure 2 we see that the approximated Fair risks (dashed
5

---

Under review as a conference paper at ICLR 2020
lines) are close to the theoretical best (solid lines). The achieved maximum risk discrimination is also close to the theoretical best. The algorithm is able to correctly trade risks from the two advantaged subgroups to reduce the risk of the worst performing subgroup.
0.08
0.8
Optimal Casster
I|all
Recovered Clazs
0.07
ParetoFair [|Fali-
0.20
0.06
0.19
0.05
0.4
2
1
group
0.04
p(X (A)
20017
group
0.03
0.16
group
0.02
0.15
Paretofair air (soli)
0.01
150
200
250
50
150
X
Epochs
Epochs
Figure 2: Synthetic data experiment. Bottom left figure shows the conditional distribution of the observa- tion variables for each of the three subgroups, while upper left shows the distribution of the target variable conditioned both the observation and sensitive attribute; it additionally shows the theoretical and empirical Pareto-fair classifier. Center figure shows the empirical validation risks (dashed lines) function of epochs during the optimization procedure, the theoretical Pareto-Fair risks are also shown (solid lines). Rightmost shows how the maximum discrimination shrinks with the number of epochs.
5.2 REAL DATASETS
We evaluate the performace of our algorithm on mortality prediction (MIMIC-III, skin lesion clas- sification HIAM10000) income prediction (Adult), and credit lending (German). We also compared with several state of the art methods (Zafar et al. (2017): Kamishima et al. 2012); Hardt et al. (2016); Feldman et a (2015) and with model trained to minimize the average risk (Naive) and one that undersamples the minority class (Naive Balanced or NaiveB). A description of these methods is provided in Section A..3 of Supplementary material. Metrics used for evaluation include accuracy (Ace), confidence (Conf), expected and maximum calibration error (ECE and MCE respectively), Brier score (BS) and cross-entropy (CE). Values for every metric are reported per sensitive attribute. The dataset average is reported as (sample mean), and the group-normalized average as (group mean). The best and worst performing groups are also shown as (group max) and (group min), and the worst case difference between subgroups is reported the the metrics and datasets used for evaluation, refer to Sections AA.4 and [A.5] of Supplementary material. Here present comparison of the performance in terms of accuracy and ECE but an extensive comparison is available additional results Section A.7 in supplementary material.
(disc). For an in-depth description of
Similarly to Chen et al. 22018); Zafar et al. (2017); Hardt et al. (016)); Woodworth et al. (017)) we omit the sensitive attribute for the entire dataset to avoid needing test-time access to sensitive attributes whenever possible. All classifiers shown in this section implemented using neural networks and/or logistic regression, we used either cross-entropy or Bricr score as our training loss depending on the dataset. For details on the architecture and hyperparameters used on each dataset, refer to the supplementary material, Section A.6 We show how the proposed Pareto-fair approach produces well calibrated models that reduce group disparities across several metrics.
from observation features. Our method trains single classifier
5.2.1 PREDICTING MORTALITY IN INTENSIVE CARE PATIENTS
Medical decisions in general and mortality prediction in particular are examples where notions of fairness among sub- cobppopulations are of paramount importance, and where ethical considerations make no-harm fairness very attractive paradigm. To that end, we used clinical notes collected from adult ICU patients at the Beth Israel Deaconess Medical Center (MIMIC-III dataset) Johnson) et al. (2016) to predict patient mortality. We study faimess with respect age (adult senior), ethnicity (white or nonwhite), and outcome (alive/deceased). This leads to a total of 8 sensitive groups. We included outcome (alive/deceased) as a sensitive sub-population criteria to demonstrate a case where sensitive attributes would not available at test-time, and because in our experiments patients who ultimately passed away on ICU were under-served by Naive classifier. Table shows empirical accuracies and expected calibration errors of all tested methodologies. note how imbalanced these groups are by looking at the 'ratio" column. Here one can see that 56.7%. 76% of the samples correspond to the majority class (alive, white, senior) and only 0.4% to the minority
is important to

---

Under review as a conference paper at ICLR 2020
(a) Accuracy comparison
type
ratio
Naive
NaiveB
ZafarCE
ParetoFair CE
ParetoFair BS
sample mean
89.5%
78.9:10.7%
78.0:10.7%%
74.3
1.17=
group mean
12.5%
1.5%
7..3
6..411.3%
7..2
77.51 %
group max
56.7%
9..2.10.%
8..14133%
9..0=114%%
86.41:22%
87.142.0%
group min
0.4%
18.1=225%
1.27%
43.7+:39%
67.9+2.5%
69.9+22.2% %
disc
56.3%
81.112.5%
22.511.5%%
49.3144.4%
18.5114.2%
17.11.2.6%
(b) Accuracy comparison after Hardt processoe
type
ratio
NaiveB+H
Zafar+H
ParetoFair CE+H
ParetoFair BS+H
sample mean
7..1
1.0%
10.6:11..34
71.9+
group mean
12.5%
11.2::11%%
6..4+1.3%
0.7%=
7..3+:10%
group max
56.7%
80.0+-2.5%
70.0+22.6%
77.6+4.199
79.9+-5.6%
group min
0.4%
61.4+:34%
52.0+336%%
65.7 1.8%
66.8++2.19 %
disc
56.3%
18.6.13.2%
18.114..3%
11.91..0%
13.11.5.6%
(c) Expected calibration error (ECE)
type
ratio
Naive
NaiveB
ZafarCE
ParetoFair CE
ParetoFair BS
sample mean
0.115:10.004
0.037--0.003
0.219:00.007
0.062:+0.021
0.052=0.012
group mean
12.5%
0.288=0.016
0.068=0.006
0.305=:00013
0.094=0.023
0.082=0.021
group max
56.7%
0.593-+00028
0.166++0.047
0.561+00.04
0.172++0.05
0.156- 0.05
group min disc
0.4%
0.042:10.002
0.025:10.003
0.069:10.014
0.039:1.0133
0.032=0.011
56.3%
0.552:+0.028
0.141+0.005
0.492+:0.045
0.133:00.03
0.125-0.044
Table 1: Performance comparison on MIMIC dataset. We show accuracies and expected calibration test set. Standard deviations computed across 5 splits.
class (deceased, nonwhite, adult). Our methodology produces low discrimination classifiers with high group accuracies, for example, ParetoFair trained with brier score loss (BS) increased the clas- sification accuracy of the most best-served group by 12% when compared the Naive classifier.
group by over 50% while reducing the accuracy of the
5.2.2 SKIN LESION CLASSIFICATION
The HAM10000 dataset Tschandl et al, (018) collects over 10, 000 dermatoscopic images of skin lesions over a diverse population. Lesions are classified according diagnostic categories. We found that a Naive classifier exhibited almost no measurable discrimination based on age or race on this dataset. We instead chose use the diagnosis class as both the target and sensitive variable, casting balanced risk minimization as a particular use-case for Pareto fairness. This is possible since methodology does not require test-time access sensitive labels. It was not possible to show comparisons against Hardt et al. (2016) since the sensitive attribute is perfectly predictive of the outcome. Table 3 shows accuracies and ECE for all tested methodologies. The Pareto-fair classi- fier has the overall best calibration results and smallest accuracy disparities across methodologies. Comparisons against other methods were not possible because the target labels are non non-binary.
Groups
ratio
ParetoFair
NaiveB
Naive
ParetoFair NaiveB
Naive
sample mean
78.6%
84.8%
83.6%
0.055
0.065
0.095
group mean
14.3%
63.4%
64.1% %
32.3%
0.206
0.223
0,416
group max
81.0%
83.5%
90.9%
96.0%
0.452
0.57
0.816
group min
0.5%
51.9%
33.3%
0.0%
0.025
0.034
0.032
disc
80.4%
31.7%
57.5%
96.0%
0.426
0.535
0.784
(a) Accuracy
(b) ECE
Table 3: Accuracies (left) and Expected calibration error (right) on test set for HAM10000 dataset.
5.2.3 INCOME PREDICTION AND CREDIT RISK ASSESMENT
We tested the proposed method on the Adult UCI dataset Dua & Graft (2017a) and on the Ger- man Credit dataset Dua & Graff (0177)). In the Adult UCI dataset the goal is predict person's
7

---

Under review as a conference paper at ICLR 2020
(a) Adult gender comparison
type
ratio
Naive
NaiveB
Feldman
Kamishima
Zafar
PairtoFair
Male
67.9%
92.3:+00.4%
92.3
92.4-10.33%
92.6+t0.4%
92.2-1-0.%%
92.4
Female
32.1%
80.5::04%
80.31.07%
80.8:103%%
80.910.4%
80.810.1%
81.0 %
sample mean
84.3110.3%
84.2.10.5%
84.5:10.3%
84.6.1.3%%
84.4:10.3%
84.7-10.%% 86.7++0.2% %
group mean
50.0%
86.4+0.2%
86.3+004%
86.6+0.3%
86.7+002% %
86,5++0.4%
disc
35.9%
11.9110.%
12.0.107%%
11.6.10.2%
11.7.10.%
11.4.1..%
11.4110.6%
(b) Adult ethnicity and gender comparison
type
ratio
Naive
NaiveB
Feldman
Kamishima
Zafar
ParetoFair
White male
39.7%
90.8:10.3%
9..%:10.3%
90.6++0.3%
90.4:t1.1%
90.5+:0.6%
90.7:10.3%
Other
60.3%
0.5%%
80.4::0.6% 79.7:
79.4:22.%
80.1=0.1%
80.6:10.6%
sample mean
84.1%..2%
84.5++03% 84.1+0.4%
83.8++18%
84.2++02%
84.6+0.49%
group mean
50.0%
85.3+
05.2++
85 0..5++03%
8.69+
85.3++0.%%
05.6+t0.4%
disc
20.5%
11.0:1.8%
11.1100.%
10.2.10.7% %
11.0.1.4%
10.3.10.%
10.2.10.6%% %
(c) German comparison
type
ratio
Naive
NaiveB
Feldman
Kamishima
Zafar
ParetoFair
Female
29.5%
64.4-66.3%
64.1+...5% %
74.2+7.7%%
68.8+668%%
72.7-16.1%
65.4-:5.199
Male
70.5%
68.91:50%
70.415.3%%
72.0+137%%
72.7+:2.6%
71.613.9%
68.7:1600%
sample mean
67.6+t4.55
68.6+44.5%
72.7+-4.1%
71.6++3.%%
71.9+:3.9%
67.0++4.4%
group mean disc
50.0%
66.6.14.7%
67.31148%%
73.114.9%%
70.811.0%
72.113.5%
67.5::..1%
41.0%
7.6%:2.0%
10.4+3.2%
6.64..3%%
6.0::4.4%
6.3+t4.3%
5.3%+228%
Table 5: Performance comparison between methods for Adult and German datasets. We show accuracies test set. Standard deviations computed across splits.
income, which can be an important factor on meaningful decisions such as credit lending. In the German Credit dataset the goal is predicting credit risk. We select gender and ethnicity as our sen- sitive attributes. To compare ourselves against state of the art methods Zafar et al. (2017); [Feldman (2012) we binarize the sensitive attributes into White-Male and Other when dealing with ethnicity and gender simultaneously. or Male and Female when dealing with gen- der, and use the unified testbed provided in Friedler al. (2019). We limit hypothesis class H to linear logistic regression to compare evenly against these standard baselines. Results on both datasets are shown in Table 5
al. (2015); Kamishima
6 DISCUSSION
There exists a rich literature of fairness machine learning in general, and risk-based fairness in particular. Here we explore a relatively untapped sub-problem where the goal is to reduce risk disparity gaps in the most ethical way possible (i.e., minimizing unnecessary harm). Unlike other works in the area, our problem investigates on how to reduce this disparity gap without collecting additional data samples, using the entirety of the available training data produce an algorithm that is maximally fair with respect to sub-populations, and that does not necessarily require test-time access to sensitive attributes.
We provide concrete algorithmic adaptation to any standard classification or regression loss to bridge this disparity gap at no unnecessary harm, and demonstrate its performance on several real- world case studies. Even for applications where the need for strict fairness outweighs the need for no-harm classifiers, this methodology can be applied before any post-hoc corrections to ensure that the risk disparity gap is closed in the most risk-efficient way for all involved aub-populations. The proposed algorithm does not sweep through different disparity constraint values, as previously done in related works, making it simpler alternative.
As an avenue future research, could be of interest to analyze if we can automatically identify high-risk uob-populations as part of the learning process and attack risk disparities as they arise, rather than relying on preexisting notions of disadvantaged groups or populations. We strongly believe that no-unnecessary-hare notions of faimess are of great interest for several applications, especially domains such healthcare and lending, where decisions are highly impactful.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reduc- tions approach fair classification, arXiv preprint arXiv: arXiv:/03.0024: 1803.0243, 2018.
Solon Barocas and Andrew D Selbst. Big data's disparate impact. Calif. Rev. 104:671, 2016.
Toon Calders and Sicco Verwer. Three naive bayes approaches for discrimination-free classification.
Data Mining and Knowledge Discovery, 21(2):277-292, 2010.
Irene Chen, Fredrik D Johansson, and David Sontag. Why is my classifier discriminatory? In Advances in Neural Information Processing Systems, pp. 3539-3550, 2018.
Elliot Creager, David Madras, Jörn-Henrik Jacohsen, Marissa A Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. arXiv preprint arXiv:1906.023: 2019.
Pedro Domingos. A unified bias-variance decomposition. In Proceedings of 17th International Conference on Machine Learning, pp. 231-238, 2000.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017a. URL htRp::/t//a/cteeee ics. uci. edu/ml
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017b. URL https://
archive. ics uci. edu /ml (ataaa/aeeee/ttttlllleeeeeeaaaaa
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science confer- ence, pp. 214-226. ACM, 2012.
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubra- manian. Certifying and removing disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 259-268. ACM, 2015.
Sorelle A Friedler, Carlos Scheidegger, Suresh Venkatasubramannian, Sonam Choudhary, Evan P Hamilton, and Derek Roth. A comparative study of fairness- tainness-nnancingg interventions in machine learning. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp- 329-338. ACM, 2019.
Arthur M Geoffrion. Proper efficiency and the theory of vector maximization. Journal of mathe- matical analysis and applicationns 22(3):618-66) 630, 1968.
Moritz Hardt, Eric Price, Nathan Srebro, et al. Equality of opportunity in supervised learning. In Advances in neural information processing systems, pp. 3315-3323, 2016.
Tatsunori B Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. arXiv preprint 1rXiv:1806.00012 2018.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4703 2017.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3:160035, 2016.
Matthew Joseph, Michael Kearn,, Jamie Morgenstern, Seth Neel, and Aaron Roth. Rawlsian fairness for machine learning. arXiv preprint arXiv: 1(2), 2016.
Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-aware classifier with prejudice remover regularizer. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 35-50. Springer, 2012.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: /4/2 2014.

---

Under review as a conference paper at ICLR 2020
Juhani Koski. Defectiveness of weighting method in multicriterion optimization of structures. Com- munications applied numerical methods, (6))3333-37, 1985.
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The variational fair 2015.
autoencoder. arXiv preprint arXiv:
Andreu Mas-Colell, Michael Dennis Whinston, Jerry R Green, et al. Microeconomic theory, vol- ume Oxford university press New York, 1995.
Kaisa Miettinen. Nonlinear multiobjective optimization, volume 12. Springer Science & Business Media, 2012.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5: 180161, 2018.
dataset, large collection
Berk Ustun, Yang Liu, and David Parkes. Fairness without harm:
Decoupled classifiers with prefer- guarantees. In International Conference Machine Learning, pp- 6373-6382, 2019.
Blake Woodworth, Suriya Gunasekar, Mesrob Ohannessian, and Nathan Srebro. Learning non- discriminatory predictors. arXiv preprint 17Xiv:1702.06081, 2017
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. Fair- ness constraints: Mechanisms for fair classification. arXiv preprint arXiv:1507.055: 05550700059 2015,
Muhammad Bilal Zafar, Isabel Valera, Manuel Rodriguez, Krishna Gummadi, and Adrian Weller. From parity to preference-based notions of fairness in classification. Advances Neural formation Processing Systems, pp. 229-239, 2017.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International Conference on Machine Learning, pp. 325-3323333333 2013.
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1 PROOFS
Here we restate the Lemmas shown in Section Sectionnalooe aaoong with sketch of the proofs.
Lemma 3.1
If h & P(H. + hh h E P(H,A) Ra (b I Ro (B Va, where ER a equality of risks classifier in such that Ro (h = max R. (h), Va and he Ro (1)r))
max R (hp),, Va.
a' d'EA
Proof. If hp dominates h + Ra Rh E = max R. (ht) S max Rh) RR( Va E A. a' EA
EA
Definition A.1. Dominant vector: A vector r' E R is said to dominate E RR noted as r' > r, if T 2 ry Vi II.... & and Rj ry > r'j (i.e., strict inequality on least one component).
Definition A.2. Pareto field: A function P Ω + IR is defined as a Pareto field over a convex set Q OCR' R if P E C¹ is continuously differentiable function such that v,P(r) > O'r 92 Vi 1....,k. k.
Lemma A.1. Let S2 C R* be a convex set, and P: Q + IR a Pareto field. Then the set D {r e Q: P(r)= 0} is proper Pareto set in Ω and the set D {r E EQ: P(r) 0} is the set of dominated points, ie., D = (r Q: Br' E eD(r r'}.
Proof. First prove that D f Ω P(r) 0} is Pareto set. Assume by contradiction
that there exists r, r' E D r' > r, then r c = E, deeq 8, 2 OVi E (I,...;ke k), Fj e (1..... b} 83 > 0. is standard basis vector.
Using the Gradient theorem we have
P(r') = P(r) + [ VP(A)dA,
-P(r) + S/ (0)....,))).(V1)))) -c)) Ac),...,&PPP(( Ac)))da,
=P(r) + Σd, 6, / Vr
>0
> P(r):
which directly contradicts r',r E D. No point in set D is dominated by another point in set D, making set D proper Pareto set.
To show that the set D is the set of dominated points, we first note that for all E Q r' E D,r > r' we have P(r)= > P(r") using the same arguments above, meaning that the set of dominated points included in D+. Similarly, for all r E Q r' E D,r 2 r we have P(r') 2 P(r). meaning that non-dominated points are not part of set D
Lemma A.2. Let Ω RIAI be convex set defined by Ω (r E RIAI (r(r) 0Vc {L...., Ch.g- continuously differentiable); let P St + IR be a convex Pareto field with corre- sponding Pareto sei D = (r { EΩ: P(r) = 0}. Let E D and Φ(r, E!AI ri+ melri 2 with r² There exists set of 4² > such that:
r arg min ((r,p") sd..: P(r) 2 0. (r) 2 OVe (1....,C}
rERIA
P(r) 2 0} are convex, so the intersection is also convex. For μ 0, ó(T,)) 4) is convex function r. Under these hypothesis, the Karush-Kuhhn- Tucker (KKT) conditions are necessary and sufficient to recover a global minimizer. So it suffices to find set 11 such that the KKT conditions are exactly satisfied r".
Proof. By hypothesis, both Ω and fr E Ω
11

---

Under review as a conference paper at ICLR 2020
Let J= {i [ hypothesis, r E D and therefore P(r*) always active. Focusing on the first KKT condition on the active set we get Vø(r", (i) (r) + ΣjeJ "i V (r*). in matrix form:
(((r)) 0} be the set of indices of active o" constraints r note that by
A
2 cc)
+
= VP Vig
V
: Ue
2( c)+
By hypothesis, we have ri c and therefore
H4 (r))(r") Ejer T3j Vgl (r*)- -1
2(r cc)
In particular, for TJ Vj and μn > (p)) 1] - -c+ Vi # we get
\P((r"
14 = [(11(r1 - +) + 1) (r:(r*)
(r((r*) 2 -c}+ > Ovi F..
111 > O,
À W1 (12 - c) +1 > 0,
(r*)
which satisfies the KKT conditions and produces the required minimizer r" for all points r E D.
Lemma A.3. Let Ω C R² be convex set defined by Ω = fr E R² : g 2 04c {1...., ,,,, of continuously differentiable): let P : Ω - I be a convex Pareto field with corre-
sponding Pareto set D (r E Ω P(r) 0}. Let > 0. r" = arg min |max; Tq miny ry] rED
then
min(r1, r2) min(ri, V(r1,2)) D.
Proof. case ri = -r:: Assume r E D minar; ri > ri. Then r r", which contradicts the hypothesis r E D.
case rf # rš: Without loss of generality, we take ri > r2. We can then prove by couterpositive
that r E implies v1 > r2- Assume by contrapositive that there exists r' D r rí. By
hypothesis D is continuous curve, and we can find path R(()= R(O) r*, R(1) = =r', R(()) E
DV' E [0, 1). which would imply there exists & R(f) = (f, f), in which case rf = r.. which contradicts ri + r2.
We then have that for r E D. ri > T2. Assume by counterpositive that T2 = min(r), r2) >
min(rī, r5) = r then 71 < rī (since both r, r.r D), but then |r] r21 Irf r which contradicts the hypothesis.
Putting both statements together, we proved that if rī > r.. then Vr D, min(r,) = 12 < min(ri,7)) = 1².
Lemma A.A. Let Ω C R2 be a convex set defined by = fr E R² : gr(r) 2 Ove e (1..... Gt.g continuously differentiable): let P : Ω - IR be a convex Pareto field with corre-
sponding Pareto set D {r E P(r)= = 0}. Let μ (21, O. μ³¹ (H1,ph); plz p2
Define @rr, c) = 2i Ti pr - c)2 and
arg min o(r, p),
s.1. P())20 reft
r' arg min o(r nl, c).
I. P(r) reB
With c T2. then, r. rı, r4 S r2, with equality and only ff
arg min T2 (P
12

---

Under review as a conference paper at ICLR 2020
Proof. By hypothesis, we have
o(r, 2 H',c)
((r,11,) (122 (2) -) 2 olr (122 122((1 c)2+
(rr,p,)) 2 o(r', (. (4/ -22--22l(15 - -c)2t -
12)l(T2
since by hypothesis @(r, ur e) S c(r' ,,, c), it follows r/ S 12, and therefore ri 2 rı, because they both belong to the Pareto set.
Case r' r. To analyze when the equality arises, note that the tangent to D is T(r) -V
((r) v P(r)) and that r + 57(r,, 5% > 0 valid search direction if r 7 (r does not minimize r2). Also note that Vø(r,
Vø(r, ,r, (0,2(14 p2) c)+)
By contradiction, assume r * A. then r + (T(r), ((87(r)), Vo(r, > O, which implies r' * arg min Φ(T, pl, and contradicts the hypoth- s.t. P())20 refl
0 a valid search direction and
esis.
Therefore, ifr'=>> r it follows = =r=-
Lemma A.5. Let Ω C RIAI be a convex set defined by S = fo RIAI gg(rr) 2 0Vc E {1...., Of continuously differentiable): let P D R be convex Pareto field with corre- sponding Pareto sei fr
P(r) = 0}. Lel r" arg min(max, v, min, ry) and
rED
Φ(r, (c. E r p:(r--r000 - -c)²+. Let nh > 0 be an initial penalty vector, r arg min E, To
rED
<> 0, e min; r. 67 > 5 E (0, 1). Define the following auxiliary variables ++ min r
r max rf min
at arg max(ra)
11 (7 1)
1 at = phat
(2)
r arg min Φlr, (rr, c(t))
s.t. P())()
rell
minr" r!
r max r!
min m)
We define the following iteration procedure
compute Eq
* i
terminate
While 1t
1
7 + Et
Recompute Eq
End While
u k+1 wl (rt-ch )+
(rt ct)
k+1 1 rl
The procedure is convergent to r* for [A| =2
13

---

Under review as a conference paper at ICLR 2020
Proof. From Lemma A.3 we have ck min; r. vk, we can then apply Lemma A.2 and state that there exists set 1u** such that r" arg min p(r, ,et).
P(r220
We split the proof in two scenarios, one where D and then 1 > 0, since they show different convergence properties.
n 0. Assume without loss of generality that rf arg max; r/, from Lemma A.4 have uf < ju] and further, for all u E (uf, pf*), we have rf r < ri. Since | and p* .r, rM
all belong to D, we also have r > 12 >> r2. which implies 1* > 1 > | = 0. The subiteration procedure is always successful, and we get sequence of iterates such that I > p strictly
decreasing sequence bounded below by 0, which implies asymptotic convergence to r". Procedure never terminates with probability (event μa E ue has probability 0).
case r*
O Assume without loss of generality that < r2. Following the arguments from Lemma we observe that < v(ri, r2) D. In these conditions, we have that d
15 X+1 and
A.3
et. We will re-derive the KKT conditions in this scenario show that convergence is guaranteed for μ > j12*, and that this j22* is independent of c², meaning that convergence to the optimal 1" can be achieved in finite number of steps.
Let VP(r") (dP1, dP2) O, with corresponding tangent vector x" (dP2, -dPa), for r* to be the risk with minimal gap, r" + 5r must be an infeasible descent direction for å > 0. Therefore, there is an active constraint from Q g(r), with Vg(r*) (dgı dg2) such that ((dg), dg2), (dP2,- )d < 0.
Deriving the first KKT conditions, get
1 1 - c) = =AdP; + ndgi,,
+ 122(ri - c) = =AdP2 + 1ıdg2,
and in terms of X, 7)
x 1+ 212(1) c) - ndg1
dP1
7 = dP 222((r - c) dd2(1 p:2(1 -c)
dg2dPi - dP2dgi
Note that dggdP; dP2dgi > from ((dg), dg2), (dP2, -dP:)) 0. To recover a valid set of
Lagrange multipliers À > 0, suffices to have
dP (1 + p22(r2 c)) > dP2(1 + 21 c)),
(1+ 212(1i - c)) dB
27.
112
2(r c)
(1 + 112(112 - c²)) aP.
11
where we used that d < c < e Vk. Therefore, for terminates.
H9 large enough, n = r" and the algorithm ae
A.2
ALGORITHMIC DETAILS
Here we provide details on how we optimize our adaptive loss in-between penalty update (paz) steps, shown in Algorithm
A.3
METHODS
We compare the performance of the following methods:
14

---

Under review as a conference paper at ICLR 2020
Algorithm 2: AdaptiveOptimize
Given: ho, T, c,, D"r Dival Mp, B
e 0, 6- +- 00
ht + he
while S mp do
// Making loss progress
for (1. E D
// Run one epoch of SGD on training set La = E (I....., B) a; a), (ho) = +.a E ((ho(z.). // Empirical risks (ho) =ooeg Rg + Na (a( cc
do
c 0 lr Vvo ( (ho)
// Gradient step 11 Evaluate improvement on Val and update target risks
end
0) (ha) 0" then
h- + ho:
0² t (ho); p 0
else
p p+ 1;
end
n (tt)
end
Return: 11", 12aa
Kamishima. Kamishima et al. (2012) uses logistic regression as a baseline classifier, and requires numerical input (observations), and binary sensitive attribute and target variable. Fairness con- trolled via regularization term with a tuning parameter that controls trade-off between fairness and overall accuracy. 7 is optimized via grid search with r) E (0, 300) as in the original paper. We report results on the hyperparameter configuration that produces the smallest accuracy disparity between sensitive subgroups.
Feldman. Feldman et al. (2015) provides a preprocessing algorithm to sanitize input observations.
It modifies each input attribute so that the marginal distribution of each coordinate is independent on the sensitive attribute. The degree to which these marginal distributions match is controlled by À parameter between 0 and 1. It can handle numerical and categorical observations, non-binary sensitive attributes, and arbitrary target variables. Following Friedler et al. (219)). we train a linear logistic regressor on top of the sanitized attributes. À is optimized via grid search with increments of 0.05. We report results on the hyperparameter configuration that produces the smallest accuracy disparity between sensitive subgroups.
well as
Zafar. Zafar et 2017) Addresses disparate mistreatment via convex relaxation. Specifically, in the implementation provided in |Fieddeer et al. (2019), they train a logistic regression classifier with fairness constraint that minimizes the covariance between the sensitive attribute and the clas- sifier decision boundary. This algorithm can handle categorical sensitive attributes and binary target variables, and numerical observations. The maximum admissible covariance is handled by hyper- parameter c, tuned by logarithmic gridsearch with values between 0.001 and 1. We report results on the hyperparameter configuration that produces the smallest accuracy disparity between sensitive subgroups.
algorithm that takes in an arbitrary predictor and the sensitive attribute as input, and produces a new, fair predictor that satisfies equalized odds. This algorithm can handle binary target variables, an arbitrary number of sensitive attributes, and any baseline predictor, but requires test-time access to sensitive attributes. does contain any tuning parameter. We apply this method on top of both the Naive Classifier and our Pareto Fair classifier.
Hardt. Hardt et al. (016)) proposes a
Naive Classifier (Naive). Standard classifier, trained to minimize an expected risk h = arg min (xx,A,(((((X)) Y)J. The baseline classifier class H. is implemented as neural network NEW and varies by experiment as described in Section A.6, the loss function also varies by experiment and is also described in Section A.6 Optimization is done via stochastic gradient descent.
15

---

Under review as a conference paper at ICLR 2020
Naive Balanced (NaiveB). Baseline classifier designed to address undersampling of minority classes, trained to mimimize a class-rebalanced expected risk h arg PAA. (L((X()), Y)].
Like the Naive classifier, it is imple-
kEW mented as a neural network and optimized via stochastic gradient descent. The sole difference with the Naive classifier is that, during training, training samples are drawn from the new input distribution A U |Al:: XIA ~ P(X,Y|A), which is achieved by re-weighted sampling of the original training dataset.
Pareto Fair. Our proposed methodology, trained to minimize an adaptive loss function using the procedure described in Algorithm [ Addresses risk disparity minimization without introducing unnecessary harm any subgroup. The baseline classifier class H. is implemented as neural network and varies by experiment as described in Section A.6 the loss function also varies by experiment and is also described in Section A.6
A.4 EVALUATION METRICS
Here we describe the metrics used to evaluate the performance of all tested methods. We are given a of test samples D, the corresponding objective. We assume that y is a finite alphabet, as in a classification problem, and we will represent the one-hot encoding of y as E. Given a trained model h X - [0. 1] the predicted output for a an input is vector h(z;) = Pi such that (M²); E [0, 11. E {1, DD and | ( (e.g.: output of softmax layer). The predicted class is ÿ = arg max, and its associated confidence is p. = max;(pi);- Ideally ý should be the same as yı- Using these definitions, compute the following metrics.
y;)) where 2i E X' realization of our model input and yi E y
Accuracy (AC): N/2 Di=l 1(y; ú:). Fraction of correct classifications in dataset.
N
Confidence (CO): N
Em fi. Average magnitude of the predicted class probability.
Brier Score (BS): N || 12.112 where & is the one-hot representation of the categorical ground truth value y;- This quantity is also known as Mean square error (MSE).
Cross-Entropy (CE): - 1/2 Σm Σ (6))glog(H)) also known as negative log- likelihoo (NLL) of the multinomial distribution.
Expected Calibration Error (ECE): N M of bins to divide the interval [o, 1] such that Bm (i (1,.., : (m X7 are the group of samples that our model assigns a confidence (p:) in the interval (mM²) 1 Ar Measures how closely
EieB., [1 (.) pill where M is the number
-
the predicted probabilities match the true base rates.
Maximum Calibration Error (MCE): ma..mm M} mm Eien, (M = ú.) - -pill. Measures
worst-case miscalibration errors.
These metries are computed independently for each sensitive subgroup on the test set and reported in Section A.7
A.5 EXPERIMENTS ON REAL DATA
The following is description of the data and experiments for each of the real datasets. The infor- mation present here summarized in Table 7
16

---

Under review as a conference paper at ICLR 2020
Outcome/ Objective
Sensitive Attribute
Dataset
Train/Val/Tess splits
Gender (F/M)
Adult Dua & Graff
Income
Ethnicitt(W/NWW)
60/20/20
5
2017a
categories
2x2 categories
German
Credit categories
Gender (F/M)
60/20/20
5
Dua & Graff
2017a
2 categories
Mortality(A/D), Age (A/S) Ethnicity (W/NW)
MIMIC-III
Mortality (A/D)
60/20/20
5
Johnson et al. (2
2 categories
2x2x2 categories
HAM1 Tschandl et al. 2018
Type of lesion 7 categories
Type of lesion
80/10/10
1
7 categories
Table 7: Basic characteristics of real datasets
This dataset consist of clinical records collected from adult ICU patients at the Beth Israel Deaconess Medical Center (MIMIC-III dataset) Johnson Jotason))ohnso et al. (2016). The goal is predicting patient mortality from clinical notes. We follow the pre-processing methodology outlined Chen et al. (2018), where we analyze clinical notes acquired during the first 48 hours of ICU admission; discharge notes where excluded, as where ICU stays under 48 hours. Tf-idf statistics on the 10, 000 most frequent words in clinical notes are taken as input features.
MIMIC-II.
We identify 8 sensitive groups the combination of age (under/over 55 years old), ethnicity as determined by the majority group (white/nonwhite) and outcome (alive/deceased). Here we will use the term adult to refer to people under 55 years old and senior otherwise. This dataset shows large sample disparities since 56.7% of corresponds the overall majority group aalivesenniorwwhit) and -adnee---dddl---nhttee)...
only 0.4% to the overall minority group ddeceased-adull nonwhite).
We used fully connected neural network as described in table 8 as the baseline classifier for our proposed Pareto Fair algorithm. We compare our results against both the Naive and Naive Balanced training loss.
algorithms using the same neural network architecture, and crossentropy (CE) We also evaluate the performance of Zafar applied on the feature embeddings learned by the Naive Balanced classifier (Results for Zafar over the original input features were not promising).
We report the performance across 5-fold split of the data, we used 60/20/20 rrii--vlldattion-ess partition as described on Table 7 and report results over the test set. We denote the overall sensitive attribute the combination of outcome A::alive/D:deceasee/ ity (W:white, NW:nonwhite) with shorthand notation of the form D/A/W to denote, for example deceased, white adult. We also note that results both Zafar and Hardt were done over only the sensitive attributes Adult/Senior and White/Nonwhite,, outcome was not considered as sensitive attribute for both methods since Hardt requires test-time access to sensitive attributes, which would not be possible in this case, and Zafar attempts to decorrelate sensitive attributes and classification decision boundaries, which is counterproductive when the sensitive attribute includes the correct decision outcome.
age (A:adult/S: senior) and ethnic-
HAM10000. This dataset contains over 10, 000 dermatoscopic images of skin Iesions over a di- verse population ['schandl et al.] (2018). Lesions are classified in 7 diagnostic categories, and the goal is learn a model capable of identifying the category from the lesion image. The dataset highly unbalanced since 8 31% of the samples correspond to a melanocytic nevi lesion (nv), and 0.5% to dermatofibroma (df).
Here we chose to use the diagnosis class as both the target and sensitive variable, casting balanced risk minimization particular use-case for Pareto fairness.
We load pre-trained 1enseNet121 network Huang et al. (2017) and train to classify skin lesions from dermatoscopic images using our Pareto fairness algorithm. We compared against the Naive and the Naive balanced training setup. Note that in the naive balance approach we use batch sampler where images from each class have the same probability, this can be seen as a naive oversampling technique. Table 8 shows the details of the implementation.
17

---

Under review as a conference paper at ICLR 2020
We used the original saai--vaiiaation-ess (80/10/10) split, and report results on the test set. Nota- tion for each group follows the original notation: Actinic keratoses and intraepithelial carcinoma / Bowen's s disease (akiec), basal cell carcinoma (bec), benign keratosis-like lesions (bkl), dermatofi- broma (df), melanoma (mel), melanocytic nevi (nv) and vascular lesions (vasc).
Adult. The Adult UCI dataset Dua & Grafi (201 is based on the 1994 U.S. Census and contains data on 32, 561 adults. The data contains 105 binarized observations representing education status, age, ethnicity, gender, and marital status and target variable indicating income status (binary at- tribute representing over or under $50, 000). Following Friedler al. (2019), we take ethnicity and gender as our target sensitive attributes, defining two subgroups (White Males and Other). We also present results taking just the gender sensitive attribute (Male/Female). To compare our Pareto Fair algorithm evenly against the other methods, we limit our hypothesis class to linear logistic regression.
German. The German credit dataset Dua & Graff (2017a) contains 20 observations collected across 1000 individuals, and a binary target variable assessing the individual's s credit score as good or bad. We take gender (Male/Female) as the sensitive attribute, which is not included in the data but can be inferred. As in the Adult dataset, we limit our hypothesis class to linear logistic regression to compare evenly across methodologies.
A.6 NEURAL ARCHITECTURES AND PARAMETERS
Table 8 summarizes network architectures and loss functions for all experiments in Section 5 Note that all networks have standard dense softmax their final layer. The training optimizer is standard ADAM/Kinggaa Ba 2014). loss functions were either crossentropy (CE) or brier score (BS), also
known categorical mean square error (MSE).
Dataset
Network Body
Gate
Loss type
Parameters
FullyConnected 64x64
batch size=32
Synthetic
ELU
BS
Ir=5e-4
Adult German
Logistic Regression (LR)
batch size=32
CE
Dua & Graff
2017a
Ir=5e-4
MIMIC-II
FullyConeected 2048x2048
batch size=512
ELU
CE/BS
Johnson et al.
2016
1r=1e-6/5c-6
HAM10000
DenseNet121 Huang al. (0117
batch size=32 1r=5e-6
ReLU
BS
Tschandl et al. (2018)
Table 8: Summaay network architectures and losses. All networks have a final softmax dense layer as the output layer. CE refers to crossentropy and BS to Brier score or categorical MSE, Training was done using ADAM optimizer with the lcarning rates (Ir) specified in the table.
A.7 SUPPLEMENTARY RESULTS
The following tables show a quantitative performance comparisons between the various fairness methods described in Section A.3 over the datasets described in Section A.5] All metrics presented are described in Section A.4 and computed per sensitive attribute. We additionally provide results for the overall mean across the dataset (sample mean), as well as results on the group- balancd and group min to denote the maximum and
mean across groups (group mean). We use group minimum metric values attained over all sensitive groups. We denote group max- group min as Discrimination, which is the worst case difference in performance across groups. When the dataset contains multiple splits, we report both split mean and split standard deviation.
18

---

Under review as a conference paper at ICLR 2020
type
ratio
Naive
NaiveB
ZafarCE
ParetoFair CE
ParetoFairr BS
A/A/NW
5.7%
86.311.5% 49.110.4%
9..0.1..4%
80.4.12.8%
83.4112.6
A/A/W
13.3%
98.%+
86.3 %
90.0=1.3%%
81.0=t1.5%
83.2+1..5%
A/S/NW
12.9%
97.5110.6%
76.5.1..7%
81.8.1..7%
69.5.12.2%%
71.4.13.0%
A/S/W
56.7%
97.9=103%%
79.0+006%
77.410.7%
1.35+
74.6+1.6%
D/A/NW
0.4%
23.4:9944%
76.14.85%%
47.71.9.6%
76.6:t:53%%
78.6:16.19
D/A/W
0.9%
32.6++3.7%
80.1+333%
60.5+6.3%
85.0++3.%%
83.3+t4.2%
D/S/NW
188%
21.42.2%%
66.91224%
48.242.0%
72.6:15.1%
73.31:29%
DIS/W
8.3%
23.4+222%
6.94+199%
57.1+2.2%
71.0++4.%%
72.5+336%
sample mean
89.5.10.2%
78.910..%
78.0.10.5%
74.31.0.%
75.711.11%
group mean
12.5%
1.5%%
7..3
6..4+133%
76.2+t0.%%
77.5 %
group
56.7%
0.33%1
88.111.3%%
93.0.114%%
86.4.12.2%
87.11.2.0%
group min
0.4%
18.14
43.7+339% 1.27%
6..9+
69.9+
disc
56.3%
2..1%
22.5+2.5%
49.31.44%%
18.51442%
1..1+2.6%%
type
ratio
NaiveB H
ZafarCE H
ParetoFair CE H
ParctoFair BS H
A/ANWW
5.7%
7..3%
68.442.2%
69.9+2.5%
71.6::3.%%
A/AW
13.3%
7..71
67.81..3%
70.8:12.4%
7,88:19%%
A/S/NW
12.9%
76.4+2.2%
67.9+2.1%
70.1+2.5%
71.3+3.1%
A/S/W
56.7%
76.2.12.2%
67.6.11.%%
70.6.1..2%
72.111.2%%
D/A/NW
0.4%
66.6+.9.%
59.7++89%
71.2+77.%%
74.1+9.2%
D/A/W
0.9%
66.4.12.4%
57.0.15.0%
72.9113.%
73.6.14.3%
DIS/NW
1.8%
64.8
55.99331%%
70.2+44.%%
71.2+t2.%
D/S/W
8.3%
66.244-2.9%
55.34
70.943.8%
72.4+33.5%
sample mean
7..1++1.8%
66.3+t.0%%
1.36+
1..9+
group mean
12.5%
7.12%
62.4:11.%%
70.8:10.7%
7..31100%
group max
56.7%
80.0++2.5%
70.0++2.6%
47.6+t4.1%
79.94-5.6%
group min
0.4%
61.4:13.4%
52.0.13.6%
6..7.1188%
6..8112.11 %
disc
56.3%
38.6%%..2%
18.14443%
5..9+
13.1
Table Accuracies on test for MIMIC dataset. Standard deviations are computed across splits.
type
ratio
Naive
NaiveB
ZafarCB
ParetoFair CE
ParetoFair BS
A/A/NW
5.7%
0.027-10.004
0.101
0.138:10.029
0.291 1..04
0.252
A/AWW
13.3%
0.027=0.005
0.196=10.013
0.240.026
0.285 0.22
0.247
A/S/NW
12.9%
0.055+00.007
0.311+0.017
0.36440.036
0.383++0.018
0.363++0.029
A/S/W
56.7%
0.049:10.003
0.28310.005
0.44910.014
0.355
0,338110.017
D/A/NW
0.4%
13007-L0.13
0.362+0.055
1.042+0.189
0.335+0.03
0.34+0.041
D/A/W
0.9%
0.922+:0.068
0.292+0.001
0.785
0.287+0.022
0.277+0.031
D/S/NW 1.8%
1.0851100019
0.41710.018
1.032.10.01
0.37110.033
0.365
D/S/W
8.3%
1.043+0.021
0.419+0.023
0.857=0.044
0,379#0.032
0.307
sample mean
0.009
0.285
0.437+0.014
0.347-0.013
0.3271
group
12.5%
0.5274.0.02
0.311:0.09
0.60810.026
0.33610.009
0.31910.002
group 56.7%
1.098+0.028
0.43150.015
1.122+0.078
0.402+-00017
0.394+-.011
group min disc
0.4%
0.025-+0.003
0.001
0.138+0.029
0.261+0.023
0.233+0.022
56.3%
1.07310.03
0.243
0.9857
0.141
0.161
Table 10: Brier Score (BS) on test set for MIMIC dataset. Standard deviations are computed across 5 splits.
19

---

Under review as a conference paper at ICLR 2020
type
ratio
Naive
NaiveB
ZafarCE
ParetoFair CE
ParetoFair BS
A/A/NW
5.7%
0.068+00005
0.329=t0.027
2.803=0.659
0.459+0.051
0.409+00003
A/A/W
13.3%
0.067-10.007
0.322-10.017
4.079-1.601
0.45100031
0.4.0.0277
A/S/NW
12.9%
0.121+0.009
0.471=0002
7.697=0.91
0,564+0.02
0.538++0.034
A/S/W
56.7%
0.111+:0003
0.434-10.007
9.238
0.531110.119
0.508:10.021
DIA/NW
0.4%
1.573+0.160
0.567=00078
21.617=3.25
0.515#0.033
0.523++0,052
DIA/W
0.9%
1.363+-0.112
0.459++0.05
17.284422502
0.46+-0.023
0.445+000033
DIS/WW
1.8%
1.58+0.046
0.60++0.027
22.097+0.961
0.552++0.037
0.544+00.037
DIS/W 8.3%
1.532:10.046
0.60910.033
18.446.00.9
0.563110.036
0.547110.41
sample mean
0.266+0.005
0.437++0.007
9.113=0.4
0.522+00018
0.495+0.0117
group mean
12.5%
0.802-L0.032
0.474L100009
12.208-10.33
0.512110011
0.4891100004
group max
56.7%
1.65
0.635+0.014
23.343=11058
0.587++00119
0.578+0.019
group min disc
0.4%
0.065-L0.005
0.313+L0.021
2.803-00659
0.426:10.031
0.388:10.032
56.3%
1.585++0.059
0.323+0.03
20.54+1.088 1.089
0.162+0.025
0.191+0.033
Table 11: Cross Entropy (CE) test set for MIMIC dataset. Standard deviations computed across 5 splits.
type
ratio
Naive
NaiveB
ZafarCE
ParetoFair CE
ParetoFair BS
A/A/NW
5.7%
0.046++0.004
0.063++0.007
0.069+0.014
0.109+0.016
0.101+0.000
A/A/W
13.3%
0.043110.002
0.05110.006
0.110.013
0.09211.027
0.08310.015
A/S/NW
12.9%
0.068-10.007
0.033110.007
0.182:10.018
0.04510.017
0.03810.01
A/S/W
56.7%
0.065++0.002
0.027+0.006
0.225=0.007
0.052#0.022
0.041
D/A/NW
0.4%
0.55:10.094
0.164
0.522.10095S
0.149110.077
0.146.10.06
DIA/W
0.9%
0.43=100057
0.09+10.03
0.39440.061
0.155=00057
0.123++00033
DIS/NW
1.8%
0.57=000017
0.008
0.516+0.021
0.09-10.029
0.07+
D/S/W
8.3%
0.535100.03
0.048:0.012
0.429=0.022
0.061
0,052100026
sample mean
0.115400004
0.03710.003
0.21940.007
0.06210.021
0.05210.012
group mean
12.5%
0.201
0.068+00006
0.305+0.013
0.094+0.023
0.082+0.021
group 56.7%
0.593110.028
0.166.10.047
0.56110.04
0.172.10.05
0.156.10055
group 0.4%
0.042++0.002
0.025+:0003
0.069#0.014
0.039+0.013
0.032+0.011
disc 56.3%
0.552-10.028
0.1411100045
0.492.0.005
0.1033
0.125
Table 12: Expected calibration error (ECE) on test set for MIMIC dataset. Standard deviations are computed across splits.
type
ratio
Naive
NaiveB
ZafarCE
ParctoFair CE
ParetoFair BS
A/A/WW
5.7%
0.322:10.090
0.125:4.02
0.425+0.301
0.147:1.017
0.166:10.016
A/A/W
13.3%
0.2511-0.061
0.088+000115
0.229-+0.133
0.127+0.026
0.124+40.019
A/S/NW
12.9%
0.223110.08
0.05810.013
0.32310.198
0.07710.03
0,0710.013
A/S/W
56.7%
0.203+-0.007
0.046+0.001
0.496+0.184
0.091+0.033
0.069+0.02
DIA/NW
0.4%
0.828++0.115
0.341+0.094
0.522+0.095
0.293
0.422+0.266
D/A/W
0.9%
0.879110.07
0.18311.074
0.50911.2001
0.206.10.033
0.19120.002
D/S/WW
1.8%
0.822+00052
0.158+0.082
0.553+0.067
0.152+0.027
0.129+0.029
D/S/W
8.3%
0.865410.027
0.11410.038
0.563-1.112
0.136:10.044
0.112-10.041
sample mean
0.294::0001
0.06710.007
0.441#0.113
0,10441..02
0.089:100115
group mean
12.5%
0.549--0.031
0.139
0.453+0.094
0.15440.024
0.16++0.447
group max
56.7%
0.914110.05
0.364110001
0.710.105
0.293-100088
0.42910.259
group min
0.4%
0.196.10.007
0.042
0.146110.0
0.07110.025
0,059100013
disc
56.3%
0.719+00044
0.554+0.097 0.322+0.065
0.222++0002
0.37-
Table 13: Maximum calibration error (MCE) on test set for MIMIC dataset. Standard deviations are computed across 5 splits.
20

---

Under review as a conference paper at ICLR 2020
Groups
ratio
ParetoFair
NaiveB
Naive
ParetoFair
NaiveB
Naive
akiec
2.5%
51.9%
55.6%
3.7%
55.4%
66.3%
55.3%
bec
2.7%
56.7%
76.7%
56.7%
57.6%
77.5%
66.7%
bkl
7.2%
59.5%
58.2%
36.7%
64.5%
71.6%
64.3%
df
0.5%
66.7%
33.3%
0.0%
59.8%
68.1% %
81.6%
nv
81.0%
83.5%
90.9%
96.0%
81.7%
89.1%
93.7%
vasc
1.3%
71.4%
85.7%
0.0%
66.9%
74.7%
76.
mel
4.8%
53.8%
48.1%
32.7%
63.6%
32.1%
63.6%
sample mean
78.6%
84.8%
83.6%
78.0%
85.6%
88,
group mean
14.3%
63.4%
64.1 %
32.3%
64.2%
73.6%
71.6%
group max
81.0%
83.5%
90.9%
96.0%
81.7%
89.1 1%
93.7%
group min
0.5%
51.9%
33.3%
0.0%
55.4%
66.3%
55.3%
disc
80.4%
31.7%
57.5%
96.0%
26.3%
22.8%
38.4%
(a) Accuracy
(b) Confidence
Table 14: Accuracies (left) and confidence (right) on test set for HAM10000 dataset.
type
ratio
ParetoFair
NaiveB
Naive
ParetoFair
NaiveB
Naive
akiec
2.5%
0.741
0.671
1.289
1.547
1.638
3,879
2.7%
0.549
0.341
0.613
1.095
0.678
1.172
bkl
7.2%
0.62
0.606
0.931
1.345
1.206
2.258
df
0.5%
0,536
0.898
1.721
0.989
1.674
6.486
81.0%
0.241
0.128
0.054
0.58
0.28
0.115
1.3%
0.36
0.246
1.604
0.747
0.499
4.564
mel
4.8%
0.586
0.657
0.941
1.213
1.28
2.137
sample
0.308
0.213
0.234
0.708
0.449
0.579
group mean
14.3%
0.519
0.507
1.074
1.037
2.944
group max
81.0%
0.741
0.898
1.721
1.547
1.674
6.486
group min
0.5%
0.241
0.128
0.054
0.58
0.28
0.115
disc
80.4%
0.501
0.769
1.667
0.967
1.394
6.371
(a) Brier Score
(b) Cross Entropy
Table 16: Brier Score (left) and Cross Entropy (right) set for HAM10000 dataset.
type
ratio
ParetoFair
NaiveB
Naive
ParetoFair
NaiveB
Naive
akiec
2.5%
0.249
0.254
0.516
0.754
0.447
0.966
bec
2.7%
0.147
0.134
0.161
0.269
0.408
0.418
bkl
7.2%
0.171
0.156
0.304
0.731
0.294
0.744
df
0.5%
0.452
0.57
0.816
0.718
0.853
0.985
nv
81.0%
0.025
0.034
0.032
0.143
0.248
0.273
vasc
1.3%
0.256
0.202
0.761
0.557
0.539
0,956
mel
4.8%
0.14
0.212
0.323
0.277
0.289
0.836
sample mean
0.055
0.065
0.095
0.219
0.27
0.367
group mean
14.3%
0.206
0.223
0.416
0.493
0.44
0.74
group max
81.0%
0.452
0.57
0.816
0.754
0.853
0.985
group min
0.5%
0.025
0.034
0.032
0.143
0.248
0.273
disc
80.4%
0.426
0.535
0.784
0.611
0.605
0.712
ECE
(b) MCE
Table 18: Expected Calibration Error (left) and Maximum Calibration Error (right) on test set for HAM10000 dataset.
21

---

Under review as a conference paper at ICLR 2020
type
ratio
Naive
NaiveB
Feldman
Kamishima
Zafar
ParetoFair LR
Male
67.9%
92.3++0.%% 92.3++04%
92.4+003%
92.6+t0.%%
92.2+007%%
92.4++0.%
Female
32.1%
80.5.10.4%
80.3.10.%%
80.8.10.%%
80.9100.%
80.8.10.1%
81.0.10.5%
sample mean
84.3+..3%
84.2:t0.5%
84.5+0.3%
84.6+0.3%
84.4+t0.%
84.7+t0.3%
group mean
50.0%
86.4:10.2%
86.3:10.4%
86.6:10.3%
86.7.10.2%
86.5:100.%
86.7:10.2%%
disc
35.9%
11.9:0.%%
12.0::0.%%
11.6:t0.2%
11.7:t0.%%
11.4:0.7%
11.4:t0.%
type
ratio
Naive
NaiveB
Feldman
Kamishima
Zafar
ParetoFair LR
White Male
39.7%
90.8:10.3%
90.8:10.3%
90.6:10.3%
90.4-11.1%
90.5:10.6%
90.7.L0.3%%
Other
60.3%
79.7+0.7% 79.%%
80.410.6%
79.4+2.3%
80.1+0.1%
80.6:t0.6%
sample mean
84.1%
84.1+0.4%
84.5++03%
83.8++18%%
84.2++0.2%
84.6++0.4%
group mean
50.0%
85.3++.119 %
85.2++03%%
85.5++03%
84.9+1.6%%
85.3++0.%%
85.6++0.4%
disc
20.5%
11.0.1..%%
11.111.8%
10.2:10.7%
11.0.1.4%
10.3:10.6%
10.2:10.6%
Table 20: Accuracies on test set for Adult dataset. Standard deviations are computed across 5 splits.
type
ratio
Naive
NaiveB
Feld
Kam
Zafar
ParetoFair LR
Male
32.1%
0.116.10.004
0.117:10.005
0.115:10.044
0.11110.005
0.115.10.013
0.114110.003
Female
67.9%
0.268=0.004
0.272::0.007
0.263::0.055
0.258=0.003
0.26=0.003
0.261=0.006
sample mean
0.22:10.003
0.202:10.006
0.215:10.004
0.21110.003
0.21410.006
0.214:10.004
group mean
50.0%
0.192:#0.00 0.003
0.19440.006
0.189=00.00
0.184=0.003
0,188
0,188=0.000
disc
35.9%
0.152=00005
0.15540.005
0.14810.005
0.147=0.005
0.144:0.011
0.147=0.007
type
ratio
Naive
NaiveB
Feld
Kam
Zafar
ParetoFair LR
White Male
39.7%
0.139=00.033
0.137::0.004
0.135::0.004
0.14=:0.2
0.13810.013
0.133
Other
60.3%
0.28=10.00
0.28110008
0.271=10.005
0.284=0.038
0.26911.000
0.267+0.007
sample
0.224=00.002
0.223+-0.005
0.217+0.003
0.227=0.031
0.217+0.006
0.204+0.000
group mean disc
50.0%
0.21:10.002
0.208110.005
0.203310.003
0.212-100299
0.20411.007
0.2:10.005
20.5%
0. 0.142:00.007
0.143++0.007
0.136::0.007
0.144
0.131#0.012
0.133++0.008
Table 21: Brier Score (BS) on test set for Adult dataset. Standard deviations are computed across splits.
type
ratio
Naive
NaiveB
Feld
Kam
Zafar
ParetoFair LR
Male
67.9%
0.20410.000
0.20410.001
0.2.10.005
0.189-100006
0.201
0.20710.011
Female
32.1%
0.401
0.41640.011
0.402+0.007
0.395=0.004
0.406:#000
0.41
sample mean
0.345:10.006
0.348-1.001
0.337-10.005
0.329=0.004
0.34=0.01
0.345
group mean
50.0%
0.308=0.007
0.31100011
0.30110.000
0.292=20.04
0.303440011
0,309110:113
disc
35.9%
0.207-00.007
0.201400.00
0.203-40.008
0.206
0.206+0.027
0.202+00.01
type
ratio
Naive
NaiveB
Feld
Kam
Zafar
ParetoFair LR
White Male
39.7%
0.238110.005
0.23310.008
0.227=10.006
0.25410.075
0.237+0.022
0.233:10013
Other
60.3%
0.408
0..01
0.413+40.007
0.463+0.116
0.418+0.007
0.415+0.015
sample
0.352-10.004
0.3510.009
0.339-10.005
0.38110.999
0.34610.0009
0.343110.013
group
50.0%
0.333110.003
0.3310.008
0.32-10.004
0.359-0.095
0.32410.013 0.32710.011
disc
20.5%
0.1999.0009
0.194+0.0009
0.1874+0.01
0.209=0.042
0.181
0.183+00013
Table 22: Cross Entropy (CE) on test set for Adult dataset. Standard deviations are computed across 5 splits.
22

---

Under review as a conference paper at ICLR 2020
type
ratio
Naive
NaiveB
Feld
Kam
Zafar
ParetoFair LR
Male
67.9%
0.026=100008
0.023110.007
0.024:10.003
0.011=0.003
0.02110.023
0.01710.007
Female
32.1%
0.0134+0004
0.014+000005
0.01=00004
0.012=00002
0.014+0.005
0.013+0.004
sample mean
0.017:10005
0.017110.005
0.014110.003
0.01110.002
0.01611.011
0.014:10.003
group mean
50.0%
0.019=0.006
0.019:10.005
0.017=:00002
0.011=0.002
0.018#0.014
0.015=#00003
disc
35.9%
0.013.10.004
0.01100006
0.014.10.006
0.003
0.01110.016
0.0091100005
type
ratio
Naive
NaiveB
Feld
Kam
Zafar
ParetoFair LR
White Male
39.7%
0.026= 0006
0.024#0.008
0.016=0.012 0.019#0.002
0.026#0.02
0.019
Other
60.3%
0.0181000005
0.016.10.004
0.014110005
0.021
0.0191100008
0.01511.004
sample mean
0.021
0.019=:0.006
0.016::0.003
0.019=00018
0.022#00013
0.017=0.002
group disc
50.0%
0.022=10.004
0.02=00006
0.016:10.003
0.019=0.017
0.023+0.014
0.017=0.003
20.5%
0.008+00.005
0.007++0.005
0.007++0.003
0.006+0.01
0.01=0.009
0.008+
Table 23: Expected calibration error (ECE) on test set for Adult dataset. Standard deviations are computed across 5 splits.
type
ratio
Naive
NaiveB
Feld
Kam
Zafar
ParetoFair LR
Male
67.9%
0.064=0.012
0.065::0.034
0.064:10.017
0.072=0.01
0.077+0.03
0.062+0.022
Female
32.1%
0.027-100133
0.0311:00009
0.03+10.01
0.031=0.008
0.035+1.0008
0.024+10.008
sample mean
0.039:10.099
0.04211.016
0.0411:00006
0.044=0.008
0.04910.011
0.03611000
group mean
50.0%
0.045+000088
0.048-00.021
0.047-10.008
0.052-00.099
0.056+0.0116
0.043++0.013
disc
35.9%
0.037100019
0.03410.027
0.034.10.024
0.041:0.004
0.042:10.03
0.03811002
type
ratio
Naive
NaiveB
Feld
Kam
Zafar
ParetoFair LR
White Male
39.7%
0.043+000088
0.053+-0.015
0.045++0.008
0.061
0.08=0.033
0.052+00014
Other
60.3%
0.05110011
0.035110.007
0.035:10.019
0.03910.034
0.044:10.01
0.03211.011
sample mean group mean
0.047=0.001
0.042++0.01
0.039+00001
0.048=0.025
0.058+0.017
0.04+00009
50.0%
0.047=0.011
0.044400011
0.04+00009
0.05+0.023
0.062++0.2
0.042+0.000
disc
20.5%
0.016
0.01710.009
0.023110009
0.032-1.02
0.037:10.029
0.023
Table 24: Maximum calibration enror (MCE) on test set for Adult dataset. Standard deviations are computed across 5 splits.
23

---

Under review as a conference paper at ICLR 2020
A.8 ANALYSIS OF PARETO-OPTIMAL CLASSIFIERS THE INFINITE DATA AND MODEL CAPACITY REGIME
In this section we analyze the form of Pareto-optimal solutions to classification (and regression) tasks in the asymptotically ideal case where we have infinite capacity hypothesis classes, that is, our hypothesis class H. contains every function mapping points from the observation space X' to the classification or regression space R!M?
We additionally assume that the joint distributions between target variables Y and observation vari- ables X given every sensitive attribute are known (i.c., P(Y,X|A) is known), and that the loss function L(h(x),,) y) is convex with respect to h(x).
Any joint loss of the form
Op ({Ro(h))) [M Σ (h)
aEA
produces solutions which are already in the Pareto front of (Ra for any distribution of A PA- Since the loss function is convex with respect to h(() and the hypothesis class is complete, it can also be shown that for any point the Pareto front, there exists value of PA such that that point reached by minimizing Φra RR (h))) Geoffrion (968); Koski (1985); Miettinen (2012)
We therefore analyze the Bayes-optimal classifier hrg which minimizes the Naive risk Sr. (f R. (R)()))) (h))), and analytically compute the sub-population risks Ra (hpx) induced. In general, we can write
E[Y[æ]]
E [
(M
Ploaass
(4)
hp (X)
= arg min E[ Y)[X],
(hpx)) = (Lhp,, (X), Y) a).
and in the particular case where target variable Y is categorical, and the classifier loss is L2 loss against the one-hot encoding of variable Y the equations reduce to
Σ P( aEd
[|
Σ P(ala)P2
aEA
(5)
hr, (X)
= =[]]])]
Ra(ha))
= Exx P(yla, Ey (hp, (X) - 8ly s/1)²11.
24


Reviews:
[{"review_id": "rygZv8oIqS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the notion of \"no-harm\" group fairness, i.e. trying to reduce the risk gap between minority and majority groups without excessive reduction in performance on the majority groups. Authors formalize the problem by defining a Pareto fair classifier, i.e. one that minimizes the risk gaps between groups and belongs to the family of Pareto classifiers containing the classifier minimizing the empirical risk. Authors suggest an optimization procedure for finding the Pareto fair classifier and demonstrate its performance on multiple datasets.\n\nPros:\nI think that studying \"no-harm\" classifiers is an important topic given the alarming tendency of some of the recent group fairness approaches to achieve fairness by essentially driving down the performance on the majority group without improving on the minority group. Decision making in medical applications is one of the prominent examples where \"no-harm\" is absolutely needed, as authors suggested. The mathematical formulation of the problem around the notion of Pareto optimality also seems reasonable.\n\nCons:\nMy concerns are related to counter-intuitive experimental results and lack of clarity in parts of the presentation.\n\nFigure 1 seems important for understanding the ideas in the paper, but is not explained in much detail. Analogous to it Figure 2 is lacking important details. In the upper left plot, what are the decision boundaries of the baselines? What are the baselines risks in the center top figure, particularly for the equal risk classifier? It is hard to see from the right figure if the proposed classifier achieves the \"no-harm\" fairness over the equal risk classifier - numerical summary in a table could help. Finally, why is it necessary to use a neural network (which seems to be the case based on the supplement A.3) for the toy problem? I would recommend working through a toy example in more detail using a linear classifier to verify the correctness of the proposed technique and improve the overall clarity. Further, absence of a toy problem with linear classifier is alarming given there is not much discussion of the algorithm and its convergence properties.\n\nRegarding the real data experiments, none seem to showcase the \"no-harm\" versus \"zero-gap\" fairness tradeoff motivating this paper.\n\nMIMIC-III results seem to contradict the main story of the paper. The minority group appears to be \"D/A/NW\", then there should be a \"harming\" group fairness classifier achieving close to 0 discrimination at the cost of lowering performance on other subgroups. The \"no-harm\" classifier should then achieve a similar or slightly lower performance on \"D/A/NW\", but much better results on other sub-groups. Despite, the \"no-harm\" classifiers seems to outperform other approaches on \"D/A/NW\" by a good margin. Next, it appears that \"Naive+Zafar\" (it also would be helpful to have a brief discussion of the baselines considered) approach was not configured to eliminate A/S disparity as suggested by poor results on the D/A/NW and D/A/W, while it performs very well on other subpopulations.\n\nSkin Lesion classification experiment departs from the problem of fairness and considers the problem of classification with unbalanced classes instead. Results are again counterintuitive. \"Rebalanced Naive\" only mildly improves over the \"Naive\" approach, while proposed algorithm seems to achieve a quite significant mean accuracy improvement. This again does not show the motivating \"harm\" vs \"zero gap\" tradeoff, but it could be interesting as the imbalanced classification is an important problem by itself. Could you please compare to more advanced imbalanced classification algorithms?\n\nResults on the Adult and German Credit datasets are very similar across competing methods.\n\nAcknowledgement and references to the individual fairness line of works are missing when presenting the problem of fairness in machine learning.\nFont size in tables and Figure 2 legends is too small.\nTypo in the last sentence on page 8: \"have highly impactful\" -> \"are highly impactful\".", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rke5tXpptH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new kind of algorithmic fairness framework where the focus is on first finding a fair classifier that does \"no harm\" and then in a subsequent step potentially allow doing harm in order to achieve even fairer outcomes. Fairness is here understood as risk disparity: how different are the risks achieved by our model in the various subgroups. The risk is task-dependent and can be something like a cross-entropy loss for classification problems. The goal is to have similar risks in the subgroups that correspond to sensitive attributes.\n\nIt is often impossible to have equal risks without doing some harm because some subgroups might have higher noise-levels or fewer samples so that it is fundamentally not possible to achieve low risks in these subgroups. The only way then to make risks equal is by *increasing* the risk in all the other subgroups. This is not always desirable, so this paper presents a method for finding a model that is as fair as possible without doing harm.\n\n---\n\nThe basic idea of this paper is solid, but the mathematical definitions don't seem to capture that idea. Definitions 3.1, 3.2 and 3.3 define together the \"optimal Pareto-fair classifier\". But this doesn't seem to correspond to what was described before.\n\nHere is an example to show what I mean:\n\nSay, we have two subgroups: $a=0$ and $a=1$ and the risk is a binary classification loss. Furthermore, we have two classifiers $h_1$ and $h_2$. Now, say the achieved risk is such that $h_1$ achieves 80% accuracy on $a=0$ and 30% accuracy on $a=1$ (to make it precise it should be classification loss instead of accuracy but those two should be basically equivalent); classifier $h_2$ achieves 60% on $a=0$ and 31% on $a=1$.\n\nAccording to definition 3.1, neither of them dominates the other. So they could both be in the Pareto front. But classifier $h_2$ clearly does harm to $a=0$. And furthermore, definition 3.3 will choose $h_2$ over $h_1$ as the optimal classifier as the gap is smaller with $h_2$. So how can you claim that the optimal Pareto-fair classifier does no harm?\n\nNow, the classifier that you train in the end is actually from a much smaller subset that happens to be in the Pareto front: it is one that minimizes the overall risk (Lemma 3.2) and this subset might really do no harm, but that is still not obvious to me.\n\nAnother problem that I see is that the proof for Lemma 3.1 is not constructive, so while there might exist a classifier $h_p$ that dominates $h$, you might not be able to find it; and just using $h$ might turn out to be a reasonable choice.\n\nMinor comments:\n\n- the plots don't seem to be vector graphics", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxXAWmVqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# 1403\n\nGeneral\n\nThe paper proposes a method to achieve the no-harm fair model that will lie on the Pareto-optimal front, but has the minimum risk discrimination gap. While the problem formulation is interesting, the paper is not very easy to follow and there are some aspects that the paper needs to get improved to get published, in my opinion. \n\nCon & Questions:\nThe notations for the mathematical expressions are not very clearly explained. I think it causes unnecessary confusions. \nDeferring the main algorithm to the Supplementary seems weird. I think the algorithm pseudo-code should be included in the main paper so that the full method can be appreciated properly. \nThe quality of the figures is very poor. Fonts are too small and hard to read. \nTable 1 is  confusing. The numbers it presenting is not clearly described. In the caption, it says they are cross-entropy risk / accuracies, but the caption also says Pareto-Fair achieves the lowest mean risk and risk disparity. Where is the information about the risk disparity? What exactly are the number for the row Discrimination? ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript outlines procedures to address fairness as measured by disparity in risk across groups. The manuscript is primarily motivated by methods that can achieve "no-harm" fairness, i.e., achieving fairness without increasing the risk in subgroups.

The reviewers and AC agree that the problem studied is timely and interesting. However, in reviews and discussion, the reviewers noted issues with clarity of the presentation, and sufficient justification of the results. The consensus was that the manuscript in its current state is borderline, and would have to be significantly improved in terms of clarity of the discussion, and possibly improved methods that result in more convincing results.

Decision: Reject