Title: PARETO OPTIMALITY IN NO-HARM FAIRNESS

Abstract: Common fairness definitions in machine learning focus on balancing various notions of disparity and utility. In this work we study fairness in the context of risk disparity among sub-populations. We introduce the framework of Pareto-optimal fairness, where the goal of reducing risk disparity gaps is secondary only to the principle of not doing unnecessary harm, a concept that is especially applicable to high-stakes domains such as healthcare. We provide analysis and methodology to obtain maximally-fair no-unnecessary-harm classifiers on finite datasets. We argue that even in domains where fairness at cost is required, no-unnecesary-harm fairness can prove to be the optimal first step. This same methodology can also be applied to any unbalanced classification task, where we want to dynamically equalize the misclassification risks across outcomes without degrading overall performance any more than strictly necessary. We test the proposed methodology on real case-studies of predicting income, ICU patient mortality, classifying skin lesions from images, and assessing credit risk, demonstrating how the proposed framework compares favorably to other traditional approaches.

1. INTRODUCTION:

Machine learning algorithms play an important role in decision making in society. When these algorithms are used to make high-impact decisions such as hiring, credit-lending, predicting mortality for intensive care unit patients, or classifying benign/malign skin lesions, it is paramount to guarantee that these decisions are both accurate and unbiased with respect to sensitive attributes such as gender or ethnicity. A model that is trained naively may not have these properties by default; see, for example Barocas & Selbst (2016) .

In these critical applications, it is desirable to impose some fairness criteria. Much of the fairness in machine learning literature attempts to produce algorithms that satisfy Demographic Parity, which aims to make algorithm's predictions independent of the sensitive populations (Louizos et al. (2015) ; Zemel et al. (2013) ; Feldman et al. (2015) ); or Equality of Odds or Equality of Opportunity, which aims to produce predictions that are independent of the sensitive attributes given the ground truth (Hardt et al. (2016) ; Woodworth et al. (2017) ). Notions of Individual Fairness have also been advanced (Dwork et al. (2012) ; Joseph et al. (2016) ; Zemel et al. (2013) ). These notions of fairness can be appropriate in many scenarios, but in domains where quality of service is paramount, such as healthcare, we argue that it is necessary to strive for models that are as close to fair as possible without introducing any unnecessary harm to any subgroup (Ustun et al. (2019) ). Even if the overall fairness goal is a potentially harmful, zero-gap classifier, pursuing first a Pareto-fair classifier and later applying other harmful methodologies ensures that all possible non-harmful trade-offs are covered before explicitly degrading performance, therefore minimal harm is introduced to the decision.

In this work we make use of the concept of Pareto optimality to measure and analyze discrimination (unfairness) in terms of the difference in predictive risks across sub-populations defined by our sensitive attributes, a fairness metric that has been explored in other recent works such as Calders & Verwer (2010) ; Dwork et al. (2012) ; Feldman et al. (2015) ; Chen et al. (2018) ; Ustun et al. (2019) . We examine the subset of models from our hypothesis class that have the best trade-offs between sub-population risks, and select from this set the one with the smallest risk disparity gap. This is in direct contrast to common post-hoc correction methods like the ones proposed in Hardt et al. (2016) ; Woodworth et al. (2017) , where noise is potentially added to the decisions of the best performing sub-population. While this latter type of approach diminishes the risk-disparity gap, it does so by degrading performance on advantaged groups, the previously disadvantaged groups do not directly benefit from this treatment. Since our proposed methodology does not require test-time access to sensitive attributes, and can be applied to any standard classification or regression task, it can also be used to reduce risk disparity between outcomes, acting as an adaptive risk equalization loss compatible with unbalanced classification scenarios.

Main Contributions. We formalize the notion of no-harm risk fairness1 using Pareto optimality (Mas-Colell et al. (1995) ), a state of resource allocations from which it is impossible to reallocate without making one subgroup worse. We show that finding a Pareto-fair classifier is equivalent to finding a model in our hypothesis class that belongs to the Pareto front (the set of all Pareto optimal allocations) with respect to the sub-population risks with the smallest possible risk disparity. This general notion is already amenable to non-binary sensitive attributes. We analyze the fairness performance trade-offs that can be expected from different approaches with an illustrative example. We provide a concrete algorithm that promotes fair solutions belonging to the Pareto front; this algorithm can be applied to any standard classifier or regression task that can be trained using (Stochastic) Gradient Descent. We show that if the goal is to obtain a zero-gap classifier, first recovering the fairest Pareto optimal solution and then adding harmful post-hoc corrections ensures the lowest risk levels across all subgroups. We demonstrate how our methodology performs on synthetic and real tasks such as inferring income status in the Adult dataset Dua & Graff (2017a) irrespective of their ethnicity or gender, predicting ICU mortality rates in the MIMIC-III dataset from hospital notes Johnson et al. (2016) , classifying skin lesions in the HAM10000 dataset Tschandl et al. (2018) , and assessing credit risk on the German Credit dataset Dua & Graff (2017b) .

2. RELATED WORK:

There is an extensive body of work on fairness in machine learning. Following Friedler et al. (2019) , we compare our methodology against the works of Feldman et al. (2015) ; Kamishima et al. (2012); Zafar et al. (2017) . Our method shares conceptual similarities with Zafar et al. ( 2017); Woodworth et al. (2017) ; Agarwal et al. (2018) , with differences on how we define our fairness objective and adapt it to work with standard neural networks. Although optimality is often discussed in the fairness literature, it is usually in the context of utility-discrimination tradeoffs. To the best of our knowledge, this is the first work to discuss optimality with respect to subgroup risks on a unified classifier, a distinction that disallows extreme performance degradation in the pursuit of fairness. Hashimoto et al. (2018) discusses decoupled classifiers as a way of minimizing group-risk disparity, but simultaneously cautions against this methodology when presented with insufficiently large datasets. The works of Chen et al. (2018) ; Ustun et al. (2019) also empirically report the disadvantages of decoupled classifiers as a way to mitigate risk disparity. Here we argue for the use of a single classifier because it allows transfer learning between diverse sub-populations. We do not need access to the sensitive attribute during test time, but in cases where this is possible, we instead choose to incorporate it as part of our observation features.

The work presented in:

The work of Chen et al. (2018) uses the unified bias-variance decomposition advanced in Domingos (2000) to identify that noise levels across different sub-populations may differ, making perfect fairness parity impossible without explicitly degrading performance on one subclass. Their methodology attempts to bridge the disparity gap by collecting additional samples from high-risk sub-populations. Here we modify our classifier loss to bridge the disparity gap without inducing unnecessary harm, which could prove to be synergistic with their methodology.

3. PROBLEM STATEMENT:

Consider we have access to a dataset D = {(x i , y i , a i )} n i=1 containing n independent triplet samples drawn from a joint distribution (x i , y i , a i ) ∼ P (X, Y, A) where x i ∈ X are our input features (e.g., images, tabular data, etc.), y i ∈ Y is our target variable, and a i ∈ A indicates group membership or sensitive status (e.g., ethnicity, gender); our input features X may or may not explicitly contain A.

Let h ∈ H be a classifier from a compact hypothesis class H trained to infer y from x, h : X → Y; and a loss function : Y × Y → R. We define the class-specific risk of classifier h on subgroup a as

R a (h) = E X,Y |A=a [ (h(X), Y )].

The risk discrimination gap between two subgroups a, a ∈ A is measured as Γ a,a (h) = |R a (h) -R a (h)|, and we define the pairwise discrimination gap vector as Γ A (h) = {Γ a,a (h)} a,a ∈A . Our goal is to obtain a classifier h ∈ H that minimizes this gap without causing unnecessary harm to any particular group in A. To formalize this notion, we define: Definition 3.1. Dominant risk vector: A vector r ∈ R k is said to dominate r ∈ R k , noted as r r , if r i ≥ r i , ∀i = 1, ..., k and ∃j : r j > r j (i.e., strict inequality on at least one component). Definition 3.2. Dominant risk classifier: Classifier h is said to dominate h, noted as h h , if the risks vector r = {R a (h )} 

P(H, A) = {h ∈ H : h ∈ H | h h }.

This means that there is no other classifier in H that is at least as good in all risks and strictly better in at least one of them. It is the set of classifiers such that improving one group's risk comes at the cost of increasing other's.

The Pareto front defines the best achievable trade-offs between population risks R a (h). This definition is already suited for classification and regression tasks where the sensitive attributes are categorical. Constraining the classifier to be in the Pareto front disallows laziness, there exists no other classifier in the hypothesis class H that is at least as good on all class-specific risks and strictly better in one of them. As shown in Chen et al. (2018) ; Domingos (2000) , the risk can be decomposed in bias, variance and noise for some loss functions, where the noise is the smallest achievable risk for infinitely large datasets (Bayes-optimal risk). If the noise differs between sensitive groups, zero-discrimination (perfect fairness) can only be achieved by introducing bias or variance.

Literature on fairness has focused on putting constraints on the norm of discrimination gaps (Zafar et al. (2017; 2015) ; Creager et al. (2019) ; Woodworth et al. (2017) ). We follow a similar criteria in Definition 3.4 and define the Pareto-fair classifier as the classifier in the Pareto front that minimizes || Γ A (h)|| ∞ (the maximum risk discrimination gap). Note that one could alternatively choose to find the Pareto classifier that minimizes the maximum subgroup risk. Definition 3.4. Pareto-fair classifier and Pareto-fair vector: A classifier h * is an optimal Paretofair classifier if it minimizes the discrimination gap among all Pareto front classifiers, h * = arg min

h∈P(H,A) || Γ A (h)|| ∞ . The Pareto-fair vector r * ∈ R |A| is defined as r * = {R a (h * )} a∈A .

Even when perfect equality of risk is desirable, Pareto classifiers still serve as useful intermediaries. To this end, Lemma 3.1, shows that applying a mechanism for reaching equality of risk on a dominated classifier h ∈ H leads to equal or worse risks than applying it to a Pareto classifier h p ∈ P(H, A) that dominates h. Lemma 3.1. If h ∈ P(H, A) → ∃ h p ∈ P(H, A) :

h p h ∧ R a (h ER p ) ≤ R a (h ER ) ∀a, with h ER an equal-risk classifier : R a (h ER ) = max a ∈A R a (h), ∀a and h ER p : R a (h ER p ) = max a ∈A R a (h p ).

To exemplify these notions graphically, Figure 1 shows a scenario with binary sensitive attributes a and binary output variable y where none of the Pareto front classifiers achieve equality of risk. Here the noise level differs between subgroups, and the Pareto-fair vector r * is not achieved by either a Naive classifier (minimizes expected global risk), or a classifier where subgroups are re-sampled to appear with equal probability (rebalanced Naive classifier). Note that the amount of performance degradation required to enforce perfect fairness starting from the Naive classifier is higher than when starting from the Pareto-fair vector.

Our objective is to find the Pareto-fair classifier h * as in Definition 3.4. In particular, we will give regularity conditions on the hypothesis class H and the loss function (h(X), Y ). It can be shown that for a sufficiently rich class of hypothesis functions H and for risk functions R a (h) that are convex with respect to h, the space of risk vectors is convex (see Geoffrion (1968); Koski (1985) ; Miettinen (2012) ). Under these conditions, we can find an auxiliary loss function φ : R |A| → R defined in terms of the subgroup risks {R a (h)} a∈A (denoted from here on as r ∈ R |A| ) that has a global minima on r * .

To prove the existence of loss function φ with the desired global minimum r * , it is convenient to think of the convex set of risk vectors as the intersection of a convex Pareto set (defined by a convex Pareto front and all risks that are dominated by it), and an additional convex set Ω. We further require both this Pareto set and Ω to be smooth, so that we can apply standard tools from smooth convex optimization and prove that for every risk vector r in the Pareto front there exists a function φ that has r as its global optima. This is formalized in Lemma 3.3. Definition 3.5. Pareto field: A function P : Ω → R is a Pareto field over a convex set Ω ⊂ R k if P ∈ C 1 is a continuously differentiable function such that ∇ i P (r) > 0 ∀r ∈ Ω, ∀i = 1, . . . , k. Where C is the number of constraints that characterized Ω. Proofs for all lemmas are given in the supplementary material, Section A.1. Lemma 3.3 motivates our loss function to take the form φ(r, µ)

= |A| i=1 r i + µ i (r i -c) +2

, with |A| the number of sensitive groups. The challenge is to find µ * such that our Pareto-fair vector r * minimizes φ(r, µ * ). In Section 4 we provide an algorithm to approximately recover the Pareto-fair classifier h * by adaptively searching for µ * and optimizing a classifier on φ(•, µ) using tools from standard Stochastic Gradient Descent.

4. OPTIMIZATION METHODS:

Recall that we wish to recover the Pareto-fair classifier h * within our hypothesis class. From Lemma 3.3, there exists a loss function of the form

EQUATION

such that h * = arg min h∈H φ(h; µ, c). Note that we can state the loss function directly on the risk vectors r, or implicitly on the classifier h. Since R a (h) is differentiable with respect to h, φ(h; µ, c) can be directly minimized using gradient descent on h for any choice of values µ, c. Let D Tr = {x, y, a} N T i=1 , D Val = {x, y, a} N V i=1 be our training and validation datasets respectively. The proposed implementation of the Pareto-fair framework is formalized in Algorithm 1 where we specify how to update the penalty coefficients µ, c. 

Algorithm 1: ParetoFairOptimization:

Given: h θ , , D Tr , D Val , nµ, np, nmax, γ > 0, k ≥ 1, ξ, ζ ∈ (0, 1), lr, B µ ← 1, µ * ← 1, µcount ← 0, ecount ← 0, c ← 0, Γ * ← ∞, h * ← h θ while ecount ≤ nmax and µcount ≤ nµ do µcount ← µcount + 1, ecount ← ecount + 1 h θ , r Val ← AdaptiveOptimize(h θ , , µ, c, D Tr , D Val ,

h * ← h θ , Γ * ← || ΓA(h)||∞, cold ← c, c ← mina r Val a k µ * ← µ • (r Val -c old ) + (r Val -c) + , a ← arg max a r Val a , µcount ← 0 else lr ← ζ lr, µ ← µ * , γ ← γξ, h θ ← h * end µ a ← (1 + γ)µ a end

// Exit loop due to excessive iterations or no improvement in fairness Return: h * We regularly check that reductions in the fairness gap generalize to the validation set; we additionally check if the trade-offs are in the non-dominated solution set (i.e., we have not observed a universally better classifier during training). Algorithm 2 (AdaptiveOptimize, shown in Section A.2) summarizes how we perform stochastic gradient descent steps with early stopping in-between µ, c updates. Lemma A.5 (shown in supplementary material, Section A.1) shows that this algorithm is convergent for |A| = 2 when the minimization step for a fixed µ, c is performed exactly.

To conclude this section, we stress that the proposed framework is independent of the desired algorithm class H and loss function ; these are kept from the original application. The Pareto-fair classifier uses the same inputs as the Naive classifier, with parameters that have been optimized towards Pareto fairness. Code will be made available.

5. EXPERIMENTS AND RESULTS:

We applied the methodology described in Section 4 to learn a Pareto-fair classifier (classifier in the group-risk Pareto front with the smallest risk disparity). We first validate our methodology on synthetic data with known Pareto-fair classifiers. Observations are drawn from a Gaussian mixture model where each sensitive attribute is encoded by a corresponding Gaussian mode, and target attributes are binary. We demonstrate our methodology on publicly available fairness datasets, and show how the risk disparity gaps are subsequently reduced. Where applicable, we compare our results against the methodologies proposed in Zafar et al. ( 2017 

5.1. SYNTHETIC DATA:

We tested our approach on synthetic data where the observations are drawn from conditional Gaussian distributions X|A = a ∼ N (µ A , 1), the target variable y is a conditional Bernoulli variable with distribution

Y |X = x, A = a ∼ Ber f a (x) with f a (x) = ρ low a 1[x ≤ c a ] + ρ high a 1[x > c a ]

, and |A| = 3. We used Brier Score (BS, refer to Section A.4) as our loss function. Under these conditions, the Bayes-optimal classifier for each subgroup is h(x) = f a (x). The subgroup risk function for any classifier h can be computed as

R a (h) = E X|A=a [Y -h(X)) 2 ] = E X|A=a [f a (X)(1 -h(X)) 2 + (1 -f a (X))h(X) 2 ].

Network details are given in the supplementary material, Section A.6. Figure 2 shows the analytically-derived Pareto-fair classifier, as well as the trade-off obtained by the proposed algorithm. We remark that the Pareto-fair classifier in this scenario cannot be achieved using linear logistic regression, something that can be seen from the functional form of the Bayes Optimal classifier derived in Section A.8 in supplementary material. From the center graph in Figure 2 we can see that the approximated Pareto-Fair risks (dashed lines) are close to the theoretical best (solid lines).The achieved maximum risk discrimination is also close to the theoretical best. The algorithm is able to correctly trade risks from the two advantaged subgroups to reduce the risk of the worst performing subgroup. 

5.2. REAL DATASETS:

We evaluate the performace of our algorithm on mortality prediction (MIMIC-III), skin lesion classification (HAM10000), income prediction (Adult), and credit lending (German). We also compared with several state of the art methods (Zafar et al. (2017) ; Kamishima et al. (2012) ; Hardt et al. (2016) ; Feldman et al. (2015) ) and with a model trained to minimize the average risk (Naive) and one that undersamples the minority class (Naive Balanced or NaiveB). A description of these methods is provided in Section A.3 of Supplementary material. Metrics used for evaluation include accuracy (Acc), confidence (Conf), expected and maximum calibration error (ECE and MCE respectively), Brier score (BS) and cross-entropy (CE). Values for every metric are reported per sensitive attribute. The dataset average is reported as (sample mean), and the group-normalized average as (group mean). The best and worst performing groups are also shown as (group max) and (group min), and the worst case difference between subgroups is reported as (disc). For an in-depth description of the the metrics and datasets used for evaluation, refer to Sections A.4 and A.5 of Supplementary material. Here we present a comparison of the performance in terms of accuracy and ECE but an extensive comparison is available in additional results Section A.7 in supplementary material.

Similarly to Chen et al. (2018); Zafar et al. (2017) ; Hardt et al. (2016) ; Woodworth et al. (2017) , we omit the sensitive attribute a ∈ A from our observation features. Our method trains a single classifier for the entire dataset to avoid needing test-time access to sensitive attributes whenever possible. All classifiers shown in this section are implemented using neural networks and/or logistic regression, we used either cross-entropy or Brier score as our training loss depending on the dataset. For details on the architecture and hyperparameters used on each dataset, refer to the supplementary material, Section A.6. We show how the proposed Pareto-fair approach produces well calibrated models that reduce group disparities across several metrics.

5.2.1. PREDICTING MORTALITY IN INTENSIVE CARE PATIENTS:

Medical decisions in general and mortality prediction in particular are examples where notions of fairness among sub-populations are of paramount importance, and where ethical considerations make no-harm fairness a very attractive paradigm. To that end, we used clinical notes collected from adult ICU patients at the Beth Israel Deaconess Medical Center (MIMIC-III dataset) Johnson et al. (2016) to predict patient mortality. We study fairness with respect to age (adult or senior), ethnicity (white or nonwhite), and outcome (alive/deceased). This leads to a total of 8 sensitive groups. We included outcome (alive/deceased) as a sensitive sub-population criteria to demonstrate a case where sensitive attributes would not be available at test-time, and because in our experiments patients who ultimately passed away on ICU were under-served by a Naive classifier. Table 1 shows empirical accuracies and expected calibration errors of all tested methodologies. It is important to note how imbalanced these groups are by looking at the 'ratio' column. Here one can see that 56.7% of the samples correspond to the majority class (alive, white, senior) and only 0.4% to the minority 2019). We limit our hypothesis class H to linear logistic regression to compare evenly against these standard baselines. Results on both datasets are shown in Table 5 .

6. DISCUSSION:

There exists a rich literature of fairness in machine learning in general, and risk-based fairness in particular. Here we explore a relatively untapped sub-problem where the goal is to reduce risk disparity gaps in the most ethical way possible (i.e., minimizing unnecessary harm). Unlike other works in the area, our problem investigates on how to reduce this disparity gap without collecting additional data samples, using the entirety of the available training data to produce an algorithm that is maximally fair with respect to sub-populations, and that does not necessarily require test-time access to sensitive attributes.

We provide a concrete algorithmic adaptation to any standard classification or regression loss to bridge this disparity gap at no unnecessary harm, and demonstrate its performance on several realworld case studies. Even for applications where the need for strict fairness outweighs the need for no-harm classifiers, this methodology can be applied before any post-hoc corrections to ensure that the risk disparity gap is closed in the most risk-efficient way for all involved sub-populations. The proposed algorithm does not sweep through different disparity constraint values, as previously done in related works, making it a simpler alternative.

As an avenue of future research, it could be of interest to analyze if we can automatically identify high-risk sub-populations as part of the learning process and attack risk disparities as they arise, rather than relying on preexisting notions of disadvantaged groups or populations. We strongly believe that no-unnecessary-harm notions of fairness are of great interest for several applications, especially so on domains such as healthcare and lending, where decisions are highly impactful.

A APPENDIX:

A.1 PROOFS Here we restate the Lemmas shown in Section 3 along with a sketch of the proofs.

Lemma 3.1 If h ∈ P(H, A) → ∃ h p h ∈ P(H, A) : R a (h ER p ) ≤ R a (h ER ), ∀a, where h ER is a equality of risks classifier in H such that R a (h ER ) = max a ∈A R a (h), ∀a and h ER p : R a (h ER p ) = max a ∈A R a (h p ), ∀a. Proof. If h p dominates h → R a (h ER p ) = max a ∈A R a (h p ) ≤ max a ∈A R a (h) = R a (h ER ) ∀a ∈ A.

Definition A.1. Dominant vector: A vector r ∈ R k is said to dominate r ∈ R k , noted as r r, if r i ≥ r i , ∀i = 1, ..., k and ∃j : r j > r j (i.e., strict inequality on at least one component). Definition A.2. Pareto field: A function P : Ω → R is defined as a Pareto field over a convex set To show that the set D + is the set of dominated points, we first note that for all r ∈ Ω | r ∈ D, r r we have P (r) > P (r ) using the same arguments as above, meaning that the set of dominated points is included in D + . Similarly, for all r ∈ Ω | r ∈ D, r r we have P (r ) ≥ P (r), meaning that non-dominated points are not a part of set D + . Proof. By hypothesis, both Ω and {r ∈ Ω : P (r) ≥ 0} are convex, so the intersection is also convex. For µ 0, φ(r, µ) is a convex function of r. Under these hypothesis, the Karush-Kuhn-Tucker (KKT) conditions are necessary and sufficient to recover a global minimizer. So it suffices to find a set µ * such that the KKT conditions are exactly satisfied at r * .

Ω ⊂ R k if P ∈ C 1 is a continuously differentiable function such that ∇ i P (r) > 0 ∀r ∈ Ω, ∀i = 1, . . . , k. Lemma A.1. Let Ω ⊂ R k be a convex

Let J * = {j : g j (r * ) = 0} be the set of indices of active g c constraints at r * , note that by hypothesis, r * ∈ D and therefore P (r * ) is always active. Focusing on the first KKT condition on the active set we get ∇φ(r * , µ) = λ∇P (r * ) + j∈J * η j ∇g j (r * ), in matrix form:

  1 . . . 1   +    2(r * 1 -c) + 0 . . . 0 2(r * A -c) +    µ = | | | ∇P ∇g j1 . . . ∇g j |J * | | | |     λ η 1 . . . η c *     .

By hypothesis, we have r i -c > 0 and therefore

µ i = λ∇P i (r * ) + j∈J * η j ∇g j i (r * ) -1 2(r * i -c) .

In particular, for η j = 0 ∀j and

µ 1 > [ ∇Pi(r * ) ∇P1(r * ) -1] 1 2(r * 1 -c) + ∀i = 1 we get µ i = [(µ 1 2(r * 1 -c) + + 1) ∇P i (r * ) ∇P 1 (r * ) -1] 1 2(r * i -c) + > 0∀i = j, µ 1 > 0, λ = µ 1 2(r * 1 -c) + + 1 ∇P 1 (r * ) > 0,

which satisfies the KKT conditions and produces the required minimizer r * for all points r * ∈ D.

Lemma A.3. Let Ω ⊂ R 2 be a convex set defined by Ω = {r ∈ R 2 : g c (r) ≥ 0 ∀ c ∈ {1, .

. . , C}, g c continuously differentiable}; let P : Ω → R be a convex Pareto field with corresponding Pareto set D = {r ∈ Ω :

P (r) = 0}. Let > 0, r * = arg min r∈D [max i r i -min j r j ] then c = min(r 1 , r 2 ) -< min(r * 1 , r * 2 ) ∀(r 1 , r 2 ) ∈ D.

Proof. case r * 1 = r * 2 : Assume r ∈ D : min i r i > r * 1 . Then r r * , which contradicts the hypothesis r ∈ D. case r * 1 = r * 2 : Without loss of generality, we can take r * 1 > r * 2 . We can then prove by couterpositive that r ∈ D implies r 1 > r 2 . Assume by contrapositive that there exists r ∈ D : r 2 > r 1 . By hypothesis D is a continuous curve, and we can find a path R(t) : R(0) = r * , R(1) = r , R(t) ∈ D∀t ∈ [0, 1], which would imply there exists t : R( t) = (r, r), in which case r * 1 = r * 2 , which contradicts r * 1 = r * 2 . We then have that for r ∈ D, r 1 > r 2 . Assume by counterpositive that

r 2 = min(r 1 , r 2 ) > min(r * 1 , r * 2 ) = r * 2 then r 1 < r * 1 (since both r, r * ∈ D), but then |r 1 -r 2 | < |r * 1 -r * 2 | which contradicts the hypothesis.

Putting both statements together, we proved that if

r * 1 > r * 2 , then ∀r ∈ D, min(r 1 , r 2 ) = r 2 ≤ min(r * 1 , r * 2 ) = r * 2 .

Lemma A.4. Let Ω ⊂ R 2 be a convex set defined by Ω = {r ∈ R 2 : g c (r) ≥ 0 ∀ c ∈ {1, . . . , C}, g c continuously differentiable}; let P : Ω → R be a convex Pareto field with corresponding Pareto set D = {r ∈ Ω : With c < r 2 , then, r 1 ≥ r 1 , r 2 ≤ r 2 , with equality if and only if r = r = arg min

P (r) = 0}. Let µ = (µ 1 , µ 2 ) > 0, µ = (µ 1 , µ 2 ); µ 2 > µ 2 . Define φ(r, µ, c) = 2 i=1 r i + µ i (r i -c)

s. t. r∈{P (r)≥0}∩Ω r 2

Proof. By hypothesis, we have φ(r, µ , c) ≥ φ(r , µ , c)

φ(r, µ, c) + (µ 2 -µ 2 )(r 2 -c) 2+ ≥ φ(r , µ, c) + (µ 2 -µ 2 )(r 2 -c) 2+ φ(r, µ, c) ≥ φ(r , µ, c) + (µ 2 -µ 2 )[(r 2 -c) 2+ -(r 2 -c) 2+

] since by hypothesis φ(r, µ, c) ≤ φ(r , µ, c), it follows r 2 ≤ r 2 , and therefore r 1 ≥ r 1 , because they both belong to the Pareto set.

Case r = r. To analyze when the equality arises, note that the tangent to D at r is τ (r) = (∇ 2 P (r), -∇ 1 P (r)) and that r + δτ (r), δ > 0 is a valid search direction if r = r (r does not minimize r 2 ). Also note that ∇φ(r, µ , c) = ∇φ(r, µ, c)

+ (0, 2(µ 2 -µ 2 )(r 2 -c) + )

By contradiction, assume r = r, then r + δτ (r), δ > 0 is a valid search direction and (δτ (r)), ∇φ(r, µ , c > 0, which implies r = arg min (max i r i -min j r j ) and

φ(r, µ, c) = |A| i=1 r i + µ i (r i -c) 2+

. Let µ 0 > 0 be an initial penalty vector, r 0 = arg min r∈D i r i , > 0, c 0 = min i r 0 i -,γ > 0, ξ ∈ (0, 1). Define the following auxiliary variables

c k+1 = min i r k i - Γ k = max i r k i -min i r k i a † = arg max a (r k a ) µ † a † = (γ + 1)µ k a † µ † \a † = µ k \a † r † = arg min s. t. P (r)≥0 r∈Ω φ(r, µ † , c k+1 ) c † = min i r † i - Γ † = max i r † i -min j r † j (2)

We define the following iteration procedure compute Eq 2

If Γ k = Γ † terminate While Γ † > Γ k γ ← ξγ Recompute Eq 2 End While µ k+1 = µ † • (r † -c k ) + (r † -c † ) + r k+1 = r †

The procedure is convergent to r * for |A| = 2.

Algorithm 2: AdaptiveOptimize 

Given: h θ , L, µ, c, D Tr , D Val , np, lr, B p ← 0, φ * ← ∞ h * ← h θ while p ≤ np //

(h θ ) = 1 |Ia| i∈Ia L(h θ (xi), yi) // Empirical risks φ B (h θ ) = a∈A RB a (h θ ) + µa( RB a (h θ ) -c) +2 θ ← θ -lr∇ θ φ B (h θ ) // Gradient step end if φ Val (h θ ) < φ *

// Evaluate improvement on Val and update target risks Kamishima et al. (2012) uses logistic regression as a baseline classifier, and requires numerical input (observations), and binary sensitive attribute and target variable. Fairness is controlled via a regularization term with a tuning parameter η that controls trade-off between fairness and overall accuracy. η is optimized via grid search with η ∈ (0, 300) as in the original paper. We report results on the hyperparameter configuration that produces the smallest accuracy disparity between sensitive subgroups.

then h * ← h θ ; φ * ← φ Val (h θ ); p ← 0 else p ← p + 1; end R Val ← RVal (h) end Return: h * , R Val Kamishima.

Feldman. Feldman et al. (2015) provides a preprocessing algorithm to sanitize input observations. It modifies each input attribute so that the marginal distribution of each coordinate is independent on the sensitive attribute. The degree to which these marginal distributions match is controlled by a λ parameter between 0 and 1. It can handle numerical and categorical observations, as well as non-binary sensitive attributes, and arbitrary target variables. Following Friedler et al. ( 2019), we train a linear logistic regressor on top of the sanitized attributes. λ is optimized via grid search with increments of 0.05. We report results on the hyperparameter configuration that produces the smallest accuracy disparity between sensitive subgroups.

Zafar. Zafar et al. (2017) Addresses disparate mistreatment via a convex relaxation. Specifically, in the implementation provided in Friedler et al. ( 2019), they train a logistic regression classifier with a fairness constraint that minimizes the covariance between the sensitive attribute and the classifier decision boundary. This algorithm can handle categorical sensitive attributes and binary target variables, and numerical observations. The maximum admissible covariance is handled by a hyperparameter c, tuned by logarithmic gridsearch with values between 0.001 and 1. We report results on the hyperparameter configuration that produces the smallest accuracy disparity between sensitive subgroups.

Hardt. Hardt et al. (2016) proposes a post-processing algorithm that takes in an arbitrary predictor and the sensitive attribute as input, and produces a new, fair predictor that satisfies equalized odds. This algorithm can handle binary target variables, an arbitrary number of sensitive attributes, and any baseline predictor, but requires test-time access to sensitive attributes. it does not contain any tuning parameter. We apply this method on top of both the Naive Classifier and our Pareto Fair classifier.

Naive Classifier (Naive). Standard classifier, trained to minimize an expected risk:

h = arg min h∈H E X,A,Y [L(h(X), Y )].

The baseline classifier class H is implemented as a neural network and varies by experiment as described in Section A.6, the loss function also varies by experiment and is also described in Section A.6. Optimization is done via stochastic gradient descent. Pareto Fair. Our proposed methodology, trained to minimize an adaptive loss function using the procedure described in Algorithm 1. Addresses risk disparity minimization without introducing unnecessary harm to any subgroup. The baseline classifier class H is implemented as a neural network and varies by experiment as described in Section A.6, the loss function also varies by experiment and is also described in Section A.6.

A.4 EVALUATION METRICS:

Here we describe the metrics used to evaluate the performance of all tested methods. We are given a set of test samples D t = {(x i , y i )} N i=1 where x i ∈ X is a realization of our model input and y i ∈ Y the corresponding objective. We assume that Y is a finite alphabet, as in a classification problem, and we will represent the one-hot encoding of y i as e i . Given a trained model h : X → [0, 1] |Y| the predicted output for a an input x i is a vector h(x i ) = p i such that ( p i ) j ∈ [0, 1], ∀j ∈ {1, ..., |Y|} and | |Y| j=1 ( p i ) j = 1 (e.g.: output of a softmax layer). The predicted class is ŷi = arg max j ( p i ) j and its associated confidence is pi = max j ( p i ) j . Ideally ŷi should be the same as y i . Using these definitions, we compute the following metrics. These metrics are computed independently for each sensitive subgroup on the test set and reported in Section A.7.

A.5 EXPERIMENTS ON REAL DATA:

The following is a description of the data and experiments for each of the real datasets. The information present here is summarized in We identify 8 sensitive groups as the combination of age (under/over 55 years old), ethnicity as determined by the majority group (white/nonwhite) and outcome (alive/deceased). Here we will use the term adult to refer to people under 55 years old and senior otherwise. This dataset shows large sample disparities since 56.7% of corresponds to the overall majority group (alive-senior-white) and only 0.4% to the overall minority group (deceased-adult-nonwhite).

We used a fully connected neural network as described in table 8 as the baseline classifier for our proposed Pareto Fair algorithm. We compare our results against both the Naive and Naive Balanced algorithms using the same neural network architecture, and crossentropy (CE) as our training loss. We also evaluate the performance of Zafar applied on the feature embeddings learned by the Naive Balanced classifier (Results for Zafar over the original input features were not promising).

We report the performance across a 5-fold split of the data, we used a 60/20/20 train-validation-test partition as described on Table 7 and report results over the test set. We denote the overall sensitive attribute as the combination of outcome (A:alive/D:deceased), age (A:adult/S: senior) and ethnicity (W:white, NW:nonwhite) with shorthand notation of the form D/A/W to denote, for example deceased, white adult. We also note that results on both Zafar and Hardt were done over only the sensitive attributes Adult/Senior and White/Nonwhite, outcome was not considered as a sensitive attribute for both methods since Hardt requires test-time access to sensitive attributes, which would not be possible in this case, and Zafar attempts to decorrelate sensitive attributes and classification decision boundaries, which is counterproductive when the sensitive attribute includes the correct decision outcome.

HAM10000. This dataset contains over 10, 000 dermatoscopic images of skin lesions over a diverse population Tschandl et al. (2018) . Lesions are classified in 7 diagnostic categories, and the goal is to learn a model capable of identifying the category from the lesion image. The dataset is highly unbalanced since 81% of the samples correspond to a melanocytic nevi lesion (nv), and 0.5% to dermatofibroma (df).

Here we chose to use the diagnosis class as both the target and sensitive variable, casting balanced risk minimization as a particular use-case for Pareto fairness.

We load a pre-trained DenseNet121 network Huang et al. (2017) and train it to classify skin lesions from dermatoscopic images using our Pareto fairness algorithm. We compared against the Naive and the Naive balanced training setup. Note that in the naive balance approach we use a batch sampler where images from each class have the same probability, this can be seen as a naive oversampling technique. 

:

In this paper we use "no-harm" to mean "no-unnecessary-harm," in other words, the system doesn't degrade performance on any class unless it is strictly necessary to improve performance in a disadvantaged class.

Figures and Tables:

a=1 dominates r = {R a (h)} |A| a=1 . Definition 3.3. Pareto front: We define the Pareto front as

Figure 1: Example of achievable risk tradeoffs for a binary regression problem with two unbalanced sensitive groups a = {0, 1} (P a=1 > P a=0 ). Bottom left and top left figures show conditional distributions of observation variable p a (x), and target variable p(y = 1|x, a) respectively. p(y = 1|x, a) are simple piece-wise constant functions with levels ρ low and ρ high . Middle figure shows the Pareto front defined by subgroup risks; we observe that noise levels differ across subgroups and perfect Pareto fairness is unattainable. Equality of Risk requires pure degradation of service for group a = 1. Both Naive and Rebalanced Naive classifiers do not attain the best possible no-harm classifier in this case. The proposed Pareto-fair point is shown in green, with its corresponding optimal Equal risk classifier shown in purple. The utopic point (R opt 0 ,R opt 1 ) can only be attained when the classifier also has access to the sensitive attribute (h(X, A)). Rightmost figure shows the trade-offs attainable between discrimination and mean risks, an alternative perspective to the risk trade-off figure.

Lemma 3.2. Let Ω ⊂ R k be a convex set, and P : Ω → R a Pareto field. Then the set D = {r ∈ Ω : P (r) = 0} is a Pareto set in Ω, and the set D + = {r ∈ Ω : P (r) > 0} is the set of dominated points, i.e., D + = {r ∈ Ω : ∃r ∈ D | r r } Lemma 3.3. Let Ω ⊂ R k be a convex set defined by Ω = {r ∈ R k : g c (r) ≥ 0 ∀ c ∈ {1, . . . , C}, g c continuously differentiable}; let P : Ω → R be a convex Pareto field with corresponding proper Pareto set D = {r ∈ Ω : P (r) = 0}. Let r ∈ D and φ(r, µ) = k i=1 r i + µ i (r i -c) +2 , with c < r, where we denote (x) +2 = max 2 (x, 0). There exists a set of vectors μ 0 such that: r = arg min r∈R k φ(r, μ) s.t. : P (r) ≥ 0, g c (r) ≥ 0 ∀c ∈ {1, . . . , C}.

);Kamishima et al. (2012);Hardt et al. (2016);Feldman et al. (2015).

Figure 2: Synthetic data experiment. Bottom left figure shows the conditional distribution of the observation variables for each of the three subgroups, while upper left shows the distribution of the target variable conditioned on both the observation and sensitive attribute; it additionally shows the theoretical and empirical Pareto-fair classifier. Center figure shows the empirical validation risks (dashed lines) as a function of epochs during the optimization procedure, the theoretical Pareto-Fair risks are also shown (solid lines). Rightmost shows how the maximum discrimination shrinks with the number of epochs.

set, and P : Ω → R a Pareto field. Then the set D = {r ∈ Ω : P (r) = 0} is a proper Pareto set in Ω, and the set D + = {r ∈ Ω : P (r) > 0} is the set of dominated points, i.e., D + = {r ∈ Ω : ∃r ∈ D | r r }. Proof. First we prove that D = {r ∈ Ω : P (r) = 0} is a Pareto set. Assume by contradiction that there exists r, r ∈ D | r r, then rr = c = i δ i e i δ i ≥ 0∀i ∈ {1, . . . , k}, ∃j ∈ {1, . . . , k} | δ j > 0. e i is a standard basis vector. Using the Gradient theorem we have P (r ) = P (r, . . . , δ k ), (∇P 0 (r + λc), . . . , ∇P k (r + λc)) dλ, r , r ∈ D. No point in set D is dominated by another point in set D, making set D a proper Pareto set.

Let Ω ⊂ R |A| be a convex set defined by Ω = {r ∈ R |A| : g c (r) ≥ 0 ∀ c ∈ {1, . . . , C}, g c continuously differentiable}; let P : Ω → R be a convex Pareto field with corresponding Pareto set D = {r ∈ Ω : P (r) = 0}. Let r * ∈ D and φ(r, µ) = |A| i=1 r i + µ i (r i -c) 2+ , with c < r * . There exists a set of µ * 0 such that: r * = arg min r∈R |A| φ(r, µ * ) s.t. : P (r) ≥ 0, g c (r) ≥ 0 ∀c ∈ {1, . . . , C} .

2+ and r = arg min s. t. P (r)≥0 r∈Ω φ(r, µ, c), r = arg min s. t. P (r)≥0 r∈Ω φ(r, µ , c).

s. t. P (r)≥0 r∈Ω φ(r, µ , c) and contradicts the hypothesis. Therefore, if r = r it follows r = r = r Lemma A.5. Let Ω ⊂ R |A| be a convex set defined by Ω = {r ∈ R |A| : g c (r) ≥ 0 ∀ c ∈ {1, . . . , C}, g c continuously differentiable}; let P : Ω → R be a Pareto field with corresponding Pareto set D = {r ∈ Ω : P (r) = 0}. Let r * = arg min r∈D

Naive Balanced (NaiveB). Baseline classifier designed to address undersampling of minority classes, trained to mimimize a class-rebalanced expected risk h = arg min h∈H E A∼U [1,...,|A|],(X,Y )∼P (X,Y |A) [L(h(X), Y )]. Like the Naive classifier, it is implemented as a neural network and optimized via stochastic gradient descent. The sole difference with the Naive classifier is that, during training, training samples are drawn from the new input distribution A ∼ U [1, . . . , |A|]; X, Y |A ∼ P (X, Y |A), which is achieved by re-weighted sampling of the original training dataset.

y i = ŷi ). Fraction of correct classifications in dataset. Confidence (CO): 1 N N i=1 pi . Average magnitude of the predicted class probability. Brier Score (BS): 1 N N i=1 || e i -p i || 2 where e i is the one-hot representation of the categorical ground truth value y i . This quantity is also known as Mean square error (MSE). Cross-Entropy (CE): -1 N N i=1 |Y| j=1 ( e i ) j log( p i ) j also known as negative log-likelihood (NLL) of the multinomial distribution. Expected Calibration Error (ECE): 1 N M m=1 i∈Bm [1(y i = ŷi ) -pi ] where M is the number of bins to divide the interval [0, 1] such that B m = {i ∈ {1, .., N } : pi ∈ ( m-1 M , m M ]} are the group of samples that our model assigns a confidence ( pi ) in the interval ( m-1 M , m M ]. Measures how closely the predicted probabilities match the true base rates. Maximum Calibration Error (MCE): max m∈{1,...,M } 1 |Bm| i∈Bm [1(y i = ŷi ) -pi ] . Measures worst-case miscalibration errors.

np, lr, B) // Optimize current loss // Check that solution is Pareto efficient and reduces fairness gap if || ΓA(h)||∞ < Γ * and r Val is not dominated by previous validation risks then

Table content: <table/>

Performance comparison on MIMIC dataset. We show accuracies and expected calibration error on test set. Standard deviations are computed across 5 splits. Naive classifier exhibited almost no measurable discrimination based on age or race on this dataset. We instead chose to use the diagnosis class as both the target and sensitive variable, casting balanced risk minimization as a particular use-case for Pareto fairness. This is possible since our methodology does not require test-time access to sensitive labels. It was not possible to show comparisons againstHardt et al. (2016) since the sensitive attribute is perfectly predictive of the outcome. Table3shows accuracies and ECE for all tested methodologies. The Pareto-fair classifier has the overall best calibration results and smallest accuracy disparities across methodologies. Comparisons against other methods were not possible because the target labels are non-binary.

Table content: <table><tr><td/><td/><td/><td/><td colspan="2">(a) Accuracy comparison</td></tr><tr><td>type</td><td colspan="2">ratio</td><td>Naive</td><td>NaiveB</td><td>ZafarCE</td><td>ParetoFair CE ParetoFair BS</td></tr><tr><td colspan="2">sample mean -</td><td/><td colspan="4">89.5±0.2% 78.9±0.7% 78.0±0.7% 74.3±0.8%</td><td>75.7±1.1%</td></tr><tr><td colspan="7">group mean 12.5% 61.8±1.5% 77.3±1.3% 69.4±1.3% 76.2±0.5%</td><td>77.5±0.7%</td></tr><tr><td>group max</td><td colspan="6">56.7% 99.2±0.3% 88.1±1.3% 93.0±1.4% 86.4±2.2%</td><td>87.1±2.0%</td></tr><tr><td>group min</td><td colspan="6">0.4% 18.1±2.5% 65.7±1.2% 43.7±3.9% 67.9±2.5%</td><td>69.9±2.2%</td></tr><tr><td>disc</td><td colspan="6">56.3% 81.1±2.5% 22.5±2.5% 49.3±4.4% 18.5±4.2%</td><td>17.1±2.6%</td></tr><tr><td/><td/><td colspan="5">(b) Accuracy comparison after Hardt post-processing</td></tr><tr><td>type</td><td/><td>ratio</td><td colspan="2">NaiveB+H Zafar+H</td><td colspan="2">ParetoFair CE+H ParetoFair BS+H</td></tr><tr><td colspan="3">sample mean -</td><td colspan="4">75.1±1.8% 66.3±1.0% 70.6±1.3%</td><td>71.9±1.2%</td></tr><tr><td colspan="7">group mean 12.5% 71.2±1.1% 62.4±1.3% 70.8±0.7%</td><td>72.3±1.0%</td></tr><tr><td colspan="2">group max</td><td colspan="5">56.7% 80.0±2.5% 70.0±2.6% 77.6±4.1%</td><td>79.9±5.6%</td></tr><tr><td colspan="2">group min</td><td colspan="5">0.4% 61.4±3.4% 52.0±3.6% 65.7±1.8%</td><td>66.8±2.1%</td></tr><tr><td>disc</td><td/><td colspan="5">56.3% 18.6±3.2% 18.1±4.3% 11.9±5.0%</td><td>13.1±5.6%</td></tr><tr><td/><td/><td/><td colspan="4">(c) Expected calibration error (ECE)</td></tr><tr><td>type</td><td>ratio</td><td colspan="2">Naive</td><td>NaiveB</td><td>ZafarCE</td><td>ParetoFair CE ParetoFair BS</td></tr><tr><td colspan="2">sample mean -</td><td colspan="5">0.115±0.004 0.037±0.003 0.219±0.007 0.062±0.021 0.052±0.012</td></tr><tr><td colspan="7">group mean 12.5% 0.288±0.016 0.068±0.006 0.305±0.013 0.094±0.023 0.082±0.021</td></tr><tr><td>group max</td><td colspan="6">56.7% 0.593±0.028 0.166±0.047 0.561±0.04 0.172±0.05</td><td>0.156±0.05</td></tr><tr><td>group min</td><td colspan="6">0.4% 0.042±0.002 0.025±0.003 0.069±0.014 0.039±0.013 0.032±0.011</td></tr><tr><td>disc</td><td colspan="6">56.3% 0.552±0.028 0.141±0.045 0.492±0.045 0.133±0.043 0.125±0.044</td></tr></table>



Table content: <table><tr><td/><td/><td/><td colspan="3">(a) Adult gender comparison</td><td/></tr><tr><td>type</td><td>ratio</td><td>Naive</td><td>NaiveB</td><td>Feldman</td><td colspan="2">Kamishima Zafar</td><td>ParetoFair</td></tr><tr><td>Male</td><td colspan="7">67.9% 92.3±0.4% 92.3±0.3% 92.4±0.3% 92.6±0.4% 92.2±0.7% 92.4±0.3%</td></tr><tr><td>Female</td><td colspan="7">32.1% 80.5±0.4% 80.3±0.7% 80.8±0.3% 80.9±0.4% 80.8±0.1% 81.0±0.5%</td></tr><tr><td colspan="2">sample mean -</td><td colspan="6">84.3±0.3% 84.2±0.5% 84.5±0.3% 84.6±0.3% 84.4±0.3% 84.7±0.3%</td></tr><tr><td colspan="8">group mean 50.0% 86.4±0.2% 86.3±0.4% 86.6±0.3% 86.7±0.2% 86.5±0.4% 86.7±0.2%</td></tr><tr><td>disc</td><td colspan="7">35.9% 11.9±0.7% 12.0±0.7% 11.6±0.2% 11.7±0.7% 11.4±0.7% 11.4±0.6%</td></tr><tr><td/><td/><td/><td colspan="3">(b) Adult ethnicity and gender comparison</td><td/></tr><tr><td>type</td><td>ratio</td><td>Naive</td><td>NaiveB</td><td>Feldman</td><td colspan="2">Kamishima Zafar</td><td>ParetoFair</td></tr><tr><td colspan="8">White male 39.7% 90.8±0.3% 90.8±0.3% 90.6±0.3% 90.4±1.1% 90.5±0.6% 90.7±0.3%</td></tr><tr><td>Other</td><td colspan="7">60.3% 79.8±0.5% 79.7±0.7% 80.4±0.6% 79.4±2.3% 80.1±0.1% 80.6±0.6%</td></tr><tr><td colspan="2">sample mean -</td><td colspan="6">84.1±0.2% 84.1±0.4% 84.5±0.3% 83.8±1.8% 84.2±0.2% 84.6±0.4%</td></tr><tr><td colspan="8">group mean 50.0% 85.3±0.1% 85.2±0.3% 85.5±0.3% 84.9±1.6% 85.3±0.3% 85.6±0.4%</td></tr><tr><td>disc</td><td colspan="7">20.5% 11.0±0.8% 11.1±0.8% 10.2±0.7% 11.0±1.4% 10.3±0.6% 10.2±0.6%</td></tr><tr><td/><td/><td/><td colspan="2">(c) German comparison</td><td/><td/></tr><tr><td>type</td><td>ratio</td><td>Naive</td><td>NaiveB</td><td>Feldman</td><td colspan="2">Kamishima Zafar</td><td>ParetoFair</td></tr><tr><td>Female</td><td colspan="7">29.5% 64.4±6.3% 64.1±7.5% 74.2±7.7% 68.8±6.8% 72.7±6.1% 65.4±5.1%</td></tr><tr><td>Male</td><td colspan="7">70.5% 68.9±5.0% 70.4±5.3% 72.0±3.7% 72.7±2.6% 71.6±3.9% 68.7±6.0%</td></tr><tr><td colspan="2">sample mean -</td><td colspan="6">67.6±4.5% 68.6±4.5% 72.7±4.1% 71.6±3.2% 71.9±3.2% 67.0±4.4%</td></tr><tr><td colspan="8">group mean 50.0% 66.6±4.7% 67.3±4.8% 73.1±4.9% 70.8±4.0% 72.1±3.5% 67.5±5.1%</td></tr><tr><td>disc</td><td colspan="4">41.0% 7.6±2.0% 10.4±3.2% 6.6±3.3%</td><td>6.0±4.4%</td><td colspan="2">6.3±4.3% 5.3±2.8%</td></tr></table>

Performance comparison between methods for Adult and German datasets. We show accuracies on test set. Standard deviations are computed across 5 splits.

Table content: <table/>



Table content: <table><tr><td>Dataset</td><td>Outcome/ Objective</td><td>Sensitive Attribute</td><td colspan="2">Train/Val/Test splits</td></tr><tr><td>Adult Dua &amp; Graff (2017a)</td><td>Income 2 categories</td><td>Gender (F/M) 2x2 categories Ethnicity(W/NW)</td><td>60/20/20</td><td>5</td></tr><tr><td>German Dua &amp; Graff (2017a)</td><td>Credit 2 categories</td><td>Gender (F/M) 2 categories</td><td>60/20/20</td><td>5</td></tr><tr><td>MIMIC-III Johnson et al. (2016)</td><td>Mortality (A/D) 2 categories</td><td>Mortality(A/D), Age (A/S) 2x2x2 categories Ethnicity (W/NW)</td><td>60/20/20</td><td>5</td></tr><tr><td>HAM10000 Tschandl et al. (2018)</td><td>Type of lesion 7 categories</td><td>Type of lesion 7 categories</td><td>80/10/10</td><td>1</td></tr></table>

Basic characteristics of real datasets MIMIC-III. This dataset consist of clinical records collected from adult ICU patients at the Beth Israel Deaconess Medical Center (MIMIC-III dataset) Johnson et al. (2016). The goal is predicting patient mortality from clinical notes. We follow the pre-processing methodology outlined inChen et al. (2018), where we analyze clinical notes acquired during the first 48 hours of ICU admission; discharge notes where excluded, as where ICU stays under 48 hours. Tf-idf statistics on the 10, 000 most frequent words in clinical notes are taken as input features.

Table content: <table/>

Table 8 shows the details of the implementation.

Table content: <table><tr><td>type</td><td>ratio</td><td colspan="2">Naive</td><td colspan="2">NaiveB</td><td>ZafarCE</td><td>ParetoFair CE ParetoFair BS</td></tr><tr><td>A/A/NW</td><td colspan="6">5.7% 99.1±0.4% 86.3±1.5% 93.0±1.4% 80.4±2.8%</td><td>83.4±2.6%</td></tr><tr><td>A/A/W</td><td colspan="6">13.3% 98.8±0.5% 86.3±1.1% 90.0±1.3% 81.0±1.5%</td><td>83.2±1.5%</td></tr><tr><td>A/S/NW</td><td colspan="6">12.9% 97.5±0.6% 76.5±1.7% 81.8±1.7% 69.5±2.2%</td><td>71.4±3.0%</td></tr><tr><td>A/S/W</td><td colspan="6">56.7% 97.9±0.3% 79.0±0.6% 77.4±0.7% 73.5±1.3%</td><td>74.6±1.6%</td></tr><tr><td>D/A/NW</td><td colspan="6">0.4% 23.4±9.4% 76.1±8.5% 47.7±9.6% 76.6±5.3%</td><td>78.6±6.1%</td></tr><tr><td>D/A/W</td><td colspan="6">0.9% 32.6±3.7% 80.1±3.3% 60.5±6.3% 85.0±3.8%</td><td>83.3±4.2%</td></tr><tr><td>D/S/NW</td><td colspan="6">1.8% 21.4±2.2% 66.9±2.4% 48.2±2.0% 72.6±5.1%</td><td>73.3±2.9%</td></tr><tr><td>D/S/W</td><td colspan="6">8.3% 23.4±2.2% 67.4±1.9% 57.1±2.2% 71.0±4.2%</td><td>72.5±3.6%</td></tr><tr><td colspan="2">sample mean -</td><td colspan="5">89.5±0.2% 78.9±0.7% 78.0±0.7% 74.3±0.8%</td><td>75.7±1.1%</td></tr><tr><td colspan="7">group mean 12.5% 61.8±1.5% 77.3±1.3% 69.4±1.3% 76.2±0.5%</td><td>77.5±0.7%</td></tr><tr><td>group max</td><td colspan="6">56.7% 99.2±0.3% 88.1±1.3% 93.0±1.4% 86.4±2.2%</td><td>87.1±2.0%</td></tr><tr><td>group min</td><td colspan="6">0.4% 18.1±2.5% 65.7±1.2% 43.7±3.9% 67.9±2.5%</td><td>69.9±2.2%</td></tr><tr><td>disc</td><td colspan="6">56.3% 81.1±2.5% 22.5±2.5% 49.3±4.4% 18.5±4.2%</td><td>17.1±2.6%</td></tr><tr><td>type</td><td colspan="2">ratio</td><td colspan="2">NaiveB H</td><td colspan="2">ZafarCE H ParetoFair CE H ParetoFair BS H</td></tr><tr><td>A/A/NW</td><td colspan="6">5.7% 76.3±1.9% 68.4±2.2% 69.9±2.5%</td><td>71.6±3.2%</td></tr><tr><td>A/A/W</td><td colspan="6">13.3% 76.7±1.6% 67.8±1.3% 70.8±2.4%</td><td>71.8±1.9%</td></tr><tr><td>A/S/NW</td><td colspan="6">12.9% 76.4±2.2% 67.9±2.1% 70.1±2.5%</td><td>71.3±3.1%</td></tr><tr><td>A/S/W</td><td colspan="6">56.7% 76.2±2.2% 67.6±1.1% 70.6±1.2%</td><td>72.1±1.2%</td></tr><tr><td>D/A/NW</td><td colspan="6">0.4% 66.6±9.9% 59.7±8.9% 71.2±7.5%</td><td>74.1±9.2%</td></tr><tr><td>D/A/W</td><td colspan="6">0.9% 66.4±2.4% 57.0±5.0% 72.9±3.8%</td><td>73.6±4.3%</td></tr><tr><td>D/S/NW</td><td colspan="6">1.8% 64.8±2.1% 55.9±3.1% 70.2±4.3%</td><td>71.2±2.9%</td></tr><tr><td>D/S/W</td><td colspan="6">8.3% 66.2±2.9% 55.3±1.9% 70.9±3.8%</td><td>72.4±3.5%</td></tr><tr><td colspan="2">sample mean -</td><td/><td colspan="4">75.1±1.8% 66.3±1.0% 70.6±1.3%</td><td>71.9±1.2%</td></tr><tr><td colspan="7">group mean 12.5% 71.2±1.1% 62.4±1.3% 70.8±0.7%</td><td>72.3±1.0%</td></tr><tr><td>group max</td><td colspan="6">56.7% 80.0±2.5% 70.0±2.6% 77.6±4.1%</td><td>79.9±5.6%</td></tr><tr><td>group min</td><td colspan="6">0.4% 61.4±3.4% 52.0±3.6% 65.7±1.8%</td><td>66.8±2.1%</td></tr><tr><td>disc</td><td colspan="6">56.3% 18.6±3.2% 18.1±4.3% 11.9±5.0%</td><td>13.1±5.6%</td></tr></table>

Accuracies on test set for MIMIC dataset. Standard deviations are computed across 5 splits.

Table content: <table><tr><td>type</td><td>ratio</td><td>Naive</td><td>NaiveB</td><td>ZafarCE</td><td colspan="2">ParetoFair CE ParetoFair BS</td></tr><tr><td>A/A/NW</td><td colspan="5">5.7% 0.027±0.004 0.198±0.019 0.138±0.029 0.291±0.04</td><td>0.252±0.033</td></tr><tr><td>A/A/W</td><td colspan="4">13.3% 0.027±0.005 0.196±0.013 0.2±0.026</td><td colspan="2">0.285±0.022 0.247±0.019</td></tr><tr><td>A/S/NW</td><td colspan="6">12.9% 0.055±0.007 0.311±0.017 0.364±0.036 0.383±0.018 0.363±0.029</td></tr><tr><td>A/S/W</td><td colspan="6">56.7% 0.049±0.003 0.283±0.005 0.449±0.014 0.355±0.014 0.338±0.017</td></tr><tr><td>D/A/NW</td><td colspan="5">0.4% 1.007±0.133 0.362±0.055 1.042±0.189 0.335±0.03</td><td>0.34±0.041</td></tr><tr><td>D/A/W</td><td colspan="6">0.9% 0.922±0.068 0.292±0.041 0.785±0.121 0.287±0.022 0.277±0.031</td></tr><tr><td>D/S/NW</td><td colspan="6">1.8% 1.085±0.019 0.417±0.018 1.032±0.041 0.371±0.033 0.365±0.03</td></tr><tr><td>D/S/W</td><td colspan="6">8.3% 1.043±0.021 0.419±0.023 0.857±0.044 0.379±0.032 0.367±0.035</td></tr><tr><td colspan="2">sample mean -</td><td colspan="5">0.159±0.003 0.285±0.006 0.437±0.014 0.347±0.013 0.327±0.013</td></tr><tr><td colspan="7">group mean 12.5% 0.527±0.022 0.31±0.009 0.608±0.026 0.336±0.009 0.319±0.002</td></tr><tr><td>group max</td><td colspan="6">56.7% 1.098±0.028 0.431±0.015 1.122±0.078 0.402±0.017 0.394±0.017</td></tr><tr><td>group min</td><td colspan="6">0.4% 0.025±0.003 0.188±0.016 0.138±0.029 0.261±0.023 0.233±0.022</td></tr><tr><td>disc</td><td colspan="6">56.3% 1.073±0.03 0.243±0.025 0.985±0.087 0.141±0.017 0.161±0.025</td></tr></table>

Brier Score (BS) on test set for MIMIC dataset. Standard deviations are computed across 5 splits.

Table content: <table><tr><td>type</td><td>ratio</td><td>Naive</td><td>NaiveB</td><td>ZafarCE</td><td colspan="2">ParetoFair CE ParetoFair BS</td></tr><tr><td>A/A/NW</td><td colspan="6">5.7% 0.068±0.005 0.329±0.027 2.803±0.659 0.459±0.051 0.409±0.043</td></tr><tr><td>A/A/W</td><td colspan="5">13.3% 0.067±0.007 0.322±0.017 4.079±0.601 0.45±0.031</td><td>0.4±0.027</td></tr><tr><td>A/S/NW</td><td colspan="4">12.9% 0.121±0.009 0.471±0.02 7.697±0.91</td><td>0.564±0.02</td><td>0.538±0.034</td></tr><tr><td>A/S/W</td><td colspan="6">56.7% 0.111±0.003 0.434±0.007 9.238±0.363 0.531±0.019 0.508±0.021</td></tr><tr><td>D/A/NW</td><td colspan="6">0.4% 1.573±0.169 0.567±0.078 21.617±3.25 0.515±0.038 0.523±0.052</td></tr><tr><td>D/A/W</td><td colspan="5">0.9% 1.363±0.112 0.459±0.05 17.284±2.502 0.46±0.023</td><td>0.445±0.033</td></tr><tr><td>D/S/NW</td><td colspan="6">1.8% 1.58±0.046 0.604±0.027 22.097±0.961 0.552±0.037 0.544±0.037</td></tr><tr><td>D/S/W</td><td colspan="4">8.3% 1.532±0.046 0.609±0.033 18.446±0.9</td><td colspan="2">0.563±0.036 0.547±0.041</td></tr><tr><td colspan="2">sample mean -</td><td colspan="3">0.266±0.005 0.437±0.007 9.113±0.4</td><td colspan="2">0.522±0.018 0.495±0.017</td></tr><tr><td colspan="7">group mean 12.5% 0.802±0.032 0.474±0.009 12.908±0.233 0.512±0.013 0.489±0.004</td></tr><tr><td>group max</td><td colspan="6">56.7% 1.65±0.055 0.635±0.014 23.343±1.058 0.587±0.019 0.578±0.019</td></tr><tr><td>group min</td><td colspan="6">0.4% 0.065±0.005 0.313±0.021 2.803±0.659 0.426±0.031 0.388±0.032</td></tr><tr><td>disc</td><td colspan="6">56.3% 1.585±0.059 0.323±0.03 20.54±1.089 0.162±0.025 0.191±0.037</td></tr></table>

Cross Entropy (CE) on test set for MIMIC dataset. Standard deviations are computed across 5 splits.

Table content: <table><tr><td>type</td><td>ratio</td><td>Naive</td><td>NaiveB</td><td>ZafarCE</td><td colspan="2">ParetoFair CE ParetoFair BS</td></tr><tr><td>A/A/NW</td><td colspan="6">5.7% 0.046±0.004 0.063±0.007 0.069±0.014 0.109±0.016 0.101±0.004</td></tr><tr><td>A/A/W</td><td colspan="4">13.3% 0.043±0.002 0.051±0.006 0.1±0.013</td><td colspan="2">0.092±0.027 0.083±0.015</td></tr><tr><td>A/S/NW</td><td colspan="6">12.9% 0.068±0.007 0.033±0.007 0.182±0.018 0.045±0.017 0.038±0.01</td></tr><tr><td>A/S/W</td><td colspan="6">56.7% 0.065±0.002 0.027±0.006 0.225±0.007 0.052±0.025 0.041±0.016</td></tr><tr><td>D/A/NW</td><td colspan="6">0.4% 0.55±0.094 0.164±0.049 0.522±0.095 0.149±0.037 0.146±0.06</td></tr><tr><td>D/A/W</td><td colspan="3">0.9% 0.43±0.057 0.09±0.03</td><td colspan="3">0.394±0.061 0.155±0.057 0.123±0.053</td></tr><tr><td>D/S/NW</td><td colspan="5">1.8% 0.57±0.017 0.068±0.022 0.516±0.021 0.09±0.029</td><td>0.07±0.02</td></tr><tr><td>D/S/W</td><td colspan="6">8.3% 0.535±0.023 0.048±0.012 0.429±0.022 0.061±0.027 0.052±0.026</td></tr><tr><td colspan="2">sample mean -</td><td colspan="5">0.115±0.004 0.037±0.003 0.219±0.007 0.062±0.021 0.052±0.012</td></tr><tr><td colspan="7">group mean 12.5% 0.288±0.016 0.068±0.006 0.305±0.013 0.094±0.023 0.082±0.021</td></tr><tr><td>group max</td><td colspan="5">56.7% 0.593±0.028 0.166±0.047 0.561±0.04 0.172±0.05</td><td>0.156±0.05</td></tr><tr><td>group min</td><td colspan="6">0.4% 0.042±0.002 0.025±0.003 0.069±0.014 0.039±0.013 0.032±0.011</td></tr><tr><td>disc</td><td colspan="6">56.3% 0.552±0.028 0.141±0.045 0.492±0.045 0.133±0.043 0.125±0.044</td></tr></table>

Expected calibration error (ECE) on test set for MIMIC dataset. Standard deviations are computed across 5 splits.

Table content: <table><tr><td>type</td><td>ratio</td><td>Naive</td><td>NaiveB</td><td>ZafarCE</td><td colspan="2">ParetoFair CE ParetoFair BS</td></tr><tr><td>A/A/NW</td><td colspan="6">5.7% 0.322±0.096 0.125±0.02 0.425±0.301 0.147±0.017 0.166±0.016</td></tr><tr><td>A/A/W</td><td colspan="6">13.3% 0.251±0.061 0.088±0.015 0.229±0.133 0.127±0.026 0.124±0.019</td></tr><tr><td>A/S/NW</td><td colspan="5">12.9% 0.223±0.038 0.058±0.013 0.323±0.198 0.077±0.03</td><td>0.07±0.013</td></tr><tr><td>A/S/W</td><td colspan="6">56.7% 0.203±0.007 0.046±0.011 0.496±0.184 0.091±0.034 0.069±0.02</td></tr><tr><td>D/A/NW</td><td colspan="6">0.4% 0.828±0.115 0.341±0.094 0.522±0.095 0.293±0.048 0.422±0.266</td></tr><tr><td>D/A/W</td><td colspan="6">0.9% 0.879±0.07 0.183±0.074 0.509±0.201 0.206±0.033 0.191±0.072</td></tr><tr><td>D/S/NW</td><td colspan="6">1.8% 0.822±0.052 0.158±0.082 0.553±0.067 0.152±0.027 0.129±0.029</td></tr><tr><td>D/S/W</td><td colspan="6">8.3% 0.865±0.027 0.11±0.038 0.563±0.182 0.136±0.044 0.112±0.041</td></tr><tr><td colspan="2">sample mean -</td><td colspan="5">0.294±0.013 0.067±0.007 0.441±0.113 0.104±0.029 0.089±0.015</td></tr><tr><td colspan="7">group mean 12.5% 0.549±0.031 0.139±0.007 0.453±0.094 0.154±0.024 0.16±0.047</td></tr><tr><td>group max</td><td colspan="4">56.7% 0.914±0.045 0.364±0.061 0.7±0.105</td><td colspan="2">0.293±0.048 0.429±0.259</td></tr><tr><td>group min</td><td colspan="6">0.4% 0.196±0.007 0.042±0.008 0.146±0.04 0.071±0.025 0.059±0.013</td></tr><tr><td>disc</td><td colspan="6">56.3% 0.719±0.044 0.322±0.065 0.554±0.097 0.222±0.062 0.37±0.262</td></tr></table>

Maximum calibration error (MCE) on test set for MIMIC dataset. Standard deviations are computed across 5 splits.

Table content: <table><tr><td>Groups</td><td>ratio</td><td colspan="2">ParetoFair NaiveB Naive</td><td colspan="2">ParetoFair NaiveB Naive</td></tr><tr><td>akiec</td><td colspan="2">2.5% 51.9%</td><td>55.6% 3.7%</td><td>55.4%</td><td>66.3% 55.3%</td></tr><tr><td>bcc</td><td colspan="2">2.7% 56.7%</td><td>76.7% 56.7%</td><td>57.6%</td><td>77.5% 66.7%</td></tr><tr><td>bkl</td><td colspan="2">7.2% 59.5%</td><td>58.2% 36.7%</td><td>64.5%</td><td>71.6% 64.3%</td></tr><tr><td>df</td><td colspan="2">0.5% 66.7%</td><td>33.3% 0.0%</td><td>59.8%</td><td>68.1% 81.6%</td></tr><tr><td>nv</td><td colspan="2">81.0% 83.5%</td><td>90.9% 96.0%</td><td>81.7%</td><td>89.1% 93.7%</td></tr><tr><td>vasc</td><td colspan="2">1.3% 71.4%</td><td>85.7% 0.0%</td><td>66.9%</td><td>74.7% 76.1%</td></tr><tr><td>mel</td><td colspan="2">4.8% 53.8%</td><td>48.1% 32.7%</td><td>63.6%</td><td>32.1% 63.6%</td></tr><tr><td colspan="2">sample mean -</td><td>78.6%</td><td>84.8% 83.6%</td><td>78.0%</td><td>85.6% 88.1%</td></tr><tr><td colspan="3">group mean 14.3% 63.4%</td><td>64.1% 32.3%</td><td>64.2%</td><td>73.6% 71.6%</td></tr><tr><td>group max</td><td colspan="2">81.0% 83.5%</td><td>90.9% 96.0%</td><td>81.7%</td><td>89.1% 93.7%</td></tr><tr><td>group min</td><td colspan="2">0.5% 51.9%</td><td>33.3% 0.0%</td><td>55.4%</td><td>66.3% 55.3%</td></tr><tr><td>disc</td><td colspan="2">80.4% 31.7%</td><td>57.5% 96.0%</td><td>26.3%</td><td>22.8% 38.4%</td></tr><tr><td/><td/><td>(a) Accuracy</td><td/><td/><td>(b) Confidence</td></tr></table>

Accuracies (left)  and confidence (right) on test set for HAM10000 dataset.

Table content: <table><tr><td>type</td><td>ratio</td><td colspan="3">ParetoFair NaiveB Naive</td><td colspan="3">ParetoFair NaiveB Naive</td></tr><tr><td>akiec</td><td colspan="2">2.5% 0.741</td><td>0.671</td><td>1.289</td><td>1.547</td><td>1.638</td><td>3.879</td></tr><tr><td>bcc</td><td colspan="2">2.7% 0.549</td><td>0.341</td><td>0.613</td><td>1.095</td><td>0.678</td><td>1.172</td></tr><tr><td>bkl</td><td colspan="2">7.2% 0.62</td><td>0.606</td><td>0.931</td><td>1.345</td><td>1.206</td><td>2.258</td></tr><tr><td>df</td><td colspan="2">0.5% 0.536</td><td>0.898</td><td>1.721</td><td>0.989</td><td>1.674</td><td>6.486</td></tr><tr><td>nv</td><td colspan="2">81.0% 0.241</td><td>0.128</td><td>0.054</td><td>0.58</td><td>0.28</td><td>0.115</td></tr><tr><td>vasc</td><td colspan="2">1.3% 0.36</td><td>0.246</td><td>1.604</td><td>0.747</td><td>0.499</td><td>4.564</td></tr><tr><td>mel</td><td colspan="2">4.8% 0.586</td><td>0.657</td><td>0.941</td><td>1.213</td><td>1.28</td><td>2.137</td></tr><tr><td colspan="2">sample mean -</td><td>0.308</td><td>0.213</td><td>0.234</td><td>0.708</td><td>0.449</td><td>0.579</td></tr><tr><td colspan="3">group mean 14.3% 0.519</td><td>0.507</td><td>1.022</td><td>1.074</td><td>1.037</td><td>2.944</td></tr><tr><td>group max</td><td colspan="2">81.0% 0.741</td><td>0.898</td><td>1.721</td><td>1.547</td><td>1.674</td><td>6.486</td></tr><tr><td>group min</td><td colspan="2">0.5% 0.241</td><td>0.128</td><td>0.054</td><td>0.58</td><td>0.28</td><td>0.115</td></tr><tr><td>disc</td><td colspan="2">80.4% 0.501</td><td>0.769</td><td>1.667</td><td>0.967</td><td>1.394</td><td>6.371</td></tr><tr><td/><td colspan="2">(a) Brier Score</td><td/><td/><td colspan="3">(b) Cross Entropy</td></tr></table>

Brier Score (left) and Cross Entropy (right) on test set for HAM10000 dataset.

Table content: <table><tr><td>type</td><td>ratio</td><td colspan="3">ParetoFair NaiveB Naive</td><td colspan="3">ParetoFair NaiveB Naive</td></tr><tr><td>akiec</td><td colspan="2">2.5% 0.249</td><td>0.254</td><td>0.516</td><td>0.754</td><td>0.447</td><td>0.966</td></tr><tr><td>bcc</td><td colspan="2">2.7% 0.147</td><td>0.134</td><td>0.161</td><td>0.269</td><td>0.408</td><td>0.418</td></tr><tr><td>bkl</td><td colspan="2">7.2% 0.171</td><td>0.156</td><td>0.304</td><td>0.731</td><td>0.294</td><td>0.744</td></tr><tr><td>df</td><td colspan="2">0.5% 0.452</td><td>0.57</td><td>0.816</td><td>0.718</td><td>0.853</td><td>0.985</td></tr><tr><td>nv</td><td colspan="2">81.0% 0.025</td><td>0.034</td><td>0.032</td><td>0.143</td><td>0.248</td><td>0.273</td></tr><tr><td>vasc</td><td colspan="2">1.3% 0.256</td><td>0.202</td><td>0.761</td><td>0.557</td><td>0.539</td><td>0.956</td></tr><tr><td>mel</td><td colspan="2">4.8% 0.14</td><td>0.212</td><td>0.323</td><td>0.277</td><td>0.289</td><td>0.836</td></tr><tr><td colspan="2">sample mean -</td><td>0.055</td><td>0.065</td><td>0.095</td><td>0.219</td><td>0.27</td><td>0.367</td></tr><tr><td colspan="3">group mean 14.3% 0.206</td><td>0.223</td><td>0.416</td><td>0.493</td><td>0.44</td><td>0.74</td></tr><tr><td>group max</td><td colspan="2">81.0% 0.452</td><td>0.57</td><td>0.816</td><td>0.754</td><td>0.853</td><td>0.985</td></tr><tr><td>group min</td><td colspan="2">0.5% 0.025</td><td>0.034</td><td>0.032</td><td>0.143</td><td>0.248</td><td>0.273</td></tr><tr><td>disc</td><td colspan="2">80.4% 0.426</td><td>0.535</td><td>0.784</td><td>0.611</td><td>0.605</td><td>0.712</td></tr><tr><td/><td/><td>(a) ECE</td><td/><td/><td/><td>(b) MCE</td></tr></table>

Expected Calibration Error (left)  and Maximum Calibration Error (right) on test set for HAM10000 dataset.

Table content: <table/>

Peer review (iclr2020):
This paper considers the notion of *no-harm* group fairness, i.e. trying to reduce the risk gap between minority and majority groups without excessive reduction in performance on the majority groups. Authors formalize the problem by defining a Pareto fair classifier, i.e. one that minimizes the risk gaps between groups and belongs to the family of Pareto classifiers containing the classifier minimizing the empirical risk. Authors suggest an optimization procedure for finding the Pareto fair classifier and demonstrate its performance on multiple datasets. Pros: I think that studying *no-harm* classifiers is an important topic given the alarming tendency of some of the recent group fairness approaches to achieve fairness by essentially driving down the performance on the majority group without improving on the minority group. Decision making in medical applications is one of the prominent examples where *no-harm* is absolutely needed, as authors suggested. The mathematical formulation of the problem around the notion of Pareto optimality also seems reasonable. Cons: My concerns are related to counter-intuitive experimental results and lack of clarity in parts of the presentation. Figure 1 seems important for understanding the ideas in the paper, but is not explained in much detail. Analogous to it Figure 2 is lacking important details. In the upper left plot, what are the decision boundaries of the baselines? What are the baselines risks in the center top figure, particularly for the equal risk classifier? It is hard to see from the right figure if the proposed classifier achieves the *no-harm* fairness over the equal risk classifier - numerical summary in a table could help. Finally, why is it necessary to use a neural network (which seems to be the case based on the supplement A.3) for the toy problem? I would recommend working through a toy example in more detail using a linear classifier to verify the correctness of the proposed technique and improve the overall clarity. Further, absence of a toy problem with linear classifier is alarming given there is not much discussion of the algorithm and its convergence properties. Regarding the real data experiments, none seem to showcase the *no-harm* versus *zero-gap* fairness tradeoff motivating this paper. MIMIC-III results seem to contradict the main story of the paper. The minority group appears to be *D/A/NW*, then there should be a *harming* group fairness classifier achieving close to 0 discrimination at the cost of lowering performance on other subgroups. The *no-harm* classifier should then achieve a similar or slightly lower performance on *D/A/NW*, but much better results on other sub-groups. Despite, the *no-harm* classifiers seems to outperform other approaches on *D/A/NW* by a good margin. Next, it appears that *Naive+Zafar* (it also would be helpful to have a brief discussion of the baselines considered) approach was not configured to eliminate A/S disparity as suggested by poor results on the D/A/NW and D/A/W, while it performs very well on other subpopulations. Skin Lesion classification experiment departs from the problem of fairness and considers the problem of classification with unbalanced classes instead. Results are again counterintuitive. *Rebalanced Naive* only mildly improves over the *Naive* approach, while proposed algorithm seems to achieve a quite significant mean accuracy improvement. This again does not show the motivating *harm* vs *zero gap* tradeoff, but it could be interesting as the imbalanced classification is an important problem by itself. Could you please compare to more advanced imbalanced classification algorithms? Results on the Adult and German Credit datasets are very similar across competing methods. Acknowledgement and references to the individual fairness line of works are missing when presenting the problem of fairness in machine learning. Font size in tables and Figure 2 legends is too small. Typo in the last sentence on page 8: *have highly impactful* -> *are highly impactful*.