Unprocessing Seven Years of Algorithmic Fairness
Authors: André Cruz
Moritz Hardt
Venue: ICLR 2024 oral (2024)
Keywords: fairness
algorithmic fairness
social computing
tabular data
meta study
Primary area: societal considerations including fairness, safety, privacy
TL;DR: A large-scale meta study shows that the simple post-processing method to achieve error rate parity is Pareto-dominant.

Abstract: Seven years ago, researchers proposed a postprocessing method to equalize the error rates of a model across different demographic groups. The work launched hundreds of papers purporting to improve over the postprocessing baseline. We empirically evaluate these claims through thousands of model evaluations on several tabular datasets. We find that the fairness-accuracy Pareto frontier achieved by postprocessing contains all other methods we were feasibly able to evaluate. In doing so, we address two common methodological errors that have confounded previous observations. One relates to the comparison of methods with different unconstrained base models. The other concerns methods achieving different levels of constraint relaxation. At the heart of our study is a simple idea we call unprocessing that roughly corresponds to the inverse of postprocessing. Unprocessing allows for a direct comparison of methods using different underlying models and levels of relaxation.

Published as a conference paper at ICLR 2024
UNPROCESSING SEVEN YEARS OF ALGORITHMIC FAIRNESS
André F. Cruz" & Moritz Hardt
Max Planck Institute for Intelligent Systems, Tübingen and Tübingen Al Center
ABSTRACT
Seven years ago, researchers proposed a postprocessing method to equalize the eтror rates of model across different demographic groups. The work launched hundreds of papers purporting to improve over the postprocessing baseline. We empirically evaluate these claims through thousands of model evaluations several tabular datasets. We find that the fairness-accuracy Pareto frontier achieved by postprocessing the predictor with highest accuracy contains all other methods we were feasibly able to evaluate. In doing so, we address two common methodological errors that have confounded previous observations. One relates to the comparison of methods with different unconstrained base models. The other concerns methods achieving different levels of constraint relaxation. At the heart of our study is simple idea we call unprocessing that roughly corresponds to the inverse of postprocessing. Unprocessing allows for direct comparison methods using different underlying models and levels of relaxation.
INTRODUCTION
Risk minimizing predictors generally have different error rates in different groups of population. When eπrоrs are costly, some groups therefore seem to bear the brunt of uncertainty. while others enjoy the benefits of optimal prediction. This fact has been the basis of intense debate in the field of algorithmic fairness, dating back to the 1950s (Hutchinson & Mitchell 2019). A difference in error rates between groups, equally deserving of resource, strikes many as a moral wrong (Angwin et al., 2016 Barocas et al. 2019).
Researchers have therefore proposed numerous algorithmic interventions to mitigate a disparity in error rates. The most basic such method is known postprocessing. Postprocessing sets group- specific acceptance thresholds so as to minimize risk while achieving an equality in error rates across desired set of groups. Postprocessing is both simple and computationally efficient.
Perhaps because of its simplicity, postprocessing has been widely assumed to be sub-optimal. Troves of academic contributions seek to improve over postprocessing by more sophisticated algorithmic means. These efforts generally fall into two categories. Preprocessing methods aim to adjust the source data in such manner that predictors trained on the data satisfy certain properties. So- called "inprocessing" methods, in contrast, modify the training algorithms itself to achieve a desired constraint during the optimization step.
1.1 OUR CONTRIBUTIONS
Through a large, computationally intensive meta study we empirically establish that postprocessing is Pareto-dominant among all methods we were feasibly able to evaluate. Whatever level of accuracy can be achieved by any method at specific level of error rate disparity, can also be achieved by setting group-specific acceptance thresholds on an unconstrained risk score.
We performed more than ten thousand model training and evaluation runs across five different prediction tasks from the folktables package (Ding et al. (202]), with two or four sensitive groups, based on tabular data from the US Census American Community Survey, We also include additional experiments on the Medical Expenditure Panel Survey (MEPS) (Blewett et 2021 dataset
"Corresponding author: andre. www.etueuutbucce wwwettubbinnnnn...m www.gngw...............w........

---

Published as a conference paper at ICLR 2024
0.5
constraint violation
model
0.4
optimal model m*
constant pred
0.3
postprocessing of m'
confidence interv.
0.2
0.0
0.625
0.650
0.675
0.700
0.725
0.750
0.775
0.800
0.825
accuracy
Figure 1: Test accuracy and constraint violation for 1000 models trained on the ACSIncome dataset (Ding et al. 2021), corresponding to a variety of preprocessing or inprocessing methods, as well as unconstrained learners. A red line shows the postprocessing Pareto frontier of the single model with highest accuracy GBM model).
in the appendix. The methods include recent state-of-ther -fr-----ttt--tttt algorithms, as well as standard baselines. While postprocessing is hyperparameter-ffee, we did extensive search for the best hyperparameters of all competing methods.
Our work addresses two common methodological errors that have confounded previous comparisons with postprocessing.
First, many preprocessing and inprocessing methods naturally do not achieve exact error rate equality, but rather some relaxation of the constraint. In contrast, postprocessing is typically applied so achieve exact equality. The primary reason for this seems to be that there is simple and efficient method based on tri-search to achieve exact equality (Hardt et al.] 2016). However, an efficient relaxation of error rate parity is more subtle and is therefore lacking from popular software packages. We contribute a linear programming formulation to achieve approximate error rate parity for postprocessing, and open- open-source our implementation in an easy-to-use Python package called error-parity This allows us to compare methods to postprocessing at the same level of slack.
Second, different methods use base models of varying performance. Observed improvements may therefore be due to a better unconstrained base model rather than better way of achieving error rate parity. How can we put different methods on level playing field? We introduce a simple idea we call unprocessing that roughly corresponds to the inverse of postprocessing. Here, we take model that satisfies error rate parity (approximately) and optimize group-specific thresholds so as to yield the best unconstrained model possible. Unprocessing maps any fairness-constrained model to a corresponding unconstrained counterpart. Both models have the same underlying risk-scor estimates, to which method we therefore do not have to come up with our own unconstrained model. We can simply steal, say, the unconstrained model implicit in any method.
then apply postprocessing. When comparing postprocessing to any given
These findings should not come as a surprise. Theory, perhaps overlooked, had long contributed an important fact: If an unconstrained predictor is close to Bayes optimal (in squared loss). then postprocessing this predictor is close to optimal among all predictors satisfying error rate parity (Hardt al. 2016, Theorem 4.5). To be sure, this theorem applies the squared loss and there are clever countercxamples in some other cases (Woodworth et al. 2017). However, our empirical evaluation suggests that these counterexamples don't arise in the real datasets we considered. This may be the case because, on the tabular datasets we consider, methods such as gradient boosting produce scores that are likely close to Bayes optimal under the squared loss. We focus on tabular data case-studies, an important basis for public policy decisions. High-dimensional datasets containing raw features (e.g.. images, text) are not explored the current paper, making for an interesting future work direction.
Limitations and broader impacts. We are narrowly concerned with evaluating algorithms that achieve error rate parity approximately. We do not contribute any new substantive insights about faimesss in different domains. Nor do we escape the many valid criticisms that have been brought
https: httpss///htthub.cco////caaa fowndat otr
2

---

Published as a conference paper at ICLR 2024
Contender
Unprocessed
Postprocessed
constraint violation
postproc
unppoc
postproc
B
B
B
B
0.2
A
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
accuracy
accuracy
accuracy
Figure 2: Example illustrating unprocessing. Left: Initial unconstrained model A postprocessed to Á. Some contender model B incomparable to Á. Middle: We unprocess B to get new model B". Right: Postprocessing B* to the same constraint level as B or A.
forward against algorithmic fairness narrowly construed (Bao al. 2021 Barocas et al., 2019, Kasy & Abebe, 202T). In particular, our work says nothing new about the question whether we should equalize error rates in the first place. Some argue that error rates should be a diagnostic, not a locus of intervention (Barocas al. 2019). Others reject the idea altogether (Corbett- et al. 2017). If, however, the goal is to equalize error rates exactly or approximately. the simplest way of doing so optimal: Take the best available unconstrained model and optimize group-specific thresholds.
1.2 RELATED WORK
Hardt et al. 2016) introduced error rate parity under the name of equalized odds in the context of machine learning and gave an analysis of postprocessing, including the aforementioned tri-search algorithm and theoretical fact. WWoodwothh et al. (2017) proposed second-moment relaxation of equalized odds and an algorithm to achieve the relaxation on linear predictors, as well as examples of specific loss functions and predictors for which postprocessing not optimal. The postprocessing
fairness-accuray trade-off has since been further detailed (Jang et al. 2022 Kim et al. (020)).
Numerous works have considered constrained empirical risk formulations to achieve fairness criteria, see, e.g., Agarwal et al. (2018); Celis et al. 2019);; Cotter et al. (2019); Cruz et al. (2023); Donini al. 2018); Menon & Williamson (2018 8 Zafar al. (2017]2019 for starting point. The work on learning fair representations (Zemel et al.) 2013) spawned much follow-up work on various preprocessing methods. See Section 2 for an extended discussion of the related work that we draw on in our experiments. We are unable to survey the vast space of algorithmic fairness methods here.
Our findings mirror several studies MI. and related fields that advocate for increased empirical rigor when proposing complex developments over simple baselines. |Armstron et al. (2009). and later Kharazmi et al. 2016)), present evidence of the widespread use weak baselines information retrieval, leading over-stated advancements in the field. Lucic et al. (2018) study state-of-the-- the-art generative adversarial networks (GAN), and find no clear-cul cut improvements over the original GAN introduced in Goodfellow et al.) (2014). Ferrari Dacrema al. 2019) reach similar conclusions for the field of recommenders systems, and Musgrave et al. (2020) for the field of metric learning. We contribute this growing body of work by showing that simple postprocessing baseline matches dominates all evaluated fairness interventions over a variety of datasets and evaluation scenarios.
2 EXPERIMENTAL SETUP
We conduct experiments four standard machine learning models, paired with five popular algorith- mic fairnss methods. The standard unconstrained models in the comparison are: gradient boosting machine (GBM), random forest (RF), neural network (NN), and logistic regression (LR).
Regarding fairness interventions, we include both pre- and inprocessing methods our experiments. We the learned fair representations (LFR) (Zemel et al. 2013) and the correlation remover
3

---

Published as a conference paper at ICLR 2024
(CR) (Bird al. 2020) preprocessing faimess methods, respectively implemented in the aif360
and fairlearn Python libraries. Additionally, we use the exponentiated gradient reduction (EG) and the grid search reduction (GS) inprocessing fairness methods (Agarwal et al. 2018) (implemented in fa r leann, as well as the FairGBM (Cruz et al. 2023) inprocessing method (implemented in fai rnglom. The preprocessing methods (CR, LFR) be paired with any other MI model (2 4 8 pairs), EG as well (4 pairs), and GS is compatible with GBM, RF, and LR models (3 pairs). FairGBM naturally only compatible with GBM. Together with the four standard unconstrained models, there is a total of 20 different methods (or pairings) in the comparison.
Although there have been numerous proposed fairness methods over the years, far fewer have available and ready-to-use open- ppen-source implementations. This is somewhat inevitable, as each new method would have to maintain a usable up-to-date implementation, as well as custom implementations for compatibility with different fairness criteria and different underlying base learners. Postprocessing approaches have practical advantage: a single implementation is compatible with any underlying learner that can produce scores of predicted probabilities, and any fairness criterion that can be expressed constraint over the joint distribution of (Y, (Y, S),, where Y is the true target, Y the predictions, and S the protected group membership.
On each dataset, we train 50 instances of cach ML algorithm in the study. For clarity, we will refer to different pairs of (unconstrained, fairness-aware) algorithms as different algorithms (e.g.. (GBM, EG) and (NN, EG) are two different algorithms). As study 20 different ML algorithms, total of 50- 20 = 1000 ML models is trained on each dataset. Each model is trained with a different randomly-sample selection of hyperparameters (e.g., learning rate of GBM, number of trees of an RF, weight regularization of an LR). This fulfills two goals: first, to accurately explore the best outcomes of competing fair ML methods, as related work has shown that a wide range of fairness values can be obtained for the same ML algorithm by simply varying its hyperparameters; and, second, to indirectly benchmark against fairness-awan aware AutoMI. approaches, which attempt train fair models by tuning the hyperparameters of unconstrained models (Cruz et al. 2021 Perrone et al. 2021 Weerts et al. 2023).
Unprocessing We define (f) the process of postprocessing predictor f to minimize some classification loss function l over the group-specific decision thresholds by E R, subject to an r-relaxed equalized odds constraint (Equation I.
(IPL 1|5 = Y = vl -PIL = 11s = b, Y = v1) ST, Va,bes (0
max
(I)
where the prediction for a sample of group s E S is given by ý = l( 2 t,), and R is its real-valued risk score. Thereby, umprocessing is defined as the unconstrained minimization of the loss E; ie., Jo (f), an oo-relaxed solution to equalized odds. As classifiers with different values of constraint violation are potentially incomparable between themselves, unprocessing emerges means to fairer comparisons between classifiers, unearthing the achievable unconstrained accuracy underlying a constrained predictor. For example, while classifiers A, B, and Å of Figure 2 are all Pareto-efficient (Pareto 1919) (i.e., incomparable), we can fairly compare the accuracy of A with that of xo (B) B* (both are unconstrained classifiers), B with (B) B, and À with म, (B).
The following subsections will detail the datasets we use (Section 2.1 and the experimental procedure we employ test our hypothesis (Section 2.2
2.1 DATASETS
We evaluate all methods on five large public benchmark datasets from the folktables Python package (Ding et al. (02T). These datasets are derived from the American Community Survey (ACS) public use microdata sample from 2018, containing variety of demographic features (c.g.. age, race, education). We also conduct a similar experiment on the MEPS dataset, shown in Appendix A.5
Each of the five ACS datasets is named after a specific prediction task: ACSIncome (1.6M rows) relates to household income prediction, ACSTravelTime (1.4M rows) relates to daily commute time prediction, ACSPublieCoverage (1.1M rows) relates to health insurance coverage prediction, ACSMobility (0.6M rows) relates to the prediction of address changes, and ACSEmployment (2.3M rows) relates employment status prediction. ACSIncome arguably carries particular weight in
4

---

Published as a conference paper at ICLR 2024
Original
Unprocessed
Postprocessed
constraint violation
0.6
RF
pastprocessing
RF,
95% conf. interv.
0.4
A
8
unprocc
0.2
+B
0.0
0.800
0.805 0.810 0.815 0.800
0.805 0.810 accuracy
0.815
0.800
0.805 0.810 0.815 accuracy
accuracy
Figure 3: Real-data version of the illustrative plot shown in Figure 2 (results shown on ACSIn- come test, models selected on validation). A and B two arbitrary incomparable models (both Pareto-efficient), which are made comparable after unprocessing. Left: original (unaltered) results; Middle: results after unprocessing all models; Right: original (unaltered) results, together with the postprocessing curve for both A and B". Additional model pairs shown in Appendix A.6
the fair ML community, as it is a larger modern-day version of the popular UCI Adult dataset (49K rows) (Dua & Graff 2017), which has been widely used for benchmarking algorithmic fairness methods over the years. We use race group membership as the protected attribute all five datasets (RACIP column); specifically, we use samples from the four largest groups: White, Black, Asian, and Other (some other race alone). Additional experiments using only samples from the two largest groups are presented as well, although not the focus of the paper results' analysis (see Appendix (A.4).
In total, 11 000 models were trained and evaluated over range of 11 different evaluation scenarios, pertaining to datasets, with sizes ranging from 49K to 2.3M samples.
2.2 EXPERIMENTAL PROCEDURE
We conduct the following procedure for each dataset, with 60%/20% /20% train/est/validation data split. First, we fit 1000 different ML models on the training data (50 per algorithm type). Second, to enable comparison of all models on an equal footing, we unprocess all 1000 trained models (on validation), and compute accuracy and equalized odds violation of the resulting classifiers. For any given classifier, its equalized odds violation is given by the left-hand side of the incquality in Equation | (or the smallest slack that fulfills the inequality). Then, we select the model with highest unprocessed accuracy, m = Txx (m'), ) obtained by the unconstrained postprocessing of the model m'. We defer the formal definition and procedure for solving the relaxed problem Section 4
We solve the r-relaxed equalized odds postprocessing on validation, T,(m",, for all values of constraint violation, E [0, c0,c) (with discrete intervals of 0.01), where c(m") is the constraint violation of m Finally, we compute accuracy and equalized odds violation on the withheld test dataset for all original 1000 models, and all post-processed versions of m', (mn'). All in all, even though the selection process for m and m entirely disregarded fairness, we expect (mm")) to be the classifier with highest accuracy at all levels of fairness r E [0.0, 1.0] - as illustrated in Figure 2
It may happen that the classifier m² with highest unprocessed accuracy is based on a pre- or inprocessing fairness method m'. Crucially, this implies that the means by which this fairness method resulted in fairer classifier was by finding more accurate risk scores in the first place. Otherwise, unprocessing cainess-constraan
models would not result in accurate unconstrained predictions.
Figure 3 shows an example of the general effect of unprocessing on the ACSIncome dataset (test results). Example models A and B are chosen respectively to maximize accuracy (A) and to maximize an average of accuracy and fairness (B) validation data. On the left plot (original, unaltered models), unconstrained (o markers) and constrained models (+ markers) are incomparable in terms of Pareto dominance: the first is generally more accurate than the latter, but the ranking reversed for constraint violation. However, after unprocessing (middle plot), we can see that the unprocessed unconstrained model A achieves higher accuracy than the unprocessed constrained
5

---

Published as a conference paper at ICLR 2024
ACSIncome (test)
ACSPublicCoverage (test)
0.6
0.6
GBM
constraint violation
0.5
constraint violation
0.5
GBM,
GBM,
0.4
0.4
GBM, GS
GBM, LFR
0.3
0.3
FairGBM
0.2
0.2
postprocessing
0.65
0.70
0.75
0.80
0.725 0.750 0.775 0.800
accuracy
accuracy
Figure 4: Pareto frontier attained by each GBM-based algorithm, together with the Pareto frontier attained by postprocessing the GBM-based model with highest unprocessed validation accuracy, m² Results for remaining ACS datasets shown in Figure Al
model B, indicating that postprocessing the first would be Pareto dominant or match the latter. Finally, the right plot confirms this hypothesis as evident by comparing postprocessing curves for both models. More extensive experimental results are shown in Section 3
RESULTS ON AMERICAN COMMUNITY SURVEY DATA
In this section we will present and discuss the results of experiments on all five ACS datasets. These experiments entail total of 1 000 models trained per dataset. Due to space constraints, plots are shown only for the ACSIncome and ACSPublieCoverage datasets. Corresponding plots for the remaining datasets are shown in Appendix [ Results for counterpart experiment using only two sensitive groups are also explored in this section, and further detailed Appendix(A.. A.4
3.1 COMPARISON BETWEEN FAIRNESS METHODS
We first analyze how each pre- or inprocessing fairness method compares with each other, without the effects of postprocessing. Figure 4 shows the Pareto frontiers achieved by each method when using GBM base models (see also Figure Al of the appendix). Overall, preprocessing methods (LFR and CR) achieved lacklustre fairness-aceuse accuracy trade-offs across all datasets, while the EG and FairGBM inprocessing methods performed best (highest area above Pareto frontiers). Specifically, L.R is indeed able achieve high fairness fulfillment, but at steep accuracy cost. To clarify: the plotted colored Pareto frontiers correspond to multiple (up 50) different underlying base models, while the black dashed line corresponds to the postprocessings of the single GBM-based model with highest accuracy. The following subsection contains more detailed analysis of postprocessing.
Interestingly, some fairness methods were able to achieve higher test accuracy than unconstrained GBM models, suggesting improved generalization performance. Figure Ssuuresshooss shows one potential reason: fairness methods can take notoriously high compute resources to trai, potentially giving them compute advantage with respect their unconstrained counterparts. Recent related work has put forth other explanations for why fairness-constraining can improve learning. Wei et al. (2023) find that fairness can improve overall performance under label noise by improving learning on tail sub-populations. On the other hand, Creager et al. (2021) establish ties between common fairness constraints and goals from the robustness literature, suggesting that fairness-aware learning can improve generalization under distribution shifts.
3.2 POSTPROCESSING OTHER METHODS
Figure 6 shows test-set results for the experimental procedure detailed in Section 2.2 zoomed on the region of interest (high accuracy and low constraint violation). The model with highest unprocessed validation accuracy, m" is shown with larger marker, while all other markers correspond to

---

Published as a conference paper at ICLR 2024
Time to fit on ACSIncome (GBM)
Time to fit on ACSPublicCoverage (GBM)
time (seconds)
10*
postproc.
time (seconds)
10*
postproc.
10a
10a
10a
10a
FGBM
GBM
CR
LFR
FGBM
LFR
CR algorithm
GBM
algorithm
Figure 5: Mean time to GBM and GBM-based preprocessing and inprocessing algorithms on the ACSIncome (left plot) and ACSPublicCoverage (right plot) datasets, with 95% confidence intervals. The time taken to run postprocessing is also shown for each algorithm as a stacked dark bar. Note the log scale: the EG inprocessing method takes order of magnitude longer to fit than the base GBM.
ACSincome (test)
ACSPublicCoverage (test)
0.5
0.40
base model
constraint violation
0.4
constraint violation
0.30
0.3
0.25
fairness method
None
0.20
CR
0.2
0.15
EG
GS
0.10
0.1
FairGBM
0.05
m" (selected on val.)
0.0
0.00
postprocessing of m"
95% confidence interv.
accuracy
accuracy
Figure 6: Detailed look at the postprocessing Pareto frontier on the ACSIncome (left) and ACSPublic- Coverage (right) datasets. shown, postprocessing m dominates or matches all 1000 trained MI. models, regardless of the underlying train algorithm (preprocessing, inprocessing, or unconstrained). Results for remaining ACS datasets shown Figure A7
unaltered non- conppostprocessed models. The Pareto frontier achieved by postprocessing m is shown with black dash-dot line, well corresponding 95% confidence intervals computed using bootstrapping (Efron & Tibshirani] 1994). Figures data, algorithms fail to show-up in the zoomed region of interest. Figures A8 A12 show the result of conducting the same experiment but using only GBM-based models, leading to identical trends.
show a wider view of the same underlying
All datasets show a wide spread of models throughout the fairness-accuracy space, although to varying levels of maximum accuracy (from 0.713 ACSTravelTime 0.831 on ACSEmployment) Unconstrained models (circles) can generally be seen to form cluster of high accuracy and low fairness (high constraint violation). Neither LFR nor LR-based methods manage produce any model within the region plotted in Figure On the ACSIncome and ACSPublicCoverage datasets, the m" model corresponds to an unconstrained GBM (blue circle), on ACSTravelTime and ACSEmployment m" is of type (GBM, CR) (blue diamond), and on ACSMobility m is of type (GBM, GS) (blue cross). Models m are GBM-based across all datasets, contributing to a wide body of literature reporting that GBM models are highly performant tabular datasets (Shwartz-Ziv & Armon 2022)
Crucially, postprocessing the single most accurate model resulted the fair optima for all values of faimess constraint violation on all datasets, either dominating matching other contender models (within 95% confidence intervals). That is, all optimal trade-offs between fairness and accuracy can be retrieved by applying different group-specifie thresholds the same underlying risk scores.
7

---

Published as a conference paper at ICLR 2024
ACSincome (test)
ACSPublicCoverage (test)
0.25
0.16
base model
0.14
GBM
constraint violation
constraint violation
NN
0.12
RF
0.10
fairness method
None
0.08
0.06
EG
GS
0.04
FairGBM
0.02
m' (selected val.)
0.00
0.00
pestprocessing
0.800
0.805
0.810
0.815
0.820
0.790
0.795
0.800
0.805 0.810
95% confidence interv.
accuracy
accuracy
Figure 7: [Binary protected groups] Results for a counterpart the main experiment, in which only samples from the two largest groups are used (White and Black). Note the significantly reduced axis range when compared with Figure Results for remaining ACS datasets shown in Figure A14
Finally, Figure 7 shows results for a similar experiment where fairness constraints were learned only on the two largest sub-groups (White and Black). This leads to arguably easier problem to solve, which is reflected on the general compression of models on the vertical axis (reduced constraint violation for all models). While previously the maximum unprocessed accuracy on ACSIncome was achieved at 0.38 constraint violation, on this Sinary-group setting it is achieved at 0.16 constraint violation. Nonetheless, the same trend is visible on all studied datasets. Unconstrained models either trained in an unconstrained manner (circles) made unconstrained via unprocessing (m*)- occupy regions of high accuracy and low fairness (high constraint violation). However, the Pareto frontier that results from postprocessing m² (the best-performing unconstrained model) to different levels of fairness relaxation again dominates or matches the remaining fairness methods.
All in all, postprocessing provides full view of the Pareto frontier derived from single predictor m. Regardless fairness violation, when this predictor is near-optimal potentially achievable on tabular data by training variety of algorithms-
will its postprocessed Pareto frontier be.
4 ACHIEVING RELAXED ERROR RATE PARITY
Error rate parity, also known equalized odds, enforces equal false positive rate (FPR) and equal true positive rate (TPR) between different protected groups (Hardt et al. 2016). This can be formalized as a constraint on the joint distribution of (Y, S)):
P = 1 = a, Y = v] = Pl- 1|5 = b, Vy E 1). Ya, E S,
(2)
where a f references two distinct groups in the set all possible groups S.
Fulfilling the strict equalized odds constraint greatly simplifies the optimization problem of finding the optimal classifier through postprocessing, as the constrained optimum must be at the intersection of the convex hulls of each group-specific ROC curve. As such, we' re left with linear optimization problem on a single 2-dimensional variable, = 71):
min %o {(1,0) Po + (1 -n) 1000)(0,11 1) P1. TED
where To is the global FPR, the global TPR, D C [0, 1)2 the optimization domain, Py, = P|Y y] the prevalence of label Y y, and {() y)) the loss incurred for predicting ý when the correct class was (we assume w.l.o.g. E(0, 0) 1), 0). Strict equalized odds fulfillment collapses the optimization domain D into single convex polygon that results from intersecting all group- specific ROC hulls; i.e., D = N Jss D., where D, is the convex hull of the ROC curve for group s. Specifically, D, = convexhull (C.(t) : RJ, and C, defines the ROC curve for group as:
C.(1) = (P ft 2 4 = Y = 0)
R 1s = s ===1 11)
8

---

Published as a conference paper at ICLR 2024
1.0
White
True Positive Rate
0.8
0.80
Black
Asian
0.75
0.6
Other
rel. global
0.4
0.70
strict global
0.2
0.65
relaxation
0.0
0.60
0.0
0.1
0.2
0.15 False Positive Rate
0.10
False Positive Rate
Figure 8: Optimal solution to strict (red star) and a 0.00-relaxed (blue star) equalized odds constraint. Right plot shows zoom on the region of interest (represented on the left by dashed black rectangle).
where E IR is a group-specific decision threshold, and the predictor's real-valued score. In this section, In order to relax the equalized odds constraint, we introduce a slack variable, g(a.b) E [0, 1J for each pair a, b S:
detail the solution to the r-relaxed equalized odds constraint defined in Equation
PI6- = 1|s = a, Y = y| PI- = 1|$ = b, = y] = 8fa.6) S Vy E (0, 1}, (5)
where r [0, 1] the maximum allowed constraint violation.
We introduce variables 7(()) = (1o%), (+) D.. E S, as the points of group-specific FPR, %0%
and group-specific TPR, (0) Equation can then be equivalently stated as:
b (0) (b)
p(a.)) Sr.
(6)
The global ROC point, is defined as:
(8)
Σ (4) 71 Ps/1
70
70
Peju>
71
where Psjy = P[S s y is the relative size of group within the of samples with label Y = Importantly, the global point is not limited to the intersection of group-specific ROC hulls. Each group-specific ROC point naturally limited be inside its group-specific ROC hull, ((e) D,, and only limited by its definition
function of all (e) S, per Equation 7
Finally, finding the r-relaxed optimum boils down to minimizing the linear objective function defined in Equation with domain D U.ES e D., subject to affine constraints defined in Equations 6 This optimization problem amounts to a linear program (LP), for which there is variety of efficient open-source solvers (Diamond & Boyd 2016). We contribute solution in open-source package!
Figure & shows an example of optimal strict and 0.05-relaxed solutions for equalized odds. Striet fulfillment of the equalized odds constraint (red star) reduces the feasible space of solutions the intersection of all group-specific ROC hulls. This fact potentially restricts all but one group to sub-optimal accuracy, achieved by randomizing some portion of the classifier's predictions. On the other hand, if we allow for some relaxation of the constraint, each group's 's ROC point will lie closer its optimum. In this example, the optimal solution to an 0.05 relaxation no longer needs to resort to randomization, placing each group's 's ROC point on the frontier of its ROC convex hull.
5 CONCLUSION
We revisit the simple postprocessing method a comprehensive empirical evaluation spanning 6 distinct datasets. 11 evaluation tasks, and more than 11 000 trained models. We find that, in all cases, any Pareto-optimal trade-off between accuracy and error rate parity can be achieved by postprocessing the model with highest accuracy. Along the way, address two confounding factors that have impaired previous comparisons of faimess methods, We hope that our study helps strengthen evaluation standards in algorithmic fairness.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We're indebted to Noam Barda, Noa Dagan, and Guy Rothblum for insightful and stimulating discussions about the project. We thank Florian Dorner, Olawale Salaudeen, and Vivian Nastl for invaluable feedback
earlier version of this paper. Lastly, thank the four anonymous reviewers for their fruitful suggestions, and the area chair for important and enriching references to related work on the undue over-complexifiatton of ML methods.
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting André F. Cruz.
REPRODUCIBILITY STATEMENT
We've made significant efforts to ease reproducibility our experiments. All source code has been open-sourced, including open-sourcing Python package to postprocess any score-based classifier to given level of fairnsss-constrain relaxation, code to run experiments using the aforementioned package (folder scripts s of the supplementary materials)), and code to generate the paper plots (folder notebooks the supplementary m Detailed hyperparameter search spaces for each algorithm included in folder hyperparameterss. materials." Furthermore, we are releasing detailed experimental results for all trained models in a series of csv files (under folder results of the supplementary material including variety of performance and fairness metrics, as well as their values at 2.5 and 97.5 bootstrapping percentiles. Appendix B details the infrastructure used to run jobs, as well as total compute usage.
rameters seseeees_ssssssssse of the supplementary
REFERENCES
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. A reductions approach to fair classification. International Conference on Machine Learning, pp. 60-69, 2018.
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias: There's software used across the country to predict future criminals. and it's biased against blacks. -awp:/eeeentni-iooommmaaaaa--eee
bstts::/www.ppoopiie org. article/ httriiieeiiss------sssseeeeeeeee
May 2016.
Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. Improvements that don't add up: ad-hoc retrieval results since 1998. In Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM '09, pp. 601-610, New York, NY, USA, 2009, doi: 10.1145/1645953. 1646031.
Association for Computing Machinery. ISBN 9781605585123.
URL / http://oooo...11..11111. 1ttp:/11..1111114101 U/LATpps///o...o//011111//555553 164411
Michelle Bao, Angela Zhou, Samantha Zottola, Brian Brubach, Brian Brubach, Sarah Desmarais, Aaron Horowitz, Kristian Lum, and Suresh Venkatasubramanian. It's compaslicated: The messy re- lationship between RAI datasets and algorithmic fairness benchmarks. Joaquin Vanschoren and Sii-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NewrlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL
www.inuuri.... pa2pp/2021/ htt.:/222d....ee....0000022222a2 -atell00---ssssssee Abstract roundl. html
Matias Barenstein. Propublica's compas data revisited, 2019.
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Faimesss and Machine Learning: Limitations and Opportunities. fairmlbook.org 2019. htttp://wwe fairriiiiiii maiimmbbook.. org
Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathleen Walker. Fairleam: toolkit for as-
sessing and improving faimess in AI. Technical Report 20-00-220 Microsoft,
May htt.:p/..........ooooo..oooo//er
wwwwww wtwp://www..ooo..oooooeeeeeeeeee
Fairlearn-a-toollee Set
10

---

Published as a conference paper at ICLR 2024
Lynn A Blewett, Julia A Rivera Drew, Risa Griffin, Natalie Del Ponte, and Pat Convey. IPUMS health surveys: Medical expenditure panel survey, version 2.1 [dataset]. Minneapolis: IPUMS, 2021. doi: 10.18128/D07L.2.11 URL/attps://abi.ooo 0Bt../.000.00..11111/00/01101100
L. Elisa Celis,, Lingxiao Huang, Vijay Keswani, and Nisheeth K. Vishnoi. Classification with fairness constraints: A meta-algorithm with provable guarantees. In Proceedings of the Conference on
Fairness, Accountability, and Transparency, FAT* 19, pp. 319-328. New York, NY. USA, 2019. Association for Computing Machinery. ISBN 9781450361255 doi: 10.1145/3887500.2287566
URL htt.hhttts://ooooo. www.oioorg/100/ 10....1155/. 321/ 3287586
Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining, pp. 797-806, 2017.
Andrew Cotter, Heinrich Jiang, Serena Wang, Taman Narayan, Seungil You, Karthik Sridharan, and Maya R. Gupta. Optimization with non-differentiable constraints with applications to fairness, recall, churn, and other goals. Journal of Machine Learning Research, 2019.
Elliot Creager, Joern-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Con- ference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2189-2200, PMLR, 18-24 Jul 2021. URL
creager21a html
André F. Cruz, Pedro Saleiro, Catarina Belém, Carlos Soares, and Pedro Bizarro. Promoting fairness through hyperparameter optimization. In 2021 IEEE International Conference on Data Mining (ICDM), pp. 1036-1041, 2021. doi: 10.1 109/ICCMM511299202000000
André F. Cruz, Catarina Belém, Sérgio Jesus, João Bravo, Pedro Saleiro, and Pedro Bizarro. FairGBM: Gradient boosting with fairness constraints. In The Eleventh International Conference on Learning Representations, 2023. URL httplppss///ppereveeeee net/ forum d= mXxBgCX3a
Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1-5,20) 2016.
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. In Advances in Neural Information Processing Systems, volume 34, pp. 6478 0798-699, 2021.
Michele Donini, Luca Oneto, Shai Ben-David, John S Shawe-Taylor, and Massimiliano Pontil. Empirical risk minimization under fairness constraints. In Advances in Neural Information Processing Systems, volume 31, 2018.
Dheeru Dua and Casey Graff. UCI Machine Learning Repository, 2017. URL htt:::tttttthhhii
ics.uci.ee www...uummm...mmm.mmm...
Bradley Efron and Robert J. Tibshirani. An Introduction to the Bootstrap. Number 57 in Monographs on Statistics and Applied Probability. Chapman & Hall/CRC, Boca Raton, Florida, USA, 1994.
Alessandro Fabris, Stefano Messina, Gianmaria Silvello, and Gian Antonio Susto. Algorithmic faimess datasets: the story so far. Data Mining and Knowledge Discovery, 36(6):2074-211) 2022.
Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. Are we really making much progress? a worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM Conference on Recommender Systems, RecSys 119, pp. 101-109, New York, NY, USA, 2019. Association for Computing Machinery, ISBN 9781450332436. doi: 10.1145/3298689. 3347058. URL httt:s: http:///....o/o/...11222226666.. 3347055
Ian Goodfellow, Jean Pbadec-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Ad- vances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL htttpa://poooeeeeiiiiiiio wwwr:us...eeu...u.uee.e cctpappppppppppppppppppepppppppe fiwes/ fiies//paper/20147 fi ffle:/eeseeeh2222555533522242222

---

Published as a conference paper at ICLR 2024
Ulrike Grömping. South german credit data: Correcting widely used data set. In Reports in Mathematics, Physics and Chemistry, 2019.
Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. In D. Lee, M. Sugiyama, Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
Ben Hutchinson and Margaret Mitchell. 50 years of test (un) fairness: Lessons for machine learning. In Proceedings of the conference on fairness, accountability, and transparency, pp. 49-58, 2019.
Taeuk Jang, Pengyi Shi, and Xiaoqian Wang. Group-aware threshold adaptation for fair classification. Proceedings of the AAAI Conference on Artificial Intelligence, 36ce::ece00000000098888866659999 2022. doi: 10. 1609/aaai, 20657. URL https:/ http://o....aaa........eeeeeee view 20657
Maximilian Kasy and Rediet Abebe. Faimess, equality, and power in algorithmic decision-makinn In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp.
576-586, 2021.
Sadegh Kharazmi, Falk Scholer, David Vallet, and Mark Sanderson. Examining additivity and weak baselines. ACM Trans. Inf. Syst., 34(4), jun 2016. ISSN 1046-8188. doi: 10.1145/2882782. URL https://dot: http:///ooo.ogg1...1111 1ttp://....11112282222
org
Joon Sik Kim, Jiahao Chen, and Ameet Talwalkar. FACT: A diagnostic for group fairness trade-offs. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 2264-52744 PMLR. 13-18 Jul 2020.
html
Lydia T. Liu, Max Simchowitz, and Moritz Hardt. The implicit fairness criterion of unconstrained learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1-660. PMLR, 09-15 Jun 2019. URL https: https::/pproceddingg mlr
#ress/v97/live 9ttm//mmmmm
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? large-scale study. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL https: http:::://ccceeeeeeeeeeee wwwp:ngg...euuuiii nwwrips cc/paper.p.p......pppp filefiiee//e paper/2018/F1le//4 e4 6de ee wew.caaccceesssleeeeeadeeaooooo. oagpaaaaeeeeeaaaa222220o--reeere
Aditya Krishna Menon and Robert C Williamson. The cost of fairness in binary classification. In Conference on Fairness, accountability and transparency, pp. 107-118. PMLR, 2018.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. metric learning reality check. In Com- puter Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV, pp. 68 681-699, Berlin, Heidelberg, 2020. Springer- ISBN 978-3- 030-58594-5, doi: 10.1100797883300000
0-188-3-00--88----- 41.
/oi. org 1007/
178-3-030-58595-- 2819--2-55555 41
Vilfredo Pareto. Manuale di economia politica: con una introduzione alla scienza sociale, volume 13.
Società editrice libraria, 1919.
Valerio Perrone, Michele Donini, Muhammad Bilal Zafar, Robin Schmucker, Krishnaram Kenthapadi, and Cédric Archambeau. Fair bayesian optimization. In Proceedings of the 2021 AAAI/ACM Conference AI, Ethics, and Society, AIES '21, pp. 854 863, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 978 4781445384 doi: DD.14453461102.34666
URL http:/ttt///oo..oo
1141114444455 70..044..22222 3462629
Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. In- formation Fusion, 81:84-90, 90, 2022. ISSN 1566-2535. doi: http:///bo.org/10.1100....
2001.11.011. URL http:://www.ciieeceeeee.ttaoaeee
61 81566258521002366
12

---

Published as a conference paper at ICLR 2024
Hilde Weerts, Florian Pfisterer, Matthias Feurer. Katharina Eggensperger, Edward Bergman, Noor Awad, Joaquin Vanschoren, Mykola Pechenizkiy, Bernd Bischl, and Frank Hutter. Can fair- be automated? guidelines and opportunities for fairness-awer automl. arXiv preprint arXiv:2303.08483 2023.
Jiaheng Wei, Zhaowei Zhu, Gang Niu, Tongliang Liu, Sijia Liu, Masashi Sugiyama, and Yang Liu. Faimnsss improves learning from noisily labeled long-tailed data, 2023.
Blake Woodworth, Suriya Gunasekar, Mesrob I Ohannessian, and Nathan Srebro. Learning non- discriminatory predictors. In Conference on Learning Theory, pp. 1920-1953, PMLR, 2017.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P. Gummadi. Fair- ness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of the 26th International Conference World Wide Web, WWW '17, pp. 1171-1180, 180, Republic and Canton of Geneva, CHE, 2017. International World Wide Web Conferences Steering Committee. ISBN 9781 07781450349330 htttps://do..ogg//// 01t....1111133311111 12 3052660
doi: 10.1145/3033812.335566 URL
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P Gummadi. Fairesss constraints: flexible approach for fair classification. The Journal of Machine Learning Research, 20(1):2737-278,, 2019.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International Conference on Machine Learning, pp. 325-333, 2013.
13

---

Published as a conference paper at ICLR 2024
A ADDITIONAL EXPERIMENTAL RESULTS
The main body of the paper discusses results on all five ACS datasets. However, due to space constraints, plots are only shown for two example datasets: ACSIncome and ACSPublicCoverage. Appendices A.I HA.3 show analogous versions of each previous plot for the remaining three ACS ACSMobility, and ACSEmployment. Plots are shown in the same order as in the main paper body. Additionally, we run a similar experiment on ACS datasets using only the two largest sensitive groups (White and Black), shown in Appendix [A.4 Appendix Appendix(AA33ppm (A.3ppresent results on the MEPS dataset (Blewett et al., 2021), entirely different data source corresponding to real-world surveys healthcare usage across the United States. Appendix A further evidence that model ranking maintained throughout all levels of constraint violation, and Appendix A.7 compares the results of unconstrained model training to unprocessing constrained models.
datasets:
We consciously refrain from evaluating the popular COMPAS dataset (Angwin et al. 2016). as related work has surfaced severe data gathering issues, including measurement biases and label leakage (Bao et al. 102 [arenstein 2019; Fabris al.) (022). The German Credit dataset (Dua & Graff 2017) another popular benchmark in the fairness literature - suffers from its small size 000 samples), the age of its data (dates back to 1973-1975), and encoding issues that make it impossible to retrieve accurate sensitive information such as the individual's 's sex (Grömping 2019). Overall, total of different evaluation scenarios were studied, pertaining
datasets, with sizes ranging from 49K to 2.3M samples. Confidence intervals and metric results are computed using bootstrapping on the respective evaluation dataset (Efron & Tibshirani] 1994). We hope the scale of our study suffices to convince the reader of the validity of our claims. Source code is made available to easily reproduce our setup on other datasets All appendix experiments are in accordance with the main findings presented in Section 3
A.1 COMPARISON BETWEEN FAIRNESS METHODS
Figure Al shows Pareto frontiers for all studied GBM-based algorithms. We observe a similar trend to that seen in Figure 4 preprocessing fairness methods can increase fairness but at dramatic accuracy costs, while EG and FairGBM inprocessing fairness methods trade Pareto-dominance between each other. Postprocessing Pareto frontier also shown for reference, but more detailed comparison between postprocessing and all other contender models is shown in the following section.
A.2 POSTPROCESSING VS OTHER METHODS
Figures A2 A6 show complete views of the Pareto frontiers obtained by postprocessing the model with highest validation accuracy m² each dataset (potentially obtained by unprocessing fairesss- aware model), together with a scatter of all other competing preprocessing. inprocessing, or uncon- strained models 000 in total per dataset). Figure/A7] shows detailed postprocessing results on each dataset, zoomed on the region of interest (maximal accuracy and minimal constraint violation, i.e., bottom right portion of the plot). Figures [88 A12 show results using only subset of models: only LBBM-based models. The main paper hypothesis is confirmed on each and every plot: we can obtain optimally fair classifiers at any level of constraint violation by postprocessing the model with highest accuracy, m", irrespective of its constraint violation.
SSupllementary materials:
https://github. wwt.c/oc//s///// iicis foundat htw.e//tii////rrooroorrtttt///ta
14

---

Published as a conference paper at ICLR 2024
ACSTravelTime (test)
ACSMobility (test)
0.20
GBM
constraint violation
0.3
constraint violation
GBM,
0.15
GBM,
GBM, GS
0.2
GBM, LFR
0.10
FairGBM
postprocessing
0.1
0.05
0.0
0.00
0.60
0.65
0.75
0.76
0.77
accuracy
accuracy
ACSEmployment (test)
GBM
constraint violation
0.15
GBM,
GBM, GS
GBM, LFR
FairGBM
postprocessing
0.05
0.00
0.6
0.7
accuracy
Figure A1: Pareto frontier attainable by each GBM-based MI. algorithm, together with the Pareto frontier attained by postprocessing m", the GBM-based model with highest unprocessed validation accuracy. Plotted Pareto curves are linearly interpolated between Pareto-efficient models.
ACSIncome (test)
constraint violation
0.5
0.3
0.1
0.625
0.650
0.675
0.700
0.725
0.750
0.775
0.800
0.825
accuracy
base model
faimess method
LFR
GBM
None
FairGBM m (accuracy=0.821)
NN
EG
postprocessing of 95% confidence interv.
RF
GS
Figure A2: Fairness and accuracy test results ACSIncome dataset. Colors portray different underlying unconstrained models and markers portray different fairness methods (or no fairness method for circle markers). The unconstrained model with highest validation accuracy, m is shown with a larger marker, and the Pareto frontier attainable by postprocessing m is shown as a black dash-dot line, together with its 95% confidence intervals in shade. This is colored and more granular version of Figure [
000 trained ML models (50 of each type) on
15

---

Published as a conference paper at ICLR 2024
ACSPublicCoverage (test)
0.6
constraint violation
0.4
0.2
0.0
0.72
0.74
0.76
0.78
0.80
accuracy
Figure A3: Fairness and accuracy test results on the ACSPublicCoverage dataset. Model m is of type (GBM) and achieves 0.808 accuracy. See legend and caption of Figure A2 for more details.
ACSTravelTime (test)
constraint violation
constant ptedictor BCC
0.1
0.0
0.56
0.58
0.60
0.64 accuracy
0.66
0.68
0.70
Figure A4: Fairness and accuracy test results on the ACSTravelTime dataset. Model m is of type (GBM, CR) and achieves 0.713 accuracy. See legend and caption of Figure A2 for more details.
ACSMobility (test)
0.20
Constraint violation
constant predictor acc.
0.15
0.10
0.05
0.00
0.735
0.740
0.745
0.750
0.755
0.760
0.765
0.770
accuracy
Figure A5: Fairness and accuracy test results on the ACSMobility dataset. Model m is type (GBM, GS) and achieves 0.770 accuracy. See legend and caption of Figure A2 for more details.
16

---

Published as a conference paper at ICLR 2024
ACS Employmmnn (test)
0.20
constraint violation
pred predictor aCC.
0.15
0.10
0.05
m
0.00
0.55
0.60
0.65
0.70
0.75
0.80
accuracy
Figure A6: Fairness and accuracy test results on the ACSEmployment dataset. Model m of type (GBM, CR) and achieves 0.831 accuracy. See legend and caption of Figure A2 for more details.
ACSTravelTime (test)
ACSMobility (test)
base model
GBM
Constraint violation
0.15
constraint violation
0.15
NN
RF
faimess method
0.10
0.10
GS
0.05
0.05
FairGBM
m" (selected val.)
0.00
0.00
postprocessing m
0.66
0.68
0.70
0.72
0.760
0.765 accuracy
0.770
95% confidence interv.
accuracy
ACSEmployment (test)
0.10
base model
GBM
constraint violation
0.08
method
0.06
None
CR
0.04
EG
GS
0.02
LFR
FairGBM
m (selected on val.)
0.00
0.8225
0.8250
postprocessing of m
accuracy
95% confidence interv.
Figure A7: Detailed view the postprocessing Pareto frontier on the ACSTrave/Time (left), AC- SMobility (right), and ACSEmployment (bottom) datasets. Respectively corresponds to zoomed-in versions of Figures A4 (left), A5 (right), and And|6 (bottom).
17

---

Published as a conference paper at ICLR 2024
ACSIncome (test)
constraint violation
0.5
0.3
0.1
0.625
0.650
0.675
0.700
0.725
0.750
0.775
0.800
0.825
accuracy
GBM
GBM, GS
(accuracy=0.821) postprocessing of m
CR
GBM, LFR FairGBM
GBM, EG
dominated by m
Figure A8: Fairnsss and accuracy test results for 300 GBM-based based ML models (50 of each algorithm type) on the ACSIncome dataset. The unconstrained model with highest validation accuracy, m² is shown with larger marker, and the Pareto frontier attainable by postprocessing m is shown black dash-dot line, together with its 95% confidence intervals in shade.
ACSPublicCoverage (test)
constraint violation
0.5
0.3
0.1
0.72
0.74
0.76
0.78
0.80
accuracy
Figure A9: Fairness and accuracy test results for GBM-based ML models on the ACSPublicCoverage dataset. Model m is of type (GBM) and achieves 0.808 accuracy. See legend and caption of Figure A8 for more details.
ACSTravelTime (test)
constraint violation
0.3
constant eredictor acc.
0.2
0.1
0.0
0.60
0.68
0.70
0.72
accuracy
Figure A10: Fairness and accuracy test results for GBM-based ML models on the ACSTravelTime dataset. Model m² is of type (GBM,CR) and achieves 0.713 accuracy. See legend and caption of Figure A8 for more details.
18

---

Published as a conference paper at ICLR 2024
ACSMobility (test)
0.20
constraint violation
consant consantt predictor DCC.
0.15
0.10
0.05
0.00
0.750
0.760
accuracy
Figure A11: Fairness and accuracy test results for GBM-based ML models on the ACSMobility dataset. Model m² is of type (GBM,GS) and achieves 0.770 accuracy.
ACS ACploploy (test)
0.20
constraint violation
constant predictor acc.
0.15
0.10
0.05
0.00
0.55
0.60
0.80
accuracy
Figure A12: Fairness and accuracy test results for GBM-based ML models on the ACSEmployment dataset. Model m² is of type (GBM,GS) and achieves 0.831 accuracy.
19

---

Published as a conference paper at ICLR 2024
A.3 TIME TO FIT EACH METHOD
Figure A13 shows the mean time to fit each GBM-based model on three separate datasets. The small increment the time taken to fit the
trend is clear all studied datasets: postprocessing base model, preprocessing methods take longer but are still within the same order of magnitude, the FairGBM inprocessing method also incurs a relatively small increment the base model time, while EG and GS take one to two orders of magnitude longer to fit.
For clarification, all times listed end-to-end process times for fitting and evaluating a given model. For example, postprocessing times include the time taken to fit the base GBM model plus the time taken solve the LP. We note that most time consumed for postprocessing simply corresponds to computing the model scores for the respective dataset where postprocessing will be fitted, while solving the usually takes only a few seconds. Likewise, preprocessing fairness methods include the time taken to fit the preprocessing method, the time taken transform the input data, and the time to fit the base model. Finally, inprocessing fairness methods include only the time taken to fit preprocessing postprocessing steps are required. Nonetheless, the
the inprocessing method, GS and EG inprocessing methods take significantly longer than any other competing method.
Time to fit on ACSTravelTime (GBM)
Time to fit on ACSMobility (GBM)
time (seconds)
104
postproc.
time (seconds)
104
postproc.
10a
103
10a
10a
101
FGBM
CR
LFR
GBM
EG
FGBM
GBM
LFR
CR
algorithm
algorithm
Time fit on ACSEmployment (GBM)
Time to on MEPS (GBM)
time (seconds)
postproc
time (seconds)
102
postproc.
104
10
10a
102
100
FGBM
GBM
LFR
CR
GS
EG
GBM
FGBM
CR
LFR
GS
EG
algorithm
algorithm
Figure A A13: Mean time to fit the base GBM model and each studied fairness method on ACSTravel- Time (top left), ACSMobility (top right), ACSEmployment (bottom left), and MEPS (bottom right), with 95% confidence intervals.
A.4 EXPERIMENTS WITH BINARY SENSITIVE GROUPS
While compatibility with more than two sensitive groups is arguably essential for real-world applica- bility fairness intervention, it is common among the fair MI. literature to propose and evaluate methods considering only two groups (Zemel et al. 2013; Agarwal et al. 2018 Cruz et al. (023).
this binary-group setting, constrained optimization methods only have to consider two constraints:
1|$ = 0, Y 0
= 1,8 o]
FPR constraint
P 118 0.Y - y -
<
TPR constraint
respectively, constraint on group-specific FPR, and another on group-specifie TPR, with some small slack. By relaxing the equalized odds problem to only two constraints we expect to provide fairness-constrained methods with the best chance at disproving paper hypothesis.
20

---

Published as a conference paper at ICLR 2024
Figure A14 (as Figure 7 shows results of applying the experimental procedure detailed in Section 2.2 to sablse the ACS datasets: only samples from the two largest sensitive groups are used (White and Black). We observe substantially lower constraint violation across the board, both for unconstrained and fairness-aware models. In fact, even unconstrained unprocessed models (m* on each plot) achieve below 0.1 constraint violation 4 datasets when using binary groups (all but ACSIncome), and below 0.01 on 2 datasets (ACSMobility and ACSEmploymen These results arguably discourage the use of binary sensitive groups the ACSMobility and ACSEmployment datasets for faimess benchmarking, as very low disparities are effortlessly achieved.
see Figure (A14).
ACSTravelTime (test)
ACSMobility (test)
0.10
0.06
base model
GBM
constraint violation
0.08
constraint violation
NN
RF
0.04
None
0.03
CR
EG
GS
0.01
(selected val.)
0.00
0.00
0,750
0.755
0.760 accuracy
0,765
0.770
postprocessing
accuracy
confidence interv.
ACSEmployment (test)
0.030
base model
GBM
0.025
constraint violation
NN
RF
0.020
fairness method
None
CR
EG
0.010
GS
FairGBM
0.005
(selected on val.)
postprocessing of
0.826
0.828 0.830 0.832 0.83 accuracy
confidence interv.
Figure A14: [Binary protected groups] Detailed view of the postprocessing Pareto frontier on the ACSTravelTime (left), ACSMobility (right), and ACSEmployment (bottom) datasets, when using only samples of the two largest groups (White and Black). Note the significantly reduced y axis range (constraint violation) when compared with results using four sensitive groups.
A.5 RESULTS ON THE MEPS DATASET
The Medical Expenditure Panel Survey (MEPS) (Blewett et al. 2021 dataset consists of large-scale surveys of families and individuals across the United States, together with their medical providers and employees. MEPS collects data on the health services used, costs and frequency of services, as well as demographic information of the respondents. The goal is predict low (< 10) or high (2 10) medical services utilization. Utilization is defined as the yearly sum total of office-basce visits, hospital outpatient visits, hospital emergency room visits, hospital inpatient stays, or home health care visits. Exact data proccescssing is made available in the supplementary materials We use survey panels 19 and for training and validation (data is shuffled and split 70%/30%) collected in 2015 and beginning of 2016 and survey panel 21 for testing collected in 2016. In total, the MEPS dataset consists of 49075 samples, 23380 of which are used for training. 10020 for validation,
and 15675 for testing, making it over order of magnitude smaller than the smallest ACS dataset in our study. We use race as the sensitive attribute, with non-overlapping groups as determined by the panel data: Hispanic, Non-Hispanic White, and Non- White.
21

---

Published as a conference paper at ICLR 2024
Figure A15 shows results of conducting the experiment detailed in Section 2.2 on the MEPS dataset. We note that the variance of results is the largest among all studied datasets, as evidenced by the wide confidence intervals. This is most likely due to the small dataset size. It is also possible that the mr model on smaller datasets (such as MEPS) could produce scores that are farther from Bayes optimality than those of m larger datasets (such ACS). We hope that study motivates additional empirical work on when exactly the optimality of postprocessing breaks in practice. We recall that, although no counter-example was observed among 000 trained models, there known edge-cases where postprocessing is sub-optimal (Woodworth et al. 2017). Overall, empirical results on the MEPS dataset are in accordance with those observed on the ACS datasets: the most accurate unconstrained model can be postprocessed to match dominate any other fairness-aware model.
MEPS (test)
0.20
constraint violation
0.15
0.10
0.05
0.00
0.835
0.840
0.845
0.850
0.855
0.860
0.865
accuracy
base model
falmess method
LFR
GBM
None
FairGBM
m (accuracy=O.Be)
NN
EG
postpracessing m 95% confidence interv.
RF
GS
Figure A15: Detailed view of the postprocessing Pareto frontier of m² on the MEPS dataset. Note the substantial variance in results, as shown by the wide postprocessing confidence intervals.
A.6 RANKING PRESERVATION BETWEEN UNPROCESSED AND POSTPROCESSED VERSIONS
Figure A16 akin to Figure shows real-data examples of the uuprocessing-pstproccstii postprocessing experimental setup described in Section The three plot panels show: (left) original results, (middle) results after unprocessing all models, and (right) original results with postprocessing curves overlaid.
We recall that the main experimental results (in Section 3) show that postprocessing the model with highest accuracy Pareto-dominates all other models (both Fairness-aware and standard models). In this section, present another perspective the same empirical insight: given specific incomparable models (A and B), the postprocessing curve of the model with highest unprocessed accuracy will Pareto-dominate the postprocessing curve of the model with lower unprocessed accuracy. That is. while Figure 6 compares postprocessing to all other fairness interventions, Figure A16 compares postprocessing to postprocessing. this scenario, the same empirical insight is confirmed: taking the model with highest accuracy is superior at all levels of fairesss constraint violation.
In summary, when near Bayes optimality model rankings are maintained across all postprocessing relaxations, i.e., if A > B", then (A) > πT,(B) Vr E [0, 1]. We know this to be true on both extremes (r = 0Vr 1) for Bayes optimal model (Hardt et al., 2016): achieves optimal accuracy, accuracy. At the same time, know and its postprocessing achieves optimal counter-examples (Woodworth et al. (017). The focus this to be false on some carefully constructed of the present work is to study whether this ranking is generally maintained in practice, on real-world data. This hypothesis is confirmed on all experiments conducted throughout the paper.
We only compare models that Pareto-dominant among their algorithm cohort.
22

---

Published as a conference paper at ICLR 2024
Original
Unprocessed
Postprocessed
0.6
constraint violation
GBM
postprocessing
0.5
GBM, EG
95% conf. interv.
0.4
J
0.3
8
uniproc
0.2
0.1
0.0
0.810
0.815 accuracy
0.820
0.810
0.815
0.820
0.810
0.815 accuracy
0.820
accuracy
Original
Unprocessed
Postprocessed
constraint violation
0.6
GBM
postprocessing 95% conf. interv.
RF, EG
0.4
unproc
0.2
+
0.80
0.82
0.80 accuracy
0.82
accuracy
accuracy
Original
Unprocessed
Pastprocessed
constraint violation
0.6
GBM
postprocessing 95% conf. interv.
NN, CR
0.4
unpro
0.2
0.0
0.80 accuracy
0.82
0.80 accuracy
0.82
accuracy
Original
Unprocessed
Postprocessed
constraint violation
0.6
RF
postpracessing 95% conf. interv.
NN, CR
0.4
unproc
0.2
0.79
0.80
0.81
0.79
0.80
0.81
0.79
0.80
0.81
accuracy
accuracy
accuracy
Figure A16: Comparison between postprocessing results between a variety of model pairs. Each model selected as maximizing accuracy (model A) maximizing a weighted average between accuracy and fairness (model B) among all models of the same algorithm cohort. Selection is performed on validation data, and results are shown on withheld test data; hence why some models may not be exactly at the Pareto frontier of their cohort. Results shown for the ACSIncome dataset.
23

---

Published as a conference paper at ICLR 2024
One final noteworthy point is that unconstrained models are not significantly affected by unprocessing, occupying approximately the same fairness-accuracy accuracy region before and after optimization over group- specific thresholds (e.g.. compare A with A in Figure A This is expected,
unconstrained learning optimizes for calibration by group (Liiu et al. 2019). Ply = 1| = r; S = 8) = r. Vs E S, which leads to the same loss moss-minimizing threshold for all groups (further details in Appendix (
A.7 UNPROCESSING VS UNCONSTRAINER LEARNING
As per Section 3 the best performing inprocessing fairness interventions are EG and FairGBM (i.e.. highest drreto-dominated area). In this section, we assess how unconstrained learning compares to unprocessing model that was trained using either of these faireess interventions. Ideally, if enforcing the fairness constraint in-training did not hinder the learning process, we'd expect unprocessed models to approximately occupy the same fairness-accuraee region as unconstrained models.
Figure A17 shows results before and after unprocessing fairness-constraine models on the AC- SIncome dataset. Unprocessing is done on validation data, and results are shown on withheld test data. The plots show that, after unprocessing, fairness-constrained models are naturally brought to similar levels of constraint violation unconstrained models. While overlap between unconstrained and coaness-constrained models was previously minimal or non-existent (left plots), these models form clearly overlapping clusters after unprocessing (right plots). Figure A18 shows similar results before and after unprocessing niimess-constrained models, as well as results after postprocessing unconstrained models. As evident in the plots, unprocessing brings fairness-constrained models to the high-accuracy and high-disparity region that was previously occupied solely by unconstrained models; while postprocessing brings unconstrained models to the low-disparity region previously occupied solely by can be seen as the inverse mapping of postprocessing. With these plots we aim to bring attention to the interchangeability of the underlying scores produced by both unconstrained and constrained models. Whether we want to deploy a fainnes--oonstaained or an unconstrained classifier can be chosen after model training, by postprocessing fulfillment. Finally, postprocessing has the added advantage of better-tuned of constrait fulfillment, models that were trained in from a wide variability of constraint fulfillment (orange markers of left-most plots).
models. This motivates the naming of unprocessing, as it
high-performing model to the appropriate value
manner suffer
Original
Unprocessed RF, EG
constraint violation
RF
0.5
RF, EG
0.3
0.2
0.1
0.0
0.78
0.79
0.80
0.81
0.78
0.79
0.80
0.81
accuracy
accuracy
Original
Unprocessed NN, EG
0.6
constraint vío violation
NN
NN, EG
0.4
0.3
0.2
0.1
0.0
0.790
0.795
0.800
0.805
0.790
0.795
0.800
0.805
accuracy
accuracy
Figure A17: ACSIncome test results before (left) and after (right) unprocessing constrained models.
24

---

Published as a conference paper at ICLR 2024
Original
Unprocessed GBM, EG
Postprocessed GBM
constraint violation
GBM GBM, EG
0.80
0.81
0.82
0.80
0.81 accuracy
0.82
0.80
0.81
0.82
accuracy
accuracy
Original
Unprocessed FairGBM
Pastprocessed GBM
constraint violation
0.5
0.4
0.3
0.2
GBM
0.1
FairGBM
0...
0.0
0.80
0.81
0.82
0.80
0.81
0.82
0.80
0.81
0.82
accuracy
accuracy
accuracy
Figure A18: ACSIncome test results using GBM as the hase model. Left: original results. Middle: after unprocessing Sairness-constrainee models. Right: after postprocessing unconstrained models.
B EXPERIMENT RUN DETAILS
All experiments
as jobs submitted to centralized cluster, running the open-source HTCondor scheduler. Each job was given the same computing resources: CPU. Compute nodes AMD EPYC 7 7666 64-core CPUs. No GPUs were used. Memory was allocated as required for each algorithm: all jobs were allocated at least 16GB of RAM; GS and EG jobs were allocated 64GB of RAM as these ensembling algorithms have increased memory requirements.
An experiment job accounts for training and evaluating single model on given dataset. That is, 1 000 models were trained on each dataset (50 per algorithm type), totaling 11 000 models trained: 5 000 for the main ACS experiment using 4 sensitive groups, 5 000 for the ACS experiment using 2 sensitive groups, and 000 for the MEPS dataset experiment. Overall, the median job finished in 10.3 minutes, while the average job lasted for 112.0 minutes (most models are fast, but some fairnsss- aware models such as EG take a long time to fit, as seen in Figures and A13). Compute usage was: 10 528 CPU hours for the main 4-group ACS experiment, 967 CPU hours for the binary group ACS experiment (Appendix A.4 and 31 CPU hours for the MEPS dataset experiment (Appendix A.5)) Total compute usage was 20 526 CPU hours, which amounts to 14 days 64-core node. Detailed per-job CPU usage is available under folder results t of the supplementary materials!!
Complete code base required to replicate experiments is provided part of the supplementary materials, together with exact evaluation results for each trained model!"
C THRESHOLDING GROUP-CALIBRATED PREDICTORS
In this section we provide proof for the following statement: for any classifier with group-calibrated scores (Equations M. the group-specific decision thresholds that minimize the classification loss among each group all take the same value, f. = to, Va, E S, which is fully determined by the loss
((1.0) function, t, ((100)40011 Vs S.
25

---

Published as a conference paper at ICLR 2024
Proof. Given joint distribution over features, labels, and sensitive attributes (X, S), binary classification loss function é {[, 1}2² + Rt, predictive scores R = f(X), and binary predictions
== 4 ( E T C R. Assume the scores R are group-calibrated (Barocas et al. 2019). i.e.:
P Y 1 r, s] =r;
Vr [0, 1], E
(8)
PP y = 0 Rt r, = s| =1-
Vr E [0, V
(9)
We want to minimize the expected loss among samples of group 8, (t) E [(Y 11
L.(1) = ((1,))
1, = o|s و ((0, = 0,
(10)
assuming w.l.o.g. no cost for correct predictions ((0,0) = -
We have:
y 1, 0ss = P y = = =
0,5 0,5 P 0|5 s) (t) P Y 0 s) 0 = 1,5
y 0, 115 =
y = -
P|Y = 1|5 = =} - 11 (1) P P 1| }]
where (1) and A. (t) are, respectively, the False Positive Rate (FPR) and the False Negative Rate (FNR) among samples of group 8, as functions of the chosen group-specific threshold L. We can trade-off FPR and FNR by varying the threshold, leading to 2-dimensional curve known as the Receiver Operating Characteristic (ROC) curve.
Furthermore, given the conditional density function of R given S = s, (r), we have:
() (1) P Y 1 0,
P R 2 1 = 0,5
P 0| 2 S,
P
P = 0
(1 r) PR. (T)
P = 0 = dr,
using calibration (Eq.
(t- 1) Pm. (t)
01
P|Y = 0|$ 8)
and.
A5 (t) = P y 0|Y 1,$=
= PR < 1 = 1,5 s]
P[Y = 1 < t, St 8) ...P.R <
P.Y = =|
(r)
J0 P = 1|S dr,
using calibration (Eq.
OREN
t (f)
0t
P 1|S =
The threshold tg that minimizes the group-specific loss (t) is a solution to aL, = 0, where: at
(t) ((1,0)- - (h,) P Y 0 s] 1). (T((() -P = 1| = s)
ÖL, ((100) (6- 1) pa. (1) + 10 (p)
21
Hence, for group-calibrated predictor (fulfilling Equations 9 for any group E S, the optimal group-specific decision threshold t, does not depend on any group quantities, and given by:
((1,0)
t, E(1,0) + E(0, 1)"
26


Reviews:
[{"review_id": "ngmy5jMDXK", "reviewer": "Reviewer_RSrR", "summary": "The paper considers the relation fairness-accuracy tradeoff. In particular, the paper considers the relation between fairness (in terms of Equalized Odds) violation and accuracy of the predictor, before and after \"unprocessing\", and claims based on empirical observations that any Pareto-optimal tradeoff between accuracy and empirical EOdds violation can be achieved by postprocessing.\n\n---\n\n**Post-rebuttal**\n\nThe authors claim that Theorem 5.6 of Hardt et al. (2016) strengthens the result of empirical studies considered in the work. It would be helpful if such discussion can be incorporated in the manuscript to help readers understand this connection. After engaging with authors and going through comments by other reviewers, I have increased my evaluation from 5 to 6.", "questions": "__Question 1__: what is the exact relation between unprocessing and postprocessing?\n\nBased on Hardt et al. (2016), the postprocessing strategy for EOdds is trading off True Positive Rates (TPRs) and False Positive Rates (FPRs) across different demographic groups. Such procedure is _oblivious_, in the sense that only the joint distribution $(A, Y, \\hat{Y})$ are utilized in the postprocessing procedure. If this specific way of postprocessing is of interest in the paper, I am not sure how to understand the relation between unprocessing and postprocessing. I can see why authors draw an analogy between unprocessing and the inverse of postprocessing. According to Equation 1, unprocessing starts from the postprocessed $\\hat{Y}$ and aims to find the unconstrained optimized predictor. How can we do that with obliviously postprocessed $\\hat{Y}$? How to make sure the unprocessed predictor has a sensible mapping from input features to target variable?\n\n\n\n__Question 2__: regarding the claim that _any_ Pareto-optimal tradeoff can be achieved by postprocessing\n\nFollow up to Question 1, if the postprocessing is defined as in Hardt et al. (2016), it would be very helpful if authors can provide a clear characterization of the relation between unprocessing and such definition of postprocessing, so that readers can understand why unprocessing is a helpful analyzing tool to understand the importance of postprocessing. Empirical evaluations can be strengthened by some certain level of theoretical analysis to make the results and message more convincing.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The strength of the paper comes from the extensive empirical experiments and the efforts to present the observation (that Pareto-optimal tradeoff can potentially be achieved by postprocessing. The experiments are conducted on a relatively new data set (compared to standard baseline data sets in the literature), and the setup includes exact and relaxed EOdds (Hardt et al., 2016).", "weaknesses": "The weakness of the paper comes from the lack of a certain level of theoretical derivation to justify the empirical findings. The proposed term \"unprocessing\", as noted by authors, \"roughly corresponds to the inverse of postprocessing\", is more of less confusing (for reasons detailed in Section __Questions__). While one can observe from extensive empirical evaluations that Pareto-optimal tradeoffs can be achieved (setting aside numerical indeterminacy), there is a worry that the results can only provide limited insight regarding the not-clearly-motivated unprocessing procedure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y3RxJ9zg4F", "reviewer": "Reviewer_QHJ8", "summary": "There have been many proposals in the recent literature to train fair ML models. This paper evaluates thousands of such models, and finds that a simple postprocessing technique achieves the fairness-accuracy Pareto frontier.", "questions": "None", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This type of comprehensive benchmarking of thousands of models adds a ton of value to the algorithmic fairness literature. I think the result that a simple postprocessing step achieves the Pareo frontier is very significant. I applaud the authors for taking on this task.", "weaknesses": "None", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4LGyD8ke3u", "reviewer": "Reviewer_tygt", "summary": "This work performs an extensive benchmark for 1000 models to compare the error rate disparity and accuracy trade-offs. To make a fair comparison, the constrained models, either trained with pre-processing techniques or in-processing learning constraints, are unprocessed to yield the corresponding optimal unconstrained model. Through these assessments, the authors convey a straightforward yet crucial finding: achieving fairness is best attained by training the most effective unconstrained model available and subsequently employing post-processing techniques to fine-tune the thresholds.", "questions": "- How efficient is it to solve the LP problem? Can I just exhaustively search all the combinations of the thresholds and plot the Pareto frontiers of the fairness-accuracy trade-offs?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- I like the way the authors pose the narrative of this work. The structure is well-defined, presenting experimental details clearly. \n-  I think the concept of \"unprocessing\" is a novel and effective method to discover the optimal unconstrained model corresponding to the constrained models.\n- In general, the evaluation is solid and can provide enough insights to the practitioners.\n- In my personal opinion, this paper satisfies my standard of acceptance but does not reach the rating of 8. So I would rather recommend a rating of 6.", "weaknesses": "- I would like to see a comparison between the real unconstrained model and the unprocessed version of the constrained model. This comparison is necessary and could enhance the claim that unprocessing can be applied to find the optimal unconstrained model.\n- Section 4 is just a standard LP problem in solving Equal Odds with post-processing. It is not novel and there is no need to write down it in the main paper.\n- The author has admitted that their evaluation is only applied to tabular data, with a focus on 5 different partitions of the FolkTables dataset. It would be interesting to see how the conclusions can still be generalized to tasks with rich representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oHTqaUcKxl", "reviewer": "Reviewer_4YF6", "summary": "The problem the paper considers is building accurate models subject to a fairness constraint. There are many ways of building models but it is difficult to compare between different methods because a) the model performance depends on the underlying classifier and b) the models satisfy the fairness constraint up to different relaxations.\n\nThis paper seeks to solve both problems and run a large experiment on many different methods and models. They start with an approach they call \"unprocessing\" which takes the underlying classifier and removes the fairness constraint. In this way, different models can then reasonably be compared to each other. They then postprocess the classifiers to achieve the fairness constraint. There is an optimal way to achieve the postprocessing so this step also lets different models be compared to each other.", "questions": "Is there anything in my assessment you disagree with?\n\nHave you considered putting your approach into a popular package so that researchers can quickly and easily compare their models?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A simple way of comparing models with different fairness constraints. I hope this becomes widely adopted and used before people introduce their XYZ fairness algorithm.\n\n2. A comprehensive evaluation of lots of models on four data sets. I especially liked two observations from their results:\n\n* Models subject to a fairness constraint can actually achieve higher accuracy than models not subject to a fairness constraint when compared fairly (pun intended). The explanation they give is that fair training can take longer and use more resources because of the complexity in the algorithms.\n\n* In their words: \n\n\"Crucially, postprocessing the single most accurate model resulted in the fair optima for all values of fairness constraint violation on all datasets, either dominating or matching other contender models (within 95% confidence intervals). That is, all optimal trade-offs between fairness and accuracy can be retrieved by applying different group-specific thresholds to the same underlying risk scores.\"\n\nI think this is intuitively obvious and it's nice to see experimental confirmation.\n\n3. A technical description of how to achieve relaxed parity.", "weaknesses": "1. I found the technical description of how to achieve relaxed parity jarring from the rest of the paper. I would have liked this section to be longer and for more explanations there. I did find the figures quite helpful in understanding it.\n\n2. A big selling point of the paper is the extent of their experiments. I think the reason they were able to do this is because they had access to a ton of compute. All the data sets and models (I believe) are easily accessible. If this is the case, I'm not sure that \"having lots of compute\" is really something we should reward as a contribution.\n\n3. I found their approach intuitively obvious: Of course given a classifier, you can vary how much it violates a reward constraint in an optimal way. So I think the contribution here would be because (it seems like) no one has done this before rather than because it is so interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)