A Near-Optimal Algorithm for Debiasing Trained Machine Learning Models
Authors: Ibrahim Alabdulmohsin
Mario Lucic
Venue: NeurIPS 2021 Poster (2021)
Keywords: Fairness
Classification
Statistical Parity
Deep Learning

Abstract: We present a scalable post-processing algorithm for debiasing trained models, including deep neural networks (DNNs), which we prove to be near-optimal by bounding its excess Bayes risk.  We empirically validate its advantages on standard benchmark datasets across both classical algorithms as well as modern DNN architectures and demonstrate that it outperforms previous post-processing methods while performing on par with in-processing. In addition, we show that the proposed algorithm is particularly effective for models trained at scale where post-processing is a natural and practical choice.

A Near-Optimal Algorithm for Debiasing Trained Machine Learning Models
Ibrahim Alabdulmohsin Google Research, Brain Team Zürich, Switzerland abomohsin@google. com
Mario Lucic Google Research, Brain Team Zürich, Switzerland lucic&google.coo
Abstract
We present a scalable post-processing algorithm for debiasing trained models, including deep neural networks (DNNss, which we prove to be near-optimal by bounding excess Bayes risk. We empirically validate its advantages on standard benchmark datasets across both classical algorithms well modern DNN architectures and demonstrate that it outperforms previous post-processing methods while performing par with in-processing. In addition, show that the proposed algorithm is particularly effective for models trained scale where post-processing is natural and practical choice.
Introduction
Background. Machine learning is increasingly applied to critical decisions which can have lasting impact individual lives, such as for credit lending [Bruckner, 2018], 181 medical applications [Deo, 2015], and criminal justice [Brennan et al., 2009]. Consequently, it is imperative to understand and improve the degree of bias of such automated decision-making.
Unfortunately, despite the fact that bias (or "fairness") is a central concept in our society today, is difficult to define it in precise terms. In fact, as people perceive ethical matters differently depending on a plethora of factors including geographical location culture [Awad al., 2018], no universally- agreed upon definition for bias exists. Moreover, bias may depend on the application and might even be ignored favor accuracy when the stakes are high, such as in medical diagnosis [Kleinberg et al., 2016]. As such, it is not surprising that several measures of bias have been introduced, such as statistical parity [Dwork et al., 2012, Zafar et al., 2017a], equality of opportunity [Hardt et al., 2016], and equalized odds [Hardt et al., 2016, Kleinberg et al., 6], and these not generally mutually compatible [Chouldechova, 2017, Kleinberg al., 2016].
Let X' be an instance space and let ) = {0, be the target set in binary classification problem. In the fair classification setting. we may further assume the existence of a sensitive attribute {1..... K}. where s(z) if and only if x E Xr for some total partition X' Jg For example, X' might correspond to the set job applicants while s indicates their sex. Then, commonly used criterion for fairness is to require similar mean outcomes across the sensitive attribute (a.k.a. statistical parity) [Dwork et al., 2012, Zafar al., 2017a, Mehrabi al., 2019]:
Definition (Statistical Parity). Let X be an instance space and X' = U be total partition of X. A predictor X -> [0, satisfies statistical parity across all groups XI,..., Xx if:
max Ex [x(()) |x Xx] min Ex (x(()) x Xx Se, e, AE(K]
kE(K]
where [K] denotes the set [1....,]..
Our main contribution is to derive a near-optimal recipe for debiasing models, including deep neural networks, according to Definition 1. Specifically, we formulate the task of debiasing learned models
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

as a regularized optimization problem that is solved efficiently using the projected SGD method. We show how the algorithm produces thresholding rules with randomization near the thresholds, where the width of randomization is controlled by a regularization hyperparamter. We also prove that randomization near the threshold is, in general, necessary for Bayes risk consistency. In Appendix D, we show how the proposed algorithm can be modified to handle weaker notion of bias as well. We refer to the proposed algorithm as Randomized Threshold Optimizer (RTO).
Besides the theoretical guarantees, we empirically validate RTO on benchmark datasets across both classical algorithms as well as modern DNN architectures. Our experiments demonstrate that the proposed algorithm significantly outperforms previous post-processing methods and performs competitively with in pr-prcessing (Section 5). While we focus on binary sensitive attributes in the experiments, our algorithm and its guarantees continue to hold for non-binary attributes as well.
In addition, we show that RTO is particularly effective for models trained at scale where post- processing is a natural and practical choice. Qualitatively speaking. for a fixed downstream task D, such as predicting facial attributes in the CelebA dataset [Liu et al., 2015]. we say that the model is "trained at scale" if it is both: (1) heavily overparameterized, and (2) pretrained on large datasets before fine-tuning on the downstream task D. We show that the impact of debiasing models on their performance using the proposed algorithm can be improved with scale.
Remark. Because "bias" is a societal concept that cannot be reduced to metrics such as statistical parity [Chouldechova, 2017, Dixon et al., 2018, Selbst et al., 2019], our conclusions do not necessarily pertain to "fairness" in its broader sense. Rather, they hold for the narrow technical definition of statistical parity. Similarly, we conduct experiments [Liu et al., 2015] and COCO [Lin et al., 2014], which are commonly used in the literature, as a way of validating the technical claims of this paper. Our experiments are, hence, not to be interpreted as an endorsement of those visions tasks, such as predicting facial attributes.
standard benchmark datasets, such as CelebA
2 Related Work
Algorithms for fair machine learning can be broadly classified into three groups: (1) pre-processing methods, (2) in- in-processing methods, and (3) post-processing methods [Zafar et al., 2019].
Preprocessing algorithms transform the data into a different representation such that any classifier trained on it will not exhibit bias. This includes methods for learning fair representation [Zemel et 2013, 2013,, Lum and Johndrow, 2016, Bolukbasi et al., 2016, Calmon et al., 2017, Madras et al., 201 8, Kamiran and Calders, 2012], label manipulation [Kamiran and Calders, 20091, data augmentation [Dixon et al., 2018], or disentanglement [Locatello et al., 2019].
On the other hand, in-processing methods constrain the behavior of learning algorithms in order to control bias. This includes methods based on adversarial training [Zhang et al., 2018] and constraint- based classification, such as by incorporating constraints on the decision margin [Zafar et al., 2019] or features [Grgié- Hlača et al., 2018]. Agarwal et al. [2018] 8] showed that the task of learning an unbiased classifier could be reduced to a sequence of cost-sensitive classification problems, which could be applied to any black-box classifier. One caveat of the latter approach is that requires solving a linear program (LP) and retraining classifiers, such as neural networks, many times before convergence.
The algorithm we propose in this paper post-processing method, which can be justified theo- retically [Corbett-Davies et al., 2017, Hardt et al., 2016, Menon and Williamson, 2018, Celis et al., 2019]. Fish al. [2016] and Woodworth et al. [2017] fall under this category. However, the former only provides generalization guarantees without consistency results while the latter proposes a two- stage approach that requires changes to the original training algorithm. Kamiran al. [2012] also proposes a post-processing algorithm, called Reject Option Classifier (ROC), without any theoretical guarantees. In contrast, our algorithm is Bayes consistent and does not alter the original classification method. In Celis et al. [2019] and Menon and Williamson [2018]. mnstance-dependen rules are also learned. However, our algorithm also learns to randomize around the threshold (Figure (a)) and this randomization is key to our algorithm both theoretically as well as experimentally (Appendix and Section 5). Hardt et al. [2016] learns a randomized post-processing rule but our proposed algorithm outperforms it in all of our experiments (Section 5). Also, [Wei et al., 2019] a post-processing method but requires solving non-linear optimization problem (for the dual variables) via ADMM and provides guarantees for approximate faimess only.
thresholding
2

---

Woodworth et al. [2017] showed that the post-processing approach can be suboptimal. Nevertheless, the latter result does not contradict the statement that our post-procsssing rule is near-optimal because we assume that the original classifier outputs a score (i.e. a monotone transformation of an approximation to the posterior p(y = 1 x such as margin or softmax output) whereas Woodworth et al. [2017] assumed that the post-processing rule had access to the binary predictions only.
We argue that the proposed algorithm has distinct advantages, particularly for deep neural networks (DNNs). First, stochastic convex optmiization methods can scale well to massive amounts of data [Bottou, 2010], which is often the case in deep learning today. Second, the guarantees provided by our algorithm hold w.r.t. the binary predictions instead of using a proxy, such as the margin as in some previous works [Zafar al., 2017b, 2019]. Third, unlike previous reduction methods that would require retraining a deep neural network several times until convergence [Agarwal et al., 2018], which can be prohibitively expensive, our algorithm does not require retraining. Also, post-processing can be the only available option, such as when using machine learning service with out-of-the-bon predictive models or due to various other constrains in data and computation [Yang et al., 2020b].
3 Near-Optimal Algorithm for Statistical Parity
Notation. We reserve boldface letters for random variables (e.g. x), small letters for instances (e.g- r), capital letters for sets (e.g. X), ), and calligraphic typeface for universal sets (e.g. the instance space X). Given a set S. 1 sx) E (0, 1) is its characteristic function. Also, we denote [N] = (1....,N7 N)
and [x] = ma æ). We reserve 17(x) for the Bayes regressor: m)x)) = p(y = 1| =x).
Algorithm. Given a classifier outputting a probability score ply = || x),, let f(x) 2p = 1 1|x= x) 1. We refer to f(x)) E [-1, +1] as the classifier's output. Our goal is to post-process the predictions made by the classifier to control statistical parity with respect to a sensitive attribute X' + [K] according to Definition 1. To this end, instead of learning a deterministic rule, we consider randomized prediction rules h hxx), where lx(x) is the probability of predicting the positive class given f(x) and sx). Note that we have the Markov chain:
(s(x), fxx) - h(x).
simple approach of achieving c statistical parity is to output constant prediction in each subpop- ulation, which is clearly suboptimal in general. As such, there is tradeoff between accuracy and fairness. The approach we take this work is to modify the original classifier such that the original predictions are matched as much as possible while satisfying the fairness constraints. Minimizing the probability of altering the binary predictions of the original classifier can be achieved by maximizing the inner product Ex [h(x) fx) (cf. Appendix B). However, maximizing this objective alone leads to deterministic thresholding rules which have a major drawback as illustrated in the following example.
Example 1 (Randomization is necessary). Suppose that X' = {-1,0, 1 where p(x =1) = 1/2.
p(x 0)- = 1/3 and p(x = 1) 1/6. Let 17(-1) = 0, n0) = 1/2 and 11(1) = 1. In addition, let s E (0.1) be a sensitive attribute, where p(s = 1/x -1) = 1/2, p(s = =I 0) = 1, and p(s
1/x = 1) O. Then, the Bayes optimal prediction rule h (z) subject to statistical parity (e 0)
satisfies: p(h*(x) 1|x 1) O. p(h"(x) (x) = 1|x 0) 7/10 and p(h" (x) 1 1) = 1.
As result, randomization close to the threshold is necessary in the general case achieve Bayes risk consistency. This conclusion in Example 1 remains true with approximate fairness (e < 12/70).
In this work, we propose to achieve this by minimizing the following regularized objective for some hyperparameter 0:
(7/2) (2 (b(x))] [xx)) fxx))
(4)
We prove in Appendix A that this regularization term leads to randomization around the threshold, which is critical, both theoretically (Section 4 and Appendix B) and experimentally (Section 5). Informally, 7 controls the width of randomization as illustrated in Figure 1.
Let X' = U2 X2 be a total partition of the instance space according to the sensitive attribute : X -> [K]. Denote finite training sample by S = ((11,)1)..... (#N, yN)} and write S₃ = S n Xg. For each group Sr, the fairness constraint in Definition I over the training sample can be written as:
Σ (h(2:)) p) s 2
(5)
S
S.ES
for some hyper-parameter p E [0, 1). Precisely, if the optimization variables Mx;)) satisfy the constraint (5), then Definition I holds over the training sample by the triangle incquality. Conversely,
3

---

Algorithm 1: Pscudocode of the Randomized Threshold Optimizer (RTO).
Input: 7 0 E 1); 2 0; f: : 1-1,1];8: [K]
Output: Prediction rule: (x)) Training:
Initialize (Az, p1)........ (AK, (K) to zeros. Then, repeat until convergence:
1. Sample instance p(z)
2. Perform the updates:
Au(s) [hu(s)- -19..co"". Ma(x) (anc))) 7902(0) I+
(1)
where:
+p a
E, (f(x) - (Macs) - -1,0s))
M).(1 = 2
d)a(s)
i
&, (f(x) (ha(s) Hate)))
gr.(s)
(p(())
and:
w²
(1w) 27
140 S w s + (w - 20 Tfw > 7
(2)
Prediction:
Given instance in the group Xx. predict the label +-1 with probability le (x), where:
(x) [mnn(1 (f(x)) As + pc)/7}*
if Definition holds, then the constraint (5) also holds where:
2p max E. [x() S&]
min [x(((x) |x E St).
KE[K]
kE[K]
Therefore, to learn the post-processing rule hxx), we solve the optimization problem:
min 0SM(2EE1
Σ
(w/2)
((z))((z))
FES
s.t.
Vk E [K]]
Σ (h(z)) p)1 S Ek: F
(6)
in which G = |S&| c/2 for all & E [K]. Using Lagrange duality we show in Appendix that solving the above optimization problem is equivalent to Algorithm In Appendix D, show that if an alternative formulation can be used to minimize the same objective while satisfying the fairness constraint but without introducing hyperparameter p. To reiterate, p [0, tuned via validation dataset and >> 0 is a hyperparameter that controls randomization.
Theoretical Analysis
Our first theoretical result is to show that RTO satisfies the desired fairness guarantees.
Theorem (Correctness). Let h : X + [0, 1) the randomized predictor Equation learned by applying the update rules in Equation on a fresh sample of size N until convergence with learning rates satisfying the Robbins and Monro condition [Robbins and Monro, 1951]. Then, h satisfies e
statistical parity the training sample. Moreover, with probability of at least 1 5, the following bound on bias holds w.r.t the underlying distribution:
2 log eN
log 2K
max E[((x) x Xx] kE[K]
min E(h(x) e Xx) S e + 8V KE[K
+ 2
Proof. The proof is in Appendix A. We make use of strong duality, which holds by Slater's condition [Boyd and Vandenberghe, 2004]. The update rules correspond to the projected SGD method on the dual problem. This establishes the guarantee on the training sample. For the underlying distribution, we bound the Rademacher complexity [Bousquet et al., 2003] of the function class H. Figure 1(a) by that of 0-1 thresholding rules over I. from which generalization bound is derived.
D

---

kx(x))
X
u0
0
0
5
10
15 Epoch
20
25
f(z
(a) Decision rule
(b) Convergence
Figure 1: (a) The learned post-process pest-processing rule (x)) in Equation 3 as a function of the classifier's
score f() over subpopulation. Randomization is applied when (x) E (0, 1). (b) The value of Ag ug is plotted against the number of epochs in projected SGD applied to the random forests classifier. The classifier trained the Adult dataset to implement statistical parity with respect to the sex attribute (cf. Section 5). We observe fast convergence in agreement with Proposition 1.
The following guarantee shows that the randomized prediction rule converges to the Bayes optimal unbiased classifier if the original classifier Bayes consistent.
Theorem 2. Let ht = arg minkeh. [x)(x) + yl, where H.e is the set of binary predictors on X' that satisfy fairness on the training sample according Definition for 2 Let h - [0, 1] be the randomized rule in Algorithm! 1. If h is trained on a fresh data of size N, then there exists value of p E [0, 1) independent of N such that the following holds with a probability of at least
8(2+ + 1) + - -1-
2K + 2 log
EI (x) fy31 [[((("(() (x) f
No
N
where n(x) ply 1|x x) is the Bayes regressor and K is the number of groups X
Proof. The full proof is in Appendix B. First, we show that minimizing the probability of error can be achieved by maximizing [x (2n)(x) -])]. We use the regularized loss instead, which is strongly convex. Using Lipschitz continuity of the decision rule when 7 > (cf. Figure 1(a)), and the robustness framework of Xu and Mannor [2012], prove a generalization bound and proceed with a series of inequalities to establish the main theorem.
Thus, if the original classifier is Bayes consistent, namely IE (2n(x) - ʃ(x) - 0 as the sample size goes to infinity, and if N + 70. + 0+ and 7NV 1/2 - oo, then EEi,, (x) y] - E(h((x) # y]. Hence, Algorithm converges the optimal prediction rule subject to the fairness constraints.
Convergence Rate, As we show in Appendix A, the update rules in Equation perform a projected stochastic gradient descent the following optimization problem:
min F = Ex (As(x) + (o0o) + (Aa(o) P.(x)) + (x( (Noco - (=())) 4.A20
(8)
where &, is given by Equation 2. The following proposition shows that the post-processing rule can be efficiently computed. In practice, we observe fast convergence as demonstrated in Figure ((b).
Proposition Let (0)) ()) 0 and write 1 X R² for the value of the optimization variables after t updates defined in Equation 1 for some fixed learning rate a = a. Let j (L/T) Et= (x) and X (1/T) D (x). Then,
E F < (1 (1+ 1I 113 11
(9)
2Ta
where F: BBK. x R + P. is the objective function in (8) using the averaged solution and X while F* is its optimal value. In particular, E|F F = (((KK/T) when O(/KTT).
The proof of Proposition is in Appendix C. As shown Figure ((a), the hyperparameter y controls the width of randomization around the thresholds. A large value of may reduce the accuracy of the classifier. On the other hand, 7 cannot be zero because randomization around the threshold is, in general, necessary for Bayes risk consistency as shown earlier in Example 1.
5

---

5 Experiments
Baselines and Experimental Setup. We compare against three post-processing methods: (1) the algorithm of Hardt et al. [2016] (2) the shift inference method, first introduced in Saerens et al. [2002] and used more recently in Wang et al. [2020b], and (3) the Reject Option Classifier (ROC) [Kamiran et al., 2012]. We also include the reduction approach of Agarwal et al. [2018] to compare the performance against in-proesssing rules. We briefly review each of these methods next.
The post-processing method of Hardt et al. [2016] is a randomized post- proccsscnssiing rule. It was originally developed for equalized odds and equality of opportunity, Nevertheless, it can be modified to accommodate other criteria, such as statistical parity [Agarwal et al., 2018, Dudik et al., 2020].
The shift inference rule, on the other hand, is a post-hoe correction that views bias as a shift in distribution, hence the name. It is based on the identity r(y|s,x) oc {(y|s,x) r(y,s)/9(x, s). which holds for any two distributions r and q on the product space of labels y. sensitive attributes s. and instances x if they share the same marginal r(x) = ((x) [Wang et al., 2020b]. By equating, g(y x) with the classifier's s output based on the biased distribution and r is x) with the unbiased classifier, the predictions of the classifier q(y|s, (s. x) can be post-hoc corrected for bias by multiplying its probability score with the ratio (()))((s///)m))))))))) ()),s).
The reject option classifier (ROC) proposed by Kamiran al. [2012] is deterministic thresholding rule. It enumerates all possible values of some tunable parameter @ up to a given precision, where 0 0 corresponds to the original classifier. Candidate thresholds are then tested on the data.
Finally, the reduction approach of Agarwal et al. [2018] is an in processing method that can be applied to black-box classifiers but it requires retraining the model several times. More precisely, let h be hypothesis in the space H and M be a matrix, Agarwal et al. [2018) 8J showed that minimizing the error of h subject to constraints of the form Mp(h)= S c, where juhh) is a vector of conditional moments on h of a particular form, can be reduced (with some relaxation) to a sequence of cost-sensitive classification tasks for which many algorithms can be employed.
We use the implementations of Hardt et al. [2016] and Agarwal et al. [2018] in the FairLearn software package [Dudik et al., 2020]. The training data used for the post-processing methods is always fresh sample, i.c. different from the data used to train the original classifiers. Specifically, we split the data that was not used in the original classifier into three subsets of equal size: (1) training data for the post-processing rules, (2) validation for hyperparameter selection, and (3) test data. The value of the hyper-parameter 0 of the ROC algorithm is chosen in the grid (0.01, 0.02, ..... 1.0]- In the proposed algorithm, the parameter 7y is chosen in the grid {0.01, 0.02, 0.05, 0.1, 0.2} while p is chosen in the gird E[y] + {0, 0.05, 0.1). All hyper-parametert are selected based on a separate validation dataset. For the in-processing approach, we used the Exponentiated Gradient method as proposed by Agarwal et al. [2018] with its default settings in the FairLearn package (e.g. max iterations of 50).
Tabular Data. We evaluate performance on two real-world datasets, namely the Adult income dataset [Kohavi, 1996] and the Default of Credit Card Clients (DCCC) dataset [Ych and Lien, 2009], both from the UCI Machine Learning Repository [Blake and Merz, 1998]. The Adult dataset contains 48.88 48.842 records with 14 attributes each and the goal is to predict if the income of an individual exceeds $50K per year. The DCCC dataset contains 30,000 records with 24 attributes, and the goal is to predict a client will default on their credit card payment. We set sex as a sensitive attribute. In DCCC, we introduce bias the training set to study the case in which bias shows up in the training data only (e.g. due to the data curation process) but the test data remains unbiased (cf. [Torralba and Efros, 2011] and [de Vries et al., 2019] who discuss similar observations in common benchmark datasets). Specifically, if 6(x) = y(x) we keep the instance and otherwise drop it with probability 0.5.
We train four classifiers: (1) random forests with depth 10, (2) K--N with k = 10, (3) two-layer neural network with 128 hidden nodes, and (4) logistic regression whose parameter C' is fine-tuned from a grid of values in logarithmic scale between 10 4 and 10t using 10-fold cross validation. The learning rate in our algorithm is fixed to 10 (K/T))/) where T is the number of steps, and 0.
Table (Top and Middle) shows the bias on test data after applying each post-processing method. The column marked as "original" corresponds to the original classifier without alteration. As shown in the table, the shift-inference method does not succeed at controlling statistical parity while ROC can fail when the original classifier's output is concentrated on a few points because it does not randomize.

---

Table 1: A comparison of four post-processing methods and the reduction approach of Agarwal et al. [2018] on 3 datasets. The classifiers are random forests (RF), k-NN, MLP, logisitic regression (LR), ResNet50 trained from scratch (R50/S), ResNet50 pretrained on ImageNet (R50/T), MobileNet trained from scratch (MN/S) and MobileNet pretrained on ImageNet (MN/I). Values in bold correspond to cases where debiasing fails. ROC may fail in k-NN and in neural networks because debiasing them can require randomization. Original bias in the dataset is provided in the leftmost column.
Bias
Dataset
Classifier
Original
RTO
Hardt, 2016
Shift Inference
ROC
Reduction
ADULT
RF
.38
.01
.01
.16
.02
.01
(Bias 19)
KNN
.24
.02
.01
.08
.08
.01
MI.P
.29
.01
.02
.10
.02
.01
LR
.39
.01
.02
.10
.01
.01
DCCC (Bias = 22)
RF
.07
.01
.01
.09
.02
.01
KNN
.10
.01
.01
.18
.02
.01
MLP
.13
.01
.01
.12
.02
.01
LR
.12
.01
.01
.13
.01
.01
CELEBA (Bias .33)
R50/S
.43
.01
.01
.38
.08
R50/1
.40
0.02
.01
.35
.15
MN/S
.35
.01
.01
.24
.01
MN/I
.38
.002
.002
.34
.10
CelebA Dataset. Our second set of experiments builds on the task of predicting the "attractiveness" attribute the CelebA dataset [Liu et al., 2015]. We reiterate that we do not endorse the usage of vision models for such tasks, and that we report these results because they exhibit sex-related bias. CelebA contains 202,599 images of celebrities annotated with 40 binary attributes, including sex. We use two standard architectures: ResNet50 [He et al., 2016] and MobileNet [Howard et al., 2017], trained from scratch or pretrained on ImageNet ILSVRC2012 [Deng et al., 2009]. We resize images to 224 x 224 and train with fixed learning rate of 0.001 until the validation error converges. We present the bias results in Table (bottom). We observe that randomization is indeed necessary: ROC and Shift Inference both fail at debiasing the neural networks because they do not learn to randomize when most scores produced by neural networks are concentrated around the set {-1, +1}.
Impact on Test Accuracy. As shown in Table 2, the proposed algorithm has a much lower impact on the test accuracy compared to Hardt et al. [2016] and even improves the test accuracy in DCCC because bias was introduced in DCCC to the training data only as discussed earlier. The tradeoff curves between accuracy and bias for both the proposed algorithm and Hardt et al. [2016) are shown in Figure 2 (LEFT). Also, for a comparison with in in-processing rules, we observe that the post-processing algorithm performs competitively with the reduction approach of Agarwal et al. [2018].
Impact of Scale. Models trained at scale transfer better and enjoy improved out-of disti--iitti robustness [Djolonga et al., 2021]. As these models are now often used in practice, we assess to which extent can these models be debiased while retaining high accuracy. We conduct 768 experiments on 16 deep neural networks architectures, pretrained on either ILSVRC2012, ImageNet-21k (a superset of ILSVRC2012 that contains k classes [Deng et al., 20091), or JFT-300M (a proprietary dataset with 300M examples and 18k classes [Sun al., 2017]). The 16 architectures are listed in Appendix E and include MobileNet [Howard et al., 2017]. DenseNet [Huang et al., 2017], Big Trasnfer (BiT) models [Kolesnikov et al., 2020], and NASNetMobile [Zoph et al., 201 8]. The classification tasks contain seven attribute prediction tasks in CelebA [Liu et al., 2015] as well as five classification tasks based on the COCO dataset [Lin et al., 2014]. We describe how the tasks were selected in Appendix E. The sensitive attribute is always sex in our experiments and all classification tasks are binary. Unless explicitly stated, we use = 0. Morcover, in the COCO dataset, we follow the procedure of [Wang et al., 2020a] in inferring the sensitive attribute based on the image caption: we use images that contain either the word "woman" or the word "man" in their captions but not both.
In every task, we build linear classifier on top of the pretrained features. Inspired by the HyperRule in [Kolesnikov et al., 2020], we train for 50 epochs with an initial learning rate of 0.003, which is dropped by factor of 10 after 20, 30, and 40 epochs. All images are resized to 224 x 224. For augmentation, we use random horizontal flipping and cropping, where we increase the dimension of the image to 248 x 248 before cropping image of size 224 224 at random.
7

---

Table 2: A comparison of the test accuracy of the proposed algorithm against the algorithms of Hardt al. [2016] 6] and the reduction approach of Agarwal et al. [2018]. Both Shift Inference and ROC failed at debiasing all models (Table 1) so they are excluded from the comparison here.
Test Accuracy
Original
RTO
Hardt, 2016
Reduction
85.7 .1%
84.4 + .1%
81.0 + .2%
83.9+t.1%
kNN
86.8 +1%
81.3
.2%
78.7 + .2%
80.2++.19
ADULT
MLP
85.5 + +22%
83.5 + .3%
79.7 + .2%
13%
LR
84.9 .2%
83.0 = .1%
79.4 1: .2%
83.3:..2%
RF
81.2 + .2%
81.8 + .1%
80.6 + .2%
81.4+..3%
ANN
79.6 =
80.4 + .2%
78.7 + .1 1%
79.5tt11%
DCCC
80.5 1 .1%
81.3 .2%
78.8 1 .2%
81.3.1..%
LR
80.6
.2%
81.7 .1%
78.3 1: .1%
8..5
R-S 77.8%
71.3%
65,9%
R-1 79.7%
71.7%
67.5%
CELEBA
M-S
76.9%
71.8%
66.4%
M-1
79.3%
72.8%
67.5%
0.68
RF
KNN
MLP
LR
Females
Adult
Frequency
10
Proposed
0.63
Hardt, al. 2016
0.58
10-3
0.2
0.4
0.2
0.4
0.2
0.4
0.2
0.4
DCCC
0.84
RF
KNN
MLP
LR
10
11.0
005
0.0
0.5
1.0
0.81
2ply=1 -1))=1x---
Males
0.78
CelebA
0.80
0.2
R-S
0.4
0.2
R-1
0.4
M-S
0.2
0.4
0.2
M-1
0.4
Frequency
10-1
Hardt,
Proposed
0.70
10
0.60
10-5
0.25
0.5
0.25
0.5
0.25
0.5
0.25
0.5
1.0
-05
0.5
1.0
2ply=1|x)
Figure 2: LEFT: The tradeoff curves are displayed for each classification problem, where blue curves for the proposed RTO algorithm and amber curves for Hardt al. [2016]. The æ-axis is bias (Definition 1) while the y-axis is test accuracy. RIGHT: The distribution of the scores produced by ResNet50 trained from scratch are shown for both subpopulations. The curves correspond to p(y = 1[x) of Hardt et al. [2016] and the proposed algorithm when y = 0.1 and p Ely].
Scaling up the Model Size. First, we examine the impact of over-parameteriiatton in pretrained models on the effectiveness of the proposed post-processing algorithm. We fix the upstream dataset to ILSVRC2012 (8 models in total, cf. Appendix E) and aggregate the test error rates across tasks by placing them on common scale using soft ranking. Specifically, given task linearly, so that the best error achieved is zero while the largest error is one. After that, we average the performance of cach model across all tasks. Aggregated results given in Figure 3. The impact of the proposed algorithm on test errors improves by scaling up the size of pretrained models.
rescale all error rates in
Scaling up the Data. Second, look into the impact of the size of the upstream data. We take the four BiT models ResNet50x1, ResNet50x3, ResNet101xl and ResNet101x3, each is pretrained on either 11212 ImageNet-21k, JFT-300K [Kolesnikov et al., 2020]. For each model and every downstream task, we rank the upstream datasets according to the test error on the downstream task and report the average ranking. Figure 4 shows that pretraining each model JFT-300M yields the best test accuracy when it is debiased using the proposed algorithm. To ensure that the improvement is not solely due the data collection process, we pretrain ResNet50 on subsets of ImageNet-211 k before fine-tuning on the 12 downstream tasks. Figure 5 shows, again, that the impact of the proposed post-processing rule test errors improve when pretraining on large datasets.

---

Data Fraction 0.1
Data Fraction 0.2
Data Fraction 0.5
Data Fraction
1.0
1.0
Normel zed Error
0.3
0.3
Normel zed Enrrr
0.3
0.3
Normal zed
Normal zed
0.4
0.4
0.4
0444
0.4
0.4
0.4
0,
10' Model Size
10
10'
10
20'
10*
20'
10*
Model Sizee
Model Size
Model Size
Figure 3: Aggregated performance of debiased DNN models pretrained ILSVRC2012 across 12 classification tasks in CelebA and COCO (see Appendix E). The æ-axis is the number of model parameters while the y-axis the aggregated error rate across all tasks after normalization (see Section 5). Figures from left to right use 10%, 20%, 50%, & 100% of downstream data, respectively.
Data Fraction D.1
Data Fraction 0.2
Data Fraction 0.5
Data Praction
1.5
1.5
1.5
1.5
Average Ranking
Awerage Ranking
Average Ranking
Average Ranking
1.0
0.5
0.5
0.5
0.5
0.0
0.0
101x1
101x3
50x1
50x3
101x1
101x3
50x1
50x3
101x1
101x3
50x1
50x3
101x1
101x3
50x1
50x3
Figure 4: Aggregated performance of debiased Big Transfer (BiT) models pretrained ILSVRC2012 (blue), ImageNet-214 (orange), or JFT-300M (green). The y-axis is the average ranking of each upstream dataset (lower is better) according to the test error rate each of the 12 downstream classification tasks in Appendix E. Figures from left to right use 10%, 20%, 50%, & 100% of downstream data, respectively. In all models, pretraining on JFT-300K yields the best performance.
Pownstream
Downstream
Downstream
Downstream
Normellze Erer
0.0
0.0
No monteeee Eree
0.0
6.0
0.6
Normelized
0.6
termalized
6.8
0.2
0.0
10"1
0.0
10"1
10
0.0
10a
0.0
10-1
Upstream Data Traction
Upstream Data Traction
Upstream Data Traction
Upstream Data traction
Figure 5: Aggregated performance of debiasing ResNet50 when pretrained on subsets of ImageNet- 21k across 12 classification tasks in CelebA and COCO (see Appendix E). The æ-axis is the fraction of ImageNet-211 lk used during pretraining while the y-axis follows the approach in Figure Figures from left to right use 10%, 20%, 50%, & 100% of downstream data, respectively. The impact of the proposed post-processing algorithm on test errors improves when pretraining on large datasets.
6 Conclusion
The post-pree post-prceessig approach in fair classification enjoys many advantages. can be applied to any classification algorithm and does not require retraining. In addition, it is sometimes the only option available, such as when using machine lcarning as a service with out-of-the-boo predictive models [Obermeyer et al., 2019] or due to other constraints in data and computation [Yang et al., 2020a].
In this paper, we propose near optimml scalable models according to statistical parity. In addition to its strong theoretical guarantees, we show that it outperforms previous post-processing methods on standard benchmark datasets across classical and modern machine learning models, and performs favorably with even in-processing methods. Finally, we show that the algorithm is particularly effective for models trained at scale, which heavily overparameterized models pretrained large datasets before fine-tuning the downstream task.
algorithm for debiasing trained

---

Acknowledgement
The authors are grateful to Lucas Dixon, Daniel Keysers, Ben Zevenbergen, Philippe Gervais, Mike Mozer and Olivier Bousquet for the valuable comments and discussions.
Funding Disclosure
This work was performed at and funded by Google, The authors declare that there is no conflict of interest.
References
A. Agarwal, A. Beygelzimer, M. Dudik, J. Langford, and H. Wallach. A reductions approach to fair classification. In International Conference on Machine Learning, 2018.
E. Awad, S. Dsouza, R. Kim, J. Schulz, J. Henrich, A. Shariff, J.-F. Bonnefon, and I. Rahwan. The moral machine experiment. Nature, 2018.
C. L. Blake and C. J. Merz. UCI repository of machine learning databases, 1998.
Bolukbasi, K.-W. Chang, J. Y. Zou, V. Saligrama, and A. T. Kalai. Man is computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems, 2016.
Bottou. Large-scale machine learning with stochastic gradient descent, International Conference on Computational Statistics. 2010.
O. Bousquet, S. Boucheron, and G. Lugosi. Introduction to statistical learning theory. Springer, 2003.
S. Boyd and A. Mutapeic. Stochastic subgradient methods. 2008. URL https: //see. stanford. adu/materze http://trrrraeeeeooooeeeeee///// httho4sssttoohh_hubbaeeeee sootggerrddonss....ommm notes.pdf.
S. Boyd and L. Vandenberghe. Convex optimization. Cambridge University Press, 2004.
T. Brennan, W. Dieterich, and B. Ehret. Evaluating the predictive validity of the COMPAS risk and needs assessment system. Criminal Justice and Behavior, 2009.
M. Bruckner. The promise and perils of algorithmic lenders' use of big data. Chi.-Kent Rev., 2018.
F. Calmon, D. Wei, B. Vinzamuri. K. N. Ramamurthy, and K. R. Varshney. Optimized mee-pocesssing for discrimination prevention. Advances in Neural Information Processing Systems, 2017.
L. E. Celis, L. Huang, V. Keswani, and N. K. Vishnoi. Classification with faimesss constraints: A meta-algorithm with provable guarantees. In Conference Fairness, Accountability, and Transparency, 2019.
A. Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5(2), 2017.
Corbett-Davies, F. Pierson, A. Feller, S. Goel, and A. Huq. Algorithmic decision making and the cost of fairness. In International Conference Knowledge Discovery and Data Mining, 2017.
T. de Vries, I. Misra, C. Wang, and L. van der Maaten. Does object recognition work for everyone? In Conference on Computer Vision and Pattern Recognition Workshops, 2019.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, 2009.
R. Deo. Machine learning in medicine. Circulation, 2015.
I. Dixon, Li, J. Sorensen, Thain, and L. Vasserman. Measuring and mitigating unintended bias in text classification. In Conference on AI, Ethics, and Society, 2018
10

---

J. Djolonga, J. Yung, M. Tschannen, R. Romijnders, L. Beyer, A. Kolesnikov, J. Puigcerver, M. Min- derer. A. D' Amour, D. Moldovan, Gelly, N. Houlsby, Zhai, and M. Lucie. On robustness and transferability of convolutional neural networks. In Conference on Computer Vision and Pattern Recognition, 2021.
M. Dudik, R. Edgar, B. Horn, and R. Lutz. fairlearn 0.4.6. 2020. URL https://pypi.orr org/ project/fairlearn//
C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel. Fairness through awareness. In Innovations in Theoretical Computer Science, 2012.
B. Fish, J. Kun, and Á. D. Lelkes. A confidence- based approach for balancing fairness and accuracy. In International Conference on Data Mining, 2016.
N. Grgić-Hlača, M. B. Zafar, K. P. Gummadi, and A. Weller. Beyond distributive fairness in algorithmic decision making: Feature selection for procedurally fair learning. In AAAI Conference on Artificial Intelligence. 2018.
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In International Conference Machine Learning. 2017.
M. Hardt, E. Price, N. Srebro, et al. Equality of opportunity in supervised learning, In Advances in Neural Information Processing Systems, 2016.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, 2016.
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang. T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 17XVv1170.00461, 2017.
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Conference on Computer Vision and Pattern Recognition, 2017.
Kamiran and T. Calders. Classifying without discriminating. International Conference on Computer, Control and Communicationn, 2009.
F. Kamiran and T. Calders. Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems, 2012.
F. Kamiran, A. Karim, and X. Zhang. Decision theory for discrimination-avan classification. In International Conference on Data Mining, 2012.
J. Kleinberg, S. Mullainathan, and M. Raghavan. Inherent trade-offs in the fair determination of risk scores, arXiv preprint arXiv: (ete:0:0900000 2016.
R. Kohavi. Scaling up the accuracy of maive-Bayes classifiers: A decision-tres hybrid. In International Conference on Knowledge Discovery and Data Mining, 1996.
A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung. S. Gelly, and N. Houlsby. Big transfer (BiT): General visual representation learning. In European Conference on Computer Vision, 2020.
T. Lin, M. Maire, J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, Doll'a r, and L. Zitiick. Microsoft COCO: common objects in context. CoRR, abs/1 ftss/1405.0311 2014. URL http://arxiv. org/ abs/ 1405. 0312.
Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In International Conference on Computer Vision, 2015.
F. Locatello, G. Abbati, T. Rainforth, S. Bauer, B. Schölkopf, and O. Bachem. On the fairness of disentangled representations. Advances in Neural Information Processing Systems, 2019.
K. Lum and Johndrow. statistical framework for fair predictive algorithms. arXiv preprint 1eX.:16610.00077, 2016.

---

D. Madras, E. Creager, T. Pitassi, and R. Zemel. Learning adversarially fair and transferable representations. In International Conference on Machine Learning, 2018.
N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan. A survey on bias and fairness in machine learning. arXiv preprini arXiv: 0ati::/9000 096309635, 2019.
A. K. Menon and R. C. Williamson. The cost of faimess in binary classification. In Conference on Fairness, Accountability and Transparency, 2018.
Z. Obermeyer, B. Powers, C. Vogeli, and S. Mullainathan. Dissecting racial bias in algorithm used to manage the health of populations, Science, 366(6464), 2019.
J. Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 1999.
H. Robbins and S. Monro. A stochastic approximation method. The annals of mathematical statistics, 1951.
M. Sacrens, P. Latinne, and C. Decaestecker. Adjusting the outputs of a classifier to new priori probabilities: simple procedure. Neural computation, 14(1), 2002.
A. D. Selbst, D. Boyd, S. Friedler, Venkatasubramaniian, and Vertesi. Fairness and abstraction in sociotechmical systems. In Conference on Fairness, Accountability, and Transparency, 2019.
C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In International Conference on Computer Vision, 2017.
A. Torralba and A. A. Efros. Unbiased look at dataset bias. In Conference on Computer Vision and Pattern Recognition. IEEE, 2011.
A. Wang, A. Narayanan, and O. Russakovsky. Revise: A tool for measuring and mitigating bias in visual datasets. In European Conference Computer Vision, 2020a.
7. Wang, K. Qinami, I. C. Karakozis, K. Genova, Nair, K. Hata, and O. Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In Conference on Computer Vision and Pattern Recognition, 2020b.
D. Wei, K. N. Ramamurthy, and F. d P. Calmon. Optimized score transformation for fair classification. arXiv preprint arXiv:1906. 000666 2019.
B. Woodworth, S. Gunasekar, M. Ohannessian, and N. Srebro. Learning non-discriminatory predictors. arXiv preprint 17X10:1002.06081, 2017.
H. Xu and S. Mannor. Robustness and generalization. Machine learning, 86(3), 2012.
K. Yang, K. Qinami, L. Fei-Fei, Deng, and O. Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In Conference Fairness, Accountability, and Transparency, 2020a.
Y. Yang, C. Zhang, C. Fan, A. Mostafavi, and X. Hu. Towards fairness-awar disaster informatics: an interdisciplinary perspective. IEEE Access, 8, 2020b.
I.C.. Yeh and C.-h. Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications, 36(2), 2009.
M. B. Zafar, Valera, M. Gomez Rodriguez, and K. Gummadi. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In International Conference on World Wide Web, 2017a.
M. B. Zafar, I. Valera, M. G. Rogriguez, and K. P. Gummadi. Faimess Constraints: Mechanisms for Fair Classification. In International Conference Artificial Intelligence and Statisties, 2017b.
M. B. Zafar, I. Valera, M. GodezzRodriguez, and P. Gummadi. Fairness Constraints: A Flexible Approach for Fair Classification. Journal of Machine Learning Research, 2019.
12

---

R. Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork. Learning fair representations. In International Conference on Machine Learning. 2013.
B. H. Zhang, B. Lemoine, and M. Mitchell. Mitigating unwanted biases with adversarial learning. In Conference on AI, Ethics, and Society, 2018.
B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. Learning transferable architectures for scalable image recognition. In Conference on Computer Vision and Pattern Recognition, 2018.
13


Reviews:
[{"review_id": "wcnVL6KkbUR", "reviewer": "Reviewer_EmDZ", "summary": "The paper introduces a post-processing method to enforce demographic parity. The authors provide theoretical guarantees for their approach as well as empirically evaluate its performance on various datasets and against several baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Post-processing fairness-enforcing approaches are convenient as they can interact with black-box models as well as enjoy faster running times compared to in-processing methods. The paper is a nice addition to the growing literature on post-processing. It is known that post-processing can be suboptimal so it is important to understand the limits of such approaches. \n\nHowever, I do have some concerns about the paper:\n\n--The bound on Theorem 2 depends on both $\\gamma$ and $1/\\gamma$. How should this parameter be selected in practice?  In particular, how should the bound in Theorem 2 be interpreted as constant $\\gamma$ is used in the experiments? Why a fresh set of data points are needed in the statement of Theorem 2? Moreover, for the theory results, it would be great to add a discussion of the applicability and limitations. When should your approach be used? Are there any properties in the dataset or the black-box that make your approach more suitable compared to prior work?\n\n--Some of the details in the experiments are not clear and some justifications are needed. I have not checked the appendix to see whether these details are provided. For example, why additional bias is introduced in DCCC? Why the performance is not reported for the reduction-based approach on CelebA? The details of averaging and normalization for the results in the scaling section are not described clearly. More particularly, I am curious about the following points:\n\n        --How does the running time compare to Hardt et. al.? Both approaches seem to achieve the same level of fairness violation while the approach of Hardt et al is much simpler.\n\n        --Why is there no comparison with Wei et. al, 2020 as they are also proposing a post-processing approach?\n\n        --Imposing post-processing seems to improve accuracy for DCCC. This is confusing. As mentioned above, why additional bias is introduced in DCCC, and how does the accuracy is measured? i.e. for the original data or the biased data?\n\n        --The results in Table 2 for DCCC are a bit confusing. Why does the error get larger as more fairness violation is allowed? Also, there seems to be no real trade-off in this dataset. So what is the takeaway?\n\n        --I cannot follow the results in the scaling section. The details of the averaging and normalization are missing. There is no mention of the bias in this section and all the reported performances are based on normalized error. What is the takeaway here? Why there is no comparison with baselines?\n \n--The choice of using CelebA dataset to predict attractiveness as a question regarding fairness is not appropriate. Please see my comments below.\n\nMinor comments and questions:\n\n--I would prefer to call this fairness notion demographic parity. Statistical parity refers to a more broad category of fairness metrics where parity on some statistical notion is enforced for different subgroups.\n\n--Can the result in Example 1 be extended to approximate statistical parity instead of exact? I am curious whether randomization is truly necessary there.\n\n--What is $n$ in Theorem 1? Is it supposed to be $N$?\n\nThis is a truly borderline paper. It can get accepted or rejected with an equal chance though I am leaning a bit towards rejection at the moment.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is not much discussion on the limitation of the work. On the societal impact question 1(c), the authors answer: \"We propose a debiasing algorithm for machine learning. We do not foresee any potential negative societal impacts of our work.\" I think at the least there should be references to the literature that highlights where demographic parity is not a reasonable notion of fairness. Moreover, I find the task of predicting the attractiveness of people from their images a bit inappropriate at the least. While the authors do not endorse using AI for such tasks, it would have been much better if other datasets are used (e.g., there are datasets for facial recognition algorithms where the prediction bias happens because of the skin color).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "QSAhaiK2jYr", "reviewer": "Reviewer_KuGk", "summary": "This paper proposes a near-optimal post-processing algorithm (via a regularized optimization objective) that is theoretically guaranteed to debias trained machine learning models for the problem of statistical parity, which is both scalable and performs favorably relative to several existing post-processing and in-processing techniques on 2 datasets from the UCI Machine Learning repo, CelebA, and COCO. The focus of the paper is on binary classification, and the key is to output randomized predictions around the classifier’s threshold.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n- The method builds upon the existing work in fair ML, where the proposed approach allows for a randomized thresholding rule that mitigates the downsides of learning a deterministic thresholding rule. This randomization is key for the paper’s nice theoretical properties. The authors adequately cited related work in Section 2 and clearly outlined how their approach differs from existing work. They also provided an overview of baselines they compared to in Section 5, which I appreciated. \n\nQuality:\n- The paper is technically sound and claims well-supported.\n\nClarity:\n- The paper was clearly written and easy to follow, barring a few minor points of confusion (listed at the bottom of this text box). Example 1 was a nice illustration of why randomization is necessary, and the notation used in the main text was clear.\n- I did have some trouble parsing the experimental results. For example, where exactly are the results for COCO? I saw some in Figures 3-5 but from the main text I expected the results to be included with all the other tables (e.g. results for CelebA, DCCC, etc.). Also how should I be interpreting Figure 2? It’s a bit cramped and the caption doesn’t really tell me what the takeaway is.\n\nSignificance:\n- The results are significant, as debiasing machine learning models is an important research area, and most existing works do not provide theoretical guarantees. Since this work explores fairness in the statistical parity setting, and mentions that the algorithm generalizes to non-binary attributes, I could see how others would build upon these ideas.\n\n------\nUPDATE: I have read the other reviews as well as the authors' response. Thanks to the authors for clearing up some points of confusion, and I'll keep my score as is.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors stated that their results are only applicable to the setting of statistical parity, and made it clear that their method is not intended to address all issues in fair machine learning. They did not discuss potential negative societal impacts of their work, as they are developing an algorithm to mitigate bias in trained machine learning models.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "7TYSCD60lrM", "reviewer": "Reviewer_Apbv", "summary": "In this paper, the authors propose a post-processing method for debiasing trained models. The authors defined statistical parity as a 'bias'. The paper proposes a constrained optimization problem that takes the input data, sensitive attribute, partitioning and a trained model to yield a probabilistic decision rule. The authors proved theoretical properties and empirically validate its advantages on standard benchmark datasets across both classical algorithms.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Fairness is an essential topic in the machine learning community. The proposed method suggests a post-processing algorithm for reducing bias from trained models according to statistical parity. Although 'bias' is a societal concept that cannot be adequately measured by statistical parity, the reviewer thinks the proposed method suggests a reasonably acceptable solution.\n\nFor a minor comment, the reviewer thinks some explanations are relatively complicated (for instance, Example 1) due to the page limit. Some revision would be necessary for readability.\n\n**Conclusion**\nThe reviewer thinks the paper is well-written and generally acceptable, which will be an excellent contribution to the NeurIPS conference. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not address the limitations and potential societal impact of their work in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The balance of the reviews is in favor of accepting the paper. I want to follow this consensus. While some of the reviewers described significant issues with the paper, I do not view these issues as disqualifying. I expect the camera-ready version of the paper will be able to address most of them as per the author response.
Decision: Accept (Poster)