Fairness without Demographics through Knowledge Distillation
Authors: Junyi Chai
Taeuk Jang
Xiaoqian Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: fairness without demographics
knowledge distillation

Abstract: Most of existing work on fairness assumes available demographic information in the training set. In practice, due to legal or privacy concerns, when demographic information is not available in the training set, it is crucial to find alternative objectives to ensure fairness. Existing work on fairness without demographics follows Rawlsian Max-Min fairness objectives. However, such constraints could be too strict to improve group fairness, and could lead to a great decrease in accuracy. In light of these limitations, in this paper, we propose to solve the problem from a new perspective, i.e., through knowledge distillation. Our method uses soft label from an overfitted teacher model as an alternative, and we show from preliminary experiments that soft labelling is beneficial for improving fairness. We analyze theoretically the fairness of our method, and we show that our method can be treated as an error-based reweighing. Experimental results on three datasets show that our method outperforms state-of-the-art alternatives, with notable improvements in group fairness and with relatively small decrease in accuracy.

Fairness without Demographics through Knowledge Distillation
Junyi Chai, Taeuk Jang, Xiaoqian Wang"
Elmore Family School of Electrical and Computer Engineering
Purdue University
West Lafayette, IN 47906 {chai 28, jang114,, joprnog
Abstract
Most of existing work on fairness assumes available demographic information the training set. In practice, due legal or privacy concerns, when demographic information is not available in the training set, is crucial to find alternative objectives ensure fairness. Existing work fairness without demographics follows Rawlsian Max-Min fairness objectives. However, such constraints could be too strict improve group fairness, and could lead to great decrease in accuracy. In light of these limitations, in this paper, we propose to solve the problem from a new perspective, i.e., through knowledge distillation. Our method uses soft label from an overfitted teacher model as an alternative, and we show from preliminary experiments that soft labelling is beneficial for improving fairness. We analyze theoretically the fairness of our method, and we show that our method can be treated as an error-based reweighing. Experimental results on three datasets show that our method outperforms state-of-the-ar alternatives, with notable improvements in group fairness and with relatively small decrease in accuracy.
Introduction
Machine learning has been widely adopted real-world scenarios that has great social influence, and fairness in automatic decision-maie making systems has become arising concern. It has been shown that without proper intervention, models can be biased against certain demographic groups (Hardt et al.) 2016 Chouldechoval 2017: Ntoutsi al., 2020). Due distributional disparities across different demographic groups, simply removing sensitive attributes during training does not improve fairness. Therefore, research fair classification has received extensive attention, and different methods have been proposed to improve fairness (Roh et al.) 2020a; Choi et al.) 2020). These methods usually focus on specified fairness notions, and most of these works assume access predefined sensitive information during training.
In practice, however, due to legal regulatory concerns, it is often infeasible to collect sensitive information, which greatly limits the usage of conventional methods on fairness. Besides, in many real-world applications, attributes. For example, Consumer Financial Protection Bureau (CFPB) requests creditors to achieve fairness, without requesting collecting information about applicant's race, color, religion, national origin, or sex. As most works on fairness assumes only one predefined sensitive attribute, due difference in distributional disparities across different sensitive attributes, improving fairness under one sensitive attribute does not always imply fairness under another sensitive attribute. Therefore, it crucial to study the problem of fairness without demographics.
expect the decision-making systems be fair w.r.t. multiple sensitive
*Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Current methods on fairness without demographies can be divided into two categories: faimess with proxy sensitive attribute and Max-Min fairness. Methods on fairness with proxy sensitive attribute (Yan et al., 2020; Grari et al., 2022) utilizes surrogate group information, including clustering information and group information through estimation as alternative, and enforces fairness under surrogate group partition. The problem with proxy attributes is that it requires a strong assumption that the clustering or surrogate group information is correlated with the sensitive attribute(s) of concern, and improving group fairness w.r.t. several sensitive arrtibutes could be infeasible under the condition of large distributional discrepancy over different sensitive attributes of interest. Methods on Max-Min fairness (Hashimoto et al. 2018 Lahoti al., 2020) focuses improving the worst utility amongst all sensitive groups. Ideally, this formulation is in accord with group fairness; however, in practice, such formulation could be too strict to obtain improvement in group faimesss as improving worst-group utility does not necessarily imply decrease in inter-group disparities, and since Max-Min fairness tends require similar utility over all groups, this could lead significant decrease in accuracy. Instead, we seek to find alternative ways to quantify disparities without observing sensitive information during training.
We draw inspiration from debiased learning (Choi et al., 2019; Nam et al. 2020: Liu et al. 2021 ), where the goal is to unlearn spurious correlations present in training set. One common way of debiased learning is through reweighing, where samples of worst groups specified by spurious features are up- up-weighed so to mitigate biased prediction. For classification tasks, one simple way to assign weight is to replace the discrete one-hot encoding with continuous labelling that are obtained based expected properties. However, for most classification tasks, continuous labelling is unrealistic in real-world scenarios. In light of this, we consider obtaining the soft labels through knowledge distillation, where training labels for student model are replaced with normalized logits of teacher model, and we expect knowledge from the overfitted teacher model to help fit hard samples in training set.
In this paper, we propose a knowledge distillation framework for improving fairness without accessing sensitive information. Compared with previous works on fairness without demographics, our method does not impose surrogate constraints on fairness. Instead, we try to find a soft labelling for samples through knowledge distillation to help the student model better fit. During training, teacher network to be overfitted to training data, and we use the normalized logits of teacher model as soft label to train student network. In this way, we seek to correct bias in training set via reweighing training samples.
first train
We summarize our contribution as follows:
1. Fairness without Demographies: We study the effectiveness of knowledge distillation in improv- ing fairness without observing sensitive information. We discuss two different mapping functions of logits by teacher model, and we discuss theoretically the impact of label smoothing and knowledge distillation on fairness.
2. Our method effectively utilizes knowledge from complex teacher model and achieves comparable classification results than unconstrained baseline student model.
3. Empirical Observation: We validate the effectiveness of label smoothing on new Adult dataset, and we show from experiments that our method achieves remarkable improvement in fairness on three benchmark datasets under different sensitive attributes.
2 Related Work
2.1 Group Fairness in Classification
Group fairness is a class of definitions to quantify fairness, which measures the disparity of predicted outcome among the subgroups with certain sensitive attributes. Several works were proposed to mitigate the disparity and ensure the independence of the performance measures between the subgroups to achieve group fairness. Equal opportunity (Hardt et al. (0016) states that true positive rates are the same for the subgroups. Similarly, predictive equality (Chouldechova 2017) requires the equality of false positive rates. Applying calibration conditions (Kleinberg et al., 2016, Pleiss et al., 2017) were proposed so that the probability estimate matches the actual ratio of positive distribution within the group. Diverse approaches are presented to meet particular group fairness notions. Pre- processing method (Chen et al. 2018 Jang et al. 2021 suggests improving the skewed sample size

---

problem. Adversarial learning (Madras et al. 2018) is proposed to learn a fair representation that is independent of sensitive attributes. In-orcsssingg methods (losifidis and Ntoutsi 2019; Roh et al. 2020b Chai and Wang, 2022) aim at regulating the training process with fairness constraints by reweighing training samples or enforcing fairnsss regularization. Post-processing the output of the biased model (Kim et al. 2020, Jang et al. (022) with multiple fairness objectives are introduced as they are more efficient than training a model from scratch. However, to ensure fairness, the aforementioned approaches require sensitive information such as race, gender, etc. This restricts their versatility because the sensitive information is not always available owing to regulations and privacy concerns. Work including (Celis et al., 2021 b. Mehrotra and Celis 2021) studies the problem of fair classification under noisy or adversarially perturbed sensitive attributes, which can be considered as relaxed version of fairness without demographies.
2.2 Fairness without Demographics
Due to the practicality and regulatory limitations, methods to achieve fairness without demographics have emerged. The methods in this category adhere to Rawlsian Max-Min faimess (Rawls) )
minimize the risk of the group with the highest risk. Distributionally Robust Optimization (DRO)
(Hashimoto et al. 2018) utilizes x divieeggee to find the worst-case distribution that is close to the empirical distribution and minimizes the risk of such distribution. ARL (Lahoti et al.] 2020) conduct weighted empirical risk minimization with adversary network to maximize the loss with the perspective of
Other methods (Zhao et al. 2022) on fairness
without demographies seek to find surrogate group information based on input features. Fair class
balancing (Yan et al.) (2020) proposes to quantify disparities w.r.t. sensitive attribute using clustering information and upsample minor class in each cluster to achieve balance.
2.3 Debiased learning
Debiased learning focuses on learning representations that are unbiased w.r.t. spurious features. Some works on debiased learning focus on learning unbiased representation through adversarial training. Choi et al. (2019) propose to augment the standard cross-entropy loss for action classification with an adversarial loss for scene types and a human mask confusion loss for background. Kim et al. (2019) proposes to unlearn the bias information by minimizing mutual information between latent representation and bias prediction with one adversarial network. Another approach employs a biased model to explore the samples with high loss. Nam et al. (2020) proposed to apply GCE loss (Zhang and Sabuncu, 2018) to train a bias amplifying classifier and parallelly train a fair classifier by referencing the biased classifier. Liu et al. (2021) suggest simple two-stage training process. After training a classifier with ERM in the first stage, during the second stage, the model emphasizes the misclassified samples in the first stage. However, the objective to minimize the maximum loss could make the training unstable and deteriorate the overall performance.
2.4 Label Augmentation for Classification
Unlike the previous works, we explore a new approach to ensure faimess without demographics, which is to utilize knowledge distillation (Hinton al., 2015). While cross entropy (Rumelhart et al.) 1986) has been dominant loss function for classification tasks, some works question its shortcomings, such being sensitive noisy label (Zhang and Sabuncu 2018), poor margin (Nar al. 2018), and vulnerable to adversarial samples (Nar et al.] 2019). To overcome this problem, perturbing the label distribution empirically verified the improvement. Label smoothing (Müller al., 2019) and label augmentation (Zhang et al.) (017)) effectively deal with noisy label and adversarial samples. Knowledge distillation (Hinton et al.) (013)) was proposed for model compression to effectively transfer knowledge from a deep teacher model to smaller student model. Surprisingly, by convex combination of classical cross entropy loss and supervision from the teacher model, the student model could obtain competitive or even a superior performance at inference. Even though family of knowledge distillation methods proposed, its impact on fairness is not actively studied yet. this work, we study the effectiveness of knowledge distillation on fairness in the absence of sensitive information, and we claim that noticeably improves fairness by distilling intrinsic information while volatilizing data bias.
3

---

3 Method
3.1 Background
Consider labelled samples (z. y, a) X x V A, with y being the ground truth labels and being the sensitive attributes, the goal of fair classification is to learn classifier that satisfies certain fairness criteria w.r.t. a. These criteria can generally be formulated as independence between sensitive information and expected statistical features. For example, disparate impact (Feldman et al.) 2015) requires the predicted label ij to independent of sensitive attributes: ý 1. a, and equalized odds is satisfied when the predictions and sensitive attributes are independent conditional on ground truth label: aly.
3.2 Preliminary Results
Conventional methods on fairness fails when no sensitive information available in the training set. Instead, we seek find surrogate training objective that is beneficial for improving fairness. We start with preliminary results on new Adult dataset.
The new Adult dataset (Ding et al., 2021) is reconstructed version of Adult dataset (Dua and Graff 2017). Instead of containing binary labels indicating whether or not an tindividual's income exceeds 50K, the new Adult dataset provides specific incomes for each individual, and the goal is to predict whether an individual's 's income exceeds certain predefined threshold. Research on new Adult dataset shows that by adjusting the income threshold for binary encoding we able to improve fairness, without introducing extra regularization terms during training (Ding et al., 2021). Intuitively, a properly adjusted threshold for binary labelling helps reduce hard samples in dataset, which could be beneficial for the unprivileged subgroup. Inspired by this, we consider reweighing samples through label smoothing. Although labelling bias have been widely studied in fairness literature (Jiang and Nachum 2020 Krasanakis et al.) 2018), the effect of label smoothing without accessing sensitive information remains unclear. Does label smoothing also help improve fairness during training?
We validate this hypothesis on new Adult dataset. Specifically, we repeat experiments three times and report average results, and in each repetition, we randomly split data into 50% training data and 50% testing data. Following the traditional setting on the Adult data, we set the binary cutoff threshold of income 50K. Let s; be the income of i-th individual, on testing data, we have y1 = 1(s; >> E and we consider four different types of labelling during training:
Binary: The training labels are determined by y 1(s; > E).
Random: The training labels are determined as follows: for s; > E, we sample from
uniform distribution on (0.5, 1]; for s; we sample y, from a uniform distribution on (0,0.
Softmax: The training labels are determined by y/ = (1 +e -1
Linear: The training labels are determined by piecewise linear function: y/s =
(aL-E) (ec-min: sc) 1(8: S ())+ + 2(max, & 3 + 0.5) 1(s: E).
2[6 2ig-ming s:)
Label Assignment
Sensitive Attribute: Race
Sensitive Attribute: Gender
Accuracy
Dis. Impact
Eq. Odds
Dis. Impact
Eq. Odds
Binary
85.1910.117 % 85.13+t0.17%
11.80:#0488%
13.22+11.00%
17.74=0.56%
16.65+1.80%%
Random
12.21+1.379) 10.75+11.15%
13.15:t1.12%
17.64=0.95%
15.57 1.49%
Linear 85.14%
10.13+1.06%
16.62=0.83%
12.37+1.649% 64%
Softmax
85.19:10.17% %
11.21
10.67+1.24%
16.37-00.8%%
13.34 1...-11.9999 1.89%
Table 1: Experimental results on new Adult dataset with race and gender as sensitive attribute, respectively. Fairness is evaluated using two metries: Disparate Impact and Equalized Odds.
Results are shown in Tab. Compared with binary labelling, both linear and softmax labelling im- proves fairness under different sensitive attributes, while random labelling achieves little improvement in faimess. This shows the effectiveness of proper label smoothing.

---

3.3 Knowledge Distillation
Teacher network)
Sont Libel (prediction teachar)
Ma-up with hard abd
Student jimp network)
bbd forstudent
Figure I: Demonstration of our knowledge distillation method. The dashed line refers to the decision boundary under soft labelling, and the solid line refers to the decision boundary under hard labelling. Points of different shapes refer to samples of different labels, different colors refer to different sensitive groups and larger pattern indicates higher weight. By focusing on hard samples that are conrectly classified (marginal samples that are correctly classified), method learns better decision boundary which improves the performance on hard samples, and disadvantaged group(s) are more likely to benefit from this.
For most classification tasks with categorical encoding, continuous labels are not available. In light of this, we instead consider knowledge distillation for obtaining soft labels. We considering the binary classification setting, although method may be readily generalized to multi-class classification.
Knowledge distillation is a form of model compression through the process of transferring the knowledge from a large pre-trained teacher model set of models to single smaller student model that can be practically deployed under real-world scenarios, and performed commonly on neural networks associated with complex architectures. We consider response- based knowledge, which focuses on the final output layer of the teacher model, and the student model learns to mimic the predictions of the teacher model.
Generally, given a training set {1 (Ix,,);),
N) X x let : - V, teacher model, and f X' + y) + ÿ be the student model, the training objective for student model can be formulated as
/ be the
N
L = N Σ
[aLa)(((())) + -1 (L=)))de((())))
where أن - (x;;) is the soft label from teacher model, Lots is the classification loss, and a E (0, 1) is the trade-off hyper-parameter For a classification task with classes, the teacher model can be decomposed into two functions, ! = Ø ht where h X - IRS +> z the nonlinear function that predicts the logit, and Φ R + j0, 1] gl is the mapping function for soft labelling.
We consider two different mapping functions d:
Softmax function. Given the output vector 2; = h(x;)) by the teacher model, we apply softmax activation aollmmux convert the logits into probability vector ūj. and the predicted probability for j-th class is computed by comparing the logit Zij with the other logits:
exp (zz/TT)
i Psottmax (z:); Ee exp (zik/T)'
where T is the temperature that controls the probability distribution over classes.
Linear function. Given the logits, Zij, computed for each class by teacher model, let Zıy maxi Zi be the largest logit prediction for x; and Sj Zilzz = zij' }. we first use linear function to convert
the largest logit zt of sample z, into a probability Tty::
2z5' minges, Zzf'
Taj' maxies, Ziz' - minges, Zij'
+ 0.5,
(3)
5

---

where the decision threshold for logits that corresponds to predicted probability 0.5 is chosen as the smallest logits values of samples whose raw logits predictions are the highest for the j-th class. Probability for the other logits can then be linearly distributed as follows:
Zik
Tik = - (1-1 Saarg max,
Vk + arg max Eut.
(4)
3.4 Theoretical Analysis
In this subsection, we provide formal analysis on the impact of label smoothing and knowledge distil- lation on fairness. We first discuss the impact of label smoothing and its connection to reweighing.
For ease of exposition, here we consider binary classification where E {0, 13. Denote as y' = ay² + (1 - oryy the soft label, the cross entropy loss under soft label can be formulated as
Ltoft yy log(f())) (1 - y²) log(1 f(x)).
(5)
And the cross entropy loss under binary label can be formulated as
Lhard = y (og([(())) (1 y) log(1 - f())).
Suppose the classifier / makes same predictions x under both binary and soft labelling at current iteration, we have
A = Lsoft Lhard (y y/) log(f(x)) - (y-
log(1 f(x))
exp (z1)
a(y 1/ log(
exp (20) texp (#1)
(21)
exp (20) texp (21)
a - v (21 20).
20)-
where 2; is the logit of student network for i-th class. Since 2 0, for I.oft Lhard < 0, we have = 0, 21 > 20 or y = 1, < zo. which shows that label smoothing down-weighs samples that are incorrectly predicted. For samples that are correctly predicted, larger discrepancy between y and y' indicates larger A, which shows that the classifier pays attention hard samples that correctly predicted, and larger a indicates larger discrepancy between L. and Lhard-
Similarly, compute the induced weight by label smoothing follows:
Lhard TiSoft =
log - f(x))
w(x)
+1 y
(8)
log(f(z))
Here only consider positive samples for ease of discussion. Let ((x) log(1-f(z)) have (og(f(x))
w(x) 101==(11-=111))000 ((==)(=))a))>> + adyl + T(x)
Since for f(x) > 0.5, we have r(x) > 1, and w decreases as y" increases. this way, samples with high soft label but low predicted probability, i.e. positive hard samples that correctly classified assigned with higher weight. For samples that are wrongly classified, we have r(x) < 1, and samples with lower soft label and lower predicted probability, i.e. hard samples that are wrongly classified
assigned with the lowest weight. Besides, since J² S y. as a increases, y/ decreases, and samples that are correctly predicted are assigned with higher weights, while samples that are incorrectly predicted are assigned with lower weights. Similar results also holds for negative samples.
We further derive theoretical analysis to show that our trained with label smoothing and knowledge distillation guarantees fairness in terms of equal odds.
Theorem 3.1. Consider a classifier f X' [0, 1] for binary classification. Denote the classification loss Lsaft. = -oflog( (()g((()))) -1- ( log(1 f(x)) with soft label y' = ogj² + + a.)y, where j' E [0, 1] is the predicted label from teacher model, y {0, 1} is the binary label, and a is the balance parameter. The equal odds fairness metrics w.r.t. classifier is upper bounded by Leoft.

---

Experiments
4.1 Experimental Setup
We validate our model on three benchmark dataset:
contains 49,531 samples with 14 attributes. The goal is to predict whether an individual 's income exceeds certain threshold. During experiments, we set the binary cutoff threshold as 50K, and we choose race and gender sensitive attribute.
New Adult: The Adult reconstruction dataset Ding al. 2021
The COMPAS dataset (Larson et al. 2016) contains 7,215 samples with 11 attributes. Following previous works on fairness (Zatar et al.) 2017), we only select black and white defendants in COMPAS dataset, and the modified dataset contains 6,150 samples. The goal is predict whether defendant reoffends within two years, and we choose sex and race as sensitive attributes.
COMPAS:
CelebA: The CelebA dataset (Liu et al. 2015) contains 202,599 face images, each of resolution 178 218, with 40 binary attributes. We consider different binary classifcation tasks on CelebA dataset: predicting attractiveness with gender as sensitive attribute, and predicting gender with age sensitive attribute.
We implement our method in PyTorch 1.10.1 with one NVIDIA RTX-3090 GPU. We use accuracy as evaluation metric, and we apply disparate impact (Kamiran and Calders 2012) and equalized odds (Hardt et al. 2016) as fairness metrics. We build the teacher model using ResNet-152 (He al. (2166) and student model using ResNet-18 (He et al. 2016). We compare our method with the following related methods:
Teacher model: Teacher network trained on binary label. The teacher model is expected to overfit training data.
Student model (binary): Student network trained on binary label.
Student model (softmaxx: Student network trained on soft label obtained by the output of teacher model with softmax activation as in (2).
Student model (linear): Student network trained soft label obtained by the output of teacher model with linear normalization as in (3).
Distributionally robust optimization (DRO): Student network with distributionally robust
optimization (Hashimoto et al. 2018). The network is trained on binary label.
Adversarially reweighted learning (ARL): Student network with adversarially reweighted learning (Lahoti et al., 2020). The network trained on binary label. The adversarial network is chosen as one linear layer as suggested by the authors (Lahoti et al. (020).
FairRF: Student network with FairRF (Zhao et al. (002). The network is trained on binary label. We include results of FairRF only on new Adult and COMPAS dataset, as it is hard to regulate sensitive-related features for images.
We repeat experiments on each dataset five times and report the average results. To avoid large discrepancies in testing data, before each repetition, we randomly spilt data into 50% training data, 10% validation data and 40% test data. All the methods evaluated are trained and tested on the same data partitions each time. For student model trained on softmax label, the temperature is tuned to find the best validation accuracy. The hyperparameters of comparing methods are tuned with binary search to find global minimum, as suggested in the original paper (Hashimoto et al. 2018).
4.2 Experimental Results
Experimental results are shown in Table On all three datasets, both DRO and ARL show a drop in accuracy, especially for DRO which strictly follows the formulation of Max-Min faimess.. However, both methods do not show notable improvement that Max-Min fairness could be too strict to achieve expected improvement in group fairness. In comparison, our method does not induce as much decrease in accuracy, but achieves significant improvement in terms of fairness on all three datasets, especially for equalized odds. This shows the
fairness. This in line with our hypothesis
7

---

Method
Accuracy
Disparate impact
Equalized odds
Teacher
84.41%
20.27%
39.64%
Student (with hard label)
64.13:10.32%
23.27:+2.43%
38.34:1..37%
DRO (Hashimoto et al. 2018
62.67-10.3%
21.41
30,43
ARL Lahotr al. 2020
63.23-10.47%
21.37::3.46%
29.46:11.744
FairRF (Zhao et al. 2022)
63.26+00.89%
21.47
25.67++2.634
Student (with softmax label)
63.47:00.4%
19.52++2.46% 20.27+22.4%%
21.32++1.9%%
Student (with linear label)
63.34+0..6%
20.31+22.62%
Table 2: Results
COMPAS dataset with sensitive attribute race.
Method
Accuracy
Disparate impact
Equalized odds
Teacher
84.41%
19.42%
34.41%
Student (with hard label)
64.13:0..2%
19.17+2.33%
20.25:2..5%%
DRO (Hashimoto et al. 2018
62.67:t0.73%
19.62+2.27%
18.75+22.8%%
ARL Lahoti al. 2020)
63.23::0.47%
18.87+3.32%
19.1442.55%
FairRF (Zhao et al. 2022)
63.26:10.83%
17.23=1.844
18.7442.21%
Student (with softmax label)
63.3710.44% 63.3440.46%
16.63#1.67%
14.32:+2.47%%
Student (with linear label)
16.14=1.33%
15.13
Table 3: Results on COMPAS dataset with sensitive attribute sex.
Method
Accuracy
Disparate impact
Equalized odds
Teacher
90.63%
14,43%
17.76%
Student (with hard label)
85.42:10.46%
12.31
14.43 1.241.244 1.24%
DRO (Hashimoto al. 2018
8..64.10.62%
12.27
13.69 1.34%
ARL Laahoti et al. 2020)
83.37-1-.46%
12.34
14.02-+: 1.43%
FairRF (Zaa al. (2022)
83.741:0.6%
11.37=1.46%
11.23#1.4299
Student (with softmax label)
84.63+00.47% 84.27+0.31%%
10.63=1.349%
10.34+1.225%
Student (with linear label)
10.21
10.57=1.64%%
Table Results on new Adult dataset with sensitive attribute race.
Method
Accuracy
Disparate impact
Equalized odds
Teacher
90.63%
19.42%
24.47%
Student (with hard label)
8..42++..32%
17.63+
18.82+1.8822
DRO (Hashimoto et al. 2018
82.64+0.83%%
17.21+1.63%
17.63++2.27%
ARL Lahoti al. 2020
83.37+00.2%
16.62+1..7%%
16.52+1..96%
FairRF (Zhao et al. 2022)
83.74+10.86%
16.37
13.54=1.2661
Student (with softmax label)
84.63::0.33% 84.2710.64%
15.63+1.469%
12.23+1.833
Student (with linear label)
15.56+
11.59+1.7496
Table 5: Results on new Adult dataset with sensitive attribute gender.
Method
Accuracy
Disparate impact
Equalized odds
Teacher
86.67%
18.46%
31.47%
Student (with hard label)
81.67++0.63%
+..32%1.34%
26.42+1.66
DRO (Hashimoto al. 2018
76.62+0..6%
17.26+1.82%%
23.26:+-268%%
ARL (Lahotr et al. 2020)
77.14:10.75%
17.23+22.48%
21.41 %
Student (with softmax label)
80.87 14%
15.27+1.71%%
11.43+11.2%%
Student (with linear label)
80.76+10.3%%
14.47
10.62+
Table 6: Results of attractiveness classification on CelebA dataset with sensitive attribute gender.
effectiveness of knowledge from teacher model and error- brsed reweighing for soft labelling. We observe that the pre-trained overfitted teacher model achieves higher accuracy on all three datasets, which is in line with the expectation of knowledge distillation. However, the overfitted teacher model does not improve fairness, and the equalized odds of teacher model for COMPAS and new Adult dataset are much worse than baseline student model, and we conclude that improvement in fairness with knowledge distillation is due to soft labelling, instead of complexity of teacher model.

---

Method
Accuracy
Disparate impact
Equalized odds
Teacher
94.42%
17.31%
17.46%
Student (with hard label)
90.43:10.23%
1..67+2.26%
17.32:11.78%%
DRO (Hashimoto et al. 2018
74.46+1:0.7%%
16.73++..5%%
16.45-+.2.66%
ARL Lahoot al. 2020)
84.27+0.94%
16.21+ 1.86%
15.54++ 1.57%
Student (with softmax label)
89.21+0.15%% 89.42++0.68%
12.23+ 1.84%
10.13+42.14%
Student (with linear label)
13.16++2.21%
9.64
Table 7: Results of gender classification on CelebA dataset with sensitive attribute age.
4.3 Parameter Analysis
205
00 Bro
100
275
12-
-076
(a) COMPAS (race)
(b) New Adult (race)
(c) CelebA, gender classification (age)
14
15-
12
(d) COMPAS (sex)
(c) New Adult (gender)
(f) CelebA, attractiveness classification (gender)
Figure 2: Change of equalized odds varies on COMPAS, new adult and CelebA dataset. The sensitive attributes for each setting are shown in parenthesis.
We move on to discuss the influence of the trade-off parameter in 1 on accuracy and fairness. is straightforward to see that as a varies, the corresponding loss term can be treated as linear interpolation between binary cross-entropy loss and soft cross-entropy loss, and increasing increase the role of soft label in knowledge distillation, which we observe to help improve fairness as Fig. 2 As shown in Fig.
increases, there is small decrease in accuracy, and all three datasets, as a increases, under both softmax and linear labelling, although with fluctuation under certain variation of a COMPAS and new Adult datasets, overall the equalized odds regarding different sensitive attributes show downtrend, which validates the effectiveness of soft labelling.
5 Conclusion
Fairness without demographies has been an important yet less studied task. In this paper, we propose to solve this problem through knowledge distillation. We consider a pre-trained complex teacher model that overfitted training data, and we use the normalized logits as soft label train simple student model. We consider two different normalization functions, the softmax function and linear function. We theoretically discuss the connection between soft labelling and reweighing, and we prove theoretically that with knowledge distillation, the training objective of student model upper-bounds several fairness metrics. We validate from preliminary experiments on new Adult dataset the effectiveness of soft labelling in improving fairness, and we show from experiments that knowledge distillation achieves remarkable improvement in fairness with relatively small sacrifice in accuracy.

---

1990
(a) COMPAS dataset
(b) New Adult dataset
04
04
0.
(c) CelebA (attractiveness classifi- (d) CelebA (gender classification)
cation)
Figure 3: Change of accuracy varies COMPAS, new Adult and CelebA dataset.
There are two main directions of interest for future work. First alternatives mapping function: we define other mapping functions for the logits such that the student model achieve better fairness? Understanding the influence of mapping functions crucial in extending our work. Another topic is robustness of our method to distributional shift, especially the robustness of fairness, as robustness to distributional shift has been an important but challenging problem in the context fairness. Other topics include few-shot tweaking with partially available sensitive attributes.
Acknowledgements
This work partially supported by the EMBRIO Institute, contract #2120200, National Science Foundation (NSF) Biology Integration Institute, Purdue's Elmore ECE Emerging Frontiers Center, and NSF IIS #1955890, IIS #2146091.
References
Celis, E., Huang, L., Keswani, V., and Vishnoi, N. K. (2021a). Fair classification with noisy protected attributes: A framework with provable guarantees. In International Conference Machine Learning, pages 1349-1361. PMLR.
Celis, L. E., Mchrotra, A., and Vishnoi, N. (2021b). Fair classification with adversarial perturbations. Advances in Neural Information Processing Systems, 34:8158-8171.
Chai, and Wang, X. (2022). Fairness with adaptive weights. In International Conference Machine Learning, pages 2853-2866. PMLR.
Chen, I,, Johansson, F. D., and Sontag, D. (2018). Why is my classifier discriminatory? In Advances in Neural Information Processing Systems, pages 3539-3550.
Choi, Gao, C., Messou, C.. and Huang, J.-B. (2019). Why can't dance in the mall? learning to mitigate scene bias in action recognition. Advances in Neural Information Processing Systems, 32.
Choi, Y., Dang, M., and Van den Broeck, G. (2020). Group faimess by probabilistic modeling with latent fair decisions. arXiv preprint 20XX:200000003111
Chouldechova, A. (2017). Fair prediction with disparate impact: study of bias in recidivism prediction instruments. Big data, 5(2):153-163.
10

---

Ding, F., Hardt, M., Miller, J. and Schmidt, L. (2021). Retiring adult: New datasets for fair machine learning, Advances Neural Information Processing Systems, 34.
Dua, D. and Graff, C. (2017). UCI machine learning repository.
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C.,, and Venkatasubramaniaa, S. (2015). Certifying and removing disparate impact. In proceedings of the 2/th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 259-268.
Grari, V., Lamprier, S., and Detyniccki, M. (2021). Fairness without the sensitive attribute via causal variational autoencoder. arXiv preprint S.3iv:2009009999
Hardt, M., Price, E.,, and Srebro. N. (2016). Equality of opportunity in supervised learning. Advances in Neural Information Processing Systems, 29.
Hashimoto, T., Srivastava, M., Namkoong, H., and Liang. P. (2018). Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning, pages 1929-19388 PMLR.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778.
Hinton, G., Vinyals, O., Dean, J., al. (2015). Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253: 2(7).
Iosifidis, V. and Ntoutsi, E. (2019). Adafair: Cumulative fairness adaptive boosting. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 781 789-
Jang. T., Shi, P., and Wang, X. (2022). Group-awar threshold adaptation for fair classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6988 6995.
Jang, T., Zheng, F., and Wang, X. (2021). Constructing a fair classifier with generated fair data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7908-7916.
Jiang, H. and Nachum, O. (2020). Identifying and correcting label bias in machine learning. In International Conference on Artificial Intelligence and Statistics, pages 702-712. PMLR.
Kamiran, F. and Calders, T. (2012). Data preprocessing techniques for classification without discrim- ination. Knowledge and Information Systems, 33(1):1-33.
Kim, B., Kim, H., Kim, K., Kim, S., and Kim, J. (2019). Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9012-9020.
Kim, J. S., Chen, J., and Talwalkar, A. (2020). Fact: A diagnostic for group fairness trade-offs. In International Conference on Machine Learning, pages 5264-5274. PMLR.
Kleinberg, J., Mullainathan, S., and Raghavan, M. (2016). Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv: 1609.0580.
Krasanakis, E., citrommitro-xiouffi E.,, Papadopoulos. S., and Kompatsiaris, Y. (2018). Adaptive sensitive reweighting to mitigate bias in faimess-aware classification. In Proceedings of the 2018 World Wide Web Conference, pages 853-862.
Lahoti, P., Beutel, A., Chen, J., Lee, K., Prost, F, Thain, N., Wang, X., and Chi, E. (2020). Fairness without demographics through adversarially reweighted learning. Advances in Neural Information Processing Systems, 33:728-7444
Larson, J., Mattu, S., Kirchner, I.., and Angwin, (2016). Compas analysis. GitHub, available at: https:/ggithub. com/ppoublicalcompsssannc
Liu, E. Z.,, Haghgoo, B., Chen, A. S., Raghunathan, A., Koh, P. W., Sagawa, S., Liang, P., and Finn, C. (2021). Just train twice: Improving group robustness without training group information. In International Conference Machine Learning, pages 6781-6792. PMLR.

---

Liu, Z., Luo, P., Wang. X., and Tang. X. (2015). Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV).
Madras, D., Creager, E., Pitassi, T., and Zemel, R. (2018). Learning adversarially fair and transferable representations. arXiv preprint 1rXiv:1802.06309
Mehrotra, A. and Celis, L. E. (2021). Mitigating bias in set selection with noisy protected attributes. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 237-248.
Müller, R., Korbblith, S., and Hinton, G. E. (2019). When does label smoothing help? Advances in Neural Information Processing Systems, 32.
Nam, J.. Cha, H., Ahn, S., Lee, J., and Shin, J. (2020). Learning from failure: De-biasing classifier from biased classifier. Advances Neural Information Processing Systems, 33:20673-20684.
Nar, K., Ocal, O., Sastry, S. S., and Ramchandran, K. (2018). Cross-entropy loss leads to poor margins.
Nar, K., Ocal, O., Sastry, S. S., and Ramchandran, K. (2019). Cross-entropy loss and low-rank features have responsibility for adversarial examples. arXiv preprint arXiv:1901.0836:
Ntoutsi, E., Fafalios, P., Gadiraju, U.. Iosifidis, V., Nejdl, W., Vidal, M.-E., Ruggieri, S., Turini, F. Papadopoulos, S., Krasanakis, E., et al. (2020). Bias in data-driven artificial intelligence systems-an introductory survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 1033):
Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J., and Weinberger, K. Q. (2017). On fairness and calibration. Advances in Neural Information Processing Systems, 30.
Rawls, J. (2009). A theory of justice. Harvard university press.
Roh, Y., Lee, K., Whang, S., and Suh, C. (2020a). Fr-train: A mutual information- approach to fair and robust training. In International Conference on Machine Learning, pages 8147-8157. PMLR.
Roh, Y., Lee, K., Whang, S. E., and Suh, C. (2020b). Fairbatch: Batch selection for model fairness. arXiv preprint 2rXiv:2012.01666.
Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back- propagating errors. nature, (3088)
Yan, S., Kao, H.-t,, and Ferrara, E. (2020). Fair class balancing: Enhancing model fairness without observing sensitive attributes. In Proceedings of the 29th ACM International Conference Information & Knowledge Management, pages 1715-1724.
Zafar, M. B., Valera, I., Gomez Rodriguez, M., and Gummadi, K. P. (2017). Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of the 26th International Conference on World Wide Web, pages 1171-1180
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. (2017). mixup: Beyond empirical risk minimization. arXiv preprint arX0v:7710004112.
Zhang. Z. and Sabuncu, M. (2018), Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in Neural Information Processing Systems, 31.
Zhao, T., Dai, E., Shu, K., and Wang. S. (2022). Towards fair classifiers without sensitive attributes: Exploring biases in related features. In Proceedings of the Fifteenth ACM International Conference Web Search and Data Mining, pages 1433-1444
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "oFbzGbOyU6A", "reviewer": "Reviewer_V6Vx", "summary": "This paper introduces a new method to achieve considerable improvement in group fairness through knowledge distillation. The solution is to use the soft labels from an overfitted teacher model to train a student model, and the paper analyzes the fairness of the solution theoretically. The experiment results on several extensively-used datasets indicate that the proposed method enhances the fairness in the outcome without loss of Accuracy. The main contribution of the paper is that the authors propose an alternative method when demographic information is not available in the training set. ", "questions": "As for the questions and suggestions, please refer to the Weakness in the “Strengths And Weaknesses”. Nevertheless, I still have some concerns:\n1. Are the teacher model and student model both trained without the demographic data? Why the teacher model is expected to overfit training data (in line 250)?\n2. Did the paper miss some important baselines or references? For example, \n(1)\tCelis, L. E., Huang, L., Keswani, V., & Vishnoi, N. K. (2021, July). Fair classification with noisy protected attributes: A framework with provable guarantees. In International Conference on Machine Learning (pp. 1349-1361). PMLR.\n(2)\tCelis, L. E., Mehrotra, A., & Vishnoi, N. (2021). Fair classification with adversarial perturbations. Advances in Neural Information Processing Systems, 34, 8158-8171.\n(3)\tTianxiang Zhao, Enyan Dai, Kai Shu, and Suhang Wang. 2022. Towards Fair Classifiers Without Sensitive Attributes: Exploring Biases in Related Features. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining (WSDM '22). Association for Computing Machinery, New York, NY, USA, 1433–1442.\nOther details are as above.\n", "limitations": "Please refer to the main reviews.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is clearly organized and well written. \n2. The authors propose an alternative method when demographic information is not available in the training set. The paper analysis the impact of label smoothing and knowledge distillation on fairness theoretically.\n3. The experiments demonstrate the effectiveness of the proposed framework, i.e. the improvement of fairness without loss of Accuracy performance.\n4. The research direction is interesting and the proposed method is relevant to many domains.\n\n\nWeaknesses：\n1. At the beginning of the paper, it needs a better description of the motivation and novelty; it's hard to know where the challenge is to solve the fairness problem through knowledge distillation. \n2. Besides, except for the theoretical proof of the knowledge distillation on the fairness problem, the paper needs to introduce the reason why it is useful empirically or intuitively to help readers understand better. \n3. The interpretability of the proposed method should be discussed. For example, when demographic information is not available, how to discriminate different groups, such as gender.\n4. The procedure of the proposed method by authors on the fairness is not shown logically and intuitively. Adding a figure and refining relevant paragraphs to describe the procedure of how the knowledge distillation can improve fairness may help readers who is not familiar with fairness understand better.\n5. There are some typos in this paper. For example, the “individual” shall be “individual” in line 148, and the formula in line 227 may be wrong.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iY5AMU62LSy", "reviewer": "Reviewer_Sqvi", "summary": "This paper proposes to use knowledge distillation to improve fairness in the student model. The key idea is to use soft labels from the teacher, which essentially poses a smoothness constraint in the student model, to improve fairness (disparate impact and equalized odds).\n\nThe authors did experiments on 3 datasets, new Adult, COMPAS, and CelebA, and showed that the proposed method improves upon existing baselines, including a DRO based approach and an adversarial re-weighting based approach.", "questions": "- Can the authors include a baseline that does direct label smoothing on hard labels in training?\n\n- Can the authors add capacity-separated baselines, like Teacher + DRO / ARL?\n\n- Would a combination of Teacher + DRO / ARL + label smoothing further improve fairness? i.e., you distill a model from Teacher + DRO/ARL through the label smoothing you proposed.\n\n- Minor: in Lahoti et al., 2020, the AUC doesn't seem to decrease significantly across Adult (the older dataset) and COMPAS, why it shows a significant drop on the student model compared to student (with hard label) across Table 2/3/4/5?", "limitations": "Discussed in Section 5.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper is well written. The key idea is clearly presented. The authors first show that label smoothing can help fairness, and then proposed a reasonable approach to practically implement this through knowledge distillation when continuous labels are not directly available.\n- The idea of imposing smoothness in the model to improve fairness through knowledge distillation is novel to the best of my knowledge.\n- The authors did experiments on 3 datasets, and the results show that the proposed method works universally well across all datasets, outperforming existing DRO and adversarial-reweighting baselines.\n\nWeaknesses:\n- Since the authors study label smoothing, one obvious baseline seems to be missing, which is training the model by smoothing the binary labels directly from [1], i.e., using $y(1-a) + a/K$ where $a$ controls the smoothness and $K$ is the number of classes. \n\n[1] When Does Label Smoothing Help? Muller et al. NeurIPS 2019.\n\n- By looking at the results, there might actually be multiple components that contribute to the final fairness which are entangled together. (1) label smoothing does help the final fairness, as mentioned in Section 3.2. (2) model capacity also seems to contribute to the fairness, as the student trained with hard label already has much better fairness across multiple datasets. \n\nFor completeness, I think the authors should disentangle those factors, i.e., separating the tables into two parts, \n- first part is high-capacity model only (i.e., the teacher), and there should be baselines like a) Teacher (with hard label), b) Teacher with direct label smoothing, c) Teacher with DRO/ARL; \n- then the second part should be low-capacity model only (i.e., the student), and you should list all the results currently presented. \n\nThis should give people a clearer picture about the contribution of each component.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WmB3klgK_d", "reviewer": "Reviewer_PRCV", "summary": "The authors argue that training on distilled versions of existing classifiers improves fairness on unobserved demographics and build upon this objective. The approach is interesting and of use to the fairness literature, improving upon previous works. The paper is well-written, the experiment methodology is sound, and results convincing. ", "questions": "Please see my comments on Strengths and Weaknesses section.", "limitations": "- Differences in Table 1 do not seem statistically significant in terms of disparate impact with a first glance due to high variance, so maybe a little more discussion is needed to address this (they are statistically significant for equalized odds though).\n- In my opinion, the fact that the hyperparameter `a` can only be tweaked means that it is currently selected blindly. I think the authors need to be a little more explicit over this shortcoming. Maybe future works can try selecting that hyperparameter with few-shot tweaking (have sensitive attribute for a few samples and then reach an acceptable trade-off)?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I have a few important concerns, though these are -in my opinion- straightforward to address:\n\n- Given that the authors already arrived at Equation 9, why not use it directly for reweighting and then new training of the base model? I guess the only difference is whether new sample weights or new equivalent objectives are introduced, in which case I would like to see some discussion of this point in the text.\n- The question of why disparate impact is improved is not really addressed. The improved theorization editing I mention in the following point could help address this, the authors can give a simple intuitive explanation / proposition, or ideally (though this could be too much work) a second theorem could be introduced. I remind that fairness is not a global objective and that efficacy over each sub-objective needs to be explored by its own. If none of the aforementioned actions are taken, I would also be happy seeing the paper emphasize more on improving equalized odds as the primary goal and mentioning that disparate impact is also improved as a side-effect (at least one hypothesis of why this holds true is needed - maybe something that starts with the assumption that parts of these biases could be correlated so partially fixing one also partially fixes the other and then working from there?).\n- The authors argue in section 3.4 that the effectiveness of their approach can be attributed to down-weighting misclassified samples, which has been shown [Krasanakis et al., 2018] to be inherently insufficient by itself. As a recommendation on how to improve this (other explanations could be possible, perhaps even more intuitive depending on phrasing) with small adjustments to the text is to acknowledge that Equation 9 effectively introduces a different degree of weight skewing for highly misclassified samples (i.e. weights are not linear with respect to errors, since t(x) is also implicitly affected by errors) . So, protected demographics (i.e. the groups of people classifiers often bias against) are expected to obtain the highest misclassification rates and this is why placing disproportionately more weights on higher misclassifications addresses bias without impacting accuracy much.\n\nJust a note:\nIt seems to me that Equation (9) is an ad-hoc variation of [Krasanakis et al., 2018]'s Equation 10 with some additional assumptions filling in for the absence of demographics. More analysis is needed to show equivalence for sure, and it is not needed by the paper's theorization, so I'm just mentioning this in case the authors find it interesting. If this holds true, there is a chance that tweaking the softmax's hyperparameter instead of selecting it with cross-validation could also create accuracy and fairness trade-offs, as already demonstrated for the hyperparmeter `a`, since the former implicitly affects the Lipshitz constant of sample weights.\n\nMinor recommendations:\n- in the abstract \"Max-Min fairness objective.\" > \"Max-Min fairness objectives.\"\n- in the abstract \"to achieve expected improvement in\" > \"to improve\"\n- I think the introduction needs a sentence to explain *why* the proposed approach is expected to trade-off fairness with accuracy.\n- It would help adding [1] as a general background reference and [2] to labelling bias references in 3.2.\n- If I understand this correctly, in section 3.2, probably explicitly mention that Binary corresponds to the original version of the Adult dataset.\n\n[1] Ntoutsi, E., Fafalios, P., Gadiraju, U., Iosifidis, V., Nejdl, W., Vidal, M. E., ... & Staab, S. (2020). Bias in data‐driven artificial intelligence systems—An introductory survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(3), e1356.\n[2] Iosifidis, Vasileios, and Eirini Ntoutsi. \"Adafair: Cumulative fairness adaptive boosting.\" Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2019.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept