Fairness via Representation Neutralization
Authors: Mengnan Du
Subhabrata Mukherjee
Guanchu Wang
Ruixiang Tang
Ahmed Hassan Awadallah
Xia Hu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Algorithmic bias
Fairness mitigation
Explainability

Abstract: Existing bias mitigation methods for DNN models primarily work on learning debiased encoders. This process not only requires a lot of instance-level annotations for sensitive attributes, it also does not guarantee that all fairness sensitive information has been removed from the encoder. To address these limitations, we explore the following research question: Can we reduce the discrimination of DNN models by only debiasing the classification head, even with biased representations as inputs? To this end, we propose a new mitigation technique, namely, Representation Neutralization for Fairness (RNF) that achieves fairness by debiasing only the task-specific classification head of DNN models. To this end, we leverage samples with the same ground-truth label but different sensitive attributes, and use their neutralized representations to train the classification head of the DNN model. The key idea of RNF is to discourage the classification head from capturing spurious correlation between fairness sensitive information in encoder representations with specific class labels. To address low-resource settings with no access to sensitive attribute annotations, we leverage a bias-amplified model to generate proxy annotations for sensitive attributes. Experimental results over several benchmark datasets demonstrate our RNF framework to effectively reduce discrimination of DNN models with minimal degradation in task-specific performance.

Fairness via Representation Neutralization
Mengnan Subhabrata Mukherjee', Guanchu Wang', Ruixiang Tang', Ahmed Hassan Awadallah', Xia Hu
Teeas A&M University ?Microsoft Research RRiee University
dumengnan@tamu. edu, {submukhe, aataaaaammmmiirooofotoooooo (guanchu.o wang, rt39, xia. werieiiee........ edu
Abstract
Existing bias mitigation methods for DNN models primarily work on learning debi- ased encoders. This process not only requires a lot of instance-level annotations for sensitive attributes. it also does not guarantee that all fairness sensitive information has been removed from the encoder. To address these limitations, we explore the following research question: Can we reduce the discrimination of DNN models by only debiasing the classification head, even with biased representations in- puts? To this end, we propose new mitigation technique, namely, Representation Neutralization for Fairness (RNF) that achieves fairness by debiasing only the task-specific classification head of DNN models. To this end, we leverage samples with the same ground-truth label but different sensitive attributes, and use their neutralized representations to train the classification head the DNN model. The key idea of RNF is to discourage the classification head from capturing undesirable correlation between fairness sensitive information in encoder representations with specific class labels. To address low-resource settings with access sensitive attribute annotations, we leverage a bias-amplified model to generate proxy an- notations for sensitive attributes. Experimental results over several benchmark datasets demonstrate our RNF framework to effectively reduce discrimination DNN models with minimal degradation in task-specific performance.
Introduction
and have been deployed in many real-world applications. However, DNNs often suffer from biases and show discrimination towards certain demographics, especially high-stake applications, such as criminal justice, employment, loan approval, credit scoring, etc 4 S 6). For example, COMPAS, an algorithmic recidivism predictor, is likely to associate African-Ami American offenders with higher risk scores compared to Caucasians while having a similar profile [7]. This brings significant harm to both society and individuals, thus leading to recent focus on mitigation techniques alleviate the adverse effects of DNN biases.
Deep neural networks (DNNs) have made significant advances in recent times 2 3
Existing debiasing methods usually work on learning debiased representations at the encoder- level. One representative family of methods perform mitigation by explicitly learning debiased representations, either through adversarial learning [ 9 10) invariant risk minimization [1] [2]
Another family of methods 13 14] [5] implicitly learn debiased representations by incorporating explanation during model training to suppress it from paying high attention to biased features in the original input. Essentially, the above methods aim to remove the bias from deep representations.
Learning debiased representations is technically challenging problem. Firstly, it is hard to remove all fairness sensitive information in the encoder. The suppression of fairness sensitive information
Part the work was done while the first author was an intern at Microsoft Research.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

might also remove useful information that is task relevant. Secondly, most existing debiasing methods assume access additional meta-data such as fairness sensitive attributes and a lot of annotations corresponding to the protected groups to guide the learning of debiased representations. However, such resources are expensive to obtain, if not unavailable, for most real world applications.
To address these limitations, we explore the following research question: Can we reduce the discrim- ination of DNN models by only debiasing the task- mask-specific classification head, even with a biased representation encoder? Our work is motivated by the empirical observation that standard training can result in the classification head capturing undesirable correlation between fairness sensitive information and specific class labels. Some recent works [16 17 [8] have explored such spurious or shortcut learning behavior of DNNs in various applications. To this end, we propose the RNF (Representation Neutralization for Fairness) framework for mitigation, motivated by the Mixup work 19 20) We first train a biased teacher network via standard cross entropy loss. the second stage, we freeze the representation encoder of the biased teacher, and only update the classification head via representation neutralization. This discourages the model from associating biased features with specific class labels, and enforces the model to focus more on task relevant information. To address low-resource settings, our RNF framework does not require any access to the protected attributes during training. To this end, we train a bias-amplified model using generalized cross entropy loss that is used to generate proxy annotations for sensitive attributes. Experimental results over several benchmark tabular and image datasets demonstrate our RNF framework to significantly reduce discrimination of DNN models with minimal degradation of the task performance. The major contributions of our work can be summarized as follows:
We analyze bias propagation from the encoder representations to the final task-specific layer demonstrating that DNN models heavily rely on undesirable correlations for prediction. We introduce RNF, a bias mitigation framework for DNN models via representation neutralization. Our RNF framework achieves mitigation without any access to instance-level sensitive attribute annotations, and instead relies on self self-generated proxy annotations.
Experimental results on several benchmark datasets demonstrate the effectiveness of our RNF framework via debiasing only the classification head while using biased representations as input. Additionally, we show RNF to be complementary to existing methods that learn debiased encoders and can be further improved within our framework.
2 Related Work
We briefly review bias mitigation and broader robustness literature which are most relevant to ours.
Bias Mitigation. Recent studies have indicated that DNN models exhibit social bias towards certain demographic groups. This has led to increased attention to bias mitigation techniques in recent times (21 22 23). Existing mitigation methods can be generally grouped into three broad categories. The first one is based on adversarial training (8 9 [0]. It leads to fair classifier as the predictions cannot carry any group discrimination information that the adversary can exploit. However, this method assumes that the sensitive attribute annotations are known, and uses those annotations to learn debiased representations. The second representative family of mitigation methods is based on explainability [13] [4] [5] 24 These methods require fine-grained feature-leved annotations to specify which subset of features are fairness sensitive. The third category falls under the umbrella of causal fairness 25 [2). The main idea is to enforce the model to concentrate more on task relevant causal features, and getting rid of superficial correlations (66.. This results in the model capturing debiased representations. For instance,
27 minimizes the correlation between sentence representations and bias words using a contrastive learning framework. We can typically decouple the classification problem into representation learning and classification as the two major parts (28). Different from the above-mentionel methods that mainly aim to train the debiased representations, our work focuses on the classification head and aims to suppress it from capturing undesirable correlation between fairesss sensitive attributes and class labels.
Shortcut Learning in DNNs. Our work is motivated by the observation that DNNs with standard training are prone to exploit undesirable correlations (or shorteuts in the dataset) for prediction [16] 29 Beyond algorithmic discrimination, recent studies show that shoricut learning [17 can result in other undesirable consequences like poor generalization and adversarial vulnerability. Specifically, this leads to high performance degradation for previously unseen inputs, especially for those data beyond
2

---

Before Neutralication Famale and male
Neutralization Neutralized feature
(dt Neutrallzation Pasitive and Nogative
Positive and Nogative
Figure I: Representation analysis of = using Kernel PCA. (a) Protected attribute a (i.e., gender) is discriminative feature. In this plot, the male group primarily lies in the lower left interval, whereas the female group is located primarily on the upper right interval. (b) Predicted positive label and negative task-label distribution. (c) Representation neutralization (please refer Sec. 3.3) reduce the discriminative power of for dimensions relevant to the sensitive information, comparing to the distribution in (a). (d) Neutralization still preserves the useful task relevant information.
held-out test set. Representative tasks depicting such behavior include reading comprehension [3], natural language inference [8]. visual question answering and deepfake detection [32].
The most similar work to ours also regularizes the model training using interpolated samples between groups [33 Similar to the aforementioned solutions, this work also heavily relies on sensitive attribute annotations in the training set. This limits the application scenarios of the mitigation algorithms, especially for many real-world applications without readily available annotations dealing with sensitive attributes. Our representation neutralization framework achieves comparable or better performance to such techniques without relying on such sensitive annotations by leveraging proxy annotations obtained from bias-intensified version of our framework, thereby, making it broadly applicable to arbitrary real-world upplications.
3 Representation Neutralization for Fairness
In this section, we first analyze the task-specific classification head of DNN to examine how bias propagated from the encoder representation layer to the task-specific output layer (Section (3.2).
We empirically demonstrate the undesirable correlation between fairness sensitive information in representations with specific class labels. Based on the observation, we introduce the Representation Neutralization for Fairness (RNF) framework to debias DNN models (Section 3.3) Finally, we pro- pose an approach to generate proxy annotations for sensitive attributes, enabling the RNF framework to be applicable to low-resouree settings with access to sensitive attribute annotations (Section 3.4).
3.1 Notations
We first introduce the notations used in this work. Let X' {x,, Vi, (,,a; .....,N N be the training set, where z; is the input feature, y/s denotes the ground truth label, and a, represents the sensitive attribute (e.g., gender, race, age). For case of notation, in the following sections, binary sensitive attributes Nevertheless, our proposed mitigation framework can also be applied to non-binary sensitive attributes (e.g., age).
consider
Consider the classification model (a. 0) (z)( parameterized with the model parameters. Here (xx)= X - Z represents the feature encoder, and g(x) = 2 is the representation for x obtained from a DNN model. The predictor c(=) - y is the multi-layer classification head, It is depicted input and maps it to
by the top layer(s) of the DNN, which takes the encoded representation softmax probability. The final class prediction is denoted by ÿj = arg max c(z). In this work, we aim to reduce the discrimination of DNN models by only debiasing the classification head c(2), with the biased representation encoder (αx) as input.
*Gender not binary in reality as there are many different gender identities, such as male, female, transgender, to name few. In this work, we consider gender as binary sensitive attribute due to the limitation of the benchmark dataset that encodes gender binary variable.

---

3.2 Analysis of the Classification Head
In this section, we examine how bias manifests in the representation space Z as well how the classification head c(z) obtained with standard training scheme propagates bias from the representa- tion layer to the model output layer. To this end, we train a biased network (x) via standard cross entropy loss, where the following experiment is performed using the Adult dataset [4].
Representation Probing Analysis. For the Adult training set, we generate representation vectors for 500 training samples using the biased network fz( and project them in 2D for visualization. To this end, we utilize the Kernel Principal Component Analysis (KPCA) 35 with a sigmoid kernel, which is a tool to visualize high-dimensional data. Since the classification head c(2) contains multiple non-linear layers, we choose kemnel PCA instead of a linear dimensionality reduction method such as linear PCA. The visualization is shown in Figure [ (a). The plot depicts that the low dimensional projection separates the two protected groups in two areas, where the male group is primarily located in the lower left area, whereas the female group primarily occupies the upper right area. Comparing the task-label y distribution in Figure I (b) with the protected group distribution in Figure (a), we observe that the protected attribute information is a discriminative feature that could be exploited by the task classification head for prediction.
Role of the Biased Classification Head. The above demonstrative analysis indicates that the model representation captures both useful task relevant classification information as well as bias information from protected attributes. Specifically, the model captures strong correlation between the fairness sensitive information and the class labels. On analyzing the data distribution, we observe this to be an artifact of the conditional label distribution with respect to the sensitive attributes being skewed. The model relies on this shorteut for prediction, resulting in bias amplification. We observe the male neurons to positively correlate to the desirable label (also refer to the experimental analysis in Sec. 4.3). whereas the female neurons positively correlate to undesirable label. This depicts an undesirable correlation between sensitive information with certain class labels in the model.
Our Motivation. Based on the above empirical observations, we propose to neutralize the training samples (Fig (c) so as to reduce the discriminative power of the fairness sensitive information, while at the same time preserving task relevant information (Fig (d)) With the neutralized training data, we propose to re-train the classification head. Our goal is to adjust the decision boundary to implicitly de-correlate the fairness sensitive information in the representation space with class labels.
3.3 Representation Neutralization for Debiasing Classification Head
Based on the aforementioned empirical observations, in this section we propose simple yet effective bias mitigation framework via Representation Neutralization for Fairness (RNF). RNF does not require any prior knowledge about existing bias in the representation space; nor does it require any knowledge about specific dimension(s) encoding the sensitive attributes making it widely useful for arbitrary applications. Our goal is to encourage the model to ignore the sensitive attributes and instead focus more on task relevant information.
RNF is implemented in two steps (see Figure 2. In the first step, we train the model using cross entropy loss, and obtain a biased teacher network fr (z). During the second step, we freeze the encoder g(x) for (I), and use it as our backbone encoder for learning representations, We then re-train only the classification head c(2) using feature neutralization (see Figure 2 (b)).
Representation Neutralization. To this end, while training the classification head, for an input sample (.1, y, 01) }. we randomly select another sample (I2, y, 02), with the same class label y but different sensitive attribute d2 compared to 01 in the input sample. Now we compute the corresponding representations 2 g(x1) and z2 = g(22)) and re-train the classification head using the neutralized representation 21 as input. For the supervision label for the classification head, we utilize
the neutralized soft probability y t2 after temperature scaling obtained as follows. Given the logit vector 21 for input x1, the probability for class is computed as P1. Σ, /TT) where
exp(s1
T21 1. This can regarded as a form of knowledge distillation (36), where > softens the softmax score. A larger temperature prevents the model from assigning over-confident prediction probabilities. special case for RNF when 1, where p1 and Pa are the standard softmax probabilities obtained from the biased teacher network fx)

---

Exiitingg methods on the debiased encoder
Z1
Encoder
Classification
P1
21
Z2
Classification head
+ Pz
1
head
2
2
Our RNF focuses
Z/
Neutralized feature
Neutralized label
dehiesed classification hredd
(a) Biased teacher network
(b) Representation neutralization
Figure 2: Debiasing with representation neutralization. (a) We first train biased teacher network using only cross entropy loss. For two inputs I1 and 22 that with the same class label y and different sensitive attribute
obtain the representations 21 and 22, and softened probabilities P1 and p2- (b) We freeze parameters of the biased encoder and only re-train the classification head using the neutralized representation +1 as input, and softened probability 2
t2a as supervision signal.
We use the knowledge distillation loss. particular, the mean squared error (MSE) loss is used as distance-bar based metric to measure the similarity between model prediction and the supervision signal.
(Ú:- y)² fel2 2 22) - (2Px + pa))²
LSE
where, is the classification head project representations to softmax prediction probability.
There are two main benefits of the aforementioned training scheme. From the input perspective, the neutralization of representations suppresses the model from capturing the undesirable correlation between fairness sensitive information in the representation with the class labels. From the output perspective, the softened label encourages the model to assign similar predictions to different sensitive groups. Optimizing Eq. (1) can lead to reduced generalization gap between the two groups.
satisfying 1|21 22l12 S X₂ Pi - Pal for 21 E X and a bounded loss function L(c(z)).p.) = (1-p:)l(c((:))) 0) p,l(c(z)),) y 1), where [(((z)) = j) S er. for x4 and 0,1; if the clas- sification head minimizes the loss between neutralized representation and soft probabilities,
Theorem Given well-trained representation encoder g(x)
i.e. V=L(c(2). 2= 21 pite
S tc. where
g(z), I1 P(x1 a, 0).
2
D2 P(12 a: =1 1). P1 P2l
S
Ep. the gap of generalization loss between groups
and a = 1 is bounded by:
E Eer~P(p: L(c(2)). Plyy
I Cp(Acce +())
Here "well-trained" means that the model has learned reasonably good representations to encode sufficient task relevant information. Essentially, after training for sufficient number of epochs until the validation loss has converged, we have access to a reasonably good representation space (although it might encode lot of fairness sensitive information). If train the classification head, the generalization gap between the different protected groups would be small. For more detailed proof, please refer to Section A in the Appendix.
the RNF loss function to further
Smoothing Neutralization. To further enforce the model ignore sensitive attributes, we construct syper-parameter À to control the degree of neutralization
augmented training samples using of the samples (æ1, Pl, Mn,yl y) and (=2.P2.
P2. )- The augmented neutralized sample is given by = λzι + (1 - 22, À E [g,1] 1). We encourage the classification head to give similar prediction scores for the augmented and the neutralized sample (with À )). The regularization loss is given by:
Smmooth
Σ xEl
lc(Azz + (1- >) al c + -=222) 22)11
By varying À we control the degree of sensitive information for the augmented samples. is utilized to penalize the large changes softmax probability when we move along the interpolation between two samples. We linearly combine the MSE loss in Eq. ( with the regularization term follows:
C = LMSE + aL
(4)
5

---

We train the classification head using the loss function in Eq. (4.. Eventually we combine the original encoder of fr(x) and re-trained classification head as the debiased student network fz(2). The teacher (a) is later discarded and the debiased student network fs( is used for prediction.
3.4 Generating Proxy Annotations for Sensitive Attributes
The aforementioned feature neutralization is limited in that it requires instance-level sensitive attribute annotations {ar)/=1 for all training samples. Such resource- eettencveett annotations are difficult to obtain for many practical applications particularly due to the nature of the sensitive attributes. To address this limitation, we propose method to generate proxy annotations {ā: J for the sensitive attributes based on the model uncertainty. The key idea is that a biased model generates over-confident predictions for one demographic group, while giving much lower scores for the alternative group. Particularly, the bias- bias-amplified model tends to assign the privileged group more desired outcome, while assigning the under-privileged group less-desired outcome. For instance in the Adult dataset, the average prediction probability of the desired label (higher income) for the male group is much higher than that of the female group. In contrast, the average probability of the less-desired label for the female group is much higher than that of the male group.
To better facilitate the model to generate uncertainty scores, we train another biased model by intentionally amplifying the bias via generalized cross entropy loss (GCE) 337. The bias-amplified
model is denoted as (T) and the loss function is given as follows:
GCE(J(Z;0),)) = 1 - (=;
(5)
q
where [a; 0) denotes the output probability for ground truth label y. The hyper-parameter q E (0, controls the degree of bias amplification. When limy--, the GCE loss in Equation (2) approaches -logp which is equivalent to standard cross entropy loss. The core idea is that for more biased samples, i... samples with larger fy (z; 8) value, the model assigns higher weights fß while updating the gradient.
a CE(p.) a GCE(P. v) = 1- 00 In this setup, the model (x) learns more from bias- stullampliiied samples compared to the model fr()) trained with standard cross entropy loss.
(6)
00
The confidence score of (I) is used to indicate whether a sample belongs to a privileged or unprivileged group. Specifically, for a desired ground truth label, samples with over-confident scores are grouped into the privileged group, whereas subsets of samples with low prediction scores are grouped into the unprivileged group. In contrast, for the undesired ground truth label, samples with over-confident scores are grouped into the unprivileged group, and vice versa. Based on this criterion, we generate proxy sensitive attribute annotation à for each training sample x to split samples into two groups, which are subsequently used for feature neutralization.
The overall process of our RNF mitigation framework is given in Algorithm 1, which con- tains two stages. In the first stage, we train
Algorithm 1: RNF mitigation framework.
Input: Training data D = {(x,, w)
the biased teacher network (z) and the bias-1
Set hyperparameter q. a.
amplified network [x)... In the second stage,
while first stage do Train teacher network (x and
we first use (x) to generate proxy sensitive at-a
tribute annotations for all training samples. We use the ratio to partition the training set to gen-4 erate proxy annotations for protected attributes that are subsequently used for feature neutraliza- tion. Note that the ratio 7 is determined by the faimess-accuraac trade-off on the validation set.? Then we use representation neutralization and the loss function in Eq. 4 to re-train the classi- fication head of fr fx)).. Eventually, we combine the original encoder s(x) of fr(a) and the re-
bias-amplified network (z(
while second stage do
Determine the splitting threshold y for f=x(w); Calculate proxy annotation à; for each training sample { based Use loss function Eq. (4) to re-train
and fa
classification head c(z) of fr(().
Output: The debiased student network (z)..
fined classification head c(z) to give us the debiased student network fx(x). is worth noting that the neutralization is merely performed during the training stage for only debiasing the classification

---

head. At inference time, the features just come in as encoded, and the classification head has learned to not exploit any of the information correlated with the sensitive attribute for prediction.
4 Experiments
In this section, we conduct experiments to evaluate the effectiveness of our RNF framework.
4.1 Experimental Setup
4.1.1 Fairness Metries, Benchmark Datasets and Baselines
We use two group fairness metrics: Demographic Parity (38 and Equalized Odds [39] Demographic Parity (DP) measures the ratio of the probability of favorable outcomes between unprivileged and privileged groups: DP
p(á=1ja=0) Equalized Odds (AEO) require favorable outcomes to be
pi p(g-Ila=T).
independent of the protected class attribute a, conditioned on the ground truth label y. Specifically, calculates the summation of the True Positive Rate difference and False Positive Rate difference:
AEO = (P( = 1 | = 0,y 1)- Pl =1 a 1, = 1)) + (P() = 1 | 0,y = P(ý=1 a 1=1,y 1.y = 0}}- (7)
Under the above metrics, it is desirable to have DP value closer to 1 and AEO value closer to 0.
We use two benchmark tabular datasets and one image dataset to evaluate the effectiveness of RNF. For the Adult income dataset (Adult), the goal is to predict whether a person's 's income exceeds S50K/yr [34]. We consider gender the protected attribute where vanilla trained models show discrimination towards the female group by predicting females to earn less. For the Medical Expenditure dataset (MEPS), we consider two groups white and non-white 44) Here the task is to predict whether person would have 'high' utilization, where vanilla DNN shows discrimination towards the non- non-white group. The CelebFaces Attributes (CelebA) dataset is used to predict whether the hair in an image is wavy or not 41 We consider two groups male and female, where vanilla trained models show discrimination towards the male group. We split all datasets into three subsets with statistics reported in Table More details of the datasets included in Sec.
Table I: Dataset statistics.
Adult
MEPS
CelebA
# Training
33120
11362
194599
&Validation
3000
1200
4000
# Test 9102
3168
8000
Sec.Ciin Ci the Appendix.
We compare two variants of our framework RNF (using proxy sensitive attribute annotations) and RNF (using ground truth sensitive attribute annotations) against baselines such as DNNs trained using only cross entropy loss (referred as Vanilla) and two regularization based mitigation methods, namely, adversarial training (Adversarial) Among them, the Adversarial method achieves fairness via learning debiased representations, whereas EOR directly optimizes the EO metric in Eq. 7 All three baselines control the faimess--accuracy trade-off via hyper ppper-paramtter... More details on the baselines are included in Sec. D in the Appendix.
42 and Equalized Odds Regularization (EOR) (43).
4.1.2 Implementation Details
For the image classification task, we use ResNet-18 [ (we add one more fully connected layer). We set the representation encoder g(x) as the convolutional layers and use the remaining two fully connected layers as the classification head c(z). For tabular datasets, we use a three-layer MI.P (multilayer perceptron) as the classification model, where the first layer is set as the encoder and the remaining two layers are used as the classification head. Dropout is used for the first two layers with the dropout probability fixed 0.2. We use the same batch size of 64 for tabular datasets and 390 for the image dataset. For selecting another random sample {I2, y, 02 } to be neutralized with current sample {a1, y. a.), we perform the selection within the current batch of training data. The hyper- parameters (e.g., learning rate and training epoches) are determined based on the model performance on the validation set, and early-stopping based on validation performance used to avoid overfitting. The optimal temperature T used to calculate the probability is set as 2.0, 5.0, 2.0 for Adult, MEPS and CelebA datasets respectively. For Eq. (3), we sample À from the list [0.6, 0.7, 0,,,0.8.... 0.9). The papee--arammttrr g in Eq. (5) is set as 0.2, 0.3 for Adult, MEPS and CelebA datasets respectively.
0.3
7

---

0.060
0.70
0.855
Accuracy
Accuracy
0.850
Accuracy
0.845
0.72
0.840
RNI
RNI
0.70
RNI
Adversarial
Adversarial
EOR
EOR
EOR
Vanilia
0.030
Vanilia
Vanila
0.75
0.80
0.95
0.820 .080
0.85
0.95
0.66
0.675 0.700 0.725 0.750 0.775 0.800
Demographic Parity
Demographic Parity
Demographic Parity
0.860
0.78
0.76
Accuracy
0.850
Accuracy
0.74
0.72
RNF-GT
RNF-GT
RNF-GT
RNF
0.040
RNF
0.70
RNF
EOR
EDR
0.60
EOR
Vanilla
0.830
Vanilla
Vanila
-0.00
-0.15
-0.30
-0.05
0.00
-0.25
-0.20
-0.15
-0.10
0.50
-0.45
-0.40
-0.35
-0.30
Equality of Odos
Equality of Odds
Equality Odds
(a) Adult
(b) MEPS
(c) CelebA
Figure 3: The fairness-an canness-accurayy curve comparison of RNF and other baselines. The first and second row depict the DP accuracy and AEO accuracy trade-off curves, respectively. Note that there is certain level of variance for each baseline method, and we report the average over 10 runs.
4.2 Mitigation Performance Analysis
We compare the mitigation performance of RNF (with proxy attribute annotations) and RNFGr (with ground-truth attribute annotations) with other competing methods and illustrate their fairness-accuracy curves for the three datasets in Figure 3 The hyper-paramette a in Eq. (4) controls the trade-off between accuracy and fairness for RNF. For Adversarial and EOR, we vary their regularization weights to obtain the corresponding performance curves. As the random seeds lead variance in the accuracy and fairness metrics (please refer to Section E.4 in the Appendix for detailed analysis of the variability effect), we train the model for 10 times with different seeds and report the average result. Overall, we make the following key observations.
Even though RNF does not rely on annotations for the sensitive attributes, it performs similar to baseline methods with access to such information, e-g, Adversarial training, or better than them in some cases. This makes RNF readily usable for real-world applications where protected attributes are not available the training set.
RNFGr improves mitigation performance over RNF by 10% on an average across all datasets and metries, thereby, demonstrating the benefit of using ground truth sensitive attribute annotations.
The soft labels of RNF and RNFGT (obtained using higher temperature discourage the model to assign overconfident predictions, thereby, suppressing it from capturing undesirable correlation between fairness sensitive information and class labels. Penalizing the large changes probability we move along the interpolation between two samples further suppresses the model from capturing the undesirable correlation.
Direct optimization of the equality of odds metric (i.e., EOR) achieves comparable performance RNF for all the datasets. However, has limited improvement in terms of the demographic parity metric. Note that EOR requires instance-level annotations for the protected attributes.
We observe that Adversarial training also performs effective mitigation by learning debiased representations. However, this happens at the expense of a higher accuracy drop for the task performance. This likely results from the loss of task relevant information while suppressing sensitive information from the representations. Additionally, we observe adversarial training to be unstable, especially for relatively complex task like image classification.

---

4.3 Classification Head Analysis
In this section, we use explainability and an auxiliary prediction task analyze the classification head c(z) for the Adult dataset. Particularly, we leverage explainability as debugging tool to analyze the attention difference between Vanilla and RNF model with respect to the representations.
Auxiliary Sensitive Attribute Prediction Task. We perform a representation analysis using an auxiliary prediction task. To this end, we train another linear classifier to predict sensitive attributes using the biased representation ((x) = 2 as input and the sensitive attribute annotations {a, } as the supervision signal. The linear classifier denoted by LSENS (z) = W + b, where W and b represent the weight matrix and bias for the linear classifier respectively. The weight matrix W can be used measure the degree of bias in each dimension of the representation 2.
Explanation Analysis. We use post-hoe explainability [44] to analyze the contribution of the classification head c(2). Our goal is to figure out the contribution of each dimension within the biased representation (xx) = 2 towards the model prediction f(z, 8) = (z)). We train a linear classifier Trexplan( 2) = W b to mimic the decision boundary of the multi-layer classification head c(z).
We compare the weight matrix of the two linear classifiers LSENS (2)
and Lexplan (z) using cosine similarity. For models trained using Adult dataset, we extract the weight matrix corresponding to the protected attribute Male and task label Positive, and then we calculate the cosine similarity between the Male vector and the Positive vector. This fol- lows from our observation in Figure I that the vanilla model makes use of male relevant information to make positive predictions. For the Vanilla model and RNF models listed in Figure B (a), calculate the cosine similarity and report the DP-Similarity performance Figure We observe that RNF dramatically reduces the cosine similarity be- tween positive predictions and male relevant information compared Vanilla (from 0.272 0.075), by adjusting the decision boundary.
0.25
Cosine SimBurity
0.20
RNF-CT
Varitb
0.05
0.75
0.80
0.85
0,50 0.95
Domographic Panty
Figure 4: Analysis for the classification head.
This eventually helps the head c(2) to shift its attention from fairness sensitive information to task relevant information.
4.4 Effectiveness of GCE Loss
We use MEPS dataset to examine the GCE loss in terms of generating proxy annotations for the protected attributes. The results are shown in Table Firstly, we compare the fairness performance of GCE loss with standard CE loss (Vanilla). As expected, we observe that GCE is more biased than Vanilla in terms of the fairness metrics with 0.2% accuracy difference. Secondly, we compare RNF with several of its variants: 1) replacing GCE loss with cross entropy (CE) loss generate proxy labels, and 2) using random annotations for the pro- tected attributes. For fair comparison, except for the protected attribute annotations, we use the same set of hyper-parameten for different variants. Even though the same level of regularization is performed for the three RNF variants, RNF-GCE has higher mitigation performance than RNF-CE with 3.0% DP improvement and only 0.3% accuracy difference, thereby, demonstrating the effective- ness of GCE terms of separating protected groups. Another observation is that even using random annotations, our RNF framework could achieve certain level of mitigation. This is because partial samples contain different sensitive attributes, which still could serve neutralization purpose.
Table Ablation analysis.
MEPS
Models
Accuracy
DP
AEO
Vanilla
0.862
0,866
-0.210
GCE
0.860
0.839
-0.249
RNF-GCE RNF-CE
0.839
0.964
-0.099
0.842
0.936
-0.128
RNF-Random
0.856
0.902
-0.163
4.5 Varying Layers for the Classification Head
We use MEPS and CelebA two datasets to study the effect of an important our framework in terms of which layers to use for the encoder. Our classification model f(x contains the encoder g(x) and task predictor e(z). In this experiment, we vary the depth of the representation layer to examine the effect of
Table 3: Varying the layers for classification head.
of
MEPS
Celeb
Models
Accuracy
DP
ABO
Accuracy
DP
ABO
RNF Last
0.839
0.964
-0.099
0.668
0.836
-0.290
0.837
0.971
-0.076
0.641
0.966
-0.052

---

layer selection. In particular, we investigate the question: can we debias only the very last layer, with the remaining layers as the biased encoder? We report the results for MEPS and CelebA datasets in Table 3 where the second row depicts the result of debiasing only the last layer. On MEPS dataset, R.FF-Last has better performance in terms of the two fairness metrics with only 0.2% accuracy difference. Similarly, for CelebA dataset, with 2.7% accuracy difference, RNF-Last has better fairness performance (with 13% DP improvement). This demonstrates that debiasing only the last layer can achieve similar performance compared to debiasing the last several layers.
4.6 Representation Neutralization with Debiased Encoder
In the discussions so far, we reported the performance of RNF while debiasing only the classification head. In this section, we analyze the impact of RNF built on top of a debiased encoder. We first use Adversarial or EOR training to learn a debiased encoder - which is subsequently used as the backbone encoder for updating the classification head using RNF. This experiment is performed on the MEPS dataset where both Adversarial and EOR methods achieve competitive performance. We use the same hyper-paramedte single point in the fairness accuuracyy curve. The results are shown in Table 4 We observe RNF EOR to achieve better fairness performance over RNF, with DP metric improvement of 1.6% and AEO metric moves closer to O. However, such improvement in the fairness metrics incur some loss in the task performance where the accuracy reduces by 0.5%. We observe a similar trend with the combination of RNF and Adversarial training, where the joint combination improves the fairness metrics DP and AEO. Similar to the previous case, this faimess improvement is achieved at the expense of some task performance degradation, where the accuracy drops by 1.3%. This indicates that our RNF is complementary to using a debiased encoder where the joint combination performs better than either of them in terms of the faimess metrics with some loss task performance.
Table 4: RNF with debiased encoder.
MEPS
Models
Accuracy
DP
AEO
Vanilla RNF
0.862
0.866
-0.210
0.839
0,964
-0.099
RNF_EOR BOR
0.834
0,980
-0.049
RNF Adv__sversarial
0.826
0.971
-0.085
for different RNF variants and report a
5 Conclusions and Future Work
In this work, we demonstrate that even when input representations are biased, we can still improve fairness by debiasing only the classification head of the DNN models. We introduce the RNF framework for debiasing the classification head by neutralizing training samples that have the same ground truth label but with different sensitive attribute annotations. To reduce the reliance on sensitive attribute annotations (as used in existing works), generate proxy annotations by training a bias- intensified model and then annotating samples based on its confidence level. Experimental results indicate our RNF framework to dramatically reduce the discrimination of DNN models, without requiring access to annotations for the sensitive attributes for all the training samples. Experimental analysis further demonstrates our RNF framework to further improve in conjunction with other debiasing methods. Specifically, our RNF framework built on top of a debiased backbone encoder leads to better mitigation performance with negligible accuracy drop in the task performance. It is worth noting that our RNF framework could help alleviate the discrimination rather than eliminate it.
On the other hand, the experimental analysis indicates a mitigation gap between RNF and RNFGr- This is because we assume zero access to the protected attribute annotations when using GCE framework. It is desirable to further boost the quality of the generated proxy annotations. In real- world applications, domain experts could be involved to annotate a small ratio of the samples for the training set. Equipped with this small ratio of high quality protected attribute annotations, generate proxy annotations for other training samples with a higher accuracy compared to the proxy annotations generated by GCE framework. As such, we can further boost the mitigation performance of RNF This is a challenging topic and would be explored in our future research.
could
Funding Transparency Statement
The work is in part supported by NSF grants views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
IIS- and 193979..... The
10

---

References
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2016.
[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. 2019 Annual Conference of the North American Chapter of the Association for Computatiional Linguistics (NAACL), 2019.
[3] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (NeurIPS), 2012.
[4] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. survey on bias and fairness in machine learning. ACM Computing Surveys, 2021.
[5] Mengnan Du, Fan Yang, Na Zou, and Xia Hu. Fairness in deep learning: computational perspective. IEEE Intelligent Systems, 2020.
[6] Rachel KE Bellamy, Kuntal Dey, Michael Hind, Samuel C Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, et al. Ai fairness 360; An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv preprint arXiv: 1810.01943, 2018.
[7] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. wwtp::/wwwwpppicccccooaaaacceeie 2016.
Machine bias.
[8] Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Jummo Kim. Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2019.
[9] Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Wei Chang, and Vicente Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. International Conference on Computer Vision (ICCV), 2019.
[10] Yanai Elazar and Yoav Goldberg. Adversarial removal of demographic attributes from text data. 2018 Conference Empirical Methods in Natural Language Processing (EMNLP), 2018.
[11] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint arXiv: 190.:02:9070089 2019.
[12] Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In International Conference on Machine Learning (ICML), 2020.
[13] Krishna Kumar Singh, Dhruv Mahajan, Kristen Grauman, Yong Jae Lee, Matt Feiszli, and Deepti Ghadiyaram. Don't judge object by its context: Learning to overcome contextual bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[14] Andrea Zunino, Sarah Adel Bargal, Riccardo Volpi, Mehrnoosh Sameki, Jianming Zhang, Stan Sclaroff, Vittorio Murino, and Kate Saenko. Explainable deep classification models for domain generalization. arXiv preprint arXi::2003.064493 2020.
[15] Laura Rieger, Chandan Singh, W James Murdoch, and Bin Yu. Interpretations are useful: penalizing explanations to align neural networks with prior on Machine Learning (ICMI.), 2020.
prio knowledge. International Conference
[16] Mohammad Pezeshki, Dékou-Oumal Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity neural networks. arXiv preprint arXiv:2011.09466 2020.

---

[17] Robert Geirhos, Jern-Kerrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning deep neural networks. Nature Machine Intelligence, 2(1 2(11):663 1:665-6733 2020.
[18] Timothy Niven and Hung- Yu Kao. Probing neural network comprehension of natural language arguments. 57th Annual Meeting of the Association for Computational Linguisties (ACL), 2019.
[19] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations (ICLR), 2018.
[20] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning (ICML), 2019.
[21] Hao Wang, Berk Ustun, and Flavio Calmon. Repairing without retraining: Avoiding disparate impact with counterfactual distributions. In International Conference on Machine Learning (ICML), 2019.
[22] Christina Wadsworth, Francesca Vera, and Chris Piech. Achieving faimess through adversarial learning: an application to recidivism prediction. Faimess,, Accountability, and Transparency in Machine Learning (FAT/ML), 2018.
[23] Harrison Edwards and Amos Storkey. Censoring representations with an adversary. International Conference on Learning Representations (ICLR), 2016.
[24] Frederick Liu and Besim Avei. Incorporating priors with feature attribution on text classification.
57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
[25] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In Advances in neural information processing systems (NeurIPS), 2017
[26] Niki Kilbertus, Mateo Rojas Carulla, Giambatiisa Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Schölkopf. Avoiding discrimination through causal reasoning. In Advances Neural Information Processing Systems (NeurIPS), 2017.
[27] Pengyu Cheng, Weituo Hao, Siyang Yuan, Shijing Si, and Lawrence Carin. Fairfil: Contrastive neural debiasing method for pretrained text encoders. International Conference on Learning Representations (ICLR), 2021.
[28] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. International Conference on Learning Representations (ICLR), 2020.
[29] Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. Learning from failure: Training debiased classifier from biased classifier. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[30] Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017.
[31] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
[32] Ali Khodabakhsh, Raghavendra Ramachandra, Kiran Raja, Pankaj Wasnik, and Christoph Busch. Fake face detection methods: Can they be generalized? In 2018 International Conference of the Biometrics Special Interest Group (BIOSIG), 2018.
[33] Ching-Yar Chuang and Youssef Mroueh. Fair mixup: Fairness via interpolation. International Conference Learning Representations (ICLR), 2021.
[34] Ron Kohavi. Scaling up the accuracy of nayee- classifiers: A decision-tree hybrid. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (KDD), 1996.
12

---

[35] Gökhan H Bakır, Jason Weston, and Bernhard Schölkopf. Learning to find pre-images. Advances in neural information processing systems (NeurIPS), 2004.
[36] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. NeurlPS 2014 Deep Learning Workshop, 2015.
[37] Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in Neural Information Processing Systems (NeurIPS), 2018.
[38] Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkata- subramanian. Certifying and removing disparate impact. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2015.
[39] Moritz Hardt, Eric Price, Nati Srebro, et Equality of opportunity supervised learning. In Advances in neural information processing systems (NeurIPS), 2016.
[40] Steven B Cohen. Design strategies and innovations in the medical expenditure panel survey. Medical care, pages III1- 2003.
[41] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings International Conference on Computer Vision (ICCV), 2015.
[42] Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society (AIES), 2018.
[43] Yahav Bechavod and Katrina Ligett. Penalizing unfairness in binary elassification. Fairness, Accountability and Transparency in Machine Learning (FAT/ML), 2017.
[44] Mengnan Du, Ninghao Liu, and Xia Hu. Techniques for interpretable machine learning. Communications of the ACM, 2019.
13


Reviews:
[{"review_id": "uAMDJyULCPM", "reviewer": "Reviewer_FmVc", "summary": "The authors proposed a new technique to reduce the discrimination of DNN models by debiasing only the task-specific classification head of DNN models  (termed as Representation Neutralization for Fairness, RNF). Furthermore, to handle the cases where sensitive attribute annotations are not available, they leveraged a bias-amplified model to generate approximate annotations for sensitive attributes. They showed that the RNF effectively reduces the bias of DNN models with a minimal decrease in model performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work is innovative as previous work on this topic mostly focused on learning debiased encoders, whereas the current approach focuses on learning debiased decoders. Also, existing methods require annotations of the sensitive attributes but the current method removes this constraint as it can generate the proxy sensitive attribute via a biased-amplification approach.\n\n The submission is technically sound. The authors not only demonstrated that their approach mitigates bias (3.2) but also conducted thorough ablation studies to confirm the necessity of different model components (3.4). The analysis of the classification head further illuminates the effect of debiasing the classification head. The author found that RNF dramatically reduced the cosine similarity between positive predictions and sensitive attributes (e.g. male) compared to the Vanilla model by adjusting the decision boundary. I do have a question related to the results: since there's a tradeoff between accuracy and fairness, I wonder if the authors could compute a single metric combining accuracy and fairness (e.g., just like how f1-score takes account of both precision and recall) so that we could compare multiple models more directly.\n\nThe submission is clearly written and well organized. The results are important in the sense that it works complementary to the debiased encoder. When combined with debiased encoders, the model performs better than either of them alone in terms of the fairness metrics albeit with some loss in task performance. The authors may think more about how to improve fairness without the performance drop.\n\n-----------\n\nUpdated: I've read the authors' responses to my review as well as the ones to the other reviewers'. I think their responses addressed my questions. I appreciate the clarifications on the single metric and the trade-off between fairness and performance. I will keep my score unchanged as it reflects my best judgment of the work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There's no foreseeable negative societal impact of this work.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}, {"review_id": "e9NGALKBqmX", "reviewer": "Reviewer_Ho2y", "summary": "This paper proposes an unfairness mitigation technique for DNN models called Representation Neutralization for Fairness (RNF), which debiases the classification head without touching the possibly-biased representations. The idea is to first train a teacher network for the representations and then a representation neutralization network that trains on neutralized features (random combinations of samples with the same label, but different sensitive attribute values). Another contribution is to generate proxy annotations for sensitive attributes in case they are not available. Experiments show that RNF outperforms several baselines in terms of accuracy and fairness.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper proposes an intriguing approach for unfairness mitigation by only debiasing the classification head. The idea is simple, yet effective. However, some critical comparisons with other methods seem to be missing.\n\nStrong points:\n\n* Simple and effective approach for debiasing without touching the representations.\n\n* Adapting Mixup techniques for generating neutralized features is a neat idea.\n\n* The proposed method can be combined with any debiased model to further improve fairness.\n\n* Experimental results look promising on multiple datasets and prominent fairness measures, even when annotations for sensitive attributes are not available.\n\n* The paper is well written.\n\nWeak points:\n\n* During representation neutralization, mixing random samples with different sensitive attributes seems rather ad-hoc. Have the authors tried limiting the samples to mix, say by distance? If that does not work, why? \n\n* The key assumption for generating proxy annotations is questionable. A biased model may be over-confident for any subgroup of the data, and there is no guarantee this happens to be one of the sensitive groups. The authors should experimentally verify that the proxy annotations really align with sensitive attribute annotations. That is, do the samples with proxy annotations 0 (1) really have gender male (female)?\n\n* Some critical comparisons seems to be missing in the experiments. \n\n     - RNF seems very close to post-processing unfairness mitigation techniques as it can be viewed as neutralization (post-processing) after training the teacher network. Hence, there should be an experimental comparison with Hardt's method, which the authors already cite ([27]). \n\n     - The authors cite the relevant work Fair mixup [40], but do not make any comparisons with it either. \n\n     - Since the authors compare with EOR (direct optimization of equalized odds), there should also be a comparison with Zafar's method, which optimizes demographic parity. Zafar et al., \"Fairness Constraints: Mechanisms for Fair Classification\", AISTATS 2017.\n\n* Although the authors claim that the proposed method can be extended to non-binary sensitive attributes, some clarifications would be helpful.\n\n     - When generating proxy annotations, one demographic group has over-confident predictions, and the alternative group has less-confident ones. If there are more than two sensitive groups, how does RNF assign proxy annotations among groups?\n\n     - Smoothening neutralization: In Eq. (3), how can we assign lambda values to multiple sensitive groups?\n\n* Optionally, it would be nice to also use the ProPublica COMPAS dataset in the experiments.\n\n======\n\nThe author response addresses my main concerns above. Thanks for the clarifications and new experiments.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the work are sufficiently discussed in the supplementary, but should probably appear in the body of the paper. There is no discussion on potential negative societal impacts. One possible discussion is whether optimizing fairness may reduce accuracy too much, which can be viewed as a negative outcome.\n", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see any ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "IVEZxJEq0k", "reviewer": "Reviewer_Aa1C", "summary": "The authors propose a mitigation technique, namely, Representation Neutralization for Fairness (RNF) that achieves fairness by debiasing only the task-specific classification head of DNN models. They leverage samples with the same ground-truth label but different sensitive attributes and use their neutralized representations to train the classification head of the DNN model. Experimental results over several benchmark datasets demonstrate our RNF framework to effectively reduce discrimination of DNN models with minimal degradation in task-specific performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The paper has dealt with the problem of bias mitigation methods for DNN models which is a very common, prevailing and still challenging problem.\n2. The model is well described with proper model figure, dataset description, problem statement, adequate references.\n3. The algorithm is clear. It provides reasonably complete information that will help reproduce the results, including the methodology, pseudo-code, and empirical evaluation.\n4. The paper is accompanied by an adequate set of experiments for evaluating the effectiveness of the solutions the authors propose.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "-NBOKJ0UG1A", "reviewer": "Reviewer_Pdht", "summary": "This paper presents a method for fine-tuning the upper layers of a DNN (the \"classification head\") with the goal that it becomes insensitive to information in the encoded representation that correlates with a protected attribute. The technical contribution leverages the *mixup* approach to present \"neutralized\" encodings to the classification head, which learns to project them to softened target probabilities. Another key contribution is that this process can be done *without* access to ground truth labelling of the protected attribute. As a proxy, the authors assume that the unneutralized model will predicatably generate biased output in favour of a specified demographic group. They use that output to generate proxy annotations to take the place of ground truth labelling of the protected attribute.\n\nThe work is generally well-written and I have many minor comments, questions, and some suggestions for improvement and clarification throughout.\n\n~However, I have concerns about the evaluation. These concerns are the primary basis for my ultimate recommendation to not accept. If these concerns are addressed or if I have missed something, I am open to adjusting my recommendation.~\n\nThe author(s) have provided very helpful responses:\n\n* they \"will rephrase the claim that 'RNF… outperform baseline methods…' in the revised manuscript to instead highlight the resource benefits of RNF over the baseline methods with similar or better performance in some cases,\"\n* while not going as far as statistical significance tests, they have provided more information that allows me to better assess whether the observations are mere noise or true effects; they will average across more (10) runs and will include error bars in the figure\n* they have agreed to re-organize some of the presentation to make more clear some of the points I had questions about\n\nBased on these commitments, I have revised my recommendation to \"7: good paper, accept.\"", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality\n\nThe proposed method seems novel and parts of it surprise me! The work seems to adequately cite its motivations and cites what it builds off of.\n\nI was most surprised by the claim that the framework \"does not require any access to the protected attributes during training.\"\n\n# Quality & Clarity\n\n**I wonder throughout whether the phrase \"representation neutralization\" is a good name for this technique.** You admit that you are debiasing *only* the classification head and accept that the encoded representation itself *may be biased*. And you position your work in *contrast* to existing methods that learn debiased representations. I do see how you \"neutralize\" the representation as part of fine-tuning the classification head, but at inference time, the representation reaching the classification head is not neutralized in the same sense. Instead, the classification head has just hopefully become insensitive to the information correlated with the protected attribute. I don't know if I have a good suggestion for an alternative name. Maybe, something like desensitized classification. In any case, it's just a name, but it did put the wrong idea in my head for the early parts of the paper.\n\n## Introduction\n\nYou seem to be alluding to COMPAS at line 25. I think works frequently cite this ProPublica article when referring to COMPAS: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing\n\nI also would have expected citations to Calo & Citron, \"The Automated Administrative State\" (https://scholarlycommons.law.emory.edu/elj/vol70/iss4/1/), and Kate Crawford, *Atlas of AI*.\n\nSuggest citing to Edwards & Storkey, \"Censoring Representations with an Adversary\" (ICLR, 2016).\n\nAt line 23, you refer to *the* recidivism algorithm. This implies that there is only one. Suggest something like: \"COMPAS, an algorithmic recidivism predictor, is likely to associate...\"\n\nAt line 34: \"The suppression of fairness sensitive information might also remove useful information that is task relevant.\" Much work in this area acknowledges that this is the trade-off that anti-discrimination regimes require: even \"rational discrimination\" is not okay. See Prince & Schwarcz, \"Proxy Discrimination in the Age of Artificial Intelligence and Big Data\" (https://ilr.law.uiowa.edu/print/volume-105-issue-3/proxy-discrimination-in-the-age-of-artificial-intelligence-and-big-data).\n\nI wonder whether it is helpful to refer to the correlation with the sensitive attribute as \"spurious.\" These correlations are often the result of embedded social structures. The correlation is not spurious in the sense that it arose from randomness in a particular sample. Rather, the correlation has arisen *for real* in the population because of our decisions as society.\n\n## Representation Neutralization for Fairness\n\nThroughout this paper, until I got to 3.1.2, I was very curious and confused about how you determined what the \"classification head\" was. This confusion arises because of the way you talk about it in the early sections. You refer to \"the task-specific classification head of a DNN\" as if it is a clearly identified add-on. In actuality, it is just the top layer(s) of the DNN that you have chosen to set aside for fine-tuning. If there is any way you can clarify this early on, that would be helpful.\n\nAt 76, you say you consider \"binary sensitive attributes\" but go on to choose gender for the examples. Consider adding a caveat (even just a footnote) that gender is only binary as coded in the data, despite it not being a binary in reality.\n\n## Analysis of the Classification Head\n\nAgain, it would be helpful to note that there is in general no such thing as a distinct \"classification head\" other than how you have decided to fragement your network(s) for analysis / fine-tuning purposes.\n\nAt line 101, you distinguish between \"useful task relevant classification information\" and \"bias information from protected attributes.\" But this isn't a true disctinction and that is part of the problem. The issue is that the protected attributes have *become correlated* with the relevant classification target and thus *become* useful task relevant classification information. The goal of anti-discrimination frameworks is to require decision-makers to ignore that correlation *even if* it would improve some performance metric.\n\n## Representation Neutralization for Debiasing Classification Head\n\nAgain, you make the distinction between \"sensitive attributes\" and \"task-relevant information.\" The problem is that sensitive attributes have *become* task-relevant information. Perhaps you mean to distinguish between \"sensitive attributes\" and \"permitted factors\"?\n\nHere, it becomes clear that you freeze the encoder *after* it has learned what you are calling a spurious correlation, which you hope that your fine-tuned classification head can ultimately ignore.\n\nA question about lines 139-141. You say, \"From the input perspective, the neutralization of representations suppresses the model from capturing the spurious correlation between fairness sensitive information in the representation with the class labels.\" Were you being careful here in saying \"suppresses\" rather than \"prevents\"? If so, that matches how I understand this would work. I don't see how your smoothing/averaging would necessarily eliminate the spurious correlation from making it through, but could suppress it to an extent. Am I misunderstanding this? Regardless, this wasn't critical to my review, but just a place where you could perhaps be clearer if I did misunderstand you.\n\nI think it would be helpful to emphasize early on that the neutralization only happens during the fine-tuning of the hopefully debiased classification head. At inference time, the features just come in as encoded, to a classification head that has hopefully learned to not exploit any of the information correlated with the sensitive attribute.\n\nThe **smoothing neutralization** seems to be introducing a penalty for large changes in soft-max probabilities as you move along the interpolation between samples. Is this correct?\n\n## Generating Proxy Annotations for Sensitive Attributes\n\nYou say, \"A biased model generates over-confident predictions for one demographic group, while giving much lower scores for the alternative group.\" Can you justify this claim? Or is this true by definition? What happens if you apply your technique to an unbiased model?\n\nI think the phrase \"over-confident\" is adding some confusion here. You don't mean merely that predictions are close to 1 or 0, but rather, that there is a demographic group that tends to get the less-desired outcome and another demographic group that tends to get the more desired outcome.\n\nTher paragraph from lines 186--202 is very dense and some aspects come out of nowhere. The \"splitting threshold\" for example. How is it used? How do you use the fairness-accuracy tradeoff to determine its value? It would be helpful if you took more space to explain this all.\n\n## Experiments\n\n## Experimental Setup\n\n## Fairness Metrics, Benchmark Datasets, and Baselines\n\nGenerally very clear.\n\nAt line 217, you say \"bias towards the female group.\" This is ambiguous, as it could mean bias in favour of female groups or bias disfavouring the female group. An alternative: \"vanilla trained models depict bias that predicts females earn less.\"\n\nAt line 220, you say \"discrimination towards.\" This is less ambiguous, so you could use this at 217 too.\n\n## Implementation details\n\nVery clear, other than the purpose of the thresholds at line 243 (as I mentioned above was also unclear at line 196).\n\nYour description here of how you split the networks into representation layer and classification head is something that would have been helpful much earier. Maybe not this full amount of detail, but just that it's a somewhat arbitrary decision and nothing inherent about the division.\n\n## Mitigation performance analysis\n\n**This is the section that gives me the most concerns about this paper.**\n\nYou claim that \"RNF... outperforms baseline methods...\" However, these figures (Figure 3) do not convince me of this. The datapoints are averages across three runs. But, just looking at how non-smooth the fairness-accuracy curves are, and how close these curves are to each other, I am not sure averaging across three runs was sufficient to be able to be confident we are not observing noise from a small set of experimental runs. In Figure 3(a), under equality-of-odds analysis, EOR might beat RNF, and RNF-Gt looks similar to EOR. In Figure 3(b), under demographic-parity analysis, RNF and RNF-GT look similar to EOR. In Figure 3(b), under equality-of-odds analysis, EOR might be beating RNF, EOR looks similar to RNF-GT, and RNF looks similar to adversarial. Figure 3(c), under demographic-parity analysis, this figure does look like it is consistent with your conclusions. Figure 3(c), under equality-of-odds analysis, EOR may be better than RNF, EOR might be similar to RNF-GT. Regardless, it is hard for me to draw conclusions from this small set of 3-run experiments.\n\n## Classification Head Analysis\n\nThis is a creative way to demonstrate correlation between fairness-sensitive information and information used by the classification head.\n\n## Hyperparameter and ablation analysis\n\nI was really hoping you would assess the effect of where you draw the line between encoder and classification head. Great to see this analysis here.\n\n## Representation Neutralization with Debiased Encoder\n\nThis reports only a single point on the fairness-accuracy curve. Were these also averages across three runs? A single point on the curve isn't enough to be able to draw a conclusion.\n\n# Revised Recommendation\n\nThe author(s) have provided very helpful responses:\n\n* they \"will rephrase the claim that 'RNF… outperform baseline methods…' in the revised manuscript to instead highlight the resource benefits of RNF over the baseline methods with similar or better performance in some cases,\"\n* while not going as far as statistical significance tests, they have provided more information that allows me to better assess whether the observations are mere noise or true effects; they will average across more (10) runs and will include error bars in the figure\n* they have agreed to re-organize some of the presentation to make more clear some of the points I had questions about\n\nBased on these commitments, I have revised my recommendation to \"7: good paper, accept.\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors could point out that some anti-discrimination regimes require the *elimination* not the mere *reduction* of disparate outcomes in government decision-making.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for debiasing classifiers. Firstly, a biased classifier is trained as normal. Then, a new classification head is trained for that encoder, that interpolates using combined representations from examples with the same label but different sensitive attributes (similarly to mix up training). This approach allows a classifier that is less biased, whilst retaining most of the performance of the original. This is clearly an important problem, and the reviewers generally found the proposed approach to be creative, soundly evaluated and well written. Therefore, I recommend acceptance.
Decision: Accept (Poster)