Fairness via Representation Neutralization
Authors: Mengnan Du
Subhabrata Mukherjee
Guanchu Wang
Ruixiang Tang
Ahmed Hassan Awadallah
Xia Hu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Algorithmic bias
Fairness mitigation
Explainability

Abstract: Existing bias mitigation methods for DNN models primarily work on learning debiased encoders. This process not only requires a lot of instance-level annotations for sensitive attributes, it also does not guarantee that all fairness sensitive information has been removed from the encoder. To address these limitations, we explore the following research question: Can we reduce the discrimination of DNN models by only debiasing the classification head, even with biased representations as inputs? To this end, we propose a new mitigation technique, namely, Representation Neutralization for Fairness (RNF) that achieves fairness by debiasing only the task-specific classification head of DNN models. To this end, we leverage samples with the same ground-truth label but different sensitive attributes, and use their neutralized representations to train the classification head of the DNN model. The key idea of RNF is to discourage the classification head from capturing spurious correlation between fairness sensitive information in encoder representations with specific class labels. To address low-resource settings with no access to sensitive attribute annotations, we leverage a bias-amplified model to generate proxy annotations for sensitive attributes. Experimental results over several benchmark datasets demonstrate our RNF framework to effectively reduce discrimination of DNN models with minimal degradation in task-specific performance.

Fairness via Representation Neutralization
Mengnan Subhabrata Mukherjee', Guanchu Wang', Ruixiang Tang', Ahmed Hassan Awadallah', Xia Hu
Teeas A&M University ?Microsoft Research RRiee University
dumengnan@tamu. edu, {submukhe, aataaaaammmmiirooofotoooooo (guanchu.o wang, rt39, xia. werieiiee........ edu
Abstract
Existing bias mitigation methods for DNN models primarily work on learning debi- ased encoders. This process not only requires a lot of instance-level annotations for sensitive attributes. it also does not guarantee that all fairness sensitive information has been removed from the encoder. To address these limitations, we explore the following research question: Can we reduce the discrimination of DNN models by only debiasing the classification head, even with biased representations in- puts? To this end, we propose new mitigation technique, namely, Representation Neutralization for Fairness (RNF) that achieves fairness by debiasing only the task-specific classification head of DNN models. To this end, we leverage samples with the same ground-truth label but different sensitive attributes, and use their neutralized representations to train the classification head the DNN model. The key idea of RNF is to discourage the classification head from capturing undesirable correlation between fairness sensitive information in encoder representations with specific class labels. To address low-resource settings with access sensitive attribute annotations, we leverage a bias-amplified model to generate proxy an- notations for sensitive attributes. Experimental results over several benchmark datasets demonstrate our RNF framework to effectively reduce discrimination DNN models with minimal degradation in task-specific performance.
Introduction
and have been deployed in many real-world applications. However, DNNs often suffer from biases and show discrimination towards certain demographics, especially high-stake applications, such as criminal justice, employment, loan approval, credit scoring, etc 4 S 6). For example, COMPAS, an algorithmic recidivism predictor, is likely to associate African-Ami American offenders with higher risk scores compared to Caucasians while having a similar profile [7]. This brings significant harm to both society and individuals, thus leading to recent focus on mitigation techniques alleviate the adverse effects of DNN biases.
Deep neural networks (DNNs) have made significant advances in recent times 2 3
Existing debiasing methods usually work on learning debiased representations at the encoder- level. One representative family of methods perform mitigation by explicitly learning debiased representations, either through adversarial learning [ 9 10) invariant risk minimization [1] [2]
Another family of methods 13 14] [5] implicitly learn debiased representations by incorporating explanation during model training to suppress it from paying high attention to biased features in the original input. Essentially, the above methods aim to remove the bias from deep representations.
Learning debiased representations is technically challenging problem. Firstly, it is hard to remove all fairness sensitive information in the encoder. The suppression of fairness sensitive information
Part the work was done while the first author was an intern at Microsoft Research.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

might also remove useful information that is task relevant. Secondly, most existing debiasing methods assume access additional meta-data such as fairness sensitive attributes and a lot of annotations corresponding to the protected groups to guide the learning of debiased representations. However, such resources are expensive to obtain, if not unavailable, for most real world applications.
To address these limitations, we explore the following research question: Can we reduce the discrim- ination of DNN models by only debiasing the task- mask-specific classification head, even with a biased representation encoder? Our work is motivated by the empirical observation that standard training can result in the classification head capturing undesirable correlation between fairness sensitive information and specific class labels. Some recent works [16 17 [8] have explored such spurious or shortcut learning behavior of DNNs in various applications. To this end, we propose the RNF (Representation Neutralization for Fairness) framework for mitigation, motivated by the Mixup work 19 20) We first train a biased teacher network via standard cross entropy loss. the second stage, we freeze the representation encoder of the biased teacher, and only update the classification head via representation neutralization. This discourages the model from associating biased features with specific class labels, and enforces the model to focus more on task relevant information. To address low-resource settings, our RNF framework does not require any access to the protected attributes during training. To this end, we train a bias-amplified model using generalized cross entropy loss that is used to generate proxy annotations for sensitive attributes. Experimental results over several benchmark tabular and image datasets demonstrate our RNF framework to significantly reduce discrimination of DNN models with minimal degradation of the task performance. The major contributions of our work can be summarized as follows:
We analyze bias propagation from the encoder representations to the final task-specific layer demonstrating that DNN models heavily rely on undesirable correlations for prediction. We introduce RNF, a bias mitigation framework for DNN models via representation neutralization. Our RNF framework achieves mitigation without any access to instance-level sensitive attribute annotations, and instead relies on self self-generated proxy annotations.
Experimental results on several benchmark datasets demonstrate the effectiveness of our RNF framework via debiasing only the classification head while using biased representations as input. Additionally, we show RNF to be complementary to existing methods that learn debiased encoders and can be further improved within our framework.
2 Related Work
We briefly review bias mitigation and broader robustness literature which are most relevant to ours.
Bias Mitigation. Recent studies have indicated that DNN models exhibit social bias towards certain demographic groups. This has led to increased attention to bias mitigation techniques in recent times (21 22 23). Existing mitigation methods can be generally grouped into three broad categories. The first one is based on adversarial training (8 9 [0]. It leads to fair classifier as the predictions cannot carry any group discrimination information that the adversary can exploit. However, this method assumes that the sensitive attribute annotations are known, and uses those annotations to learn debiased representations. The second representative family of mitigation methods is based on explainability [13] [4] [5] 24 These methods require fine-grained feature-leved annotations to specify which subset of features are fairness sensitive. The third category falls under the umbrella of causal fairness 25 [2). The main idea is to enforce the model to concentrate more on task relevant causal features, and getting rid of superficial correlations (66.. This results in the model capturing debiased representations. For instance,
27 minimizes the correlation between sentence representations and bias words using a contrastive learning framework. We can typically decouple the classification problem into representation learning and classification as the two major parts (28). Different from the above-mentionel methods that mainly aim to train the debiased representations, our work focuses on the classification head and aims to suppress it from capturing undesirable correlation between fairesss sensitive attributes and class labels.
Shortcut Learning in DNNs. Our work is motivated by the observation that DNNs with standard training are prone to exploit undesirable correlations (or shorteuts in the dataset) for prediction [16] 29 Beyond algorithmic discrimination, recent studies show that shoricut learning [17 can result in other undesirable consequences like poor generalization and adversarial vulnerability. Specifically, this leads to high performance degradation for previously unseen inputs, especially for those data beyond
2

---

Before Neutralication Famale and male
Neutralization Neutralized feature
(dt Neutrallzation Pasitive and Nogative
Positive and Nogative
Figure I: Representation analysis of = using Kernel PCA. (a) Protected attribute a (i.e., gender) is discriminative feature. In this plot, the male group primarily lies in the lower left interval, whereas the female group is located primarily on the upper right interval. (b) Predicted positive label and negative task-label distribution. (c) Representation neutralization (please refer Sec. 3.3) reduce the discriminative power of for dimensions relevant to the sensitive information, comparing to the distribution in (a). (d) Neutralization still preserves the useful task relevant information.
held-out test set. Representative tasks depicting such behavior include reading comprehension [3], natural language inference [8]. visual question answering and deepfake detection [32].
The most similar work to ours also regularizes the model training using interpolated samples between groups [33 Similar to the aforementioned solutions, this work also heavily relies on sensitive attribute annotations in the training set. This limits the application scenarios of the mitigation algorithms, especially for many real-world applications without readily available annotations dealing with sensitive attributes. Our representation neutralization framework achieves comparable or better performance to such techniques without relying on such sensitive annotations by leveraging proxy annotations obtained from bias-intensified version of our framework, thereby, making it broadly applicable to arbitrary real-world applications.
3 Representation Neutralization for Fairness
In this section, we first analyze the task-specific classification head of DNN to examine how bias propagated from the encoder representation layer to the task-specific output layer (Section (3.2).
We empirically demonstrate the undesirable correlation between fairness sensitive information in representations with specific class labels. Based on the observation, we introduce the Representation Neutralization for Fairness (RNF) framework to debias DNN models (Section 3.3) Finally, we pro- pose an approach to generate proxy annotations for sensitive attributes, enabling the RNF framework to be applicable to low-resouree settings with access to sensitive attribute annotations (Section 3.4).
3.1 Notations
We first introduce the notations used in this work. Let X' {x,, Vi, (,,a; .....,N N be the training set, where z; is the input feature, y/s denotes the ground truth label, and a, represents the sensitive attribute (e.g., gender, race, age). For case of notation, in the following sections, binary sensitive attributes Nevertheless, our proposed mitigation framework can also be applied to non-binary sensitive attributes (e.g., age).
consider
Consider the classification model (a. 0) (z)( parameterized with the model parameters. Here (xx)= X - Z represents the feature encoder, and g(x) = 2 is the representation for x obtained from a DNN model. The predictor c(=) - y is the multi-layer classification head, It is depicted input and maps it to
by the top layer(s) of the DNN, which takes the encoded representation softmax probability. The final class prediction is denoted by ÿj = arg max c(z). In this work, we aim to reduce the discrimination of DNN models by only debiasing the classification head c(2), with the biased representation encoder (αx) as input.
*Gender not binary in reality as there are many different gender identities, such as male, female, transgender, to name few. In this work, we consider gender as binary sensitive attribute due to the limitation of the benchmark dataset that encodes gender binary variable.

---

3.2 Analysis of the Classification Head
In this section, we examine how bias manifests in the representation space Z as well how the classification head c(z) obtained with standard training scheme propagates bias from the representa- tion layer to the model output layer. To this end, we train a biased network (x) via standard cross entropy loss, where the following experiment is performed using the Adult dataset [4].
Representation Probing Analysis. For the Adult training set, we generate representation vectors for 500 training samples using the biased network fz( and project them in 2D for visualization. To this end, we utilize the Kernel Principal Component Analysis (KPCA) 35 with a sigmoid kernel, which is a tool to visualize high-dimensional data. Since the classification head c(2) contains multiple non-linear layers, we choose kemnel PCA instead of a linear dimensionality reduction method such as linear PCA. The visualization is shown in Figure [ (a). The plot depicts that the low dimensional projection separates the two protected groups in two areas, where the male group is primarily located in the lower left area, whereas the female group primarily occupies the upper right area. Comparing the task-label y distribution in Figure I (b) with the protected group distribution in Figure (a), we observe that the protected attribute information is a discriminative feature that could be exploited by the task classification head for prediction.
Role of the Biased Classification Head. The above demonstrative analysis indicates that the model representation captures both useful task relevant classification information as well as bias information from protected attributes. Specifically, the model captures strong correlation between the fairness sensitive information and the class labels. On analyzing the data distribution, we observe this to be an artifact of the conditional label distribution with respect to the sensitive attributes being skewed. The model relies on this shorteut for prediction, resulting in bias amplification. We observe the male neurons to positively correlate to the desirable label (also refer to the experimental analysis in Sec. 4.3). whereas the female neurons positively correlate to undesirable label. This depicts an undesirable correlation between sensitive information with certain class labels in the model.
Our Motivation. Based on the above empirical observations, we propose to neutralize the training samples (Fig (c) so as to reduce the discriminative power of the fairness sensitive information, while at the same time preserving task relevant information (Fig (d)) With the neutralized training data, we propose to re-train the classification head. Our goal is to adjust the decision boundary to implicitly de-correlate the fairness sensitive information in the representation space with class labels.
3.3 Representation Neutralization for Debiasing Classification Head
Based on the aforementioned empirical observations, in this section we propose simple yet effective bias mitigation framework via Representation Neutralization for Fairness (RNF). RNF does not require any prior knowledge about existing bias in the representation space; nor does it require any knowledge about specific dimension(s) encoding the sensitive attributes making it widely useful for arbitrary applications. Our goal is to encourage the model to ignore the sensitive attributes and instead focus more on task relevant information.
RNF is implemented in two steps (see Figure 2. In the first step, we train the model using cross entropy loss, and obtain a biased teacher network fr (z). During the second step, we freeze the encoder g(x) for (I), and use it as our backbone encoder for learning representations, We then re-train only the classification head c(2) using feature neutralization (see Figure 2 (b)).
Representation Neutralization. To this end, while training the classification head, for an input sample (.1, y, 01) }. we randomly select another sample (I2, y, 02), with the same class label y but different sensitive attribute d2 compared to 01 in the input sample. Now we compute the corresponding representations 2 g(x1) and z2 = g(22)) and re-train the classification head using the neutralized representation 21 as input. For the supervision label for the classification head, we utilize
the neutralized soft probability y t2 after temperature scaling obtained as follows. Given the logit vector 21 for input x1, the probability for class is computed as P1. Σ, /TT) where
exp(s1
T21 1. This can regarded as a form of knowledge distillation (36), where > softens the softmax score. A larger temperature prevents the model from assigning over-confident prediction probabilities. special case for RNF when 1, where p1 and Pa are the standard softmax probabilities obtained from the biased teacher network fx)

---

Exiitinng methods on the debiased encoder
Z1
Encoder
Classification
P1
21
Z2
Classification head
+ Pz
1
head
2
2
Our RNF focuses
Z/
Neutralized feature
Neutralized label
dehiesed classification hredd
(a) Biased teacher network
(b) Representation neutralization
Figure 2: Debiasing with representation neutralization. (a) We first train biased teacher network using only cross entropy loss. For two inputs I1 and 22 that with the same class label y and different sensitive attribute
obtain the representations 21 and 22, and softened probabilities P1 and p2- (b) We freeze parameters of the biased encoder and only re-train the classification head using the neutralized representation +1 as input, and softened probability 2
t2a as supervision signal.
We use the knowledge distillation loss. particular, the mean squared error (MSE) loss is used as distance-bar based metric to measure the similarity between model prediction and the supervision signal.
(Ú:- y)² fel2 2 22) - (2Px + pa))²
LSE
where, is the classification head project representations to softmax prediction probability.
There are two main benefits of the aforementioned training scheme. From the input perspective, the neutralization of representations suppresses the model from capturing the undesirable correlation between fairness sensitive information in the representation with the class labels. From the output perspective, the softened label encourages the model to assign similar predictions to different sensitive groups. Optimizing Eq. (1) can lead to reduced generalization gap between the two groups.
satisfying 1|21 22l12 S X₂ Pi - Pal for 21 E X and a bounded loss function L(c(z)).p.) = (1-p:)l(c((:))) 0) p,l(c(z)),) y 1), where [(((z)) = j) S er. for x4 and 0,1; if the clas- sification head minimizes the loss between neutralized representation and soft probabilities,
Theorem Given well-trained representation encoder g(x)
i.e. V=L(c(2). 2= 21 pite
S tc. where
g(z), In P(x1 a, 0).
2
D2 P(12 a: =1 1). P1 P2l
S
Ep. the gap of generalization loss between groups
and a = 1 is bounded by:
E Eer~P(p: L(c(2)). Plyy
I Cp(Acce +())
Here "well-trained" means that the model has learned reasonably good representations to encode sufficient task relevant information. Essentially, after training for sufficient number of epochs until the validation loss has converged, we have access to a reasonably good representation space (although it might encode lot of fairness sensitive information). If train the classification head, the generalization gap between the different protected groups would be small. For more detailed proof, please refer to Section A in the Appendix.
the RNF loss function to further
Smoothing Neutralization. To further enforce the model ignore sensitive attributes, we construct syper-parameter À to control the degree of neutralization
augmented training samples using of the samples (æ1, Pl, Mn,yl y) and (=2.P2.
P2. )- The augmented neutralized sample is given by = λzι + (1 - 22, À E [g,1] 1). We encourage the classification head to give similar prediction scores for the augmented and the neutralized sample (with À )). The regularization loss is given by:
Smmooth
Σ xEl
lc(Azz + (1- >) al c + -=222) 22)11
By varying À we control the degree of sensitive information for the augmented samples. is utilized to penalize the large changes softmax probability when we move along the interpolation between two samples. We linearly combine the MSE loss in Eq. ( with the regularization term follows:
C = LMSE + aL
(4)
5

---

We train the classification head using the loss function in Eq. (4.. Eventually we combine the original encoder of fr(x) and re-trained classification head as the debiased student network fz(2). The teacher (a) is later discarded and the debiased student network fs( is used for prediction.
3.4 Generating Proxy Annotations for Sensitive Attributes
The aforementioned feature neutralization is limited in that it requires instance-level sensitive attribute annotations {ar)/=1 for all training samples. Such resource- eettencveett annotations are difficult to obtain for many practical applications particularly due to the nature of the sensitive attributes. To address this limitation, we propose method to generate proxy annotations {ā: J for the sensitive attributes based on the model uncertainty. The key idea is that a biased model generates over-confident predictions for one demographic group, while giving much lower scores for the alternative group. Particularly, the bias- bias-amplified model tends to assign the privileged group more desired outcome, while assigning the under-privileged group less-desired outcome. For instance in the Adult dataset, the average prediction probability of the desired label (higher income) for the male group is much higher than that of the female group. In contrast, the average probability of the less-desired label for the female group is much higher than that of the male group.
To better facilitate the model to generate uncertainty scores, we train another biased model by intentionally amplifying the bias via generalized cross entropy loss (GCE) 337. The bias-amplified
model is denoted as (T) and the loss function is given as follows:
GCE(J(Z;0),)) = 1 - (=;
(5)
q
where [a; 0) denotes the output probability for ground truth label y. The hyper-parameter q E (0, controls the degree of bias amplification. When limy--, the GCE loss in Equation (2) approaches -logp which is equivalent to standard cross entropy loss. The core idea is that for more biased samples, i... samples with larger fy (z; 8) value, the model assigns higher weights fß while updating the gradient.
a CE(p.) a GCE(P. v) = 1- 00 In this setup, the model (x) learns more from bias- stullampliiied samples compared to the model fr()) trained with standard cross entropy loss.
(6)
00
The confidence score of (I) is used to indicate whether a sample belongs to a privileged or unprivileged group. Specifically, for a desired ground truth label, samples with over-confident scores are grouped into the privileged group, whereas subsets of samples with low prediction scores are grouped into the unprivileged group. In contrast, for the undesired ground truth label, samples with over-confident scores are grouped into the unprivileged group, and vice versa. Based on this criterion, we generate proxy sensitive attribute annotation à for each training sample x to split samples into two groups, which are subsequently used for feature neutralization.
The overall process of our RNF mitigation framework is given in Algorithm 1, which con- tains two stages. In the first stage, we train
Algorithm 1: RNF mitigation framework.
Input: Training data D = {(x,, w) Set hyperparameter q. a.
the biased teacher network (z) and the bias-1
amplified network [x)... In the second stage,
while first stage do Train teacher network (x and
we first use (x) to generate proxy sensitive at-a
tribute annotations for all training samples. We use the ratio to partition the training set to gen-4 erate proxy annotations for protected attributes that are subsequently used for feature neutraliza- tion. Note that the ratio 7 is determined by the faimess-accuraac trade-off on the validation set.? Then we use representation neutralization and the loss function in Eq. 4 to re-train the classi- fication head of fr fx)).. Eventually, we combine the original encoder s(x) of fr(a) and the re-
bias-amplified network (z(
while second stage do
Determine the splitting threshold y for f=x(w); Calculate proxy annotation à; for each training sample { based Use loss function Eq. (4) to re-train
and fa
classification head c(z) of fr(().
Output: The debiased student network (z)..
fined classification head c(z) to give us the debiased student network fx(x). is worth noting that the neutralization is merely performed during the training stage for only debiasing the classification

---

head. At inference time, the features just come in as encoded, and the classification head has learned to not exploit any of the information correlated with the sensitive attribute for prediction.
4 Experiments
In this section, we conduct experiments to evaluate the effectiveness of our RNF framework.
4.1 Experimental Setup
4.1.1 Fairness Metries, Benchmark Datasets and Baselines
We use two group fairness metrics: Demographic Parity (38 and Equalized Odds [39] Demographic Parity (DP) measures the ratio of the probability of favorable outcomes between unprivileged and privileged groups: DP
p(á=1ja=0) Equalized Odds (AEO) require favorable outcomes to be
pi p(g-Ila=T).
independent of the protected class attribute a, conditioned on the ground truth label y. Specifically, calculates the summation of the True Positive Rate difference and False Positive Rate difference:
AEO = (P( = 1 | = 0,y 1)- Pl =1 a 1, = 1)) + (P() = 1 | 0,y = P(ý=1 a 1=1,y 1.y = 0}}- (7)
Under the above metrics, it is desirable to have DP value closer to 1 and AEO value closer to 0.
We use two benchmark tabular datasets and one image dataset to evaluate the effectiveness of RNF. For the Adult income dataset (Adult), the goal is to predict whether a person's 's income exceeds S50K/yr [34]. We consider gender the protected attribute where vanilla trained models show discrimination towards the female group by predicting females to earn less. For the Medical Expenditure dataset (MEPS), we consider two groups white and non-white 44) Here the task is to predict whether person would have 'high' utilization, where vanilla DNN shows discrimination towards the non- non-white group. The CelebFaces Attributes (CelebA) dataset is used to predict whether the hair in an image is wavy or not 41 We consider two groups male and female, where vanilla trained models show discrimination towards the male group. We split all datasets into three subsets with statistics reported in Table More details of the datasets included in Sec.
Table I: Dataset statistics.
Adult
MEPS
CelebA
# Training
33120
11362
194599
&Validation
3000
1200
4000
# Test 9102
3168
8000
Sec.Ciin Ci the Appendix.
We compare two variants of our framework RNF (using proxy sensitive attribute annotations) and RNF (using ground truth sensitive attribute annotations) against baselines such as DNNs trained using only cross entropy loss (referred as Vanilla) and two regularization based mitigation methods, namely, adversarial training (Adversarial) Among them, the Adversarial method achieves fairness via learning debiased representations, whereas EOR directly optimizes the EO metric in Eq. 7 All three baselines control the faimess--accuracy trade-off via hyper ppper-paramtter... More details on the baselines are included in Sec. D in the Appendix.
42 and Equalized Odds Regularization (EOR) (43).
4.1.2 Implementation Details
For the image classification task, we use ResNet-18 [ (we add one more fully connected layer). We set the representation encoder g(x) as the convolutional layers and use the remaining two fully connected layers as the classification head c(z). For tabular datasets, we use a three-layer MI.P (multilayer perceptron) as the classification model, where the first layer is set as the encoder and the remaining two layers are used as the classification head. Dropout is used for the first two layers with the dropout probability fixed 0.2. We use the same batch size of 64 for tabular datasets and 390 for the image dataset. For selecting another random sample {I2, y, 02 } to be neutralized with current sample {a1, y. a.), we perform the selection within the current batch of training data. The hyper- parameters (e.g., learning rate and training epoches) are determined based on the model performance on the validation set, and early-stopping based on validation performance used to avoid overfitting. The optimal temperature T used to calculate the probability is set as 2.0, 5.0, 2.0 for Adult, MEPS and CelebA datasets respectively. For Eq. (3), we sample À from the list [0.6, 0.7, 0,,,0.8.... 0.9). The papee--arammttrr g in Eq. (5) is set as 0.2, 0.3 for Adult, MEPS and CelebA datasets respectively.
0.3
7

---

0.060
0.70
0.855
Accuracy
Accuracy
0.850
Accuracy
0.845
0.72
0.840
RNI
RNI
0.70
RNI
Adversarial
Adversarial
EOR
EOR
EOR
Vanilia
0.030
Vanilia
Vanila
0.75
0.80
0.95
0.820 .080
0.85
0.95
0.66
0.675 0.700 0.725 0.750 0.775 0.800
Demographic Parity
Demographic Parity
Demographic Parity
0.860
0.78
0.76
Accuracy
0.850
Accuracy
0.74
0.72
RNF-GT
RNF-GT
RNF-GT
RNF
0.040
RNF
0.70
RNF
EOR
EDR
0.60
EOR
Vanilla
0.830
Vanilla
Vanila
-0.00
-0.15
-0.30
-0.05
0.00
-0.25
-0.20
-0.15
-0.10
0.50
-0.45
-0.40
-0.35
-0.30
Equality of Odos
Equality of Odds
Equality Odds
(a) Adult
(b) MEPS
(c) CelebA
Figure 3: The fairness-an canness-accurayy curve comparison of RNF and other baselines. The first and second row depict the DP accuracy and AEO accuracy trade-off curves, respectively. Note that there is certain level of variance for each baseline method, and we report the average over 10 runs.
4.2 Mitigation Performance Analysis
We compare the mitigation performance of RNF (with proxy attribute annotations) and RNFGr (with ground-truth attribute annotations) with other competing methods and illustrate their fairness-accuracy curves for the three datasets in Figure 3 The hyper-paramette a in Eq. (4) controls the trade-off between accuracy and fairness for RNF. For Adversarial and EOR, we vary their regularization weights to obtain the corresponding performance curves. As the random seeds lead variance in the accuracy and fairness metrics (please refer to Section E.4 in the Appendix for detailed analysis of the variability effect), we train the model for 10 times with different seeds and report the average result. Overall, we make the following key observations.
Even though RNF does not rely on annotations for the sensitive attributes, it performs similar to baseline methods with access to such information, e-g, Adversarial training, or better than them in some cases. This makes RNF readily usable for real-world applications where protected attributes are not available the training set.
RNFGr improves mitigation performance over RNF by 10% on an average across all datasets and metries, thereby, demonstrating the benefit of using ground truth sensitive attribute annotations.
The soft labels of RNF and RNFGT (obtained using higher temperature discourage the model to assign overconfident predictions, thereby, suppressing it from capturing undesirable correlation between fairness sensitive information and class labels. Penalizing the large changes probability we move along the interpolation between two samples further suppresses the model from capturing the undesirable correlation.
Direct optimization of the equality of odds metric (i.e., EOR) achieves comparable performance RNF for all the datasets. However, has limited improvement in terms of the demographic parity metric. Note that EOR requires instance-level annotations for the protected attributes.
We observe that Adversarial training also performs effective mitigation by learning debiased representations. However, this happens at the expense of a higher accuracy drop for the task performance. This likely results from the loss of task relevant information while suppressing sensitive information from the representations. Additionally, we observe adversarial training to be unstable, especially for relatively complex task like image classification.

---

4.3 Classification Head Analysis
In this section, we use explainability and an auxiliary prediction task analyze the classification head c(z) for the Adult dataset. Particularly, we leverage explainability as debugging tool to analyze the attention difference between Vanilla and RNF model with respect to the representations.
Auxiliary Sensitive Attribute Prediction Task. We perform a representation analysis using an auxiliary prediction task. To this end, we train another linear classifier to predict sensitive attributes using the biased representation ((x) = 2 as input and the sensitive attribute annotations {a, } as the supervision signal. The linear classifier denoted by LSENS (=) = W + b, where W and b represent the weight matrix and bias for the linear classifier respectively. The weight matrix W can be used measure the degree of bias in each dimension of the representation 2.
Explanation Analysis. We use post-hoe explainability [44] to analyze the contribution of the classification head c(2). Our goal is to figure out the contribution of each dimension within the biased representation (xx) = 2 towards the model prediction f(z, 8) = (z)). We train a linear classifier Trexplan( 2) = W b to mimic the decision boundary of the multi-layer classification head c(z).
We compare the weight matrix of the two linear classifiers LSENS (2)
and Lexplan (z) using cosine similarity. For models trained using Adult dataset, we extract the weight matrix corresponding to the protected attribute Male and task label Positive, and then we calculate the cosine similarity between the Male vector and the Positive vector. This fol- lows from our observation in Figure I that the vanilla model makes use of male relevant information to make positive predictions. For the Vanilla model and RNF models listed in Figure B (a), calculate the cosine similarity and report the DP-Similarity performance Figure We observe that RNF dramatically reduces the cosine similarity be- tween positive predictions and male relevant information compared Vanilla (from 0.272 0.075), by adjusting the decision boundary.
0.25
Cosine SimBurity
0.20
RNF-CT
Varitb
0.05
0.75
0.80
0.85
0,50 0.95
Domographic Panty
Figure 4: Analysis for the classification head.
This eventually helps the head c(2) to shift its attention from fairness sensitive information to task relevant information.
4.4 Effectiveness of GCE Loss
We use MEPS dataset to examine the GCE loss in terms of generating proxy annotations for the protected attributes. The results are shown in Table Firstly, we compare the fairness performance of GCE loss with standard CE loss (Vanilla). As expected, we observe that GCE is more biased than Vanilla in terms of the fairness metrics with 0.2% accuracy difference. Secondly, we compare RNF with several of its variants: 1) replacing GCE loss with cross entropy (CE) loss generate proxy labels, and 2) using random annotations for the pro- tected attributes. For fair comparison, except for the protected attribute annotations, we use the same set of hyper-parameten for different variants. Even though the same level of regularization is performed for the three RNF variants, RNF-GCE has higher mitigation performance than RNF-CE with 3.0% DP improvement and only 0.3% accuracy difference, thereby, demonstrating the effective- ness of GCE terms of separating protected groups. Another observation is that even using random annotations, our RNF framework could achieve certain level of mitigation. This is because partial samples contain different sensitive attributes, which still could serve neutralization purpose.
Table Ablation analysis.
MEPS
Models
Accuracy
DP
AEO
Vanilla
0.862
0,866
-0.210
GCE
0.860
0.839
-0.249
RNF-GCE RNF-CE
0.839
0.964
-0.099
0.842
0.936
-0.128
RNF-Random
0.856
0.902
-0.163
4.5 Varying Layers for the Classification Head
We use MEPS and CelebA two datasets to study the effect of an important our framework in terms of which layers to use for the encoder. Our classification model f(x contains the encoder g(x) and task predictor e(z). In this experiment, we vary the depth of the representation layer to examine the effect of
Table 3: Varying the layers for classification head.
of
MEPS
Celeb
Models
Accuracy
DP
ABO
Accuracy
DP
ABO
RNF Last
0.839
0.964
-0.099
0.668
0.836
-0.290
0.837
0.971
-0.076
0.641
0.966
-0.052

---

layer selection. In particular, we investigate the question: can we debias only the very last layer, with the remaining layers as the biased encoder? We report the results for MEPS and CelebA datasets in Table 3 where the second row depicts the result of debiasing only the last layer. On MEPS dataset, R.FF-Last has better performance in terms of the two fairness metrics with only 0.2% accuracy difference. Similarly, for CelebA dataset, with 2.7% accuracy difference, RNF-Last has better fairness performance (with 13% DP improvement). This demonstrates that debiasing only the last layer can achieve similar performance compared to debiasing the last several layers.
4.6 Representation Neutralization with Debiased Encoder
In the discussions so far, we reported the performance of RNF while debiasing only the classification head. In this section, we analyze the impact of RNF built on top of a debiased encoder. We first use Adversarial or EOR training to learn a debiased encoder - which is subsequently used as the backbone encoder for updating the classification head using RNF. This experiment is performed on the MEPS dataset where both Adversarial and EOR methods achieve competitive performance. We use the same hyper-paramedte single point in the fairness accuuracyy curve. The results are shown in Table 4 We observe RNF EOR to achieve better fairness performance over RNF, with DP metric improvement of 1.6% and AEO metric moves closer to O. However, such improvement in the fairness metrics incur some loss in the task performance where the accuracy reduces by 0.5%. We observe a similar trend with the combination of RNF and Adversarial training, where the joint combination improves the fairness metrics DP and AEO. Similar to the previous case, this faimess improvement is achieved at the expense of some task performance degradation, where the accuracy drops by 1.3%. This indicates that our RNF is complementary to using a debiased encoder where the joint combination performs better than either of them in terms of the faimess metrics with some loss task performance.
Table 4: RNF with debiased encoder.
MEPS
Models
Accuracy
DP
AEO
Vanilla RNF
0.862
0.866
-0.210
0.839
0,964
-0.099
RNF_EOR BOR
0.834
0,980
-0.049
RNF Adv__sversarial
0.826
0.971
-0.085
for different RNF variants and report a
5 Conclusions and Future Work
In this work, we demonstrate that even when input representations are biased, we can still improve fairness by debiasing only the classification head of the DNN models. We introduce the RNF framework for debiasing the classification head by neutralizing training samples that have the same ground truth label but with different sensitive attribute annotations. To reduce the reliance on sensitive attribute annotations (as used in existing works), generate proxy annotations by training a bias- intensified model and then annotating samples based on its confidence level. Experimental results indicate our RNF framework to dramatically reduce the discrimination of DNN models, without requiring access to annotations for the sensitive attributes for all the training samples. Experimental analysis further demonstrates our RNF framework to further improve in conjunction with other debiasing methods. Specifically, our RNF framework built on top of a debiased backbone encoder leads to better mitigation performance with negligible accuracy drop in the task performance. It is worth noting that our RNF framework could help alleviate the discrimination rather than eliminate it.
On the other hand, the experimental analysis indicates a mitigation gap between RNF and RNFGr- This is because we assume zero access to the protected attribute annotations when using GCE framework. It is desirable to further boost the quality of the generated proxy annotations. In real- world applications, domain experts could be involved to annotate a small ratio of the samples for the training set. Equipped with this small ratio of high quality protected attribute annotations, generate proxy annotations for other training samples with a higher accuracy compared to the proxy annotations generated by GCE framework. As such, we can further boost the mitigation performance of RNF This is a challenging topic and would be explored in our future research.
could
Funding Transparency Statement
The work is in part supported by NSF grants views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
IIS- and 193979..... The
10

---

References
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2016.
[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. 2019 Annual Conference of the North American Chapter of the Association for Computatiional Linguistics (NAACL), 2019.
[3] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (NeurIPS), 2012.
[4] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. survey on bias and fairness in machine learning. ACM Computing Surveys, 2021.
[5] Mengnan Du, Fan Yang, Na Zou, and Xia Hu. Fairness in deep learning: computational perspective. IEEE Intelligent Systems, 2020.
[6] Rachel KE Bellamy, Kuntal Dey, Michael Hind, Samuel C Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, et al. Ai fairness 360; An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv preprint arXiv: 1810.01943, 2018.
[7] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. wwtp::/wwwwpppicccccooaaaacceeie 2016.
Machine bias.
[8] Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Jummo Kim. Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2019.
[9] Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Wei Chang, and Vicente Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. International Conference on Computer Vision (ICCV), 2019.
[10] Yanai Elazar and Yoav Goldberg. Adversarial removal of demographic attributes from text data. 2018 Conference Empirical Methods in Natural Language Processing (EMNLP), 2018.
[11] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint arXiv: 190.:02:9070089 2019.
[12] Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In International Conference on Machine Learning (ICML), 2020.
[13] Krishna Kumar Singh, Dhruv Mahajan, Kristen Grauman, Yong Jae Lee, Matt Feiszli, and Deepti Ghadiyaram. Don't judge object by its context: Learning to overcome contextual bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[14] Andrea Zunino, Sarah Adel Bargal, Riccardo Volpi, Mehrnoosh Sameki, Jianming Zhang, Stan Sclaroff, Vittorio Murino, and Kate Saenko. Explainable deep classification models for domain generalization. arXiv preprint arXi::2003.064493 2020.
[15] Laura Rieger, Chandan Singh, W James Murdoch, and Bin Yu. Interpretations are useful: penalizing explanations to align neural networks with prior on Machine Learning (ICMI.), 2020.
prio knowledge. International Conference
[16] Mohammad Pezeshki, Dékou-Oumal Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity neural networks. arXiv preprint arXiv:2011.09466 2020.

---

[17] Robert Geirhos, Jern-Kerrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning deep neural networks. Nature Machine Intelligence, 2(1 2(11):663 1:665-6733 2020.
[18] Timothy Niven and Hung- Yu Kao. Probing neural network comprehension of natural language arguments. 57th Annual Meeting of the Association for Computational Linguisties (ACL), 2019.
[19] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations (ICLR), 2018.
[20] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning (ICML), 2019.
[21] Hao Wang, Berk Ustun, and Flavio Calmon. Repairing without retraining: Avoiding disparate impact with counterfactual distributions. In International Conference on Machine Learning (ICML), 2019.
[22] Christina Wadsworth, Francesca Vera, and Chris Piech. Achieving faimess through adversarial learning: an application to recidivism prediction. Faimess,, Accountability, and Transparency in Machine Learning (FAT/ML), 2018.
[23] Harrison Edwards and Amos Storkey. Censoring representations with an adversary. International Conference on Learning Representations (ICLR), 2016.
[24] Frederick Liu and Besim Avei. Incorporating priors with feature attribution on text classification.
57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
[25] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In Advances in neural information processing systems (NeurIPS), 2017
[26] Niki Kilbertus, Mateo Rojas Carulla, Giambatiisa Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Schölkopf. Avoiding discrimination through causal reasoning. In Advances Neural Information Processing Systems (NeurIPS), 2017.
[27] Pengyu Cheng, Weituo Hao, Siyang Yuan, Shijing Si, and Lawrence Carin. Fairfil: Contrastive neural debiasing method for pretrained text encoders. International Conference on Learning Representations (ICLR), 2021.
[28] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. International Conference on Learning Representations (ICLR), 2020.
[29] Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. Learning from failure: Training debiased classifier from biased classifier. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[30] Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017.
[31] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
[32] Ali Khodabakhsh, Raghavendra Ramachandra, Kiran Raja, Pankaj Wasnik, and Christoph Busch. Fake face detection methods: Can they be generalized? In 2018 International Conference of the Biometrics Special Interest Group (BIOSIG), 2018.
[33] Ching-Yar Chuang and Youssef Mroueh. Fair mixup: Fairness via interpolation. International Conference Learning Representations (ICLR), 2021.
[34] Ron Kohavi. Scaling up the accuracy of nayee- classifiers: A decision-tree hybrid. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (KDD), 1996.
12

---

[35] Gökhan H Bakır, Jason Weston, and Bernhard Schölkopf. Learning to find pre-images. Advances in neural information processing systems (NeurIPS), 2004.
[36] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. NeurlPS 2014 Deep Learning Workshop, 2015.
[37] Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in Neural Information Processing Systems (NeurIPS), 2018.
[38] Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkata- subramanian. Certifying and removing disparate impact. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2015.
[39] Moritz Hardt, Eric Price, Nati Srebro, et Equality of opportunity supervised learning. In Advances in neural information processing systems (NeurIPS), 2016.
[40] Steven B Cohen. Design strategies and innovations in the medical expenditure panel survey. Medical care, pages III1- 2003.
[41] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings International Conference on Computer Vision (ICCV), 2015.
[42] Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society (AIES), 2018.
[43] Yahav Bechavod and Katrina Ligett. Penalizing unfairness in binary elassification. Fairness, Accountability and Transparency in Machine Learning (FAT/ML), 2017.
[44] Mengnan Du, Ninghao Liu, and Xia Hu. Techniques for interpretable machine learning. Communications of the ACM, 2019.
13


Reviews:
[{"review_id": "kaMFKn-rkyW", "reviewer": "Reviewer_s8dC", "summary": "This paper proposes a two stage scheme for learning models that do not rely on sensitive attributes such as race, gender etc for making predictive decisions. In the first stage of the scheme, a bias amplified 'teacher' network is trained and the confidence scores for this network is used to segment samples into groups. In the second stage, a mixup-like strategy that combines the representations and predictions for input examples in the same class but with different sensitive attributes is used to 'debias' the classification head and ultimately the model.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\nThe RNF scheme proposed in this paper combines several ideas previously known. For example, the combination scheme used for 'neutralizing' the representations is essentially mixup applied to the representations. An intriguing insight in this work is in section 2.3, which seems to suggest that one could use a model's confidence as a way to segment input samples into different protected categories.\n\n### Quality\nThis work contains several components, which are not entirely well justified and tested. The entire paragraph starting from lines 100-108 consist of a series of assertions that ought to be justified. I go into more detail about this in the later sections of this review. The biggest issue regarding quality that I have with the work as it stands is that several of the key components are assumed to work as needed without testing or justification. Please see the additional comments section for more detail on these comments.\n\n### Clarity\nI found it quite challenging to digest sections 2.2 and 2.3. It took several readings and I am not quire sure I understand the entire scheme presented here. It has several moving parts, so it can be difficult to keep these straight. For example, the qualifiers teacher and student are used several times but for different models at the different stages of the RNF scheme. I would suggest that these sections be revised such that the entire scheme is first presented end-to-end. Afterwards, the successive portions can dive into the details of each portion of the scheme. The term 'classification head' while intuitive should also be defined, early on, when it is first used.\n\n### Significance \nThe question that the paper seeks to answer is to reduce a model's dependence on spurious correlation. The spurious correlation problem is considered though in the setting of model debiasing. The paper proposes a scheme for automatically learning the sensitive attribute of an input, if this scheme turns out to be effective in practice, then this component of the work will high impact in other settings. \n\n### Additional Comments\n1) How do we know  NRF 'works'? My biggest concern is that each stage of the scheme is presented and assumed to work without testing or justification. For example, how to we know that segmenting inputs based on the model confidence is reliable for predicting the sensitive attribute for a particular input? In addition, why should $\\frac{z_1 + z_2}{2}$ with the mixup strategy 'neutralize' representations? The insights presented in section 2.1 are all based on an analysis of 500 samples for the adult dataset, should we expect this to also hold for others? My point here is that I would suggest that the authors construct a toy setting or a semi-synthetic dataset where the ground truth is known a priori and then use this setting to thoroughly test and validate every component of the NRF scheme. This is my biggest issue with the paper and it would allay most of my concerns if this is done.\n\n2) How are $f_T(x)$, the teacher network, and $f_B(x)$ trained exactly? What are the inputs and what are the outputs of these models? Specifically, is any sensitive attribute annotation used at all for training $f_B(x)$?\n\n3) I am confused about Theorem 1. What does 'well-trained' mean here and is it reasonable to expect representations to satisfy the lipschitz like requirement stated in this theorem? Additional discussion should also be provided about why this theorem is helpful.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper does not include an explicit limitations section in the draft.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see any ethical concerns with this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6.5"}, "extra_text": {}}, {"review_id": "Vh6CBYeglg", "reviewer": "Reviewer_DyXC", "summary": "This work proposes RNF, an approach to train debiased classifier on top of an (likely biased) feature extractor. The method is a mixup-style data-augmentation approach, where given a feature exactor g and data $x_1$, RNF trains the classifier c by sample another example $x_2$ that correspond to same label but different sensitive attribute, and train classifier p=c(z) on the input-output pair $(\\frac{z_1+z_2}{2}, \\frac{p_1 + p_2}{2}$) for $z_k=g(x_k)$. In the case that the sensitive attribute is not available, author proposed an approach to generate proxy annotations using uncertainty from a bias-augmented classifier. Authors conducted experiment on categorical and image benchmarks, showing RNF compare favorably against baselines. Detailed ablation analysis on the effect of hyperparameters, encoders are also conducted.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality & Significance**\nThis work proposed an simple, empirically effective, and theory backed approach to debiasing model in face of unknown bias. Although ideas of similar flavor has been represent in recent literature, the specific mix-up style training method proposed here appears to be novel, and can be of interest to practitioners and researchers in the fairness community.\n\n**Quality**:\nThe proposed approach and technical derivation appears to be sound. \n\n**Clarity**: \nThe paper is clearly structured and well written. The Related work section seems missing some recent works (e.g., [1] that decouple representation and classifier learning, and [2] using GCE loss to debias classifier). Please add them and related approaches to Section 4. I have some minor comment / concerns listed below, and would appreciate if authors can clarity / adding relevant discussion to the paper. \n\n(1) **Statement of Theorem 1**: If I understand correctly, theorem 1 states that the between-group generalization gap is bounded by three terms: (1) RNF loss $\\epsilon_c$, (2) original loss $\\epsilon_L$ (3) maximum difference of between-group prediction $\\epsilon_p$. My understanding is while (1) guaranteed to be small due to minimizing RNF loss, (2) is not (e.g., accuracy-fairness trade-off), and it is not clear to me that there's guarantee that (3) would be small either. Therefore, if there's accuracy-fairness trade-off with low-value of $\\epsilon_c$ and high-value of $\\epsilon_L$, this upper can be rather loose. I wonder if authors can comment on if this is a plausible scenario? More generally, it would be helpful to provide more detailed interpretation of Theorem 1, and specifically stating roughly in what situation the desired guarantee is known to hold, and discuss possible scenarios the guarantee / method breaks down.\n\n(2) **Justification for uncertainty-based proxy annotation**: Contrary to section 2.1., the proxy-generation  in Section 2.3. seems to receive less justification. While I am convinced of the usefulness of this approach in __some__ situations. I wonder if author can be more precise in either what assumption on the data is need for this approach to work well. Alternatively, comment on in what situation the approach breaks down. In particular, high uncertainty can correspond to ambiguous examples or label noise, rather than the membership of a under-privileged group. \n\n(3) **different label but same sensitive attribute?**. The current implementation of the method selects contrastive examples by selecting those with same label but different sensitive attribute. I wonder how authors arrived at this particular choice. For example, how about selecting contrastive examples by picking those from different label but same sensitive attribute, and adapt RNF correspondingly? Intuitively, this would be an sensible approach as well since we forced the classifier to ignore the sensitive attribute in the prediction. Some comment / discussion around alternative choices like this would strengthen the paper further.\n\n**Related Work**:\n[1] Kang et al (2020) Decoupling Representation and Classifier for Long-Tailed Recognition. ICLR 2020.\n[2] Nam et al (2020) Learning from Failure: Training Debiased Classifier from Biased Classifier. NeurIPS 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "M4M5-Wq29_m", "reviewer": "Reviewer_5yfA", "summary": "This paper proposes a method to improve the fairness metric, e.g. Demographic Parity and Equalized Odds, by neutralizing the classification head to prevent it from relying on spurious attribute. To do this, they use the interpolation of pretrained encoder representations from a pair of sample that share the same label but different attributes and learn an interpolated target. To remove the reliance on the sensitive attributes, authors propose to train a biased classifier can use that to assign pseudo attributes to examples.\n\n**Contributions**: the proposed method contributes to the line of fairness research, and it performs well compared to baseline methods that require sensitive attributes at training time. In contrast, the proposed method does not rely on sensitive attributes. In addition, the paper is very well written. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: the idea resembles the fair mixup (Chuang et al., 2021), but it additionally train a biased encoder to make predictions on the sensitive attributes which makes the method free of demographic attributes. In terms of decoupling the classification with the encoder, it resembles the work in reference [1], which however focuses on long-tailed recognition.\n\n**Weakness and Questions**:\nThe paper is clear and well written; however, this work does not take into account a relatively large body of work on fairness/debiasing work recently that also do not have to rely on sensitive attributes. A large body of work w.r.t. unlearning spurious correlations is group DRO [4, 5, 6] / DRO [7]; and there is also adversarial method that do not need group information [2] and reweighing method [3]. Although, most of these work focus on the worst-group accuracy (Rawlsian fairness), some of them should be included as the baseline methods and discussed, instead of only comparing with methods that use sensitive attributes.\n\nAlso, could you discuss the relationship of your method with [8, 9]? It looks that they are very relevant to the proposed approach.\n\n(a) Although it is discussed in the paper that the proposed method can be applied to multi-attribute setting, I strongly recommend to perform experiments in a more realistic setting, e.g. the toxicity detection task in [4] and CivilComments dataset in [3].\n\n(b) Is there a typo in equation (1)? where the model prediction \\hat{y}_i is expressed as the average of the representations.\n\n(c) The proposed method is very sensitive to hyperparameters, and it looks that there are multiple hyperparameters to tune (L242-245).\n\n(d) Could you elaborate more on why \\alpha in Eq. (4) controls the trade-off between accuracy and fairness?\n\n**References**:\n\n[1] DECOUPLING REPRESENTATION AND CLASSIFIER FOR LONG-TAILED RECOGNITION, Kang et al., ICLR 2021\n\n[2] Fairness without Demographics through Adversarially Reweighted Learning, Lahoti et al., NeurIPS 2020\n\n[3] Just Train Twice: Improving Group Robustness without Training Group Information, Liu et al., ICML 2021\n\n[4] Examining and Combating Spurious Features under Distribution Shift, Zhou et al., ICML 2021\n\n[5] No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems, Sohoni et al., NeurIPS 2020\n\n[6] DISTRIBUTIONALLY ROBUST NEURAL NETWORKS FOR GROUP SHIFTS, Sagawa et al., ICLR 2020\n\n[7] Fairness Without Demographics in Repeated Loss Minimization, Hashimoto et al., ICML 2018\n\n[8] Debiasing representations by removing unwanted variation due to protected attributes, Bower et al., FAT 2018\n\n[9] Learning De-biased Representations with Biased Representations, Bahng et al., ICML 2020", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "AMJ9Lpt9xaT", "reviewer": "Reviewer_Ehds", "summary": "The paper makes the argument that when mitigating biases from an existing dataset most current methods rely on producing input encoders that produce features where bias has been minimized. However, the proposed approach instead aims to remove biases in the final classifier that takes such features as input. The method consists in training a regular classifier and a bias-amplified classifier. The bias-amplified classifier is carefully constructed by considering groups of samples based on the confidence of their predictions. The method neutralizes features by  cleverly leveraging ideas from mixup [20,21], where samples with the same target label but different sensitive attributes are combined by linear interpolation of its input representation and labels. In the absence of sensitive attributes, the predictions from the bias-amplified classifier can be used.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ An alternative method for bias mitigation is proposed that is complementary to other bias mitigation strategies that aim to mitigate bias at the representation-level.\n+ An alternative method for bias mitigation is proposed that does not rely on sensitive attribute annotations but rather creates its own groups by training a bias-amplified classifier. \n+ Innovative approach leveraging the idea of mixing samples by simple interpolation.\n\nWeaknesses:\n+ Experiments on rather simple datasets, Adult and MEPS, and only one dataset that relies on a more complex encoder CelebA. Other fairness works addressing biases in deep learning classifiers go beyond these settings to multi-label and multi-class classification. e.g. COCO, Imsitu [10], CIFAR-10S (Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation by Wang et al CVPR 2020). It is unclear whether the presented method will be competitive in these settings -- it might be but then it seems like such a missed opportunity.\n+ There's a good justification for the method presented when labels for sensitive attributes are presented (Theorem 1), however it is unclear why the method is working when proxy annotations for sensitive attributes are used. Overconfidence might work in the datasets used in the submission but it might not be competitive in complex images with multiple objects with attributes such as gender.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not explicitly but the submission itself addresses a societal issue.", "review_text": "", "extra_scores": {"ethical_concerns": "Not that I can tell. Perhaps a warning on the limitations of the work as it is restricted to mitigation and not bias elimination.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main merit of this work seems to be a novel approach for debiasing and for creating proxy annotations. The reviewers thought that the experimental setup was a bit weak, missing (a) comparisons with other baselines that don't require sensitive attributes; (b) experiments with more datasets; (c) discussing related work. These issues were partly answered during the discussion phase, and I strongly encourage the authors to include the new experiments and revise their discussions in their next revision. 

The reviews are still split and there is still unclarity about the neutralization scheme that uses an average. The reviewers would have liked additional clarification beyond the toy setting, and demonstration that it works with multiple attributes. Relatedly, there was some discussion whether experimenting with multiple attributes is a must, and the general feeling is that it's not a must have. However, a discussion of this extension is necessary. And, further clarification/justification for the neutralization scheme is also needed.
More generally, the reviewers pointed out many parts that were unclear or not justified, most of which were answered by the authors. So these should be good pointers on how to improve the paper in the next revision. 

Decision: Accept (Poster)