Fairness via Representation Neutralization

Abstract: Existing bias mitigation methods for DNN models primarily work on learning debiased encoders. This process not only requires a lot of instance-level annotations for sensitive attributes, it also does not guarantee that all fairness sensitive information has been removed from the encoder. To address these limitations, we explore the following research question: Can we reduce the discrimination of DNN models by only debiasing the classification head, even with biased representations as inputs? To this end, we propose a new mitigation technique, namely, Representation Neutralization for Fairness (RNF) that achieves fairness by debiasing only the task-specific classification head of DNN models. To this end, we leverage samples with the same ground-truth label but different sensitive attributes, and use their neutralized representations to train the classification head of the DNN model. The key idea of RNF is to discourage the classification head from capturing spurious correlation between fairness sensitive information in encoder representations with specific class labels. To address low-resource settings with no access to sensitive attribute annotations, we leverage a bias-amplified model to generate proxy annotations for sensitive attributes. Experimental results over several benchmark datasets demonstrate our RNF framework to effectively reduce discrimination of DNN models with minimal degradation in task-specific performance.


Reviews and author responses:

This paper proposes a method to improve the fairness metric, e.g. Demographic Parity and Equalized Odds, by neutralizing the classification head to prevent it from relying on spurious attribute. To do this, they use the interpolation of pretrained encoder representations from a pair of sample that share the same label but different attributes and learn an interpolated target. To remove the reliance on the sensitive attributes, authors propose to train a biased classifier can use that to assign pseudo attributes to examples.

**Contributions**: the proposed method contributes to the line of fairness research, and it performs well compared to baseline methods that require sensitive attributes at training time. In contrast, the proposed method does not rely on sensitive attributes. In addition, the paper is very well written.   **Originality**: the idea resembles the fair mixup (Chuang et al., 2021), but it additionally train a biased encoder to make predictions on the sensitive attributes which makes the method free of demographic attributes. In terms of decoupling the classification with the encoder, it resembles the work in reference [1], which however focuses on long-tailed recognition.

**Weakness and Questions**:
The paper is clear and well written; however, this work does not take into account a relatively large body of work on fairness/debiasing work recently that also do not have to rely on sensitive attributes. A large body of work w.r.t. unlearning spurious correlations is group DRO [4, 5, 6] / DRO [7]; and there is also adversarial method that do not need group information [2] and reweighing method [3]. Although, most of these work focus on the worst-group accuracy (Rawlsian fairness), some of them should be included as the baseline methods and discussed, instead of only comparing with methods that use sensitive attributes.

Also, could you discuss the relationship of your method with [8, 9]? It looks that they are very relevant to the proposed approach.

(a) Although it is discussed in the paper that the proposed method can be applied to multi-attribute setting, I strongly recommend to perform experiments in a more realistic setting, e.g. the toxicity detection task in [4] and CivilComments dataset in [3].

(b) Is there a typo in equation (1)? where the model prediction \hat{y}_i is expressed as the average of the representations.

(c) The proposed method is very sensitive to hyperparameters, and it looks that there are multiple hyperparameters to tune (L242-245).

(d) Could you elaborate more on why \alpha in Eq. (4) controls the trade-off between accuracy and fairness?

**References**:

[1] DECOUPLING REPRESENTATION AND CLASSIFIER FOR LONG-TAILED RECOGNITION, Kang et al., ICLR 2021

[2] Fairness without Demographics through Adversarially Reweighted Learning, Lahoti et al., NeurIPS 2020

[3] Just Train Twice: Improving Group Robustness without Training Group Information, Liu et al., ICML 2021

[4] Examining and Combating Spurious Features under Distribution Shift, Zhou et al., ICML 2021

[5] No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems, Sohoni et al., NeurIPS 2020

[6] DISTRIBUTIONALLY ROBUST NEURAL NETWORKS FOR GROUP SHIFTS, Sagawa et al., ICLR 2020

[7] Fairness Without Demographics in Repeated Loss Minimization, Hashimoto et al., ICML 2018

[8] Debiasing representations by removing unwanted variation due to protected attributes, Bower et al., FAT 2018

[9] Learning De-biased Representations with Biased Representations, Bahng et al., ICML 2020 N/A

 We thank the reviewer for the insightful follow-up questions.

**1. Did you freeze the CNN parameters and only train the classification head?**

Yes, we freezed the CNN parameters and only trained the classification head when implementing representation neutralization. We mentioned this point on line 46-48 in the Introduction section, and on line 122-124 in Section 2.3 as well as in the caption for Figure 2. We would like to emphasize this in the revised manuscript to make it more clear for readers.

**2. Have you tried to finetune the CNN parameters as well?**

Thanks for this nice question. In this work, we explore the following research question: Can we reduce the discrimination of DNN models by only debiasing the task-specific classification head, even with a biased representation encoder? Thus, we exclusively focused on debiasing the classification head and did not finetune the parameters of CNN, i.e., the representation encoder. We appreciate the reviewer for proposing this insightful research question, and would like to explore it in our future research.


This paper proposes a two stage scheme for learning models that do not rely on sensitive attributes such as race, gender etc for making predictive decisions. In the first stage of the scheme, a bias amplified 'teacher' network is trained and the confidence scores for this network is used to segment samples into groups. In the second stage, a mixup-like strategy that combines the representations and predictions for input examples in the same class but with different sensitive attributes is used to 'debias' the classification head and ultimately the model.  ### Originality
The RNF scheme proposed in this paper combines several ideas previously known. For example, the combination scheme used for 'neutralizing' the representations is essentially mixup applied to the representations. An intriguing insight in this work is in section 2.3, which seems to suggest that one could use a model's confidence as a way to segment input samples into different protected categories.

### Quality
This work contains several components, which are not entirely well justified and tested. The entire paragraph starting from lines 100-108 consist of a series of assertions that ought to be justified. I go into more detail about this in the later sections of this review. The biggest issue regarding quality that I have with the work as it stands is that several of the key components are assumed to work as needed without testing or justification. Please see the additional comments section for more detail on these comments.

### Clarity
I found it quite challenging to digest sections 2.2 and 2.3. It took several readings and I am not quire sure I understand the entire scheme presented here. It has several moving parts, so it can be difficult to keep these straight. For example, the qualifiers teacher and student are used several times but for different models at the different stages of the RNF scheme. I would suggest that these sections be revised such that the entire scheme is first presented end-to-end. Afterwards, the successive portions can dive into the details of each portion of the scheme. The term 'classification head' while intuitive should also be defined, early on, when it is first used.

### Significance 
The question that the paper seeks to answer is to reduce a model's dependence on spurious correlation. The spurious correlation problem is considered though in the setting of model debiasing. The paper proposes a scheme for automatically learning the sensitive attribute of an input, if this scheme turns out to be effective in practice, then this component of the work will high impact in other settings. 

### Additional Comments
1) How do we know  NRF 'works'? My biggest concern is that each stage of the scheme is presented and assumed to work without testing or justification. For example, how to we know that segmenting inputs based on the model confidence is reliable for predicting the sensitive attribute for a particular input? In addition, why should $\frac{z_1 + z_2}{2}$ with the mixup strategy 'neutralize' representations? The insights presented in section 2.1 are all based on an analysis of 500 samples for the adult dataset, should we expect this to also hold for others? My point here is that I would suggest that the authors construct a toy setting or a semi-synthetic dataset where the ground truth is known a priori and then use this setting to thoroughly test and validate every component of the NRF scheme. This is my biggest issue with the paper and it would allay most of my concerns if this is done.

2) How are $f_T(x)$, the teacher network, and $f_B(x)$ trained exactly? What are the inputs and what are the outputs of these models? Specifically, is any sensitive attribute annotation used at all for training $f_B(x)$?

3) I am confused about Theorem 1. What does 'well-trained' mean here and is it reasonable to expect representations to satisfy the lipschitz like requirement stated in this theorem? Additional discussion should also be provided about why this theorem is helpful. This paper does not include an explicit limitations section in the draft.

 I have read the author rebuttal, and it does clarify my major concerns. Thank you.

 I thank the authors for additional experiments on Civil-Comments and baseline experiment JTT, hence I increase my score to 5 given that authors revise their related work and experimental sections. 

One additional question I have is that, did you freeze the CNN parameters and only train the classification head? If so, have you tried finetune the CNN parameters as well?

 We thank the reviewer for the follow-up questions and helpful discussions. Please refer below for experiments with additional baseline and dataset as per the recommendation by reviewer 5yfA as well as reviewer Ehds.

**1. Comparing with baselines that do not require known sensitive attributes.**

Following the reviewer’s suggestion, we compare with the baseline JTT [2] that does not rely on sensitive attributes. JTT is essentially a reweighting method, and is implemented using two stages. In the first stage, it trains the identification model for several epochs. In the second stage, JTT trains a debiased model by upweighting the training samples that have been  misclassified by the identification model. Given the fairness-accuracy trade-off for the different models, for a fair comparison, we tune the hyper-parameter $\lambda_{up}$ in Equation (8) of the JTT baseline [2] such that JTT and RNF-GCE have similar accuracy. Consider the MEPS dataset as an example where we set $\lambda_{up}$ as 2.7. The fairness metric performance is shown below:

|Models| Accuracy        | DP           | $\Delta$EO  |
| -------------| ------------- |:-------------:| -----:|
|Vanilla| 0.862      | 0.866 | -0.210 |
|JTT baseline | 0.837      | 0.945      |   -0.105 |
|RNF-GCE (ours)| 0.839      | 0.964      |   -0.099 |

The first row denotes the result for the vanilla model trained using cross entropy loss, whereas the second row is the result for the JTT baseline. The last row is the result for the proposed method which indicates that our RNF-GCE method can achieve better fairness metric scores compared to the JTT baseline.

**2. Conducting additional experiments in more realistic and complex datasets.**

Thanks for this suggestion. We conducted experiments using the CivilComments dataset in [3], as per the reviewer recommendation. The task is to predict whether a comment text is severely toxic or not. We focus on alleviating the gender bias, and consider “male”, “female”, and “homosextual_gay_or_lesbian” three categories of gender attribute information in the dataset. Note that in reality there are far more categories of gender identities, and we only consider the above three categories. In total we consider 8221 samples with clear gender attribute annotations. It is further split into training set, validation set and test set with 5986, 745, 1490 samples respectively. We use the CNN as the classification model [r1] and treat the last fully connected layer as the classification head. Please refer to Section 3.1.2 for hyper-parameters and other experimental settings. Due to the skewed distribution of the dataset, the trained CNN model with cross entropy loss shows discrimination towards the “homosextual_gay_or_lesbian” group. In other words, both males and females are privileged groups and the homosextual_gay_or_lesbian is the unprivileged group. The fairness and accuracy metric performance are given as follows:

|Models|Accuracy   | DP           | $\Delta$EO  |
| -------------| ------------- |:-------------:| -----:|
|homosextual_gay_or_lesbian vs male| 93.7%      | 0.844 | -0.560 |
|homosextual_gay_or_lesbian vs female| 93.7%      | 0.836      |   -0.549 |

The results indicate that the CNN model trained with cross entropy loss shows significant discrimination towards the homosextual_gay_or_lesbian group. After mitigation with our RNF framework, the fairness and accuracy performance of the CNN model are shown as follows:

|Models|Accuracy   | DP           | $\Delta$EO  |
| -------------| ------------- |:-------------:| -----:|
|homosextual_gay_or_lesbian vs male| 92.0%      | 0.927 | -0.271 |
|homosextual_gay_or_lesbian vs female| 92.0%      | 0.929      |   -0.244 |

It indicates that our RNF framework has substantially reduced the discrimination towards the homosextual_gay_or_lesbian group, with slight accuracy drop.

**3. How do you tune the hyperparameters and how sensitive the proposed method is to the hyperparameters?**

***3.1 How are the hyperparameters tuned?***

The hyper-parameters $q$ and $\gamma$ are tuned based on the fairness accuracy performance on the validation dataset. Considering the Adult dataset as an example, we randomly sample 20% of the data from the training set. We generate proxy annotations for this sampled training dataset to train our RNF model. Based on the fairness-accuracy performance of the model on the validation dataset, we select the $q$ and $\gamma$ corresponding to the best AUC score. This is based on the observation that with better proxy annotations, there is less trade-off between the fairness and accuracy metrics and thus a higher AUC score (see the performance gap between RNF and RNF-GT in Figure 3). After selecting $q$ and $\gamma$, we generate the proxy annotations for the whole training dataset. Similarly, the hyperparameter $T$ is also tuned based on the fairness accuracy performance on the validation dataset.

***3.2 How sensitive is the RNF method to hyperparameters?***

Our representation neutralization framework is relatively insensitive to the hyper-parameter $T$. On the other hand, our framework is relatively sensitive to $q$ and $\gamma$. Note that $q$ and $\gamma$ are used to generate proxy annotations. Since different datasets have totally different characteristics in terms of the data distribution, our proxy annotation algorithm needs to tune $q$ and $\gamma$ for each dataset. This is similar to the several reference works [2-5] which assign pseudo group membership to training samples. 


### References

[r1] Yoon Kim. "Convolutional Neural Networks for Sentence Classification." EMNLP, 2014.


 Thank you for your response and answering my questions! 

First, thanks for considering related work I pointed out! However, I am not fully convinced by the current state of the experiments given these additional experiments are only promised to be added upon acceptance. And I do hope to see additional experiments in more realistic datasets / multi-class classification problems as other reviewers have also pointed out.
Besides, if I am not mistaken, both [4] and [5] do not require known sensitive attributes but just requiring some clustering method to assign pseudo group membership to each training example.

Second, could you provide experiments to demonstrate how sensitive the proposed method is to the hyperparameters ($q$, $\gamma$ and $T$) for different datasets. It seems that these hyperparameters are hardly tuned for different datasets from Section 3.1.2. Are there any intuitions how you set these hyperparameters.


 Thanks authors for answering my questions and I am satisfied with the response. Conditional on the additional discussions / relevant work being added to the paper, I still vote for acceptance and maintain my score of 7. 

 We thank the reviewer for providing helpful feedback and constructive suggestions.

**1 Applicability to more complex datasets.**

Thanks for this suggestion. Although we use a binary classification task as a running example, our framework is generally applicable to more complex datasets and settings, e.g., multi-class classification problems. There are no constraints in the formulations from Equation (1) - (6) that the label $y$ should be binary. 

Our framework can also be extended to settings with multiple attributes. Considering we have three groups with corresponding ground truth sensitive annotations, the loss function in Equation 3 can be modified as follows:
$|c(\frac{\lambda_1 z_1 + \lambda_2 z_2 +\lambda_3 z_3}{\lambda_1+\lambda_2+\lambda_3}) - c(⅓ z_1 + ⅓ z_2 +⅓ z_3)|$,
where all three lambda values are sampled from the interval [0, 1].

We will emphasize these points in the revised manuscript and add experimental evaluations for multi-class and multi-attribute classification datasets.

**2 Justification for generating proxy annotations for sensitive attributes.**

Generating proxy annotations for sensitive attributes is based on the observation that the bias-amplified model tends to assign the privileged group more desired outcome, while assigning the under-privileged group less-desired outcome. For instance in the Adult dataset, the average prediction probability of the desired label (higher income) for the male group is 0.078 higher than that of the female group. Note that this comparison is performed for both groups with the same desired ground truth label. In contrast, the average probability of the less-desired label for the female group is 0.113 higher than that of the male group. Similarly for the MEPS dataset, the average probability of assigning the desired label to the priviledged ‘white’ group is 0.128 higher than that of the under-priviledged ‘non-white’ group; whereas the average probability of assigning the less-desired label to the ‘non-white’ group is 0.075 higher than that of the ‘white’ group. Note that these numbers are statistically significant, given that the prediction probability for each instance is within [0, 1].

To verify the effectiveness of proxy annotations, we calculate their alignment with ground truth sensitive attributes, using the metrics of precision, recall and F-Measure with results as follows:


|Dataset| Precision        | Recall           | F-Measure  |
| -------------| ------------- |:-------------:| -----:|
|Adult| 0.647      | 0.592 | 0.604 |
|MEPS| 0.651      | 0.643      |   0.646 |


We further validate the effectiveness of the GCE framework in Table 2, where RNF-GCE (with bias-amplified proxy annotations) has better performance than RNF-Random (with random sensitive attribute annotations). However, such proxy annotations, as noted by the reviewers, are still noisy and we observe a performance gap between them and the ground truth annotations. This performance gap is illustrated in Figure 3 between the RNF and RNF-GT (described in lines 255-256). We also mention the limitation of the proxy annotations in Section E at Appendix (line 635-645).

We leverage proxy annotations to train the GCE framework since we assume zero access to the ground truth annotations for the protected attributes. In real-world applications, we can leverage domain experts to annotate a small amount of data for the sensitive attributes (say, 5%) that can be used to train our model to generate better proxy annotations in contrast to the completely unsupervised GCE setup.  


**3 The limitations of the work.**

We had incorporated the limitation section in Appendix E (on page 17, line 635-645). In the revised manuscript, we will move the limitation discussions to Section 5 (page 9) to make the limitations more clear to the readers. Furthermore, we will point out that our RNF framework could help alleviate the discrimination rather than eliminate it in the revised manuscript.


 We thank the reviewer for the insightful comments. We first answer the major concern regarding the justification of each stage of the scheme, and then reply to other questions.

**1. Each stage of the scheme is assumed to work without testing or justification.**

While we motivate the need for each stage and module via some toy tasks and examples, during experiments we extensively evaluate different components as discussed below.
 
***1.1. Why should $\frac{z_1 + z_2}{2}$ with the mixup strategy ‘neutralize’ representations?***

In Figure 1 (a), we observe that after we reduce the representation dimension to 2, one of the dimensions would correspond to the protected attribute. Based on this observation, we construct a toy example where the representations $z_1$ and $z_2$ are 2-dimensional vectors, where the first dimension encodes the protected attribute, and the second dimension encodes task relevant information. Consider we have four samples: [0.1, 0.9], [0.9, 0.8], [0.05, 0.1], [0.95, 0.2], where the first two samples have ground truth labels “positive”, whereas the other two samples have ground truth labels “negative”. Also consider, the first and the third samples correspond to the protected attribute “male”, whereas the second and the fourth one correspond to the protected attribute “female”. After representation neutralization, we have two new samples: [0.5, 0.85], and [0.5, 0.15] -- where our scheme effectively neutralizes the dimension corresponding to the protected attribute, while still preserving well task relevant information. Using the new sample representation to train the classification head, we are essentially aiming to encourage the model to utilize more task-relevant information for prediction, and suppress the undesirable correlation between protected attribute and the predictive task.

***1.2. How to demonstrate that our RNF solution indeed works?***

In Section 3.3, we use explainability and an auxiliary prediction task to analyze the behavior of the classification head $c(z)$. The experimental results are shown in Figure 4. It indicates that the vanilla model trained using only cross entropy loss highly relies on the fairness sensitive information in the representation for prediction. In contrast, our RNF framework adjusts the decision boundary and encourages the classification head to shift its attention from fairness sensitive information to task relevant information.


**2. How are $f_T(x)$, the teacher network, and $f_B(x)$ trained exactly?**

For the teacher network $f_T(x)$, we train it using only cross entropy (CE) loss. In terms of the bias-amplified model $f_B(x)$, we use the generalized cross entropy loss (GCE) function (see Equation (5)). For training both $f_T(x)$ and $f_B(x)$, we use only the task input and label pairs $[x_i, y_i]_{i=1}^N$. When training $f_B(x)$, we do not use any sensitive attribute annotations.

**3. Concerns about Theorem 1.**

Here “well-trained” depicts that the model has learned reasonably good representations to encode sufficient task relevant information. Essentially, after training for a sufficient number of epochs until the validation loss has converged, we have access to a reasonably good representation space (although it might encode a lot of fairness sensitive information). Note that we expect $||z_1 - z_2||_2 \leq \lambda_z |p_1 - p_2| $ to hold true for a well-trained model, where the two samples with the same ground truth label would have small distance. In contrast, if the model has only been trained with only a couple of epochs and is far from convergence, then the distance between $z_1$ and  $z_2$ would be large.

The key idea of Theorem 1 is as follows. We need to first well-train a biased model to encode sufficient task relevant information. If we use the RNF loss function to further train the classification head, the generalization gap between the two protected groups would be small. In this way, the debiased model would reduce discrimination for the under-priviledged group. 


**4. Presentation clarity.**

Thanks for this suggestion. We will introduce the entire scheme first and then introduce its detailed components. We defined our problem setup and semantics of the “classification head” in the third paragraph of Section 2 (lines 78-83). We will emphasize the description in the revised manuscript so as to reduce confusion for the readers.


**5. This paper does not include an explicit limitations section in the draft.**

We had incorporated the limitation section in Appendix E (on page 17, line 635-645). In the revised manuscript, we will move the limitation discussions to Section 5 (page 9) to make the limitations more clear to the readers. 


 Thank the reviewer for the detailed feedback and helpful suggestions. 

**1. Comparing with references [2-7].**

Thanks to the reviewer for suggesting these references [2-7]. In general, fairness-related works can be broadly categorized into the following two categories: parity based notion of fairness and Rawlsian Max-Min fairness [2]. The first category tries to minimize the gap between groups, and aims to optimize fairness metrics such as demographic parity and equality of opportunity -- which we follow in this work. In contrast, the second type of fairness tries to improve the utility of the worst-off groups. Representative examples for the second type of fairness include works addressing: darker skin-tone female groups having much higher misclassification rates [r1], error rate of the black speakers being much higher than that of white speakers for automated speech recognition (ASR) systems [r2], to name a few. Note that all the three works [2,3,7] belong to the second group of fairness, whereas our method addresses the first kind of fairness problem. Besides, [4-6] address the group DRO (distributionally robust optimization) problem. That is the reason why originally we did not include these references in our baselines and discussions. As per the reviewer recommendation, we will add these discussions and references [2-7] in related work.

On the other hand, some of the suggested references such as [2, 3] are relevant to our work in that they do not rely on sensitive attributes, although they work on a different notion of fairness as outlined above. We would like to adapt their solutions from the domain of Rawlsian Max-Min fairness to the domain of parity based notion of fairness, and then include them as baselines in our updated version. 

**2. Relationship with literature [8, 9].**

Firstly, reference [8] is a preprocessing mitigation method, aiming to remove the inherent bias in data and create debiased data. This is related but complementary to our problem setup where we are learning to debias the model, specifically the task classification head. We will discuss the joint combination of debiasing both the data and the model as future work. Secondly, the difference of our method with [9] is two-fold. On one hand, [9] addresses the general spurious learning problem (see Section 4 of our manuscript, lines 363-369), rather than the fairness problem as in our setup. On the other hand, the philosophy of [9] is to debias representations. They train bias-only models that capture biased representations and then train the debiased model by encouraging it to be different with the biased representations. In contrast, our method aims to learn the debiased classification head rather than the debiased representations. The two approaches are complementary and we show via experiments in Section 3.5 that their joint combination can further improve the fairness-utility trade-off.

**3. Perform experiments in a more realistic setting with multiple-attributes.**

Our framework can be extended to settings with multiple attributes. Considering we have three groups with corresponding ground truth sensitive annotations, the loss function in Equation 3 can be modified as follows:
$|c(\frac{\lambda_1 z_1 + \lambda_2 z_2 +\lambda_3 z_3}{\lambda_1+\lambda_2+\lambda_3}) - c(⅓ z_1 + ⅓ z_2 +⅓ z_3)|$,

where all three lambda values are sampled from the interval [0, 1]. We will add experimental evaluations for datasets with multiple protected attributes in the revised manuscript, e.g. the toxicity detection task in [4] and CivilComments dataset in [3].

**4. Why $\hat{y}_i$ is expressed as the average of the representations in equation (1)?**

In Equation (1), $\hat{y}_i = c(\frac{1}{2} z_1 + \frac{1}{2} z_2)$. Besides the average of the representations, the c(.) function corresponds to the classification head. We will emphasize this point in the revised manuscript to make it more clear to the readers.

**5. The proposed method is very sensitive to hyperparameters.**

In total we have three hyperparameters to tune. Firstly, for the representation neutralization algorithm, we have two hyper-parameters: \lambda list in Equation (3) and temperature T. We fix the \lambda list as [0.6, 0.7, 0.8, 0.9] for all experiments and do not need to tune this hyperparameter. We only need to tune the temperature T. Secondly, for the proxy annotation generation framework, we need to tune two hyper-parameters respectively for each dataset.

**6. Why \alpha in Eq. (4) controls the trade-off between accuracy and fairness?**

For Eq. (4), we aim to penalize large changes in softmax probability when we move along the interpolation between two samples for regularization. Here $\alpha$ denotes the degree of penalization. A large $\alpha$ will lead the model to have better fairness performance and slightly worse accuracy performance.

### References

[r1] Buolamwini, Joy, and Timnit Gebru. "Gender shades: Intersectional accuracy disparities in commercial gender classification." ACM FAccT, 2018.

[r2] Koenecke, Allison, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R. Rickford, Dan Jurafsky, and Sharad Goel. "Racial disparities in automated speech recognition." PNAS, 2020.


 Thank the reviewer for your comprehensive summary and encouraging words on our work.

**1 Statement of Theorem 1.**

Yes, theorem 1 states that the between-group generalization gap is bounded by three terms: $\epsilon_c$, $\epsilon_L$, and $\epsilon_p$. $\epsilon_L$ denotes the loss for the original biased model trained using only cross entropy loss and it is a small value. $p_1$ and $p_2$ correspond to the probability of samples with the same ground-truth label, and therefore $\epsilon_p$ is also not a large value. The key idea of Theorem 1 is as follows. We need to first train a biased model to make it well trained. It means that the model has learned reasonably good representations to encode sufficient task relevant information. If we use the RNF loss function to further train the classification head, the generalization gap between two protected groups would be also small. In this way, the debiased model would reduce discrimination for the under-priviledged group. 

We agree with the reviewer that in certain cases the upper bound could be large, and this happens when the original model is not well trained. That is to say, the model is only trained for a couple of epochs and far from being converged. In this case, $\epsilon_L$ would be large. Since we do not have a reasonably good representation space now, $\lambda_z$ would also be large. As such, this upper bound $\epsilon_p(\lambda_z\epsilon_c+\epsilon_L)$ would be loose. Thus we need to train the original biased model for a sufficient number of epochs, in order to make the desirable guarantee to hold. We will add the detailed descriptions to the updated manuscript.

**2. Justification for uncertainty-based proxy annotation.**

The bias-amplified model tends to assign the privileged group more desired outcome, while assigning the under-privileged group less-desired outcome. For the Adult dataset, the average prediction probability of the desired label (higher income) for the male group is 0.078 higher than that of the female group. Note that this comparison is performed for both groups with the same desired ground truth label. In contrast, the average probability of the less-desired label for the female group is 0.113 higher than that of the male group. Similarly for the MEPS dataset, the average probability of the desired label for the ‘white’ group is 0.128 higher than the ‘non-white’ group; while the average probability for less-desired label of the ‘non-white’ group is 0.075 higher than the ‘white’ group. This is statistically significant given that the prediction probability for each instance is in the interval [0, 1].

We agree with the reviewer that the assumption can break down, when the discrimination for a certain protected attribute is not sufficiently significent. Consider the MEPS datasets as an example where we consider two protected attributes: race and gender. We found that the DNN model trained with cross entropy loss has relatively large race bias and very minor gender bias. In this case, we could generate relatively high quality proxy race annotations based on the GCE framework. In contrast, the uncertainty based framework would not work well for the gender attribute, and the proxy gender annotations would have relatively poorer quality.

**3. Different labels but same sensitive attribute?**

We agree with the reviewer that selecting samples with different labels but the same protected attribute, together with other constraints could possibly force the classifier to ignore the sensitive attribute. However, this representation mixup scheme with different task labels can potentially   sacrifice the model’s task accuracy by losing its discrimination power and thus might hurt the utility of the model. We will add the pros and cons of this alternative scheme in the revised manuscript.

**4. Adding references to the Related Work section.**

Although [1] addresses the long-tailed recognition problem, it shares a similar philosophy as ours in decoupling representation and classifier learning. [2] uses the GCE loss to debias classiifer. Both works are relevant and we will add them in Section 4 in revised manuscript.


 We thank all the reviewers for providing many constructive comments and helpful feedback. We are encouraged to find that they have found our contributions to be technically sound and clearly structured (DyXC), innovative (Ehds), clear and well written (5yfA, DyXC).

We have additionally performed experiments to address some of the evaluation concerns. Please find below our detailed response to the questions and any concern raised by the reviewers. We will incorporate all these comments and comprehensive experimental evaluations into the revised manuscript. We are grateful to the reviewers for all the suggestions to improve our work.

Paper6197 Authors


This work proposes RNF, an approach to train debiased classifier on top of an (likely biased) feature extractor. The method is a mixup-style data-augmentation approach, where given a feature exactor g and data $x_1$, RNF trains the classifier c by sample another example $x_2$ that correspond to same label but different sensitive attribute, and train classifier p=c(z) on the input-output pair $(\frac{z_1+z_2}{2}, \frac{p_1 + p_2}{2}$) for $z_k=g(x_k)$. In the case that the sensitive attribute is not available, author proposed an approach to generate proxy annotations using uncertainty from a bias-augmented classifier. Authors conducted experiment on categorical and image benchmarks, showing RNF compare favorably against baselines. Detailed ablation analysis on the effect of hyperparameters, encoders are also conducted.  **Originality & Significance**
This work proposed an simple, empirically effective, and theory backed approach to debiasing model in face of unknown bias. Although ideas of similar flavor has been represent in recent literature, the specific mix-up style training method proposed here appears to be novel, and can be of interest to practitioners and researchers in the fairness community.

**Quality**:
The proposed approach and technical derivation appears to be sound. 

**Clarity**: 
The paper is clearly structured and well written. The Related work section seems missing some recent works (e.g., [1] that decouple representation and classifier learning, and [2] using GCE loss to debias classifier). Please add them and related approaches to Section 4. I have some minor comment / concerns listed below, and would appreciate if authors can clarity / adding relevant discussion to the paper. 

(1) **Statement of Theorem 1**: If I understand correctly, theorem 1 states that the between-group generalization gap is bounded by three terms: (1) RNF loss $\epsilon_c$, (2) original loss $\epsilon_L$ (3) maximum difference of between-group prediction $\epsilon_p$. My understanding is while (1) guaranteed to be small due to minimizing RNF loss, (2) is not (e.g., accuracy-fairness trade-off), and it is not clear to me that there's guarantee that (3) would be small either. Therefore, if there's accuracy-fairness trade-off with low-value of $\epsilon_c$ and high-value of $\epsilon_L$, this upper can be rather loose. I wonder if authors can comment on if this is a plausible scenario? More generally, it would be helpful to provide more detailed interpretation of Theorem 1, and specifically stating roughly in what situation the desired guarantee is known to hold, and discuss possible scenarios the guarantee / method breaks down.

(2) **Justification for uncertainty-based proxy annotation**: Contrary to section 2.1., the proxy-generation  in Section 2.3. seems to receive less justification. While I am convinced of the usefulness of this approach in __some__ situations. I wonder if author can be more precise in either what assumption on the data is need for this approach to work well. Alternatively, comment on in what situation the approach breaks down. In particular, high uncertainty can correspond to ambiguous examples or label noise, rather than the membership of a under-privileged group. 

(3) **different label but same sensitive attribute?**. The current implementation of the method selects contrastive examples by selecting those with same label but different sensitive attribute. I wonder how authors arrived at this particular choice. For example, how about selecting contrastive examples by picking those from different label but same sensitive attribute, and adapt RNF correspondingly? Intuitively, this would be an sensible approach as well since we forced the classifier to ignore the sensitive attribute in the prediction. Some comment / discussion around alternative choices like this would strengthen the paper further.

**Related Work**:
[1] Kang et al (2020) Decoupling Representation and Classifier for Long-Tailed Recognition. ICLR 2020.
[2] Nam et al (2020) Learning from Failure: Training Debiased Classifier from Biased Classifier. NeurIPS 2020. Yes.

The paper makes the argument that when mitigating biases from an existing dataset most current methods rely on producing input encoders that produce features where bias has been minimized. However, the proposed approach instead aims to remove biases in the final classifier that takes such features as input. The method consists in training a regular classifier and a bias-amplified classifier. The bias-amplified classifier is carefully constructed by considering groups of samples based on the confidence of their predictions. The method neutralizes features by  cleverly leveraging ideas from mixup [20,21], where samples with the same target label but different sensitive attributes are combined by linear interpolation of its input representation and labels. In the absence of sensitive attributes, the predictions from the bias-amplified classifier can be used.  Strengths:
+ An alternative method for bias mitigation is proposed that is complementary to other bias mitigation strategies that aim to mitigate bias at the representation-level.
+ An alternative method for bias mitigation is proposed that does not rely on sensitive attribute annotations but rather creates its own groups by training a bias-amplified classifier. 
+ Innovative approach leveraging the idea of mixing samples by simple interpolation.

Weaknesses:
+ Experiments on rather simple datasets, Adult and MEPS, and only one dataset that relies on a more complex encoder CelebA. Other fairness works addressing biases in deep learning classifiers go beyond these settings to multi-label and multi-class classification. e.g. COCO, Imsitu [10], CIFAR-10S (Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation by Wang et al CVPR 2020). It is unclear whether the presented method will be competitive in these settings -- it might be but then it seems like such a missed opportunity.
+ There's a good justification for the method presented when labels for sensitive attributes are presented (Theorem 1), however it is unclear why the method is working when proxy annotations for sensitive attributes are used. Overconfidence might work in the datasets used in the submission but it might not be competitive in complex images with multiple objects with attributes such as gender. Not explicitly but the submission itself addresses a societal issue.

Meta-review: The main merit of this work seems to be a novel approach for debiasing and for creating proxy annotations. The reviewers thought that the experimental setup was a bit weak, missing (a) comparisons with other baselines that don't require sensitive attributes; (b) experiments with more datasets; (c) discussing related work. These issues were partly answered during the discussion phase, and I strongly encourage the authors to include the new experiments and revise their discussions in their next revision. 

The reviews are still split and there is still unclarity about the neutralization scheme that uses an average. The reviewers would have liked additional clarification beyond the toy setting, and demonstration that it works with multiple attributes. Relatedly, there was some discussion whether experimenting with multiple attributes is a must, and the general feeling is that it's not a must have. However, a discussion of this extension is necessary. And, further clarification/justification for the neutralization scheme is also needed.
More generally, the reviewers pointed out many parts that were unclear or not justified, most of which were answered by the authors. So these should be good pointers on how to improve the paper in the next revision. 


Decision: accept