Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a mitigation technique, namely, Representation Neutralization for Fairness (RNF) that achieves fairness by debiasing only the task-specific classification head of DNN models. They leverage samples with the same ground-truth label but different sensitive attributes and use their neutralized representations to train the classification head of the DNN model. Experimental results over several benchmark datasets demonstrate our RNF framework to effectively reduce discrimination of DNN models with minimal degradation in task-specific performance.
main_review: 1. The paper has dealt with the problem of bias mitigation methods for DNN models which is a very common, prevailing and still challenging problem.
2. The model is well described with proper model figure, dataset description, problem statement, adequate references.
3. The algorithm is clear. It provides reasonably complete information that will help reproduce the results, including the methodology, pseudo-code, and empirical evaluation.
4. The paper is accompanied by an adequate set of experiments for evaluating the effectiveness of the solutions the authors propose.
limitations_and_societal_impact: The authors adequately addressed the limitations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a method for fine-tuning the upper layers of a DNN (the "classification head") with the goal that it becomes insensitive to information in the encoded representation that correlates with a protected attribute. The technical contribution leverages the *mixup* approach to present "neutralized" encodings to the classification head, which learns to project them to softened target probabilities. Another key contribution is that this process can be done *without* access to ground truth labelling of the protected attribute. As a proxy, the authors assume that the unneutralized model will predicatably generate biased output in favour of a specified demographic group. They use that output to generate proxy annotations to take the place of ground truth labelling of the protected attribute.

The work is generally well-written and I have many minor comments, questions, and some suggestions for improvement and clarification throughout.

~However, I have concerns about the evaluation. These concerns are the primary basis for my ultimate recommendation to not accept. If these concerns are addressed or if I have missed something, I am open to adjusting my recommendation.~

The author(s) have provided very helpful responses:

* they "will rephrase the claim that 'RNF… outperform baseline methods…' in the revised manuscript to instead highlight the resource benefits of RNF over the baseline methods with similar or better performance in some cases,"
* while not going as far as statistical significance tests, they have provided more information that allows me to better assess whether the observations are mere noise or true effects; they will average across more (10) runs and will include error bars in the figure
* they have agreed to re-organize some of the presentation to make more clear some of the points I had questions about

Based on these commitments, I have revised my recommendation to "7: good paper, accept."
main_review: # Originality

The proposed method seems novel and parts of it surprise me! The work seems to adequately cite its motivations and cites what it builds off of.

I was most surprised by the claim that the framework "does not require any access to the protected attributes during training."

# Quality & Clarity

**I wonder throughout whether the phrase "representation neutralization" is a good name for this technique.** You admit that you are debiasing *only* the classification head and accept that the encoded representation itself *may be biased*. And you position your work in *contrast* to existing methods that learn debiased representations. I do see how you "neutralize" the representation as part of fine-tuning the classification head, but at inference time, the representation reaching the classification head is not neutralized in the same sense. Instead, the classification head has just hopefully become insensitive to the information correlated with the protected attribute. I don't know if I have a good suggestion for an alternative name. Maybe, something like desensitized classification. In any case, it's just a name, but it did put the wrong idea in my head for the early parts of the paper.

## Introduction

You seem to be alluding to COMPAS at line 25. I think works frequently cite this ProPublica article when referring to COMPAS: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

I also would have expected citations to Calo & Citron, "The Automated Administrative State" (https://scholarlycommons.law.emory.edu/elj/vol70/iss4/1/), and Kate Crawford, *Atlas of AI*.

Suggest citing to Edwards & Storkey, "Censoring Representations with an Adversary" (ICLR, 2016).

At line 23, you refer to *the* recidivism algorithm. This implies that there is only one. Suggest something like: "COMPAS, an algorithmic recidivism predictor, is likely to associate..."

At line 34: "The suppression of fairness sensitive information might also remove useful information that is task relevant." Much work in this area acknowledges that this is the trade-off that anti-discrimination regimes require: even "rational discrimination" is not okay. See Prince & Schwarcz, "Proxy Discrimination in the Age of Artificial Intelligence and Big Data" (https://ilr.law.uiowa.edu/print/volume-105-issue-3/proxy-discrimination-in-the-age-of-artificial-intelligence-and-big-data).

I wonder whether it is helpful to refer to the correlation with the sensitive attribute as "spurious." These correlations are often the result of embedded social structures. The correlation is not spurious in the sense that it arose from randomness in a particular sample. Rather, the correlation has arisen *for real* in the population because of our decisions as society.

## Representation Neutralization for Fairness

Throughout this paper, until I got to 3.1.2, I was very curious and confused about how you determined what the "classification head" was. This confusion arises because of the way you talk about it in the early sections. You refer to "the task-specific classification head of a DNN" as if it is a clearly identified add-on. In actuality, it is just the top layer(s) of the DNN that you have chosen to set aside for fine-tuning. If there is any way you can clarify this early on, that would be helpful.

At 76, you say you consider "binary sensitive attributes" but go on to choose gender for the examples. Consider adding a caveat (even just a footnote) that gender is only binary as coded in the data, despite it not being a binary in reality.

## Analysis of the Classification Head

Again, it would be helpful to note that there is in general no such thing as a distinct "classification head" other than how you have decided to fragement your network(s) for analysis / fine-tuning purposes.

At line 101, you distinguish between "useful task relevant classification information" and "bias information from protected attributes." But this isn't a true disctinction and that is part of the problem. The issue is that the protected attributes have *become correlated* with the relevant classification target and thus *become* useful task relevant classification information. The goal of anti-discrimination frameworks is to require decision-makers to ignore that correlation *even if* it would improve some performance metric.

## Representation Neutralization for Debiasing Classification Head

Again, you make the distinction between "sensitive attributes" and "task-relevant information." The problem is that sensitive attributes have *become* task-relevant information. Perhaps you mean to distinguish between "sensitive attributes" and "permitted factors"?

Here, it becomes clear that you freeze the encoder *after* it has learned what you are calling a spurious correlation, which you hope that your fine-tuned classification head can ultimately ignore.

A question about lines 139-141. You say, "From the input perspective, the neutralization of representations suppresses the model from capturing the spurious correlation between fairness sensitive information in the representation with the class labels." Were you being careful here in saying "suppresses" rather than "prevents"? If so, that matches how I understand this would work. I don't see how your smoothing/averaging would necessarily eliminate the spurious correlation from making it through, but could suppress it to an extent. Am I misunderstanding this? Regardless, this wasn't critical to my review, but just a place where you could perhaps be clearer if I did misunderstand you.

I think it would be helpful to emphasize early on that the neutralization only happens during the fine-tuning of the hopefully debiased classification head. At inference time, the features just come in as encoded, to a classification head that has hopefully learned to not exploit any of the information correlated with the sensitive attribute.

The **smoothing neutralization** seems to be introducing a penalty for large changes in soft-max probabilities as you move along the interpolation between samples. Is this correct?

## Generating Proxy Annotations for Sensitive Attributes

You say, "A biased model generates over-confident predictions for one demographic group, while giving much lower scores for the alternative group." Can you justify this claim? Or is this true by definition? What happens if you apply your technique to an unbiased model?

I think the phrase "over-confident" is adding some confusion here. You don't mean merely that predictions are close to 1 or 0, but rather, that there is a demographic group that tends to get the less-desired outcome and another demographic group that tends to get the more desired outcome.

Ther paragraph from lines 186--202 is very dense and some aspects come out of nowhere. The "splitting threshold" for example. How is it used? How do you use the fairness-accuracy tradeoff to determine its value? It would be helpful if you took more space to explain this all.

## Experiments

## Experimental Setup

## Fairness Metrics, Benchmark Datasets, and Baselines

Generally very clear.

At line 217, you say "bias towards the female group." This is ambiguous, as it could mean bias in favour of female groups or bias disfavouring the female group. An alternative: "vanilla trained models depict bias that predicts females earn less."

At line 220, you say "discrimination towards." This is less ambiguous, so you could use this at 217 too.

## Implementation details

Very clear, other than the purpose of the thresholds at line 243 (as I mentioned above was also unclear at line 196).

Your description here of how you split the networks into representation layer and classification head is something that would have been helpful much earier. Maybe not this full amount of detail, but just that it's a somewhat arbitrary decision and nothing inherent about the division.

## Mitigation performance analysis

**This is the section that gives me the most concerns about this paper.**

You claim that "RNF... outperforms baseline methods..." However, these figures (Figure 3) do not convince me of this. The datapoints are averages across three runs. But, just looking at how non-smooth the fairness-accuracy curves are, and how close these curves are to each other, I am not sure averaging across three runs was sufficient to be able to be confident we are not observing noise from a small set of experimental runs. In Figure 3(a), under equality-of-odds analysis, EOR might beat RNF, and RNF-Gt looks similar to EOR. In Figure 3(b), under demographic-parity analysis, RNF and RNF-GT look similar to EOR. In Figure 3(b), under equality-of-odds analysis, EOR might be beating RNF, EOR looks similar to RNF-GT, and RNF looks similar to adversarial. Figure 3(c), under demographic-parity analysis, this figure does look like it is consistent with your conclusions. Figure 3(c), under equality-of-odds analysis, EOR may be better than RNF, EOR might be similar to RNF-GT. Regardless, it is hard for me to draw conclusions from this small set of 3-run experiments.

## Classification Head Analysis

This is a creative way to demonstrate correlation between fairness-sensitive information and information used by the classification head.

## Hyperparameter and ablation analysis

I was really hoping you would assess the effect of where you draw the line between encoder and classification head. Great to see this analysis here.

## Representation Neutralization with Debiased Encoder

This reports only a single point on the fairness-accuracy curve. Were these also averages across three runs? A single point on the curve isn't enough to be able to draw a conclusion.

# Revised Recommendation

The author(s) have provided very helpful responses:

* they "will rephrase the claim that 'RNF… outperform baseline methods…' in the revised manuscript to instead highlight the resource benefits of RNF over the baseline methods with similar or better performance in some cases,"
* while not going as far as statistical significance tests, they have provided more information that allows me to better assess whether the observations are mere noise or true effects; they will average across more (10) runs and will include error bars in the figure
* they have agreed to re-organize some of the presentation to make more clear some of the points I had questions about

Based on these commitments, I have revised my recommendation to "7: good paper, accept."
limitations_and_societal_impact: Authors could point out that some anti-discrimination regimes require the *elimination* not the mere *reduction* of disparate outcomes in government decision-making.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors proposed a new technique to reduce the discrimination of DNN models by debiasing only the task-specific classification head of DNN models  (termed as Representation Neutralization for Fairness, RNF). Furthermore, to handle the cases where sensitive attribute annotations are not available, they leveraged a bias-amplified model to generate approximate annotations for sensitive attributes. They showed that the RNF effectively reduces the bias of DNN models with a minimal decrease in model performance.
main_review: The work is innovative as previous work on this topic mostly focused on learning debiased encoders, whereas the current approach focuses on learning debiased decoders. Also, existing methods require annotations of the sensitive attributes but the current method removes this constraint as it can generate the proxy sensitive attribute via a biased-amplification approach.

 The submission is technically sound. The authors not only demonstrated that their approach mitigates bias (3.2) but also conducted thorough ablation studies to confirm the necessity of different model components (3.4). The analysis of the classification head further illuminates the effect of debiasing the classification head. The author found that RNF dramatically reduced the cosine similarity between positive predictions and sensitive attributes (e.g. male) compared to the Vanilla model by adjusting the decision boundary. I do have a question related to the results: since there's a tradeoff between accuracy and fairness, I wonder if the authors could compute a single metric combining accuracy and fairness (e.g., just like how f1-score takes account of both precision and recall) so that we could compare multiple models more directly.

The submission is clearly written and well organized. The results are important in the sense that it works complementary to the debiased encoder. When combined with debiased encoders, the model performs better than either of them alone in terms of the fairness metrics albeit with some loss in task performance. The authors may think more about how to improve fairness without the performance drop.

-----------

Updated: I've read the authors' responses to my review as well as the ones to the other reviewers'. I think their responses addressed my questions. I appreciate the clarifications on the single metric and the trade-off between fairness and performance. I will keep my score unchanged as it reflects my best judgment of the work.
limitations_and_societal_impact: There's no foreseeable negative societal impact of this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an unfairness mitigation technique for DNN models called Representation Neutralization for Fairness (RNF), which debiases the classification head without touching the possibly-biased representations. The idea is to first train a teacher network for the representations and then a representation neutralization network that trains on neutralized features (random combinations of samples with the same label, but different sensitive attribute values). Another contribution is to generate proxy annotations for sensitive attributes in case they are not available. Experiments show that RNF outperforms several baselines in terms of accuracy and fairness.
main_review: Overall, the paper proposes an intriguing approach for unfairness mitigation by only debiasing the classification head. The idea is simple, yet effective. However, some critical comparisons with other methods seem to be missing.

Strong points:

* Simple and effective approach for debiasing without touching the representations.

* Adapting Mixup techniques for generating neutralized features is a neat idea.

* The proposed method can be combined with any debiased model to further improve fairness.

* Experimental results look promising on multiple datasets and prominent fairness measures, even when annotations for sensitive attributes are not available.

* The paper is well written.

Weak points:

* During representation neutralization, mixing random samples with different sensitive attributes seems rather ad-hoc. Have the authors tried limiting the samples to mix, say by distance? If that does not work, why? 

* The key assumption for generating proxy annotations is questionable. A biased model may be over-confident for any subgroup of the data, and there is no guarantee this happens to be one of the sensitive groups. The authors should experimentally verify that the proxy annotations really align with sensitive attribute annotations. That is, do the samples with proxy annotations 0 (1) really have gender male (female)?

* Some critical comparisons seems to be missing in the experiments. 

     - RNF seems very close to post-processing unfairness mitigation techniques as it can be viewed as neutralization (post-processing) after training the teacher network. Hence, there should be an experimental comparison with Hardt's method, which the authors already cite ([27]). 

     - The authors cite the relevant work Fair mixup [40], but do not make any comparisons with it either. 

     - Since the authors compare with EOR (direct optimization of equalized odds), there should also be a comparison with Zafar's method, which optimizes demographic parity. Zafar et al., "Fairness Constraints: Mechanisms for Fair Classification", AISTATS 2017.

* Although the authors claim that the proposed method can be extended to non-binary sensitive attributes, some clarifications would be helpful.

     - When generating proxy annotations, one demographic group has over-confident predictions, and the alternative group has less-confident ones. If there are more than two sensitive groups, how does RNF assign proxy annotations among groups?

     - Smoothening neutralization: In Eq. (3), how can we assign lambda values to multiple sensitive groups?

* Optionally, it would be nice to also use the ProPublica COMPAS dataset in the experiments.

======

The author response addresses my main concerns above. Thanks for the clarifications and new experiments.

limitations_and_societal_impact: The limitations of the work are sufficiently discussed in the supplementary, but should probably appear in the body of the paper. There is no discussion on potential negative societal impacts. One possible discussion is whether optimizing fairness may reduce accuracy too much, which can be viewed as a negative outcome.
review_title: null

Meta-review: The paper proposes a method for debiasing classifiers. Firstly, a biased classifier is trained as normal. Then, a new classification head is trained for that encoder, that interpolates using combined representations from examples with the same label but different sensitive attributes (similarly to mix up training). This approach allows a classifier that is less biased, whilst retaining most of the performance of the original. This is clearly an important problem, and the reviewers generally found the proposed approach to be creative, soundly evaluated and well written. Therefore, I recommend acceptance.
Decision: Accept (Poster)