Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper shows how research on methods to enforce demographic parity can be more predictive of protected attributes. Considering the legal implications of disparate treatment, the work merges a second classifier that explicitly predicts the protected attribute. The addition of this second classifier helps further identify individuals systematically disadvantaged.
strengths_and_weaknesses: The authors provide a simple yet novel addition to the fairness literature. And the work is properly positioned in the literature.
questions: The legal discussion seems almost disjointed from the other work. It seems like the motivation for the work stems from the legal discussion, and while the legal implications nicely complement the results, how does the approach presented contribute to the presented legal analysis?
limitations: The authors discuss limitations of scope and only having tested on a few datasets.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper is largely an observational paper, with a particular implementation of a two-headed network, which conducts experiments related to the fairness of classifiers. Specifically, the endeavor to conceptualize whether “fair” classification systems may violate US equal protection laws, like protection against disparate treatment or impact. These authors look at a network trained with a subgroup performance difference regularizer and observe that the more a fairness metric is enforced, the easier it is for the protected attribute of an intermediate representation of a data point to be predicted. They propose a two-headed classification task to create classifiers which adhere to some demographic parity constraints. Consequently, they believe that a fair model can be approximated by their approach which leads them to conclude that fair models exhibit disparate treatment and thus may violate US equal protection laws.

strengths_and_weaknesses: Strengths
1. The topics very important the the authors take an interesting and thought-provoking position on the legality of these approaches.
2. Section 4 proposes and demonstrates an approach with compelling results. 
3. The paper is written clearly and the authors take a methodical approach to arguing their conclusions. 

Weaknesses
1. The literature review in this paper should be improved. For example, since the authors are only discussing only some of the algorithmic approaches to address mistreatment in classification, the authors should broader place the regularization and preprocessing approaches within the larger algorithmic fairness work. Additionally, the authors do not describe other two-headed approaches to fair classification like (https://arxiv.org/pdf/1707.00075.pdf, http://proceedings.mlr.press/v80/madras18a/madras18a.pdf)
2. I have some methodological concerns which I have questions for in the next section. Once I have clarity on these, I’ll be better able to evaluate the conclusions of your paper. 
3. It appears that this approach is only looking at binary protected attributes and would be hard to extend to multi-category labels.
4. The claims made in the beginning of the paper about “extensive results” could be reasonably argued for Section 3, but become increasingly less convincing in Sections 4 and 5. For instance, the conclusion drawn in Section 5.1 is for just one protected attribute and predicted variable pair. 

questions: 1. I see the authors include as a reference this paper (https://arxiv.org/pdf/1707.00075.pdf, http://proceedings.mlr.press/v80/madras18a/madras18a.pdf) but do not discuss it in their main text. 
2. Why do the authors omit any codebase or output data in their submission?
3. Can the authors describe how \lambda is used? It is merely described as a fairness parameter in the paper and the appendix does not provide any more clarity. Is this like the Lagrangian so minimize the loss + \lambda \hat{R}_{DP}(f)?
4. In Section 3, why do you care about the rank of the \lambdas and the overall accuracy? i.e., why perform a rank sum test (ordinality) and not just a Spearman’s rank test? 
5. I’m further not convinced by Section 3 given the very large class imbalances that generally exist in these protected attribute categories. Can you speak to how these class imbalances impact your results? Two other cofounders are (1) the overall performance of the ResNet model, and (2) the co-occurance of the protected and the predicted variables. Can you update your statistical analysis to account for these through regression or anova?
6. I’d appreciate more clarity in the main body of the paper on the definition of g in Lines 150-151. I feel this is an important enough topic the reader shouldn’t have to search in the Appendix for this.
7. Can you speak about the sensitivity of the a_1, a_2, b_1, and b_2 values that are chosen by the logistic regression in Section 5.1? I’m concerned about how these values would fare under new datapoint. 

Typo: penultimate line of Figure 4’s caption: “iMale”. There may be others.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper observes that the fair training model may use sensitive group information in the internal state of the network. Such a phenomenon may be undesirable, as fair training fundamentally aims to not use group information. To support this main claim, the paper gives several interesting observations; for example, the fairer model can predict the groups more accurately. Also, the paper proposes a two-headed model training strategy that can demonstrate the disparate treatment of fair training. The main experimental results are based on fairness regularizer and massaging techniques, which represent fairness in-processing and pre-processing, respectively. The paper also connects this observation with a meaning of fairness from the legality point of view. 
strengths_and_weaknesses: <Strengths>

- The main observation that the fair training model may rather use sensitive group information is very interesting. Figures 1 and 2 show these results well. As many fair algorithms have been proposed, I believe such an underlying analysis is now important to further extend model fairness literature. 
- The idea of the two-headed model is simple and intuitive, and it is reasonable to illustrate the paper’s claims. Using the unfair model to explain the undesirable property of the fair algorithms gives new viewpoints.

<Weaknesses>

Although I appreciate their observations and enjoyed reading this paper, there are also several weak aspects in the paper.

- The paper contains various phenomena, but the in-depth discussions to give intuitions on the phenomena are not enough. Detailed questions are in Q1 in the next section.
- Several details in the two-headed model are not clearly explained. Detailed questions are in Q2 in the next section. 

questions: Q1: As briefly mentioned in the previous section, the in-depth discussion on the observations seems not enough. 

- First, it would be very helpful if the paper gives any intuition on why some fair training models use group information to achieve fairness. 
- Also, regularizer and massaging are quite different techniques, but there is no explanation on 1) how similar phenomena are observed in these different techniques or 2) if there is any different trend between the two methods. Since regularizer is an in-processing method, it will actively guide the model to consider fairness during the training. Thus, it is a bit imaginable why the regularized method may result in undesirable information usage. However, massaging is a pre-processing method, so it is not clear why the fixed training data make a similar effect on the model. It would be great if the paper clearly explain such details.

Q2: It seems various details on the two-headed model are missing. 

- It would be helpful if the paper gives a rationale for the loss function design. For example, the loss function in Section 4 contains two separate terms (i.e., L_bce and g loss), but there is no tuning knob to adjust the impact between the two terms. Since the roles of the two terms are completely different, I was wondering whether we do not need any tuning knob in that loss. 

- Also, why the two-headed model is a post-processing method? I understand that the method can adjust the a_1 and a_2 values of the trained model to achieve the target (fairness) performances. However, it seems this method initially needs model training by using the loss in Section 4. Thus, I believe that the paper needs to clarify the category of their methods. I am not saying that the paper should change the category, but a more careful representation is needed.

limitations: The paper discusses the limitations and potential negative societal impacts.
review_title: null

Meta-review: metareview: This paper identifies and addresses an interesting phenomenon in fair ML, where "fairer" networks are in turn more discriminatory in a different sense.  The paper is an insightful read and, while I don't agree that prior literature has bee done due diligence (please see reviews/discussion), I do believe the work stands alone as a solid contribution.  I would ask the authors to continue to pay attention to Reviewer qBCV, who unfortunately did not participate in the discussion session, but with whom I do agree; many of their points are valid and should be addressed either in the camera ready for this conference or in a future version of the work.
confidence: Less certain
Decision: Accept