Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The goal of the paper is to design mechanisms to explain the unfairness in the outcomes of a ML model and propose methods to mitigate unfairness. The paper uses the Shapley value framework. The main idea is to alter the prediction function so that instead of providing the classification score, an "unfairness" score is returned. An out of the box application of the Shapley value framework on this unfairness score now returns the "unfairness" feature attribution. These feature attributions can be used to explain the unfairness of the model. The paper then proposes to learn a linear perturbation, which when combined with the additive property of Shapley framework results in updated "unfairness" attributions.

While the paper tackles an interesting and timely problem, it lacks clarity at several important points. Additionally, some of the claims seem to be a little overreaching. The experimental evaluation can also use some more thoughtful analysis. Please see detailed comments and suggestions for improvement below:

1- First of all, I would highly recommend setting aside a separate (sub)section to describe the setup. Right now, the details about whether f(x) is a real number of a probability, whether y is {0,1} or {-1,1}, or what "a" is are scattered across the text, reducing the readability of the paper.

2- Going from Eq. 2 to Eq. 5, the paper somehow switches from prediction probabilities to accuracies. Shouldn't there be a threshold function to convert the probabilities into predictions? Or is the paper only focusing on randomized classifiers? If only the randomized classifiers are the focus, if/how does it limit the extensibility of the proposed technique?

3- Does it make any difference for the explanations when the probabilities are high non-calibrated (https://arxiv.org/pdf/1706.04599.pdf)?

4- What was the accuracy/generalizability of this $\delta$ in the experiments?

5- After reading Section 2.2, a reader would think: Why even train the linear perturbation? Why not simply train a separate fair model (from the same model class) and get the Shapley predictions of that second model? I think the main idea here is that the paper tries to leverage the Shapley additive property here. How precisely does the additive property help?

6- The results in Figure 1 are interesting, but not very surprising. Specifically, given that one achieves roughly the same accuracy on Adult data regardless of whether the sensitive feature is used or not, it would have been more interesting (and insightful) to show the Shapley plots for the cases when the sensitive feature is indeed not used for classification.

7- The claim made in the paper that fairness is achieved at "no loss of accuracy" is perhaps a bit too bold given that there are well-known results about fairness-accuracy tradeoffs (e.g., https://arxiv.org/abs/1701.08230 and https://arxiv.org/abs/1609.05807).

--------------------
Post rebuttal comments:

Thanks to the authors for the helpful comments -- they indeed help clarify some of the confusions. As a result, I have upgraded my score. However, I am still leaning towards reject because I feel there are still open questions that may hinder the adaptability of the proposed method in the real world. Specifically, given the response to question 3 above, it would help to know what are the real world situations where one uses a randomized classifier and is still interested in model interpretability (the two seem to be at odds with each other as randomness inherently seems a bit arbitrary). Another concern that I have is about Eq. 3 in the paper: Why is the sum function chosen to compute global explanations from local ones? There seem to be multiple ways to do this (e.g., median, sum of absolute values) and it would help to know what are the (dis)advantages of not using other aggregation functions.
review_title: Important topic, but lacks calrity

final_score: rating: 6: Marginally above acceptance threshold
review_content: Quality
- This paper has defined a well-scoped problem: explaining the unfairness of an ML model in terms of the features used, as well as explaining an additive perturbation that will make the model more “fair”.
- This paper acknowledges that definitions of fairness can be fraught and should be applied with care and domain knowledge. Therefore it has proposed an “explanation” procedure that works with multiple statistical definitions of fairness. This is a strength of the work.


Clarity
- Clarity of the “section 2” could be improved. What is the support of $a$, $y$ and $f(x)$?

Originality
- There has not been a lot of work on fairness and explainability. As one of the first forays into these questions showing a positive result (as claimed in introduction, and also to my best knowledge), I think the paper is sufficiently original. Even though it is adapting a well-known construct (Shapley value), its originality also lies in connecting explanations to work on post-hoc fairness corrections (section 2.2).

Significance
- As stated above, I think the paper makes a significant contribution to research on fairness and explainability by developing some basic tools to help with “explaining” group fairness issues with ML models. 
- That being said, I think one must be careful to claim that the explanations offered by shapley value suggest any particular intervention. For example, it is not true that the features that contribute most to unfairness should be always be removed. I think a careful discussion on how to use the insights from the shapley values in practice, or some open questions regarding the interpretation of the shapley values would improve this paper.
review_title: A good contribution connecting the fields of explainability and fairness.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a method for feature attribution for fairness of the classifier.  They also demonstrate a feature augmentation technique to mitigate unfairness.  They connect their attribution method to to the augmentation technique and demonstrate that their method can attribute the necessary changes to achieve fairness. They evaluate their approach on a few tabular data sets.

Introduction + Methods

- I’m slightly confused about equation (2), is the correct interpretation that y is the label we’re explaining? So, say, if we’re explaining the -1 class and f(x) is 0.25, then f_y(x) yields 0.75? 
- I think some clarity on the notation would be useful in general.  Is y the ground truth label from the data? That seems to be described this way in section 1.  However, in equation (2) is seems to refer to the label we’re selecting to explain, which feels slightly different.  I think some of this notation could be cleared up a bit.
- “Splicing disjoint sets of features” is this common terminology? I think what this is trying to say is that this step is where we substitute values from the data into a point to create a perturbation.  Let me know if I am correct.
- For the description for equation (5): “…the expected accuracy for a model which samples a predicted label according to the predicted probability” what does this mean? The notation in equation (5) looks like it’s the expected prediction of the model, where is the relation to accuracy here? I’m not quite seeing this.
-  I think that equation (5) could be clarified in general.  It would be good to explicitly state this significance of this property.  (It’s stated clearly for equation 9, and stating it explicitly here would help readers)
- Is g a value function? If so, it’d be useful to state this leading up to the equation. Also, we’ve introduced the term “a” at this point which refers to the protected attribute (going back to the introduction).  What values can a take on? Are we assuming $a \in \{ 0, 1\}? It would be good to clarify this. Last what is p(a) meant to refer to? Is this the overall proportion of individuals with a certain protected attribute? If so, why do we need this term?  
- “The linearity axiom of the Shapley values guarantees that the fairness Shapley values of a linear ensemble of models are the corresponding linear combination of Shapley values of the underlying models” — it would be good to clarify this sentence as it’s slightly confusing right now.
- For equation (10) assuming that f Is a linear model, is $\delta_\theta$ meant to be a vector of values added to each of it’s coefficients?
- Is is slightly contradictory that in equation (10) $\delta_\theta$ is a perturbation (and can be explicitly added to f) but then becomes a function in equation (11)? I get what is being said but perhaps the notation could be improved here.

Experiments
- Figure 1 is hard to read, can you make the labels bigger for each graph. Also, it might be good to place the y axis values on each graph so that its easier to see the scale.  I see what the graph is saying, but this would make it much easier to figure out.
- I think figure 1 is a nice visualization overall and gets the point across well.
- I’m not fully understanding the point being made by section 3.2.. Is the idea that we know this suppressed model doesn’t rely on sex at all, so the fairness Shapley values shouldn’t say it does? 

General Questions + Comments:
- I liked the ideas presented by this paper overall.  The idea of using Shapley values to provide feature attributions for fairness is interesting — particularly the applications for explaining the differences between two models, one of which has been corrected for fairness.
- I do feel however that the paper can be improved in a number of places in order to strengthen the work as mentioned in my comments.  It would be very useful if the authors provided answers to some of my questions and took some of the comments into consideration for a revision. Particularly, I'm somewhat confused about the contribution of section 3.2 and would appreciate clarification.
- One related question is that the authors demonstrate their method on the suppression techniques from Dimanov et al.  Would their techniques help at all with the related methods (more so phrased as attacks) from https://arxiv.org/abs/1911.02508?
- Overall, my sentiments right now are borderline, leading towards reject. There’s a number of points which could warrant further clarification right now in the paper.  
review_title: Review 

Meta-review: All the reviewers found interesting the use of Shapley values to provide feature attributions for fairness, however, the reviewers brought up a number of issues, particularly in terms of presentation and clarity. While the authors' responses did clarify some of these concerns, this was not enough for the reviewers to broadly support acceptance.
Decision: Reject