Explainability for fair machine learning
Authors: Tom Begley
Tobias Schwedes
Christopher Frye
Ilya Feige
Venue:  (2021)
Keywords: explainability
fairness
Shapley

Abstract: As the decisions made or influenced by machine learning models increasingly impact our lives, it is crucial to detect, understand, and mitigate unfairness. But even simply determining what ``unfairness'' should mean in a given context is non-trivial: there are many competing definitions, and choosing between them often requires a deep understanding of the underlying task. It is thus tempting to use model explainability to gain insights into model fairness, however existing explainability tools do not reliably indicate whether a model is indeed fair. In this work we present a new approach to explaining fairness in machine learning, based on the Shapley value paradigm. Our fairness explanations attribute a model's overall unfairness to individual input features, even in cases where the model does not operate on sensitive attributes directly. Moreover, motivated by the linearity of Shapley explainability, we propose a meta algorithm for applying existing training-time fairness interventions, wherein one trains a perturbation to the original model, rather than a new model entirely. By explaining the original model, the perturbation, and the fair-corrected model, we gain insight into the accuracy-fairness trade-off that is being made by the intervention. We further show that this meta algorithm enjoys both flexibility and stability benefits with no loss in performance.

EXPLAINABILITY FOR FAIR MACHINE LEARNING
Anonymous authors Paper under doublc-blind review
ABSTRACT
As the decisions made or influenced by machine learning models increasingly impact our lives, it is crucial to detect, understand, and mitigate unfairness. But even simply determining what "unfairness" should mean a given context is non-trivial: there are many competing definitions, and choosing between them often requires a deep understanding of the underlying task. It is thus tempting to use model explainability to gain insights into model faimess, however existing explainability tools do not reliably indicate whether a model is indeed fair. In this work we present new approach explaining fairesss in machine learning, based on the Shapley value paradigm. Our fairness explanations attribute a model's 's overall unfairness to individual input features, even in cases where the model does not operate on sensitive attributes directly. Moreover, motivated by the linearity of Shapley explainability, we propose a meta algorithm for applying existing training- time fairness interventions, wherein one trains a perturbation to the original model, rather than a new model entirely. By explaining the original model, the perturbation, and the fair-corrected model, we gain insight into the accuracy-fairnes that is being made by the intervention. We further show that this meta algorithm enjoys both flexibility and stability benefits with no loss in performance.
trade-off
INTRODUCTION
Machine learning has repeatedly demonstrated astonishing predictive power due to its capacity to learn complex relationships from data. However, it well known that machine learning models risk
perpetuating or even exacerbating unfair biases learnt from historical data (Barocas & Selbsi [0166] Bolukbasi et al. 2016; Caliskan et al. [017]] Lum & Isaac, 2016). As such models are increasingly used for decisions that impact our lives, we are compelled to ensure those decisions are made fairly,
In the pursuit of training a fair model, one encounters the immediate challenge of how fairness should be defined. There exist wide variety of definitions of fairness some based on statistical measures, others on causal reasoning, some imposing constraints on group outcomes, others at the individual level and each notion is often incompatible with its alternatives (Berk et al. 2018 Corbett-Davee ct al. 2017: Kleinberg et al. 2017 Lipton et al. (0118) Pleiss et al.) (017). Deciding which measure of fairness impose thus requires extensive contextual understanding and domain knowledge. Further still, one should understand the downstream consequences of fairness intervention before imposing
it on the model's lss decisions (Hu al. 2019 Liu et al. 2018).
To help understand whether a model is making fair decisions, and choose an appropriate notion of fairness, one might be tempted to turn to model explainability techniques. Unfortunately, it has been shown that many standard explanation methods can be manipulated to suppress the reported importance of the protected attribute without substantially changing the output of the model (Dimanov et al. 2020). Consequently such explanations are poorly suited to assessing or quantifying unfairness.
In this work, we introduce new explainability methods for fairness based on the Shapley value framework for model explainability (Datta et al. 2016 Strumbelj & Kononenko 2010 Lipovetsky & Conklin 2001 Lundberg & Lee| 2017 Strumbelj & Kononenko 2014). We consider a broad set of widely applied group-fairness criteria and propose a unified approach to explaining unfairness within any one of them. This of fairness criteria includes demographic parity, equalised odds, equal opportumity and conditional demographic parity see Sec. 2.T. We show that for each of these definitions it is possible choose Shapley value functions which capture the overall unfairness in the model, and attribute it to individual features. We also show that because the fairness Shapley values collectively must sum to the chosen fairness metric, we cannot hide unfairness by manipulating the explanations of individual features, thereby overcoming the problems with accuracy- based explanations observed by Dimanov et al. (2020).

---

Motivated by the attractive linearity properties of Shapley value explanations, we also introduce a meta algorithm for training fair model. Rather than learning fair model directly, we propose instead learning an additive correction to an existing unfair model. We use training-time fairness algorithms train the correction, thereby ensuring the corrected model fair. We show that this approach gives new perspectives helpful for understanding fairness, benefits from greater flexibility due to modess-gnos and enjoys improved stability, all while maintaining the performance of the chosen training-time time algorithm.
2 EXPLAINABLE FAIRNESS
In this section we give an overview of the Shapley value paradigm for machine learning explainability. and show how it can be adapted to explain fairness. Motivated by the axiomatic properties of Shapley values, we also introduce a meta algorithm for applying training-time faimesss algorithms to a perturbation rather than a fresh model, giving us multiple perspectives on fairness.
2. BACKGROUND AND NOTATION
We consider fairness in the context of supervised classification, where the data consists of triples (x, a, y). where x E X' are the features, a E .A is a protected attribute (e.g. sex or race), and y E y is the target. We allow, but do not require, be component of æ. The task is train a model to predict from æ while avoiding unfair discrimination with respect to a. We assume A and y are both finite, discrete sets.
Our fairness explanations apply to any definition that can be formulated as (conditional) independence of the model output and the protected attribute. This includes demographic parity (Calders al.)
2009; Feldman et al. 2015 Kamiran & Calders [2012] Zafar et al. 2017]. conditional demographic
parity (Corbett-Davve Davies et al. 2017). and equalised odds and equal opportunity (Hardt al., 2016).
Definition 1. DEMOGRAPHIE PARITY The model I satisfies demographic parity if (xx is independent of a, or equivalently P([(x) = Tla)= = P(f(x) = V) for all ÿj E y and a E A.
Definition 2. CONDITIONAL DEMOGRAPHIE PARITY The model J satisfies conditional demo- graphic parity if with respect to of legitimate risk factors {uI..... 2n } if fx) is independent of a conditional on the Un, or equivalently ((f = y a, U1 UI...., (fx) = gl......, Un) for all i y and a E A.
Definition 3. EQUALISED ODDS The model f satisfies equalised odds if ((x is independent of conditional on y, or equivalently P(f(x) = j(a,y) P(f(=) = Gly) for all g. y E y and A.
If {0, 1) is binary, then equalised odds implies that the true and false positive rates each protected group should agree. Furthermore, assuming that y = corresponds to the "privelidged outcome", we can define equal opportunity as follows
Definition 4. EQUAL OPPORTUNITY The model f satisfies equal opportunity if f(x) is indepen- dent of a conditional on = 1, or equivalemtly P(f(x) = 1 a, y = 1) = P(f(x) = My = 1) for all ÿ E y and A.
2.2 ADAPTING EXPLAINARILTY TO FAIRNESS
Fairness in decision making- automated or not - is a subtle topic. Choosing an appropriate definition of fairness requires both context and domain knowledge. In seeking to improve our understanding of the problem, we might be tempted to use model explainability methods. However Dimanov et al. (2020) show that such methods are poorly suited for understanding fairness. In particular we should not try to quantify unfairness by looking at the feature importance of the protected attribute, such measures can be easily manipulated. Part of the problem that most explainability methods attempt to determine which features important contibutors to the model's accuracy. We seek to introduce explanations that instead determine which features contributed to unfairness in the model.
Toward this end, we work within the Shapley value paradigm, which is widely used as model- agnostic and theoretically principled approach to model explainability (Datta et al. 22066] Strumbelj & Kononenko 2010 Lipovetsky & Conklin 2001 Lundberg & Lee, 2017] Strumbelj & Kononenko 2014). We will first review the application of Shapley values to explaining model accuracy, then show how this can be adapted to explaining model unfairness. See Frye et al. (2020b) for a detailed analysis of the axiomatic foundations of Shapley values in the context of model explainability.
2

---

EXPLAINING MODEL ACCURACY
Shapley values provide method from cooperative game theory to attribute value to the individual players on a team N {I.....m) n (Shapley 1953). If the team earns a total value v(N), the Shapley value (i) attributes portion to player according to:
[s |S| - 1)! n!
(.(1) = Σ
[u(su =(5)
(I)
SCNVIO
Here eSS) is the value coalition S of players generates when playing on their own. The Shapley value (i) is thus the average marginal contribution that player i makes upon joining a coalition, averaged over all coalitions and all orders in which those coalitions can form.
To apply Shapley values to model explainability, one interprets the input features as the players of the game and defines appropriate value function (e.g. the model's output) to insert into Eq. (T) Let (z) denote the predicted probability that x belongs to class We define a value function by marginalising over out-of-coalition features:
(=) = Ep(er) (1 L [(s)s)
where sy is the set of feature values with indices in S. s Lie'' is a new data point formed by filling the missing features in is with values from z', and where p(z') represents the data distribution One computes local Shapley values (z)(x) (i) by inserting (x) into Eq. (1). These can be aggregated to obtain a global explanation of the model that maintains the underlying Shapley axioms:
f Epta.) [or,()))
where p(z, y) is the joint distribution from which the labelled data is sampled, and so fx) is the probability the model assigns to the true outcome. Aggregating global Shapley values in this way provides the desirable property that
-
- Eple') [x)(
(4)
The first term the right-hand side is the average probability assigned to the true outcome. It can be interpreted as the expected accuracy of a randomised classifier that samples a predicted label according the probabilities predicted by the model. The second is an offset term corresponding to the expected accuracy if we were to sample predicted label at random according to the average prediction probabilities for each class. This offiset not attributable to any of the features and is related to the class balance. We remark that randomised classifiers are often used when training fair models, for example by the reductions approach of of[Agal et al. (2018), so expected accuracy coincides with commonly used deterministic accuracy. More generally. the expected accuracy is closely related to usual notions of accuracy, but additionally captures the confidence with which the classifier makes predictions.
EXPLAINING MODEL FAIRNESS
To explain fairness in model's decisions, we detine a new value function that captures faimesss then proceed as in Sec. 2.2 To simplify the exposition, we assume that y = A {0, 1}. See App. A.2 for straightforward generalisations to other notions of faimess, multiclass classification, and non-binary protected attribute. Without loss of generality we assume f f(x) is scalar and corresponds to the predicted probability that 1. To begin, let
gx(x)= = f(x)- (-1)a
(5)
p(a)
where p(a) denotes the proportion of individuals with protected attribute a, and in which the sign of (x) is controlled by the true value of a, whether or not the a-component in is altered. The value function on coalitions defined through marginalisation
Uga. (x) (S) = Ep(er) 9aa (as U ss)
(6)
1 more natural value function would marginalise over the conditional data distribution p(ø'x s(r)) as discussed in Aas et al. (2019) and Frye et al, (2020a). Eq. 2 is used here for clarity of exposition as it is more commonly found in the literature. Thii marginalisation could also be performed conditionally, as discussed in footnote
3

---

Next, we insert (x) into Eq. 11 to obtain local Shapley values (o)( Finally, we obtain the corresponding global Shapley values through aggregation:
( (x) (s)
where p(z: a) is the joint distribution of features and protected attribute from which the data is sampled. The definitions above are motivated by the resulting sum rule for the global Shapley values:
- - f(x)
" -
This is the expected demographic parity difference for randomised classifier that samples labels according to the model's predicted probabilities. The global Shapley values Φ, (i) can thus be interpreted each feature's marginal contribution to the overall demographic parity difference. The local Shapley values are less relevant, as group notions of fairness are defined terms of aggregate statistics over each group, and local Shapley values only explain individual predictions.
2.3 LEARNING CORRECTIVE PERTURBATIONS
The Shapley linearity axiom guarantees that the fairesss Shapley values of a sum of models is the sum of the fairness Shapley values for each model. Motivated by this we consider the problem of learning an additive perturbation to an existing model in order to impose faimess. That is, given f we would like to learn a parametric perturbation dg such that
fo(x)) f(x) (x),
is fair. We would then gain three different perspectives from the explanations of each component: the original sources of unfairness from f. the correction or trade-off that was made from 8G, and how the corrected model makes fair predictions from fa. Moreover, the explanations are mutually consistent.
We continue to focus on binary classification here, see App. A. App.A.] for generalisation to the multiclass classification case. Since we require that the output of the corrected model should also be valid probability, we define óg via an auxiliary perturbation óø the logit scale:
8a (n x. a) =o (f=)((())) + (f 2.)) f(x).
(10)
where or (Ix) (1 exp(-x:)) We can apply a training-time fairness algorithm of our choice to fe in order learn the parameters of the auxiliary perturbation ōg. In Sec. 3 we show results for the algorithms of Agarwal et al. 2018) and Zhang et al. (2018) applied to the perturbed model.
Note that while in Eq. I0 write 6g as function of x, a and f(x). it is possible to use only a subset of these inputs. If we only gave óg access to a and f(x) then this becomes true post- processing algorithm. Alternatively we may require fair algorithm that does not have access the protected attribute at inference time, in which case we could omit a. We would then need to select training- time algorithm that is able make predictions without access to the protected attribute.
RESULTS
In this section we showease the fairness Shapley values introduced in Sec. We further show that the models corrected through learning perturbation suffer no loss in performance when compared to their unconstrained analogues, and enjoy increased flexibility and stability addition to the explainability benefits. Our experiments are conducted on the Adult dataset from the UCI Machine Learning Repository (Dua & Graff 2017). where the task is to predict whether an individual carns more than $50K per year based their demographics, and the COMPAS recidivism dataset (Larson) et af., 2016), where the task is to predict recidivism risk based on demographics. We use sex as the protected attribute in our experiments Adult, and race in our experiments on COMPAS. Full details for all experiments are in App. A.3
3.1 EXPLAINABILITY
To test the effectiveness of the fairness Shapley values, we train baseline-unfair model on the Adult dataset, then train correction by applying the adversarial approach of Zhang et al. (2018) to an additive perturbation as described in Sec. 2 From left to right Fig. I shows the accuracy and fairness

---

Unfair model
Perturbation
Corrected model fg
0.80
0.80
0.80
Accuracy
0.75
0.75
0.75
0.55
0.55
martal status capital edtcaation
relattonanigg
occure uretton hours
wwerk
day capatel
Wor cicaas
rative country Rital
Originsl mortel intatorutig
So hours par woat
worl class
Pece capiaa love
der eecuuitiion
Captoar gaki pntiictiio
Cornected ayle
capital gatn cntication
martal age
acc upcuuitiin
hours mint mint enteecapii
caplias fird
native country
carc
dae
Dem. partt difference
0.20
0.20
0.20
0.15
0.15
0.10
0.10
0.05
0.05
0.05
0.00
status hours par wear capital pain
age adtccation. capiral Foot
worfccias nation reintionshig Riial
Original modell
0.00
intatoortig occupatin
Hours por Pace Cyottal
age
prrkciiss capioal lous native educatoo
Corrected mintel
marital status per week Capital gain
0.00
edtecation foos iors
Chuntry occupation
reinteenshig Filal
natiee
Figure I: Explaining accuracy and unfairness (demographic parity) using Shapley values.
global Shapley values for the original model, perturbation, and corrected model respectively. We use waterfall plots emphasise that they sum to the expected model accuracy expected demographic parity difference respectively. The offset term for the accuracy Shapley values described in Sec. 2 is used offset the y-axis in the top row, fairness Shapley values have no offset term.
The accuracy Shapley values tell us which features contribute to the model accuracy, while the fairness Shapley values tell which features contribute to the demographic parity difference. Comparing the two can give us useful insights into the legitimacy of feature. For example we see that marital-statu is the largest individual contributor to accuracy for the unfair model, but also makes a major contribution to unfairness. Weighing up the effects of each feature on both accuracy and fairness helps us determine whether legitimate to include them in the model.
We gain different insights into the problem from each of the three models. The explanation of the original-unfair model shows us why an unconstrained model might learn to make predictions in an unfair way, the explanation of the corrected ffir model shows us how it is able to make fair predictions, and the explanation of the perturbation captures the trade-off that has been made to impose fairness.
For example, in Fig. we that the unfair model uses all features to improve accuracy, as we would expect from a model that has been trained only to maximise accuracy. The accuracy Shapley values of the perturbation show that some predictive power from sex and its proxies marital all-status and re relat i Lonship has been sacrificed, but the fairness Shapley values show that as a result these three features were able to be used to heavily reduce the demographic parity difference. The fairness Shapley values for the corrected model show that the effects ma r ttllssttus and other features are almost perfectly counter-balanced by sex and re lettiosship.
The complementary fairness and accuracy explanations, and the explanations offered from the three models have the potential to be powerful tool for model development and fine-tuning of fairesss. In App. A.5 we repeat this experiment with a model that does not have access to the protected attribute and compare results.
3.2 ROBUSTNESS OF FAIRNESS EXPLANATIONS
Dimanov et (2020) show that existing explainability methods are poorly suited answering questions about fairness. Specifically they show that by retraining model with an additional penalty term corresponding to the influence the protected attribute has on the output, the explanation for that feature can be suppressed without substantially affecting the model predictions. Hence the feature importance of the protected attribute a poor measure of fairness.
Fig. 2 shows the accuracy and fairness Shapley values for the baseline model of the previous section, as well as the same model after retraining to suppress the influence of sex on the output. We see

---

Accuracy Shapley values
0.04
Original model Suppressed
Shapley Values
0.02
0.01
marial status
capital gain
education
age
occupation
miationship
hours per week
say
capital loss
workclass
race
native country
Faimess Shapley values
Original model Suppressed
Shapley Value
0.06
0.04
0.02
marital status
hours week
capital gain
age
occupation
education
race
workclass
native country
miltionship
capital
Figure 2: Suppressing the importance of the sex feature.
that retraining has completely eliminated functional dependence the output on sex, however we should not conclude that the model now satisfies demographic parity. Indeed the two models predict the same outcome on more than 98% of the test data points, and demographic parity difference only improved fractionally, falling from 0.193 0.184
The fairness Shapley values give us more complete picture. Because the fairness Shapley values must sum the expected demographic parity difference, they will always capture unfairness if present. If we suppress unfairness attributed to any one feature, the attributions of other features must increase to compensate. The only way hide unfairness from the fairness Shapley values is to eliminate it.
Since we have manipulated f to no longer depend on sex, the faimesss Shapley values for sex has also been suppressed. However, the fairness Shapley values for re relatio iosshisp have increased, showing that the model has simply shifted focus to proxies of the protected attribute. We see the same effect in the accuracy Shapley values, however without the context its impact on fairness it's hard to know whether the increased accuracy attributable to relat ionshipp is legitimate.
and
3.3 LEARNT PERTURBATIONS
In this section we investigate properties of applying existing training-time algorithms to additive perturbations of Sec. 2.3 We find that applying the reductions approach of Agarwal et al. (2018) and the adversarial approach of Zhang et al. (0118) to a perturbation leads to no loss in performance compared applying the algorithms new model, but gain flexibility, stability and the benefit of being able to compare explanations between original model, perturbation and corrected model.
PERFORMANCE
We proposed the learnt perturbations in Sec. primarily due to their explainability benefits. It is important however that we verify that in doing so we have not compromised the performance of the algorithms. In order check this we use the adversarial approach of of et al. (2018) and the reductions approach of Agarwal et al. (2018) to impose both demographic parity and equalised odds

---

Table 1: Accuracy associated with decreasing demographic parity thresholds.
Accuracy [%)] demographic parity difference
Method
0.1
0.08
0.06
0.04
0.02
0.01
0.005
Agarwal et al.
84.71
84.32
83,94
83.82
83.29
83.29
Adult
Agarwal et al. perturbed
84,69
84.43
83.82
83.82
83.35
83.23
Zhang et
84.65
84.18
84.06
83,58
83.18
83.15
83.15
Zhang al. perturbed
84.74
84.48
83.78
83,61
83.14
82.99
82,96
Feldman al. (post)
84.69
84.35
84.12
83.67
83.32
83.30
83.01
COMPAS
Agarwal et al.
74.05
74,05
73.77
73.67
73.11
73.11
73.01
Agarwal et al. perturbed
74.24
74.24
73.86
73.86
73,20
72.73
72.73
Zhang ct al.
75.19
75.19
75.19
74.62
74.15
74.15
74.15
Zhang et al. perturbed
74.24
74.24
74.24
73.30
73.30
73.20
72.73
Feldman al. (post)
74.81
74.81
74.81
74.24
74.24
73.20
72.35
Table Accuracy associated with decreasing equalised odds thresholds.
Accuracy [%) at equalised odds difference
Method
0.1
0.08
0.06
0.04
0.02
0.01
0.005
Agarwal Agarwal et al. perturbed
85.32
85.32
85.13
84.30
84.18
Adult
85.43
85.43
85.31
84.34
84.21
Zhang et al.
85.13
85.04
85.04
84,86
84.33
75.43
75.43
7hang al. perturbed
85.26
85.11
85.06
84.97
84,23
83.53
Hardt et al.
82.77
82.77
82.77
82.77
82.77
82.77
82.77
COMPAS
Agarwal et
75.19
75.19
74.05
74.05
73.86
73.39
73.39
Agarwal et perturbed
74.43
74.43
74.43
73.86
73.39
73.39
73.39
Zhang
74.62
74.62
74.62
74.62
74.62
73.48
53.12
Zhang perturbed
74.34
74.34
74.34
74.34
73.48
72.44
72.44
Hardt al.
71.31
71.31
71.31
70.45
68.75
on the Adult and COMPAS datasets. In both cases we apply the algorithms
models, and also to perturbed models as described in Sec. Sec. We also compare to simpler post-processing approaches of Feldman et al. (2015) for demographic parity, and Hardt et al. (2016) for equalised odds.
Table I shows the results of our demographic parity experiments, and Table shows the results of the equalised odds experiments. We report for each method the highest observed test-set accuracy where the corresponding fairness metric did not exceed specified threshold. All methods are seen lose accuracy as the fairness requirement becomes more stringent, some methods performing better on one task than the other. The perturbed models perform competitively on both datasets and for both definitions of fairness, showing no significant reduction in accuracy at each fairness threshold over their training- time counterparts. For demographic parity we note that while the simple post-processing method of Feldman et al. 2015) performs competitively with the more sophisticated algorithms, it requires access to the protected attribute inference time, which is not true of the approaches of Agarwal et al. (2018) and Zhang et al. (2018) which only require access to the protected attribute during training.
FLEXIBILITY
The meta algorithm presented in See. Doffers flexibility benefits that are typical of post-processing algorithms while inheriting the performance benefits of the training-time algorithms it utilises to impose fairness. Specifically, we can be fully model-agnostic with respect to the original model, as any model structure or access requirements of the training-time algorithm apply only to the perturbation, and not the original model. Indeed we could learn perturbation available for inference over a network, or correct rules based classifier that does not lend itself to conventional optimisation techniques. Furthermore, if the original model is complex, option of training a lightweight perturbation to the complex model, and may not need to rerun an expensive training procedure.
model that is only
have the
STABILITY
Since accuracy and fairness objectives are typically in tension, some optimisation methods exhibit training-time instability where the classifier does not converge to a local minimum, but instead jumps

---

0.05 = dp 0.1
0,025 0.05
0.01 = 0.025
0.0 0.01
0.86
0.64
Accuracy
0.82
Adult
0.78
0.76
0.74
Perturbed Zhang et Mell st
Perturbed Zhang etal. Adel et
Perturbed Zhang et s. Ade etal.
Perturbed Zheng Adel et
0.75
Accuracy
SOPPAD
0.65
0.55
030
Parturbed Zhang
Perturbed
Perturbed
Perturbed Zhang Adel
Figure 3: Accuracy violin plots of experimental outcomes binned by achieved level of fairness.
between them. This is particularly true of the adversarial method of Zh al. (2018 where the tension between accuracy and fairness manifests itself as the tension between the model and the adversary. Indeed training-time stability is generic problem for adversarial methods, Adel et al. (2019) introduce variant of adversarial debiasing that attempts to address this instability.
Our perturbation-basee approach combats the instability introduced by this tension between compet- ing objectives: the fair-corrected model remains anchored near the original-unfan model, which has already found local-accuracy optimum. This is also evident in Table where observe that make the fairness threshold more stringent, classifiers trained with the method of Zhang et al. (2018) collapse to a constant predictor, predicting according class balance, whereas the perturbed model. optimised in the same way, does not exhibit the same behaviour and is able to learn a non- non-trivial classifier with low equalised odds difference.
We illustrate the improved stability by comparing the application of adversarial training to a perturba- tion, and the original approaches of Zhang et al. (2018) and Adel al. (2019). For each approach we train multiple models with variety of hyperparameter choices. The resulting outcomes show spread of model behaviour during training. We visualise the results in Fig. 3 which shows distribution of accuracy outcomes in every experiment, binned by the achieved level of fairness. The plots show that while the optimal accuracy in each bin is similar across the three methods, our perturbative approach generally has less variance and higher mean accuracy across the range of hyperparameters considered. We found similar results when imposing equalised odds (see App. thus showing clear training-time stability advantage of perturbation-base approach across multiple notions of fairness.
4 CONCLUSIONS
In this work we introduced new approach to explaining fairness in machine learning, based on the Shapley value paradigm. The fairness Shapley values are able to attribute unfairness in model's predictions to individual features, and complement the already well-used accuracy Shapley values. We also showed that because these cxplanations directly capture unfairness through their sum, they are robust to manipulation. Additionally, motivated by the lincarity properties enjoyed by Shaplcy values, introduced meta algorithm for learning fairness-imposte imposing perturbation to unfair model using existing training-time algorithms for fairness. This gives us new perspectives from the corresponding explanations:
able to explain the unfair model understand why model predictions might be at risk of being unfair, the corrected model to understand how it is able to make fair predictions, and the perturbation understand the trade-off that has been made between the two models. Moreover, we showed that training a perturbation of this kind results in no loss in performance compared training fresh model, but increases flexibility and can improve the training- time stability by anchoring the corrected model to local-accuracy optimum. We believe that our approach to explaining fairness will highly valuable to practitioners, and we hope will contribute to the development of fairer models, and a greater understanding of unfaimess in different machine learning problems.

---

REFERENCES
K. Aas, M. Jullum, and A. Loland. Explaining individual predictions when features are dependent: More accurate approximations to shapley values, 2019. arXiv:1903.10460
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving,
M. Isard, et al. Tensorflow: system for large-seal machine learning, In 12th USENIX Symposium on Operating Systems Design and Implementation, 2016.
T. Adel, I. Valera, Z. Ghahramani, and A. Weller. One-network adversarial faimesss. In AAAI Conference on Artificial Intelligence, 2019.
A. Agarwal, A. Beygelzimer, M. Dudik, Langford, and H. Wallach. A reductions approach to fair classification. In International Conference Machine Learning, 2018.
S. Barocas and A. D. Selbst. Big data's disparate impact. California Law Review, 2016.
R. K. E. Bellamy, K. Dey, M. Hind, S. C. Hoffman, S. Houde, K. Kannan, P. Lohia, J. Martino, S. Mehta, A. Mojsilovic, et AI Fairness 360: an extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias, 2018. 8. arXiv:1810.01943
R. Berk, H. Heidari, S. Jabbari, M. Kearns, and A. Roth. Fairness in criminal justice risk assessments: the state the art. Sociological Methods & Research, 2018
Bird, M. Dudík, R. Edgar, B. Horn R. Lutz, V. Milan, M. Sameki, H. Wallach, and K. Walker. Fairleam: A toolkit for assessing and improv- ing fairness AI. Technical Report MSR-TR-2020-32. Microsoft, 2020. URL
https: www.w/wwwwww....o.....om.m.
T. Bolukbasi, K. Chang, J. Y. Zou, V. Saligrama, and A. T. Kalai. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems, 2016.
T. Calders, F. Kamiran, and M. Pechenizkiy. Building classifiers with independency constraints. In IEEE International Conference Data Mining Workshops, 2009.
A. Caliskan, J. Bryson, and A. Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 2017.
S. Corbett-Davies, E. Pierson, A. Feller, S. Goel, and A. Huq, Algorithmic decision making and the cost of fairness. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2017.
A. Datta, S. Sen, and Y. Zick. Algorithmic transparency via quantitative input influence: theory and experiments with learning systems. IEEE Symposium on Security and Privacy, 2016.
B. Dimanov, U. Bhatt, M. Jamnik, and A. Weller. You shouldn's trust me: Learning models which conceal unfairness from multiple explanation methods. In SafeAl @ AAAI, 2020.
D. Dua and C. Graff. UCI machine learning repository, 2017. attp: ics uci edu/ml
M. Feldman, S. A. Friedler, Moeller, Scheidegger, and S. Venkatasuurammanin. Certifying and removing disparate impact. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015.
C. Frye, D. de Mijolla, Cowton, M. Stanley, and Feige. Shapley-hased explainability on the data manifold, 2020a. arXiv:2006.01272.
C. Frye, C. Rowat, and I. Feige. Asymmetric shapley values: incorporating causal knowledge into -odel-agnostic explainability. In Advances Neural Information Processing Systems, 2020b
M. Hardt, Price, and N. Srebro. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems, 2016.
L. Hu, N. Immorlica, and J. W. Vaughan. The disparate effects of strategic manipulation. In Conference on Fairness, Accountability, and Transparency, 2019.

---

Kamiran and T. Calders. Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems, 2012.
Kleinberg, S. Mullainathan, and M. Raghavan. Inherent trade-ofts in the fair determination of risk scores, In Conference on Innovations in Theoretical Computer Science, 2017.
J. Larson, S. Mattu, L. Kirchner, and J. Angwin. How we analyzed the COMPAS recidivism algorithm. ProPublica (5 2016), 2016.
S. Lipovetsky and M. Conklin. Analysis of regression in game theory approach. Applied Stochastic Models in Business and Industry, 2001.
Z. Lipton, McAuley, and Chouldechova. Does mitigating ML's impact disparity require treatment disparity? In Advances in Neural Information Processing Systems, 2018.
L. T. Liu, S. Dean, E. Rolf, M. Simchowitz, and M. Hardt. Delayed impact of fair machine learning. In International Conference on Machine Learning, 2018.
K. Lum and W. Isaac. To predict and serve? Significance, 2016.
S. M. Lundherg and S. Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, 2017.
G. Pleiss, M. Raghavan, Wu, Kleinberg, and K. Q. Weinberger. On fairness and calibration. In Advances in Neural Information Processing Systems, 2017.
L. S. Shapley. A value for n-person games. Contributions to the Theory of Games, 1953.
E. Strumbelj and Kononenko. An efficient explanation of individual classifications using game theory. Journal of Machine Learning Research, 2010.
E. Štrumbelj and Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and Information Systems, 2014.
M. B. Zafar, I. Valera, M. G. Rodriguez, and K. P. Gummadi. Fairness constraints: mechanisms for fair classification. In International Conference on Artificial Intelligence and Statistics, 2017.
B. H. Zhang, B. Lemoine, and M. Mitchell. Mitigating unwanted biases with adversarial learning. In AAAI/ACM Conference on AI, Ethics, and Society, 2018
10

---

A APPENDIX
A.1 PERTURBING MULTICLASS CLASSIFIER
In Sec. 2 we introduced a meta algorithm for correcting an unfair model by learning a perturbation assuming binary classification task, and remarked that was straight-forward to generalise to multiclass classification. In this section we give the details.
We assume labels y E {I.....k) and model X [0, 1]th that maps each datapoint categorical probability distribution over the & classes. We denote by fa()) the probability that belongs class y. By assumption, we have Σ [z( for all x.
As in Eq. 9 we seek introduce perturbation óe produce fair model fo. As in Eq. (10) will define óg via an auxiliary perturbation on an unconstrained scale in order that we maintain valid probabilities. It is conventional use the softmax function convert k-vector of logits categorical probability distribution
exp(2;)
softmax
(11)
Ej exp(22)
The softmax function invariant under addition of a constant to all arguments. As a result, there is a translation invariance in the space of perturbations, which is not desirable for efficient optimisation. We circumvent this issue by pinning the first logit to zero. Specifically we define
[(s); := log(s;) log(s1) i ==
(12)
which as one easily verifies is right-inverse for softmax on the space of k-simplices, i.e. softmax(!(s)) = s for any k-simplex 8. With this in hand we define the perturbation analogously to Eq. (10) as follows
dq (f()),,, z, a) softmax(((f(z))) + (o( a)) - f(x)
(13)
where we require that (op) = 0. Thus we perturb only
of the logits for k-category classification problem. With the corrected model defined, we proceed precisely as before, applying training- time algorithm to fe in order to learn the parameters of 8g.
We note that this setup is easily seen to be equivalent to the formulation in Eq. 10 in terms of the sigmoid and logit functions when k 2. Indeed that case we have ((2); = for all z, and
1(=)2 = (2g(22) - log(z)) ) = log(22) - log(1 - 22) = x (22). Moreover by choice of 6n the first component of the argument of softmax the right hand side of Eq. (13) is and have softmax (0, z))2 = -1) =
=z)).
A.2 EXPLAINABILITY FOR OTHER FAIRNESS NOTIONS
To generalise the fairness explanations to other notions of fairness, we need only to adjust the weighting in Eq. (5) and the distribution in Eq. (7) over which we aggregate to produce global Shapley
values. We present the details below. in App. A. we assume multiclass classification task where (z) denotes the probability that belongs to class
EQUALISED ODDS
The fairness Shapley values presented Sec. 2 can also used to determine each feature's marginal contribution to equality of odds. Specifically we modify Eq. (5) create new functions
(a)
if aa 0
(x) =
Vyy f
(14)
if
P(a)00y)
for each y. As before we then create a value function by marginalising over out-of-coalition features
(x) (S) = Ep((') [ga.y(as [a/g]]
(15)
Shapley values are then calculated using Eq. 1 to average over all possible coalitions. To compute
global Shapley values we aggregate using Eq. but replacing p(x, a) with p(x, aly), from which obtain the following analogue of Eq.
Σ Da (i) da p ()()))) - dxp(alu,a + (x)
(16)
11

---

This corresponds to the (signed) difference in expected sensitivity to label y between the protected group a 0 and all other groups for classifier that samples predicted labels according to the predicted probabilities. In the case of binary classification, we can interpret each integral as the expected true positive and true negative rates when y and y respectively.
For equality of odds, we proceed exactly as above, but we are concerned only with the case where y
CONDITIONAL DEMOGRAPHIC PARITY
We proceed similarly to obtain explanations of fairness for conditional demographic parity by first defining
fy (a) P(a= Pn
if =
90,01 N0,01 U Un (x)
(17)
if o
P(az0 by ...
for each possible combination of resolving variables ei..... Un and value of y, then creating a value function by marginalising over out-of-coalinn features
Ug v o (2) (S) Ep(xr) [9agg.r...... (Is x'g)]
(18)
Shapley values are then calculated using Eq. to average over all possible coalitions. To compute global Shapley values aggregate using Eq. but replacing p(z, a) with p(z, ..., C,), from which we obtain the following analogue of Eq.
f dx do p(z|a 0, 1...... (,) - f p(rlaa + ....... Un (z)
Σ gv.vi
(i)
(19)
which represents the signed expected conditional demographic parity difference for the protected groups = 0 and + (conditional on Ul...., Un).
A.3 EXPERIMENTAL SETUP
In this section we give details on the datasets and experimental results presented in Sec. 3 All experiments were written in Python, we used TensorFlow to specify and train the models (Abadi al. 2016) and ran all of our experiments on AWS EC2 instances.
DATASETS
Adult
The Adult dataset (Dua & Graff 2017) contains 48842 rows with 14 features, and is pre-split into 66.7% training and 33.3% testing data. We drop the fnlwgt feature as it represents weights used in the original census application not relevant to the task hand, well the education--n feature since it a different representation of the existing education feature. This leads to 12 features, namely: age, workelass, education, marital-status. occupation, re relationship, race, sex, capital-gain, capital-lose hours-per-weet native-country. We drop all rows with missing values. When encoding the native-count feature we group into "other" all countries other than US and Mexico. We obtain a validation by randomly splitting off 20% of the remaining training data, leading final split of 53.3% training, 33.3 % testing and 13.3% validation data from total of 45,222 rows. We one-hot encode all categorical features, and standardise the continuous features by subtracting the train set mean and scaling by train set standard deviation.
COMPAS
We follow the pre-prove pre-processing steps used Larson et al. 2016) applied to the original datase which contains 7214 rows and 52 features: We remove from the arrest date to ensure we relate to the right offence, and only select individuals of African- American and Caucasian race. Some features contain information contained in others, relevant such as defendants' names. We hence select only a small subset of semantically most rele- vant features, namely: age, sex, race, curceechhhhreeeeeeeeeeee priors_counts c_jail_in, in,
with charge dates not within 30 days
not
12

---

1eiil_ttm__oo _out, juv_fel _fll__cunt,, sww_misd_coune juv_ paw_other_courte The final dataset contains 5278 rows and 10 features, which randomly split into 60% training, 20% val- idation and 20% testing data. The categorical variables are one-hot encoded, while the remaining features are standardised by subtracting the train mean and scaling by train standard deviation.
EXPERIMENTS
We here give the details of experimental setup for each of the methods consider in experiments in Sec. 3 As much as possible when comparing different methods we attempted to use similar model structures.
Explainability
To produce the results in Sec. 3.1 we trained a feedforward neural network (one hidden layer, 50 hidden units, and ReLU activations) on the Adult dataset. We then introduced a second neural network with the same architecture to serve as the perturbation. The original model is frozen and we apply the algorithm Zhang al. (2018) to train the permutation. We then calculate Shapley values for each of the original model, the perturbation and the corrected model by first sampling coalitions of features, then approximating Eq. (2) resp. Eq. (6) via Monte Carlo approximation with the empirical data distribution.
Robustness
To produce the results of Sec. 3.2 we start with the baseline model of the previous section. We then retrain it to suppress the importance of sex. We motivated by, but deviate slightly from, the work of Dimanov et al. (2020). Whereas they propose adding a penalty term corresponding to the gradient of the loss with respect sex, we instead use finite difference of the model output with respect to sex due to the discrete nature of the feature. Thus the modified loss becomes
1 Σ c(f(x)). f(. do 1)) f(a do(sex -O)
(20)
where L is the original cross-entropy loss, is hyperparameter controlling the trade-off between optimising the accuracy and minimising the effect of sex, and f(x; do(sex = j]) denotes, via slight abuse of notation, f evaluated on the data point æ; with the value for sex replaced with j. We train the baseline model for an additional 200 batches with a = 3. The resulting model agrees with the baseline on over 98.5% of the data, and has the same test set accuracy. We calculate Shapley values for the retrained model as before.
Performance
For Sec. 3.3 we trained single-layer feedforward neural network on each dataset to serve as a baseline. On the Adult dataset we use 50 hidden units, on COMPAS we used 32. In both cases we gave the network ReLU activations.
We use the training-time algorithms of Agarwal et al. (2018) ZZang et al. (2018) to train additional neural networks with the same architecture while imposing demographic parity and equalised odds. We used the implementation of the reductions approach of Agarwal et al. from the fairle mairlearn an library of Bird et al. (2020), with some superficial modifications to make it compatible with our TensorFlow models. We implemented the adversarial approach of Zhang al. (2018) ourselves. Our implementation runs for a fixed number of iterations, then restores the weights corresponding to the best validation set loss during the second half of training. All models then evaluated the unseen test set.
We reapplied both these algorithms the problem of learning a perturbation the bascline model. Again the perturbations were single layer neural networks with the same architecture as the bascline.
Finally we consider two post-procsssing methods, those of Feldman et al. 2016)), to impose demographic parity and equalised odds respectively. We use our own implementation the former, and the AI Fairness 360 library to apply the latter (Bellamy al.) 2018). each applied to the aforementioned baseline.
2015 and Hardt
'In the original paper, Feldman et al. (015)) present their algorithm pre-processing method. The idea of instead applying it as post- processing algorithm not ours, having been previously observed by Hardt https: http://tt...ggg////////eeee/e///
We weren't able to find any additional sources for this idea.
13

---

Table 3: Values for hyperparameter grid used in our simulations
Dataset
Variables
Adult
COMPAS
Number of hidden layers Width of hidden layers
{2, 3}
{0, 1,
Model
32
16. 24)
Learning rate Iterations
{0.0001, 0.001) {2500, 5000}
{0.0001. 0.001) {2500, 5000}
Adversary
Number hidden layers Width of hidden layers
(2,
1,
32
{16, 24)
Learning rate
0.01
0.01
Iterations per model iteration
{1,
{1,
Batch size
512
128
Table 4: Number of observations associated with intervals of increasing fairness
Number of observations in unfairness interval
Method
[0.05, 0.1)
[0.025, 0.05)
(0.01. 0.025)
[0. 0.01)
Adult
Perturbed
423
201
146
103
Demographi
Zhang et al.
550
281
168
183
parity
Adel et al.
606
214
117
39
COMPAS
Perturbed
4009
2548
1956
1459
Zhang et al.
2776
2771
2526
2892
Adel et al.
10015
3157
1482
945
COMPAS Adult
Perturbed
509
1083
231
76
Equalised odds
Zhang et al.
384
1140
278
112
Adel et al.
1554
1902
334
14
Perturbed
3125
2761
2854
3228
Zhang et al. Adel et al.
1986
2381
3437
4087
6421
6827
6330
3756
Stability
To demonstrate instability during training of the adversarial methods, compare the approaches of Adel et al. (2019), and Zhang et al. (2018), in the latter case applied both as specified by the authors, and also to perturbation specified in Sec. [2] Table which we search. We chose them to cover a wide range of plausible experimental configurations. For each combination the relative importance of the fairness objective compared to the accuracy objective. We run experiments with each combination and record the results after fixed number of iterations to construct the violin plots in Sec.
the selection of hyperparameters
additionally vary the weight of the discriminator term in the loss controlling
A.4 STABILITY RESULTS FOR EQUALISED ODDS
We also ran grid search over hyperparameters with repeats to test the stability of the adversarial training of perturbations compared to regular adversarial approaches. Fig. 4hoow the distribution of accuracy outcomes for each method on the two datasets we have been considering throughout, Adult and COMPAS. We see similar results the demographic parity experiments that the optimal accuracy in each fairesss bin is comparable, but there is a much smaller spread of outcomes for the perturbation- based approach. This because the perturbation remains anchored to local accuracy optimum throughout, discouraging the model from jumping between local minima. Table the number experimental results that fell in each bin for both the demographic parity, and equalised odds experiments.
A.5 EXPLAINING MODELS THAT DON'T HAVE ACCESS TO PROTECTED ATTRIBUTES
In section Sec. 3.1 we showed model explanations for an unfair model corrected with an additive perturbation trained using the adversarial method of Zhang et al. (2018). In that example, both the
14

---

0.05 s 0o 0.1
0.025 0.05
0.01 s 20.02.
0.0 co x 0.02
0.66
0.84
Accuracy
0.62
0.78
0.76
0.74
Perturbed Zhang et Mel st
Peturbed Zhang etal. Adel et
Perturbed Zheng et s. Ade et
Perturbed Zheng Adel et
0.75
Accuracy
COAPPKS S
0.65
0.00
0.5
0.50
Perturbed Zhang
Zhang
Zhang et Zheng
Figure Accuracy violin plots of experimental outcomes binned by achieved level of fairness.
Unfair model
Perturbation
Corrected &
0.80
0.80
0.80
Accuracy
3.75
0.75
0.75
3.70
0.70
0.70
copital palo
merital coarus
mllatonssio
edfication
ecospation
hours per Heal
Capital loos
wortccisn
native country
Rite!
Original model
bestamnii hours worl
Corital firs
DEss capital country
educcation
Correctedd del
cupital gam eslecation
merital ciatus
erciparion
hours wroel
Capital leoss
woricians nutive country
race
Dem, pariy differe nce
3.20
0.20
0.20
3.15
0.15
0.15
3.5
0.5
0.05
maritl statum
2.00
hours capital gais
Piptionship
whek
occupetion occppetion
age
capital lox
educatin
worrcies e
native coutry
race
Total
Orgginal morbll mtrriaa Stauu aatennkii hours want Capital glant
0.00
eccundal tars
age
adter Hort class rkce ratliv country Contect mouil
ediccation
0.0
mariaal stotis
capiaal county
Nours per whek
race
age occupation melationtiig Capiiaal
Rital
Figure 5: Accuracy violin plots of experimental outcomes binned by achieved level of fairness.
original-unfair model and perturbation had access the protected attribute. However, this is not requirement of the adversarial mitigation strategy, and indeed in some real-world scenarios may not have access to the protected attribute at inference time. In this section we repeat the experiment with unfair-model and perturbation which do not take the protected attribute as input, and compare the resulting explanations. The experimental setup is unchanged except for the fact that the models no longer take sex input.
The results show some interesting patterns. First we note that both models had comparable accuracy both before and after correction. This is not surprising the Shapley values in Fig. show that sex is relatively unimportant to accuracy, and highly correlated to other features in any case. The baseline model trained without access the protected attribute was slightly fairer, and the perturbation was slightly less effective in imposting fairness. This is also not so surprising. Witholding the protected attribute from the baseline model should make it more unfair, but also we expect the perturbation to benefit from access to the protected attribute so that it can make active corrections.
Looking Fig. 5 it is notable that the fairness Shapley values for the corrected model are all zero. The same Shapley values for the corrected model which had access the protected attribute were non-zero, but in perfect balance so that their sum was zero. This is because the model that has access to the protected attribute able to take "affirmative action", which to say it can actively use the
15

---

protected attribute to cancel out the effects of other features. In contrast, the model which does not have access the protected attribute has such option, and instead learns only use information contained in each feature that is "orthogonal" to the protected attribute. As a result no individual feature increases or decreases the disparity between protected groups.
16


Reviews:
[{"review_id": "zvsCC1-UBl", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality\n- This paper has defined a well-scoped problem: explaining the unfairness of an ML model in terms of the features used, as well as explaining an additive perturbation that will make the model more “fair”.\n- This paper acknowledges that definitions of fairness can be fraught and should be applied with care and domain knowledge. Therefore it has proposed an “explanation” procedure that works with multiple statistical definitions of fairness. This is a strength of the work.\n\n\nClarity\n- Clarity of the “section 2” could be improved. What is the support of $a$, $y$ and $f(x)$?\n\nOriginality\n- There has not been a lot of work on fairness and explainability. As one of the first forays into these questions showing a positive result (as claimed in introduction, and also to my best knowledge), I think the paper is sufficiently original. Even though it is adapting a well-known construct (Shapley value), its originality also lies in connecting explanations to work on post-hoc fairness corrections (section 2.2).\n\nSignificance\n- As stated above, I think the paper makes a significant contribution to research on fairness and explainability by developing some basic tools to help with “explaining” group fairness issues with ML models. \n- That being said, I think one must be careful to claim that the explanations offered by shapley value suggest any particular intervention. For example, it is not true that the features that contribute most to unfairness should be always be removed. I think a careful discussion on how to use the insights from the shapley values in practice, or some open questions regarding the interpretation of the shapley values would improve this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "tX-3TJRSzmm", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for feature attribution for fairness of the classifier.  They also demonstrate a feature augmentation technique to mitigate unfairness.  They connect their attribution method to to the augmentation technique and demonstrate that their method can attribute the necessary changes to achieve fairness. They evaluate their approach on a few tabular data sets.\n\nIntroduction + Methods\n\n- I’m slightly confused about equation (2), is the correct interpretation that y is the label we’re explaining? So, say, if we’re explaining the -1 class and f(x) is 0.25, then f_y(x) yields 0.75? \n- I think some clarity on the notation would be useful in general.  Is y the ground truth label from the data? That seems to be described this way in section 1.  However, in equation (2) is seems to refer to the label we’re selecting to explain, which feels slightly different.  I think some of this notation could be cleared up a bit.\n- “Splicing disjoint sets of features” is this common terminology? I think what this is trying to say is that this step is where we substitute values from the data into a point to create a perturbation.  Let me know if I am correct.\n- For the description for equation (5): “…the expected accuracy for a model which samples a predicted label according to the predicted probability” what does this mean? The notation in equation (5) looks like it’s the expected prediction of the model, where is the relation to accuracy here? I’m not quite seeing this.\n-  I think that equation (5) could be clarified in general.  It would be good to explicitly state this significance of this property.  (It’s stated clearly for equation 9, and stating it explicitly here would help readers)\n- Is g a value function? If so, it’d be useful to state this leading up to the equation. Also, we’ve introduced the term “a” at this point which refers to the protected attribute (going back to the introduction).  What values can a take on? Are we assuming $a \\in \\{ 0, 1\\}? It would be good to clarify this. Last what is p(a) meant to refer to? Is this the overall proportion of individuals with a certain protected attribute? If so, why do we need this term?  \n- “The linearity axiom of the Shapley values guarantees that the fairness Shapley values of a linear ensemble of models are the corresponding linear combination of Shapley values of the underlying models” — it would be good to clarify this sentence as it’s slightly confusing right now.\n- For equation (10) assuming that f Is a linear model, is $\\delta_\\theta$ meant to be a vector of values added to each of it’s coefficients?\n- Is is slightly contradictory that in equation (10) $\\delta_\\theta$ is a perturbation (and can be explicitly added to f) but then becomes a function in equation (11)? I get what is being said but perhaps the notation could be improved here.\n\nExperiments\n- Figure 1 is hard to read, can you make the labels bigger for each graph. Also, it might be good to place the y axis values on each graph so that its easier to see the scale.  I see what the graph is saying, but this would make it much easier to figure out.\n- I think figure 1 is a nice visualization overall and gets the point across well.\n- I’m not fully understanding the point being made by section 3.2.. Is the idea that we know this suppressed model doesn’t rely on sex at all, so the fairness Shapley values shouldn’t say it does? \n\nGeneral Questions + Comments:\n- I liked the ideas presented by this paper overall.  The idea of using Shapley values to provide feature attributions for fairness is interesting — particularly the applications for explaining the differences between two models, one of which has been corrected for fairness.\n- I do feel however that the paper can be improved in a number of places in order to strengthen the work as mentioned in my comments.  It would be very useful if the authors provided answers to some of my questions and took some of the comments into consideration for a revision. Particularly, I'm somewhat confused about the contribution of section 3.2 and would appreciate clarification.\n- One related question is that the authors demonstrate their method on the suppression techniques from Dimanov et al.  Would their techniques help at all with the related methods (more so phrased as attacks) from https://arxiv.org/abs/1911.02508?\n- Overall, my sentiments right now are borderline, leading towards reject. There’s a number of points which could warrant further clarification right now in the paper.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "DjGaS9AQKoS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The goal of the paper is to design mechanisms to explain the unfairness in the outcomes of a ML model and propose methods to mitigate unfairness. The paper uses the Shapley value framework. The main idea is to alter the prediction function so that instead of providing the classification score, an \"unfairness\" score is returned. An out of the box application of the Shapley value framework on this unfairness score now returns the \"unfairness\" feature attribution. These feature attributions can be used to explain the unfairness of the model. The paper then proposes to learn a linear perturbation, which when combined with the additive property of Shapley framework results in updated \"unfairness\" attributions.\n\nWhile the paper tackles an interesting and timely problem, it lacks clarity at several important points. Additionally, some of the claims seem to be a little overreaching. The experimental evaluation can also use some more thoughtful analysis. Please see detailed comments and suggestions for improvement below:\n\n1- First of all, I would highly recommend setting aside a separate (sub)section to describe the setup. Right now, the details about whether f(x) is a real number of a probability, whether y is {0,1} or {-1,1}, or what \"a\" is are scattered across the text, reducing the readability of the paper.\n\n2- Going from Eq. 2 to Eq. 5, the paper somehow switches from prediction probabilities to accuracies. Shouldn't there be a threshold function to convert the probabilities into predictions? Or is the paper only focusing on randomized classifiers? If only the randomized classifiers are the focus, if/how does it limit the extensibility of the proposed technique?\n\n3- Does it make any difference for the explanations when the probabilities are high non-calibrated (https://arxiv.org/pdf/1706.04599.pdf)?\n\n4- What was the accuracy/generalizability of this $\\delta$ in the experiments?\n\n5- After reading Section 2.2, a reader would think: Why even train the linear perturbation? Why not simply train a separate fair model (from the same model class) and get the Shapley predictions of that second model? I think the main idea here is that the paper tries to leverage the Shapley additive property here. How precisely does the additive property help?\n\n6- The results in Figure 1 are interesting, but not very surprising. Specifically, given that one achieves roughly the same accuracy on Adult data regardless of whether the sensitive feature is used or not, it would have been more interesting (and insightful) to show the Shapley plots for the cases when the sensitive feature is indeed not used for classification.\n\n7- The claim made in the paper that fairness is achieved at \"no loss of accuracy\" is perhaps a bit too bold given that there are well-known results about fairness-accuracy tradeoffs (e.g., https://arxiv.org/abs/1701.08230 and https://arxiv.org/abs/1609.05807).\n\n--------------------\nPost rebuttal comments:\n\nThanks to the authors for the helpful comments -- they indeed help clarify some of the confusions. As a result, I have upgraded my score. However, I am still leaning towards reject because I feel there are still open questions that may hinder the adaptability of the proposed method in the real world. Specifically, given the response to question 3 above, it would help to know what are the real world situations where one uses a randomized classifier and is still interested in model interpretability (the two seem to be at odds with each other as randomness inherently seems a bit arbitrary). Another concern that I have is about Eq. 3 in the paper: Why is the sum function chosen to compute global explanations from local ones? There seem to be multiple ways to do this (e.g., median, sum of absolute values) and it would help to know what are the (dis)advantages of not using other aggregation functions.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers found interesting the use of Shapley values to provide feature attributions for fairness, however, the reviewers brought up a number of issues, particularly in terms of presentation and clarity. While the authors' responses did clarify some of these concerns, this was not enough for the reviewers to broadly support acceptance.
Decision: Reject