Fair Feature Importance Scores for Interpreting Decision Trees
Authors: Camille Olivia Little
Debolina Halder Lina
Genevera I. Allen
Venue: Accepted by TMLR ()

Abstract: Across various sectors such as healthcare, criminal justice, national security, finance, and technology, large-scale machine learning (ML) systems are being deployed to make critical data-driven decisions. Many have asked if we can and should trust these ML systems to be making these decisions.  Two critical components are prerequisites for trust in ML systems: interpretability, or the ability to understand why the ML system makes the decisions it does, and fairness, which ensures that ML systems do not exhibit bias against certain individuals or groups. While both interpretability and fairness have garnered substantial attention in the ML literature, methods directly interpreting models in terms of fairness remain limited. This paper considers a popular interpretation for a widely used class of ML models: feature importance scores for decision trees and tree-based models. We introduce a novel Fair Tree Feature Importance Score to assess each feature's impact on fairness or bias in decision trees. Analogous to the mean decrease in impurity for trees, our score quantifies the mean increase (or decrease) in group bias, and extends to interpret tree-based ensembles or surrogates of complex ML systems. Through simulations and real examples on benchmark fairness datasets, we show the validity of our Fair Tree Feature Importance Score, offering meaningful interpretations for both tree-based ensembles and tree-based surrogates of other ML systems.

Published in Transactions on Machine Learning Research (06/2024)
Fair Feature Importance Scores for Interpreting Decision Trees
Camille Olivia Littlef
coll @rice.ede edu
Deportment of Electrical and Computer Engineering
Rice University
Debolina Halder Linal
d173
Deportment Computer Science Rice University
Genevera Allen
genevera. wwnceeeww.llee@@@oummmiccc edu
Department of Statistics
Columbia University
Reviewed on OpenReview: https: // openreview. net/ forum? id-
Abstract
Across various sectors such as healthcare, criminal justice, national security, finance, and technology, large-scale machine learning (ML) systems are being deployed make critical data-driven decisions. Many have asked if we and should trust these ML systems to be making these decisions. Two critical components are prerequisites for trust in MT systems: interpretability, or the ability to understand why the ML system makes the decisions it does, and fairness, which ensures that ML systems do not exhibit bias against certain individuals or groups. While both interpretability and fairness have garnered substantial attention in the MI. literature, methods directly interpreting models in terms of fairness remain limited. This paper considers a popular interpretation for widely used class of ML models: feature importance scores for decision trees and tree-based models. We introduce novel Fair Tree Feature Importance Score to assess each feature's impact fairness bias in decision trees. Analogous to the mean decrease in impurity for trees, our score quantifies the mean increase (or decrease) in group bias, and extends lo interprel tree-based ensembles or surrogates complex ML systems. Through simulations and real examples on benchmark fairness datasets, we show the validity of our Fair Tree Feature Importance Score, offering meaningful interpretations for both tree- breed ensembles and tree- based surrogates of other MT systems.
1 Introduction
The adoption of machine learning models in high-stakes decision-making has witnessed a remarkable surge in recent years. Employing these models to assist in human decision processes offers significant advantages, such as managing vast datascts and uncovering subtle trends and patterns. However, it has become increasingly evident that the utilization of these models can lead to biased outcomes. Even when users can discern bias within the model's results, they frequently encounter substantial hurdles when attempting to rectify this bias, primarily due to their inability to comprehend the inner workings of the model and the factors contributing to its bias. When machine learning models impact high-stakes decisions, trust is paramount (Toreiní al.
2020 Rasheed et al. 2022 Broderick et al. 2023 Users, stakeholders, and the general public need to have
* EEqual contribution.

---

Published in Transactions on Machine Learning Research (06/2024)
confidence in the fairness and interpretability of these models. Without comprehensible explanations and the ability to audit model decisions, trust can degrade rapidly.
An incident with the Apple Credit Card in 2019 is a prime example of this. The wife of a long-time married couple applied for an increased credit limit for her card (Vigdor 2019). Despite having a better credit score and other positive factors in her favor, her application for an increased line of credit was denied. The husband, who had filed taxes together with his wife for years, wondered why he deserved a credit limit 20 times that of his wife. When the couple inquired as to why the credit limit was so different, no one was able to explain the decision to the couple, which created consternation amongst these and other clients social media who also demanded explanations (Knight) 2019). This led to an investigation by the New York State Department of Financial Services. While the investigation showed that the card did not discriminate based on gender (Campbell, 2021), the inability to provide an interpretation or explanation about the fairness of the algorithm used to determine credit limits created significant mistrust. Moving forward, we must have ways of interpreting ML systems based not only on the accuracy of predictions but also on the fairness of the predictions. As a particular example, we have many ways to interpret how features affect model's predictions through feature importance scores (Du et al. 2019 Murdoch et al. 2019 Yet, we have no current way of understanding how a feature affects the fairness of the model's predictions. The goal of this paper is to fill in this critical gap by developing a simple and interpretable fair feature importance score.
Countless works have proposed methods to improve fairness in existing models (Zemel et al. 2013 Calmon et al. 2017 Agarwal et al. 2018 Zhhang et al. 2018: Lohia et al., 2019 Caton & Haas 2020), but few have focused on how to interpret models with regards to fairness. We adopt a simple approach and consider interpreting features in the context of fairness in decision trees. Why trees? For one, decision trees are the base ensemble for one of the most widely used algorithms, random forests (RFs). Renowned for their robust predictive capabilities, RF's provide an easily computable intrinsic feature importance score known as mean decrease in impurity (MDI) (Breiman 1973 Caruana et al. 2008). Beyond RFs, tree-based ensembles such as AdaBoost and XGBoost are widely used machine learning models, especially for tabular data (Fernándoz- Delgado el a 2014). Moreover, trees have also more recently been proposed as interpretable surrogales for deep learning systems (Guidotti et al., 2018 Schaaf et al.) 2019).
In this work, develop straightforward and intuitive metric for calculating fair feature importance scores in decision trees. Our Fair Tree Feature Importance Score (FairTreeF1S) reveals which features lead to improvements in the fairness of decision tree's predictions and which degrade fairness or contribute to the tree's bias. Additionally, we show how FairTreeFII can be used to explain the fairness of predictions in tree-based ensembles and through
surrogates of other complex ML systems.
1.1 Related Works
To promote trust, transparency, and accountability, there has been a surge recent. research in interpretable ML; see reviews of this literature by Ishwaran (2007): Kazemitabar et al. (2017 1 Lipton (2018) for more details. Interpretable MI. (or explainable AI) seeks to provide human understandable insights into the data, the model or model's output and decisions (Allen et al.) 2023 Murdoch et al. 2019 One of the most popular interpretations is feature importance, which measures how each feature contributes to a model's 's predictions. There are a wide variety of model-specific feature importance measures like the popular mean decrease in impurity (MDI) for decision trees (Louppe et al. 2013 or layer-wise relevance propagation (LRP) for deep learning (Samek et al., (021), ; among many others. Several proposed model agnostic measures of feature importance include Shapley values, feature permutations, and feature occlusions (Mase ct al. 2021:
Chen et al., 2018 Lundberg et al., 2018). This work is inspired by the MDI metric for decision trees which is
widely used and also has been the subject of much recent research (Strobl et al., 2007 Li et al. 2019 Zhou & Hooker
2021
Scornet 2023).
Another notable category of interpedaabilly--nnaannnng Lechniques involves surrogate models. A surrogate model is a simplified and more interpretable representation of a complex, often black-box box model (Samek & Müller 2019). Surrogate models are designed to approximate the behavior of the original model while being easier to understand, faster to compute, or more suitable for specific tasks such as optimization, sensitivity analysis, interpretability; examples include linear models, decision trees or Gaussian processes.
2

---

Published in Transactions on Machine Learning Research (06/2024)
One of the most well-known surrogates for interpretability is LIME (Local Interpretable Model-Agnostic Explanations) (Ribeiro et al., 2016); this approach builds simple and interpretable (usually linear) model to interpret local sub-region of the input space. Global surrogates, on the other hand, build second surrogate model to approximate the global behavior and all the predictions of the original model. Decision trees have been proposed as potential global surrogales as they are fast, simple and interpretable, and as a fully grown decision tree can exactly reproduce the predictions of the original model on the training data (Blanco-Justicia & Domingo--Prreer 2019). On a related note, decision trees have played a crucial role in an associated field known as knowledge distillation, where simplified surrogates of complex models are crafted to mimic the complex model's predictions (Hinton et al., 2015 Gou et al.. 2021). Although knowledge
distillation focuses on prediction, it is worth noting that if predictions from surrogate decision trees prove
to be accurate, they can also be harnessed for interpretation (Yang el al. 2018 Sagi & Rokach 2021 Wan
et al. 2020).
Separate from interpretability, fairness is another critical component to promote trust in ML systems. There
has been surge of recent literature on fairness (Chouldechova & Roth 2018 Friedler et. al.
2019). And
while many methods have been developed to mitigate bias in ML systems (Zhang et al.) 2018
Grari et al.
2019 Agarwal ct al. 2018), very few of these papers have additionally focused on interpretability.
Yet, many
have called for improving interpretability in the context of fairness (Jain et al. 2020 Agarwal
2021
Dai
itt al.
2021 Wang et al. 2023 Notably, there are a few recent examples that seek to address this. Begley
itt al.
2020 introduces a new value function that measures fairness for use within Shapley values; although
this is an interesting and relevant approach, no code is publicly available and computing these Shapley values
requires significant computational time. Another relevant. example is LimeOut. (Bhargava et al. 2020) which
uses LIME explanations to determine which features to drop to make a classifier fairer. This is local and not global method, however, and the focus is on selecting features, not directly interpreting them via a feature importance score. In this paper, we are motivated to address these issues by proposing very simple, intuitive, fast, and easy-to-compute fair feature importance score.
1.2 Contributions
We make three major contributions that allow us to interpret a tree or tree-based model in terms of the fairness of its features. First, we propose and develop the first fair feature importance score ( ThirTreeFIS) for interpreting decision trees. Sccond, we outline how to use FairTreeFI5 to interpret tree-based ensembles and tree-based global surrogates of complex ML systems. Finally, we empirically validate Fair TIS for interpreting trees, tree-based ensembles, and based surrogates of deep learning models on both synthetic and benchmark datasets.
2 FairTreeFIS: Fair Feature Importance Score for Trees
2.1 Review: Feature Importance Score (TreeFIS) for Trees
One of the many benefits of decision trees is that they have a straightforward mechanism for interpretation. Perhaps the most popular feature importance score for trees based on the Mean Decrease in Impurity (MD1), measured by a decrease in variance in regression or in the Gini Index or other metrics for classification Breiman 1973): ; we refer to the MDI as TreeFIS in this paper. Let us first introduce some notation to formally define and review TreeFIS; this definition will help us in defining our novel fairness version of TreeFTS in the next section. Suppose we have a response and the decision tree is built from data X based on n samples. Additionally, let t == 0 be the root node of the tree and T be the total number of nodes in the tree; let n, be the number of samples falling in node t. Next, let (t be the left child of t and (t) be the right child of node 1. Let S. = (i E t} be the set of samples belonging to node t; let ys, be the response associated with those samples in node i, we denote as yt for ease of notation. Let St denote the predictions for samples in node L. As an example, for binary classification with y E {0, 1} or for regression with y E recall that d.- 1.. Eves, yi that is, de is the proportion of successes in node t in the classification setting and the mcan of node in the regression setting. Additionally, let we represent the weighted number of samples " at node and 1 ((.)))) denote the indicator that. feature j was split upon in node t. Let C(y.)) be the loss
3

---

Published in Transactions on Machine Learning Research (06/2024)
lev(t)
Jev
Bias 0.17 Impur 0.38
c(t)
c(t)
c,(t)
#
Blas 0.09
Impur 0.26
C
D Label =
Label
Pred = 0.75
Pred 0.64
Blas 0.17
+++++
+++
Impur = 0.38
+++4+4
Label 11 Pred 0.43
Label Pred 0.88
++++
Bias = 0.29 Impur = 0.34
Figure 1: Schematic trees illustrate TreeFIS and FairTreeFIS. Panel illustrates the level of node and the child level of t that are used to calculate FairTreeFIS. Panels B-D illustrate classification trees with pluses and minuses denoting positive and negative labels respectively and red and blue denoting the majority and minority groups respectively. Panels B and C show the Bias and weighted impurity (Gini Index) at node or level and that the children node f. Tn Panel B, notice the Bias decreases between the parent and child level, resulting a positive Fair TreeFIS for that split. Differently, in Panel C, the Bias increases, resulting in negalive TreeFFI for that split. Panel D illustrates why we must use soft predictions versus hard labels when computing FIS..
function employed to built the decision tree (e.g. MSF loss for regression the Gini Index Cross Entropy for classification). Now, we formally define TreeFIS:
Definition 1. For decision tree, the TreeF1S (MDI) for feature is defined as:
T-1
FIS, = Σ t=
(wn 0t) (Wer(t) Der(t)) + ((ye,co) (e. De.())))
(1)
If feature j is used to split node I, then the TreeFIS calculates the change in the loss function before and after the split, or more precisely, the change in the loss between the predictions at node and the predictions of node t's children. Hence, TheeFIS uses the accuracy of the predictions to determine fcature importance.
2.2 Fair
Inspired by TreeFIS, we seek to define a tree- bree-based feature importance score for group fairness that is based upon the bias of the predictions instead of the accuracy of the predictions. To do this, we first need to define group bias measures. Let 2 E {0, 1} for i= 1,... n be an indicator of the protected attribute (e.g. gender, race or etc.) for each observation. We propose work with two popular motrics bias, Demographic Parity (DP) and Equality of Opportunity (EQOP), although we note that our framework is conducive to other group metrics as well. In brief, DP measures whether the predictions are different conditional on the protected attribute whereas EQOP is typically only defined for classification tasks and measures whether the predictions are different conditioned on a positive outcome and the protected attribute Hardt et al. 2016 Beutel et al. 2017).
measure the group
One might consider simply replacing the loss function equation | with these bias metrics, but constructing our fair metric is not that simple. Consider that for TreeFIS, we can calculate the loss between Ye and i for particular node I, hence calculate the difference in loss after split. We cannot this same

---

Published in Transactions on Machine Learning Research (06/2024)
process, however, for bias as the predictions in each node of the decision tree are the same by construction. Thus, for given node i, there are never any differences between the predictions based on protected group status. Hence, the bias calculated at node t must always be zero. To remedy this and keep the same spirit as TreeFIS, we propose to consider the difference in bias between the split that produced node and the split at node I that produces node I's children. Thus, we propose to calculate the bias that results from each split of the tree. To formalize this, notice that the result of each split in a tree a right and left node. We call this set the level of the tree for node and denote this as lev(t); this level includes the right and left node denoted leey(t) and (t), respectively. We also let c(t) denote all the children of t, or in other words, the child level of node t. Now, we can define our bias metries for the split that produced node t, or in other words, the level of node t. The Bias of leo(t) in terms of DP and EQOP are defined as follows:
Bias DP (lev(()))))))))))))))))))))))) |( = 1, e lev(1)) - E(D: 21 = 0, lev())));
(2)
Bias (OoUO(l EE 1 = 1, 1, E lev(t))
E(j: = 1ly: = 1, = 0, i E lex(t)))
(3)
These group bias metrics range between zero and one, with higher values indicating larger amounts of bias in the predictions. Armed with these definitions, we now soek to replace the loss function & in TreeFIS with this Bias metric to obtain our FairTreeFFe To do so, calculate the difference in bias between the level of node t and node t's children:
Definition 2. The FairTreeF1S for feature j is defined as:
T-1
PairTreeFIS,- Σ t=
munsy (Bias(lev(()) - Bias(c(())
(4)
Note that at the root node, = O, the level the tree consists of only the root; then, (ias(lev(O)) = 0 for this constant model at the root in our definition. Finally, as the scale of TreeFIS is not always interpretable, it is common to normalize TreeFIS so that is sums to one across all features. We analogously do so for TreeFSS by rescaling so that the sum of the absolute values across all features is one; in this manner, FairTreeFFs and TreeFIS are on the same scale and can be directly interpreted.
Our FairTreeFI1 formulation is an analogous extension of TreeFIS as it calculates the Bias of the parent minus the Bias of the children summed over all splits that split upon feature j. But unlike TreeFIS which is always positive, FairTreeFIS can be both positive and negative. As decision trees are constructed with each split minimizing the loss, the difference splits do not consider Bias, however, so the Bias of the parent level could be higher or lower than that of the child level. Thus, FairTheeFIS will be positive when the split at node f improved the bias and negative when the split at node I made the bias worse. Fai is then positive for features that improve the fairness (or decrease the bias) and negative for features that are less fair (or increased the bias). This is a particularly advantageous aspect of Fair TreePIS that improves the interpretability of each feature with respect to fairness.
loss between parent and children is always positive. The
Figure I illustrates our FreeFFeFFI definition for a binary classification example. Panel A highlights our notation and calculation of Bias for levels of the tree. In Panel B, the bias improves from the parent level to the child level and hence Fair TIeeFIS is positive, indicating the split improved the fairness of the predictions. The opposite happens in Panel C where the bias is worse in the child level and hence ThirTreeFI is negative indicating worsening fairness as a result of the split.
In regression settings, FairTreeFIS can be easily applied with the demographic parity metric equation 2 which is most commonly used for regression tasks. The bias can be calculatod directly as the empirical mean of the predictions in each sensitive group. For classification settings, however, more care needs to taken in computing the Bins and our FairTreeFIS metric, discussed in the next section.
5

---

Published in Transactions on Machine Learning Research (06/2024)
Linear
Nonlinear
Nonlinear, Interaction
Decision Tree
0.-
Feature Importance Score
Group
Boosting
TreeFis
3.0
FairtroaFis
FairtroaFis (iG)
-02
FalrTreeF15 illty)
FairTreeF15 (Vi-)
Random Forest
-0.2
-0.6
Feature
Figure 2: Classification results for TreeFIS (MD1 using the Gini Index) and FairTreeFFh (DP) on the three major simulation types and for a docision tree, gradient boosting, and random forest classifier. We consider four feature groups: features in Gj (red) and G2 (blue) are correlated with the protected attribute, features in G₁ and G3 (green) are signal, and features in G4 (purple) are noise. The magnitudes and directions of the Fair TreeF18 scores for each group align with what would expect from the simulation construction, thus validating our metric.
2.3 Fair TrieFlFF::: Classification Setting
Typically for classification tasks, people use hard label predictions to compute the DP and EQOP Bias metrics. However, for decision trees, this presents problem as both the left and right node of level t could predict the same hard label; the parent and child levels could also predict the same hard label. In these settings, using hard labels with equation 2 and equation B would result in zero a misleading Bias measure even when the split might be unfair. This phenomenon is illustrated in Figure μ Panel D. To remedy this issue, we are left with two options: employ measures of Bias that take soft predictions employ probabilistic decision trees that return stochastic hard label predictions based the soft label probabilities. So that our Bias metrics are interpretable and comparable with others that typically employ hard label predictions, we choose the latter option. Let levz(() and lev,(1) denote the left and right nodes of the level of node and let (t)r and (t),, denote the proportion of positive samples in these nodes, respectively. Then for Bernoulli random variable with probability of success (t)) and that of the right node is defined analogously. Given this, we can directly apply equation 2 and equation 3 to compute the expectation necessary for Bias metrics:
probabilistic troes, ÿú for E levl(t)

---

Published in Transactions on Machine Learning Research (06/2024)
Proposition 1. Consider binary classification with probabilistic trees, then our Bias measures given by the following:
Σ, 1 [
Σ, 1 0,
Bias MaasP((len (lev(()) = (leve(1)
1l Σ;
Σ, (a=0,ietev(())
E, ((s=
(te
1 (x
E, 1 (z, -,iElee((())
(5)
Σ, [x.00,y-,,,eee0((0))
Bias EQOP (lev(!)) = (leve(1)
11 (
Σ, (a=0O.u_l,jEtee(t)
E: 1 (1
Σ, (1=000==1))0000,0111))
(t)
(6)
Σ (2=00,,1,,,ee000))
Thus, even when employing probabilistic trees, our Bias measures and hence FairTreeFIS is casy to compute. The proof / calculation for Proposition is in the Supplemental materials. Note also that these results for the Bias and also FairTreeFIS can easily be extended to multi-class classification settings, which we present in the Supplemental materials.
2.4 FairTreeFI5 for Tree- Based Ensembles and Decision Tree Global Surrogates
Decision trees widely used due to their ability to break down complex problems into simpler solutions, thus making them more interpretable (Loh 2011). Further, they are commonly employed in various popular ensemble-basee based classifiers such as random forest, gradient boosting, XGBoost, and others. For these tree- based ensembles, TreeFIS is averaged (or averaged with weights) over all the trees in the ensemble (Breiman) 1996). We propose to extend Fis in the exact same manner to interpret all tree- based ensembles.
Decision trees have also gained attention for their role in knowledge distillation transfer knowledge from large, complex models to smaller models that easier to deploy (Hinton al. 2015 Buciluu et. al. 2006). Here, decision trees are not fit to the original labels outcomes, but instead to the complex model's 's predicted labels or outcomes. Recently, others have proposed to use decision trees in a similar manner for global interpretation surrogales [Blanco-Justicia & Domingo-Ferrer 2019 Yang et al. 2018 Sagi & Rokach 2021 Wan et al. 2020). Decision trees are often an ideal surrogate in this scenario as a fully grown tree can exactly reproduce the predictions the complex, black-box model. Hence, the predictions match precisely, we can be more confident in the feature interpretations that the decision tree surrogate produces. Here, we propose to employ FairTreeFIS to interpret features in decision tree surrogate in the exact same manner that of TreeF18. In this way, FIi provides simple, intuitive, and computationally efficient way to interpret any large, complex, and black-box ML system.
3 Empirical Studies
3.1 Simulation Setup and Results
We design simulation studies to validate our proposed FairTreeFIS metric; these simulations are an important test since there are not other comparable fair feature interpretation methods to which we can compare our approach. We work with four groups of features: features in G₁ and G2 are correlated with the protected attribute and are hence biased, features in G1 and G3 signal features associated with the outcome y, and features in G4 are purely noise. We simulate the protected attribute, 2,, as z ~ Bernoulli(z) and take
i.id
क = 0.2. Then, the data is generated as X,,j i,i.d N(a, * 2,, Σ) with a, = if j E G1 or G2 and aj = 0 if j E G2 or G4 in the classification scenarios and a, = 0.4 for j E G₁ or G2 and aj the regression simulations. Hence, all features in G2 and G2 are strongly associated with a and hence should be identified biased features with negative Fair TreeFFS. Then, we consider three major simulation
for j E G3 G4
7

---

Published in Transactions on Machine Learning Research (06/2024)
Linear
Nonlinear
Nonlinear, Interaction
0.50
Decision Tree
0.23
-0.25
Feature Importance Score
-0.50
0.50
Group
Boosting
TreeFis
FairtroaFis
0.00
FairtroaFi5 (iQ)
-0.25
FalrTreeF15 illty)
FairTreeF15 (Vi-)
-0.50
Random Forest
0.02
0.25
Feature
Figure 3: Regression results for TreeFIS (MDI using the Gini Index) and FairTreeF1S (DP) on the three major simulation types and for a decision tree regressor, gradient boosting regressor, and random forest regressor. We consider four feature groups: features in Gi (red) and G2 (blue) are correlated with the protected attribute, features in G2 and G3 (green) are signal, and features in G4 (purple) are noise. The magnitudes and directions of the Fri scores for each group align with what we would expect from the simulation construction, thus validating our metric.
scenarios for both classification and regression settings: a linear model where f(x;) = 30 + Σ5 Bysij a non-linear additive scenario where f(x;) 30 + En (z,in( and finally a non-linear scenario with pairwise interactions where f(x,) = 8 + E By + Ei=1,s= s(sir(()) and with Yuu 1 for the first two features in cach group and zero otherwise. We also let 33
for j E G. or G3 and Bg = 0 for E G2 or G4 in the classification scenario and 3, = 3 for j E G1 G3 and B3 = 0 for j E G2 or Ga in the regression simulations. For regression scenarios, we let yi = f(x)) +e where iid NO0, 1), and for classification scenarios, we employ a logistic model with y i,dd (f function. We present our binary classification and regression results for the DP metric with N = 1000, p == 12 features, and Σ in Figure tasks with N = 500 1000, larger p, correlated fcatures with Σ E I, and for the EQOP metric are presented in the Supplemental Materials.
where or is the sigmoid
/ Additional simulation results for both classification and regression
Figure 2 presents the TreeFIS and FairTreeFSS metric for each of the twelve features colored according to their group status, and averaged over ten replicates. We present all three classification simulation scenarios for decision tree, gradient boosting, and random forest classifiers. First, notice that the sign of FairTreeFIS is correct in all scenarios; that is, features G1 (red) and G2 (blue) are biased and FairTreeFIS accurately reflects this bias with negative score while the features in G3 (green) and G4 (purple) exhibit no bias and FairTheeFF is positive. FairTreeFIS also accurately captures the magnitude of each feature's contributions the magnitude of FairTreeFIS and TreeFIS values are low for non-signal features in trees and gradient boosting, as non-signal features are likely not split upon and hence do not. contribute to bias or fairness. Because random forests use random splits, however, ninnsignal features are split upon more often and we see that FoirTreeFIS accurately determines that features in G2 are biased. Additionally, decision tree regressor, gradient boosting regressor, and random forest regressor. We see similar behavior to the classification results. Overall, these results (and the many additional simulations in the Supplement)
comparable in all scenarios. Note here that FairTreeFIS
Figure 3 we present all three regression scenarios for

---

Published in Transactions on Machine Learning Research (06/2024)
Adult (Gender)
Law (Race)
0.4
0.5
0.0
0.0
-0.4
-0.6
Age
Cap Gain
Cap Loss
Edu Num
Hrs/Week
Married
Self
Self No
US NC
Wife
Age
Income
Fulltime
LSAT
Undergrad GPA
Yr1 GPA
GPA
Feature Importance
Treefis
COMPAS (Race)
Communities Crime (Race)
0.2
0.2
0.00
-05
-050
Age-25
Age:s45
Age:25-55
Chargef
Charge:
Num Prics3
Num Pri30
Num Pri:1-3
% No Phone
Asian Per Cap
Pers/Rented Hse
Retired
Large
Kids
%
Feature
Figure 4: Random Forest interpretation using FrirTreeFIS for the Adult, Law, COMPAS, and Communtiies and Crime datasets. The importance scores' magnitudes and directions align with other studies done on these datasets.
strongly validate the use of Fair TreeFIS for interpreting features in trees and tree- based ensembles for the bias or fairness that the feature induces in the predictions.
3.2 Case Studies
To align work with the existing fairness literature, we evaluate our metric five popular benchmark datasets. We examine: (i) the Adult Income dataset (Dua & Graff 2017) containing 14 features and approximately 48,000 individuals with class labels stating whether their income is greater than $50,000 and Gender as the protected attribute; (ii) the COMPAS dataset (Larson et al. 2022), which contains 13 attributes of roughly 7,000 convicted criminals with class labols that state whether the individual will recidivate within two years of their most recent crime and we use Race as the prolected attribute; (iii) the Law School dataset (Whiteman 1998), which has 8 features and 22,121 law school applicants with class labels stating whether individual will pass the Bar exam when finished with law school and Race as the protected attribute; (iv) the Communities and Crimes (C & C) dataset (Dua & Graff 2017). which contains 96 features of 2,000 cities with regression task of predicting the number of violent crimes per capita and Race encoded as the protected attribute; and (v) the German Credit dataset, which classifies people with good or bad credit risks based on 20 features and 1,000 observations and we use Gender as the protected attribute.
We begin by evaluating the quality of FairTreeP1S interpretations on four benchmark datasets in Figure additional interpretations of all benchmarks provided in the Supplemental Material. Figure shows scores for random forest classifier the Adult dataset with Gender as the protected attribute, the Law dataset with race as the protected attribute, the C & C datasct with Race as the protected attribute, and the COMPAS dataset with Race as the protected altribute. In the Adult datasel, the "Married" feature contributes most to both accuracy and bias. This indicates that while the "Married" feature is highly predictive, its use the model could lead more unfair prodictions. This aligns with studies done showing that in men higher wages were associated with a higher proportion of being married, whereas in women higher wages associated with lower proportion of being married and the fact that many women

---

Published in Transactions on Machine Learning Research (06/2024)
not employed outside the home (Mincy et al. 2009). Figure shows that "Yr 3 GPA" contributes most to both the accuracy and the bias of the model in the Law dataset. In the C & C dataset, the percentage of kids who grew up with two parents in the household, denoted as "% Kids 2 Par", has the highest magnitude for both FIS and Free,reeFIS although Fair FrS shows that this feature contributes to the bias seen in the predictions. Studies have shown that. black young adults are disproportionately impacted by family structure (Wilcox (021). Specifically, black young adults are less likely to go to college and more likely to be imprisoned if they grow up in a single-parent household. In contrast, white young adults are significantly less affected by family structure. Thus, our FairTreeFISS interpretations are consistent with these studies. Looking at the results for the COMPAS dataset, the number of priors greater than 3, denoted as "Num Pri > 3" has the highest magnitude for both TreeF'S and Tree/I, and again FairTreeFIS reveals that this feature strongly influences the biased outcomes. These interpretations are consistent with other studies on the COMPAS data set (Rudin et al. 2020), again validating our results.
Next, we further validate our Free interpretations of Random Forest models by removing features that contribute most to the bias of the model based on their FairTreeFIS values. Figure 5 shows fairness and accuracy metrics over 10 runs on the Adult, Law, COMPAS, and Communities and Crime datasets. The gray bar represents all features included in the model, the red bar represents when the feature with the most negative FairTheeFF value was removed from the model, and the blue bar represents when the two features with the most negative FairTreeFIS values were removed. We expect that models containing all features would yield the most biased results and the bias to decrease as features that contribute to the bias most are removed. We see that this is indeed the trend for all of the datasets. Furthermore, we see in the C & C dataset that the accuracy also slightly increases as features that contribute to bias are removed. Tn the Adult datasct, we notice little movement between fairness and accuracy included. We suspect that this is due in part to the fact that the Adult and C & C datasets have a large number of features and many of them are highly correlated. As a result, it is unsurprising to see only small changes when removing 1 or 2 features at a time.
matter which features are
We move on to validating the use of FrerTreFFIS for interpreting tree-based global surrogates. To do this, in Figure g we compare TreeFIS and FairTreeFIS results on a gradient boosting classifier (where these scores were calculated by averaging over all tree ensemble members) to TreeFTS and FairTreeFIS results for tree-based surrogate of the same gradient boosting classifier (where fully grown decision tree was fit to the model's predictions). Generally, we see that the TreeFTS and FairTreeFIS scores bet the top row (boosting) and bottom row (surrogate) are similar in magnitude and direction. Specifically looking at the Adult dataset, we see that "Married" is an important feature according to TreeFIS but FairTreeFIS indicates that the model is using "Married" in a way that contributes to bias; these results are reflected in both the boosting model and the tree surrogate. While the scores for some of the less important features may vary slightly between the original model and the surrogate, notice that the most important features are always consistent between the two approaches. This indicates that our FairTheeFIS scores effective when used to interpret tree-based global surrogates. Additional case studies on tree-based surrogates including validating TreeFIS compared to model-specific deep learning feature importance scores are provided in the Supplemental Material.
Lastly, in Figure 7 we examine TreeF1S and FairTreeFIS scores for a tree-based surrogate of a deep learning model (multi-layer perception with two hidden layers cach with p units and ReLU activation) as well as a tree-based surrogate for a bias mitigation method, the Adversarial Debiasing approach (Zhang al.] 2018) for the Adult dataset with Gender as the protected attribute. The Adversarial Debiasing method (Zhang ct al.) 2018) applics adversarial learning to improve fairness by learning how to prevent an adversary from predicting the protected attribute. Looking at the Adult dataset scores of the tree-based surrogate of the deep learning model, we see that the "Cap. Gain", "Edu Num", and "Married" features are most important in terms of accuracy and US Native Country (*US NC"), "Married", and "Age" are most influential in terms of bias. Specifically, "US NC" and "Marriod" hurt the overall fairness of the model. In the debiasing method, the magnitude of both FairTreeFTs and TreeFIS for the feature "Married" decreases substantially, showing that using this feature likely would result in more biased predictions. Additionally, the "Cap. Gain" feature becomes more important in terms of accuracy in the debiasing model, as this feature contributes less to bias. The accuracy and fairness go from 0.84 and 0.83 in the deep learning model to 0.80 and 0.92
10

---

Published in Transactions on Machine Learning Research (06/2024)
Adult (Gender)
Law (Race)
0.20
0:85
0.20
0.90
0.19
Accuracy
0.84
Accuracy
Blas
0.83
0.82
0.10
0.80
0.81
0.16
0.80
0.05
0.75
AlI
One
Two
All
One
Two
All
One
Two
All
One
Two
COMPAS (Gender)
C & (Race)
Remove O
0.100
0.225
Remove Two
0.075
Accuracy
0.55
0.0075
Bass
0.050
0.50
MSE
0.0050
0.025
0.45
0.0025
AlI
One
Two
All
One
Two
All One Two
AlI
One
Two
Figure 5: Random Forest model interpretations using FairTreeFIS when features contributing to most to bias are removed (See Figure 4h. Each panel shows accuracy and fairness (DP) metries in three scenarios for the Adult, Law, COMPAS, and & C datasets. The gray bar represents random forest model trained with the features, the red bar represents random forest model trained without the feature with the most negative FairTreeFIS value, and the blue bar represents model trained without the two most negative TreeFFS values. Note that removing the features with the most negative TairTreeFF values decreases bias, therefore validating our Fair Tr metric.
Adult (Gender)
COMPAS (Race)
Law (Race)
0.5
0.-
0.0
-02-
-0.5
-04
Festure Importance
Age
Cap Gain
Cap Loss
Hrs/Week
Married
Seif
Seif No Inc
US NC
Wife
Age:-25
Age:45
Age:25 45
Charger
Charge:M
Num Pric33
Num Prii
Num Pri:1-3
Age
Income
Fulltime
LSAT
Undergrad GPA
Yr1 GPA
GPA
FaiTreuFiS
0.4
0.5
0.5
0.2
0.0
0.0-
-0.5
Age
Cap Loss
Hrs/Wekk
Married
Seif
Seif Inc
US NC
Wife
Age:-c 25
Age:045
Age:25-40
Charge:F
Charge:M
Num Prix3
Num
Num Prit1
Fam Fam Income
Fulliine
Undergrad GPA
Yr Yr1 GPA
GPA
Cap
Age
LSAT
Feature
Figure 6: Global surrogate validation. The top row shows TreeFIS and FairTreeFIS results on a gradi- ent boosting classifier for the Adult, COMPAS and Law datasets. The bottom row shows TreeFIS and FairTreeFFh results for treo-basod surrogate a boosting classifier. The scores between the top and bottom similar in magnitude and direction, indicating that our are elfective when used to interpret tree- based global surrogates.
11

---

Published in Transactions on Machine Learning Research (06/2024)
Adult (Gender): MLP
Adult (Gender): Adversarial Debiasing
0.4
Festure Importance
FaifrreF
Age
Cap Gain
Cap
Edu Num
Hrs/Wook
Married
Self
Self No
US NC
Wife
Age
Cap Gain
Cap Loss
Hrs/Weck
Married
Self
US NC
Wife
Edu
Self
Feature
Figure 7: Interpretation of features for models with bias mitigation. The Fuir TreeFIS values show the difference in importance scores between a tree-based surrogate of deep learning (two hidden layer MLP) model and a tree-based surrogate Note that the "Married" feature has the most negative TreeFFS in the tree-based surrogale model, but is significantly less negative in the debiased model.
bias mitigation approach, deep learning adversarial debiasing method.
in the Adversarial Debiasing model, indicating that the approach is successful at mitigating bias. Sceing as the features that strongly negatively contribute to fairness have less of an impact when the model becomes more fair indicates that our fair feature importance scores accurately capture when features are helping or hurting the overall fairness of the model. Note also that strongly predictive features often hurt fairness, and fairness increases, accuracy decreases. This trend is a sign of the well-known and studied tradeoff ween fairness and accuracy (Zliobaite 2015 Little et al. 2022). Further results on all five benchmark datasets are included in the Supplemental material.
4 Discussion
In this work, we proposed a fair tree feature importance score, FairTreeFIS, for interpreting trees, tree- based ensembles, and tree-based surrogates of complex ML systems. We extend the traditional accuracy- based TreeFIS (MDI), which calculates the change in loss between parent and child nodes, to consider lairness, where we calculate the difference group bias between the parent and child levels. We empirically demonstrated that FairTreeeFSS accurately captures the importance of features concerning fairness in various simulation and benchmark studies. Crucially, we showed that we can employ this method to interpret complex deep-learning models when trees used surrogates.
Our FairTreeFFS metric is inspired by TreeFIS (MDI) and recently, there have been many papers studying theoretical properties of MDI and some of its limitations Strobl et al. (2007) Li et al. 2019 Zhou & Hooker
h021 0. For example, MDI known to favor features with high entropy and less correlation Strobl et al.
20077; ; is also known be consistent using out-ol-sample data for additive models (ccrnet (2023); Li al.
2019 Zhou & Hooker 22021), but its properties for other model classes are unknown. Given this recent research on MDI, may ask whether our FairTreeFIS metric inherits these properties and limitations. Note that FairTreeF1S metric performed well in simulations, including when there are correlated features (see Appendix), suggesting that perhaps these limitations do not apply. Further, note that many of the limitations of MDI arise from the fact that it utilizes the same loss function to compute feature importance as was used to greedily build the tree. In contrast, FairTreeF1l is based upon the differences in proportions across groups after each split and hence less subject to such loss function specific biases. Nonetheless, just as MDI has been studied extensively and theoretically, the properties of our FairTreeFIS metric are important to investigate in future work.
12

---

Published in Transactions on Machine Learning Research (06/2024)
Additionally, there are several other possible directions to investigate in future work. FairTreeFIS is flexible and can be easily extended to work with other group fairness metrics, including those that consider several groups in the protected attribute or even multiple protected attributes. Further, just as people have studied feature interactions using trees Basu et al. (2018), FriSTreeFIS can possibly be extended to interpret bias that might arise from feature interactions. Finally, one could also investigate extending other popular feature importance scores help interpret features in the context of fairness.
Our FoirTreeFIS metric offers valuable insights for transparency and auditing. While our score does not indicate if features are fair or unfair, it does help users understand how the model is using features in a manner that may help or hurt fairness. Combined with traditional TreeFTS, it enables users to under- stand the relative importance of features for both accuracy and fairness. Imagine if, in 2019, Apple and Goldman Sachs had employed FairTreeFIS to explain the biases exhibited by their models toward women. This could have significantly reduced public distrust and facilitated mitigation efforts. Furthermore, the increasing emphasis on fairness in AI models, exemplified by the October 2023 exccutive order on Safe, Secure, and Trustworthy AI, underscores the critical need for tools like FairTree By enabling fair model interpretation, FIS facilitates transparency and builds trust among users and consumers.
Acknowledgments
COL acknowledges support from the NSF Graduate Research Fellowship Program under grant number 1842494. GIA, COL and DL acknowledge support from the JP Morgan Faculty Research Awards and NSF DMS-2210837.
References
Alckh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. A reductions approach to fair classification. In ICML 2018: Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 60-69, 2018. URL httts::/prroceednggg.. mll. http:///nn......rrrr//////////// awteess/v80/ggarwalll. .8tmlll
Sushant Agarwal. Trade-offs between fairness and interpretability in machine learning. Tn IJCAT 2021
Workshop on Al for Social Good, 2021.
Genevera I Allen, Luqin Gan, and Lili Zheng. Interpretable machine learning for discovery: Statistical challenges), & opportunities. arXiv preprint arXiv:2308.014 2023.
Sumanta Basu, Karl Kumbier, James Brown, and Bin Yu. Iterative random forests discover predictive and stable high-order interactions. Proceedings of the National Academy of Sciences, 115(8):1943-19)) 2018.
Tom Begley, Tobias Schwedes, Christopher Frye, and Tya Feige. Explainability for fair machine learning. arXiv preprint vrXi::2010.0738, 2020.
Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H. Chi. Data decisions and theoretical implications when adversarially lcarning fair represontations. ArXiv Pre-Print 1 1707.00075, 2017. doi: 10.48550/arXiv.1707 00075.
Vaishnavi Bhargava, Miguel Couceiro, and Amedeo Napoli. Limeout: an ensemble approach to improve process fairness. In Joint. European conference on machine learning and knowledge discovery in databases, pp. 475 491. Springer, 2020.
Alberto Blanco-Justicia and Josep Domingo-Feri Ferrer. Machine learning explainability through comprehensible docision trees. In Machine Learning and Knowledge Extraction: Third IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2019, Canterbury, UK, August 26 26-29, 2019, Proceedings 9, pp. 15-26. Springer, 2019.
Leo Breiman. Classification and regression Irees. Routledge, 1973.
Leo Breiman. Bagging predictors. Machine Learning, 24(2):123-144) 1996. doi: 10.1023/ A:1118544314350.
13

---

Published in Transactions on Machine Learning Research (06/2024)
Tamara Broderick, Andrew Gelman, Rachael Meager, Anna L Smith, and Tian Zheng. Toward taxonomy of trust for probabilistic machine learning. Science Advances, 9(7)=eabn3999, 2023.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizzl. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-5555550 355-541, 2006.
Flavio P Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. Optimized pre-processing for discrimination prevention. In NewrIPS 2017: Advances in Neural Information Processing Systems, volume 30, pp. 3995 4004, 2017. doi: https:/ http://ppooceedigg.eeeeeee httpa//////aa//////aaaaaaaaaaaaa 1ww.9aa5cSSSaattffoeeeeeeeaaaaa3 Abstractttbttttttthhhh paper/2017/
lan Carlos Campbell. The apple card doesn't actually discriminate against womenn, in- vestigators say, The Verye, 2021. URL
goldman- so serpp---aaadd----eeee---dd------
Rich Caruana, Nikos Karampatziakis, and Ainur Yessenalina. An empirical evaluation of supervised learning in high dimensions. In Proceedings of the 25th international conference on Machine learning, pp. 96-103,
2008.
Simon Caton and Christian Haas. Fairness in machine learning: survey. ArXiv Pre-Print 2010.04055. 2020. doi: (18550/arXi..2100.0000
Jianbo Chen, Le Song, Martin J Wainwright, and Michael I Jordan. L-shapley and c-shapley: Efficient model interpretation for structured data. arXiv preprint 18Xx::8888.02. 2018.
Alexandra Chouldechova and Aaron Roth. The frontiers of fairness in machine learning. ArXiv Pre-Print 1810.08810, 2018. doi:
Jessica Dai, Sohini Upadhyay, Stephen H Bach, and Himabindu Lakkaraju. What will it take generate fairness-preserving explanations? arXiv preprint XrXix:2106. 19346, 2021.
Mengnan Du, Ninghao Liu, and Xia Hu. Techniques for interpretable machine learning. Communications of the ACM, 63(1):68-77, 2019.
Dheeru Dua and Casey Graff, UCI machine learning repository, 2017. URL http://archive. ics. uci auci.edi. edu/ m1
Manuel Fernández- Delgadd, Eva Cernadas, Senén Barro, and Dinani Amorim. Do we need hundreds of classifiers to solve real world classification problems? The journal of machine learning research, 15(1):
2014.
Sorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary, Evan P. Hamilton, and Derek Roth. A comparative study of fairness-enhancing interventions in machine learning. In FAceT 2019: Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency, pp. 999-338, 2019. doi: ///////28767227777777
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey, International Journal of Computer Vision, 129:1789-1819, 2021.
Vincent Grari, Boris Ruf, Sylvain Lamprier, and Marcin Detyniecki. Fair adversarial gradient tree boosting. In ICDM 2019: Proceedings of the 2019 IEEE International Conference Data Mining, pp. 1060-1065, 2019. doi:
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM Computing Surveys, 51(5), aug 2018. ISSN 0360-0300. doi: 10.1145/3236009. URL httpp:://oo..rrg/11o..o 111////20114//3330000
Moritz Hardt, Eric Price, and Nathan Srebron. Equality of opportunity in supervised learning. In NewrIPS 2016: Advances in Neural Information Processing Systems Systems 29, volume 29, 2016. URL ltttps:/paapes
14

---

Published in Transactions on Machine Learning Research (06/2024)
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:15330025%) 2015.
Hemant Ishwaran. Variable importance in binary regression trees and forests. 2007.
Aditya Jain, Manish Ravula, and Joydeep Ghosh. Biased models have biased explanations. orXiv preprint arXiv:2012 10986, 2020.
Jalil Kazemitabar, Arash Amini, Adam Bloniarz, and Ameet Talwalkar. Variable importance us- ing decision trees. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL httpp::/ nwwripe.uu/enineeeeeeeeee hitapprrfflee/ppppr////////LL
373
Will Knight. The apple card didn't see gender and that's the problem. WIRED, 2019. URL https:
//ww.wirred http://ww............aaaaaaaaaaa bhee
Jeff Larson, Marjorie Roswell, and Vaggelis Atlidakis. Compas recidivism risk score data and analysis, 2022.
URL https:://github. http:///rrrroppolaaa///aaaaaaaaa analysiss/
Xiao Li, Yu Wang, Sumanta Basu, Karl Kumbier, and Bin Yu. debiased mdi feature importance measure for random forests. Advances in Neural Information Processing Systems, 32, 2019.
Zachary C Lipton. The mythos of model interpretability: machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31 57, 2018.
Camille Olivia Little, Michael Weylandt, and Genevera I. Allen. To the fairness frontier and beyond: Identi- fying, quantifying, and optimizing the accuracyy Pareto frontier. ar preprint wxXiv222660000 2022.
Wei-Yin Loh. Classification and regression trees. Wiley intendisciplinory reviews: data mining and knowledge discovery, pp. 14, 2011.
Pranay K. Lohia, Karthikeyan Natesan Ramamurthy, Manish Bhide, Diptikalyan Saha, Kush R. Varshney, and Ruchir Puri. Bias mitigation post-processing for individual and group fairness. In ICASSP 2019: Proceedings of the 2019 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2019. doi: 10. 10.1109/ICASS)) SEEAASS..2019.880 0.68222......
Gilles Louppe, Louis Wehenkel, Antonio Sutera, and Pierre Geurts. Understanding variable importances in
forests of randomized trees. Advances in neural information processing systems, 26, 2013.
Scott M. Lundberg, Gabriel G. Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. ArXio Pre- Pre-Print 1802.03888 2018. doi: 00.45500/ariiv ww...i......000 1802.03888.
Masayoshi Mase, Art B Owen, and Benjamin B Seiler. Cohort shapley value for algorithmic fairness. arXiv preprint arXiv:2105.071 07168, 2021.
Ronald Mincy, Jennifer Hill, and Marilyn Sinkewicz. Marriage: Cause or mere indicator of future earnings growth? Journal of Policy Analysis and Management, 28(3):417 439, 2009.
W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. Definitions, mothods, and applications interpretable machine learning. Proceedings of the National Academy of Sciences, 116 011:20771-20800 2019.
Khansa Rasheod, Adnan Qayyum, Mohammed Ghaly, Ala Al-Fuqaha, Adeel Razi, and Junaid Qadir. Ex- plainable, trustworthy, and ethical machine learning for healthcare: A survey. Computers in Biology and Medicine, pp. 106043, 2022.
15

---

Published in Transactions on Machine Learning Research (06/2024)
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 'why should i trust you?": Explaining the pre-
dictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowl- edge Discovery and Data Mining, pp. 1135 1144. Association for Computing Machinery, 2016. ISBN 9781450342822. doi: 10.1145 2ttp///999922229999972 URL https:: /tttps://oo..or/11... 1144/2939//2 2939778
Cynthia Rudin, Caroline Wang, and Beau Coker. The age of secrecy and unfairness in recidivism prediction. Harvard Data Science Review, 2(1):1, 2020.
Omer Sagi and Lior Rokach. Approximating xgboost with an interpretable decision tree. Information Sciences, 572:522 -4252 2021.
Wojeiech Samek and Klaus-Robert Müller. Towards explainable artificial intelligence. Explainable AT: interpreting, explaining and visualizing deep learning, pp. 5 22, 2019.
Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J Anders, and Klaus-Robert Müller. Explaining deep neural networks and beyond: A review of methods and applications. Proceedings of the IEEE, (3):3)2247 278, 2021.
Nina Schaal, Marco Huber, and Johannes Maucher. Enhancing decision tree based interpretation of deep neural networks through 11-orthogonal regularization. In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pp. 42 49. IEEE, 2019.
Erwan Scornet. Trees, forests, and impurity-based variable importance in regression. In Annales de l'Institut Henri Poincare (B) Probobililes el stalistiques, volume 59, pp. 21 52. Institut Henri Poincaré, 2023.
Carolin Strobl, Anne-Laure Boulesteix, Achim Zeileis, and Torsten Hothorn. Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC bioinformatics, 8:1-21, 2007.
Ehsan Toreini, Mhairi Aitken, Kovila Coopamootoo, Karen Elliott, Carlos Gonzalez Zelaya, and Aad Van Moorsel. The relationship between trust in ai and trustworthy machine learning technologics. In Proceedings of the 2020 conference on fairness, accomnlabillyy, and transparency, pp. 272 283, 2020.
Neil Vigdor. Apple card investigated after gender discrimination complaints. New York Times, 2019. URL html
htttps//www..yyyinee..co//////// http:////w0.../o//aa///////aaaat
Alvin Wan, Lisa Dunlap, Daniel Ho, Jihan Yin, Scott Lee, Henry Jin, Suzanne Petryk, Sarah Adel Bargal, and Joseph E Gonzalez. Nbdt: neural-backed decision trees. orXiv preprint arXiv:2004.00221, 2020.
Caroline Wang, Bin Han, Bhrij Patel, and Cynthia Rudin. In pursuit of interpretable, fair and accurate machine learning for criminal recidivism prediction. Journal of Quantitative Criminology, 39 581, 2023.
Linda Whiteman. The scale and effects of admissions preferences in higher education (SEAPHE), 1998. URL ptggddaabaasss php
W. Bradford Wilcox. Less poverty, less prison, more
college: What two par-
ents mean for black and white children, 2021. URL https://ifstudies wwwpe/ddddes..ro//bb/ blog/
Yongxin Yang, Irene Garcia Morillo, and Timothy M Hospodales. Deep neural decision trecs. arXiv preprint arXiv:1806.0 06988, 2018.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In In- ternational conference on machine learning, pp. 325-333, 2013. URL https:://proceeeenggs.ttt mle.sssssss/rsss////// v28 08//emee13.. html
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In AIES 2018: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 335 340, 2018. doi: 10. (0.1145/3278721.327755
16

---

Published in Transactions on Machine Learning Research (06/2024)
Zhengze Zhou and Giles Hooker. Unbiased measurement of feature importance in tree-based methods. ACM Transactions on Knowledge Discovery from Data (TKDD), 15(2):1-21, 2021.
Indre Zliobaite. On the relation between accuracy and fairness in binary classification. ArXio Pre-Prit 1505.05723. 2015. doi: 005550//XXiv.555..57777 Presented at the 2nd Workshop Fairness, Account- ability, and Transparency in Machine Learning.
17


Reviews:
[{"review_id": "wPLmN4qjdO", "reviewer": "Reviewer_oq7Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-----------------\n+ Generally important idea\n+ Extensive empirical evaluation\n+ Through surrogate models the score is very widely applicable\n\nProblems:\n---------------\n- There could be more focus on the limitations of the method, at least in the Discussion Section. In particular, the score is somewhat ad hoc. Nowadays, new importance measures are often developed in a principled way, by defining a population quantity to which the estimate should converge in large samples. Not doing that can have the disadvantage of very unexpected behavior of the measure, as what we are actually measuring might not be understood (see my comments below). While I think this is ok for this paper, I also think it should be highlighted more.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper develops a new feature importance for trees and forests that measures fairness of the features\nwith respect to some protected attribute. To this end, the paper adapts the MDI measure of Random Forests to instead measure \nthe additional bias when splitting on a certain covariate $j$. After introducing the measure, an extensive empirical analysis studies the behavior of the measure in simulated and real data scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I think the limitations of the method should be better addressed: The MDI is known to have several problems, both in theory and practice, see e.g., the references in Section 2.1 in Bénard et al (2022): \"... in the last two decades, many empirical analyses have highlighted the flaws of the MDI (Strobl et al., 2007). Although Li et al. (2019), Zhou and Hooker (2021), and Loecher (2020) recently improved the MDI to partially remove its bias, Scornet (2020) demonstrated that the MDI is consistent under a strong and restrictive assumption: the regression function is additive and the covariates are independent.\" These problems arise especially when features are correlated (i.e. almost always in practice). Having a look at this literature, does the FairTreeFIS have the same problems? (I am especially thinking of the empirical flaws of MDI). If yes, they should be at the very least discussed as shortcomings in the discussion. If not, what is the difference?\n\n- Related to the above, what happens if a feature that contributes to bias is highly correlated with another one? Will FairTreeFIS (falsely) flag both as having a negative fairness value? I tried to understand this from the supplementary material, but even just a simpler simulation with two correlated covariates might be helpful here.\n\n- Is it possible to have several protected features in z?\n\n- Detail: I find the colors in the plots a bit confusing: It could help to refer again that the colors correspond to different groups G_1 to G_4 in the caption or on the side.\n\n\nBénard, C., Da Veiga, S., and Scornet, E. (2022).\nMean decrease accuracy for random forests: inconsis-\ntency, and a practical solution via the Sobol-MDA.\nBiometrika, 109(4):881–900.", "broader_impact_concerns": "There are no concerns from my side.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qchwg5Iduj", "reviewer": "Reviewer_3jQx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The article deals with fairness in machine learning, a topic of high importance in the community. The structure of the paper is sound, the FairTreeFIS methodology is clearly introduced after the introduction, and the third section presents various experiments to assess the introduced algorithm. \n\nFairTreeFIS is strongly based on the MDI, an existing importance measure for tree-based models (Breiman, 2003). There is a vast literature about the MDI, which is missing in the article. This point is critical because many papers have highlighted flaws of the MDI, and some have introduced improved MDI algorithms. More precisely, practical problems have long been identified by Strobl et al (2007), with strong biases for categorical inputs, or masking effects. Li et al (2019) and Zhou & Hooker (2021) have suggested to use out of sample data to recompute the MDI, and thus improve the accuracy of the MDI. These approaches should be considered and discussed in the article for the MDI adaptation to fairness.\n\nMore importantly, the theoretical groundings of the MDI have been  strongly criticized by Scornet (2023), which shows that the MDI is intrinsically ill-defined: only in the very specific case of independent inputs and additive regression functions, the MDI provides a variance decomposition of the output. Otherwise, we do not even know what the MDI really estimates. Consequently, the MDI cannot provide any precise quantification of input importance, but only a very rough idea whether a given feature has a strong or low influence. FairTreeFIS inherits the same problem, which is even worse in the case of fairness, which involves more complex conditional quantities. In our opinion, this point is critical and should be discussed in the paper. In particular, the baseline procedure were the MDI is run, and the correlations (or non-linear dependance metrics) of features with the protected variables are computed, may provide similar information as FairTreeFIS. \n\nThere is a growing consensus in the community that algorithmic definition of variable importance are unclear, and that theoretical quantities should be first defined instead. Then, algorithms should be built to estimate these quantities. In particular, this approach is strongly supported by Williamson et al (2021) and Candes et al. (2018), where targeted quantities are clearly stated.\n\n\n\n# References\n\nB.D. Williamson, P.B. Gilbert, N.R. Simon, and M. Carone. A general framework for inference on algorithm-agnostic variable importance. Journal of the American Statistical Association, pages 1–38, 2021.\n\nE. Candes, Y. Fan, L. Janson, and J. Lv. Panning for gold:‘model-X’knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society Series B: Statistical Methodology, 80:551–577, 2018.\n\nScornet, E. Trees, forests, and impurity-based variable importance in regression.\"Annales de l'Institut Henri Poincare (B) Probabilites et statistiques. Vol. 59. No. 1. Institut Henri Poincaré, 2023.\n\nBreiman, L. (2003). Setting up, using, and understanding random forests v3.1. Technical report, UC Berkeley, Department of Statistics.\n\nC. Strobl, A.-L. Boulesteix, A. Zeileis, and T. Hothorn. Bias in random forest variable importance measures: illustrations, sources and a solution. BMC Bioinformatics, 8:25, 2007.\n\nX. Li, Y. Wang, S. Basu, K. Kumbier, and B. Yu. A debiased MDI feature importance measure for random forests. In Advances in Neural Information Processing Systems, volume 32, pages 8049–8059, New York, 2019. Curran Associates, Inc.\n\nZ. Zhou and G. Hooker. Unbiased measurement of feature importance in tree-based methods.\nACM Transactions on Knowledge Discovery from Data, 15:1–21, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The article introduces a variable importance measure for decision trees and tree ensembles to quantify the impact of each input feature on the algorithm fairness. This new importance score identifies features that are unfair with respect to protected features (typically gender or race in the experiments).\n\nThe proposed algorithm is called FairTreeFIS, and is directly based on the Mean Decrease Impurity (MDI), introduced by Breiman (2003). The MDI is defined as the sum of weighted decreases of impurity over all tree nodes that split on a given input variable. FairTreeFIS uses the bias decrease at each node instead of the impurity in the case of the MDI, to quantify feature fairness. Both regression and classification problems are handled by FairTreeFIS.\n\nFinally, Section 3 presents the results of several experiments using both simulated and real data, for decision trees, random forests, and boosted tree ensembles. Experiments show that FairTreeFIS identifies features that are influential and unfair with respect to the protected features.\n\nBreiman, L. (2003). Setting up, using, and understanding random forests v3.1. Technical report, UC Berkeley, Department of Statistics.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Strengths And Weaknesses.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RoEPity1ob", "reviewer": "Reviewer_x23E", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. This is (probably) the first metric that is used to measure the contribution of a feature on improving or exacerbating discriminations against different populations.\n2. The introduction is well-written and motivation. The Apple Credit Card example is a good example for the need of this research.\n\n**Weaknesses**\n1. The proposed Fair Feature Importance Score is a direct application of DP or EQOD on the mean decrease in purity. Despite that very intuitive and easy to understand, the novelty of the proposed FairFIS seems not high here.\n2. There is a lack of discussion and study that combine the proposed FairFIS with explanability metrics in existing XAI literatures. For example, how to compare FairFIS with some famous metrics such as LIME (Local Interpretable Model-Agnostic Explanations) or the Shapley values? \n3. In the illustration, the authors plot the FairFIS for each feature. Is FairFIS scalable for large datasets (e.g., with several million rows and hundreds of columns)? The authors claim that FairFIS could also be applied on XGBoost; however, no empirical study is provided.\n4. Figure 1 could be improved for clarify the difference between TreeFIS and FairTreeFIS.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In applications of critical consequences, how to explain why a machine learning model makes fair or unfair decisions across different populations has been an important aspect in responsible machine learning. The authors introduce Fair Feature Importance Score (FairFIS) that measures the contribution of a feature toward or against improving fairness metrics such as demographic parity or equalized odds, especially for decision trees and models originated from them (e.g., boosting algorithms or random forests). The authors measure the Fair Feature Importance Score on synthetic and several real-world datasets to pinpoint features that cause potential discriminations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I will use this space to list some questions I have. I would be great if the authors could address the weaknesses above and my questions here.\n\n**Questions**\n1. In the empirical study, the authors seem to completely overlook the fairness-accuracy trade-off. In practice, it is often undesirable to trade in too much accuracy for fairness. In Figure 5, COMPAS dataset as an example, removing two features cause a 5% accuracy drop. Therefore, directly removing features with the most negative FairTreeFIS values may not be a practical treatment. Moreover in Figure 5, why for most of the datasets, removing one will not decrease the bias as claimed? Also the acc is not affected?\n2. Could the authors clarify what are the importance scores in other studies in Figure 4? \n3. The proposed FairTreeFIS is a single variate analysis; however, in many cases, unfairness or discrimination could be originated from a (non-linear) combination/transformation of several features. How could FairTreeFIS adopt to those scenarios?", "broader_impact_concerns": "There is no ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: One of the main criticism from reviewers is that the proposed metric is derived from MDI, which has been criticized in recent work for its flaws in reliably assessing variable importance in the standard setting (without fairness in mind). I believe the authors have properly addressed this issue (in the sense of TMLR's acceptance criteria) by adding a discussion on the limitations of MDI and the possibility to design other, perhaps more reliable metrics for fair variable importance. As a first attempt at proposing such a metric, the paper may elicit more work on this topic.
Decision: Accept (poster)