COTENABILITY AND CAUSALITY: EXPLAINING FEATURE IMPORTANCE USING SHAPLEY VALUES

SHAP has become a popular post-hoc model interpretation method because of its flexibility and desirable theoretical axioms. However, in practice calculating Shapley values either assumes independence among model features which violates any correlation among features (which we define as cotenability), or assumes a conditional probability structure which does not capture causality and fails to satisfy key axioms. Here we present a graphical interpretation of Shapley values which clarifies assumptions made during Shapley value calculations. We then consider grouping features as a framework to make causal Shapley values cotenable, and explore their properties in both simulated and real data. The goal is to extend Shapley feature importances so that both cotenability and causality are captured, ultimately increasing interpretability of these explanations. 1 Motivation As complex machine learning models are increasingly used in high-stakes settings such as criminal justice and healthcare, being able to explain model output is critical. Post-hoc additive explanation methods like LIME ([15]) are an attractive class of interpretability algorithms because of their flexibility in explaining individual predictions regardless of the underlying model and their seemingly simple and intuitive interpretation. With the SHAP framework unifying additive explanation methods with concepts from game theory ([12]), these additive algorithms can achieve theoretic axioms that are desirable properties for interpretable explanations, further evidence for their usefulness. However, in practice the calculation of Shapley values is under-specified. Empirical Shapley values can fail some desirable theoretic axioms depending on different implementation methods [16]. The essence of the issue centers on whether to use a conditional or marginal distribution to sample features when generating an explanation for a feature. The recent discussion surrounding this choice([1, 2]) has highlighted two desiderata of model explanations that have to be traded off: 1. We want our explanation models to capture the causal effect of a feature on model output (satisfied in SHAP by using marginal distributions to sample features) 2. We want our explanation models to respect feature correlation structure in order to produce what we call cotenable explanations (satisfied in SHAP by using conditional distributions to sample features) This trade-off has been implicitly explored by work studying the axiomatic properties of different Shapley value formulations. Our goal here is to clarify the trade-off using graphical interpretations of Shapley values, and explore how we can move towards satisfying both desiderata by making additional assumptions about the relationships between model features. Related Work The problem of Shapley values failing axiomatic properties depending on their formulation has been examined in detail in [16], including worked examples and proofs of correctness. We review the differences between their Conditional Expectation and Random Baseline Shapley value formulations, which respectively correspond to conditioning and not CIS 700 XAI FINAL PROJECT conditioning on features, in Section 2.1. [9] provide a causal justification for the same arguments in distinguishing conditional and marginal distributions of features, framing them as the difference between observational vs. interventional distributions. [20] similarly take a graphical approach in justifying causal interpretations of partial dependence plots. Both [16] and [9] note that the original formulation of SHAP was ambiguous in its choice of feature distributions, without clear justification of using one over the other. The most recent work by the authors of SHAP have incorporated these critiques of their original formulation into their most recent implementation of the TreeSHAP algorithm [11]. In terms of desirable properties of model explanation, [13] propose a taxonomy of non-causal and causal interpretability where our desiderata loosely correspond to their categories of causal interpretability for "model-based interpretations" (desiderata 1), and causal interpretability for "verifying causal relationships discovered from data" (desiderata 2). Note however that in our desiderata, respecting feature correlation structure does not necessitate a causal interpretation – features may simply be correlated with one another, or grouped based on preferences of the practitioner (e.g. aggregating individual features into a composite metric such as a risk score). [3] modify the KernelSHAP algorithm to better respect feature correlations (desiderata 2), though as [9] note this work approximates Conditional Expectation Shapley, which is known to fail some Shapley axioms. Aims Our work presented here most closely follows the arguments presented in [9] and [20], building off their graphical interpretation of "interventional" Shapley values and extending them to accommodate contenable feature relationships. To do so, we make an assumption that features come in groups, either provided by underlying causal knowledge or correlations present within the data. We then report on how SHAP values can be interpreted in groups as well as explore their empirical properties through simulations and a case study. The rest of this work is organized as follows: • Section 2 gives the graphical foundations of both model-based causality (desiderata 1) and cotenability (desiderata 2) in Shapley values. • Section 3 explores the behavior of Shapley values when features follow a group correlation structure through simulated data. • Section 4 explores the interpretability benefits of grouping Shapley values through a case study using a real-world medical dataset. • Section 5 concludes with discussions on limitations and future work. 2 Graphical Framework 2.1 Shapley Value Preliminaries Here we review notation used for calculating Shapley values for model feature explanation. More details can be found in [12, 16, 11]. Given a collection of N features, a model f : R|N | → R, an example x, and a set function vf,x : 2 |N | → R parameterized by f and x, we are interested in calculating the Shapley value φi for feature i: