Explanations of Black-Box Models based on Directional Feature Interactions

1 INTRODUCTION . The ability to interpret and understand the reasoning behind black box decision-making increases user trust ; it provides insights into how a model is working and , as a consequence , how a model can be improved . This has led to a large body of work on the development of explanation methods ( Ribeiro et al. , 2016 ; Chen et al. , 2018 ; Yoon et al. , 2018 ; Lundberg & Lee , 2017 ) applied to blackbox models . Intuitively , such methods aim to explain black-box behavior by understanding how individual features influence prediction outcomes . Recently , Covert et al . Covert et al . ( 2020a ) proposed a unifying mathematical framework capturing a broad array of explainability techniques , termed Removal-based Explanation methods . Nevertheless , the overwhelming majority of existing literature on explainability has a significant drawback : present methods only provide univariate explanations and , as a result , they do not take into account feature interactions . This is problematic precisely because many black box models , such as deep neural networks , perform well by creating complex structures and combining features in their latent layers . To address this , recent methods have been proposed to learn the interaction between features ( Sundararajan et al. , 2020a ; Maas et al. , 2011 ) . Their definition of interaction assumes features affect each other symmetrically ; however , in many real-world applications , feature interactions may be asymmetrical . We also observe this experimentally ( Fig . 1 ) , and argue of the importance of developing black-box interactions that not only capture interactions , but also incorporate asymmetry . Overall , we make the following contributions : • We propose a method to extend any given univariate removal-based explanation to a bivariate explanation model that can capture asymmetrical feature interactions , represented as a directed graph . Our method is general , and can be applied to a broad array of univariate removal-based explanations , as defined by Covert et al . ( 2020a ) . • We show that analyzing this graph gives a semantically-rich interpretation of black boxes . In particular , beyond the ability to identify most influential features , the graph can identify directionally redundant features , i.e. , features whose presence negates the influence of other features , as well as mutually redundant features , i.e. , features that are interchangeable . These two concepts can not be captured by neither univariate nor symmetric bivariate explanations in existing literature . • We systematize the analysis of the directed explanation graph , providing both formal definitions of the aforementioned notions as well as algorithms for scrutinizing and explaining black-box model behavior . We also provide theoretical justification for these definitions in the context of SHAP , the Shapley value explanation map introduced by Lundberg & Lee ( 2017 ) . • Finally , extensive experiments on MNIST , CIFAR 10 , IMDB , Census , Divorce , Drug , and gene expression data show that our explanation graph outperforms prior symmetrical interaction explainers as well as univariate explainers with respect to post-hoc accuracy , AUC , and time . 2 RELATED WORK . Many methods have been proposed for explaining black box models ( Guidotti et al. , 2018 ) . For instance , LIME ( Ribeiro et al. , 2016 ) explains the prediction of a model by learning a linear model locally around a sample , through which it quantifies feature influences on the prediction . SHAP ( Lundberg & Lee , 2017 ) learns an influence score for each feature based on the Shapley value ( Shapley , 2016 ) . L2X ( Chen et al. , 2018 ) learns a set of most influential features per sample based on the mutual information between features and labels . Recently , Covert et al . ( 2020a ) unified many of such explanation methods under a single framework ; we present this in detail in Sec . 3 . However , all of these methods only capture the univariate influences of each feature ; i.e. , they do not explain feature interactions . Discovering feature interactions has drawn recent interest in machine learning ( Bondell & Reich , 2008 ; Chormunge & Jena , 2018 ; Zeng & Figueiredo , 2014 ; Janizek et al. , 2021 ; Zhang et al. , 2021 ; Tsang et al. , 2018 ; 2020 ) . Tsang et al . ( 2017 ) proposed a framework to detect statistical interactions in a feed-forward neural network by directly interpreting the weights of the model . Cui et al . ( 2019 ) proposed a non-parametric probabilistic method to detect global interactions . However , most such methods study feature interactions globally ( for all instances ) . In contrast , our work detects interactions per individual instance . The work more related to explainability of a black box via feature interaction is Shapley interaction . Grabisch & Roubens ( 1999 ) proposed a Shapley interaction value to explore the interaction between features rather than feature influence . Lundberg et al . ( 2018 ) and Sundararajan et al . ( 2020a ) applied Shapley interaction value to explain black box predictions . Instance-wise Feature Grouping ( Masoomi et al. , 2020 ) explored the effects of feature interaction by allocating features to different groups based on the similarity of their contribution to the prediction task . These methods assume a symmetrical interaction between features ; in contrast , our method provides instance-wise explanations that can capture asymmetrical ( directional ) interactions . Another type of explainers are Graph Neural Network ( GNN ) explainers ( Yuan et al. , 2020 ) . These methods assume that the black-box model has a GNN architecture ; i.e . the model incorporates the input graph structure in its predictions . In contrast , our method allows the black box to be any type of function ( e.g. , CNN , GNN , Random Forest ) and does not assume access to a graph structure : we learn the feature interactions directly from the data . A small subset of GNN explainers , especially local , perturbation-based methods such as ( Yuan et al. , 2021 ; Duval & Malliaros , 2021 ; Luo et al. , 2020 ; Ying et al. , 2019 ) can be applied to black-box models . This can be done by assuming a non-informative interaction structure on the data and allowing the explainers to mask or perturb the interaction edges . However , non-GNN black box models are unable to utilize the graph structure . Causal methods provide explanations through feature influence by utilizing knowledge about causal dependencies . Frye et al . ( 2020 ) generalized the Shapley-value framework to incorporate causality . In particular , they provided a new formulation for computing Shapley value when a partial causal understanding of data is accessible , which they called Asymmetric Shapley values . Wang et al . ( 2021 ) extend this idea to incorporate the entire casual graph to reason about the feature influence on the output prediction . Causal methods rely on prior access to casual relationships ; in contrast , our method learns the asymmetrical interaction between features rather than the casual dependencies . 3 BACKGROUND . In general , explainability methods aim to discover the reason why a black box model makes certain predictions . In the local interpretability setting ( Chen et al. , 2018 ; Lundberg & Lee , 2017 ; Ribeiro et al. , 2016 ; Sundararajan et al. , 2017 ) , which is our main focus , explanations aim to interpret predictions made by the model on an individual sample basis . Typically , this is done by attributing the prediction to sample features that were most influential on the model ’ s output . The latter is discovered through some form of input perturbation ( Lundberg & Lee , 2017 ; Zeiler & Fergus , 2014 ) . In a recent paper , Covert et al . ( 2020a ) proposed a unifying framework for a variety of different explainability methods . As our work generalizes this framework , we formally describe the removal-based explanation framework by Covert et al . below . Univariate Removal-Based Explanations . The unifying framework of Covert et al . identifies three stages in a removal-based explanation method . The first , feature-removal , defines how the method perturbs input samples by removing features ; the second , termed model-behavior , captures the effect that this feature removal has on the black-box model predictions ; finally , the summary stage abstracts the impact of feature subset selection to a numerical score for each feature , capturing the overall influence of the feature in the output . Formally , a black-box model is a function f : X → Y , mapping input features x ∈ X ⊆ Rd to labels y ∈ Y . Let D ≡ { 1 , . . . , d } be the coordinates of the feature space . Given an input x ∈ X and a subset of features S ⊆ D , we denote by xS = [ xi ] i∈S ∈ Rd the projection of x to the features in S. In the local interpretability setting , we are given a black-box model f , an input x ∈ X and in some methods the additional ground truth label y ∈ Y , and wish to interpret the output f ( x ) produced by the model . The three stages of the removal based model are defined by a triplet of functions ( F , u , E ) , which we define below . First , the feature removal stage is defined by a subset function F : X × P ( D ) → Y , ( 1 ) where P ( D ) = 2D is the power set of D. Given an input x ∈ X , and a set of features S ⊆ D , the map F ( x , S ) indicates the label generated by the model when feature subset S is given . For example , several interpretability methods ( Yoon et al. , 2018 ; Chen et al. , 2018 ) set F ( x , S ) = f ( [ 0 ; xS ] ) , i.e. , replace the “ removed ” coordinates with zero . Other methods ( Lundberg et al. , 2020 ; Covert et al. , 2020c ) remove features by marginalizing them out using their conditional distribution p ( XS̄ |XS = xS ) , where S̄ = D \ S , i.e. , F ( x , S ) = E [ f ( X ) |XS = xs ] . Having access to the subset function F , the model behavior stage defines a utility function u : P ( D ) → R ( 2 ) quantifying the utility of a subset of features S ⊆ D. For instance , some methods ( Covert et al. , 2020c ; Schwab & Karlen , 2019 ) calculate the prediction loss between the true label y for an input x , using a loss function ℓ , i.e. , u ( S ) = −ℓ ( F ( x , S ) , y ) . Other methods ( Chen et al. , 2018 ; Yoon et al. , 2018 ) compute the expected loss for a given input x using the label ’ s conditional distribution , i.e. , u ( S ) = −Ep ( Y |X=x ) [ ℓ ( F ( x , S ) , Y ) ] . The utility function can be difficult to interpret due to the exponential number of possible feature subsets . This is addressed by the summary stage as follows . Let U = { u : P ( D ) → R } be the set of all possible utility functions . An explanation map is a function E : U → Rd ( 3 ) mapping a utility function to a vector of scores , one per feature in D. These scores succinctly summarize each feature ’ s value and are the final explanations produced by the removal-based explainability algorithm ( F , u , E ) . For instance , some methods ( Zeiler & Fergus , 2014 ; Petsiuk et al . ; Schwab & Karlen , 2019 ) define E ( u ) i = u ( D ) − u ( D \ { i } ) , or E ( u ) i = u ( { i } ) − u ( ∅ ) ( Guyon & Elisseeff , 2003 ) . Some methods learn E ( u ) by solving an optimization ( Chen et al. , 2018 ; Ribeiro et al. , 2016 ; Yoon et al. , 2018 ) . For instance L2X defines E ( u ) = argmax S : |S|=k u ( S ) , for a given k. The Shapley Value Explanation Map . In our experiments , we focus on Shapley value explanation maps . Shapley ( Shapley , 2016 ) introduced the Shapley value in coalition/cooperative games as a means to compute “ payouts ” , i.e. , ways to distribute the value of a coalition to its constituent members . Treating features as players in such a coalition , the Shapley value has been used by many research works on explainability to compute feature influence ( Datta et al. , 2016 ; Lundberg et al. , 2020 ; Covert et al. , 2020b ) . For instance , Lundberg & Lee ( 2017 ) proposed SHAP as a unified measure of feature influence which uses Shapley value for summarization . They also showed that explainers such as LIME , DeepLIFT ( Shrikumar et al. , 2017 ) , LRP ( Bach et al. , 2015 ) , QII ( Datta et al. , 2016 ) can all be described by SHAP under different utility functions . Formally , given a utility function u , the SHAP explanation map E ( u ) has coordinates : E ( u ) i = ∑ S⊆D\ { i } |S| ! ( d−|S|−1 ) ! d ! ( u ( S ∪ { i } ) − u ( S ) ) ( 4 ) where S ∈ P ( D ) . Direct computation of Eq . ( 4 ) is challenging , as the summands grow exponentially as the number of features increases . Štrumbelj & Kononenko ( 2014 ) proposed an approximation with Monte Carlo sampling , known as Shapley sampling values . Later on , Lundberg & Lee ( 2017 ) introduced KernelSHAP and DeepSHAP to compute the shapley values using kernel and neural network approaches respectively , and showed that such methods require fewer evaluations of the original model compared to other methods to obtain similar approximation accuracy .

Summary: In previous studies, Shapley value has been widely used to explain model predictions, in which previous studies generally characterized importance for each feature instance separately. While some works have explored the effect of combinatorial features on prediction, no works have explored how features interact to influence each other, such as whether feature A is redundant if feature B is present.   In order to achieve this goal, this paper offers an intuitive solution. The degree of feature interaction is estimated using a matrix called a bivariate Shapley explanation map. The j-th column and i-th row of the matrix represents how feature i influences the prediction if feature j has been included. Item ij can be seen as a revised Shapley value: rather than summing up the marginal contributions of all coalitions, it only considers them when feature j appears. They then propose four different types of interactions based on this definition: least/most influential features, directional/mutual redundancy. The experimental results indicate that the proposed method can efficiently discover these four types of feature interactions.