Explanations of Black-Box Models based on Directional Feature Interactions

1 INTRODUCTION . The ability to interpret and understand the reasoning behind black box decision-making increases user trust ; it provides insights into how a model is working and , as a consequence , how a model can be improved . This has led to a large body of work on the development of explanation methods ( Ribeiro et al. , 2016 ; Chen et al. , 2018 ; Yoon et al. , 2018 ; Lundberg & Lee , 2017 ) applied to blackbox models . Intuitively , such methods aim to explain black-box behavior by understanding how individual features influence prediction outcomes . Recently , Covert et al . Covert et al . ( 2020a ) proposed a unifying mathematical framework capturing a broad array of explainability techniques , termed Removal-based Explanation methods . Nevertheless , the overwhelming majority of existing literature on explainability has a significant drawback : present methods only provide univariate explanations and , as a result , they do not take into account feature interactions . This is problematic precisely because many black box models , such as deep neural networks , perform well by creating complex structures and combining features in their latent layers . To address this , recent methods have been proposed to learn the interaction between features ( Sundararajan et al. , 2020a ; Maas et al. , 2011 ) . Their definition of interaction assumes features affect each other symmetrically ; however , in many real-world applications , feature interactions may be asymmetrical . We also observe this experimentally ( Fig . 1 ) , and argue of the importance of developing black-box interactions that not only capture interactions , but also incorporate asymmetry . Overall , we make the following contributions : • We propose a method to extend any given univariate removal-based explanation to a bivariate explanation model that can capture asymmetrical feature interactions , represented as a directed graph . Our method is general , and can be applied to a broad array of univariate removal-based explanations , as defined by Covert et al . ( 2020a ) . • We show that analyzing this graph gives a semantically-rich interpretation of black boxes . In particular , beyond the ability to identify most influential features , the graph can identify directionally redundant features , i.e. , features whose presence negates the influence of other features , as well as mutually redundant features , i.e. , features that are interchangeable . These two concepts can not be captured by neither univariate nor symmetric bivariate explanations in existing literature . • We systematize the analysis of the directed explanation graph , providing both formal definitions of the aforementioned notions as well as algorithms for scrutinizing and explaining black-box model behavior . We also provide theoretical justification for these definitions in the context of SHAP , the Shapley value explanation map introduced by Lundberg & Lee ( 2017 ) . • Finally , extensive experiments on MNIST , CIFAR 10 , IMDB , Census , Divorce , Drug , and gene expression data show that our explanation graph outperforms prior symmetrical interaction explainers as well as univariate explainers with respect to post-hoc accuracy , AUC , and time . 2 RELATED WORK . Many methods have been proposed for explaining black box models ( Guidotti et al. , 2018 ) . For instance , LIME ( Ribeiro et al. , 2016 ) explains the prediction of a model by learning a linear model locally around a sample , through which it quantifies feature influences on the prediction . SHAP ( Lundberg & Lee , 2017 ) learns an influence score for each feature based on the Shapley value ( Shapley , 2016 ) . L2X ( Chen et al. , 2018 ) learns a set of most influential features per sample based on the mutual information between features and labels . Recently , Covert et al . ( 2020a ) unified many of such explanation methods under a single framework ; we present this in detail in Sec . 3 . However , all of these methods only capture the univariate influences of each feature ; i.e. , they do not explain feature interactions . Discovering feature interactions has drawn recent interest in machine learning ( Bondell & Reich , 2008 ; Chormunge & Jena , 2018 ; Zeng & Figueiredo , 2014 ; Janizek et al. , 2021 ; Zhang et al. , 2021 ; Tsang et al. , 2018 ; 2020 ) . Tsang et al . ( 2017 ) proposed a framework to detect statistical interactions in a feed-forward neural network by directly interpreting the weights of the model . Cui et al . ( 2019 ) proposed a non-parametric probabilistic method to detect global interactions . However , most such methods study feature interactions globally ( for all instances ) . In contrast , our work detects interactions per individual instance . The work more related to explainability of a black box via feature interaction is Shapley interaction . Grabisch & Roubens ( 1999 ) proposed a Shapley interaction value to explore the interaction between features rather than feature influence . Lundberg et al . ( 2018 ) and Sundararajan et al . ( 2020a ) applied Shapley interaction value to explain black box predictions . Instance-wise Feature Grouping ( Masoomi et al. , 2020 ) explored the effects of feature interaction by allocating features to different groups based on the similarity of their contribution to the prediction task . These methods assume a symmetrical interaction between features ; in contrast , our method provides instance-wise explanations that can capture asymmetrical ( directional ) interactions . Another type of explainers are Graph Neural Network ( GNN ) explainers ( Yuan et al. , 2020 ) . These methods assume that the black-box model has a GNN architecture ; i.e . the model incorporates the input graph structure in its predictions . In contrast , our method allows the black box to be any type of function ( e.g. , CNN , GNN , Random Forest ) and does not assume access to a graph structure : we learn the feature interactions directly from the data . A small subset of GNN explainers , especially local , perturbation-based methods such as ( Yuan et al. , 2021 ; Duval & Malliaros , 2021 ; Luo et al. , 2020 ; Ying et al. , 2019 ) can be applied to black-box models . This can be done by assuming a non-informative interaction structure on the data and allowing the explainers to mask or perturb the interaction edges . However , non-GNN black box models are unable to utilize the graph structure . Causal methods provide explanations through feature influence by utilizing knowledge about causal dependencies . Frye et al . ( 2020 ) generalized the Shapley-value framework to incorporate causality . In particular , they provided a new formulation for computing Shapley value when a partial causal understanding of data is accessible , which they called Asymmetric Shapley values . Wang et al . ( 2021 ) extend this idea to incorporate the entire casual graph to reason about the feature influence on the output prediction . Causal methods rely on prior access to casual relationships ; in contrast , our method learns the asymmetrical interaction between features rather than the casual dependencies . 3 BACKGROUND . In general , explainability methods aim to discover the reason why a black box model makes certain predictions . In the local interpretability setting ( Chen et al. , 2018 ; Lundberg & Lee , 2017 ; Ribeiro et al. , 2016 ; Sundararajan et al. , 2017 ) , which is our main focus , explanations aim to interpret predictions made by the model on an individual sample basis . Typically , this is done by attributing the prediction to sample features that were most influential on the model ’ s output . The latter is discovered through some form of input perturbation ( Lundberg & Lee , 2017 ; Zeiler & Fergus , 2014 ) . In a recent paper , Covert et al . ( 2020a ) proposed a unifying framework for a variety of different explainability methods . As our work generalizes this framework , we formally describe the removal-based explanation framework by Covert et al . below . Univariate Removal-Based Explanations . The unifying framework of Covert et al . identifies three stages in a removal-based explanation method . The first , feature-removal , defines how the method perturbs input samples by removing features ; the second , termed model-behavior , captures the effect that this feature removal has on the black-box model predictions ; finally , the summary stage abstracts the impact of feature subset selection to a numerical score for each feature , capturing the overall influence of the feature in the output . Formally , a black-box model is a function f : X → Y , mapping input features x ∈ X ⊆ Rd to labels y ∈ Y . Let D ≡ { 1 , . . . , d } be the coordinates of the feature space . Given an input x ∈ X and a subset of features S ⊆ D , we denote by xS = [ xi ] i∈S ∈ Rd the projection of x to the features in S. In the local interpretability setting , we are given a black-box model f , an input x ∈ X and in some methods the additional ground truth label y ∈ Y , and wish to interpret the output f ( x ) produced by the model . The three stages of the removal based model are defined by a triplet of functions ( F , u , E ) , which we define below . First , the feature removal stage is defined by a subset function F : X × P ( D ) → Y , ( 1 ) where P ( D ) = 2D is the power set of D. Given an input x ∈ X , and a set of features S ⊆ D , the map F ( x , S ) indicates the label generated by the model when feature subset S is given . For example , several interpretability methods ( Yoon et al. , 2018 ; Chen et al. , 2018 ) set F ( x , S ) = f ( [ 0 ; xS ] ) , i.e. , replace the “ removed ” coordinates with zero . Other methods ( Lundberg et al. , 2020 ; Covert et al. , 2020c ) remove features by marginalizing them out using their conditional distribution p ( XS̄ |XS = xS ) , where S̄ = D \ S , i.e. , F ( x , S ) = E [ f ( X ) |XS = xs ] . Having access to the subset function F , the model behavior stage defines a utility function u : P ( D ) → R ( 2 ) quantifying the utility of a subset of features S ⊆ D. For instance , some methods ( Covert et al. , 2020c ; Schwab & Karlen , 2019 ) calculate the prediction loss between the true label y for an input x , using a loss function ℓ , i.e. , u ( S ) = −ℓ ( F ( x , S ) , y ) . Other methods ( Chen et al. , 2018 ; Yoon et al. , 2018 ) compute the expected loss for a given input x using the label ’ s conditional distribution , i.e. , u ( S ) = −Ep ( Y |X=x ) [ ℓ ( F ( x , S ) , Y ) ] . The utility function can be difficult to interpret due to the exponential number of possible feature subsets . This is addressed by the summary stage as follows . Let U = { u : P ( D ) → R } be the set of all possible utility functions . An explanation map is a function E : U → Rd ( 3 ) mapping a utility function to a vector of scores , one per feature in D. These scores succinctly summarize each feature ’ s value and are the final explanations produced by the removal-based explainability algorithm ( F , u , E ) . For instance , some methods ( Zeiler & Fergus , 2014 ; Petsiuk et al . ; Schwab & Karlen , 2019 ) define E ( u ) i = u ( D ) − u ( D \ { i } ) , or E ( u ) i = u ( { i } ) − u ( ∅ ) ( Guyon & Elisseeff , 2003 ) . Some methods learn E ( u ) by solving an optimization ( Chen et al. , 2018 ; Ribeiro et al. , 2016 ; Yoon et al. , 2018 ) . For instance L2X defines E ( u ) = argmax S : |S|=k u ( S ) , for a given k. The Shapley Value Explanation Map . In our experiments , we focus on Shapley value explanation maps . Shapley ( Shapley , 2016 ) introduced the Shapley value in coalition/cooperative games as a means to compute “ payouts ” , i.e. , ways to distribute the value of a coalition to its constituent members . Treating features as players in such a coalition , the Shapley value has been used by many research works on explainability to compute feature influence ( Datta et al. , 2016 ; Lundberg et al. , 2020 ; Covert et al. , 2020b ) . For instance , Lundberg & Lee ( 2017 ) proposed SHAP as a unified measure of feature influence which uses Shapley value for summarization . They also showed that explainers such as LIME , DeepLIFT ( Shrikumar et al. , 2017 ) , LRP ( Bach et al. , 2015 ) , QII ( Datta et al. , 2016 ) can all be described by SHAP under different utility functions . Formally , given a utility function u , the SHAP explanation map E ( u ) has coordinates : E ( u ) i = ∑ S⊆D\ { i } |S| ! ( d−|S|−1 ) ! d ! ( u ( S ∪ { i } ) − u ( S ) ) ( 4 ) where S ∈ P ( D ) . Direct computation of Eq . ( 4 ) is challenging , as the summands grow exponentially as the number of features increases . Štrumbelj & Kononenko ( 2014 ) proposed an approximation with Monte Carlo sampling , known as Shapley sampling values . Later on , Lundberg & Lee ( 2017 ) introduced KernelSHAP and DeepSHAP to compute the shapley values using kernel and neural network approaches respectively , and showed that such methods require fewer evaluations of the original model compared to other methods to obtain similar approximation accuracy .

Summary: This paper attempts to alleviate the shortcomings of existing local-feature-interaction explainers that assume symmetrical feature interactions by proposing a bivariate feature-explanation map that can capture asymmetrical (directional) feature interactions; this analysis provides evidence of mutual and directional redundancy, offering a comprehensive understanding of the features most influential for a given prediction. This proposed approach can also be instantiated using any univariate feature-based explanation method. Empirical results on image, text, and tabular data show the ability of the proposed method to accurately identify mutual and directional redundancies.