Can I trust you more? Model-Agnostic Hierarchical Explanations

1 INTRODUCTION . State-of-the-art machine learning models , such as deep neural networks , are exceptional at modeling complex dependencies in structured data , such as text ( Vaswani et al. , 2017 ; Tai et al. , 2015 ) , images ( He et al. , 2016 ; Huang et al. , 2017 ) , and DNA sequences ( Alipanahi et al. , 2015 ; Zeng et al. , 2016 ) . However , there has been no clear explanation on what type of dependencies are captured in the black-box models that perform so well ( Ribeiro et al. , 2018 ; Murdoch et al. , 2018 ) . In this paper , we make one of the first attempts at solving this important problem through interpreting two forms of structures , i.e. , context-dependent representations and context-free representations . A context-dependent representation is the one in which a model ’ s prediction depends specifically on a data instance level ( such as a sentence or an image ) . In order to illustrate the concept , we consider an example in image analysis . A yellow round-shape object can be identified as the sun or the moon given its context , either bright blue sky or dark night . A context-free representation is one where the representation behaves similarly independent of instances ( i.e. , global behaviors ) . In a hypothetical task of classifying sentiment in sentences , each sentence carries very different meaning , but when “ not ” and “ bad ” depend on each other , their sentiment contribution is almost always positive - i.e. , the structure is context-free . To investigate context-dependent and context-free structure , we lend to existing definitions in interpretable machine learning ( Ribeiro et al. , 2016 ; Kim et al. , 2018 ) . A context-dependent interpretation is a local interpretation of the dependencies at or within the vicinity of a single data instance . Conversely , a context-free interpretation is a global interpretation of how those dependencies behave in a model irrespective of data instances . In this work , we study a key form of dependency : an interaction relationship between the prediction and input features . Interactions can describe arbitrarily complex relationships between these variables and are commonly captured by state-of-the-art models like deep neural networks ( Tsang et al. , 2018 ; Murdoch et al. , 2018 ) . Interactions which are context-dependent or context-free are therefore local or global interactions , respectively . We propose Mahé , a framework for explaining the context-dependent and context-free structures of any complex prediction model , with a focus on explaining neural networks . The context-dependent explanations are built based on recent work on local intepretations ( such as ( Ribeiro et al. , 2016 ; Murdoch et al. , 2018 ; Singh et al. , 2018 ) ) . Specifically , Mahé takes as input a model to explain and a data instance , and returns a hierarchical explanation , a format proposed by Singh et al . ( 2018 ) to show local group-variable relationships used in predictions ( Figure 1 ) . To provide context-free explanations , Mahé generalizes those context-dependent interactions with consistent behavior in a model and determines whether a local representation in the model is responsible for the global behavior . In this case , Mahé takes as input a model and representative data corresponding to an interaction of interest and returns whether or not that interaction is context-free . We conduct experiments on both synthetic datasets and real-world application datasets , which shows that Mahé ’ s context-dependent explanations can significantly outperform state-of-the-art methods for local interaction interpretation , and Mahé is capable of successfully finding context-free explanations of interactions . In addition , we identify promising cases where the methodology for context-free explanations can successfully edit models . Our contributions are as follows : 1 ) Mahé achieves the task of improved context-dependent explanations based on interaction detection and fitting performance and model-agnostic generality , compared to state-of-the-art methods for local interaction interpretation , 2 ) Mahé is the first to provide context-free explanations of interactions in deep learning models , and 3 ) Mahé provides a promising direction for modifying context-free interactions in deep learning models without significant performance degradation . 2 RELATED WORKS . Attribution Interpretability : A common form of interpretation is feature attribution , which is concerned with how features of a data instance contribute to a model output . Within this category , there are two distinct approaches : additive and sensitivity attribution . Additive attribution interprets how much each feature contributes to the model output when these contributions are summed . In contrast , sensitivity attribution interprets how sensitive a model output is to changes in features . Examples of additive attribution techniques include LIME ( Ribeiro et al. , 2016 ) and CD ( Murdoch et al. , 2018 ) . Examples of sensitivity attribution methods include Integrated Gradients ( Sundararajan et al. , 2017 ) , DeepLIFT ( Shrikumar et al. , 2017 ) , and SmoothGrad ( Smilkov et al. , 2017 ) . Unlike previous approaches , Mahé provides additive attribution interpretations that consist of non-additive groups of variables ( interactions ) in addition to the normal additive contributions of each variable . Interaction Interpretability : An interaction in its generic form is a non-additive effect between features on an outcome variable . Only until recently has there been development in interpreting nonadditive interactions despite often being learned in complex machine learning models . The difficulty interpreting non-additive interactions stems from their lack of exact functional identity compared to , for example , a multiplicative interaction . Methods that exist to interpret non-additive interactions are NID ( Tsang et al. , 2018 ) and Additive Groves ( Sorokina et al. , 2008 ) . In contrast , many more methods exist to interpret specific interactions , namely multiplicative ones . Notable methods include CD ( Murdoch et al. , 2018 ) , Tree-Shap ( Lundberg & Lee , 2017 ) , and GLMs with multiplicative interactions ( Purushotham et al. , 2014 ) . Unlike previous methods , our approach provides local interpretations of the more challenging non-additive interaction . Locally Interpretable Model-Agnostic Explanations ( LIME ) : LIME ( Ribeiro et al. , 2016 ) is a very popular type of model interpretation . Its popularity comes from additive attribution interpretations to explain the output of any prediction model . The original and most popular version of LIME uses a linear model to approximate model predictions in the local vicinity of a data instance . Since its introduction , variants of LIME have been proposed , for example Anchors ( Ribeiro et al. , 2018 ) and LIME-SUP ( Hu et al. , 2018 ) . While Anchors generates a form of context-free explanation , its method of selecting fully representative features for a prediction does not consider interactions . For example , Anchors assumes that ( not , bad ) “ virtually guarentees ” a sentiment prediction to be positive , whereas in Mahé this is not necessarily true ; only their interaction is positive ( See Table 6 for an example ) . LIME-SUP touches upon interactions but does not study their interpretation . 3 INTERACTION EXPLANATIONS . Let f ( · ) be a target function ( model ) of interest , e.g . a classifier , and φ ( · ) be a local approximation of f and is interpretable in contrast to f . A common choice for φ is a linear model , which is interpretable in each linear term . Namely , for an data instance x ∈ Rp , weights w ∈ Rp and bias b , interpretations are given by wixi , known as additive attributions ( Lundberg & Lee , 2017 ) from φ ( x ) = p∑ i=1 wixi + b . ( 1 ) Given a set of n data points { x ( i ) } ni=1 that are infinitesimally close or local to x , a linear approximation of D = { ( x ( 1 ) , f ( x ( 1 ) ) ) , . . . , ( x ( n ) , f ( x ( n ) ) ) } will accurately fit to the functional surface of f at the data instance , such that φ ( x ) = f ( x ) . Because it is possible in such scenarios that φ ( x ) = f ( x ) ≈ b , there must be some nonzero distances between x and x ( i ) to obtain informative attribution scores . LIME , as it was originally proposed , uses a linear approximation as above where samples are generated in a nonzero local vicinity of x ( Ribeiro et al. , 2016 ) . The drawback of linear LIME is that there is often an error = |f ( x ) − φ ( x ) | > 0 . For complex models f , the functional surface at x can be nonlinear . Because D consists of x ( i ) with distance d > 0 from x , a closer fit to f ( x ) in its nonlinear vicinity , i.e . { f ( x ( i ) ) } ni=1 , can be achieved with the following generalization of Eq . 1 : φ ( x ) = p∑ i=1 gi ( xi ) + b , ( 2 ) where gi ( · ) can be any function , for example one that is arbitrarily nonlinear . This function is called a generalized additive model ( GAM ) ( Hastie & Tibshirani , 1990 ) , and now attribution scores can be given by gi ( xi ) for each feature i . For the purposes of interpreting individual feature attribution , the GAM may be enough . However , if we would like broader explanations , we can also obtain nonadditive attributions or interactions between variables ( Lou et al. , 2013 ) , which can provide an even better fit to the complex local vicinity . Expanding Eq . 2 with interactions yields : φK ( x ) = p∑ i=1 gi ( xi ) + K∑ i=1 g′i ( xI ) + b , ( 3 ) where g′i ( · ) can again be any function , xI ∈ R|I| are interacting variables corresponding to the variable indices I , and { Ii } Ki=1 is a set ofK interactions . Attribution scores are now generated from both g′i and g ′ i . In this paper , we learn gi and g ′ i using Multilayer Perceptrons ( MLPs ) . φ or φK can be converted to classification by applying a sigmoid function . Adding non-additive interactions , I , that are truly present in the local vicinity increases the representational capacity of φK ( x ) . I corresponds to non-additive interacting features if and only if g′ ( · ) ( Eq . 3 ) can not be decomposed into a sum of |I| arbitrary subfunctions δ , each not depending on a corresponding interacting variable ( Tsang et al. , 2018 ) , i.e . g′ ( xI ) 6= ∑ i∈I δi ( xI\ { i } ) . 4 Mahé FRAMEWORK In this section , we introduce our Mahé framework , which can provide context-dependent and context-free explanations of interactions . To provide context-dependent explanations , we propose to use a two-step procedure that first identifies what variables interact locally , then learns a model of interactions ( as Eq . 3 ) to provide a local interaction score at the data instance in question . The procedure of first detecting interactions then building non-additive models for them has been studied previously ( Lou et al. , 2013 ; Tsang et al. , 2018 ) ; however , previous works have not focused on using the same non-additive models to provide local interaction attribution scores , which enable us to visualize interactions of any size as demonstrated later in §5.2.3 .

Summary: This paper proposes a method named Mahe that can provide hierarchical explanations for a model: including both context-dependent(instance level) and context-free (global) explanations by a local interpretation algorithm. It obtains context-free explanations through generalizing context-dependent interactions to explain global behaviors. The effectiveness is shown through a number of synthetic and real-world data experiments.