Can I Trust the Explainer? Verifying Post-Hoc Explanatory Methods

1 INTRODUCTION . A large number of post-hoc explanatory methods have recently been developed with the goal of shedding light on highly accurate , yet black-box machine learning models ( Ribeiro et al. , 2016a ; Lundberg & Lee , 2017 ; Arras et al. , 2017 ; Shrikumar et al. , 2017 ; Ribeiro et al. , 2016b ; 2018 ; Plumb et al. , 2018 ; Chen et al. , 2018 ) . Among these methods , there are currently at least two widely used perspectives on explanations : feature-additivity ( Ribeiro et al. , 2016a ; Lundberg & Lee , 2017 ; Shrikumar et al. , 2017 ; Arras et al. , 2017 ) and feature-selection ( Chen et al. , 2018 ; Ribeiro et al. , 2018 ; Carter et al. , 2018 ) , which we describe in detail in the sections below . While both shed light on the overall behavior of a model , we show that , when it comes to explaining the prediction on a single input in isolation , i.e. , instance-wise explanations , the two perspectives lead to fundamentally different explanations . In practice , explanatory methods adhering to different perspectives are being directly compared . For example , Chen et al . ( 2018 ) and Yoon et al . ( 2019 ) compare L2X , a feature-selection explainer , with LIME ( Ribeiro et al. , 2016a ) and SHAP ( Lundberg & Lee , 2017 ) , two feature-additivity explainers . We draw attention to the fact that these comparisons may not be coherent , given the fundamentally different explanation targets , and we discuss the strengths and limitations of the two perspectives . Secondly , while current explanatory methods are successful in pointing out catastrophic biases , such as relying on headers to discriminate between pieces of text about Christianity and atheism ( Ribeiro et al. , 2016a ) , it is an open question to what extent they are reliable when the model that they aim to explain ( which we call the target model ) has a less dramatic bias . This is a difficult task , precisely because the ground-truth decision-making process of neural networks is not known . Consequently , when applied to complex neural networks trained on real-world datasets , a prevalent way to evaluate the explainers is to assume that the target models behave reasonably , i.e. , that they did not rely 1Code , generated datasets , and trained models will be released . on irrelevant correlations . For example , in their morphosyntactic agreement paradigm , Pörner et al . ( 2018 ) assume that a model that predicts if a verb should be singular or plural given the tokens before the verb , must be doing so by focusing on a noun that the model had identified as the subject . Such assumptions may be poor , since recent works show a series of surprising spurious correlations in human-annotated datasets , on which neural networks learn to heavily rely ( Gururangan et al. , 2018 ; Glockner et al. , 2018 ; Carmona et al. , 2018 ) . Therefore , it is not reliable to penalize an explainer for pointing to tokens that just do not appear significant to us . We address the above issue by proposing a framework capable of generating evaluation tests for the explanatory methods under the feature-selection perspective . Our tests consist of pairs of ( target model , dataset ) . Given a pair , for each instance in the dataset , the specific architecture of our model allows us to identify a subset of tokens that have zero contribution to the model ’ s prediction on the instance . We further identify a subset of tokens clearly relevant to the prediction . Hence , we test if explainers rank zero-contribution tokens higher than relevant tokens . We instantiated our framework on three pairs of ( target model , dataset ) on the task of multi-aspect sentiment analysis . Each pair corresponds to an aspect and the three models ( of same architecture ) have been trained independently . We highlight that our test is not a sufficient test for concluding the power of explainers in full generality , since we do not know the whole ground-truth behaviour of the target models . Indeed , we do not introduce an explanation generation framework but a framework for generating evaluation tests for which we provide certain guarantees on the behaviour of the target model . Under these guarantees we are able to test the explainers for critical failures . Our framework therefore generates necessary evaluation tests , and our metrics penalize explainers only when we are able to guarantee that they produced an error . To our knowledge , we are the first to introduce an automatic and non-trivial evaluation test that does not rely on speculations on the behavior of the target model . Finally , we evaluate L2X ( Chen et al. , 2018 ) , a feature-selection explainer , under our test . Even though our test is specifically designed for feature-selection explanatory methods , since , in practice , the two types of explainers are being compared , and , since LIME ( Ribeiro et al. , 2016a ) and SHAP ( Lundberg & Lee , 2017 ) are two very popular explainers , we were interested in how the latter perform on our test , even though they adhere to the feature-additivity perspective . Interestingly , we find that , most of the time , LIME and SHAP perform better than L2X . We will detail in Section 5 the reasons why we believe this is the case . We provide the error rates of these explanatory methods to raise awareness of their possible modes of failure under the feature-selection perspective of explanations . For example , our findings show that , in certain cases , the explainers predict the most relevant token to be among the tokens with zero contribution . We will release our test , which can be used off-the-shelf , and encourage the community to use it for testing future work on explanatory methods under the feature-selection perspective . We also note that our methodology for creating this evaluation is generic and can be instantiated on other tasks or areas of research . 2 RELATED WORK . The most common instance-wise explanatory methods are feature-based , i.e. , they explain a prediction in terms of the input unit-features ( e.g. , tokens for text and super-pixels for images ) . Among the feature-based explainers , there are two major types of explanations : ( i ) feature-additive : provide signed weights for each input feature , proportional to the contributions of the features to the model ’ s prediction ( Ribeiro et al. , 2016a ; Lundberg & Lee , 2017 ; Shrikumar et al. , 2017 ; Arras et al. , 2017 ) , and ( ii ) feature-selective : provide a ( potentially ranked ) subset of features responsible for the prediction ( Chen et al. , 2018 ; Ribeiro et al. , 2018 ; Carter et al. , 2018 ) . We discuss these explanatory methods in more detail in Section 3 . Other types of explanations are ( iii ) example-based ( Koh & Liang , 2017 ) : identify the most relevant instances in the training set that influenced the model ’ s prediction on the current input , and ( iv ) human-level explanations ( Camburu et al. , 2018 ; Park et al. , 2018 ; Kim et al. , 2018 ; Bekele et al. , 2018 ) : explanations that are similar to what humans provide in real-world , both in terms of arguments ( human-biases ) and form ( full-sentence natural language ) . In this work , we focus on verifying feature-based explainers , since they represent the majority of current works . While many explainers have been proposed , it is still an open question how to thoroughly validate their faithfulness to the target model . There are four types of evaluations commonly performed : 1 . Interpretable target models . Typically , explainers are tested on linear regression and decision trees ( e.g. , LIME ( Ribeiro et al. , 2016a ) ) or support vector representations ( e.g. , MAPLE ( Plumb et al. , 2018 ) ) . While this evaluation accurately assesses the faithfulness of the explainer to the target model , these very simple models may not be representative for the large and intricate neural networks used in practice . 2 . Synthetic setups . Another popular evaluation setup is to create synthetic tasks where the set of important features is controlled . For example , L2X ( Chen et al. , 2018 ) was evaluated on four synthetic tasks : 2-dim XOR , orange skin , nonlinear additive model , and switch feature . While there is no limit on the complexity of the target models trained on these setups , their synthetic nature may still prompt the target models to learn simpler functions than the ones needed for real-world applications . This , in turn , may ease the job for the explainers . 3 . Assuming a reasonable behavior . In this setup , one identifies certain intuitive heuristics that a high-performing target model is assumed to follow . For example , in sentiment analysis , the model is supposed to rely on adjectives and adverbs in agreement with the predicted sentiment . Crowd-sourcing evaluation is often performed to assert if the features produced by the explainer are in agreement with the model ’ s prediction ( Lundberg & Lee , 2017 ; Chen et al. , 2018 ) . However , neural networks may discover surprising artifacts ( Gururangan et al. , 2018 ) to rely on , even when they obtain a high accuracy . Hence , this evaluation is not reliable for assessing the faithfulness of the explainer to the target model . 4 . Are explanations helping humans to predict the model ’ s behaviour ? In this evaluation , humans are presented with a series of predictions of a model and explanations from different explainers , and are asked to infer the predictions ( outputs ) that the model will make on a separate set of examples . One concludes that an explainer E1 is better than an explainer E2 if humans are consistently better at predicting the output of the model after seeing explanations from E1 than after seeing explanations from E2 ( Ribeiro et al. , 2018 ) . While this framework is a good proxy for evaluating the real-world usage of explanations , it is expensive and requires considerable human effort if it is to be applied on complex real-world neural network models . In contrast to the above , our evaluation is fully automatic , the target model is a non-trivial neural network trained on a real-world task and for which we provide guarantees on its inner-workings . Our framework is similar in scope with the sanity check introduced by Adebayo et al . ( 2018 ) . However , their test filters for the basic requirement that an explainer should provide different explanations for a model trained on real data than when the data and/or model are randomized . Our test is therefore more challenging and requires a stronger fidelity of the explainer to the target model . 3 INSTANCE-WISE EXPLANATIONS . As mentioned before , current explanatory methods adhere to two major perspectives of explanations : Perspective 1 ( Feature-additivity ) : For a model f and an instance x , the explanation of the prediction f ( x ) consists of a set of contributions { wxi ( f ) } i for each feature i of x such that the sum of the contributions of the features in x approximates f ( x ) , i.e. , ∑ i w x i ( f ) ≈ f ( x ) . Many explanatory methods adhere to this perspective ( Arras et al. , 2017 ; Shrikumar et al. , 2017 ; Ribeiro et al. , 2016a ) . For example , LIME ( Ribeiro et al. , 2016a ) learns the weights via a linear regression on the neighborhood ( explained below ) of the instance . Lundberg & Lee ( 2017 ) unified this class of methods by showing that the only set of feature-additive contributions that verify three desired constraints ( local accuracy , missingness , and consistency—we refer to their paper for details ) are given by the Shapley values from game theory : wxi ( f ) = ∑ x′∈x\ { i } |x′| ! ( |x| − |x′| − 1 ) ! |x| ! [ f ( x′ ∪ { i } ) − f ( x′ ) ] , ( 1 ) where the sum enumerates over all subsets x′ of features in x that do not include the feature i , and | · | denotes the number of features of its argument . Thus , the contribution of each feature i in the instance x is an average of its contributions over a neighborhood of the instance . Usually , this neighborhood consists of all the perturbations given by masking out combinations of features in x ; see , e.g. , ( Ribeiro et al. , 2016a ; Lundberg & Lee , 2017 ) . However , Laugel et al . ( 2018 ) show that the choice of the neighborhood is critical , and it is an open question what neighborhood is best to use in practice . Perspective 2 ( Feature-selection ) : For a model f and an instance x , the explanation of f ( x ) consists of a sufficient ( ideally small ) subset S ( x ) of ( potentially ranked ) features that alone lead to ( almost ) the same prediction as the original one , i.e. , f ( S ( x ) ) ≈ f ( x ) . Chen et al . ( 2018 ) ; Carter et al . ( 2018 ) , and Ribeiro et al . ( 2018 ) adhere to this perspective . For example , L2X ( Chen et al. , 2018 ) learns S ( x ) by maximizing the mutual information between S ( x ) and the prediction . However , it assumes that the number of important features per instance , i.e. , |S ( x ) | , is known , which is usually not the case in practice . A downside of this perspective is that it may not always be true that the model relied only on a ( small ) subset of features , as opposed to using all the features . However , this can be the case for certain tasks , such as sentiment analysis . To better understand the differences between the two perspectives , in Figure 1 , we provide the instance-wise explanations that each perspective aims to provide for a hypothetical sentiment analysis regression model , where 0 is the most negative and 1 the most positive score . We note that our hypothetical model is not far , in behaviour , from what real-world neural networks learn , especially given the notorious biases in the datasets . For example , Gururangan et al . ( 2018 ) show that natural language inference neural networks trained on SNLI ( Bowman et al. , 2015 ) may heavily rely on the presence of a few specific tokens in the input , which should not even be , in general , indicators for the correct target class , e.g. , “ outdoors ” for the entailment class , “ tall ” for the neutral class , and “ sleeping ” for the contradiction class . In our examples in Figure 1 , we clearly see the differences between the two perspectives . For the instance x1 , the feature-additive explanation tells us that “ nice ” was the most relevant feature , with a weight of 0.4 , but also that “ good ” had a significant contribution of 0.3 . While for this instance alone , our model relied only on “ nice ” to provide a positive score of 0.7 , it is also true that , if “ nice ” was not present , the model would have relied on “ good ” to provide a score of 0.6 . Thus , we see that the feature-additive perspective aims to provide an average explanation of the model on a neighborhood of the instance , while the feature-selective perspective aims to tell us the pointwise features used by the model on the instance in isolation , such as “ nice ” for instance x1 . An even more pronounced difference between the two perspectives is visible on instance x2 , where the ranking of features is now different . The feature-selective explanation ranks “ good ” and “ nice ” as the two most important features , while on the instance x2 in isolation , the model relied on the tokens “ very ” and “ good ” , that the feature-selection perspective would aim to provide . Therefore , we see that , while both perspectives of explanations give insights into the model ’ s behavior , one perspective might be preferred over the other in different real-world use-cases . In the rest of the paper , we propose a verification framework for the feature-selection perspective of instance-wise explanations .

Summary: The paper proposes a verification method for instance wise feature explanations. The verification framework uses an RCNN to identify two types of tokens a) the tokens that are not predictive of outcome b) the subset of clearly relevant tokens for prediction. The data used for RCNN is a pruned version of the data used to train the  black-box. The pruning eliminates data points to ensure that the tokens not selected by RCNN have no contribution to the outcome and that the model does not exhibit suffer from learning "handshakes". A handshake is defined as the set of tokens that may be spuriously missed because their information is encoded in another relevant token. This proof to identify such data points is shown and the RNN is therefore expected to be able to reliably identify 3 kinds of tokens a) Those that have zero contribution to the outcome. b) Those that definitely have some contribution to the outcome and c) those that could be relevant or noisy. Three instance-wise feature selection methods are compared. Results are provided on 3 metrics. a) % instances for which the most important tokes provided by the explainer is among the non-selected tokens, b) % of instances for which at least one non-selected token is ranked higher than a relevant token, and c) Average number of non-selected tokens ranked higher than any obviously relevant tokens.