Evaluations and Methods for Explanation through Robustness Analysis

1 INTRODUCTION With the significant progress of recent machine learning research , various machine learning models have been being rapidly adopted to countless real-world applications . This rapid adaptation increasingly questions the machine learning model ’ s credibility , fairness , and more generally interpretability . In the line of this research , researchers have explored various notions of model interpretability . Some researchers directly answer the trustability ( Ribeiro et al. , 2016 ) or the fairness of a model ( Zhao et al. , 2017 ) , while some other researchers seek to actually improve the model ’ s performance by understanding the model ’ s weak points ( Koh & Liang , 2017 ) . Even though the goal of such various model interpretability tasks varies , vast majority of them are built upon extracting relevant features for a prediction , so called feature-based explanation . Feature-based explanation is commonly based on measuring the fidelity of the explanation to the model , which is essentially how close the sum of attribution scores for a set of features approximates the function value difference before and after removing the set of features . Depending on their design , the fidelity-based attribution evaluation varies : completeness ( Sundararajan et al. , 2017 ) , sensitivity-n ( Ancona et al. , 2018 ) , infidelity ( Yeh et al. , 2019 ) , and causal local explanation metric ( Plumb et al. , 2018 ) . The idea of smallest sufficient region ( SSR ) and smallest destroying region ( SDR ) ( Fong & Vedaldi , 2017 ; Dabkowski & Gal , 2017 ) is worth noting because it considers the ranking of the feature attribution scores , not the actual score itself . Intuitively , for a faithful attribution score , removing the most salient features would naturally lead to a large difference in prediction score . Therefore , SDR-based evaluations measure how much the function value changes when the most high-valued salient features are removed . Although the aforementioned attribution evaluations made success in many cases , setting features with an arbitrary reference values to zero-out the input is limited , in the sense that it only considers the prediction at the reference value while ignoring the rest of the input space . Furthermore , the choice of reference value inherently introduces bias . For example , if we set the feature value to 0 in rgb images , this introduces a bias in the attribution map that favors the bright pixels . As a result , explanations that optimize upon such evaluations often omit important dark objects and the pertinent negative features in the image , which is the part of the image that does not contain object but is crucial to the prediction ( Dhurandhar et al. , 2018 ) . An alternative way to remove pixels is to use sampling from some predefined distribution or a generative model ( Chang et al. , 2018 ) , which nevertheless could still introduce some bias with respect to the defined distribution . Moreover , they require a generative model that approximates the data distribution , which may not be available in certain domains . In this paper , we remove such inherit bias by taking a different perspective on the input perturbation . We start from an intuition that if a set of features are important to make a specific prediction , keeping them in the same values would preserve the prediction even though other irrelevant features are modified . In other words , the model would be more sensitive on the changes of those important or relevant features than the ones that are not . Unlike the foremost approaches including SDR and SSR that perturbs features to a specific reference point , we consider the minimum norm of perturbation to arbitrary directions , not just to a reference point , that can change model ’ s prediction , also known as “ minimum adversarial perturbation ” in the literature ( Goodfellow et al. , 2014 ; Weng et al. , 2018b ) . Based on this idea , we define new evaluation criteria to test the importance of a set of features . By computing the minimum adversarial perturbation on the complementary set of features that can alter the model ’ s decision , we could test the degree of importance of the set . Although explicitly computing the importance value is NP-hard ( Katz et al. , 2017 ) , Carlini & Wagner ( 2017 ) and Madry et al . ( 2017 ) showed that the perturbations computed by adversarial attacks can serve as reasonably tight upper bounds , which lead to an efficient approximation for the proposed evaluation . Furthermore , we can derive a new explanation framework by formulating the model explanation to a two-player min-max game between explanator and adversarial attacker . The explanator aims to find a set of important features to maximize the minimum perturbation computed by the attacker . This framework empirically performs much better than previous approaches quantitatively , with very inspiring examples . To summarize our contributions : • We define new evaluation criteria for feature-based explanations based on robustness analysis . The evaluation criteria consider the worst case perturbations when a set of features are anchored , which does not introduce bias into the evaluation . • We design efficient algorithms to generate explanations that maximize the proposed criteria , which perform favorably against baseline methods on the proposed evaluation criteria . • Experiments in computer vision and NLP models demonstrate that the proposed explanation can indeed identify some important features that are not captured by previous methods . Furthermore , our method is able to extract a set of features that contrasts the current prediction to a target class . 2 ROBUSTNESS ANALYSIS FOR EVALUATING FEATURE-BASED EXPLANATIONS 2.1 PROBLEM NOTATION Let us consider the following setting : a general K-way classification problem with input space X ⊆ Rd , output space Y = { 1 , . . . , K } , and a predictor function f : X → Y where f ( x ) denotes the output class for some input example x = [ x1 , . . . , xd ] ∈ X . Then , for a particular prediction f ( x ) = y , despite the different forms of existing feature-based explanations ranging from attributing an importance value to each feature , ranking the features by their importance , to simply identify a set of important features , a common goal of them is to extract a compact set of relevant features with respect to the prediction . 2.2 EVALUATION THROUGH ROBUSTNESS ANALYSIS We note that however , given an explanation that identifies a set of said to be relevant features , how can we evaluate the quality of such explanation , or in other words , justify whether the distinguished features are truly relevant to the prediction ? While one generally has no ground truth about the underlying true relevance of the features , recent studies take an axiomatic approach to define what properties the relevant features should hold and evaluate the explanations through verifying if the identified relevant features satisfy the properties . One such properties that is widely adopted in the literature is to assume that the importance of a set of features corresponds to the degree of change in prediction when the features are removed from the original input . Nevertheless , as we discussed in the previous section , the practice of approximating removal of features by setting their value to some reference point poses the risk of introducing bias in the evaluation . As a result , to escape from the caveat , we follow a similar concept but propose two new criteria to evaluate the importance of features based on the following assumptions . Assumption 1 When the values of the most salient features are anchored ( fixed ) , perturbation on the complementary set of features has weaker influence on the model ’ s prediction . In other words , the model could tolerate a larger degree of perturbation on the less important and non-anchored features . Assumption 2 If perturbation is allowed on a set of important features , a small perturbation could easily change the model prediction even when we fix the values for the rest of the features . Based on these two assumptions , we propose a new framework for evaluating explanations . The evaluation is based on the adversarial robustness when a set of features are fixed , which is formally defined below . Definition 2.1 The minimum adversarial perturbation norm on a set of features S , which we will also name as Robustness-S , can be defined as : ∗S = g ( x , S ) = { min δ δp s.t . f ( x+ δ ) ∕= y , δS = 0 } , ( 1 ) where S = U \ S is the complementary set of features , and δS = 0 means that the perturbation value on features in S is constraint to be 0 . Assume that we are given an explanation that partitions the input features into a relevant set Sr and an irrelevant set Sr . Assumption 1 implies that the quality of the relevant set can be measured by ∗ Sr – the robustness of irrelevant set when the relevant set is anchored . Specifically , Assumption 1 infers that a higher robustness on Sr follows from a larger coverage of pertinent features in set Sr ; and thus an explanation is considered better if it leads to a higher robustness against perturbation in Sr. On the other hand , based on Assumption 2 , an explanation that has included important salient features in Sr should lead to a smaller robustness level on ∗Sr . Therefore , Assumption 1 and 2 build up our proposed evaluation criteria Robustness-Sr and Robustness-Sr respectively , as listed below . Robustness-Sr measures the minimum adversarial distortion Sr when the set of important features Sr , typically represented by the high-weight features in an attribution map , are anchored and perturbation is only allowed in low-weight regions . The higher the score the better the explanation . To measure Robustness-Sr , we would need to first determine the size of |Sr| . We can set |Sr| to the amount of anchors that an user is interested in or we may vary the size of |Sr| and evaluate the corresponding Robustness-Sr at different points . By varying the size of |Sr| , we could plot an evaluation curve for each explanation and in turn measure the area under curve ( AUC ) , which corresponds to the average Robustness-Sr at different sizes of relevant set . Robustness-Sr measures the minimum distortion distance Sr when the set of important features Sr are the only region that is perturbable , and the rest of feature values are anchored . Contrary to Robustness-Sr , lower scores on this metric indicate better explanation . We similarly define AUC of Robustness-Sr as the average of Robustness-Sr when we vary the size of |Sr| . Evaluation Procedure Note that both Robustness-Sr and Robustness-Sr are sensitive to the cardinality of Sr. For instance , including all features in Sr will make ∗Sr = 0 . Therefore , we will only use these criteria to evaluate relevant sets with the same cardinality . For example , to evaluate a feature attribution method that assigns a weight with each feature , we can sort the features by the decending order of weights and then for each set of top-K features with K = 1 , 2 , . . . , d , we evaluate Robustness-Sr ( Sr ) and plot a curve . A larger ( smaller ) area under curve indicates a better feature attribution ranking . ( See examples in Figure 1 ) . Untargeted v.s . Targeted Explanation Definition 2.1 corresponds to the untargeted adversarial robustness – a perturbation that changes the predicted class to any label except y is considered as a successful attack . Instead of doing this , our formulation can also extend to targeted adversarial robustness , where we replace ( 1 ) by ∗S , t = { min δ δp s.t . f ( x+ δ ) = t ; δS = 0 } , ( 2 ) where t is the targeted class . Using this definition , our approach will try to address the question “ Why is this example classified as y instead of t ” , and the important features that optimize this criterion will highlight the contrast between class y and t. We will give several interesting results in the experiment section . Comparing to existing measurement The proposed criteria at the first glance look similar to SSR- and SDR-based measurements . We note that , however , the key differences between our proposed criteria and SSR- ( SDR- ) based criteria are in two-folds : 1 ) Conceptually , to measure whether a set of features is important , instead of concerning the prediction change before and after removing the features , we consider whether perturbation on the feature values would significantly alter the prediction . 2 ) Practically , our proposed criteria allow us to eschew the difficulty of modeling feature removal as discussed in section 1 . In fact , as most implementations of removal-based criteria set the values of the features of interest to some fixed reference point , our criteria could be viewed as generalized versions where we consider all possible reference points by allowing perturbations in any directions . As a result , the proposed criteria enjoys a broader view of prediction behavior around the input , and in turn could capture a broader range of important features like the pertinent negative features in Dhurandhar et al . ( 2018 ) , as we shall show in the experiment section . Robustness Evaluation under Fixed Anchor Set It is known that computing the exact minimum distortion distance in modern neural networks is intractable ( Katz et al. , 2017 ) , so many different methods have been developed to estimate the value . Adversarial attacks , such as C & W ( Carlini & Wagner , 2017 ) and PGD attack ( Madry et al. , 2017 ) , aim to find a feasible solution of ( 1 ) , which leads to an upper bound of ∗S . They are based on gradient based optimizers which are usually efficient . On the other hand , neural network verification methods aim to provide a lower bound of ∗S to ensure that the model prediction will not change within certain perturbation range ( Singh et al. , 2018 ; Wong & Kolter , 2018 ; Weng et al. , 2018a ; Gehr et al. , 2018 ; Zhang et al. , 2018 ; Wang et al. , 2018 ; Zhang et al. , 2019 ) . However , these methods are usually time consuming ( often > 50 times slower than a backpropagation ) . The proposed framework can be combined with any method that aims to approximately compute ( 1 ) , including attack , verification , and some other statistical estimations . However , for simplicity we only choose to evaluate ( 1 ) by the state-of-the-art projected gradient descent ( PGD ) attack ( Madry et al. , 2017 ) , since the verification methods are too slow and often lead to much looser estimation as reported in some recent studies ( Salman et al. , 2019 ) . 3 NEW EXPLANATIONS TOWARDS OPTIMIZING THE CRITERIA Given the new evaluation criteria , a natural follow-up question is how to design explanations that optimize the measurements . Recall that under the proposed criteria the goal of an optimal explanation is to maximize ( minimize ) robustness-S̄r ( robustness-Sr ) under the cardinality constraint on Sr. Searching for such explanations thus leads to the following optimization problems , ( 3 ) for Robustness-Sr and ( 4 ) for Robustness-Sr : maximize Sr∈ { 0,1 } d g ( x , Sr ) subject to Sr0 ≤ K , ( 3 ) minimize Sr∈ { 0,1 } d g ( x , Sr ) subject to Sr0 ≤ K , ( 4 ) where g ( x , S ) computes the value in Eq . ( 1 ) , the minimum distortion distance when the features in set S̄r is not allowed to be perturbed , and K is a pre-defined size constraint on the set Sr. 3.1 GREEDY ALGORITHM Directly solving ( 3 ) and ( 4 ) is challenging since g is an implicit function computed by solving ( 1 ) approximately , and furthermore , the discrete input constraint makes it intractible to find the optimal solution . As a result , we propose a greedy-styled algorithm , where we iteratively add the most promising feature into Sr that optimizes the objective at each local step until Sr reaches the size constraint . In other words , we initialize the set Sr as empty , and sequentially solve the following subproblem at every step t : argmax i g ( x , Str ∪ i ) , or argmin i g ( x , Str ∪ i ) , ∀i ∈ Sr ( 5 ) where Str is the anchor set at step t , and S 0 r = ∅ . We repeat this subprocedure until the size of set Str reaches K. We name this method as Greedy . A straightforward way for solving ( 5 ) is to exhaustively search over every single feature . However , considering a single feature at a time ignores the correlation between features , which tends to introduce noise ( see our experimental results ) . If

Summary: The manuscript proposes a method for model explanation and two metrics for the evaluation of methods for model explanation based on robustness analysis. More specifically, two complementary, yet very related, criteria are proposed: i) robustness to perturbations on irrelevant features and ii) robustness to perturbations in relevant features. Moreover, different from existing works which defined the perturbation values following different somewhat-fixed procedures, the proposed method aims at allowing perturbations in any directions.