Locally Invariant Explanations: Towards Causal Explanations through Local Invariant Learning

1 INTRODUCTION . Deployment and usage of neural black-box models has significantly grown in industry over the last few years creating the need for new tools to help users understand and trust models ( Gunning , 2017 ) . Even well-studied application domains such as image recognition require some form of prediction understanding in order for the user to incorporate the model into any important decisions ( Simonyan et al. , 2013 ; Lapuschkin et al. , 2016 ) . An example of this could be a doctor who is given a cancer diagnosis based on an image scan . Since the doctor holds responsibility for the final diagnosis , the model must provide sufficient reason for its prediction . Even new text categorization tasks ( Feng et al. , 2018 ) are becoming important with the growing need for social media companies to provide better monitoring of public content . Twitter recently began monitoring tweets related to COVID-19 in order to label tweets containing misleading information , disputed claims , or unverified claims ( Roth & Pickles , 2020 ) . Laws will likely emerge requiring explanations for why red flags were or were not raised in many examples . In fact , the General Data Protection and Regulation ( GDPR ) ( Yannella & Kagan , 2018 ) act passed in Europe already requires automated systems that make decisions affecting humans to be able to explain them . Given this acute need , a number of methods have been proposed to explain local decisions ( i.e . example specific decisions ) of classifiers ( Ribeiro et al. , 2016 ; Lundberg & Lee , 2017 ; Simonyan et al. , 2013 ; Lapuschkin et al. , 2016 ; Dhurandhar et al. , 2018a ) . Locally interpretable model-agnostic explanations ( LIME ) is arguably the most well-known local explanation method that requires only query ( or black-box ) access to the model . Although LIME is a popular method , it is known to be sensitive to certain design choices such as i ) ( random ) sampling to create the ( perturbation ) neighborhood1 , ii ) the size of this neighborhood ( number of samples ) and iii ) ( local ) fitting procedure to learn the explanation model ( Molnar , 2019 ; Zhang et al. , 2019b ) . The first , most serious issue could lead to nearby examples having drastically different explanations making effective recourse a challenge . One possible mitigation is to increase the neighborhood size , however , one can not arbitrarily do so as it not only leads to higher computational cost , but in today ’ s cloud computing-driven world it could have direct monetary implications where every query to a black-box model has an associated cost ( Dhurandhar et al. , 2019 ) . There have been variants suggested to overcome some of these limitations ( Botari et al. , 2020 ; Shrotri et al. , 2021 ; Plumb et al. , 2018 ) primarily through mechanisms that create realistic neighborhoods or through adversarial training ( Lakkaraju et al. , 2020 ) , however , their efficacy is restricted to certain settings and modalities based on their assumptions and training strategies . In this paper we introduce a new method called Locally INvariant EXplanations ( LINEX ) inspired by the invariant risk minimization ( IRM ) principle ( Arjovsky et al. , 2019 ) , that produces feature based explanations that are robust to neighborhood sampling and can recover faithful ( i.e . mimic black-box behavior ) , stable ( i.e . similar for closeby examples ) and unidirectional ( i.e . same sign attributions for closeby examples , see section 4.1 ) explanations across tabular , image , and text modalities . In particular , we show that our method performs better than the competitors for random as well as realistic neighborhood generation , where in some cases even with the prior strategy our explanation quality is close to methods that employ the latter . Qualitatively , our method highlights ( local ) features as important that in the particular locality i ) have consistently high gradient with respect to ( w.r.t . ) the black-box function and ii ) where the gradient does not change significantly , especially in sign . Such stable behavior for LINEX is visualized in Figure 1 , where we get similar explanations for nearby examples in the IRIS dataset . The ( in ) fidelity of LINEX is still similar to LIME ( see Table 3 ) , but of course our explanations are much more stable . 2 RELATED WORK . Posthoc explanations can typically be partitioned into two broad categories global and local . Global explainability avers to trying to understand a black-box model at a holistic level where the typical tact is knowledge transfer ( Hinton et al. , 2015 ; Dhurandhar et al. , 2018b ; 2020 ) where ( soft/hard ) labels of the black-box model are used to train an interpretable model such as a decision tree or rule list ( Rudin , 2019 ) . Local explanations on the other hand avers to understanding individual decisions . These explanations are typically in two forms , either exemplar based or feature based . For exemplar based as the name suggests similar but diverse examples ( Kim et al. , 2016 ; Gurumoorthy et al. , 2019 ) are provided as explanations for the input in question . While for feature based ( Ribeiro et al. , 2016 ; Lundberg & Lee , 2017 ; Dhurandhar et al. , 2018a ; Lapuschkin et al. , 2016 ) , which is the focus of this work , important features are returned as being important for the decision made for the input . There are some methods that do both ( Plumb et al. , 2018 ) . Moreover , there are methods which provide explanations that are local , global as well as at a group level ( Ramamurthy et al. , 2020 ) . All of these methods though may not still provide stable and robust local feature based explanations which can be desirable in practice ( Ghorbani et al. , 2019 ) . 1By perturbation neighborhood – referred to as simply neighborhood – we mean neighborhoods generated for local explanations . By exemplar neighborhood , we mean nearest examples in a dataset to a given example . Given this there have been more recent works that try to learn either robust or even causal explanations . In ( Lakkaraju et al. , 2020 ) the authors try to learn robust and stable local explanations relative to distribution shifts and adversarial attacks . However , the distribution shifts they consider are linear shifts and adversarial training is performed which can be slow and sometimes unstable ( Zhang et al. , 2019a ) . Moreover , the method seems to be applicable primarily to tabular data . Works on causal explanations ( Frye et al. , 2020 ; Heskes et al. , 2020 ) mainly modify SHAP and assume access to a partial causal graph . Some others ( Vig et al. , 2020 ) assume white-box access . In this work we do not assume availability of such additional information . There are also works which show that creating realistic neighborhoods by learning the data manifold for LIME ( Botari et al. , 2020 ; Shrotri et al. , 2021 ) can lead to better quality explanations , where in a particular work ( Anders et al. , 2020 ) it is suggested that projecting explanations themselves on to the manifold can also make them more robust . The need for stability in a exemplar neighborhood for LIME like methods has been highlighted in ( Zhang et al. , 2019b ) , with the general desire for stable explanations being also expressed in prior works ( Yeh et al. , 2019 ; Visani et al. , 2020 ) . 3 PRELIMINARIES . Invariant Risk Minimization : Given a collection of training datasets D = { De } e∈Etr gathered from a set of environments Etr , where De = { xie , yie } ne i=1 is the dataset gathered from environment e ∈ Etr and ne is the number of points in environment e. The feature value for data point i is xie ∈ X and the corresponding label is yie ∈ Y , where X ⊆ Rd and Y ⊆ R. Each point ( xie , yie ) in environment e is drawn i.i.d from a distribution Pe . Define a predictor f : X → R. The goal of IRM is to use these collection of datasets D to construct a predictor f that performs well across many unseen environments Eall , where Eall ⊇ Etr . Define the risk achieved by f in environment e as Re ( f ) = Ee [ ` ( f ( Xe ) , Ye ) ] , where ` is the square loss when f ( Xe ) is the predicted value and Ye is the corresponding label , ( Xe , Ye ) ∼ Pe and the expectation Ee is defined w.r.t . the distribution of points in environment e. An invariant predictor is composed of two parts a representation Φ ∈ Rd×n and a predictor ( with the constant term ) w ∈ Rd×1 . We say that a data representation Φ elicits an invariant predictor wTΦ across the set of environments Etr if there is a predictor w that achieves the minimum risk for all the environments w ∈ arg minw̃∈Rd×1 Re ( w̃TΦ ) , ∀e ∈ Etr . IRM may be phrased as the following constrained optimization problem ( Arjovsky et al. , 2019 ) : min Φ∈Rd×n , w∈Rd×1 ∑ e∈Etr Re ( w TΦ ) s.t . w ∈ arg min w̃∈Rd×1 Re ( w̃ TΦ ) , ∀e ∈ Etr ( 1 ) If wTΦ solves the above , then it is an invariant predictor across the training environments Etr . Local explainability setup vs IRM setup : There are multiple differences between IRM setup and the setup we have in this paper . First , IRM is meant to learn global models directly from the dataset . This also applies to the game theoretic version proposed in ( Ahuja et al. , 2020 ) , while for local explainability we are trying to learn local models that explain ( per example ) a given black-box model . Second , given that we want to explain a black-box model which typically is a function of the input features , we want to highlight features in our explanation that may be spurious from the domain perspective , but nonetheless the black-box model uses them to make decisions . Third , the IRM model does not have to be interpretable as its learned representation Φ can be arbitrary . Fourth , the IRM learning procedure is quite inefficient as it tries to solve a bi-level optimization problem with a highly non-convex constraint . Fifth , IRM assumes that one has access to multiple environments ( viz . multiple data sources ) , but in our case we have to figure out how to appropriately create them for each example that we want to explain . Last , since a black-box model is trained on the input features all or a subset of them are causal to its prediction , unlike the standard IRM setup where a predictor is expected to model the true underlying mechanism . Hence in our case , it is sufficient to limit ourselves to some interpretable representation , which in many cases is just the input feature space ( i.e . Φ = X ) . Thus IRM can not be directly applied to our problem and we propose a novel game theoretic approach that is suitable for local explainability consistent with the above points . With our approach it is interesting that we are able to obtain similar style theoretical results for our setting that have been seen in out-of-distribution generalization type of works ( Ahuja et al. , 2021 ) , thus justifying our choice of leveraging such frameworks . Nash Equilibrium ( NE ) : To understand how certain key aspects of our method function let us revisit the notion of Nash Equilibrium ( Dutta , 1999 ) . A standard normal form game is written as a tuple Ω = ( N , { ui } i∈N , { Si } i∈N ) , where N is a finite set of players . Player i ∈ N takes actions from a strategy set Si . The utility of player i is ui : S → R , where we write the joint set of actions of all the players as S = Πi∈NSi . The joint strategy of all the players is given as s ∈ S , the strategy of player i is si and the strategy of the rest of players is s−i = ( si′ ) i′ 6=i . Definition 1 . A strategy s† ∈ S is said to be a pure strategy Nash equilibrium ( NE ) if it satisfies ui ( s † i , s † −i ) ≥ ui ( k , s † −i ) , ∀k ∈ Si , ∀i ∈ N NE thus identifies a state where each player is using the best possible strategy in response to the rest of the players leaving no incentive for any player to alter their strategy . In seminal work by ( Debreu , 1952 ) it was shown that for a special class of games called concave games such a pure NE always exists . This is relevant because the game implied by Algorithm 1 falls in this category .

Summary: This paper approaches the problem of locally explaining predictions of black box models. They attempt to do so by creating local environments around a data unit. They propose their method, LINEX to overcome the shortcomings of the method, LIME. LINEX learns locally invariant predictors and aggregates their explanation for locally explaining the original model. The authors show some theoretical properties as well as empirical evidence.