Locally Invariant Explanations: Towards Causal Explanations through Local Invariant Learning

1 INTRODUCTION . Deployment and usage of neural black-box models has significantly grown in industry over the last few years creating the need for new tools to help users understand and trust models ( Gunning , 2017 ) . Even well-studied application domains such as image recognition require some form of prediction understanding in order for the user to incorporate the model into any important decisions ( Simonyan et al. , 2013 ; Lapuschkin et al. , 2016 ) . An example of this could be a doctor who is given a cancer diagnosis based on an image scan . Since the doctor holds responsibility for the final diagnosis , the model must provide sufficient reason for its prediction . Even new text categorization tasks ( Feng et al. , 2018 ) are becoming important with the growing need for social media companies to provide better monitoring of public content . Twitter recently began monitoring tweets related to COVID-19 in order to label tweets containing misleading information , disputed claims , or unverified claims ( Roth & Pickles , 2020 ) . Laws will likely emerge requiring explanations for why red flags were or were not raised in many examples . In fact , the General Data Protection and Regulation ( GDPR ) ( Yannella & Kagan , 2018 ) act passed in Europe already requires automated systems that make decisions affecting humans to be able to explain them . Given this acute need , a number of methods have been proposed to explain local decisions ( i.e . example specific decisions ) of classifiers ( Ribeiro et al. , 2016 ; Lundberg & Lee , 2017 ; Simonyan et al. , 2013 ; Lapuschkin et al. , 2016 ; Dhurandhar et al. , 2018a ) . Locally interpretable model-agnostic explanations ( LIME ) is arguably the most well-known local explanation method that requires only query ( or black-box ) access to the model . Although LIME is a popular method , it is known to be sensitive to certain design choices such as i ) ( random ) sampling to create the ( perturbation ) neighborhood1 , ii ) the size of this neighborhood ( number of samples ) and iii ) ( local ) fitting procedure to learn the explanation model ( Molnar , 2019 ; Zhang et al. , 2019b ) . The first , most serious issue could lead to nearby examples having drastically different explanations making effective recourse a challenge . One possible mitigation is to increase the neighborhood size , however , one can not arbitrarily do so as it not only leads to higher computational cost , but in today ’ s cloud computing-driven world it could have direct monetary implications where every query to a black-box model has an associated cost ( Dhurandhar et al. , 2019 ) . There have been variants suggested to overcome some of these limitations ( Botari et al. , 2020 ; Shrotri et al. , 2021 ; Plumb et al. , 2018 ) primarily through mechanisms that create realistic neighborhoods or through adversarial training ( Lakkaraju et al. , 2020 ) , however , their efficacy is restricted to certain settings and modalities based on their assumptions and training strategies . In this paper we introduce a new method called Locally INvariant EXplanations ( LINEX ) inspired by the invariant risk minimization ( IRM ) principle ( Arjovsky et al. , 2019 ) , that produces feature based explanations that are robust to neighborhood sampling and can recover faithful ( i.e . mimic black-box behavior ) , stable ( i.e . similar for closeby examples ) and unidirectional ( i.e . same sign attributions for closeby examples , see section 4.1 ) explanations across tabular , image , and text modalities . In particular , we show that our method performs better than the competitors for random as well as realistic neighborhood generation , where in some cases even with the prior strategy our explanation quality is close to methods that employ the latter . Qualitatively , our method highlights ( local ) features as important that in the particular locality i ) have consistently high gradient with respect to ( w.r.t . ) the black-box function and ii ) where the gradient does not change significantly , especially in sign . Such stable behavior for LINEX is visualized in Figure 1 , where we get similar explanations for nearby examples in the IRIS dataset . The ( in ) fidelity of LINEX is still similar to LIME ( see Table 3 ) , but of course our explanations are much more stable . 2 RELATED WORK . Posthoc explanations can typically be partitioned into two broad categories global and local . Global explainability avers to trying to understand a black-box model at a holistic level where the typical tact is knowledge transfer ( Hinton et al. , 2015 ; Dhurandhar et al. , 2018b ; 2020 ) where ( soft/hard ) labels of the black-box model are used to train an interpretable model such as a decision tree or rule list ( Rudin , 2019 ) . Local explanations on the other hand avers to understanding individual decisions . These explanations are typically in two forms , either exemplar based or feature based . For exemplar based as the name suggests similar but diverse examples ( Kim et al. , 2016 ; Gurumoorthy et al. , 2019 ) are provided as explanations for the input in question . While for feature based ( Ribeiro et al. , 2016 ; Lundberg & Lee , 2017 ; Dhurandhar et al. , 2018a ; Lapuschkin et al. , 2016 ) , which is the focus of this work , important features are returned as being important for the decision made for the input . There are some methods that do both ( Plumb et al. , 2018 ) . Moreover , there are methods which provide explanations that are local , global as well as at a group level ( Ramamurthy et al. , 2020 ) . All of these methods though may not still provide stable and robust local feature based explanations which can be desirable in practice ( Ghorbani et al. , 2019 ) . 1By perturbation neighborhood – referred to as simply neighborhood – we mean neighborhoods generated for local explanations . By exemplar neighborhood , we mean nearest examples in a dataset to a given example . Given this there have been more recent works that try to learn either robust or even causal explanations . In ( Lakkaraju et al. , 2020 ) the authors try to learn robust and stable local explanations relative to distribution shifts and adversarial attacks . However , the distribution shifts they consider are linear shifts and adversarial training is performed which can be slow and sometimes unstable ( Zhang et al. , 2019a ) . Moreover , the method seems to be applicable primarily to tabular data . Works on causal explanations ( Frye et al. , 2020 ; Heskes et al. , 2020 ) mainly modify SHAP and assume access to a partial causal graph . Some others ( Vig et al. , 2020 ) assume white-box access . In this work we do not assume availability of such additional information . There are also works which show that creating realistic neighborhoods by learning the data manifold for LIME ( Botari et al. , 2020 ; Shrotri et al. , 2021 ) can lead to better quality explanations , where in a particular work ( Anders et al. , 2020 ) it is suggested that projecting explanations themselves on to the manifold can also make them more robust . The need for stability in a exemplar neighborhood for LIME like methods has been highlighted in ( Zhang et al. , 2019b ) , with the general desire for stable explanations being also expressed in prior works ( Yeh et al. , 2019 ; Visani et al. , 2020 ) . 3 PRELIMINARIES . Invariant Risk Minimization : Given a collection of training datasets D = { De } e∈Etr gathered from a set of environments Etr , where De = { xie , yie } ne i=1 is the dataset gathered from environment e ∈ Etr and ne is the number of points in environment e. The feature value for data point i is xie ∈ X and the corresponding label is yie ∈ Y , where X ⊆ Rd and Y ⊆ R. Each point ( xie , yie ) in environment e is drawn i.i.d from a distribution Pe . Define a predictor f : X → R. The goal of IRM is to use these collection of datasets D to construct a predictor f that performs well across many unseen environments Eall , where Eall ⊇ Etr . Define the risk achieved by f in environment e as Re ( f ) = Ee [ ` ( f ( Xe ) , Ye ) ] , where ` is the square loss when f ( Xe ) is the predicted value and Ye is the corresponding label , ( Xe , Ye ) ∼ Pe and the expectation Ee is defined w.r.t . the distribution of points in environment e. An invariant predictor is composed of two parts a representation Φ ∈ Rd×n and a predictor ( with the constant term ) w ∈ Rd×1 . We say that a data representation Φ elicits an invariant predictor wTΦ across the set of environments Etr if there is a predictor w that achieves the minimum risk for all the environments w ∈ arg minw̃∈Rd×1 Re ( w̃TΦ ) , ∀e ∈ Etr . IRM may be phrased as the following constrained optimization problem ( Arjovsky et al. , 2019 ) : min Φ∈Rd×n , w∈Rd×1 ∑ e∈Etr Re ( w TΦ ) s.t . w ∈ arg min w̃∈Rd×1 Re ( w̃ TΦ ) , ∀e ∈ Etr ( 1 ) If wTΦ solves the above , then it is an invariant predictor across the training environments Etr . Local explainability setup vs IRM setup : There are multiple differences between IRM setup and the setup we have in this paper . First , IRM is meant to learn global models directly from the dataset . This also applies to the game theoretic version proposed in ( Ahuja et al. , 2020 ) , while for local explainability we are trying to learn local models that explain ( per example ) a given black-box model . Second , given that we want to explain a black-box model which typically is a function of the input features , we want to highlight features in our explanation that may be spurious from the domain perspective , but nonetheless the black-box model uses them to make decisions . Third , the IRM model does not have to be interpretable as its learned representation Φ can be arbitrary . Fourth , the IRM learning procedure is quite inefficient as it tries to solve a bi-level optimization problem with a highly non-convex constraint . Fifth , IRM assumes that one has access to multiple environments ( viz . multiple data sources ) , but in our case we have to figure out how to appropriately create them for each example that we want to explain . Last , since a black-box model is trained on the input features all or a subset of them are causal to its prediction , unlike the standard IRM setup where a predictor is expected to model the true underlying mechanism . Hence in our case , it is sufficient to limit ourselves to some interpretable representation , which in many cases is just the input feature space ( i.e . Φ = X ) . Thus IRM can not be directly applied to our problem and we propose a novel game theoretic approach that is suitable for local explainability consistent with the above points . With our approach it is interesting that we are able to obtain similar style theoretical results for our setting that have been seen in out-of-distribution generalization type of works ( Ahuja et al. , 2021 ) , thus justifying our choice of leveraging such frameworks . Nash Equilibrium ( NE ) : To understand how certain key aspects of our method function let us revisit the notion of Nash Equilibrium ( Dutta , 1999 ) . A standard normal form game is written as a tuple Ω = ( N , { ui } i∈N , { Si } i∈N ) , where N is a finite set of players . Player i ∈ N takes actions from a strategy set Si . The utility of player i is ui : S → R , where we write the joint set of actions of all the players as S = Πi∈NSi . The joint strategy of all the players is given as s ∈ S , the strategy of player i is si and the strategy of the rest of players is s−i = ( si′ ) i′ 6=i . Definition 1 . A strategy s† ∈ S is said to be a pure strategy Nash equilibrium ( NE ) if it satisfies ui ( s † i , s † −i ) ≥ ui ( k , s † −i ) , ∀k ∈ Si , ∀i ∈ N NE thus identifies a state where each player is using the best possible strategy in response to the rest of the players leaving no incentive for any player to alter their strategy . In seminal work by ( Debreu , 1952 ) it was shown that for a special class of games called concave games such a pure NE always exists . This is relevant because the game implied by Algorithm 1 falls in this category .

Summary: The paper proposes an algorithm LINEX for learning a robust LIME-like explanation for a black box model. The algorithm is based on the Invariant Risk Minimization Games framework proposed by [1],[2]. LIME is notorious for being very sensitive to its hyperparameters, and the paper posits that a robust variant of LIME is one which is invariant to these hyperparameters. The primary contribution of the paper is to formulate the perturbation-neighbourhoods of the point to be explained as ‘environments’ in an IRM setup. The paper also notes that characteristics of such invariant predictors (Theorem 2 of [2]) make them well-suited for explanations.