SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Amit Dhurandhar*
Yorktown Heights, USA
adhuran@us.ibm.com
Kartik Ahuja
kartik.ahuja@mila.quebec
Karthikeyan Natesan Ramamurthy*
knatesa@us.ibm.com
Vijay Arya
vijay.arya@in.ibm.com
Locally interpretable model agnostic explanations (LIME) method is one of the most popular methods used to explain black-box models at a per example level. Although many variants have been proposed, few provide a simple way to produce high fidelity explanations that are also stable and intuitive. In this work, we provide a novel perspective by proposing a model agnostic local explanation method inspired by the invariant risk minimization (IRM) principle – originally proposed for (global) out-of-distribution generalization – to provide such high fidelity explanations that are also stable and unidirectional across nearby examples. Our method is based on a game theoretic formulation where we theoretically show that our approach has a strong tendency to eliminate features where the gradient of the black-box function abruptly changes sign in the locality of the example we want to explain, while in other cases it is more careful and will choose a more conservative (feature) attribution, a behavior which can be highly desirable for recourse. Empirically, we show on tabular, image and text data that the quality of our explanations with neighborhoods formed using random perturbations are much better than LIME and in some cases even comparable to other methods that use realistic neighbors sampled from the data manifold. This is desirable given that learning a manifold to either create realistic neighbors or to project explanations is typically expensive or may even be impossible. Moreover, our algorithm is simple and efficient to train, and can ascertain stable input features for local decisions of a black-box without access to side information such as a (partial) causal graph as has been seen in some recent works.
Deployment and usage of neural black-box models has significantly grown in industry over the last few years creating the need for new tools to help users understand and trust models [24]. Even well-studied application domains such as image recognition require some form of prediction understanding in order for the user to incorporate the model into important decisions [46, 33]. An example of this could be a doctor who is advised by a model of a positive cancer diagnosis based on an image
scan. Since the doctor holds responsibility for the final diagnosis, the model must provide sufficient reason for its prediction. Even new text categorization tasks [21] are becoming important with the growing need for social media companies to provide better monitoring of public content. Twitter was monitoring tweets related to COVID-19 in order to label tweets containing misleading information, disputed claims, or unverified claims [42]. Laws will likely emerge requiring explanations for why red flags were or were not raised in many examples. In fact, the General Data Protection and Regulation (GDPR) [51] act passed in Europe already requires automated systems that make decisions affecting humans to be able to explain them. Given this acute need, a number of methods have been proposed to explain local decisions (i.e. example specific decisions) of classifiers [41, 35, 46, 33, 13]. Locally interpretable model-agnostic explanations (LIME) is arguably the most well-known local explanation method that requires only query (or black-box) access to the model. Although LIME is a popular method, it is known to be sensitive to certain design choices such as i) (random) sampling to create the (perturbation) neighborhood $^{2}$ , ii) the size of this neighborhood (number of samples) and iii) (local) fitting procedure to learn the explanation model [37, 54]. The first, most serious issue could lead to nearby examples having drastically different explanations making effective recourse a challenge. One possible mitigation is to increase the neighborhood size but one cannot arbitrarily do so as it not only leads to higher computational cost, but also in today's cloud computing-driven world it could have direct monetary implications where every query to a black-box model has an associated cost [14].
There have been variants suggested to overcome these limitations [8, 45, 39, 15] primarily through mechanisms that create realistic neighborhoods or through adversarial training [32], however, their efficacy is restricted to certain settings and modalities based on their assumptions and training strategies.
In this paper we introduce a new method called Locally INvariant EXplanations (LINEX) inspired by the invariant risk minimization (IRM) principle [7], that produces explanations in the form of feature attributions that are robust to neighborhood sampling and can recover faithful (i.e. mimic black-box behavior), stable (i.e. similar for closeby examples) and unidirectional (i.e. same sign attributions a.k.a. feature importances) for closeby examples, see section 4.1) explanations across tabular, image, and text modalities. In particular, we show that our method performs better than the competitors for random as well as realistic neighborhood generation, where in some cases even with the prior strategy our explanation quality is close to methods that employ the latter. Qualitatively, our method highlights (local) features as important that in the particular locality i) have consistently high gradient with respect to (w.r.t.) the black-box function and ii) where the gradient does not change significantly, especially in sign. Such stable behavior for LINEX is visualized in Figure 1, where we get similar explanations for nearby examples in the IRIS dataset. The (in)fidelity of LINEX is still similar to LIME (see Table 2), but of course our explanations are much more stable.
Figure 1: Above we visualize for the IRIS dataset the Coefficient Inconsistency (CI) (see Section 5 for exact definition and setup details) between the explanation (top two features) for an example and its nearest neighbor in the dataset. Each circle denotes an example and a rainbow colormap depicts the degree of inconsistency w.r.t. its nearest neighbor where red implies least inconsistency, while violet implies the most. As can be seen LINEX explanations are much more consistent than LIME's.
Posthoc explanations can typically be partitioned into two broad categories global and local. Global explainability avers to trying to understand a black-box model at a holistic level where the typical tact is knowledge transfer [28, 17, 16] where (soft/hard) labels of the black-box model are used to train an interpretable model such as a decision tree or rule list [43]. Local explanations on the other hand avers to understanding individual decisions. These explanations are typically in two
<sup>2</sup>By perturbation neighborhood or simply neighborhood, we mean neighborhoods generated for local explanations. By exemplar neighborhood, we mean closest in dataset examples.
forms, either exemplar based or feature based. For exemplar based as the name suggests similar but diverse examples [30, 25] are provided as explanations for the input in question. While for feature based [41, 35, 13, 33, 55], which is the focus of this work, important features are returned as being important for the decision made for the input. There are some methods that do both [39]. Moreover, there are methods which provide explanations that are local, global as well as at a group level [40]. All of these methods though may not still provide stable and robust local feature based explanations which can be desirable in practice [23].
Given this there have been more recent works that try to learn either robust or even causal explanations. In [32] the authors try to learn robust and stable local explanations relative to distribution shifts and adversarial attacks. However, the distribution shifts they consider are linear shifts and adversarial training is performed which can be slow and sometimes unstable [53]. Moreover, the method seems to be applicable primarily to tabular data. There are also works [18, 19] which try to robustify gradient based explanations assuming white box access to the model. Works on causal explanations [22, 27] mainly modify SHAP and assume access to a partial causal graph. Some others [48] assume white-box access. In this work we do not assume availability of such additional information. There are also works which show that creating realistic neighborhoods by learning the data manifold for LIME [8, 45] can lead to better quality explanations, where in a particular work [6] it is suggested that projecting explanations themselves on to the manifold can also make them more robust. The need for stability in a exemplar neighborhood for LIME like methods has been highlighted in [54], with the general desire for stable explanations being also expressed in [52, 49]. Furthermore, it was recently surmised through expert and crowd worker user studies that stability is a key factor when it comes to assessing capability of a model or when learning a new domain [34].
Given that our approach is inspired from IRM we now describe, how it is novel w.r.t. to it. It is important to realize that IRM approaches such as [3, 4] are designed for the out-of-distribution (OOD) generalization, which learn global models directly from the data. The main similarity of these works to ours is only that they also are game theory based approaches, but with the details being quite different. For one, they assume accessibility to environments which (ideally) correspond to different interventional distributions and with assumptions on the structural causal model derive results on how the true causal factors could be divulged. In our case, we propose ways to generate environments as they are not given, and have  $l_{1}$  and  $l_{\infty}$  constraints on the entire and environment specific parts of the model respectively, which is not the case with these prior works. As such those algorithms do not produce sparse unidirectional models that are also consumable. Moreover, the perspective we provide is novel in the context of local posthoc explanations where a priori it is not obvious that approaches from OOD generalization could be extended and adapted. Additionally, we propose a novel metric Unidirectionality which is not part of any of these works, but as we have argued it is a desirable property for explanations.
Invariant Risk Minimization: Given a collection of training datasets  $D = \{D_e\}_{e \in \mathcal{E}_{tr}}$  gathered from a set of environments  $\mathcal{E}_{tr}$ , where  $D_e = \{\pmb{x}_e^i, y_e^i\}_{i=1}^{n_e}$  is the dataset gathered from environment  $e \in \mathcal{E}_{tr}$  and  $n_e$  is the number of points in environment  $e$ . The feature value for data point  $i$  is  $\pmb{x}_e^i \in \mathcal{X}$  and the corresponding label is  $y_e^i \in \mathcal{Y}$ , where  $\mathcal{X} \subseteq \mathbb{R}^d$  and  $\mathcal{Y} \subseteq \mathbb{R}$ . Each point  $(\pmb{x}_e^i, y_e^i)$  in environment  $e$  is drawn i.i.d from a distribution  $\mathbb{P}_e$ . Define a predictor  $f: \mathcal{X} \to \mathbb{R}$ .
The goal of IRM is to use these collection of datasets  $D$  to construct a predictor  $f$  that performs well across many unseen environments  $\mathcal{E}_{all}$ , where  $\mathcal{E}_{all} \supseteq \mathcal{E}_{tr}$ . Define the risk achieved by  $f$  in environment  $e$  as  $R_{e}(f) = \mathbb{E}_{e}\big[\ell (f(\boldsymbol{X}_{e}),Y_{e})\big]$ , where  $\ell$  is the square loss when  $f(\boldsymbol{X}_e)$  is the predicted value and  $Y_{e}$  is the corresponding label,  $(\boldsymbol{X}_e,Y_e)\sim \mathbb{P}_e$  and the expectation  $\mathbb{E}_e$  is defined w.r.t. the distribution of points in environment  $e$ .
An invariant predictor is composed of two parts: a representation  $\Phi \in \mathbb{R}^{d\times n}$  and a predictor (with the constant term)  $\boldsymbol{w} \in \mathbb{R}^{d\times 1}$ . We say that a data representation  $\Phi$  elicits an invariant predictor  $\boldsymbol{w}^{\top}\Phi$  across the set of environments  $\mathcal{E}_{tr}$  if there is a predictor  $\boldsymbol{w}$  that achieves the minimum risk for all the environments  $\boldsymbol{w} \in \operatorname*{argmin}_{\tilde{\boldsymbol{w}} \in \mathbb{R}^{d\times 1}} R_{e}(\tilde{\boldsymbol{w}}^{\top}\Phi)$ ,  $\forall e \in \mathcal{E}_{tr}$ . IRM may be phrased as the following constrained optimization problem [7]:
$$
\min  _ {\boldsymbol {\Phi} \in \mathbb {R} ^ {d \times n}, \boldsymbol {w} \in \mathbb {R} ^ {d \times 1}} \sum_ {e \in \mathcal {E} _ {t r}} R _ {e} \left(\boldsymbol {w} ^ {\top} \boldsymbol {\Phi}\right) \quad \text {s . t .} \boldsymbol {w} \in \underset {\tilde {\boldsymbol {w}} \in \mathbb {R} ^ {d \times 1}} {\operatorname {a r g m i n}} R _ {e} \left(\tilde {\boldsymbol {w}} ^ {\top} \boldsymbol {\Phi}\right), \forall e \in \mathcal {E} _ {t r} \tag {1}
$$
If  $\pmb{w}^{\top}\pmb{\Phi}$  solves the above, then it is an invariant predictor across the training environments  $\mathcal{E}_{tr}$ .
Nash Equilibrium (NE): To understand how certain key aspects of our method function let us revisit the notion of Nash Equilibrium [20]. A standard normal form game is written as a tuple  $\Omega = (\mathcal{N},\{u_i\}_{i\in \mathcal{N}},\{S_i\}_{i\in \mathcal{N}})$ , where  $\mathcal{N}$  is a finite set of players. Player  $i\in \mathcal{N}$  takes actions from a strategy set  $S_{i}$ . The utility of player  $i$  is  $u_{i}:\mathcal{S}\to \mathbb{R}$ , where we write the joint set of actions of all the players as  $\mathcal{S} = \Pi_{i\in \mathcal{N}}\mathcal{S}_i$ . The joint strategy of all the players is given as  $\boldsymbol {s}\in S$ , the strategy of player  $i$  is  $\boldsymbol{s}_i$  and the strategy of the rest of players is  $\boldsymbol{s}_{-i} = (\boldsymbol{s}_{i^{\prime}})_{i^{\prime}\neq i}$ .
Definition 1. A strategy  $\pmb{s}^{\dagger} \in S$  is said to be a pure strategy Nash equilibrium (NE) if it satisfies,  $u_{i}(s_{i}^{\dagger}, s_{-i}^{\dagger}) \geq u_{i}(k, s_{-i}^{\dagger}), \forall k \in S_{i}, \forall i \in \mathcal{N}$ , where  $u_{i}(s_{i}^{\dagger}, s_{-i}^{\dagger}) = u_{i}(s_{1}^{\dagger}, s_{2}^{\dagger}, \dots, s_{\mathcal{N}}^{\dagger}) = u_{i}(s^{\dagger})$ .
NE thus identifies a state where each player is using the best possible strategy in response to the rest of the players leaving no incentive for any player to alter their strategy. In seminal work by [11] it was shown that for a special class of games called concave games such a pure NE always exists. This is relevant because the game implied by Algorithm 1 falls in this category.
We first define desirable properties for our explanation methods. The first three have been seen in previous works, while the last Unidirectionality is new. We then describe our method where the goal is to explain a black-box model  $f: \mathcal{X} \to \mathbb{R}$  for individual inputs  $\mathbf{x}$  based on predictors  $\mathbf{w}$  by looking at their corresponding components, also termed as feature attributions.
We take inspiration from IRM since, our goal here too is to extract robust features that are ideally stable and unidirectional. The main difference is that we do not learn a new (possibly invariant) representation since, we desire interpretability and this new representation may not be interpretable. We hence, are restricted to the provided input or some other interpretable representation. Thus, given that  $\Phi \subseteq \mathcal{X}$  where  $n = 1$  (since local explanations) in our setup, our goal is to find the best predictor  $\boldsymbol{w}$  (viz. high fidelity) for an input that will eliminate or at least mitigate the effect of unstable features. In other words, we want to identify features in the input space that will (roughly) have the same importance (i.e. are invariant) in the neighborhood of the example we want to explain. Our approach as we will see is similar in spirit to IRM games [4], where we adopt a game theoretic strategy to obtain such explanations. The differences with IRM games are mentioned in the last paragraph of Section 2.
4.1 Desirable Properties
We now discuss certain properties we would like our explainability method to have in order to provide robust explanations that could potentially be used for recourse. Let  $D_{t}$  denote a (test) dataset with examples  $(x,y)$  where  $y_{b}(x)$  is the black-box models prediction on  $x$  and  $y_{e}^{x^{\prime}}(x)$  is the prediction on  $x$ $(\in \mathcal{X})$  using the explanation model at  $x^{\prime}$ . The feature attributions (or coefficients) for the explanation model at  $x$  are denoted by  $c_e^x, \mathcal{N}_x$  denotes the exemplar neighborhood of  $x$  with  $|.|_{\mathrm{card}}$  denoting cardinality and  $|.|$  denoting absolute value.
Fidelity: This is the most standard property which all proxy model based explanation methods are evaluated against [41, 35, 32] as it measures how well the proxy model simulates the behavior of the black-box (i.e. faithfulness to the black box) it is attempting to explain. Here we define inverse of it, that is Infidelity (INFD), as the MAE between the black-box and explanation model predictions across all the test points:
$$
\text {I N F D} = \frac {1}{| D _ {t} | _ {\text {c a r d}}} \sum_ {(x, y) \in D _ {t}} | y _ {b} (x) - y _ {e} ^ {x} (x) |. \tag {2}
$$
We also define another metric here called Generalized Infidelity (GI), which also been used in previous works [40] to measure the generalizability of local explanations to neighboring test points. It is defined as:
$$
\mathrm {G I} = \frac {1}{| D _ {t} | _ {\text {c a r d}}} \sum_ {(x, y) \in D _ {t}} \frac {1}{| \mathcal {N} _ {x} | _ {\text {c a r d}}} \sum_ {x ^ {\prime} \in \mathcal {N} _ {x}} | y _ {b} (x) - y _ {e} ^ {x ^ {\prime}} (x) |. \tag {3}
$$
Stability: This is also a popular notion [26, 40, 52] to evaluate robustness of explanations. Largely, stability can be measured at three levels. One is prediction stability, which measures how much the predictions of an explanation model change for the same example subject to different randomizations within the method or across close by examples. The second is the variance in the feature attributions again for the same or close by examples. It is good for a method to showcase stability w.r.t. both even though in many cases the latter might imply the former. An interesting third notion of stability is the correlation between the feature attributions of an explanation model and average feature values of examples belonging to a particular class. This measures how much does the explanation method pick features that are important for the class, rather than spurious ones that seem important for just the example. We thus define two stability metrics.
Coefficient Inconsistency (CI): This notion has been used before [26] to measure an explanation methods robustness. It can be defined as the MAE between the attributions of the test points and their respective neighbors:
$$
\mathrm {C I} = \frac {1}{| D _ {t} | _ {\text {c a r d}}} \sum_ {(x, y) \in D _ {t}} \frac {1}{| \mathcal {N} _ {x} | _ {\text {c a r d}}} \sum_ {x ^ {\prime} \in \mathcal {N} _ {x}} \left| c _ {e} ^ {x} - c _ {e} ^ {x ^ {\prime}} \right| _ {1}. \tag {4}
$$
Class-Attribution Consistency (CAC): For local explanations of classification black-boxes, we expect certain important features to be highlighted across most of the explanations of a class. This is codified by this metric which is defined as follows:
$$
\mathrm {C A C} = \frac {1}{| \mathcal {Y} | _ {\text {c a r d}}} \sum_ {y \in \mathcal {Y}} r \left(\mu_ {e} ^ {y}, \mu_ {y}\right), \tag {5}
$$
where  $\mathcal{V}$  denotes the set of class labels in the dataset,  $\mu_y$  the mean (vector) of all inputs in class  $y\in \mathcal{V}$ ,  $\mu_e^y$  the mean explanation for class  $y$  and  $r$  the Pearson's correlation coefficient. This metric quantifies the consistency between the important features for a class and attributions provided by the explanations.
Black-box Invariance: This is the same as implementation invariance defined in [47]. Essentially, if two models have exactly the same behavior on all inputs then their explanations should also be the same. Since, our method is model agnostic with only query access to the model it is easy to see that it satisfies this property if the same environments are created.
Unidirectionality: This is a new property, but as we argue that this is a natural one to have. Loosely speaking, unidirectionality would measure how consistently the sign of the predictor for a feature is maintained for the same or close by examples by an explanation method. This is a natural metric [36], which from an algorithmic recourse [29] perspective is also highly desirable. For instance, recommending a person to increase their salary to get a loan and then recommending to another person with a very similar profile to decrease their salary for the same outcome makes little sense.
We define the unidirectionality  $\Upsilon$  as a measure of how consistent the sign of the attribution for a particular feature in a local explanation is when varying neighborhoods for the same example or when considering different close by examples. As such, given  $m$  attributions for each of  $d$  features denoted by  $w_{1}^{(1)},\ldots ,w_{m}^{(d)}$  the metric for an example is:
$$
\Upsilon = \frac {1}{m d} \sum_ {i = 1} ^ {d} \left| \sum_ {j = 1} ^ {m} \operatorname {s g n} \left(w _ {j} ^ {(i)}\right) \right| \tag {6}
$$
where  $|.|$  stands for absolute value. Clearly, the more consistent the signs for the attribution of a particular feature across  $m$  attributions the higher the value, where the maximum value can be one. If equal number of attributions have different signs for all features then  $\Upsilon$  will be zero, the lowest possible value. This property thus measures how intuitively consistent (ignoring magnitude) the explanations are. Given its sole focus on the sign of the attributions it compliments the above metrics along with attributional robustness metrics [9, 44].
4.2 Method
4.2.1 Description
In Algorithm 1, we show the steps of our method LINEX. The input is the example we want to explain  $x$ , the black-box predictor, a few thresholds that we describe next and  $k$  (local) environments
Algorithm 1: Locally Invariant EXplanations (LINEX).
Input: example  $\pmb{x}$ , black-box predictor  $f(\cdot)$ , number of environments to be created  $k$ ,  $(l_{\infty})$
threshold  $\gamma >0$ $(l_{1})$  threshold  $t > 0$  and convergence threshold  $\epsilon >0$
Initialize:  $\forall i\in \{1,\dots,k\}$ $\tilde{\pmb{w}}_i = \mathbf{0}$  and  $\Delta = 0$
Let  $\xi_1(.),\dots,\xi_k(.)$  be  $k$  environment creation functions as described in section 4.2.2
$\Delta = 0$
$$
\tilde {\boldsymbol {w}} _ {- i} ^ {+} = \sum_ {j \in \{1, \dots , k \}, j \neq i} \tilde {\boldsymbol {w}} _ {j}
$$
$$
\tilde {\boldsymbol {w}} _ {i} ^ {\text {p r e v}} = \tilde {\boldsymbol {w}} _ {i}
$$
$$
\tilde {\boldsymbol {w}} _ {i} = \underset {\boldsymbol {x} \ddot {\boldsymbol {x}}} {\operatorname {a r g m i n}} \sum_ {\tilde {\boldsymbol {x}} \in \xi_ {i} (\boldsymbol {x})} \left(f (\tilde {\boldsymbol {x}}) - \tilde {\boldsymbol {w}} _ {- i} ^ {+ ^ {\intercal}} \tilde {\boldsymbol {x}} - \tilde {\boldsymbol {w}} ^ {\intercal} \tilde {\boldsymbol {x}}\right) ^ {2} \text {s . t .} | \tilde {\boldsymbol {w}} _ {- i} ^ {+} + \tilde {\boldsymbol {w}} | _ {1} \leq t \text {a n d} | \tilde {\boldsymbol {w}} | _ {\infty} \leq \gamma
$$
$$
\Delta = \max  \left(\left| \tilde {\boldsymbol {w}} _ {i} ^ {\text {p r e v}} - \tilde {\boldsymbol {w}} _ {i} \right| _ {2}, \Delta\right)
$$
while  $\Delta \geq \epsilon$
Output:  $\pmb{w} = \sum_{i\in \{1,\dots ,k\}}\tilde{\pmb{w}}_i$