SlowGuess/NAACL_2022_OCR huggingface dataset:
ExSUM: From Local Explanations to Model Understanding
Yilun Zhou
Julie Shah
{yilun,julie_a_shah}@csail.mit.edu marcotcr@microsoft.com
https://yilunzhou.github.io/exsum/
Interpretability methods are developed to understand the working mechanisms of black-box models, which is crucial to their responsible deployment. Fulfilling this goal requires both that the explanations generated by these methods are correct and that people can easily and reliably understand them. While the former has been addressed in prior work, the latter is often overlooked, resulting in informal model understanding derived from a handful of local explanations. In this paper, we introduce explanation summary (EXSUM), a mathematical framework for quantifying model understanding, and propose metrics for its quality assessment. On two domains, EXSUM highlights various limitations in the current practice, helps develop accurate model understanding, and reveals easily overlooked properties of the model. We also connect understandability to other properties of explanations such as human alignment, robustness, and counterfactual similarity and plausibility.
Understanding a model's behavior is often a prerequisite for deploying it in the real world, especially in high-stake scenarios such as financial, legal, and medical domains. Unfortunately, most high-performing models, such as neural networks, are black-boxes. Thus, model-agnostic interpretability techniques have been developed, with the majority being "local" – algorithms that produce an explanation for a specific input at a time (e.g., Li et al., 2016; Ribeiro et al., 2016).
Even with these local explanations, there are still two hurdles to overcome before achieving the ultimate goal of complete understanding of a model. First, some local explanations may not correctly (or faithfully) represent the model's reasoning process (Jacovi and Goldberg, 2020), as has been demonstrated both theoretically (Nie et al., 2018) and empirically (Adebayo et al., 2018) in prior work. As a
Figure 1: Local model explanations need to be both correct and easily understandable. While much prior work (e.g., Zhou et al., 2022) has studied the former property, this paper focuses on the latter, which has thus far been largely ignored.
result, correctness evaluation has received much attention in the community (e.g., Samek et al., 2016; Arras et al., 2019; Zhou et al., 2022).
Another mostly overlooked property of explanations is their understandability. As the model understanding pipeline depicted in Fig. 1 shows, explanations need to be both correct and easily understandable, since even correct explanations are not as valuable if they lead to incorrect understanding. However, the concept of understandability has yet to be formalized, and instead users often derive model understanding from few examples in a non-rigorous (and potentially incorrect) manner.
Consider the sentiment classification task shown in Fig. 2. On a test input, the model makes the correct prediction of positive sentiment. Obviously, this evidence is insufficient to conclude that "in general, the model classifies positive inputs correctly", because even a random-guess model is correct  $50\%$  of the time on a single instance. Instead, statistics such as the confusion matrix serve to rigorously support (or refute) generalization claims about model performance – for example, "the model is correct  $97.6\%$  of the time on positive inputs" – ensuring an accurate understanding of model performance.
Human Language Technologies, pages 5359 - 5378
Figure 2: An analogy between understanding model prediction (top route) and model explanation (bottom route). A test input (A) is fed into a fine-tuned RoBERTa model (B), which generates a correct prediction (C1) and reasonable explanation (C2). While generalized claims of understanding model performance (D1) are made rigorously from quantitative statistics such as the test set confusion matrix (E1), claims of understanding model behavior (D2) are predominantly derived informally from one or few explanations (C2). In this paper, we argue the necessity of formalizing this process, and propose the explanation summary (EXSUM) framework (E2), which reveals the severe limitations of the ad hoc model understanding (D2).
Do we understand model behaviors in the same rigorous way? Fig. 2 shows that the SHAP score (Lundberg and Lee, 2017) of the word "memorable" is highest at 0.48, while that of "for" is negligible at -0.02. Therefore, it is tempting to conclude that "in general, the model recognizes the high positive contribution of highly positive words and ignores stop words" – as expected for an accurate sentiment classifier. However, this is a generalization from a single instance, and thus potentially unreliable. We need the "confusion matrix" analogue for such claims, which to the best of our knowledge does not exist, making it hard to derive model understanding from local explanations.
In this paper, we propose ExSUM, a mathematical framework to formalize model understanding. In ExSUM, each piece of "model understanding" is specified precisely via a rule that links inputs to attribution values. For example, the tentative understanding described in the previous paragraph could be formalized as "words more positive than memorable (as measured by the word sentiment score given in the dataset, e.g., flawless, charming, etc) have SHAP attribution value in the [0.48, 1] range." This precise definition allows for quantitative evaluations. For example, this rule covers  $1.6\%$  of all words in the corpus, and is only correct  $3.1\%$  of the time. For the rule to be  $90\%$  correct, we need a wide and uninformative range of [-0.01, 1], indicating that a hasty generalization from "memorable" is unwarranted. Similarly, a saliency range of [-0.05, 0.05] for stop words is only correct  $64\%$  of the time: over  $1/3$  of stop words have non-negligible saliency – an understanding that is easily available with ExSUM, but might be missed with informal
explanation inspection. We define metrics to establish the quality profile of each rule and present a tool that makes it easy for users to construct EXSUM rules from local explanations. Finally, we demonstrate how EXSUM reveals the various drawbacks in the current practices of ad hoc model understanding, and allows for better understanding of model behavior in two separate tasks.
2 On Generalized Model Understanding
Besides the practical example above, we start from first principles and argue that generalized model understanding is the central concept for explanation usefulness. Local explanations are mathematical descriptions (MD) of some aspect of model behavior; for specific inputs. For example, gradient saliency (in the embedding space) is the sensitivity of the prediction to infinitesimal changes in the token embedding; occlusion saliency is the prediction change if individual embeddings are zeroed out. It is with these mathematical descriptions that people associate high-level interpretations (HL) of model behavior, such as associating the above two metrics with word importance. This (unconscious) train of thought can be described as follows:
$$
x \rightarrow \mathrm {M D} \rightarrow \mathrm {H L}.
$$
Crucially, people rarely study MD or HL for one specific input, as explanations are often used to understand broader model behaviors, such as reliance upon spurious correlation, non-discrimination of a protected class, or usage of unknown scientific principles. We elaborate upon these use cases in App. A to demonstrate that people implicitly or explicitly seek generalized model understanding.
From another perspective, analogous to why people ultimately focus on the generalization accuracy of a model, they (should) focus on generalized model understanding derived from local explanations.
For example, after observing that some highly polar words have high contribution for a sentiment classification model, people conclude that all highly polar words have high contribution. This process can be formalized as follows:
$$
\left. \begin{array}{c} x _ {1} \to \mathrm {M D} _ {1} \to \mathrm {H L} _ {1} \\ \dots \\ x _ {n} \to \mathrm {M D} _ {n} \to \mathrm {H L} _ {n} \end{array} \right\} \to \mathrm {H L} ^ {\mathrm {(g)}},
$$
where  $\mathrm{HL}^{(\mathrm{g})}$  is the generalized high-level model understanding. This generalization is too informal, not least because the step from  $\mathrm{MD}_i$  to  $\mathrm{HL}_i$  is itself already informal. Alternatively, we propose to generalize at the MD level, as follows:
$$
\left. \begin{array}{c} x _ {1} \to \mathrm {M D} _ {1} \\ \dots \\ x _ {n} \to \mathrm {M D} _ {n} \end{array} \right\} \to \mathrm {M D} ^ {\mathrm {(g)}} \to \mathrm {H L} ^ {\mathrm {(g)}}.
$$
Since MDs are rigorously defined mathematical quantities (e.g., the prediction of the sentence drops by  $32\%$  after the embedding of "great" is zeroed out), we can define and evaluate the quality of their generalization, and  $\mathrm{HL}^{(\mathrm{g})}$  can also include any failures and anomalies. As each MD is a local explanation, we call  $\mathrm{MD}^{(\mathrm{g})}$  the explanation summary (ExSUM), and proceed by instantiating this principle for feature attribution explanations.
3 The ExSum Framework
3.1 Setup and Notation
We focus on the classification setting, but all the ideas below can extend straightforwardly to regression. We have an input space  $\mathcal{X}$  and output space  $\mathcal{Y} = \{1,\dots,K\}$  of  $K$  classes. A data point is an input-output pair  $d = (x,y)\in \mathcal{D} = \mathcal{X}\times \mathcal{Y}$ , distributed as  $\mathbb{P}_D$ . We consider a model  $m:\mathcal{X}\to \Delta^{K - 1}$  where  $m(x)$  is the predicted class distribution on the probability simplex.
Feature attribution explainers assign an attribution, also known as saliency or importance, to each input feature, such as a token in a text input. For an instance  $(x,y)$ , each feature of  $x$  is called a fundamental explanation unit (FEU), defined as  $u = (x,y,l) \in \mathcal{U}$  with  $1 \leq l \leq L_x$  as the feature index.  $e(u) \in \mathcal{E}$  represents the attribution value assigned to it, where  $\mathcal{E}$  is the attribution space, such as  $[-1,1]$  for normalized explanations.
[ e(u_{-}) = \left(e_{x}^{(1)},\dots,e_{x}^{(l - 1)},e_{x}^{(l + 1)},\dots,e_{x}^{(L_{x})}\right) \in \mathcal{E}_{-}^{*} ] denotes the explanations on all other FEUs of  $x$ .
We define a distribution  $\mathbb{P}_U$  over  $\mathcal{U}$  such that the probability (or probability density) of  $u = (x,y,l)$  is  $1 / L_{x}$  of that of  $d$  under the data distribution  $\mathbb{P}_D$ . In other words, sampling of  $u$  can be performed in two steps: first draw an instance  $d = (x,y)\sim \mathbb{P}_D$ , then a feature index  $l\sim \mathrm{Unif}(\{1,\dots,L_x\})$ .
3.2 ExSUM Rules
An ExSUM rule formalizes a piece of model understanding, such as that for positive words in Fig. 2, which we use as the running example.
Definition 3.1 (ExSUM rule). An ExSUM rule  $r$  is defined by two functions. A binary-valued applicability function  $a: \mathcal{U} \to \{0,1\}$  determines whether the rule applies to a given FEU, with 1 being applicable and 0 otherwise. We use  $a(\mathcal{U}) = \{u \in \mathcal{U}: a(u) = 1\}$  to denote the applicability set. A set-valued behavior function is defined as  $b: a(\mathcal{U}) \times \mathcal{E}_{-}^{*} \to \mathcal{P}(\mathcal{E})$  where  $\mathcal{P}(\mathcal{E})$  is the power set (i.e., the set of all subsets) of  $\mathcal{E}$ . This function predicts a set of possible explanation values for the FEU, called the behavior range. The rule is written as  $r = \langle a, b \rangle$ . We abbreviate  $b(u, e(u_{-}))$  as  $b(u)$  and refer to the two functions as  $a$ - and  $b$ -functions.
For FEU  $u = (x, y, l)$ , the  $a$ -function typically depends only on  $x_l$ , but could depend on the entire input  $x$  (e.g., for long sentences) or the output  $y$  (e.g., for positive class). In our example, it tests whether the sentiment score is greater than that of the word "memorable" (0.638). The  $b$ -function usually outputs a constant range. Since "memorable" has a saliency of 0.479, the range is [0.479, 1.0].
3.3 Additional Examples
While we expect most rules to use rather simple  $a$ - and  $b$ -functions, they can also be more complex with more nuanced aspects. For the following examples, recall that  $u = (x,y,l)$ . An applicability function can target words only in long sentences using a conjunction with  $\mathrm{len}(x) \geq L$  where  $L$  is the threshold. We can also target inputs with ambivalent predictions with  $\max_c m(x)_c \leq 0.6$ , where  $\max_c m(x)_c$  is the probability of the predicted class. For behavior functions, to indicate the first word of the sentence has higher saliency than the rest, we can define  $b(u,e_{-}) = (\max_{l' \geq 2} e_{-}^{(l')}, 1.0]$ , where the  $a$ -function selects the first word (i.e.  $a(u) = \mathbb{1}_{l=1}$ ). Similarly, to describe that an FEU has higher saliency than all
$$
\begin{array}{l} \text {t h e v e r b s i n a s e n t e n c e , w e c a n c a u s e} b (u, e _ {-}) = \\ \left(\max  _ {l ^ {\prime}: \text {i s \_ v e r b} (x _ {l ^ {\prime}})} \{e _ {-} ^ {(l ^ {\prime})} \}, + \infty\right). \end{array}
$$
3.4 ExSUM Rule Unions
Since a single EXSUM rule is designed to capture one aspect of model understanding, multiple rules are often necessary for comprehensive understanding. However, conflicts can occur when multiple rules apply to the same FEU but the  $b$ -functions are different. We resolve them by defining the composition of two or more rules into a rule union.
Definition 3.2 (Precedence-Mode Composition). Two rules,  $r = \langle a, b \rangle$  and  $r' = \langle a', b' \rangle$ , can be composed into a precedence-mode rule union  $r^* = r > r'$  defined as  $r^* = \langle a^*, b^* \rangle$  where
$$
a ^ {*} (u) = \mathbb {1} \left\{a (u) + a ^ {\prime} (u) \geq 1 \right\}, \tag {1}
$$
$$
b ^ {*} (u) = \left\{ \begin{array}{l l} b (u) & \text {i f} a (u) = 1, \\ b ^ {\prime} (u) & \text {i f} a (u) = 0, a ^ {\prime} (u) = 1, \end{array} \right. \tag {2}
$$
represent the  $a$ - and  $b$ -functions of rule union  $r^*$ , with semantics similar to those for rules.
For example, if we want to split positive adjectives into a separate rule from other positive words, we create a rule to test for part-of-speech and sentiment score, and assign a higher precedence to this rule, such that the original rule is only applicable to the remaining non-adjectives. One useful practice is to include a lowest-precedence catch-all rule that covers everything not addressed by other rules, with a constant  $a(u) = 1$  function, which leaves no FEUs unaccounted for.
Definition 3.3 (Intersection-Mode Composition). Two rules,  $r = \langle a, b \rangle$  and  $r' = \langle a', b' \rangle$ , can be composed into an intersection-mode rule union  $r^* = r \& r'$  defined as  $r^* = \langle a^*, b^* \rangle$  where
$$
a ^ {*} (u) = \mathbb {1} \left\{a (u) + a ^ {\prime} (u) \geq 1 \right\}; \tag {3}
$$
$$
b ^ {*} (u) = \left\{ \begin{array}{l l} b (u) & \text {i f} a (u) = 1, a ^ {\prime} (u) = 0, \\ b ^ {\prime} (u) & \text {i f} a (u) = 0, a ^ {\prime} (u) = 1, \\ b (u) \cap b ^ {\prime} (u) & \text {i f} a (u) = a ^ {\prime} (u) = 1. \end{array} \right. \tag {4}
$$
Unlike precedence-mode, intersection-mode composition is symmetric with respect to the two rules. This mode is helpful when each property of an FEU has a corresponding behavior range, and the final behavior range of an FEU depends on FEU's properties. For example, if verbs have a behavior range of [-0.4, 0.4] and strongly positive words have a behavior range of [0.3, 1], a strongly positive verb would have a behavior range
[0.3, 0.4], or the intersection of the two constituent ranges. In our case studies, however, we do not encounter any situations in which intersection-mode compositions were preferable.
Since rule unions are also defined by  $a$ - and  $b$ -functions, they can form other rule unions in the same way. Recursively, this results in a list of rules composed into a single rule union, written as  $r^{*} = (r_{3} > r_{1})\& ((r_{4}\& r_{2}) > r_{5})$ . This rule union represents our generalized model understanding.
3.5 Quality Metrics
We propose three metrics for establishing the quality profiles of ExSUM rules or rule unions.
Definition 3.4 (Coverage). The coverage of a rule (union)  $r = \langle a, b \rangle$  is defined as follows:
$$
\kappa (r) = \mathbb {E} _ {U \sim \mathbb {P} (U)} [ a (U) ]. \tag {5}
$$
This represents the fraction of FEUs that we attempt to understand. While individual rules may have low coverage because they specialize in aspects of the model behavior, we want their union to have high coverage to achieve a comprehensive understanding of the model and prevent model prediction from being excessively affected by the uncovered (i.e. unexplained) input features. For our positive word rule, the coverage is the frequency of those words in the corpus and not surprisingly is only  $1.6\%$ . By contrast, including a catch-all rule in the union maximizes out its coverage value at  $100\%$ .
Definition 3.5 (Validity). Let  $\mathbb{P}_{a(U)}$  be  $\mathbb{P}_U$  truncated to the set of applicable FEUs. The validity of a rule (union)  $r = \langle a,b\rangle$  is then defined as follows, capturing the intuitive notion of a "correct" understanding:
$$
\nu (r) = \mathbb {E} _ {U \sim \mathbb {P} _ {a (U)}} [ \mathbb {1} \{e (U) \in b (U) \} ]. \tag {6}
$$
For our example, we compute it as the frequency that the saliency of those words is actually in the range of [0.479, 1] – which turns out to be only  $3.1\%$  of the time. However, validity alone is not sufficient, as it increases with wider behavior range. We thus establish sharpness as a competing metric.
Definition 3.6 (Sharpness). Let  $\mathbb{P}_E$  be the probability measure corresponding to the marginal distribution over explanation values generated by the explainer on  $u\sim \mathbb{P}_U$ . The sharpness of a rule (union)  $r = \langle a,b\rangle$  is defined as follows:
$$
\sigma (r) = \mathbb {E} _ {U \sim P _ {a (U)}} \left[ 1 - \mathbb {P} _ {E} (b (U) \backslash U) \right], \tag {7}
$$
where  $b(U)_{\backslash U} = b(U) \backslash \{U\}$  removes the actual attribution value  $U$  from the behavior range to pre
vent penalizing sharpness simply because the attribution value is very common (e.g., zero for sparse explanations), in which case  $\mathbb{P}_E$  is discrete at  $U$ .
Sharpness represents precision in the understanding, as  $1 - \sigma (r)$  gives the probability that a random FEU explanation value is correct. Thus, a lack of precision represented by a wide behavior range has minimal sharpness. We use the probability measure  $\mathbb{P}_E$  to define the "size", as it is consistent across all explanation distributions, most of which are non-uniform. A more general interpretation of sharpness is the consistency of the described model behavior: if a behavior range is wide (e.g., containing very positive and negative saliencies), then it is less sharp, and hence less useful.  $\mathbb{P}_E$  could be replaced by an application-specific diversity measure, though the precision notion may be lost.
There is generally a trade-off between validity and sharpness, as more precise rules (i.e., those with narrower behavior ranges) are less likely to be valid. For our rule, the probability of a random word saliency being in [0.479, 1.0] is  $0.2\%$ , indicating that explanation values are rarely higher than 0.479. This makes sharpness very high at  $99.8\%$ . However, the rule is not useful because of its low validity; i.e., it is almost never correct. By comparison, the looser range of  $[-0.01, 1.0]$  has  $90.4\%$  validity but  $28.6\%$  sharpness. There is another trade-off between coverage and the two, since a larger set of covered FEUs tends to be more diverse, making it harder to write a  $b$ -function that remains as valid and sharp simultaneously.
Since these metrics are all expected values, we can estimate them by their empirical estimate from a dataset (i.e., a simple average), and  $\mathbb{P}_E$  can be constructed by kernel density estimation.
4 EXSUM Development Process and GUI
We describe a systematic procedure for authoring EXSUM rule unions from scratch and utilize it in Sec. 5. Starting from an empty rule union with
Figure 3: ExSUM inspection GUI.
no FEUs covered, we iteratively create rules that target uncovered FEUs. Each rule describes one model behavior, such as that for highly positive words. For a rule, the  $a$ - and  $b$ -functions need to be defined, which may involve setting and tuning parameters, such as the sentiment threshold. Last, we add a lowest precedence catch-all rule if any FEUs remain uncovered. During this process, we may also merge or split rules and change the composition structure according to the metric values.
To support these steps, we developed a Python Flask-based (Grinberg, 2018) graphical user interface (GUI, Fig. 3). Users can visualize the FEUs, with font formatting for their coverage and validity. Users can also filter for uncovered or invalid FEUs, iteratively constructing and refining the rule union. EXSUM rule definitions usually include parameters such as the sentiment threshold. Manually selecting correct values for the parameters is tedious, so the lower middle panel of the GUI implements automatic parameter tuning for a given target metric value. Installation and usage instructions for the GUI are available on the project page<sup>1</sup>.
We construct ExSUM rule unions for SST and QQP models (details in App. B). We split the test set into a construction set to create the rule union and tune its parameters (analogous to the training and validation set in supervised model training) and an evaluation set to compute unbiased estimates of the metric values (analogous to the test set).
5.1 Sentiment Classification
Setup We use SHAP explanations (Lundberg and Lee, 2017) for fine-tuned RoBERTa (Liu et al., 2019), and take 300 random sentences as the construction set, with the remaining 1910 sentences as the evaluation set. We compute five features for each FEU: sentiment score, part of speech (POS), named entity recognition (NER), dependency tag (DEP) and word frequency. For example, the word "same" in the sentence "They felt like the same movie to me." has sentiment score of 0.028, POS = ADJ, NER = O, DEP = amod, and frequency of 7.14e-4, with SHAP saliency of -0.82.
Current Practice We evaluate the current practice of extracting informal model understanding from local explanation inspection against the three
<https://yilunzhou.github.io/exsum/
Figure 4: Coverage and validity metrics for the three current practice modes. Tab. 4 of App. B.1.1 presents the complete numerical data (also with sharpness).
metrics. We assess three values of  $K$ , the number of inspected instances: 1, for the typical ad hoc setting of generalization from a single explanation, 10, for a more careful investigation, and 30, which is quite cumbersome for manual inspection. These examples are selected either randomly or by submodular pick (Ribeiro et al., 2016). Next, we consider three ways to extract model understanding – belief-guided (BG), quantile-fitting (QF) and word-level (WL) – and apply them to create rules on strongly positive words and stop words introduced in Sec. 1. For the strongly positive word rule, BG mandates that words more positive than the average sentiment score should have an above-average saliency score, representing the belief of a positive correlation between the two. For the stop word rule, a saliency range belief of  $[-0.05, 0.05]$  is averaged with the observed range. For both rules, QF extracts the  $5\% -95\%$  quantile interval of the saliencies for words covered by the respective rule. WL, by contrast, creates a behavior range for each word seen, with 0.03 margin on both sides. App. B.1.1 presents technical details for these.