SlowGuess/NAACL_2022_OCR huggingface dataset:
We formalize the understanding derived from the selected instances and plot their coverage and validity metrics on the evaluation set in Fig. 4. For BG and QF, the bars represent the average metric value of the positive word and stop word rules. For WL, the bars represent the metric for the rule union consisting of an individual rule for each unique word. Error bars for the random pick represent the standard deviation across five iterations. Tab. 4 of App. B.1.1 presents the complete statistics for all metric values, and we highlight several findings.
- A very small number of samples (e.g., 1) exhibit large variance for random pick, and low validity for both pick methods. This confirms the intuition that model understanding from very few explanations should be avoided.
Figure 5: Behavior ranges can vary widely and unpredictably on similar words for WL rules.
- BG overall yields low validity, because its "beliefs" turn out to be quite incorrect. This suggests a strong prior belief about how the model works could lead to incorrect conclusions.
While submodular pick can select a more diverse set of words, to the particular benefit of the coverage of  $\mathrm{WL}^2$ , its validity is generally lower due to under-representation of common words.
- Although WL achieves highest coverage and validity, it has  $>500$  rules at  $K = 30$ , with similar words having very different ranges, as shown in Fig. 5 - a conglomerate (almost) impossible to make sense of. It also overfits, as the evaluation set validity is much lower than the construction set validity (which is  $100\%$  by construction).
- At  $K = 10$ , only the stop word rule with random pick QF achieves validity  $>80\%$ , indicating that even the more careful practices are unreliable.
All the drawbacks call for a principled way to derive robust model understanding with enforceable metric values (e.g. validity). As we demonstrate next, given a large construction set and automatic parameter tuning assistance, we can create such a ExSUM rule union. Finally, as a meta-point, the above discussion above of various limitations would not be possible without the proposed ExSUM formalization and metric definitions.
ExSUM Construction We create a rule union consisting of nine rules, with target validity of  $90\%$  and tune the sharpness accordingly. Tab. 1 summarizes the individual and aggregate metrics.
Clearly, high validity comes at the cost of low sharpness. Since (1 - sharpness) is the probability that a random FEU has an explanation value within the behavior range, this around  $90.7\%$  validity should be put into a context where the random baseline achieves a validity of around  $75\%$ . In this sense, we attain only a crude understanding of the local explanations that misses many subtleties.
Nonetheless, Rule 3 (strongly positive words)
2The other two are less affected because the subject of the rule (e.g., stop words) largely dictates which words it covers.
Idx	Rule	Cov%	Val%	Shp%
1	Negation	1.2	89.5	65.1
2	Strongly neg. adj	3.2	91.6	83.5
3	Strongly pos. words	5.1	91.9	40.0
4	Strongly neg. non-adj	1.2	89.9	71.4
5	Person name	2.4	90.9	28.4
6	Stop words	47.5	90.8	23.5
7	Zero-sentiment words	17.1	90.0	15.6
8	Weakly pos. words	15.4	91.2	11.3
9	Weakly neg. words	5.7	91.7	31.4
Un-ion	On construction set	100	90.7	26.1
On evaluation set	100	89.4	26.2
and Rule 6 (stop words) achieve better validity-sharpness trade-off than their counterparts created using the ad hoc BG and QF methods above. Moreover, the WL rules cover all words seen in the analyzed instances - analogous, in a sense, to our ExSUM rule union. While the validity-sharpness trade-off is comparable between the two, ours has  $100\%$  coverage due to the effectively "catch-all" Rule 7, while WL rules have less than  $60\%$ . Most importantly, as our rule union is composed of nine semantically organized rules, it is much more interpretable than WL, which include more than 500 unpredictably varying rules (Fig. 5).
The fact that the EXSUM rule union reveals the imprecision and limitations of our model understanding while still performing better than current practice emphasizes the need for more formal and quantitative model understanding, as well as the development of methods that are easier to understand, in addition to being correct. Below, we highlight two sets of rules that quantitatively support or refute our intuition, and cover the rest in App. B.1.2. Rule 2, 3, 4, 8, 9: Sentiment-carrying words. We expect a sentiment classifier to recognize sentiment-laden words. To test our intuition, we create rules for positive and negative words, and further split each set of words into two according to sentiment strength, resulting in four rules. For the two rules on strong words, we find that wide behavior ranges of [0.01, 1] and [-1, -0.01] are necessary to achieve  $90\%$  validity, suggesting the looseness of the model understanding. However, we do observe that negative adjectives (but not positive ones) are modeled much better, where a range of [-1, -0.06] is sufficient for the same validity. Thus, we create a separate Rule 2, with very high sharpness of  $84.2\%$ . For the two rules on words of weaker sentiment,
Table 1: Metrics for SST rules and rule union.
Idx	Rule	Cov%	Val%	Shp%
1	Matching words neg. pred	11.7	90.9	39.5
2	Matching words pos. pred	12.4	90.3	38.6
3	Non-matching words neg. pred	18.7	90.0	35.5
4	Question mark neg. pred	5.2	90.2	36.5
5	Question mark pos. pred	3.8	90.0	23.1
6	Stop words neg. pred	22.3	90.0	32.8
7	Stop words pos. pred	12.6	90.5	12.5
8	Negation words neg. pred	0.3	90.0	36.0
9	Negation words pos. pred	0.1	95.7	7.2
10	All else neg. pred	4.0	92.1	23.5
11	All else pos. pred	8.8	90.3	5.7
Un- ion	On construction set	100	90.3	29.3
On evaluation set	100	90.0	29.1
Word Avg	On construction set	100	90.8	29.4
On evaluation set	82.3	84.4	29.4
Table 2: Metrics for QQP rules and rule union. The last two rows are for the baseline at the end of Sec. 5.2.
even wider ranges of [-0.11, 1] and [-1, 0.05] are necessary. Since both ranges encroach upon the other side, the model often considers these words to have an impact opposite to their intrinsic meaning, but we fail to extract further understanding. In addition, negative rules are much sharper than positive ones, suggesting that the model considers a negative word to be stronger evidence for a negative prediction than its positive counterpart.
Rule 6: Stop words. While stop words (e.g., "the", "of") should have negligible impact on prediction (and saliency values close to zero), a narrow behavior range of  $[-0.05, 0.05]$  only has  $64\%$  validity. We create this rule for all stop words with  $90\%$  target validity and use different ranges on different words for better sharpness. On average, we get  $[-0.07, 0.12]$ , demonstrating that they can sometimes be more influential than even strong sentiment words. The ranges also tilt to the positive side, uncovering a grammaticality bias wherein prediction is more negative for grammatically incorrect sentences with stop words masked out by SHAP.
5.2 Paraphrase Detection
Setup We use LIME explanations (Ribeiro et al., 2016) for a fine-tuned BERT model (Devlin et al., 2019), with 500 random test sentences as the construction set and the remaining  $\approx 40\mathrm{k}$  as the evaluation set. We remove the word sentiment feature but add the question ID (1 or 2) of each FEU.
ExSUM Construction QQP is a more complex domain than SST, since the label is the semantic equivalence of two sentences. The metric values for the ExSUM are summarized in Tab. 2. Below,
we describe how expectations for the model are validated, but a hidden – and somewhat surprising – phenomenon is also uncovered. All other rules are documented in App. B.2.1.
Rule 1, 2: Matching words. Due to the nature of the task, we expect the model to rely heavily on matching words. For such a word  $u$ , defined as (proper) noun, verb, adjective or pronoun that has exactly one case-insensitive match  $v$  in the other question, we expect similar saliency to their match due to symmetry, or formally its saliency  $s_u \in [s_v - \alpha, s_v + \beta]$ , where  $\alpha$  and  $\beta$  are lower and upper margins. This behavior function is nonconstant, with output depending on the saliency values of other words in the sentence.
For the same margin, FEUs for pairs of negative predictions have much higher validity than positive ones, so we split the rule into two based on the prediction. Despite a less than  $1\%$  difference in sharpness (Tab. 2), we have  $\alpha = \beta = 0.07$  for the negative rule, but 0.18 for the positive rule, suggesting that the matching words make a much larger and more unpredictable contribution to positive predictions. Interestingly, all other rules had wider intervals for positive predictions as well.
Rule 3: Non-matching words. Next we study model behaviors for non-matching words, defined analogously to matching ones. Following the previous split based on predicted label, we designed two rules. The negative rule has a reasonably sharp behavior range of  $[-0.35, 0.01]$  at  $90\%$  validity. Given that LIME saliency is the linear regression coefficient on a neighborhood created by word erasure, we conclude that the presence of these non-matching words mostly causes the prediction to tilt toward the non-paraphrase (i.e. negative) class, indeed a very reasonable behavior. However, we cannot find a range with  $10\%$  sharpness at  $90\%$  validity for the positive rule and thus discard it.
With regard to the sharpness contrast by predicted label, one explanation is that the model defaults to a negative prediction, since many negative pairs consist of completely unrelated questions and the model decision is largely insensitive to input perturbations, leading to stable LIME coefficients. On the other hand, a positive prediction requires the cooperation of all parts of both questions. Depending on the exact sentence structure, the importance of each word to the match are different and hard to predict, which prevents the rules from being sharp.
Word Average Baseline Here, we introduce a new baseline as an "automated" version of WL rules in SST. Specifically, for each word in the construction set, we compute a behavior range around its average saliency, with sharpness of  $29.4\%$  (matching that of our ExSUM rule union). As Tab. 2 shows, the resulting rule union is much worse than our manual one on both evaluation set coverage and validity, which is not surprising as the word saliency should be more context-dependent, due to the matching mechanism of paraphrase detection. Moreover, with more than 2,000 constituent rules, the rule union barely qualifies as any sort of generalized model understanding.
As discussed in Sec. 1, explanation evaluation usually has a focus on correctness (or faithfulness) – i.e., whether the explanation truly reflects the model's reasoning process. This includes sanity checks (Adebayo et al., 2018), proxy metrics (Samek et al., 2016; Arras et al., 2019), and explicit ground truth (Zhou et al., 2022). The understandability issue has been much less studied, with the exception by Zheng et al. (2021), who proposed an evaluation specifically for rationale models (Lei et al., 2016). EXSUM, however, addresses post hoc explanations of general black-box models.
In addition, a few prior works have attempted to capture the "end-to-end" utility of explanations: whether access to explanations leads to performance increase in certain tasks. Hase and Bansal (2020) propose a model-teaching-human setup, subsequently extended by Pruthi et al. (2022) into an automated evaluation procedure. Bansal et al. (2021) study whether explanations can improve human-machine teaming performance. While these studies report mostly negative results, pinpointing the root cause is difficult due to their end-to-end nature. Poor understanding of the explanations may be a major reason, as indicated by ExSUM.
Last, some authors have proposed methods for understanding model predictions beyond individual instances. For example, the anchor method (Ribeiro et al., 2018) generates an explicit domain of applicability for each explanation, while Lakkaraju et al. (2016) and Lakkaraju et al. (2019) proposed to learn "patches" of the input space specified by logical predicates. EXSUM also emphasizes the need to understand models that generalize across instances, and uses logical predicates in
the formulation, but focuses on model understanding via explanations instead of direct predictions, which can capture a wider variety of behaviors (e.g. the matching and non-matching behaviors of the QQP model). Furthermore, the fine-grained analysis of behaviors allows us to investigate whether models are "correct for the correct reason."
7 The Many Faces of Understandability
The central thesis of this paper is quite simple and intuitive: in order to understand a model from local explanations, we need to understand those local explanations. While ExSUM is the first framework to explicitly formalize and quantify the notion of understandability, we argue that it is connected to many often-discussed and desirable properties of explanation (further details in App. C).
Human Alignment Users sometimes expect explanations that are aligned with their expectations. For example, the fact that highly salient words convey strong sentiment is taken as evidence for the quality of an explanation method by Li et al. (2016). In image classification, this concept is typically implemented as a pointing game between the high-saliency region and the segmentation mask of the predicted class (Fong and Vedaldi, 2017). However, alignment does not imply correctness, as the model could use any spurious correlations, which should be faithfully highlighted by the explainer. However, higher-alignment explanations are indeed more understandable, since by definition they agree more with human intuition. Thus, an alignment-based evaluation can be considered as one of understandability. Nonetheless, understandability can also be achieved by correcting human expectations, e.g., users realizing that punctuations are actually important for predictions (contrary to expectations).
Robustness It is often argued that explanations should be robust (Ghorbani et al., 2019) – similar inputs should induce similar explanations. However, robustness can be at odds with correctness: if the model truly applies vastly different logic for two very close inputs – such as a pair of inputs that only differ in the root feature of a decision tree – then their explanations should be distinct, as they are routed down two different sub-trees. Nonetheless, slow-varying explanations are generally easier to understand than those that change erratically and unpredictably (independent of their correctness), and thus robustness is related to understandability.
Counterfactual Similarity and Plausibility Counterfactual explanations (e.g. Ross et al., 2021) indicate how the input should change in order to alter the model prediction. Besides the success rate of achieving target prediction, they are often evaluated on similarity (the magnitude of input change) and plausibility (the naturalness of the changed input). Both properties can serve as proxies for understandability: it is easier to relate an input to another similar and natural input than to a totally different or abnormal one. However, App. C presents two cases where they should not be similar or plausible but remain understandable, to highlight certain model behaviors.
Traditionally, model explanations are evaluated on correctness (or faithfulness), i.e., whether they correspond to how models actually make predictions, e.g., reliance on spurious correlations (Zhou et al., 2022; Adebayo et al., 2022). Such evaluation, however, does not answer the equally important question of whether these (presumably correct) explanations are understandable. Even faithful explanations can lead users into error, if misunderstood (e.g., trusting a model incorrectly).
In a sense, the most correct explanation for an input is the literal trace of model computation, but it is also arguably the least understandable (or useful). As we abstract away from low-level details and use higher-level concepts such as word sentiment, the resulting explanation loses correctness but gains understandability. At the other extreme are explanations that are trivially understandable but completely wrong, such all attribution values being 0 (i.e., no feature impacts the model prediction). Thus, a trade-off often occurs between these two desiderata, and we need to choose a sweet spot.
Concretely, we propose ExSUM rules and rule unions, along with three quality metrics to formalize and evaluate understandability. Such rigorous investigations stand in contrast to current ad hoc practices, which are prone to yielding unreliable and coarse model understanding. For SST and QQP datasets, ExSUM demonstrates that our model understanding is quite limited and imprecise, even with very reasonable explanations. Being aware of this is an asset. While ExSUM helps us to recognize that our understanding is incomplete, it still helps uncover unexpected model behaviors that warrant further investigation.
Limitations and Ethical Impacts
One notable requirement of EXSUM is the extensive human involvement in constructing and optimizing its rules. However, this process is necessary, as the alternative of generalizing from a few explanations has various flaws, depicted in Fig. 2 and Fig. 4. Practically, we spent about 3 hours on each rule union in Sec. 5, and our effort was streamlined by the systematic process and GUI presented in Sec. 4, which could be further improved by methods that automatically propose candidate rules.
In addition, another area requiring human involvement is the FEU feature definitions, which are often domain-dependent: both the sentiment score and the matching word features reflect the nature of the tasks. Other features may be necessary for other tasks. For example, in question-answering, one important FEU feature could be the kind of interrogative word used in the question (e.g., "what" vs. "when" vs. non-interrogative words). If important features are missed, the quality of the ExSUM rules – and, hence, the model understandings – will suffer accordingly.
Last, the difficulty of obtaining overall high-quality model understanding may result from the fundamental limitations of word-level attribution-based explanations, which cannot account for higher-level interactions. EXSUM could aid in the development of new explanation methods that are easier for humans to understand. As a first step, we explore defining and evaluating model understanding obtained from instance-based explanations with whole input as FEUs. App. D details the investigation, which raises questions such as the reliability of such explanations.
Ethical Impacts
As interpretability methods are increasingly deployed for quality assurance, auditing and knowledge discovery purposes, it is important to ensure the legitimacy of any conclusions drawn from explanations. While the correctness of these explanations is often studied, we argue their understandability should be equally emphasized, and evaluations with our newly proposed ExSUM framework and GUI reveal many problems of existing ad hoc procedures. Thus, a more careful treatment on the understandability aspect is necessary for well-calibrated model understandings and responsible model deployment in the real world.
This research is supported by the National Science Foundation (NSF) under the grant IIS-1830282. We thank the reviewers for their efforts.
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity checks for saliency maps. Advances in Neural Information Processing Systems, 31.
Julius Adebayo, Michael Muelly, Harold Abelson, and Been Kim. 2022. Post hoc explanations may be ineffective for detecting unknown spurious correlation. In International Conference on Learning Representations.
David Alvarez-Melis and Tommi S Jaakkola. 2018a. On the robustness of interpretability methods. In ICML Workshop on Human Interpretability in Machine Learning.
David Alvarez-Melis and Tommi S Jaakkola. 2018b. Towards robust interpretability with self-explaining neural networks. Advances in Neural Information Processing Systems, 31.
Leila Arras, Ahmed Osman, Klaus-Robert Müller, and Wojciech Samek. 2019. Evaluating recurrent neural network explanations. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 113-126, Florence, Italy. Association for Computational Linguistics.
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? The effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-16.
Jasmijn Bastings, Wilker Aziz, and Ivan Titov. 2019. Interpretable neural predictions with differentiable binary variables. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 2963-2977.
Ian Covert, Scott M Lundberg, and Su-In Lee. 2020. Understanding global feature contributions with additive importance measures. In Advances in Neural Information Processing Systems, volume 33.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ruth C Fong and Andrea Vedaldi. 2017. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference on Computer Vision, pages 3429-3437.
Amirata Ghorbani, Abubakar Abid, and James Zou. 2019. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3681-3688.
Miguel Grinberg. 2018. Flask web development: Developing web applications with Python. " O'Reilly Media, Inc."
Peter Hase and Mohit Bansal. 2020. Evaluating explainable AI: Which algorithmic explanations help users predict model behavior? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5540-5552, Online. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In International Conference on Learning Representations.
Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198-4205, Online. Association for Computational Linguistics.
Himabindu Lakkaraju, Stephen H Bach, and Jure Leskovec. 2016. Interpretable decision sets: A joint framework for description and prediction. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1675-1684.
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. 2019. Faithful and customizable explanations of black box models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 131-138.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107-117, Austin, Texas. Association for Computational Linguistics.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 681-691, San Diego, California. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.