SlowGuess/NAACL_2022_OCR huggingface dataset:
Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5356-5371, Online. Association for Computational Linguistics.
Weili Nie, Yang Zhang, and Ankit Patel. 2018. A theoretical explanation for perplexing behaviors of backpropagation-based visualizations. In International Conference on Machine Learning, pages 3809-3818. PMLR.
Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, and William W. Cohen. 2022. Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? Transactions of the Association for Computational Linguistics, 10:359-375.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why should I trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135-1144.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Anchors: High-precision model-agnostic explanations. In Proceedings of the AAAI conference on Artificial Intelligence, volume 32.
Alexis Ross, Ana Marasovic, and Matthew Peters. 2021. Explaining NLP models via minimal contrastive editing (MiCE). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3840-3852, Online. Association for Computational Linguistics.
Wojciech Samek, Alexander Binder, Gregoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. 2016. Evaluating the visualization of what a deep neural network has learned. IEEE Transactions on Neural Networks and Learning Systems, 28(11):2660-2673.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Conference on Learning Representations.
Robert Wolfe and Aylin Caliskan. 2021. Low frequency names exhibit bias and overfitting in contextualizing language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 518-532, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2021. Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6707-6723, Online. Association for Computational Linguistics.
Yiming Zheng, Serena Booth, Julie Shah, and Yilun Zhou. 2021. The irrationality of neural rationale models. arXiv preprint arXiv:2110.07550.
Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, and Julie Shah. 2022. Do feature attribution methods correctly attribute features? In AAAI Conference on Artificial Intelligence.
A Real World Use Cases for Explanations
Here, we discuss several scenarios in which people use local explanations to understand models, and argue that people invariably derive generalized model understanding from these explanations.
A.1 Spurious Correlation Identification
Natural datasets can contain many spurious correlations. For example, in a COVID-19 chest X-ray dataset, most positive images (i.e., patients diagnosed with COVID-19) come from a pneumonia-specializing hospital and contain a watermark of the hospital name, while most negative images from other hospitals do not. Thus, a model could achieve very high accuracy by simply detecting the watermark rather than genuine medical signals. Similar spurious correlations could also be present in the text domain, such as the correlation between an exclamation mark and the positive sentiment class, or between the word "not" and the contradiction class in natural language inference.
It is crucial for people to be aware of the shortcuts that models may take, and one possible way to highlight such behaviors is via feature attribution, which in the examples above would assign an abnormally high score to the watermark region, exclamation mark, or the word "not." Assuming the explanations do indeed exhibit such patterns, when people claim a model relies on spurious correlation, they mean this in a general sense: for example, the model is likely to focus on the watermark in any image that contains it, rather than in only a specific set of images.
A.2 Fairness Assurance
Similar to spurious correlation features, other features should not have a high impact, but for reasons of fairness. For example, decisions made by a loan approval model should not be affected by gender<sup>3</sup>, therefore the gender feature should not have a high attribution score.
If we observe that the gender of one applicant heavily impacts the model's decision, we may suspect the model is discriminative; conversely, observing that it has minimal impact could increase our assurance of the model's fairness. However, such single-instance observations are fundamentally exploratory, and claims about the model's fairness or discrimination must be established using a population of instances to determine whether the trend persists generally.
A.3 Model-Guided Human Learning
In some cases, a very accurate and "super-human" model could be a source for knowledge discovery. Consider the task of early-stage cancer detection from CT scans, which is challenging for doctors. If a label is generated from follow-up visits tracking whether patients develop cancer after a certain number of years, a model achieving better test accuracy than doctors is likely to use certain cues that would be missed by humans or not known to be linked to cancer.
For these models, explanation methods such as saliency maps could be used to help doctors make better diagnoses, or assist scientists in the creation of new pathological theories. Similarly to the above two use cases, generalized model understanding across different inputs are necessary, because doctors need to apply what they have learned to new patients, and scientists require new theories to hold broadly.
B Additional Evaluation Details
Tab. 3 summarizes the key parameters of our experiment. Both saved models are publicly accessible from Huggingface Hub, and the model names in the table are links to the respective model checkpoints. For normalization, we divided all explanation values for all test set instances by a single scaling factor such that the maximum magnitude of new explanations is 1.
B.1 SST Sentiment Classification
For the explainer, we used the PartitionSHAP algorithm implemented by the shap repository<sup>4</sup>. Fig. 6 shows the explanations on three sentences (after normalization).
There could be other features that correlate with gender, such as job title, but we ignore such possibilities for simplicity.
4https://shap.readthedocs.io/en/latest/
Task	Dataset	Model	Acc.	F1	Explainer
Sentiment	SST-2	RoBERTa	95.6%	0.957	SHAP
Paraphrase	QQP	BERT	90.7%	0.875	LIME
Table 3: A summary of tasks, models (fine-tuned on respective datasets), and explainers for the two case studies.
Figure 6: SHAP explanation visualization for three SST inputs.
B.1.1 Details of Current Practices
Here, we provide an extended description of the three current practices, and how they are applied on the handful of selected examples, collectively called the "sample" below.
The first method, "belief guided" (BG), represents the practice wherein the user has some expectations (or beliefs) about the attributions of certain words, and modifies (or updates) them after observing explanations on some actual test inputs. It operates differently for the two rules on positive-sentiment and stop words, as follows.
1. For positive-sentiment words, the prior belief is that a word with a higher sentiment score (one of the FEU features provided by the SST dataset) should also receive more positive attribution. This leads to a rule that applies to all words with a sentiment score greater than  $\alpha$ , and has a behavior function that outputs a constant range of  $[\beta, 1]$  (recall that SHAP attribution values are normalized to  $[-1, 1]$  range). It then computes the value of  $\alpha$  as the mean sentiment score and  $\beta$  as the mean attribution value for all words in the sample with positive sentiment scores.
2. For the stop words - defined as those with parts of speech AUX, DET, ADP, CCONJ, SCONJ, PRON, PART, and PUNCT - it has a prior belief that they should have a attribution value range of [-0.05, 0.05] (i.e., not important to model prediction), and computes the observed attribution range  $[\alpha, \beta]$  for stop words in the sample. The final behavior range as predicted by the behavior function of this rule is the average of these two:  $[- (0.05 + \alpha) / 2, (0.05 + \beta) / 2]$ .
The second method, "quantile fitting" (QF), represents the practice wherein the user fully follows the observed data without any prior beliefs. Specifically, for a set of words, it collects all attribution values for words within the set and then creates a rule that applies to this set, with the behavior function predicting a constant range of  $5\%$  to  $95\%$  quantile of these attribution values. For the two rules for positive-sentiment and stop words, the set of words (and hence the applicability functions) is defined in the same way as for the BG method above.
The last method, "word-level" (WL), can be considered a more extreme version of QF, where the user not only lacks any prior expectations for the explanations but also considers each word individually. For example, if the user observes that the word "brilliant" has an attribution value of 0.5 in one sentence and the word "fantastic" has attribution of 0.8 in another, they would not conclude that other, similarly positive words would have attributions approximately within the range of [0.5, 0.8]. Specifically, for every distinct word  $w$  in the sample, this method builds a rule that applies only to that word, with a constant behavior
function that outputs a range of  $\left[\min (s_w) - 0.03,\max (s_w) + 0.03\right]$ , where  $s_w$  is the list of attributions received by different occurrences of  $w$ . In many cases, especially given a small sample, word  $w$  only appears once, in which case  $s_w$  is a list containing only that attribution value.
Tab. 4 presents the metric values of the above methods. Fig. 4 of Sec. 5.1 depicts a graphic summary.
K	pick	belief-guided	quantile-fitting	word-level seen words
positive	stop word	positive	stop word
1	SP	10, 72, 50	49, 45, 65	10, 63, 44	49, 63, 45	28, 51, 61
RND μ	12, 63, 57	49, 58, 53	12, 45, 56	49, 73, 33	17, 41, 68
RND σ	6, 25, 27	0, 9, 6	6, 32, 29	0, 24, 21	6, 12, 10
10	SP	10, 61, 61	49, 47, 63	10, 78, 34	49, 72, 38	49, 66, 48
RND μ	10, 71, 52	49, 56, 56	10, 75, 32	49, 84, 25	41, 68, 48
RND σ	0, 6, 7	0, 4, 4	0, 9, 10	0, 3, 2	2, 1, 2
30	SP	10, 64, 59	49, 50, 60	10, 88, 17	49, 82, 29	57, 73, 42
RND μ	10, 66, 56	49, 57, 55	10, 82, 26	49, 86, 24	51, 78, 39
RND σ	0, 4, 5	0, 1, 2	0, 6, 7	0, 2, 2	2, 3, 2
Table 4: Coverage, validity, and sharpness (percentage) of model understanding with ad hoc current practice. "SP" refers to the submodular pick procedure, and "RND" refers to the random sampling procedure. The latter also shows mean  $\mu$  and stdev  $\sigma$  across five runs.
B.1.2 Complete Rule Union Description
Below, we present the details of the construction process for rules not discussed in Sec. 5.1.
- **Rule 1: Negation words have negative saliency.** We found that negation words – not, n’t, no, nothing and those with NEG dependency tag – almost invariably receive (sometimes highly) negative saliency, regardless of the sentence label or sentiment of the word being modified. We create a rule that predicts a constant behavior range  $[-1.0, 0.002]$ , with  $89.5\%$  validity and  $65.1\%$  sharpness. Although the validity is under our  $90\%$  target, we found that to make it higher, the upper limit of the behavior range needs to be 0.1, which results in an extremely low sharpness of  $11\%$ . Thus, we decided against it.
- **Rule 5: Person names have positive saliency.** During our initial inspection, we found several cases where the name of a person (e.g. director or actor) have positive saliency values. Thus, we create this rule from the NER tag, covering  $2.3\%$  of words. However, after parameter tuning, we found that while many of the words have positive saliency, the correct characterization is that they all have small saliency values, as a behavior range of  $[-0.06, 0.1]$  achieves  $91.6\%$  validity. However, since SHAP saliencies are mostly concentrated around 0, this range achieves a meager sharpness of  $26.8\%$ . Despite this, we still decide to keep it.
- Rule 7: Zero-sentiment words have small saliency. Besides stop words, we should expect words that do not carry sentiment, such as most nouns and verbs (e.g., movie and get), to have small saliency magnitudes. Due to the wide range of words applicable under this rule, we choose the saliency range to be  $[-0.15, 0.15]$  for  $\geq 90\%$  validity, but this range yields lowest sharpness of  $13.5\%$ .
- Rule 8, 9: Weakly positive/negative words have weakly positive/negative saliency. Finally, we set up two rules to capture words that have sentiment of neither zero (covered by Rule 19) nor high-magnitude (covered by Rule 3 - 5). To achieve  $90\%$  validity, we require a behavior range of  $[-0.11, 1]$  for weakly positive words and  $[-1, 0.05]$  for weakly negative words, unfortunately again with quite low sharpness. Notably, both ranges need to "spill over" to the other side of zero for the required validity.
B.2 QQP Paraphrase Detection
For the explainer, we used the LIME algorithm implemented by the lime repository<sup>5</sup>. Fig. 7 depicts the explanations on two pairs (after normalization).
Figure 7: LIME explanation visualization for two QQP pairs.
B.2.1 Complete Rule Union Description
Below, we present details of the construction process for rules not discussed in Sec. 5.2.
- Rule 4, 5: Saliency for trailing question marks. Since the dataset is composed of pairs of questions, the vast majority of sentences conclude with question marks. These should be purely decorative and syntactic, and so should have small saliency, similar to stop words. However, we observe that the saliencies assigned to them for positive and negative predictions are very different, so we create two rules for these two cases. With a  $90\%$  validity target, the saliency range is  $[-0.04, 0.03]$  for negative predictions and  $[-0.07, 0.06]$  for positive predictions. Again, the saliencies for positive predictions demonstrate more variation than those for negative ones.
- Rule 6, 7: Saliency for stop words. Similar to SST, we use these two rules to ensure stop words should not be influential. We split the stop word group into finer segments by part of speech, to achieve higher sharpness. On average, the range is  $[-0.07, 0.03]$  for negative predictions and  $[-0.09, 0.1]$  for positive predictions, which again demonstrate a much higher degree of variation.
- Rule 8.9: Saliency for negation words. In the SST case, we found that negation words typically have negative saliency regardless of the sentiment label, and test whether this holds for QQP as well. Following on our previous findings, we use two rules to separately model inputs of positive and negative predictions. We find that the range is  $[-0.1, 0.24]$  for positive predictions and  $[-0.21, 0.01]$  for that for negative predictions. Curiously, the same negative saliency trend is preserved here as well, but only for inputs with negative predictions.
- Rule 10, 11: Saliency for everything else. Finally, we designed two lowest-precedence "catch-all" rules to complete the coverage. The range for positive prediction FEUs is [-0.13, 0.25]. For negative prediction inputs, we find that breaking them according to different parts of speech (nouns, verbs, adjectives, and everything else) is helpful, with verbs having a particularly narrow saliency range of [-0.05, 0.05]. On average, the saliency range is approximately [-0.09, 0.05].
C Understandability as a Unified Theme
In this section, we elaborate on how understandability is the unified theme behind many properties of explanations that seem "orthogonal" to correctness. Specifically, we discuss three properties: human alignment, robustness, and counterfactual similarity and plausibility.
<sup>5</sup>https://github.com/marcotcr/lime/
C.1 Human Alignment
Many prior works have assessed how much explanations agree with human expectation. For example, Li et al. (2016) observed that the word "hate" contributes the most to a negative sentiment prediction in many inputs, and used it to argue the explanation is correct. In a similar sentiment classification task, Bastings et al. (2019) used the high degree of overlap between the extracted rationale and strong-sentiment words to argue the superior quality of a neural rationale model (Lei et al., 2016). In computer vision, this alignment is often implemented as a pointing game that computes the intersection-over-union (IoU) metric between the salient region and the semantic segmentation mask of the predicted class (Simonyan et al., 2014; Fong and Vedaldi, 2017), as shown in Fig. 8. For a model that predicts breast cancer onset using patients' genetic information, Covert et al. (2020) demonstrated that many of the influential genes identified by their explainer were indeed known to be associated with the disease.
Figure 8: A pointing game used to quantify human alignment for visual explanations.
As discussed in App. A.1, models could use any unexpected spurious correlation, such as the green background in Fig. 8. For these models, correct explanations should have low alignment scores. When correctness (or faithfulness) is the sole desideratum of interpretability methods, it is unclear what purposes these alignment evaluations serve. Some authors (e.g. Jacovi and Goldberg, 2020) have even argued they are fundamentally misleading and flawed in nature as they focus on *plausibility*, which is sometimes at odds with the goal of correctness.
However, from the perspective of understandability, high-alignment explanations are arguably very understandable, simply because they align closely with human expectation. Thus, given the same level of correctness, a higher-alignment explainer may be preferable.
C.2 Robustness
Besides human alignment, robustness – i.e. that similar inputs should have similar explanations – is also argued to be a favorable property for explanation. For example, Ghorbani et al. (2019) argued that explanations are fragile due to their adversarial vulnerability, Alvarez-Melis and Jaakkola (2018a) empirically estimated the Lipschitz constant for many explainers, and Alvarez-Melis and Jaakkola (2018b) proposed an inherently interpretable model that is explicitly regularized for explanation robustness.
Figure 9: A decision tree that splits on the second feature at the root node.
Robustness generally conflicts with correctness. If, for two inputs, the model is using distinct reasoning patterns, the correct explainer should faithfully report distinct explanations for them. One straightforward example is the decision tree model shown in Fig. 9, where the root node splits on the second feature at a threshold value of 3. For two inputs  $x_{1}$  and  $x_{2}$  that agree on all features except the second one, with  $x_{1}^{(2)} = 2.99$  and  $x_{2}^{(2)} = 3.01$ , since they are sent down two different sub-trees at the very beginning, the model is likely to use for totally different features.
Nonetheless, as implicitly argued by the works above, erratic model behaviors are less understandable because they make it more difficult to identify generalizable patterns compared with slowly varying explanations in the input space. Thus, robustness is another aspect of the same understandability desideratum.
C.3 Counterfactual Similarity and Plausibility
Unlike feature attribution explainers that assign importance to individual features, counterfactual (CF) explainers (e.g., Ross et al., 2021) directly generate whole inputs but for a target predicted class. Thus, a CF explanation indicates how to cross the decision boundary from the input.
Naturally, the fundamental requirement of CF explanations is achieving the target prediction, which is typically known as validity. However, this is trivially satisfiable by simply finding a training instance with the target prediction, along with other ways such as creating adversarially perturbed or nonsensical inputs. Thus, two additional requirements are often enforced: similarity and plausibility. The former says that the CF explanation should be close to the original input (with regard to, for example, edit distance), and the latter says the CF explanation should be plausible, or natural. Tab. 5 depicts various CF explanations and their satisfaction of the three requirements.
Input: This restaurant is the best I have been, with especially great food.
CF	Type	Val.	Sim.	Plau.
This restaurant is the worst I have been, with especially terrible food.	“good” CF	✓	✓	✓
Rude service!	training set look-up	✓	×	✓
This restaurant is the best I have been, with especially great food.	adversarial typo injection	✓	✓	×
Fjwpeaf fawekl fka erj sfdlk erjlm adl erio fd	nonsensical inputs	✓	×	×
Table 5: CF explanations that are all valid but differ in similarity and plausibility metrics.
Validity for CF can be considered as the correctness analogy for feature attribution, but the purposes of similarity and plausibility are not readily apparent. As CF explanations represent ways to cross the decision boundary, people need to meaningfully understand how the CF instance is related to the original input. It is difficult to relate two dissimilar instances, and an implausible CF instance is generally unexpected. Thus, similarity and plausibility are required to make CF explanations more understandable.
Interestingly, if our true goal is the understandability of the relationship between the input and its CF explanation, there are cases where similarity or plausibility is not desirable. First, consider a sentence length classifier that predicts positive for sentences of at least 10 words, and negative otherwise. Given an input of three words, the CF explainer should generate dissimilar CF instances of at least 10 words in order to correctly illustrate the decision boundary, while instances of even more words would be helpful for understanding the "at least 10 words" logic. Second, consider a classifier trained on a typo-free dataset and having high probability of making mistakes on inputs that contain typos. To illustrate this behavior, CF explanations should contain randomly (not adversarially) injected typos, which are implausible, but useful as long as the typo injection is understood by people.
D Additional Details on Instance-Based Explanations
In this section, we describe our initial attempt at extending the EXSUM framework to another type of explanations: instance-based explanations (IBE). The IBE for an input  $x$  is a set of instances and their predictions  $\{(x_i, \widehat{y}_i)\}$ , where  $x$  and  $x_i$  are semantically related (e.g., negation). We define  $\widehat{y}_i$  as the predicted probability of positive class.
Type	b(ŷ)	ν	σ
Entity change	ŷ ± 0.05	91.4	56.5
Minor insert	ŷ ± 0.05	89.1	57.3
Negation	other-side(ŷ)	30.4	50.0
Negation	same-side(ŷ)	69.6	49.9
Negation (≤ 6 words)	other-side(ŷ)	56.2	49.7