SlowGuess/NAACL_2021_OCR huggingface dataset:
Shuoyang Ding Philipp Koehn
{dings，phi}@jhu.edu
Saliency methods are widely used to interpret neural network predictions, but different variants of saliency methods often disagree even on the interpretations of the same prediction made by the same model. In these cases, how do we identify when are these interpretations trustworthy enough to be used in analyses? To address this question, we conduct a comprehensive and quantitative evaluation of saliency methods on a fundamental category of NLP models: neural language models. We evaluate the quality of prediction interpretations from two perspectives that each represents a desirable property of these interpretations: *plausibility* and *faithfulness*. Our evaluation is conducted on four different datasets constructed from the existing human annotation of syntactic and semantic agreements, on both sentence-level and document-level. Through our evaluation, we identified various ways saliency methods could yield interpretations of low quality. We recommend that future work deploying such methods to neural language models should carefully validate their interpretations before drawing insights.
While neural network models for Natural Language Processing (NLP) have recently become popular, a general complaint is that their internal decision mechanisms are hard to understand. To alleviate this problem, recent work has deployed interpretation methods on top of the neural network models. Among them, there is a category of interpretation methods called saliency method that is especially widely adopted (Li et al., 2016a,b; Arras et al., 2016, 2017; Mudrakarta et al., 2018; Ding et al., 2019). At a very high level, these methods assign an importance score to each feature in the input feature set  $F$ , regarding a specific prediction  $y$  made by a neural network model  $M$ . Such feature importance scores can hopefully shed light on the neural network models' internal decision mechanism.
V	U.S.	companies	wanting	to	expand	in	Europe
SG	U.S.	companies	wanting	to	expand	in	Europe
IG	U.S.	companies	wanting	to	expand	in	Europe
Table 1: An example from our evaluation where different saliency methods assign different importance scores for the same model (Transformer language model) and the same next word prediction (are). V, SG and IG are different saliency methods (see Section 2). The tints of green and yellow mark the magnitude of positive and negative importance scores, respectively.
While analyzing saliency interpretations uncovers useful insights for their respective task of interest, different saliency methods often give different interpretations even when the internal decision mechanism remains the same (with  $F$ ,  $y$  and  $M$  held constant), as exemplified in Table 1. Even so, most existing work that deploys these methods often makes an ungrounded assumption that a specific saliency method can reliably uncover the internal model decision mechanism or, at most, relies merely on qualitative inspection to determine their applicability. Such practice has been pointed out in Adebayo et al. (2018); Lipton (2018); Belinkov and Glass (2019) to be potentially problematic for model interpretation studies – it can lead to misleading conclusions about the deep learning model's reasoning process. On the other hand, in the context of NLP, the quantitative evaluation of saliency interpretations largely remains an open problem (Belinkov and Glass, 2019).
In this paper, we address this problem by building a comprehensive quantitative benchmark to evaluate saliency methods. Our benchmark focuses on a fundamental category of NLP models: neural language models. Following the concepts proposed by Jacovi and Goldberg (2020), our benchmark evaluates the credibility of saliency interpretations from two aspects: plausibility and faithfulness. In short, plausibility measures how much these interpretations align with basic human intuitions about
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5034-5052 June 6-11, 2021. ©2021 Association for Computational Linguistics
the model decision mechanism, while faithfulness measures how consistent the interpretations are regarding perturbations that are supposed to preserve the same model decision mechanism on either the input feature  $F$  or the model  $M$ .
With these concepts in mind, our main contribution is materializing these tests' procedure in the context of neural language modeling and building four test sets from existing linguistic annotations to conduct these tests. Our study covering SOTA-level models on three different network architectures reveals that saliency methods' applicability depends heavily on specific choices of saliency methods, model architectures, and model configurations. We suggest that future work deploying these methods to NLP models should carefully validate their interpretations before drawing conclusions.
This paper is organized as follows: Section 2 briefly introduces saliency methods; Section 3 describes the plausibility and faithfulness tests in our evaluation; Section 4 presents the datasets we built for the evaluation; Section 5 presents our experiment setup and results; Section 6 discusses some limitations and implications of the evaluation; Section 7 concludes the paper.
2 Saliency
The notion of saliency discussed in this paper is a category of neural network interpretation methods that interpret a specific prediction  $y$  made by a neural network model  $M$ , by assigning a distribution of importance  $\Psi(F)$  over the input feature set  $F$  of the original neural network model.
The most basic and widely used method is to assign importance by the gradient (Simonyan et al., 2013), which we refer to as vanilla gradient method (V). For each  $x \in F$ ,  $\psi(x) = \frac{\partial p_y}{\partial x}$ , while  $p_y$  is the score of prediction  $y$  generated by  $M$ . We also examine two improved versions of gradient-based saliency: SmoothGrad (SG) (Smilkov et al., 2017) and Integrated Gradients (IG) (Sundararajan et al., 2017). SmoothGrad reduces the noise in vanilla gradient-based scores by constructing several corrupted instances of the original input by adding Gaussian noise, followed by averaging the scores. Integrated Gradients computes feature importance by computing a line integral of the vanilla saliency from a "baseline" point  $F_0$  to the input  $F$  in the feature space. We refer the readers to the cited papers for details of these saliency methods.
There is a slight complication in the meaning of
$F$  when applying these methods in the context of NLP: all the methods above will generate one importance score for each dimension of the word embedding, but most applications of saliency to NLP want a word-level importance score. Hence, we need composition schemes to combine scores over word embedding dimensions into a single score for each word. In the rest of this paper, we assume the "features" in the feature set  $F$  are input words to the language model, and word-level importance scores are composed using the gradient  $\cdot$  input scheme (Denil et al., 2014; Ding et al., 2019).
3 Evaluation Paradigm
In this section, we first introduce the notion of plausibility and faithfulness in the context of neural network interpretations (following Jacovi and Goldberg (2020)), and then, respectively, introduce the test we adopt to evaluate them.
3.1 Plausibility
Concept An interpretation is plausible if it aligns with human intuitions about how a specific neural model makes decisions. For example, intuitively, an image classifier can identify the object in the image because it can capture some features of the main object in the image. Hence, a plausible interpretation would assign high importance to the area occupied by the main object. This idea of comparison with human-annotated ground-truth (often as "bounding-boxes" signaling the main object's area) is used by various early studies in computer vision to evaluate saliency methods' reliability (Jiang et al., 2013, inter alia). However, the critical challenge of such evaluations for neural language models is the lack of such ground-truth annotations.
Test To overcome this challenge, we follow Poerner et al. (2018) to construct ground-truth annotations from existing lexical agreement annotations. Consider, for example, the case of morphological number agreement. Intuitively, when the language model predicts a verb with a singular morphological number, the singular nouns in the prefix should be considered important features, and vice versa. Based on this intuition, we divide the nouns in the prefix into two different sets: the cue set  $\mathcal{C}$ , which shares the same morphological number as the verb in the sentence; and the attractor set  $\mathcal{A}$ , which has
<sup>1</sup>We also experimented with the vector norm Li et al. (2016a) scheme in our preliminary study, and we find it performing much worse. See details in Appendix B.1.
a different morphological number than the verb in the sentence.
Then, according to the prediction  $y$  made by the model  $M$ , the test will be conducted under one of the two following scenarios:
- Expected: when  $y$  is the verb with the correct number, the interpretation passes the test if  $\max_{w \in \mathcal{C}} \psi(w) > \max_{w \in \mathcal{A}} \psi(w)$ .
- Alternative: when  $y$  is the verb with the incorrect number, the interpretation passes the test if  $\max_{w\in \mathcal{C}}\psi (w) < \max_{w\in \mathcal{A}}\psi (w)$
However, this test has a flaw: while the evaluation criteria focus on a specific category of lexical agreement, the prediction of a word could depend on multiple lexical agreements simultaneously. To illustrate this point, consider the verb prediction following the prefix "At the polling station people ..." Suppose the model  $M$  predicts the verb vote. One could argue that people is more important than polling station because it needs the subject to determine the morphological number of the verb. However, the semantic relation between vote and polling station is also important because that is what makes vote more likely than other random verbs, e.g. sing.
To minimize such discrepancy and constrain the scope of agreements used to make predictions, we draw inspiration from the previous work on representation probing and make adjustment to the model  $M$  we are evaluating on (Tenney et al., 2019a,b; Kim et al., 2019; Conneau et al., 2018; Adi et al., 2017; Shi et al., 2016). The idea is to take a language model that is trained to predict words (e.g., vote in the example above) and substitute the original final linear layer with a new linear layer (which we refer to as a probe) fine-tuned to predict a binary lexical agreement tag (e.g., PLURAL) corresponding to the word choice. By making this adjustment, the final layer extracts a subspace in the representation that is relevant to the prediction of particular lexical agreement during the forward computation, and reversely, filters out gradients that are irrelevant to the agreement prediction in the backward pass, creating an interpretation that is only subject to the same agreement constraints as to when the annotation for the test set is done.
Apart from the adjustment made on the model  $M$  above, we also extend Poerner et al. (2018) in the other two aspects: (1) we evaluate on one more lexical agreement: gender agreements between pronouns and referenced entities, and on both natural
and synthetic datasets; (2) instead of evaluating on small models, we evaluate on large SOTA-level models for each architecture. We also show that evaluation results obtained on smaller models cannot be trivially extended to larger models.
3.2 Faithfulness
Concept An interpretation is faithful if the feature importance it assigns is consistent with the internal decision mechanism of a model. However, as Jacovi and Goldberg (2020) pointed out, the notion of "decision mechanism" lacks a standard definition and a practical way to make comparisons. Hence, as a proxy, we follow the working definition of faithfulness as proposed in their work, which states that an interpretation is faithful if the feature importance it assigns remains consistent with changes that should not change the internal model decision mechanism. Among the three relevant factors for saliency methods (prediction  $y$ , model  $M$ , and input feature set  $F$ ), we focus on consistency upon changes in model  $M$  (model consistency) and input feature set  $F$  (input consistency). Note that these two consistencies respectively correspond to assumptions 1 and 2 in the discussion of faithfulness evaluation in Jacovi and Goldberg (2020).
Model Consistency Test To measure model consistency, we propose to measure the consistency between feature importance  $\Psi_{M}(F)$  and  $\Psi_{M^{\prime}}(F)$ , which is respectively generated from the original model  $M$  and a smaller model  $M^{\prime}$  that is trained by distilling knowledge from  $M$ . In this way, although  $M$  and  $M^{\prime}$  have different architectures,  $M^{\prime}$  is trained to mimic the behavior of  $M$  to the extent possible, and thus having similar underlying decision mechanisms.
Input Consistency Test To measure input consistency, we perform substitutions in the input and measure the consistency between feature importance  $\Psi(F)$  and  $\Psi(F')$ , where  $F$  and  $F'$  are input features sets before/after the substitution. For example, the following prefix-prediction pairs should have the same feature importance distribution:
- The nun bought the son a gift because (she...)
- The woman bought the boy a gift because (she...)
We measure consistency by Pearson correlation between pairs of importance score over the input
2 Although evaluating interpretation consistency over similar predictions  $y$  is also possible, it is not of interest as most applications expect different interpretations for different predictions.
feature set  $F$  for both tests. Also, note that although we can theoretically conduct faithfulness tests with any model  $M$  and any dataset, for the simplicity of analysis and data creation, we will use the same model  $M$  (with lexical agreement probes) and the same dataset as plausibility tests.
4 Data3
Following the formulation in Section 3, we constructed four novel datasets for our benchmark, as exemplified in Table 2. Two of the datasets are concerned with number agreement of a verb with its subject. The other two are concerned with gender agreement of a pronoun with its anteceding entity mentions. For each lexical agreement type, we have one synthetic dataset and one natural dataset. Both synthetic datasets ensure there is only one cue and one attractor for each test instance, while for natural datasets, there are often more than one.
For number agreement, our synthetic dataset is constructed from selected sections of Syneval, a targeted language model evaluation dataset from Marvin and Linzen (2018), where the verbs and the subjects could be easily induced with heuristics. We only use the most challenging sections where strongly interceding attractors are involved. Our natural dataset for this task is filtered from Penn Treebank (Marcus et al., 1993, PTB), including training, development, and test. We choose PTB because it offers not only human-annotated POS-tags necessary for benchmark construction but also dependent subjects of verbs for further analysis.
For gender agreement, our synthetic dataset comes from the unambiguous Winobias coreference resolution dataset used in Jumelet et al. (2019), and we only use the 1000-example subset where there is respectively one male and one female antecedent. Because this dataset is intentionally designed such that most humans will find pronouns of either gender equally likely to follow the prefix, no such pronoun gender is considered to be "correct". Hence, without loss of generality, we assign the female pronoun to be the expected case. Our natural dataset for this task is filtered from CoNLL-2012 shared task dataset for coreference resolution (Pradhan et al., 2012, also including training, develop
ment, and test). The prefix of each test example covers a document-level context, which usually spans several hundred words.
Plausibility Test For number agreement, the cue set  $\mathcal{C}$  is the set of all nouns that have the same morphological number as the verb. In contrast, the attractor set  $\mathcal{A}$  is the set of all nouns with a different morphological number. For gender agreement, the cue set  $\mathcal{C}$  is the set of all nouns with the same gender as the pronoun, while the attractor set  $\mathcal{A}$  is the set of all nouns with a different gender.
Model Consistency Test No special treatment to data is needed for this test. We conduct model consistency tests on all datasets we built.
Input Consistency Test We recognize that generating interpretation-preserving input perturbations for natural datasets is quite tricky. Hence, unlike the model consistency test, we focus on the two synthetic datasets for faithfulness tests because they are generated from templates. As can be seen from the examples, when the nouns in the cue/attractor set are substituted while maintaining the lexical agreement, the underlying model decision mechanism should be left unchanged; hence they can be viewed as interpretation-preserving perturbations. We identified 24 and 254 such interpretation-preserving templates from our Syneval and Winobias dataset and generated perturbations pairs by combining the first example of each template with other examples generated from the same template.
Interpretation Methods For SmoothGrad (SG), we set sample size  $N = 30$  and sample variance  $\sigma^2$  to be 0.15 times the L2-norm of word embedding matrix; for Integrated Gradients (IG), we use step size  $N = 100$ . These choices are made empirically and verified on a small held-out development set.
Interpreted Model Our benchmark covers three different neural language model architectures, namely LSTM (Hochreiter and Schmidhuber, 1997), QRNN (Bradbury et al., 2017) and Transformer (Vaswani et al., 2017; Baevski and Auli, 2019; Dai et al., 2019). All language models are trained on WikiText-103 dataset (Merit et al., 2017). For the first two architectures, we use the implementation as in awd-lstm-lm toolkit (Merit et al., 2018). For Transformer, we use the imple
3More details on data filtering are in Appendix A.
4Note that this assumption will not change the interpretations we generate or the benchmark test conducted for interpretations, as we always interpret the argmax decision of the model, which is not affected by this assumption. It will only affect the breakdown of the result we report.
PTB	U.S. Trade Representative Carla Hills said the first dispute-settlement panel set up under the U.S.-Canadian " free trade" agreement has ruled that Canada 's restrictions on exports of Pacific salmon and herring (PLURAL...)
Syneval	the consultant that loves the parents (SINGULAR...)
CoNLL	Israeli Prime Minister Ehud Barak says he is freezing tens of millions of dollars in tax payments to the Palestinian Authority . Mr. Barak says he is withholding the money until the Palestinians abide by cease - fire agreements . Earlier Thursday Mr. Barak ruled out an early resumption of peace talks , even with the United States acting as intermediary . Eve Conette reports from Jerusalem . Defending what (MASCULINE...)
Winobias	The bride examined the son for injuries because (FEMININE...)
Table 2: Examples prefixes from the four evaluation datasets, followed by the probing tag prediction under the expected scenario. The cue and attractor sets are marked with solid Green and yellow, respectively.
Number Agreement	Gender Agreement
all	PTB exp.	alt.	all	Syneval exp.	alt.	all	CoNLL exp.	alt.	all	Winobias exp.	alt.
Random	-	0.546	0.454	-	0.500	0.500	-	0.519	0.481	-	0.500	0.500
Nearest	-	0.502	0.498	-	0.140	0.860	-	0.038	0.962	-	0.500	0.500
LSTM	(0.858)	(0.142)	(0.596)	(0.404)	(0.730)	(0.270)	(0.584)	(0.416)
V	0.452	0.484	0.259	0.304	0.371	0.206	0.288	0.266	0.348	0.403	0.440	0.351
SG	0.780	0.805	0.629	0.950	0.951	0.949	0.799	0.767	0.880	0.984	0.981	0.988
IG	0.816	0.856	0.571	0.888	0.941	0.811	0.585	0.561	0.652	0.881	0.853	0.921
QRNN	(0.818)	(0.182)	(0.558)	(0.442)	(0.712)	(0.288)	(0.715)	(0.285)
V	0.463	0.501	0.289	0.511	0.536	0.480	0.669	0.638	0.546	0.242	0.269	0.175
SG	0.575	0.599	0.468	0.707	0.692	0.726	0.503	0.436	0.669	0.790	0.801	0.761
IG	0.697	0.728	0.555	0.797	0.764	0.838	0.737	0.700	0.828	0.768	0.730	0.863
Transformer	(0.919)	(0.081)	(0.594)	(0.406)	(0.761)	(0.239)	(0.219)	(0.781)
V	0.551	0.551	0.551	0.723	0.785	0.632	0.674	0.693	0.614	0.781	0.799	0.766
SG	0.842	0.851	0.737	0.895	0.879	0.920	0.956	0.951	0.971	0.994	1.00	0.992
IG	0.734	0.741	0.652	0.849	0.786	0.940	0.829	0.843	0.786	0.806	0.865	0.775
Table 3: Plausibility benchmark result. Each number is the fraction of cases the interpretation passes the benchmark test, while the numbers in brackets for each architecture are the fraction of times these scenarios occur for predictions generated by the corresponding model. Results from the best interpretation method for each architecture are boldfaced. The exp. and alt. columns are breakdown of evaluation results into expected scenarios and alternative scenarios as defined in Section 3. V, SG, IG stands for the vanilla saliency, SmoothGrad, and Integrated Gradients, respectively.
mentation in fairseq tookit (Ott et al., 2019).
For all the task-specific "probes", the fine-tuning is performed on examples extracted from WikiText-2 training data. A tuning example consists of an input prefix and a gold tag for the lexical agreement in both cases. For number agreement, we first run Stanford POS Tagger (Toutanova et al., 2003) on the data, and an example is extracted for each present tense verb and each instance of was or were. For gender agreement, an example is extracted for each gendered pronoun. During finetuning, we fix all the other parameters except the final linear layer. The final layer is tuned to minimize cross-entropy, with Adam optimizer (Kingma and Ba, 2015) and initial learning rate of  $1e - 3$  with ReduceLROnPlateau scheduler.
We follow the setup for DistillBERT (Sanh et al., 2019) for the distillation process involved during the model consistency test, which reduces the depth of models but not the width. For our LSTM (3 layers) and QRNN model (4 layers), the  $M^{\prime}$  we distill is one layer shallower than the original model  $M$ . For our transformer model (16 layers), we distill a 4-layer  $M^{\prime}$  largely due to memory constraints.
Plausibility According to our plausibility evaluation result, summarized in Table 3, both SG and IG consistently perform better than the vanilla saliency method regardless of different benchmark datasets and interpreted models. However, the comparison between SG and IG interpretations varies depend
ing on the model architecture and test sets.
Across different architectures, Transformer language model achieves the best plausibility except on the Syneval dataset. LSTM closely follows Transformer for most benchmarks, while the plausibility of the interpretation from QRNN is much worse. Another trend worth noting is that the gap between Transformer and the other two architectures is much larger on the CoNLL benchmark, which is the only test that involves interpreting document-level contexts. However, these architectures' prediction accuracy is similar, meaning that there is no significant modeling power difference for gender agreements in this dataset. We hence conjecture that the recurrent structure of LSTM and QRNN might diminish gradient signals with increasing time steps, which causes the deterioration of interpretation quality for long-distance agreements – a problem that Transformer is exempt from, thanks to the self-attention structure.