SlowGuess/ACL_2020_OCR huggingface dataset:
Xiaochuang Han
xiaochuh@cs.cmu.edu
Byron C. Wallace
Northeastern University b.wallace@northeastern.edu
Carnegie Mellon University ytsvetko@cs.cmu.edu
Modern deep learning models for NLP are notoriously opaque. This has motivated the development of methods for interpreting such models, e.g., via gradient-based saliency maps or the visualization of attention weights. Such approaches aim to provide explanations for a particular model prediction by highlighting important words in the corresponding input text. While this might be useful for tasks where decisions are explicitly influenced by individual tokens in the input, we suspect that such highlighting is not always suitable for tasks where model decisions should be driven by more complex reasoning. In this work, we investigate the use of influence functions for NLP, providing an alternative approach to interpreting neural text classifiers. Influence functions explain the decisions of a model by identifying influential training examples. Despite the promise of this approach, influence functions have not yet been extensively evaluated in the context of NLP, a gap addressed by this work. We conduct a comparison between influence functions and common word-saliency methods on representative tasks. As suspected, we find that influence functions are particularly useful for natural language inference, a task in which 'saliency maps' may not provide clear interpretation. Furthermore, we develop a new quantitative measure based on influence functions that can reveal artifacts in training data.<sup>1</sup>
Deep learning models have become increasingly complex, and unfortunately their inscrutability has grown in tandem with their predictive power (Doshi-Velez and Kim, 2017). This has motivated efforts to design example-specific approaches to interpreting black box NLP model predictions, i.e.,
indicating specific input tokens as being particularly influential for a given prediction. This in turn facilitates the construction of saliency maps over texts, in which words are highlighted with intensity proportional to continuous 'importance' scores. Prominent examples of the latter include gradient-based attribution (Simonyan et al., 2014; Sundararajan et al., 2017; Smilkov et al., 2017), LIME (Ribeiro et al., 2016), and attention-based (Xu et al., 2015) heatmaps.
While widely used and potentially useful for some lexicon-driven tasks (e.g., sentiment analysis), we argue that by virtue of being constrained to highlighting individual input tokens, saliency maps will necessarily fail to explain predictions in more complex semantic tasks involving reasoning, such as natural language inference (NLI), where fine-grained interactions between multiple words or spans are key (Camburu et al., 2018). Moreover, saliency maps are inherently limited as a model debugging tool; they may tell us which inputs the model found to be important, but not why.
To address these shortcomings, we investigate the use of what Lipton (2018) referred to as explanation by example. Instead of constructing importance scores over the input texts on which the model makes predictions, such methods rank training examples by their influence on the model's prediction for the test input (Caruana et al., 1999; Koh and Liang, 2017; Card et al., 2019). Specifically, we are interested in the use of influence functions (Koh and Liang, 2017), which are in a sense inherently 'faithful' in that they reveal the training examples most responsible for particular predictions. These do not require any modifications to the model structure.
This paper presents a series of experiments intended to evaluate the potential utility of influence functions for better understanding modern neural NLP models. In this context, our contributions in
$^{1}$ Code is available at https://github.com/xhan77/influence-function-analysis.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5553-5563 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: A sentiment analysis example interpreted by gradient-based saliency maps (left) and influence functions (right). Note that this example is classified incorrectly by the model. Positive saliency tokens and highly influential examples may suggest why the model makes the wrong decision; tokens and examples with negative saliency or influence scores may decrease the model's confidence in making that decision.
clude answering the following research questions.
RQ1 We empirically assess whether the approximation to the influence functions (Koh and Liang, 2017) can be reliably used to interpret decisions of deep transformer-based models such as BERT (Devlin et al., 2019).
RQ2 We investigate the degree to which results from the influence function are consistent with insights gleaned from gradient-based saliency scores for representative NLP tasks.
RQ3 We explore the application of influence functions as a mechanism to reveal artifacts (or confounds) in training data that might be exploited by models.
To the best of our knowledge, this is the first work in NLP to compare interpretation methods that construct saliency maps over inputs with methods that explain predictions via influential training examples. We also propose a new quantitative measurement for the effect of hypothesized artifacts (Gururangan et al., 2018; McCoy et al., 2019) on the model's prediction using influence functions.
2 Explaining Black-box Model Predictions
Machine learning models in NLP depend on two factors when making predictions: the input text and the model parameters. Prior attempts to interpret opaque NLP models have typically focused on the input text. Our work investigates the complementary approach of interpreting predictions by analyzing the influence of examples in training data. Saliency maps aim to provide interpretability by highlighting parts of the input text, whereas
influence functions seek clues in the model parameters, eventually locating interpretations within the training examples that influenced these estimates. In this section we explain the two interpretation methods in detail.2
2.1 Gradient-based saliency maps
As a standard, illustrative 'explanation-by-input-features' method, we focus on gradient-based saliency maps, in which the gradient of the loss  $\mathcal{L}$  is computed with respect to each token  $t$  in the input text, and the magnitude of the gradient serves as a feature importance score (Simonyan et al., 2014; Li et al., 2016a). Gradients have the advantage of being locally 'faithful' by construction: they tell us how much the loss would change, were we to perturb a token by a small amount. Gradient-based attributions are also agnostic with respect to the model, as long as it is differentiable with respect to inputs. Finally, calculating gradients is computationally efficient, especially compared to methods that require post-hoc input perturbation and function fitting, like LIME (Ribeiro et al., 2016).
We are interested in why the model made a particular prediction. We therefore define a loss  $\mathcal{L}_{\hat{y}}$  with respect to the prediction  $\hat{y}_i$  that the model actually made, rather than the ground truth  $y_i$ . For each token  $t \in x_i$ , we define a saliency score  $-\nabla_{e(t)}\mathcal{L}_{\hat{y}} \cdot e(t)$ , where  $e(t)$  is the embedding of  $t$ . This is also referred as the "gradient  $\times$  input" method in Shrikumar et al. (2017). The "gradient"  $\nabla_{e(t)}\mathcal{L}_{\hat{y}}$  captures the sensitivity of the loss to the change in the input embedding, and the "input"
2 Here we focus on interpretability approaches which are faithful (Wiegreffe and Pinter, 2019; Jacovi and Goldberg, 2020; Jain et al., 2020) by construction; other approaches are discussed in §6.
$e(t)$  leverages the sign and magnitude of the input. The final saliency score of each token  $t$  would be L1-normalized across all tokens in  $x_{i}$ .
Unlike Simonyan et al. (2014) and Li et al. (2016a), when scoring features for importance, we do not take the absolute value of the saliency score, as this encodes whether a token is positively influencing the prediction (i.e., providing support the prediction) or negatively influencing the prediction (highlighting counter-evidence). We show an example in the left part of Figure 1.
2.2 Influence functions
In contrast to explanations in the form of token-level heatmaps, the influence function provides a method for tracing model predictions back to training examples. It first approximates how upweighting a particular training example  $(x_{i},y_{i})$  in the training set  $\{(x_1,y_1),\ldots ,(x_n,y_n)\}$  by  $\epsilon_{i}$  would change the learned model parameters  $\hat{\theta}$ :
$$
\frac {d \hat {\theta}}{d \epsilon_ {i}} = - (\frac {1}{n} \sum_ {j = 1} ^ {n} \nabla_ {\hat {\theta}} ^ {2} \mathcal {L} (x _ {j}, y _ {j}, \hat {\theta})) ^ {- 1} \nabla_ {\theta} \mathcal {L} (x _ {i}, y _ {i}, \hat {\theta})
$$
We can then use the chain rule to measure how this change in the model parameters would in turn affect the loss of the test input (as in saliency maps, w.r.t. the model prediction):
$$
\frac {d \mathcal {L} _ {\hat {y}}}{d \epsilon_ {i}} = \nabla_ {\theta} \mathcal {L} _ {\hat {y}} \cdot \frac {d \hat {\theta}}{d \epsilon_ {i}}
$$
More details (including proofs) can be found in Koh and Liang (2017).
We define the influence score for each training example  $(x_{i},y_{i})$  as  $-\frac{d\mathcal{L}_{\hat{y}}}{de_i}$ , and then  $z$ -normalize it across all examples in the training set. Note that since  $\mathcal{L}_{\hat{y}}$  is defined with respect to a particular test input, influence scores of training examples are also defined for individual test instances.
Intuitively, a positive influence score for a training example means: were we to remove this example from the train set, we would expect a drop in the model's confidence when making the prediction on the test input. A negative influence score means that removing the training example would increase the model's confidence in this prediction. We show an example in the right part of Figure 1.
We are interested in analyzing and comparing the two interpretation approaches (gradient-based attributions and influence functions) on relatively shallow, lexicon-driven tasks and on more complex,
reasoning-driven tasks. We focus on sentiment analysis and natural language inference (NLI) as illustrative examples of these properties, respectively. Both models are implemented on top of BERT encoders (Devlin et al., 2019). In particular we use BERT-Base, with the first 8 of the 12 layers frozen, only fine-tuning the last 4 transformer layers and the final projection layer.<sup>3</sup>
It is worth noting that influence functions are guaranteed to be accurate only when the model is strictly convex (i.e., its Hessian is positive definite and thus invertible) and is trained to convergence. However, deep neural models like BERT are not convex, and one often performs early stopping during training. We refer to Koh and Liang (2017) for details on how influence functions can nonetheless provide good approximations. To summarize briefly: for the non-convexity issue, we add an appropriate 'damping' term to the model's Hessian so that it is positive definite and invertible. Concerning non-convergence: the approximated influence may still be interpretable as the true influence of each training example plus a constant offset that does not depend on the individual examples. Aside from this theory, we also perform a sanity check in §4 to show that influence functions can be applied to BERT in practice on the two tasks that we consider.
Sentiment analysis We use a binarized version of the Stanford Sentiment Treebank (SST-2) (Socher et al., 2013). Our BERT-based model is trained on 10k examples; this achieves  $89.6\%$  accuracy on the SST-2 dev set of 872 examples. We randomly sample 50 examples from the SST-2 dev set as the set for which we extract explanations for model predictions.
Natural language inference Our deeper 'semantic' task is NLI, a classification problem that concerns the relationship between a premise sentence and a hypothesis sentence. NLI is a ternary task with three types of premise-hypothesis relations: entailment, neutral, and contradiction. We train our BERT model on the Multi-Genre NLI (MNLI) dataset (Williams et al., 2018), which contains 393k
<sup>3</sup>We used smaller BERT models because influence functions are notoriously expensive to compute. We also resort to the same stochastic estimation method, LiSSA (Agarwal et al., 2017), as in Koh and Liang (2017), and we deliberately reduce the size of our training sets. Even with these efforts, computing the influence scores of 10k training examples w.r.t. one typical test input would take approximately 10 minutes on one NVIDIA GeForce RTX 2080 Ti GPU.
premise and hypothesis pairs of three relations from 10 different genres. We collapse the neutral and contradiction labels to a single non-entailment label and only use  $10\mathrm{k}$  randomly sampled examples for training. On the MNLI dev set of 9815 examples, the model achieves an accuracy of  $84.6\%$ .
To evaluate model interpretations in a controlled manner, we adopt a diagnostic dataset, HANS (McCoy et al., 2019). This contains a balanced number of examples where hypotheses may or may not entail premises with certain artifacts that they call 'heuristics' (e.g., lexical overlap, subsequence). The original HANS dataset contains 30k examples that span 30 different heuristic sub-categories. We test our model and interpretation methods on 30 examples covering all the sub-categories.
4 Evaluating Influence Functions for NLP
RQ1: Is influence function approximation reliable when used for deep architectures in NLP?
Influence functions are designed to be an approximation to leave-one-out training for each training example. But the theory only proves that this works on strictly convex models. While Koh and Liang (2017) show that influence functions can be a good approximation even when the convexity assumption is not satisfied (in their case, a CNN for image classification), it is still not obvious that the influence function would work for BERT.
Therefore, we conduct a sanity check: for each instance in our test set, we by turns remove the most positively influential  $10\%$ , the most negatively influential  $10\%$ , the least influential (where influence scores are near zero)  $10\%$ , and a random  $10\%$  of training examples. We are interested in how these removals in retraining would affect the confidence of model predictions. Table 1 and Table 2 show the result of experiments on sentiment analysis and NLI, repeated with 5 random initialization seeds.
Removal type	Avg. Δ in prediction confidence
Positively influential	-6.00% (±1.12%)
Negative influential	+0.17% (±0.50%)
Least influential	-1.30% (±0.54%)
Random	-1.67% (±0.54%)
The results are largely in accordance with our
Table 1: Sanity check for influence function result on BERT in sentiment analysis.
Removal type	Avg. Δ in prediction confidence
Positively influential	-11.62% (±2.09%)
Negative influential	+2.01% (±1.44%)
Least influential	+1.01% (±0.97%)
Random	+0.13% (±1.07%)
Table 2: Sanity check for influence function result on BERT in NLI.
expectation in both tasks: removing the most positively influential training examples would cause the model to have a significantly lower prediction confidence for each test example; removing the most negatively influential examples makes the model slightly more confident during prediction; and removing the least influential examples leads to an effect that is closest to removing a same amount of random examples (although we note that deleting the least influential features still yields a larger  $\Delta$  than choosing features at random to remove in NLI). We therefore conclude that the influence function behaves reasonably and reliably for BERT in both sentiment analysis and NLI tasks.
RQ2. Are gradient-based saliency maps and 'influential' examples compatible? Comparing saliency maps and outputs from application of the influence function is not straightforward. Saliency maps communicate the importance of individual tokens in test instances, while influence functions measure the importance of training examples. Still, it is reasonable to ask if they seem to tell similar stories regarding specific predictions. We propose two experiments that aim to estimate the consistency between these two interpretation methods.
The first experiment addresses whether a token with high saliency also appears more frequently in the training examples that have relatively high influence. For each example in the test set, we find the tokens with the most positive, most negative, and median saliency scores. We then find all the influential training examples w.r.t. the test inputs that contain one of these tokens. These training examples could have any labels in the label set. We further only consider examples whose label is the same as the test prediction, because the token saliency scores, whether positive or negative, are directly w.r.t. the test prediction, and the effect of a token in an oppositely labeled training example is therefore indirect.
We compute the average influence score of these
5556
training examples and report the results on top  $10\%$ ,  $20\%$ ,  $50\%$ , and all training examples for both sentiment analysis and NLI tasks in Figure 2 and Figure 3 respectively. The reason we have results at different granularity is that from empirical results in Koh and Liang (2017), we see that the influence function approximations tend to be less accurate when going from the most influential to the less influential examples down in the spectrum.
Figure 2: Average influence score of top sentiment analysis training examples that contain a token in test example with most positive, most negative, or median saliency. Error bars depict standard errors.
Figure 3: Average influence score of top NLI training examples that contain a token in test example with most positive, most negative, or median saliency. Standard error is shown in error bars.
In the task of sentiment analysis, we observe that training examples containing the most positively salient token in the test example generally have a higher influence to the test prediction. However, we do not see this trend (in fact, it is the opposite) in the task of natural language inference.
The second experiment answers the question of whether the influence result would change significantly when a salient token is removed from the
Saliency of the removed token	@0.1%	@0.2%	@0.5%	@1%
Most negative	75.6%	77.4%	80.0%	82.4%
Median	84.2%	86.7%	88.9%	89.1%
Most positive	65.2%	68.8%	71.4%	72.0%
Table 3: Average overlap rate of top influential sentiment analysis training examples before and after removal of a token with the most positive, most negative, or median saliency.
Saliency of the removed token	@0.1%	@0.2%	@0.5%	@1%
Most negative	33.0%	33.5%	37.5%	40.9%
Median	79.3%	78.0%	80.5%	84.0%
Most positive	46.0%	48.3%	49.9%	54.9%
Table 4: Average overlap rate of top influential NLI training examples before and after removal of a token with the most positive, negative, or median saliency.
input. Again, for each of the test examples, we identify the tokens with the most positive, most negative, and median saliency score. We by turns remove them from the input and compute the influence distribution over all training examples. We compare these new influence results with the one on the original input, and report an overlap rate of the top  $0.1\%$ ,  $0.2\%$ ,  $0.5\%$ , and  $1\%$  influential training examples before and after the token removal. Table 3 and Table 4 show results for sentiment analysis and NLI, respectively.
When removing a token with the most positive saliency score, we expect the model to be less confident about its current prediction; it could possibly make a different prediction. Therefore, we expect to see a most different influence distribution from the original influence result compared to removing the token with median or the most negative saliency score. This is exactly what we observe in Table 3 for sentiment analysis. However, for NLI, we again see a rather opposite trend: removing the most negatively salient token (might make the prediction more confident but should not change the prediction itself) leads to the most different influence distribution.
We conclude from the above two experiments that gradient-based saliency maps and influential examples are compatible and consistent with each other in sentiment analysis. However, for NLI the two approaches do not agree with each other and
5557
could potentially tell very different stories. To this end, we take a closer look at the task of NLI.
5 Interpreting NLI Predictions with Influence Functions
Are saliency-based explanations useful for NLI? Gradient-based saliency maps are faithful by construction, but this does not mean that they will highlight input tokens that humans find plausible or useful. We hypothesize that highlighting individual input tokens as important is likely most useful for 'shallow' classification tasks like sentiment analysis, and less so for more complex reasoning tasks such as NLI.
To contrast the types of explanations these methods offer in this context, we show explanations for a prediction made for a typical example in HANS in the form of a saliency map and influential examples in Table 5. The tokens that get the most positive and most negative saliency scores are marked in cyan and red, respectively. The training examples with the most positive and most negative influence scores are presented as supporting and opposing instances, respectively.
Test input
P: The manager was encouraged by the secretary. H: The secretary encouraged the manager.	{entail}
Most supporting training examples
P: Because you’re having fun. H: Because you’re having fun.
P: I don’t know if I was in heaven or hell, said Lillian Carter, the president’s mother, after a visit. H: The president’s mother visited.
P: Inverse price caps. H: Inward caps on price.
P: Do it now, think ‘bout it later. H: Don’t think about it now, just do it.	[entail]
[entail]
[entail]
Most opposing training examples
P: H’m, yes, that might be, said John. H: Yes, that might be the case, said John.
P: This coalition of public and private entities undertakes initiatives aimed at raising public awareness about personal finance and retirement planning. H: Personal finance and retirement planning are initiatives aimed at raising public awareness.	[non-entail]
[non-entail]
Table 5: A correctly predicted example in HANS interpreted by saliency map and influence function.
The relationship classification decision in NLI
is often made through an interaction between multiple words or spans. Therefore, an importance measure on each individual token might not give us much useful insight into model prediction. Though influence functions also do not explicitly tell us which latent interactions between words or spans informed the model prediction, we can test whether the model is relying on some hypothesized artifacts in a post-hoc way by looking at patterns in the influential training examples.
In Table 5, though the most influential examples (both supporting and opposing) are ostensibly far from the test input, they all exhibit lexical overlap between the premise and hypothesis. Some of the influential training examples (e.g., the 4th supporting example and 2nd opposing example) capture a reverse ordering of spans in the premise and hypothesis. We note that our test input also has a high lexical overlap and similar reverse ordering. This exposes a problem: the model might be relying on the wrong artifacts like word overlap during the decision process rather than learning the relationship between the active and passive voice in our case. This problem was surfaced by finding influential examples.
5.1 Quantitatively measuring artifacts