SlowGuess/ACL_2020_OCR huggingface dataset:
McCoy et al. (2019) hypothesize that the main artifact NLI models might learn is lexical overlap. In fact, for all of the examples in HANS, every word in the hypothesis would appear in the corresponding premise (100% lexical overlap rate). Half of the examples would have an entailment relationship while the other half have an non-entailment relationship. McCoy et al. (2019) compare four models with strong performance in MNLI, and all of them predict far more entailments than non-entailments. Because of this imbalance in prediction, they conclude that the models are perhaps exploiting artifacts in data when making decisions.
We see one potential problem out of the above method: it can only be applied to a certain group of examples and imply a general model behavior by examining the prediction imbalance. However, model behavior should depend on the actual example it sees each time. The extent to which the model exploits the artifact in each individual example remains unclear.
To analyze the effect of artifacts on individual examples, we propose a method using influence functions. We hypothesize that if an artifact in
5558
forms the model's predictions for a test instance, the most influential training examples for this test example should contain occurrences of said artifact. For instance, if our model exploits 'lexical overlap' when predicting the relation between a premise and a hypothesis, we should expect the most influential training examples found by the influence function to have a highly overlapping premise and hypothesis.
In Figure 4a, we plot each training example's influence score and lexical overlap rate between its premise and hypothesis for a typical example in the HANS dataset. In linen with our expectation, the most influential (both positively and negatively) training examples tend to have a higher lexical overlap rate. Note that we also expect this trend for the most negatively influential examples, because they influence the model's prediction as much as the positively influential examples do, only in a different direction.
To quantify this bi-polarizing effect, we find it natural to fit a quadratic regression to the influence-artifact distribution. We would expect a high positive quadratic coefficient if the artifact feature appears more in the most influential examples. For an irrelevant feature, we would expect this coefficient to be zero. With this new quantitative measure, we are ready to explore the below problems unanswered by the original diagnostic dataset.
For test examples predicted as non-entailment, did the model fail to recognize that they have a lexical overlap feature? Was the artifact not exploited in these cases? Figure 4a and Figure 4b show two examples in HANS, one predicted as entailment and the other predicted as non-entailment. We observe that the example predicted as non-entailment does not have a significantly different influence-artifact pattern from the entailment example. In fact, the average quadratic coefficients for all examples predicted as entailment and non-entailment are  $+3.28 \times 10^{-3}$  and  $+3.30 \times 10^{-3}$  respectively. Therefore, for predicted non-entailment examples, we still see that the most influential training examples tend to have a high rate of lexical overlap, indicating that the model still recognizes the artifact in these cases.
The model relies on training examples with high lexical overlap when predicting in the artificial HANS dataset. Would it still exploit the same artifact for natural examples? Apart
from finding the most influential training examples for each HANS example, we also apply influence functions on 50 natural MNLI examples, not controlled to exhibit any specific artifacts. A typical example is shown in Figure 4c. The average quadratic coefficient over all 50 natural examples is  $+0.65 \times 10^{-3}$ , which is considerably smaller than the above cases in HANS dataset. The model therefore does not rely on as much lexical overlap in natural examples as in the diagnostic dataset.
We have been analyzing scenarios focusing on one data artifact. What if we have a second artifact during prediction possibly indicating a contradicting decision? How will the model recognize the two artifacts in such a scenario? We know that lexical overlap could be a data artifact exploited by NLI models for making an entailment prediction in HANS. On the other hand, as briefly pointed out by McCoy et al. (2019), other artifacts like negation might be indicative of non-entailment. We are interested in how two contradicting artifacts might compete when they both appear in an example. We take all examples in HANS labeled as entailment and manually negate the hypothesis so that the relationship becomes non-entailment. For example, a hypothesis "the lawyers saw the professor" would become "the lawyers did not see the professor".
Figure 5a and Figure 5b show the influence-artifact distributions on both lexical overlap and negation for an original HANS example. Figure 5c and Figure 5d show the distributions for the same HANS example with negated hypothesis. The average quadratic coefficients on all examples are shown in Table 6. We observe that in the original HANS example, negation is actually a negative artifact: the training examples with negation tend to be the least influential ones. In the negated HANS example, we see the effect of negations becomes positive, while the effect of lexical overlap is drastically weakened. This confirms that the model recognizes the new set of artifacts, and the two are competing with each other.
Importantly, observing an artifact in the most influential training examples is a necessary but not sufficient condition to conclude that it was truly exploited by the model. However, it can serve as a first step towards identifying artifacts in black-box neural models and may be complemented by probing a larger set of hypothesized artifacts.
5559
(a) HANS example predicted as entailment.  $(P$  : The athlete by the doctors encouraged the senator.  $H$  : The athlete encouraged the senator.) Quadratic coefficient:  $+3.74\times 10^{-3}$
(b) HANS example predicted as non-entailment.  $(P$  : Since the author introduced the actors, the senators called the tourists.  $H$  : The senators called the tourists.) Quadratic coef:  $+3.59\times 10^{-3}$
(c) A typical MNLI example.  $(P$ : And uh as a matter of fact he's a draft dodger.  $H$ : They dodged the draft, I'll have you know.) Quadratic coefficient:  $+0.74 \times 10^{-3}$ .
(a) Lexical overlap in original HANS example. Quadratic coefficient:  $+3.13 \times 10^{-3}$ .
Figure 4: Influence-artifact distribution for different test examples.
(b) Negation in original HANS example. Quadratic coefficient:  $-0.92 \times 10^{-3}$ .
(c) Lexical overlap in negated HANS example. Quadratic coefficient:  $+0.76 \times 10^{-3}$ .
(d) Negation in negated HANS example. Quadratic coefficient:  $+0.55 \times 10^{-3}$ .
Figure 5: Influence-artifact distribution for an original and negated HANS example. (P: The lawyers saw the professor behind the bankers. H: The lawyers saw / did not see the professor.)
Lexical overlap coef	Negation coef
Original	+3.05 × 10-3	-1.13 × 10-3
Negated	+0.53 × 10-3	+0.27 × 10-3
Table 6: Average quadratic coefficients of the influence-artifact distribution for all original HANS examples and all negated HANS examples.
Interpreting NLP model predictions by constructing importance scores over the input tokens is a widely adopted approach (Belinkov and Glass, 2019). Since the appearance and rise of attention-based models, many work naturally inspect attention scores and interpret with them. However, we are aware of the recent discussion over whether attention is a kind of faithful explanation (Jain and Wallace, 2019; Wegreffe and Pinter, 2019). Using vanilla attention as interpretation could be more problematic in now ubiquitous deep transformer-based models, such as we use here.
Gradient-based saliency maps are locally 'faithful' by construction. Other than the vanilla gradients (Simonyan et al., 2014) and the "gradient
$\times$  input" method (Shrikumar et al., 2017) we use in this work, there are some variants that aim to make gradient-based attributions robust to potential noise in the input (Sundararajan et al., 2017; Smilkov et al., 2017). We also note that Feng et al. (2018) find that gradient-based methods sometimes yield counter-intuitive results when iterative input reductions are performed.
Other token-level interpretations include input perturbation (Li et al., 2016b) which measure a token's importance by the effect of removing it, and LIME (Ribeiro et al., 2016) which can explain any model's decision by fitting a sparse linear model to the local region of the input example.
The main focus of this work is the applicability of influence functions (Koh and Liang, 2017) as an interpretation method in NLP tasks, and to highlight the possibility of using this to surface annotation artifacts. Other methods that can trace the model's decision back into the training examples include deep weighted averaging classifiers (Card et al., 2019), which make decisions based on the labels of training examples that are most similar to the test input by some distance metrics. Croce et al. (2019) use kernel-based deep architectures
that project test inputs to a space determined by a group of sampled training examples and make explanations through the most activated training instances. While these methods can similarly identify the 'influential' training examples, they require special designs or modifications to the model and could sacrifice the model's performance and generalizability.
Other general methods for model interpretability include adversarial-attack approaches that identify that part of input texts can lead to drastically different model decisions when minimally edited (Ebrahimi et al., 2018; Ribeiro et al., 2018), probing approaches that test internal representations of models for certain tasks and properties (Liu et al., 2019b; Hewitt and Liang, 2019), and generative approaches that make the model jointly extract or generate natural language explanations to support predictions (Lei et al., 2016; Camburu et al., 2018; Liu et al., 2019a; Rajani et al., 2019).
Specific to the NLI task, Gururangan et al. (2018) recognize and define some possible artifacts within NLI annotations. McCoy et al. (2019) create a diagnostic dataset that we use in this work and suggest that the model could be exploiting some artifacts in training data based on its poor performance on the diagnostic set. Beyond NLI, the negative influence of artifacts in data was explored in other text classification tasks (Pryzant et al., 2018; Kumar et al., 2019; Landeiro et al., 2019), focusing on approaches to adversarial learning to demote the artifacts.
We compared two complementary interpretation methods—gradient-based saliency maps and influence functions—in two text classification tasks: sentiment analysis and NLI. We first validated the reliability of influence functions when used with deep transformer-based models. We found that in a lexicon-driven sentiment analysis task, saliency maps and influence functions are largely consistent with each other. They are not consistent, however, on the task of NLI. We posit that influence functions may be a more suitable approach to interpreting models for such relatively complex natural language 'understanding' tasks (while simpler attribution methods like gradients may be sufficient for tasks like sentiment analysis).
We introduced a new potential use of influence functions: revealing and quantifying the effect of
data artifacts on model predictions, which have been shown to be very common in NLI. Future work might explore how rankings induced over training instances by influence functions can be systematically analyzed in a stand-alone manner (rather than in comparison with interpretations from other methods), and how these might be used to improve model performance. Finally, we are interested in exploring how these types of explanations are actually interpreted by users, and whether providing them actually establishes trust in predictive systems.
Acknowledgments. We thank the anonymous ACL reviewers and members of TsvetShop at CMU for helpful discussions of this work. This material is based upon work supported by NSF grants IIS1812327 and SES1926043, and by Amazon MLRA award. Wallace's contributions were supported by the Army Research Office (W911NF1810328). We also thank Amazon for providing GPU credits.
Naman Agarwal, Brian Bullins, and Elad Hazan. 2017. Second-order stochastic optimization for machine learning in linear time. Journal of Machine Learning Research (JMLR), 18:116:1-116:40.
Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natural language inference with natural language explanations. In Proc. NeurIPS.
Dallas Card, Michael Zhang, and Noah A. Smith. 2019. Deep weighted averaging classifiers. In  $FAT^{*}$ .
Rich Caruana, Hooshang Kangaroo, John David N. Dionisio, Usha S. Sinha, and David B. Johnson. 1999. Case-based explanation of non-case-based learning methods. Proc. AMIA Symposium, pages 212-5.
Danilo Croce, Daniele Rossini, and Roberto Basili. 2019. Auditing deep learning processes through kernel-based explanatory models. In Proc. EMNLP.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT.
Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608.
5561
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial examples for text classification. In Proc. ACL.
Shi Feng, Eric Wallace, Alvin Grissom, Mohit Iyyer, Pedro Rodriguez, and Jordan L. Boyd-Graber. 2018. Pathologies of neural models make interpretation difficult. In Proc. EMNLP.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proc. NAACL-HLT.
John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proc. EMNLP, pages 2733-2743.
Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? ArXiv, abs/2004.03685.
Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In Proc. NAACL-HLT.
Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, and Byron C. Wallace. 2020. Learning to faithfully rationalize by construction. In Proc. ACL.
Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In Proc. ICML.
Sachin Kumar, Shuly Wintner, Noah A. Smith, and Yulia Tsvetkov. 2019. Topics to avoid: Demoting latent confounds in text classification. In Proc. EMNLP, pages 4151-4161.
Virgile Landeiro, Tuan Tran, and Aron Culotta. 2019. Discovering and controlling for latent confounds in text classification using adversarial domain adaptation. In Proc. SIAM International Conference on Data Mining, pages 298-305.
Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2016. Rationalizing neural predictions. In Proc. EMNLP.
Jiwei Li, Xinlei Chen, Eduard H. Hovy, and Dan Jurafsky. 2016a. Visualizing and understanding neural models in NLP. In Proc. HLT-NAACL.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016b. Understanding neural networks through representation erasure. *ArXiv*, abs/1612.08220.
Zachary Chase Lipton. 2018. The mythos of model interpretability. Commun. ACM, 61:36-43.
Hui Liu, Qingyu Yin, and William Yang Wang. 2019a. Towards explainable NLP: A generative explanation framework for text classification. In Proc. ACL.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019b. Linguistic knowledge and transferability of contextual representations. In Proc. NAACL-HLT.
R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proc. ACL.
Reid Pryzant, Kelly Shen, Dan Jurafsky, and Stefan Wagner. 2018. Deconfounded lexicon induction for interpretable social science. In *NAACL-HLT*.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02361.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should i trust you?": Explaining the predictions of any classifier. In Proc. HLT-NAACL (System Demonstrations).
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proc. ACL.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. In Proc. ICML.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Conference on Learning Representations.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. 2017. Smooth-Grad: removing noise by adding noise. ArXiv, abs/1706.03825.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. EMNLP.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proc. ICML.
Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner, and Sameer Singh. 2019. AllenNLP interpret: A framework for explaining predictions of NLP models. In Proc. EMNLP (System Demonstrations), pages 7-12.
Sarah Wiegrefe and Yuval Pinter. 2019. Attention is not not explanation. In Proc. EMNLP, pages 11-20.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proc. NAACL-HLT.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
5562
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proc. ICML.
The main model we used for experiments is a BERT-Base model (Devlin et al., 2019), adapted from Wolf et al. (2019). We "froze" the embedding layer and the first 8 transformer layers and only fine-tuned the last 4 transformer layers and the final projection layer. We used the default BERT optimizer with default hyperparameters: a learning rate of  $5\mathrm{e} - 5$ , a total of 3 epochs, a max sequence length of 128, and a training batch size of 32.
For gradient-based saliency maps, we used a "vanilla" version implemented by Wallace et al. (2019). For influence functions, we adapted code from Koh and Liang (2017) to PyTorch and used the same stochastic estimation trick, LiSSA (Agarwal et al., 2017). Since our model is not convex, we used a "damping" term (as mentioned in §3) of  $3\mathrm{e} - 3$ . This value was picked so that the recursive approximation to the inverse Hessian-vector product can be finished (converged) in a reasonable time. More specifically, we chose the recursion depth to be 2500 (with a total of 10k training examples), the number of recursions to be 1, and a scaling factor to be 1e4. In each step estimating the Hessian-vector product, we took a batch of 8 training examples for stability. We empirically checked that the inverse Hessian-vector product converges after the recursive estimation for all test examples on which we performed the analysis.
5563