SlowGuess/NAACL_2021_OCR huggingface dataset:
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016b. Understanding neural networks through representation erasure. CoRR, abs/1612.08220.
Zachary C. Lipton. 2018. The mythos of model interpretability. Commun. ACM, 61(10):36-43.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. Comput. Linguistics, 19(2):313-330.
Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202, Brussels, Belgium. Association for Computational Linguistics.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing LSTM language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan, and Kedar Dhamdhere. 2018. Did the model understand the question? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1896-1906, Melbourne, Australia. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairseq: A fast, extensible toolkit for sequence modeling.* In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)*, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Nina Poerner, Hinrich Schütze, and Benjamin Roth. 2018. Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 340-350, Melbourne, Australia. Association for Computational Linguistics.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning - Proceedings of the Shared Task:
Modeling Multilingual Unrestricted Coreference in OntoNotes, EMNLP-CoNLL 2012, July 13, 2012, Jeju Island, Korea, pages 1-40.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108.
Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural MT learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526-1534, Austin, Texas. Association for Computational Linguistics.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312.6034.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. 2017. Smooth-grad: removing noise by adding noise. CoRR, abs/1706.03825.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 3319-3328.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4593-4601.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanian Das, and Ellie Pavlick. 2019b. What do you learn from context? probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 252-259.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008.
Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Suggala, David I Inouye, and Pradeep K Ravikumar. 2019. On the (in) fidelity and sensitivity of explanations. In Advances in Neural Information Processing Systems, pages 10965-10976.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15-20, New Orleans, Louisiana. Association for Computational Linguistics.
A Data Filtering Details
A.1 Penn Treebank (PTB)
A potential candidate for a test case is extracted every time a word with POS tag VBZ (Verb, 3rd person singular present) or VBP (Verb, non-3rd person singular present), or a copula that is among is, are, was, were, shows up. The candidate will then be filtered subjecting to the following criteria:
1. The prefix has at least one attractor word (a noun that has a different morphological number as the verb that is predicted). This is to ensure that evaluation could be conducted in the alternative scenario.
2. The verb cannot immediately follow its grammatical subject (note: it may still immediately follow a cue word that is not a grammatical subject). This is to ensure that the signal of the subject is not overwhelmingly strong compared to the attractors.
3. Not all attractors occur earlier 10 words than the grammatical subject. Same reason as the previous criteria.
Overall, we obtained 1448 test cases out of 49168 sentences in PTB (including train, dev, and test set). We lose a vast majority of sentences mostly because of the last two criteria.
A.2 Syneval
We use the following sections of the original data (followed by their names in the data dump, Marvin and Linzen, 2018):
- Agreement in a sentential complementet: sent_comp
- Agreement across a prepositional phrase: prep_anim and prep_inanim
- Agreement across a subject relative clause: subj_rel
- Agreement across an object relative clause: obj_rel_across_anim, obj_rel_across_inanim, obj_rel_no_comp_across_anim, obj_rel_no_comp_across_inanim
- Agreement within an object relative clause: obj(rel_within_anim, obj(rel_within_inanim,
obj(rel_no_comp_within Anim, obj(rel_no_comp_within_inanim
We select these sections because they all have strong interfering attractors or have cues that may potentially be mistaken as attractors. We obtained much fewer examples (6280) than the original data (249760) because lots of examples only differ in the verb or the object they use, which become duplicates when we extract prefix before the verb.
The original dataset does not come with cue/attractor annotations, but it can be easily inferred because they are generated by simple heuristics.
Note that most of these sections have only around  $50\%$  prediction accuracy with RNNs in the original paper. Our results on large-scale language models corroborate the findings in the original paper.
A.3 CoNLL
We use the dataset (Pradhan et al., 2012) with gold parses, entities mentions, and mention boundaries. A potential candidate for a test case is extracted every time a pronoun shows up. The male pronouns are he, him, himself, his, while the female pronouns are she, her, herself, hers. We don't include cases of epicene pronouns like it, they, etc. because they often involve tricky cases like entity mentions covering a whole clause. We break prefixes according to the document boundaries as provided in the original dataset unless the prefix is longer than 512 words, in which case we instead break at the nearest sentence boundary.
The annotation for this dataset does not cover the gender of entities. We are aware that the original shared task provides gender annotation, but to this day, the documentation for the data is missing and hence we cannot make use of this annotation. Hence, we instead used several heuristics to infer the gender of an entity mention, in descending order:
- If an entity mention and a pronoun have a coreference relationship, they should share the same gender.
- If an entity mention starts with "Mr." or "Mrs." or "Ms.", we assign the corresponding gender.
- If the entity mention has a length of two tokens, we assume it's a name and use gender inference tools to guess its gender. Note that the gender
<sup>6</sup>https://github.com/lead-ratings/gender-guesser
guesser may also indicate that it's not able to infer the gender, in that case, we do not assign a gender.
- If a mention is co-referenced with another mention that is not a pronoun, they should also have the same gender.
Manual inspection of the resulting data indicates that the scheme above covers the gender of most entity mentions correctly. We hope that our dataset could be further perfected by utilizing higher quality annotation on entity genders.
Since each entity mention could span more than one word, we add all words within the span into their corresponding cue/attractor set. A tricky case is where two entity mention spans are nested or intersected. For the first case, we exclude a smaller span from the larger one to create two unintersected spans as the new span for the cue/attractor set. For the second case, we exclude the intersecting parts from both spans.
Finally, all candidates are filtered subject to the following two criteria:
1. The prefix should include one attractor entity.
2. The entity mention that is closest to the verb should be of different gender (either the opposite or epicene).
We obtained 586 document segments from the 2280 documents in the original data. As pointed out in Zhao et al. (2018), the CoNLL dataset is significantly biased towards male entity mentions. Nevertheless, our filtering scheme generated a relatively balanced test set: among 586 test cases, 258 are male pronouns, while 328 are female pronouns.
A.4 Winobias
We used the same data as the unambiguous coreference resolution dataset in Jumelet et al. (2019), which is in turn generated by a script from Zhao et al. (2018), except that we excluded cases where both nouns in the sentence are of the same gender. Similar to Syneval dataset, the cue and attractors could easily be inferred with heuristics.
We leave some results that we cannot fit into the main paper here.
B.1 Vector Norm (VN) Composition Scheme
In this section, we explain why we chose not to cover the vector norm composition scheme (mentioned in 2) in our main evaluation results.
We would like to argue first that even mathematically, VN is not a good fit for our evaluation paradigm. Vector norm composition scheme will only indicate the importance of a feature, but will not indicate the polarity of the importance because it cannot generate a negative word importance score, which is important for our evaluation. The reason why it is important is that our plausibility evaluation does distinguish between input words that should have positive/negative importance scores by placing them in cue and attractor sets, respectively. For example, in Table 1, the singular proper noun U.S. and Europe are important input words because they could potentially lead the model to make the alternative prediction is instead of the expected prediction are. Hence, they are placed into the attractor set, and when interpreting the next word prediction are, our plausibility test expects that they should have large negative importance scores.
Besides, we did run the plausibility evaluation with vector norm composition scheme under some settings, as shown in Table 8. For the vanilla gradient saliency method, the VN composition scheme performs on-par with the gradient  $\cdot$  input (GI) scheme (which is used for our main results). However, with SmoothGrad, the plausibility result does not significantly change like the case with the gradient  $\cdot$  input (GI) scheme. This corroborates with the results in (Ding et al., 2019), where they also show that SmoothGrad does not improve the interpretation quality with VN composition scheme.
With these theoretical and empirical evidence, we decided to drop vector norm composition scheme for our evaluation.
B.2 Patterns for Changes of Interpretation Quality with Varying Model Configurations
As mentioned in Section 5.3, we would like to know if there are any predictable patterns in how interpretation quality changes with varying model configurations. To answer this question, we build smaller Transformer language models with various depth, number of heads, embedding size, and feedforward layer width settings, while keeping other hyperparameters unchanged.
Number Agreement	Gender Agreement
all	PTB exp.	alt.	all	Syneval exp.	alt.	all	CoNLL exp.	alt.	all	Winobias exp.	alt.
LSTM	(0.858)	(0.142)	(0.596)	(0.404)	(0.730)	(0.270)	(0.584)	(0.416)
V+VN	0.683	0.719	0.463	0.643	0.466	0.903	0.459	0.393	0.639	0.680	0.807	0.502
SG+VN	0.543	0.540	0.561	0.549	0.271	0.959	0.394	0.234	0.829	0.625	0.587	0.678
QRNN	(0.818)	(0.182)	(0.558)	(0.442)	(0.712)	(0.288)	(0.715)	(0.285)
V+VN	0.630	0.673	0.437	0.579	0.456	0.735	0.427	0.309	0.716	0.526	0.650	0.214
SG+VN	0.559	0.567	0.521	0.556	0.352	0.813	0.398	0.230	0.811	0.539	0.538	0.540
Transformer	(0.919)	(0.081)	(0.594)	(0.406)	(0.761)	(0.239)	(0.219)	(0.781)
V+VN	0.604	0.620	0.424	0.671	0.525	0.885	0.507	0.511	0.493	0.481	0.840	0.380
SG+VN	0.592	0.596	0.542	0.654	0.504	0.872	0.529	0.538	0.500	0.437	0.836	0.325
Table 8: Plausibility benchmark result for Vector Norm (VN) composition scheme. Refer to the caption of Table 3 for notations.
Figure 1: Analysis of model configuration vs. plausibility on PTB and CoNLL benchmark. Each model configuration is color-coded, while the parameter size (in millions) is shown with circle size. l, w, e, h stands for model depth, width of feed-forward layers after self-attention, embedding size, and the number of heads.
We show two different groups of comparison here. Figure 1 shows our investigation on the interaction between model configuration and interpretation plausibility on PTB and CoNLL test sets. In general, Integrated Gradients method works better for deeper models, while SG works better for shallower models on the PTB test set, but remains roughly the same performance for all architectures on the CoNLL test set. This indicates the noisiness of the trend we are investigating, as both interpretability methods and evaluation dataset choice can influence the trend. As for the other factors of the model configurations, the trend is even noisier (note how much rankings of different configurations change moving from shallow to deep models) and do not show any clear patterns.
Figure 2, on the other hand, focuses on one specific dataset and investigates the trend on both
the plausibility and input faithfulness with varying model configurations. For plausibility results, we largely see the same trend as on PTB dataset. For faithfulness results, the trend for SG is largely the same as plausibility. For IG, the variance across other factors of configurations tends to be different on shallower models vs. deeper models, but overall still shows higher numbers for deeper models like plausibility.
Overall, these analyses further support our conclusion in the main paper, that interpretation qualities are sensitive to model configuration changes, and we reiterate that evaluations of saliency methods should be conducted on the specific model configurations of interest, and trends of interpretation quality on a specific model configuration should not be over-generalized to other configurations.
(a) Plausibility benchmark with SmoothGrad
(c) Faithfulness benchmark with SmoothGrad
(b) Plausibility benchmark with Integrated Gradients
(d) Faithfulness benchmark with Integrated Gradients
Figure 2: Analysis of model configuration vs. plausibility and faithfulness on Syneval benchmark. Each model configuration is color-coded, while the parameter size (in millions) is shown with circle size. l, w, e, h stands for model depth, width of feed-forward layers after self-attention, embedding size, and the number of heads. Note that the faithfulness numbers plotted here are the ones interpreted with expected scenario predictions.
C Language Model Perplexities
Parameter size and perplexity on WikiText-103 dev set for all language models are shown in Table 9 for reference.
Below are the respective commands to reproduce these results.
Architectures	Layers	Config	Params (M)	dev ppl
LSTM	3	-	162	37.65
2	-	130	41.97
QRNN	4	-	154	32.12
3	-	135	36.54
Transformer
16	4096w_1024e_8h	247	17.97
4	4096w_1024e_8h (Distill Student)	96.1	24.92
4	4096w_1024e_8h	96.1	28.96
8	1024w_512e_2h	39.3	32.63
8	1024w_512e_4h	39.3	32.09
8	1024w_512e_8h	39.3	31.38
8	512w_512e_8h	35.1	33.99
8	2048w_512e_8h	47.7	30.19
8	1024w_256e_8h	17.4	41.56
8	1024w_1024e_8h	96.1	27.01
4	1024w_512e_2h	30.8	37.03
4	1024w_512e_4h	30.8	35.67
4	1024w_512e_8h	30.8	35.82
4	512w_512e_8h	28.7	38.34
4	2048w_512e_8h	35.0	33.70
4	1024w_256e_8h	14.3	48.47
4	1024w_1024e_8h	70.9	30.46
2	1024w_512e_2h	26.6	44.45
2	1024w_512e_4h	26.6	42.23
2	1024w_512e_8h	26.6	41.86
2	512w_512e_8h	25.6	44.97
2	2048w_512e_8h	28.7	38.99
2	1024w_256e_8h	12.7	59.16
2	1024w_1024e_8h	58.3	36.06
Table 9: Parameter size (in millions) and perplexity on WikiText-103 dev set for all language models we trained.
D Additional Interpretation Examples
We show some additional interpretations generated by the state-of-the-art LSTM (Table 10), QRNN (Table 11) and Transformer (Table 12) models on PTB and CoNLL dataset, with their respective best-performing interpretation method.
CoNLL
Table 12: Addition interpretation examples with Transformer.