SlowGuess/ACL_2020_OCR huggingface dataset:
We first look at overall changes. Table 3 shows the average performance change across all 144 heads of BERT-base, and for just the best head (layer 8, head 4). Table A3 shows average performance changes for the other models. When BERT-base is fine-tuned on the control task, the  $F_{1}$  for most heads is similar to what it was before fine-tuning. When BERT is fine-tuned on the negation task, both the average  $F_{1}$  and the  $F_{1}$  of the best attention head increase. The Wilcoxon test shows that both the average  $F_{1}$  ( $p = 7.578 \times 10^{-5}$ ) and the  $F_{1}$  of the best head ( $p = 0.002089$ ) fine-tuned on the negation task are significantly higher than when fine-tuned on the control task. Table A3 shows that all negation-finetuned models improve over the pretrained models, but only BERT-base and RoBERTa-base improve over the controls.
We next look at changes at the level of individual attention heads.
Figure 4 plots the average  $F_{1}$  performance gain for each of BERT-base's 144 attention heads after fine-tuning on either the negation or control task. Figure A5 in the Appendix plots the same for the other models. These plots show that in negation-finetuned models the mid-to-late layers of attention heads improve their sensitivity to negation scope, while in control-finetuned models the changes are less positive and spread more broadly. Figure 4 shows that when BERT-base is fine-tuned on the negation task, the biggest gains in  $F_{1}$  are on attention heads in layers 6 through 10, while no such pattern is visible when BERT-base is fine-tuned on the control task.
Do the rich heads get richer? Are attention heads that are already good predictors of negation
3Random restarts with the exact same hyperparameters can induce a surprising amount of instability in performance (Reimers and Gurevych, 2017; Devlin et al., 2019).
Training Task	Testing Task
Negation	Control
P ± sd	R ± sd	F1± sd	A ± sd
Negation	96.1± 1.3	89.7 ± 1.3	92.8 ± 1.1	-
Control	34.8 ± 0.4	36.1 ± 2.2	35.4 ± 1.2	95.9 ± 3.0
Table 2: Performance of fine-tuned BERT-base models on the supervised negation scope detection and control tasks in terms of precision (P), recall (R) and  $F_{1}$  for negation scope and accuracy (A) for the control task. We report the average performance of 10 runs and 1 standard deviation.
Attention Head	Fine-Tune	P ± sd	R ± sd	F1 ± sd
Average	None	49.5	5.2	9.0
Average	Control	48.6 ± 1.7	5.3 ± 0.2	9.0 ± 0.4
Average	Negation	52.2 ± 2.2	6.6 ± 0.8	11.1 ± 1.2
Best (8-4)	None	76.2	41.5	53.8
Best (8-4)	Control	65.0 ± 8.9	47.5 ± 11.7	53.1 ± 6.7
Best (8-4)	Negation	82.3 ± 4.1	58.6 ± 10.8	67.7 ± 7.9
Table 3: Performance of unsupervised BERT-base attention-based classifiers on the scope detection task in terms of precision (P), recall (R) and  $F_{1}$  after the BERT model has been fine-tuned on different downstream tasks.
scope improve more after fine-tuning? That is, if an attention head has a high negation-scope prediction performance before fine-tuning, will it increase in performance more than other attention heads that had lower performance before fine-tuning? To test this, we measure the Kendall rank correlation between an attention head's performance before fine-tuning on the downstream negation task, and its change in performance after fine-tuning. For the BERT-base model, most coefficients are very small and many of the runs show no significant correlation: the average  $\tau$  coefficient for precision is -0.07 and only 3 out of 10 runs show a significant correlation, the average  $\tau$  coefficient for recall is 0.10 and only 5 out of 10 runs show a significant correlation, and the  $\tau$  coefficient for  $F_{1}$  is 0.08 and only 5 out of 10 runs show a significant correlation. Table A4 in the Appendix shows that in other models the rich on average get poorer: we find weak negative correlations. This suggests fine-tuning, even on a relevant downstream task, does not focus on improving the attention heads that are already good at the problem.
Which layers improve the most? Are attention heads at certain layers more sensitive to fine-tuning than other layers? We measure the average performance gain for attention heads in each layer of BERT-base, and plot how these vary across the 10
runs in Figure 5. Figure A6 in the Appendix plot the same for the other models. After the model is fine-tuned on the negation task, we see that attention heads in mid-to-later layers (e.g., layers 6 through 10 in BERT-base) become more sensitive to negation scope. The models fine-tuned on the control task generally show smaller changes. The exception is BERT-large, whose pattern is very different, perhaps because it is the only model to have perfectly memorized the control task.
Is the change consistent across datasets? We have seen that fine-tuning on a downstream negation task increases the negation sensitivity broadly across the many attention heads. Do these changes truly represent a better understanding of the linguistic phenomenon of negation, or are they simply a form of better fitting the training data? If a more general understanding is being learned, when looking across several different types of negation problems, there should be greater consistency in which attention heads are paying attention to negation than in the pretrained model or control task.
We thus take models after fine-tuning on the ConanDoyle-neg downstream negation scope task, treat each of the attention heads as unsupervised negation-scope classifiers as in Section 4.1, and calculate performance on both the ConanDoyle-neg data (the same type of data as was used for
Figure 4: Change in  $F_{1}$  for each attention head in BERT-base (averaged across 10 runs) before and after fine-tuning.
Figure 5: Average change in  $F_{1}$  for the attention heads in each layer in BERT-base, repeated for 10 runs.
fine-tuning) and the SFU-review data (a different text genre and annotation scheme). We then run Kendall rank correlation tests between the two sets of attention-head performances and report them in Table 4 for BERT-base and Table A5 in the Appendix for the other models. Fine-tuning BERT-base on the downstream negation task indeed yields more similar performance across datasets (0.516  $F_{1}$ ) than for the original model before fine-tuning (0.415  $F_{1}$ ) or the model fine-tuned on the downstream control task (0.409  $F_{1}$ ). A Wilcoxon test shows that the  $\tau$  coefficients fine-tuned on the negation task are significantly higher compared to those fine-tuned on the control task ( $p = 1.083 \times 10^{-5}$ ). RoBERTa-base patterns similarly. For BERT-large the negation-tuned models show a marginal consistency improvement over the pretrain and the attention head consistency in the negation-tuned RoBERTa-large models does not exceed that of the control-tuned ones.
We have presented a methodology for looking for explanations of transformer models, where a hypothesized encoding of knowledge within the transformer is measured before and after fine-tuning and the changes are compared to those seen when fine-tuning on a control task. We considered a specific
linguistic phenomenon, negation scope detection, proposed an intuitive way that attention may encode negation-scope (in-scope words pay attention to the negation cue), and applied our methodology to test whether the hypothesized encoding was indeed an explanation of the behavior of BERT and/or RoBERTa models. We found evidence that BERT-base and RoBERTa-base encode some negation knowledge in the proposed way as both average negation sensitivity and cross-dataset consistency improved over the pretrained model and the control task. Evidence for the large versions of the models was weaker, suggesting that they may be representing negation knowledge in other ways.
Other works have explored the effects of fine-tuning on attention without testing for specific linguistic knowledge. Serrano and Smith (2019), Jain and Wallace (2019) and Wegreffe and Pinter (2019) found many redundancies in the attention of sequence-to-sequence models, suggesting that attention may encode knowledge in many ways. Kovaleva et al. (2019) found that removal of attention heads in transformers does not necessarily damage downstream performance. Our results suggest an explanation for this finding: knowledge sensitivity spreads broadly, so recovering from a small number of missing heads should be easy.
Htut et al. (2019) investigated the role of gram
Fine-Tune	Precision	Recall	F1
mean τ ± sd	sig	mean τ ± sd	sig	mean τ ± sd	sig
Pretrain	0.440	0.418	0.415
Control	0.438 ± 0.020	10/10	0.406 ± 0.034	10/10	0.409 ± 0.026	10/10
Negation	0.469 ± 0.025	10/10	0.519 ± 0.020	10/10	0.516 ± 0.020	10/10
Table 4: Kendall rank correlation  $(\tau)$  between an attention head's performance on the ConanDoyle-neg dataset and its performance in the SFU-review dataset. For the fine-tuning settings, we report the average  $\tau$  across 10 runs with 1 standard deviation, and the number of runs where there was a significant correlation.
matical relations in BERT's changes before and after fine-tuning. They found that long distance grammatical relations such as advcl and csubj improved greatly after finetuning on a semantically related task, but other relations did not. They included no control task and did not report changes for individual attention heads (only changes in the maximum performance) so their work inspires some questions: Do advcl and csubj improve more than expected by chance? For the other relations, does performance not improve because they are irrelevant? Or maybe performance of one of the non-maximal heads improved quite a bit, but not enough to exceed the maximal head? Applying our methodology for comparing against a control task and examining changes in individual heads could address these questions.
Other work has tested for specific linguistic knowledge in pretrained models, but not explored how the encoding of that knowledge changes during fine-tuning. For instance, Clark et al. (2019) identified several syntactic relationships that are encoded in an intuitive way: the dependent's primary attention is on its grammatical head. We argue that testing whether this hypothesized encoding of grammatical relations survives fine-tuning is critical if this is to be an explanation of how transformer models make predictions.
We found no past work that considered the cross-dataset consistency of attention. We believe measuring such consistency is important for differentiating between an attention head that learned to encode a linguistic phenomenon for a single dataset vs. an attention head that learned an encoding of the true linguistic phenomenon. For example, it could have been the case that fine-tuning improves sensitivity to negation in both datasets, but the improvements happen at different heads. We see this for example in BERT-large on the control task, where there is essentially zero consistency in which atten
tion heads are active across the two datasets.
Some limitations of our current work suggest future research directions. First, we have focused on one interpretable way of encoding of negation scope knowledge but one can hypothesize many other ways. For instance, instead of assuming that all in-scope words directly pay attention to negation cue, it is possible that the head of in-token words are organized in a tree of attention that leads to the negation cue. We use a single nonlinguistic control task, but one could imagine exploring attention head changes in the face of a gradient of fine-tuning tasks that are more or less relevant to the linguistic phenomenon of interest. We also focus primarily on the attention mechanism, but it would be useful to explore the value vectors that transformers apply the attention to, since these form the outputs and are thus more directly tied to classification decisions.
In this paper, we propose a basic procedure and analysis methods that take a hypothesis of how a transformer-based model might encode a linguistic phenomenon, and test the validity of that hypothesis based on unsupervised probes, downstream control tasks, and measurement of cross-dataset consistency. We hypothesize an interpretable encoding of negation scope, where in-scope words attend to the negation cue, and find evidence of such an encoding in BERT-base and RoBERTa-base.
Thanks to the anonymous reviewers for their helpful suggestions. This work was supported in part by National Institutes of Health grant R01LM012918 from the National Library of Medicine (NLM). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861-872, Vancouver, Canada. Association for Computational Linguistics.
Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733-2743, Hong Kong, China. Association for Computational Linguistics.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
Phu Mon Htut, Jason Phang, Shikha Bordia, and Samuel R. Bowman. 2019. Do attention heads in bert track syntactic dependencies? ArXiv, abs/1911.12246.
Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543-3556, Minneapolis, Minnesota. Association for Computational Linguistics.
Natalia Konstantinova, Sheila C.M. de Sousa, Noa P. Cruz, Manuel J. Maña, Maite Taboada, and Ruslan Mitkov. 2012. A review corpus annotated for negation, speculation and their scope. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 3190-3195, Istanbul, Turkey. European Language Resources Association (ELRA).
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365-4374, Hong Kong, China. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
R. Thomas McCoy, Junghyun Min, and Tal Linzen. 2019. Berts of a feather do not generalize together: Large variability in generalization across models with similar test set performance. CoRR, abs/1911.02969.
Roser Morante and Walter Daelemans. 2012. Conandoyle-neg: Annotation of negation in co-nan doyle stories. In Proceedings of the Eighth International Conference on Language Resources and Evaluation, Istanbul.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018b. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499-1509, Brussels, Belgium. Association for Computational Linguistics.
Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 7-14, Florence, Italy. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL
https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageeunsupervised/language understanding paper.pdf.
Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338-348, Copenhagen, Denmark. Association for Computational Linguistics.
Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931-2951, Florence, Italy. Association for Computational Linguistics.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Noah Weber, Leena Shekhar, and Niranjan Balasubramanian. 2018. The fine line between linguistic generalization and failure in Seq2Seq-attention models. In Proceedings of the Workshop on Generalization in the Age of Deep Learning, pages 24-27, New Orleans, Louisiana. Association for Computational Linguistics.
Sarah Wiegrefe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11-20, Hong Kong, China. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
The main text of the paper focused on the results for BERT-base. This appendix contains detailed results for all four models: BERT-base, RoBERTa-base, BERT-large, and RoBERTa-large.
Models	P	R	F1
baseline	all in scope	34.0	100.0	50.7
baseline	average fixed offset	66.1	8.6	15.2
baseline	best fixed offset (-1)	83.5	11.6	20.4
BERT-base attention	average head	49.5	5.2	9.0
BERT-base attention	best head (8-4)	76.2	41.5	53.8
BERT-large attention	average head	45.4	3.3	5.9
BERT-large attention	best head (14-4)	74.9	28.3	41.0
RoBERTa-base attention	average head	56.0	6.9	12.1
RoBERTa-base attention	best head (9-12)	92.9	19.1	31.1
RoBERTa-large attention	average head	50.2	5.3	9.4
RoBERTa-large attention	best head (15-15)	66.7	21.3	32.3
Table A1: Performance of unsupervised attention-based classifiers and baselines on the negation scope detection task in terms of precision (P), recall (R) and  $F_{1}$ . The best fixed offset and attention head according to their  $F_{1}$  score are reported. Finding: all models have attention heads that know more about negation than the simple baselines.
Training Task	Testing Task
Negation	Control
P ± sd	R ± sd	F1± sd	A ± sd
BERT-base Negation	96.1± 1.3	89.7 ± 1.3	92.8 ± 1.1	-
BERT-base Control	34.8 ± 0.4	36.1 ± 2.2	35.4 ± 1.2	95.9 ± 3.0
BERT-large Negation	97.3± 0.9	93.0 ± 1.1	95.1 ± 0.6	-
BERT-large Control	39.2 ± 0.9	33.1 ± 1.0	35.9 ± 0.6	100.0 ± 0.0
RoBERTa-base Negation	97.2± 0.9	92.9 ± 1.0	95.9 ± 0.3	-
RoBERTa-base Control	43.4 ± 0.7	45.4 ± 1.2	44.4 ± 0.7	98.3 ± 0.4
RoBERTa-large Negation	97.9± 0.9	93.5 ± 1.2	95.7 ± 0.9	-
RoBERTa-large Control	44.1 ± 0.6	45.2 ± 1.8	44.6 ± 1.0	97.9 ± 2.2
Table A2: Performance of fine-tuned models on the supervised negation scope detection and control tasks in terms of precision (P), recall (R) and  $F_{1}$  for negation scope and accuracy (A) for the control task. We report the average performance of 10 runs and 1 standard deviation. Finding: All models successfully learned both supervised tasks.
Attention Head	Fine-Tune	P ± sd	R ± sd	F1 ± sd
BERT-base Average	None	49.5	5.2	9.0
BERT-base Average	Control	48.6 ± 1.7	5.3 ± 0.2	9.0 ± 0.4
BERT-base Average	Negation	52.2 ± 2.2	6.6 ± 0.8	11.1 ± 1.2
BERT-large Average	None	45.4	3.3	5.9
BERT-large Average	Control	44.8 ± 0.3	4.6 ± 0.1	8.3 ± 0.2
BERT-large Average	Negation	46.0 ± 3.7	4.8 ± 1.5	8.0 ± 2.3
RoBERTa-base Average	None	56.0	6.9	12.1
RoBERTa-base Average	Control	53.7 ± 1.7	7.0 ± 0.3	12.0 ± 0.5
RoBERTa-base Average	Negation	55.5 ± 1.9	7.9 ± 0.9	13.4 ± 1.4
RoBERTa-large Average	None	50.2	5.3	9.4
RoBERTa-large Average	Control	48.2 ± 2.2	7.0 ± 1.0	11.5 ± 1.3
RoBERTa-large Average	Negation	54.2 ± 3.4	8.0 ± 1.8	13.2 ± 2.7
Table A3: Performance of unsupervised attention-based classifiers on the scope detection task in terms of precision (P), recall (R) and  $F_{1}$  after models have been fine-tuned on different downstream tasks. All models fine-tuned on negation-scope significantly outperformed their pretrained counterparts in  $F_{1}$ , but only two (in bold) significantly outperformed the controls. Finding: In BERT-base and RoBERTa-base, attention can be a explanation of negation.
Negation change	Precision	Recall	F1
τ	pos/neg	sig	τ	pos/neg	sig	τ	pos/neg	sig
BERT-base	-0.065	0/3	3/10	0.096	5/0	5/10	0.085	5/0	5/10
BERT-large	-0.098	2/5	7/10	-0.132	0/7	7/10	-0.132	0/8	8/10
RoBERTa-base	-0.134	0/7	7/10	-0.107	0/5	5/10	-0.113	0/6	6/10
RoBERTa-large	-0.155	0/8	8/10	-0.142	0/8	8/10	-0.144	0/8	8/10
Table A4: Kendall rank correlation  $(\tau)$  between the change of an attention head after fine-tuning on the negation task and its performance in the pretrained model. We report the average  $\tau$  across 10 runs, the number of runs where there was a significant correlation, and the direction (positive or negative) of the significant correlations. Finding: The rich do not get richer: attention heads that had the top  $F_{1}s$  in the pretrained model do not have the top-ranked improvements after fine-tuning on negation scope.