SlowGuess/ACL_2020_OCR huggingface dataset:
Consistency	Precision	Recall	F1
mean τ ± sd	sig	mean τ ± sd	sig	mean τ ± sd	sig
BERT-base Pretrain	0.440	0.418	0.415
BERT-base Control	0.438 ± 0.020	10/10	0.406 ± 0.034	10/10	0.409 ± 0.026	10/10
BERT-base Negation	0.469 ± 0.025	10/10	0.519 ± 0.020	10/10	0.516 ± 0.020	10/10
BERT-large Pretrain	0.295	0.487	0.482
BERT-large Control	0.0005 ± 0.057	3/10	0.007 ± 0.039	1/10	0.006 ± 0.039	1/10
BERT-large Negation	0.474 ± 0.038	10/10	0.523 ± 0.082	10/10	0.530 ± 0.066	10/10
RoBERTa-base Pretrain	0.438	0.472	0.471
RoBERTa-base Control	0.456 ± 0.022	10/10	0.502 ± 0.023	10/10	0.487 ± 0.021	10/10
RoBERTa-base Negation	0.521 ± 0.024	10/10	0.538 ± 0.033	10/10	0.531 ± 0.033	10/10
RoBERTa-large Pretrain	0.377	0.504	0.493
RoBERTa-large Control	0.389 ± 0.031	10/10	0.579 ± 0.029	10/10	0.561 ± 0.026	10/10
RoBERTa-large Negation	0.516 ± 0.037	10/10	0.593 ± 0.056	10/10	0.584 ± 0.054	10/10
Table A5: Kendall rank correlation  $(\tau)$  between an attention head's performance on the ConanDoyle-neg dataset and its performance in the SFU-review dataset. For the fine-tuning settings, we report the average  $\tau$  across 10 runs with 1 standard deviation, and the number of runs where there was a significant correlation. Only in two models (in bold) was the correlation for the negation-trained model significantly higher than the correlation for both the pretrained model and the control model. Finding: In BERT-base and RoBERTa-base, attention performance finetuned on a negation task is more consistent scope across different domains and annotation schemes.
(a) Precision
(b) Recall
(c)  $F_{1}$
Figure A1: The heatmap of unsupervised negation-scope classification performance for BERT-base's 12 layers x 12 heads across two different datasets. The consistency (measure by Kendall rank correlation) between the two datasets for precision, recall and  $F_{1}$  are 0.440, 0.418 and 0.415 respectively.
Figure A2: The heatmap of unsupervised negation-scope classification performance for BERT-large's 24 layers x 16 heads across two different datasets. The consistency (measure by Kendall rank correlation) between the two datasets for precision, recall and  $F_{1}$  are 0.295, 0.487 and 0.482 respectively.
Figure A3: The heatmap of unsupervised negation-scope classification performance for RoBERTa-base's 12 layers x 12 heads across two different datasets. The consistency (measure by Kendall rank correlation) between the two datasets for precision, recall and  $F_{1}$  are 0.438, 0.472 and 0.471 respectively.
Figure A4: The heatmap of unsupervised negation-scope classification performance for RoBERTa-large's 24 layers x 16 heads across two different datasets. The consistency (measure by Kendall rank correlation) between the two datasets for precision, recall and  $F_{1}$  are 0.377, 0.504 and 0.493 respectively.
Figure A5: Change in  $F_{1}$  for each attention head (averaged across 10 runs) before and after fine-tuning.
4746
Figure A6: Change in  $F_{1}$  for each attention head (averaged across 10 runs) before and after fine-tuning.