SlowGuess/ACL_2020_OCR huggingface dataset:
Yiyun Zhao
yiyunzhao@arizona.edu
Steven Bethard
bethard@arizona.edu
Large pretrained language models like BERT, after fine-tuning to a downstream task, have achieved high performance on a variety of NLP problems. Yet explaining their decisions is difficult despite recent work probing their internal representations. We propose a procedure and analysis methods that take a hypothesis of how a transformer-based model might encode a linguistic phenomenon, and test the validity of that hypothesis based on a comparison between knowledge-related downstream tasks with downstream control tasks, and measurement of cross-dataset consistency. We apply this methodology to test BERT and RoBERTa on a hypothesis that some attention heads will consistently attend from a word in negation scope to the negation cue. We find that after fine-tuning BERT and RoBERTa on a negation scope task, the average attention head improves its sensitivity to negation and its attention consistency across negation datasets compared to the pre-trained models. However, only the base models (not the large models) improve compared to a control task, indicating there is evidence for a shallow encoding of negation only in the base models.
As large-scale pre-trained language models such as BERT and ELMo have achieved high performance in a variety of natural language processing tasks (Peters et al., 2018a; Radford et al., 2018; Devlin et al., 2019), a growing body of research is devoted to understanding what linguistic properties these language models have acquired. Recent work uses probes, which are supervised models trained to predict linguistic properties including morphology (Belinkov et al., 2017), syntax (Hewitt and Manning, 2019) and semantics (Peters et al., 2018b), etc. (See Belinkov and Glass (2019) for a complete survey.) A good probing performance is considered
as evidence that the language models have learned the linguistic knowledge.
What is not yet well understood is how this encoded linguistic knowledge changes when a pretrained language model is fine-tuned for a downstream task. Peters et al. (2019) applies a supervised probe both before and after fine-tuning BERT, and suggests that fine-tuning makes the internal representation task-sensitive. But with supervised probes it can be difficult to disentangle what was learned by the probe from what was present in the internal representation (Hewitt and Liang, 2019).
Recent studies have thus turned to unsupervised probes that require no additional training of the model and instead look directly at the attention mechanism, i.e., how much to care about other words when computing the next version of the current word. Clark et al. (2019) inspected pretrained transformers and found several syntactic properties encoded in an intuitive way, where the maximum attention from a dependent is on its syntactic head. But only the pretrained models were considered, not what happened to these intuitive encodings after fine-tuning to a downstream task.
We argue that if some interpretable encoding of linguistic knowledge is a good explanation of a model, rather than showing it in the pretrained model, it is more important to show it will be enhanced by fine-tuning on a task where that linguistic knowledge is necessary. If the encoding is not enhanced by such fine-tuning, then the model must be using some other mechanism to encode that linguistic knowledge. We therefore propose the following methodology for testing whether a hypothesized encoding of a linguistic phenomenon is a good explanation for a transformer's predictions.
1. Hypothesize an attention representation of the knowledge of interest and design an unsupervised probe, such that each attention head can
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4729-4747 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
make its own prediction.
2. Identify a downstream task related to the knowledge of interest, and design a control task that is learnable and has a similar input and output space but is not related to the knowledge of interest.
3. Fine-tune on both the downstream and control tasks, and measure the unsupervised probe performance of each attention head before and after fine-tuning.
Applying this methodology and a variety of analyses that it enables, and focusing on the phenomenon of linguistic negation scope in a intuitive encoding (the maximal attention from a word in negation scope will be on the negation cue), we find that:
1. Before fine-tuning, several attention heads are sensitive to negation scope. The best heads are better than a fixed-offset baseline, with the best BERT-base head achieving an  $F_{1}$  of 53.8 in a fully unsupervised setting.
2. There is consistency in which heads are negation-sensitive across different datasets.
3. After fine-tuning on a negation scope task, the average sensitivity of attention heads improved over the pretrained model for all four models (BERT-base, BERT-large, RoBERTa-base, RoBERTa-large) but only the two base models improved more than the control task.
4. The rich do not get richer: attention heads that had the top  $F_{1}$ s in the pretrained model do not have the top-ranked improvements after fine-tuning on negation scope.
5. The behavior of individual attention heads becomes more consistent across datasets after fine-tuning on the negation task, compared to the pretrained model and the control task, except for RoBERTa-large.
Items 1 and 2 suggest that in the pretrained models negation scope may be encoded via attention to negation cues. Items 3 to 5 indicate that during fine-tuning, this encoding continues to play a role in BERT-base and RoBERTa-base, but RoBERTa-large and BERT-large may rely on other mechanisms to represent negation scope. The analysis code is available at https://github.com/yiyunzhao/negation-scope-probing
Though our findings are specific to the linguistic phenomenon of negation scope and the specific attention encoding we hypothesized, our proposed methodology and analyses are general, and can easily be applied to other linguistic phenomena or
other encoding hypotheses to discover the role they play in modern pre-trained neural network models.
2.1 BERT and attention heads
We performed our analysis on the attention mechanism of uncased BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), large Transformer models (Vaswani et al., 2017). In the following text, we primarily focus on BERT-base and refer the reader to the appendix for detailed results on the other models. BERT-base contains 12 layers and each layer contains 12 attention heads. Each attention head takes a sequence of input vectors  $h = [h_1,..,h_n]$  that correspond to the n tokens. An attention head transforms each  $h_i$  into query  $(q_i)$ , key  $(k_i)$  and value  $(v_i)$  vectors and computes an output vector  $(o_i)$  via a weighted sum of value vectors based on attention weights  $(a_i)$ :
$$
a _ {i j} = \frac {\exp \left(q _ {i} ^ {T} k _ {j}\right)}{\sum_ {l = 1} ^ {n} \exp \left(q _ {i} ^ {T} k _ {l}\right)} \tag {1}
$$
$$
o _ {i} = \sum_ {j = 1} ^ {n} a _ {i j} v _ {j} \tag {2}
$$
Attention weights can be viewed as the amount of contribution from other tokens to the new representation of the current token.
2.2 Negation scope
Negation is a grammatical structure that reverses the truth value of a proposition. The tokens that express the presence of negation are the negation cue and the tokens that are affected by the negation cue belong to the negation scope. For example, in the following sentence, not is the negation cue and the underlined tokens are the negation scope.
Holmes was sitting with his back to me, and I had given him  $\{no\}$  sign of my occupation.
Knowledge about negation and its scope is important for tasks such as sentiment anlaysis and logical inference. And as a linguistic phenomenon that bridges between syntax and semantics, it is a good candidate for exploring BERT's attention, as related phenomena have already been found in BERT (Tenney et al., 2019; Clark et al., 2019).
3 Methodology and Analyses
In this section, we explain our proposed methodology and analyses, and illustrate their application to the linguistic phenomenon of negation scope.
Step 1: hypothesize an interpretable representation of the phenomenon of interest. Transformer models could represent linguistic knowledge in many ways: attention, contextualized embeddings, etc. To apply our methodology, one must first hypothesize a specific encoding of the phenomenon of interest. For negation scope, we hypothesize that for some subset of attention heads, words in negation scope will attend primarily to the negation cue, while words out of negation scope will attend primarily to other words (see Section 4.1). Under this hypothesis, each attention head is an unsupervised negation scope classifier.
Step 2: Identify a downstream task that requires the phenomenon of interest. To infer that a transformer model is explainable in terms of the hypothesized encoding, we must see evidence that the encoding is strengthened when fine-tuning on a task that requires the phenomenon of interest. If the encoding is visible in the pre-trained model but disappears during fine-tuning, then the model is handling the phenomenon through some other mechanism. For negation scope, our downstream tasks are supervised negation scope prediction problems (see Section 5.1).
Step 3: Design a control task where the phenomenon of interest is irrelevant. The control task should have input and output spaces that match those of the downstream task but should be learnable without any knowledge of the phenomenon. For negation scope, we arbitrarily assign word types to binary labels (see Section 5.1).
Step 4: Analyze differences between models fine-tuned on the downstream and control tasks. If the hypothesized encoding explains the model predictions, changes observed when fine-tuning on the downstream task must be greater than changes observed when fine-tuning on the control task. For negation scope, we analyze changes in performance of individual attention heads as unsupervised negation classifiers.
Figure 1: Example text with true negation scope on top and layer 8 head 4's maximally-attended word for each input on the bottom. Dashed lines are precision errors and dotted lines are recall errors.
4 Does BERT pay 'attention' to negation scope before fine-tuning?
We start by hypothesizing a way that negation scope could be encoded in transformer models. This hypothesis must not rely on any negation-specific training data, as we want to be able to measure evidence of the encoding equally well both before and after fine-tuning. Our hypothesized encoding treats each attention head as an unsupervised negation scope classifier.
4.1 Attention as a negation classifier
Our goal is to see if any individual attention head is good at detecting negation scope. Because attention heads by definition compare two tokens to each other, we formulate negation scope detection as a pair-wise task. We treat each attention head as an unsupervised classifier that considers each token in the sentence, and if the maximum attention from that token is to the negation cue, we classify the token as within the negation scope. Formally, the prediction of an attention head for token  $i$  is:
$$
a t t e n d n e g (i) = \left\{ \begin{array}{l l} 1 & \text {i f} j _ {n e g} = \underset {j = 1} {\operatorname {a r g m a x}} ^ {n} a _ {i j} \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {3}
$$
where  $j_{\text{neg}}$  is the index of the negation cue, and  $a_{ij}$  is attention as defined in Equation (1).
The quality of each attention head as such a negation classifier can be evaluated based on how often it agrees with the true negation scope, as shown in Figure 1. We use the standard measures of precision, recall, and  $F_{1}$ :
$$
\begin{array}{l} p r e c i s i o n = \frac {\sum_ {i = 1} ^ {n} a t t e n d n e g (i) \wedge n e g s c o p e (i)}{\sum_ {i = 1} ^ {n} a t t e n d n e g (i)} \\ r e c a l l = \frac {\sum_ {i = 1} ^ {n} \text {a t t e n d n e g} (i) \wedge \text {i n s c o p e} (i)}{\sum_ {i = 1} ^ {n} \text {n e g s c o p e} (i)} \\ F _ {1} = \frac {2 \cdot p r e c i s i o n \cdot r e c a l l}{p r e c i s i o n + r e c a l l} \\ \end{array}
$$
where  $\text{attendneg}(i)$  is the unsupervised classifier of Equation (3) and  $\text{negscope}(i)$  is 1 if  $i$  is within the annotated negation scope and 0 otherwise.
4.2 Checking for confounds
If we find an attention head that achieves a high  $F_{1}$  for negation detection, are we sure that BERT has learned negation? Or could the head be doing something simpler to achieve that  $F_{1}$ ? If most negation scopes were just one word after the negation cue, simply attending to the previous word would achieve high performance on the negation task.
To build confidence that attention heads that achieve high  $F_{1}$  in negation detection aren't somehow cheating, we (1) look at several baselines to establish the difficulty of the task, (2) use a regression to see which factors explain the attention, and (3) look for consistency in attention head performance across different datasets. We use the baselines:
all in-scope: Always attend to the negation token, regardless of the input word. This guarantees  $100\%$  recall, but is somewhat unrealistic, since the attention mechanism doesn't know where the negation word is<sup>1</sup>.
fixed offset: Always attend to a fixed position relative to the input word. For example, a fixed offset of  $+1$  would mean to always attend to the next word in the sentence, and therefore, according to Equation (3), to only predict a token is in the negation scope if it is immediately followed by the negation cue. Clark et al. (2019) observed several of BERT's attention heads displaying such behavior. We considered fixed offsets from  $-3$  to  $+3$ .
Predictors of attention If an attention head has truly learned something about negation, its attention should not be easily explainable by something simpler like the proximity in the text. We thus build a simple regression model using the token's negation scope label (in-scope or out-of-scope) and the distance to the negation cue as predictors, and the attention of the token to the negation cue as the dependent variable. If an attention head is truly detecting negation scope, we expect that scope label will be a significant predictor in this model, and token distance will be much less important.
Consistency across domains If an attention head has truly learned something about negation,
Models	P	R	F1
baseline	all in scope	34.0	100.0	50.7
baseline	average fixed offset	66.1	8.6	15.2
baseline	best fixed offset (-1)	83.5	11.6	20.4
attention	average head	49.5	5.2	9.0
attention	best head (8-4)	76.2	41.5	53.8
Table 1: Performance of unsupervised BERT-base attention-based classifiers and baselines on the negation scope detection task in terms of precision (P), recall (R) and  $F_{1}$ . The best fixed offset and attention head according to their  $F_{1}$  score are reported.
we would expect it to perform reasonably well regardless of changes in text genre or style of negation annotation. Several studies show that generalization ability to a different dataset is not always guaranteed despite a good test performance on the same dataset (Weber et al., 2018; McCoy et al., 2019). We thus consider two different corpora annotated for negation: ConanDoyle-neg (Morante and Daelemans, 2012) and SFU Review (Konstantinova et al., 2012) $^{2}$ . These datasets differ in genre (Sherlock Holmes stories vs. movie, book, and consumer product reviews) and in annotation schema (e.g., they have different rules for what sentences are considered to contain negation, and how to deal with coordination structure).
To see whether the same attention heads are performing well at negation scope detection across the two corpora, we measure kendall rank correlation:
$$
\tau = \frac {2}{n (n - 1)} \sum_ {i <   j} s g n \left(x _ {i} - x _ {j}\right) s g n \left(y _ {i} - y _ {j}\right)
$$
where  $x_{i}$  is the performance of attention head  $i$  on the Conan Dolye dataset and  $y_{i}$  is the performance of head  $i$  on the SFU-review dataset.
Table 1 shows the performance of BERT-base's attention heads and the baselines. Table A1 in the Appendix shows the results for other models. BERT-base attention heads on average are not good predictors of negation scope (49.5% in precision, 5.2% in recall, 9.0% in  $F_{1}$ ) but the 4th attention head in layer 8 stands out (76.2% in precision, 41.5% in recall, 53.8% in  $F_{1}$ ). This performance is unlike either the best fixed offset baseline (-1) or the
<sup>1</sup>Note that our classifier in Equation (3) does know where the negation word is, since it is given  $j_{neg}$  as an input. But a standalone transformer model is not given such information.
2We exclude cases in these datasets where the negation cue is part of a word (e.g., im in impossible) because such subword segmentation does not always align to BERT's tokenization.
Figure 2: The heatmap of unsupervised negation-scope classification  $F_{1}$  for BERT-base's 12 layers x 12 heads across two different datasets. The consistency (measure by Kendall rank correlation) between the two datasets for precision, recall and  $F_{1}$  are 0.440, 0.418 and 0.415 respectively. See fig. A1 for precision and recall.
all-in-scope baseline, exceeding both of these in  $F_{1}$ , and with very different precision/recall trade-offs. When we fit a regression model to predict layer 8 head 4's attention based on token distance and the true negation scope label, we found that both distance  $(\beta = 0.043, p < 2 \times 10^{-16})$  and label  $(\beta = 0.310, p < 2 \times 10^{-16})$  were significant predictors for the attention, but the true negation scope label had a much larger coefficient. Anova tests comparing the full model with a model leaving out distance or label found that true negation scope explains more variance (207.7) than distance (1.5). This suggests that a large part of what the best attention head is doing can be best explained as detecting negation.
Figure 2 shows that there is consistency in the  $F_{1}$  of BERT-base's attention heads across the two negation scope datasets, e.g., BERT-base's layer 8 head 4 has the best  $F_{1}$  in both. Kendall correlation tests confirm that the similarities across attention heads of BERT-base are significant: 0.440 tau coefficient  $(p = 5.24 \times 10^{-15})$  in precision, 0.418 tau coefficient  $(p = 1.20 \times 10^{-13})$  in recall and 0.415 tau coefficient  $(p = 1.56 \times 10^{-13})$  in  $F_{1}$ . Figures A1 to A4 in the Appendix show plots for precision and recall, and that similar results hold for the other models. Seeing that attention heads that are predictive of negation in one dataset continue to be predictive in another differently annotated dataset from a different text genre suggests that these most successful heads are indeed learning some form of linguistic negation during the BERT pre-training.
5 What happens to negation-sensitive attention heads when you fine-tune?
We have seen that without any explicit training on a negation task, some attention heads are sensi
Figure 3: Negation scope detection as a word-piece-by-word-piece binary classification task.
tive to negation scope in an intuitive way (in-scope words attend primarily to the negation cue). What happens to the attention when we fine-tune (i.e., continue training the pre-trained model) on a downstream task that requires an understanding of negation scope? Will this attention-based encoding of negation scope be strengthened? Or will the model choose to represent negation-scope knowledge in some other way during fine-tuning? What about for a downstream task that is unrelated to negation? We answer these questions and others in the following sections by fine-tuning models on downstream tasks, and measuring how this changes the negation-sensitivity of different attention heads.
5.1 Downstream Tasks
Downstream negation task We construct a downstream negation scope detection task from the ConanDoyle-neg dataset. As shown in Figure 3, we formulate the problem as a word-piece-by-word-piece binary classification problem, where a word-piece should be labeled 1 if it is in a negation scope and 0 otherwise. To provide the location of the negation cue as an input to the classifier, we add two tokens to the input, surrounding the cue with “{}” and “}”. As is standard for BERT token classification models, a fully-connected layer with sigmoid activation connects BERT's contextual embedding for each token with the binary outputs that
4733
must be predicted. This model can then be trained with BERT's standard back-propagation procedure.
Downstream control task Inspired by the control tasks of Hewitt and Liang (2019), we construct a downstream control task on the ConanDoyle-neg dataset that has the same input space and output space as the downstream negation task, but is constructed to be irrelevant to negation and most other linguistic phenomena. We arbitrarily assign each unique token in the training vocabulary to be always in-scope or always out-of-scope, with a distribution close to the empirical in-scope and out-of-scope distribution. To succeed in this control task, the model must memorize the category (in-scope or out-of-scope) for each token type. Since the assignment is arbitrary, there is no way for the model to generalize to unseen tokens, and thus when we evaluate performance on this task, we consider performance only on the tokens seen during training.
5.2 Fine-tuning classifiers
We split the data into 662 negation frames for training and 200 negation frames for testing. We use the same data split for both the downstream negation scope task and the downstream control task. For each task, we take pre-trained BERT base as our starting point. We fine-tune this model for 50 epochs with a learning rate of  $4 \times 10^{-5}$  using the transformers library (Wolf et al., 2019), and pick the best epoch based upon its performance on the testing data. For the negation scope task, performance is measured in  $F_{1}$ . For the control task, performance is measured in accuracy on the testing data tokens that have been seen in the training data. We repeat this process 10 times, generating 10 different fine-tuned BERT models for each task, to allow us to quantify variance due to the inherent randomness in neural network training $^{3}$ .
Table 2 and Table A2 in the Appendix show that after fine-tuning all models achieve very high performance in both downstream tasks. BERT-base achieves on average  $92.8\%$ $F_{1}$  for the negation scope task and on average  $95.9\%$  accuracy for the control task. The BERT-base model trained on the control task has learned essentially nothing about negation scope relationship, achieving an average
$35.4\% F_{1}$ . These results show that both tasks are learnable from their data, and that the control task is irrelevant to negation scope.
How does fine-tuning change attention? Finetuning changes many parameters to make a model better at a downstream task. Will the change be reflected in our hypothesized encoding, i.e., will in-scope words increase their attention to negation cues? And what will the patterns of such a change be? Will sensitivity to negation be spread throughout the attention heads of the model? Will just the attention heads that were already sensitive to negation improve? Or maybe no individual attention heads will get better at negation; the model will only becomes sensitive to negation in aggregate?