SlowGuess/ACL_2020_OCR huggingface dataset:
Md Mosharaf Hossain* Kathleen Hamilton** Alexis Palmer** Eduardo Blanco*
*Department of Computer Science and Engineering, University of North Texas
mdmosharafhossain@my.unt.edu, eduardo.blanco@unt.edu
\*\*Department of Linguistics, University of North Texas
kathleenhamilton@my.unt.edu, alexis.palmer@unt.edu
The focus of a negation is the set of tokens intended to be negated, and a key component for revealing affirmative alternatives to negated utterances. In this paper, we experiment with neural networks to predict the focus of negation. Our main novelty is leveraging a scope detector to introduce the scope of negation as an additional input to the network. Experimental results show that doing so obtains the best results to date. Additionally, we perform a detailed error analysis providing insights into the main error categories, and analyze errors depending on whether the model takes into account scope and context information.
Negation is a complex phenomenon present in all human languages. Horn (2010) put it beautifully when he wrote "negation is what makes us human, imbuing us with the capacity to deny, to contradict, to misrepresent, to lie, and to convey irony." Broadly speaking, negation "relates an expression  $e$  to another expression with a meaning that is in some way opposed to the meaning of  $e$ " (Horn and Wansing, 2017). The key challenge to understanding negation is thus to figure out the meaning that is in some way opposed to  $e$ —a semantic and highly ambiguous undertaking that comes naturally to humans in everyday communication.
Negation is generally understood to carry positive meaning, or in other words, to suggest an affirmative alternative. For example, John didn't leave the house implicates that John stayed inside the house. Hasson and Glucksberg (2006) show that comprehending negation involves considering the representation of affirmative alternatives. While not fully understood, there is evidence that negation involves reduced access to the affirmative mental representation (Djokic et al., 2019). Orenes et al. (2014) provide evidence that humans switch
to the affirmative alternative in binary scenarios (e.g., from not red to green when processing The figure could be red or green. The figure is not red). In such multary scenarios, however, humans keep the negated representation unless the affirmative interpretation is obvious from context (e.g., humans keep not red when processing The figure is red, green, yellow or blue. The figure is not red.).
From a linguistic perspective, negation is understood in terms of scope and focus (Section 2). The scope is the part of the meaning that is negated, and the focus is the part of the scope that is most prominently or explicitly negated (Huddleston and Pullum, 2002). Identifying the focus is a semantic task, and it is critical for revealing implicit affirmative alternatives. Indeed, the focus of negation usually contains only a few tokens, and it is rarely grammatically modified by a negation cue such as never or not. Only the focus of a negation is actually intended to be negated, and the resulting affirmative alternatives range from implicatures to entailments as exemplified below (focus is underlined, and affirmative alternatives are in italics):
- He didn't report the incident to his superiors until confronted with the evidence.
He reported the incident to his superiors, but not until confronted with the evidence.
- The board didn't learn the details about the millions of dollars wasted in duplicate work.
The board learnt about the millions of dollars wasted in duplicate work, but not the details.
In this paper, we experiment with neural networks for predicting the focus of negation. We work with the largest corpus annotating the focus of negation (PB-FOC, 3,544 negations), and obtain the best results to date. The main contributions of this paper are: (a) neural network architecture taking into account the scope of negation and context, (b) experimental results showing that scope information as predicted by an automated scope detector is more
8389
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8389-8401 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
beneficial than context, (c) quantitative analysis profiling which foci are easier and harder to predict, and (d) detailed qualitative analysis providing insights into the errors made by the models. Crucially, the scope detector we leverage to predict focus is trained with CD-SCO, a corpus created independently of PB-FOC (Section 2). Our results suggest that negation scopes may transfer across (a) genres (short stories vs. news) and (b) negation types (all negations vs. only verbal negations, i.e., when the negation cue modifies a verb).
It is generally understood that negation has scope and focus. Scope is "the part of the meaning that is negated" and includes all elements whose individual falsity would make the negated statement strictly true (Huddleston and Pullum, 2002). Consider the following statement (1) John doesn't know exactly how they met. This statement is true if one or more of the following propositions are false: (1a) Somebody knows something, (1b) John is the one who knows, (1c) exactly is the manner of knowing, and (1d) how they met is what is known. Thus, the scope of the negation in statement (1) is (1a-d). The focus of a negation is "the part of the scope that is most prominently or explicitly negated", or in other words, the element of the scope that is intended to be interpreted as false to make the overall negative true (Huddleston and Pullum, 2002). Determining the focus consists in pinpointing which parts of the scope are intended to be interpreted as true and false given the original statement. Without further context, one can conclude that the intended meaning of statement (1) is John knows how they met, but not exactly, or alternatively, that (1a-b, 1d) are intended to be interpreted as true, and (1c) as false. This interpretation results from selecting as focus (1c), i.e., the manner of knowing.
We summarize below corpora annotating scope and focus of negation, emphasizing the ones we work with. The survey by Jiménez-Zafra et al. (2020) provides a more comprehensive analysis including corpora in languages other than English. Corpus Annotating Scope. In the experiments described here, we work with a scope detector trained with CD-SCO (Morante and Daelemans, 2012), which annotates negation cues and negation scopes in two stories by Conan Doyle: The Hound of the Baskervilles and The Adventure of Wisteria Lodge. The corpus contains 5,520 sentences, 1,227
%foci	%verb with	%role is focus
ARG0	4.09	67.44	6.06
ARG1	43.76	90.47	48.36
ARG2	5.53	14.24	38.81
ARG3	0.39	1.49	26.42
ARG4	0.51	0.79	64.29
M-NEG	26.08	99.89	26.11
M-TMP	7.16	16.80	42.62
M-MNR	5.50	7.36	74.71
M-ADV	3.30	13.53	24.38
M-LOC	1.01	3.72	27.27
M-EXT	0.45	0.56	80.00
M-DIR	0.25	1.07	23.68
M-PNC	1.49	2.42	61.63
M-DIS	0.28	7.81	3.61
M-CAU	0.11	2.88	3.92
Table 1: Analysis of PB-FOC: overall percentages of foci per role, percentages of negated verbs having each role, and percentage of each role being the focus.
of which contain a negation. CD-SCO annotates all negations, including verbs (e.g., I fail to see how you could have done more), adverbs (e.g., It was never proved that [...]), determiners (e.g., There is no friend like [...]), pronouns (e.g., [...]) has yielded nothing to a careful search), affixes (e.g., The inexplicable tangle seemed [...]), and others.
Other corpora annotating scope in English include efforts with biomedical texts (Vincze et al., 2008) and working with reviews (Council et al., 2010; Konstantinova et al., 2012).
Corpora Annotating Focus. Although focus of negation is defined as a subset of the scope, there is no corpus annotating both of them in the same texts. We work with PB-FOC, the largest publicly available corpus annotating focus of negation (Blanco and Moldovan, 2011). PB-FOC annotates the focus of the negations marked with M-NEG role in PropBank (Palmer et al., 2005), which in turn annotates semantic roles on top of the Penn TreeBank (Taylor et al., 2003). As a result, PB-FOC annotates the focus of 3,544 verbal negations (i.e., when a negation cue such as never or not syntactically modifies a verb). As per the authors, the annotation process consisted of selecting the semantic role most likely to be the focus. Therefore, focus annotations in PB-FOC are always all the tokens corresponding to a semantic role of the (negated) verb. Finally, M-NEG role is chosen when the focus is the verb. The annotations in PB-FOC were carried out taking into account the previous and next sentences. We provide examples below, and Section 5 provides ad
8390
ditional examples. We indicate the semantic roles in PropBank with square brackets, and the role selected as focus is underlined.
- Even if [that deal]  $\mathrm{ARG}_{1}$  is  $[\mathrm{n}^{\prime}\mathrm{t}]_{\mathrm{M - NEG}}$  [revived]verb, NBC hopes to find another.
- [A decision]  $\mathrm{ARG}_1$  is  $[\mathrm{n}^{\prime}\mathrm{t}]_{\mathrm{M - NEG}}$  [expected]verb [until some time next year]M-TMP.
- But [quite a few money managers]ARG0 are[n't]M-NEG [buying]verb [it]ARG1.
Table 1 presents basic statistics for PB-FOC.  $\mathrm{ARG}_1$  is the most frequent role to be focus (43.76%) followed by M-NEG (26.08%) and a relatively long list of infrequent roles (  $\mathrm{ARG}_0$ ,  $\mathrm{ARG}_2$ , M-TMP, M-MNR: 4.09–7.16%). More interestingly, the last two columns in Table 1 indicate (a) how often a negated verb has each semantic role, and (b) how often a role of a negated verb is the focus—if a negated verb-argument structure does not have a particular role, that role obviously cannot be the focus. These percentages reveal that role presence does not uniquely identify foci, but some semantic roles, although infrequent overall, are likely to be the focus if present (M-EXT: 80.00%, M-MNR: 74.71%,  $\mathrm{ARG}_4$ : 64.29%, M-PNC: 61.63%).
Other corpora annotating the focus in English redefine the annotation guidelines (Anand and Martell, 2012), use dependency trees instead of roles (Sarabi and Blanco, 2016), target non-verbal negations (Sarabi and Blanco, 2017), and work with tutorial dialogues (Banjade and Rus, 2016).
3 Previous Work
In addition to identifying negation cues and resolving the scope and focus of negation, there is work showing that processing negation is important for natural language understanding in general. In particular, sentiment analysis benefits from processing negation (Wiegand et al., 2010). For example, like generally carries positive sentiment, but not when modified by a negation cue (e.g., don't like). Wilson et al. (2005) introduce the idea of contextual polarity, and note that negation may intensify rather than change polarity (e.g., not good vs. not only good but amazing). Jia et al. (2009) present a set of heuristic rules to determine sentiment when negation is present, and Councill et al. (2010) show that information about the scope of negation is beneficial to predict sentiment. Outside sentiment analysis, Bentivogli et al. (2016) point out that neural machine translation struggles translating negation, and point to focus detection as a possible solution.
Neural networks are hard to interpret, but there is evidence that they learn to process negation—to a certain degree—when trained to predict sentiment analysis. Li et al. (2016) visually show that neural networks are capable of meaning composition in the presence of, among others, negation and intensification. Wang et al. (2015) show that an LSTM architecture is capable of determining sentiment of sequences containing negation such as not good and not bad. These previous works train a model for a particular task (i.e., sentiment analysis) and then investigate whether the model learnt anything related to negation that is useful for that task. Unlike them, we target focus of negation detection—and the resulting affirmative alternatives—and work with task-independent negations.
Scope Identification. Compared to focus identification, scope identification has received substantially more attention. The first proposals (Morante and Daelemans, 2009) were trained in the biomedical domain with BioScope (Szarvas et al., 2008). The *SEM-2012 Shared Task (Morante and Blanco, 2012) included scope identification with CD-SCO (Section 2), and the winner proposed an SVM-based ranking of syntactic constituents to identify the scope (Read et al., 2012). More recently, Fancellu et al. (2016) present neural networks for this task, and Packard et al. (2014) present a complementary approach that operates over semantic representations obtained with an off-the-shelf parser. Finally, Fancellu et al. (2017) present an error analysis showing that scope is much easier to identify when delimited by punctuation. In this paper, we use a scope detector trained with CD-SCO to predict the focus of negation. While we only incorporate small modifications to previously proposed architectures, our scope detector outperforms previous work (Section 4).
Focus Identification. Although focus is part of the scope, state-of-the-art approaches to identify the focus of negation ignore information about scope. Possible reasons are that (a) existing corpora annotating scope and focus contain substantially different texts (Section 2), and (b) incorporating scope information is not straightforward with traditional machine learning and manually defined features. The initial proposals obtain modest results and only consider the sentence containing the negation (Blanco and Moldovan, 2011), including scope information in a rule-based system (Rosenberg and Bergler, 2012). Zou et al. (2014, 2015) propose
8391
Figure 1: Neural network to predict the focus of negation. The core of the architecture (NN, all components except those inside dotted shapes) takes as input the sentence containing the negation, and each word is represented with its word embedding and specialized embeddings for the negated verb and semantic roles. The additional components inside dotted shapes incorporate information about (a) the scope and (b) context (previous and next sentences).
graph-based models that incorporate discourse information and obtain improvements over previous works. In addition, Shen et al. (2019) present a neural model that leverages word-level and topic-level attention mechanisms to utilize contextual information. We compare our results and theirs in Section 4.2. In this paper, we show that (a) neural networks considering the scope of negation obtain the best results to date and (b) context is not beneficial if scope is available (Section 4).
4 Predicting the Focus of Negation
We approach the task of predicting focus of negation as a sequence labeling task with a neural network. We first describe the network architecture, and then present quantitative results. Section 5 presents a detailed error and qualitative analysis.
4.1 Neural Network Architecture
The network architecture (Fig. 1) consists of a base NN (all components except those inside dotted shapes) plus additional components to include information about the scope and context of negation. Base NN. The base network is inspired by Huang et al. (2015) and Reimers and Gurevych (2017). It is a 3-layer Bidirectional Long Short-Term Memory (BiLSTM) network with a Conditional Random Field (CRF) layer. The network takes as input the sentence containing the negation whose focus is to be predicted, where each word is represented with the concatenation of (a) its pre-trained ELMo embedding Peters et al. (2018), (b) a specialized em
bedding indicating whether a token is the negated verb (not the negation cue), and (c) a specialized embedding indicating semantic roles (one per role label). The specialized embeddings are trained from scratch as part of the tuning of the network.
Scope Information. We add an extra input at the token level indicating whether a token belongs to the scope of the negation whose focus is to be predicted. This new input is then mapped to a third specialized embedding (two values: inside or outside the scope), and concatenated to the word representation prior to feeding it to the 3-layer BiLSTM.
Scope information is taken from a scope detector inspired by Fancellu et al. (2016). Our modifications are as follows. First, we add a CRF layer on top of the 2-layer BiLSTM. Second, we use GloVe embeddings instead of word2vec embeddings. We train the scope detector with CD-SCO (Section 3), and our simple modifications yield the best results to date predicting the scope of negation: 79.41 F1 (vs. 77.77 F1). We do not elaborate on the scope detector as we only leverage it to predict focus.
Context. We also experiment with an additional component to add contextual information (previous and next sentences), as previous work has shown empirically that doing so is beneficial (Zou et al., 2014). While we tried many strategies (e.g., concatenating sentence embeddings to the representations from the 3-layer BiLSTM), we present only the one yielding the best results. Specifically, we use 2-layer Bi-LSTMs with an attention mechanism (Bahdanau et al., 2014; Yang et al., 2016). The attention weights  $(\mathsf{a_p}$  and  $\mathsf{a_n}$  for the previous
8392
P	R	F1	Acc
Zou et al. (2014)	71.67	67.43	69.49	67.1
Zou et al. (2015)	n/a	n/a	n/a	69.4
Shen et al. (2019)	n/a	n/a	n/a	70.5
NN (baseline)	72.14	71.63	71.88	71.6
NN + S	75.92	75.7	75.81	75.7
NN + Cntxt	73.69	73.17	73.43	73.2
NN + S + Cntxt	74.15	73.74	73.94	73.7
Table 2: Focus prediction results of the best performing previous works and our neural network (baseline network and adding components). S and Cntxt refer to Scope and Context, respectively. Note that Zou et al. (2014) do not report the accuracy of their model, but they do in their follow-up work (Zou et al., 2015).
and next sentences respectively) are concatenated to the representations from the 3-layer BiLSTM.
Hyperparameters and Training Details. The cell states of all BiLSTMs have size 350 and we use dropout with a ratio of 0.6. We use the stochastic gradient descent algorithm with Adam optimizer (Kingma and Ba, 2014) and a learning rate of 0.001 for tuning weights. We set batch size to 24 and stop the training process after the F1 on the development split does not increase for 50 epochs. The final model is the one which yields the highest F1 on the development split. We combined the original train and development splits from PB-FOC and used  $95\%$  of the result as training split and the remaining  $5\%$  as development split. The implementation uses PyTorch (Paszke et al., 2019).
We refer the readers to the supplemental material for additional details on the neural architecture.
Table 2 presents the results obtained with the *SEM Shared Task test split and evaluation script. Our best network architecture (NN + Scope) outperforms all previous works (Accuracy: +5.2, 7.4%).
Not all components of the architecture we experiment with are beneficial. Our main finding is that scope information, as predicted by a scope detector trained on CD-SCO, is very useful. Indeed, the core of the network (3-layer BiLSTM and CRF layer) obtains 75.81 F1 (vs. 71.88) when the input includes scope information. Disabling other specialized embeddings—indicating the negated verb and semantic roles—results in substantial drops in performance (not shown in Table 2).
%insts.	P	R	F1
ARG0	4.07	92.9	44.8	60.5
ARG1	43.82	77.9	90.4	83.7
ARG2	4.92	62.0	88.6	72.9
ARG3	0.42	16.7	33.3	22.2
ARG4	0.56	60.0	75.0	66.7
M-NEG	25.98	83.8	50.3	62.8
M-TMP	7.16	71.2	92.2	80.3
M-MNR	5.76	88.6	95.1	91.8
M-ADV	3.09	85.0	77.3	81.0
M-LOC	1.12	60.0	75.0	66.7
M-EXT	0.84	100.0	100.0	100.0
M-DIR	0.28	50.0	100.0	66.7
M-PNC	1.69	84.6	91.7	88.0
M-DIS	0.14	100.0	100.0	100.0
M-CAU	0.14	0.0	0.0	0.0
Table 3: Results per role with our best system (NN + Scope, Figure 1). % insts. indicates the percentage of foci per role in the test set.
According to the creators of PB-FOC and more recent work (Zou et al., 2014, 2015), context is important to determine the focus of negation. Our results confirm this observation: adding the previous and next sentences via attention mechanisms improves the results: 73.43 vs. 71.88 F1. Our results also show, however, that the scope of negation—not previously considered—is more beneficial than context. As a matter of fact, adding context is detrimental if scope is taken into account.
Table 3 presents the results of the best system (NN + Scope) per role. We observe that all roles obtain relatively high F1 scores (>60.5) with two exceptions:  $\mathrm{ARG}_3$  (22.2) and M-CAU (0.0). Many roles are rarely the focus ( $\leq 5\%$ :  $\mathrm{ARG}_0$ ,  $\mathrm{ARG}_2$ ,  $\mathrm{ARG}_3$ ,  $\mathrm{ARG}_4$ , etc.), yet the F1 scores with those roles are similar or even higher than more frequent roles (e.g.,  $\mathrm{ARG}_1$ ). In other words, the neural model is able to predict the focus with similar F1 scores, regardless of what role is the focus.
In Table 4, we provide a quantitative analysis of the results obtained with the best system (NN + Scope). We split the test set into four categories and subcategories, and then evaluate the test instances that fall into each subcategory. Specifically, we consider the focus length measured in tokens, the sentence length measured in tokens, the number of roles in the verb-argument structure of the negated verb (intuitively, the more roles to choose from, the harder to predict the right one), and the verb class of the negated verb. We obtained verb classes from the lexical files in WordNet (Miller, 1995).
$^{1}$ Code available at https://github.com/mosharafhossain/focus-of-negation
8393
%insts.	P	R	F1
focus length	1	39.47	85.2	61.2	66.0
2-5	33.85	92.2	85.5	87.7
6-15	21.91	95.3	93.6	93.7
>15	4.78	89.7	82.4	84.1
sent. length	5-10	7.44	82.6	73.6	74.1
11-15	10.39	88.0	85.1	85.5
16-30	46.63	79.8	77.7	76.7
>30	35.53	77.9	75.9	75.1
#roles	2, 3 roles	10.81	90.3	89.6	89.7
4 roles	35.25	80.4	79.3	77.2
5 roles	37.50	77.8	77.5	76.1
>5 roles	16.43	72.9	65.8	64.7
verb class	possession	17.70	75.1	73.0	71.0
commun.	14.04	80.0	80.0	79.7
cognition	12.36	88.9	85.2	81.8
social	10.81	77.2	75.3	74.3
Table 4: Quantitative analysis of the results in the test set. We measure focus and sentence lengths in tokens. We provide weighted averages per label, thus the F1 scores may not fall between P and R.
Regarding focus length, we observe that single-word foci are the hardest followed by long foci (over 15 tokens). This leads to the conclusion that the network struggles to represent single words and long sequences of words. We note that many foci are single words (39.47%) despite this subcategory obtaining the worst results (F1: 66.0). Regarding sentence length, we observe comparable F1 scores (74.1-76.7) except with sentences between 11 and 15 tokens (85.5). These results lead to the conclusion that since the focus prediction task is defined at the semantic role level, role length is more important than sentence length.
Unsurprisingly, the model obtains worse results depending on the number of roles in the verbargument structure of the negated verb—effectively, the model suffers when it has more roles to choose from. Negated verbs with up to three roles obtain the highest F1 scores (89.7), and results drop significantly (64.7) when there are more than 5 roles (only  $16.43\%$  of instances).
Finally, we provide detailed results for the verbs belonging to the most frequent verb classes: possession (buy, take, get, etc.), communication (say, al-lege, etc.), cognition (think, believe, imagine, etc.), and social (meet, party, etc.). Communication and cognition verbs obtain the best results; this is due in part to the fact that verbs belonging to those verb classes tend to have fewer semantic roles.
5 Error and Qualitative Analysis