SlowGuess/ACL_2020_OCR huggingface dataset:
To better understand the strengths and weaknesses of our models, we perform a detailed qualitative analysis of the errors made in predicting focus. Negation is a complex semantic phenomenon which interacts with other aspects of the meaning and structure of sentences, and this complexity is reflected in the diversity of errors. We perform the analysis over all 712 negations in the test set, investigating how linguistic properties of the negated sentences influence performance across the four models (baseline, scope, context, and combined); we consider nearly 3,000 predictions in total. The counts in this section reflect instance-model pairings; it could happen, for example, that three of the four models predict the wrong focus for a sentence with a particular linguistic property. For some sentences, multiple error types are relevant.
We identify three broad categories of errors: syntactic (5.1), semantic (5.2), and other (5.3). There are multiple error types within each category, and each error type is associated with a particular linguistic property of the negated sentence. Here we focus on the most frequently-occurring error types per category, as these offer the greatest insight into specific strengths and weaknesses of the models.
The distribution of error categories across the four models is shown in Table 8 and discussed in more detail below (5.4).
Representative examples from PB-FOC for each error type appear in Tables 5, 6, and 7. For each example, we show the full sentence, with predicted scope (as output by the scope detector trained with CD-SCO) between double angle brackets and semantic roles in square brackets. For each negated sentence, the table shows the gold focus  $(\mathbf{GF})^2$  and the predicted focus (PF), along with the model(s) responsible for the incorrect prediction.
5.1 Syntactic Error Types
Our analysis reveals three prominent error types related to the structure of negated sentences.
1. Complex verb errors occur when the target verb is part of a complex verb constellation, due to passivization, complex tense constructions, or modal constructions. These constructions result in multi-word verb constellations, such as can't be cured in example 1.1 (Table 5). These are challeng-
2Gold focus annotations come from the PB-FOC corpus and may include some errors. Some properties of the PB-FOC annotations are discussed in Section 2.
Syntactic Error Type	Examples from PB-FOC
1.1. Complex verb(all models) →	There is [nothing]ARG2wrong with the market << [that]ARG2[ca]M-MOD n't be[cured]verb [by a little coherence and common sense in Washington.]ARG3≫
GF: [nothing ... that]ARG2PF: [a little coherence and common sense in Washington]ARG3
1.2. Complex sentence(NN + context) →	Since production costs were guaranteed, it didn't matter that << [a program]ARG1[could]M-MOD n't be [sold]verb [abroad]M-LOC or put into syndication, >> [as mostAmerican programs are.]M-ADV
GF: [abroad]M-LOCPF: [as most American programs are]M-ADV
1.3. Role adjacency(NN + scope + context) →	It was an overreaction to [an event (the failure of a management and union group to get bank financing for a takeover of UAL) that]ARG0 << doesn't [mean]verb[that much]ARG1[to lots of stocks.]M-MNR≫
GF: [that much]ARG1PF: [much]ARG1[to lots of stocks]M-MNR
Table 5: Syntactic error types. KEY: [semantic role], «predicted scope», GF: gold focus, PF: predicted focus.
ing for all models, but especially for the baseline, with 56 error cases (vs. 36, 43, and 41 for the scope, context, and combined models).
2. Complex sentence structure errors are even more common, with 116/73/87/63 occurrences for the four models. Instances triggering this error type are sentences with relative clauses or complement clauses, as well as sentences with non-canonical linking between argument structure and grammatical function, such as passives and questions. According to Horn (2010), relative and complement clauses can alter the behavior of negation, compared to simple declarative sentences. Example 1.2 in Table 5 shows scope helping with complex sentence structure—both models which incorporate scope predict the correct focus, which occurs within the predicted scope. The other two models choose an argument outside of the predicted scope.
Our third type of syntactic error occurs due to 3. Role adjacency in the sentence, leading to errors in span prediction. The property associated with this error type is linear adjacency of semantic roles, with no textual material in between. Example 1.3 in Table 5 shows that the model predicts part of the correct role but then extends the span to incorporate a second role.
In summary, models with access to predicted scope make fewer syntactic errors than models without scope.
5.2 Semantic Error Types
Three different types of errors related to meaning occur with high frequency.
1. Errors due to distractors are the most fre
quent individual error type. The term distractor is most familiar from pedagogical discussion of multiple-choice questions, where a distractor is an incorrect option that test-takers are likely to mistake for a correct answer. We use the term here to refer to textual material which leads the neural network away from the gold focus. Specifically, distractors are found in two aspects of the input representation for a given instance: the predicted scope, and the adjacent sentences (previous and next) provided as part of the models which incorporate context. This error type is, by definition, not applicable for the baseline model.
We identify 124 occurrences of distractor errors for the scope model, 87 for the context model, and 130 for the combined model, making this the largest error category. Example 2.1 in Table 6 marks distractors in bold-face type. In this case, all models predict after the last crash as the focus. $^{3}$  The predicted focus occurs in the predicted scope, and the head noun crash appears in the surrounding context. In addition to the direct repetition the 1987 crash in the sentence following, we see the synonym market plunge in the previous sentence.
2. Lack of referential specificity in the gold focus is a less-frequent and more speculative error type. The idea is that focus is difficult to predict correctly when the focused semantic role is pronominal or otherwise requires additional information for reference resolution. Across the models, we count 22 occurrences. In most of these cases, the gold focus is a pronoun (it, ex. 2.2). All models seem to
An argument could be made for M-NEG as the negated role; however, we show the gold focus according to PB-FOC.
Semantic Error Type	Examples from PB-FOC
2.1. Distractors(all models) →	A further slide also would resurrect debate over a host of [other, more sweeping changes]ARG1proposed - but [not]M-NEG << [implemented]verb [after the last crash.]M-TMP >>
GF: [other, more sweeping changes]ARG1PF: [after the last crash]M-TMP
PrevSent: A deeper market plunge today could give them ...NextSent: Most notably, several of the regulatory steps recommended by the Brady Task Force, which analyzed the 1987 crash, would be revived ...
2.2. Lack of specificity(all models) →	The main advantage of a convertible mortgage is that << [it]ARG0is >> not a sale and [therefore]M-DIS << does not [trigger]verb [costly transfer taxes and reappraisal.]ARG1 >>
GF: [it]ARG0PF: [costly transfer taxes and reappraisal]ARG1
2.3. Neg. Polarity Items(NPIs)(all models) →	[And]M-DIS [unlike IBM's water-cooled mainframes]M-ADV, << [it]ARG0doesn't [need]verb [any plumbing.]ARG1 >>
GF: [it]ARG0PF: [any plumbing]ARG1
Table 6: Semantic error types. KEY: [semantic role], «predicted scope», GF: gold focus, PF: predicted focus.
Other Error Types	Examples from PB-FOC
3.1. Quotations
(all models)→	[No,]M-DIS to my mind, << [the Journal]ARG0 did not ["defend"verb [sleaze,]ARG1 >> [fraud, waste, embezzlement, influence-peddling and abuse of the public trust.]ARG1
GF: [not]M-NEG
PF: [sleaze, fraud, waste, ... public trust]ARG1
3.2. Particle verbs, PPs and inf. complements
(all models)→	[But]M-DIS << don't [pay]verb [30 times earnings]ARG1 [for a company that's expected to grow at 15% a year.]ARG3 >>
GF: [for a company that's expected to grow at 15% a year]ARG3
PF: [30 times earnings]ARG1
Table 7: Other error types. KEY: [semantic role], «predicted scope», GF: gold focus, PF: predicted focus.
disprefer predicting bare pronouns as focus.
Occurrence of 3. negative polarity items (NPIs) also influences the accuracy of the model. Negative polarity items (such as any or yet, see Horn (2010)) are licensed in the scope of negation but ungrammatical elsewhere. For example, it's ungrammatical to say *I have eaten any fish. Given the strong association between negation and NPIs, it is not surprising that our models tend to predict as focus any role which contains an NPI (example 2.3). This error type occurs roughly twice as often in models with scope than in models without scope.
5.3 Other Error Types.
Two other error types occur often enough to deserve mention. 1. Quotation errors generally involve quoted direct speech, which seems to be especially problematic when only part of a clause is quoted speech. In example 3.1, the quoted speech is the verb plus its direct object, and all models select the
role of the direct object as predicted focus. The final error type is a sort of catch-all: 2. Particle verbs, prepositional phrases, and infinitival complements. As with complex sentence structures, these error types reflect complex verbal argument structure.
Table 8 shows the distribution of error types across the four systems. Errors due to particular syntactic structures are the most common, with the subtype of complex sentences making up the bulk of these (339). The baseline network deals very poorly with both complex verb constellations and complex sentence structures, and incorporating predicted scope consistently reduces the number of errors
An error count is incremented whenever the relevant linguistic property is identified in a sentence for which the relevant system has made an incorrect prediction. Note that one sentence may present more than one linguistic property.
Synt.	Sem.	Other
n	593	407	164
NN (baseline)	32.2	3.0	25.6
NN + Scope	21.3	35.4	24.4
NN + Context	25.6	24.3	25.6
NN + Scope + Context	20.9	37.3	24.4
Total	100.0	100.0	100.0
Table 8: Number of errors made by all systems per category  $(n)$ , and percentage made by each system.
Synt.	Sem.	Other
NN (baseline)	78.6	4.5	16.9
NN + Scope	40.5	46.6	12.9
NN + Context	51.7	33.9	14.4
NN + Scope + Ctx	39.4	47.9	12.7
Table 9: Percentages per error category, for each system.
of this type. This suggests that considering scope helps the system to deal with complex sentences.
For errors related to semantics, the picture is reversed. The systems which consider scope are especially prone to distractor errors, the most common error type over all (341). When we have both scope and context, the system has even more potential distractor candidates and makes more errors. The two error types in the Other category are distributed roughly evenly across the models, suggesting that none of the current models is any better than the others at dealing with these error types.
In Table 9 we see a second view on the error distributions, now considering each category as a proportion of the errors made by the system. Again we see that predicted scope shifts the balance of error types from syntactic to semantic. By reinforcing a subsection of the text in the input representation, the search space for complex sentences narrows and the system has a better chance of selecting the correct focus. This same behavior is a disadvantage when the gold focus is not part of the predicted scope, as the scope distracts attention away from other plausible candidate roles. Similarly, including context through adjacent sentences sometimes reinforces the correct focus through introduction of other semantically-related terms, and sometimes clutters the field through the very same mechanism.
Negation is generally understood to carry positive meaning, or in other words, to suggest affirmative alternatives. Predicting the focus of negation (i.e.,
pinpointing the usually few tokens that are actually negated) is key to revealing affirmative alternatives.
In this paper, we have presented a neural architecture to predict the focus of negation. We work with PB-FOC, a corpus of verbal negations (i.e., when a negation cue grammatically modifies a verb) in which one semantic role is annotated as focus. Experimental results show that incorporating scope of negation information yields better results, despite the fact that we train the scope detector with data in a different domain (short stories vs. news). These results suggest that scope of negation transfers across domains. Our best model (NN + Scope) obtains the best focus prediction results to date. A quantitative analysis shows that this model is robust across most role labels (Table 3), sentence lengths, and verb classes (Table 4). The model obtains worse results, however, when the role that is the focus is only one token, or the negated verb has more than 5 roles (Table 4).
In addition to state-of-the-art results, we have presented a detailed qualitative analysis. We discover three main error categories (syntactic, semantic, and other) and 8 error types after manual analysis of the predictions made by the four models with all test instances. We draw two main insights from the qualitative analysis. First, including scope information solves many syntactic errors but introduces semantic errors (recall that scope information is beneficial from a quantitative point of view). Second, the lower results after including context, at least with the current architecture, are largely due to additional semantic errors via distractors in the previous and next sentences.
Thanks to the anonymous reviewers for their insightful comments. This material is based upon work supported by the NSF under Grant No. 1845757. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF. The Titan Xp used for this research was donated by the NVIDIA Corporation. Computational resources were also provided by the UNT office of High-Performance Computing.
Pranav Anand and Craig Martell. 2012. Annotating the focus of negation in terms of questions under
8397
discussion. In Proceedings of the Workshop on Extra-Propositional Aspects of Meaning in Computational Linguistics, pages 65–69, Jeju, Republic of Korea. Association for Computational Linguistics.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Rajendra Banjade and Vasile Rus. 2016. Dt-neg: Tutorial dialogues annotated for negation scope and focus in context. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France. European Language Resources Association (ELRA).
Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016. Neural versus phrase-based machine translation quality: a case study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 257-267, Austin, Texas. Association for Computational Linguistics.
Eduardo Blanco and Dan Moldovan. 2011. Semantic representation of negation using focus detection. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 581-589, Portland, Oregon, USA. Association for Computational Linguistics.
Xavier Carreras and Lluis Marquez. 2005. Introduction to the CoNLL-2005 shared task: Semantic role labeling. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005), pages 152-164, Ann Arbor, Michigan. Association for Computational Linguistics.
Isaac Councill, Ryan McDonald, and Leonid Velikovich. 2010. What's great and what's not: learning to classify the scope of negation for improved sentiment analysis. In Proceedings of the Workshop on Negation and Speculation in Natural Language Processing, pages 51-59, Uppsala, Sweden. University of Antwerp.
Vesna Djokic, Jean Maillard, Luana Bulat, and Ekaterina Shutova. 2019. Modeling affirmative and negated action processing in the brain with lexical and compositional semantic models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5155-5165, Florence, Italy. Association for Computational Linguistics.
Federico Fancellu, Adam Lopez, and Bonnie Webber. 2016. Neural networks for negation scope detection. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 495-504, Berlin, Germany. Association for Computational Linguistics.
Federico Fancellu, Adam Lopez, Bonnie Webber, and Hangfeng He. 2017. Detecting negation scope is easy, except when it isn't. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 58-63, Valencia, Spain. Association for Computational Linguistics.
Uri Hasson and Sam Glucksberg. 2006. Does understanding negation entail affirmation?: An examination of negated metaphors. Journal of Pragmatics, 38(7):1015-1032.
Laurence R Horn, editor. 2010. The expression of negation. Mouton de Gruyter.
Laurence R. Horn and Heinrich Wansing. 2017. Negation. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, spring 2017 edition. Metaphysics Research Lab, Stanford University.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
Rodney D. Huddleston and Geoffrey K. Pullum. 2002. The Cambridge Grammar of the English Language. Cambridge University Press.
Lifeng Jia, Clement Yu, and Weiyi Meng. 2009. The effect of negation on sentiment analysis and retrieval effectiveness. In Proceedings of the 18th ACM conference on Information and knowledge management, pages 1827-1830. ACM.
Salud María Jiménez-Zafra, Roser Morante, María Teresa Martín-Valdivia, and L. Alfonso Ureña-López. 2020. Corpora annotated with negation: An overview. Computational Linguistics, 46(1):1-52.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization.
Natalia Konstantinova, Sheila C.M. de Sousa, Noa P. Cruz, Manuel J. Maña, Maite Taboada, and Ruslan Mitkov. 2012. A review corpus annotated for negation, speculation and their scope. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 3190-3195, Istanbul, Turkey. European Language Resources Association (ELRA).
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 681-691, San Diego, California. Association for Computational Linguistics.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39-41.
Roser Morante and Eduardo Blanco. 2012. *SEM 2012 Shared Task: Resolving the Scope and Focus of Negation. In Proceedings of the First Joint Conference on
8398
Lexical and Computational Semantics (*SEM 2012), pages 265-274, Montréal, Canada.
Roser Morante and Walter Daelemans. 2009. A metalearning approach to processing the scope of negation. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 21–29, Boulder, Colorado. Association for Computational Linguistics.
Roser Morante and Walter Daelemans. 2012. ConanDoyle-neg: Annotation of negation cues and their scope in Conan Doyle stories. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012). European Language Resources Association (ELRA).
Isabel Orenes, David Beltran, and Carlos Santamaría. 2014. How negation is understood: Evidence from the visual world paradigm. Journal of Memory and Language, 74:36-45.
Woodley Packard, Emily M. Bender, Jonathon Read, Stephan Oepen, and Rebecca Dridan. 2014. Simple negation scope resolution through deep parsing: A semantic solution to a semantic problem. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 69-78, Baltimore, Maryland. Association for Computational Linguistics.
Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71-106.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL.
Jonathon Read, Erik Velldal, Lilja Øvrelid, and Stephan Oepen. 2012. UiO1: Constituent-based discriminative ranking for negation resolution. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 310–318, Montréal, Canada. Association for Computational Linguistics.