SlowGuess/NAACL_2022_OCR huggingface dataset:
Md Mosharaf Hossain, $^{e}$  Luke Holman, $^{u}$  Anusha Kakileti, $^{e}$  Tiffany Iris Kao, $^{r}$  Nathan Raul Brito, $^{e}$  Aaron Abraham Mathews, $^{e}$  and Eduardo Blanco $^{u}$
$^{\mathrm{a}}$ University of North Texas  $^{\mathrm{b}}$ Arizona State University  $^{\mathrm{c}}$ University of Texas at Austin
{mdmosharafhossain, nathanbrito, aaronmathews} @my.unt.edu lholman2@asu.edu akakileti@gmail.com tiffanyiris1004@gmail.com eduardo.blanco@asu.edu
This paper explores a question-answer driven approach to reveal affirmative interpretations from verbal negations (i.e., when a negation cue grammatically modifies a verb). We create a new corpus consisting of 4,472 verbal negations and discover that  $67.1\%$  of them convey that an event actually occurred. Annotators generate and answer 7,277 questions for the 3,001 negations that convey an affirmative interpretation. We first cast the problem of revealing affirmative interpretations from negations as a natural language inference (NLI) classification task. Experimental results show that state-of-the-art transformers trained with existing NLI corpora are insufficient to reveal affirmative interpretations. We also observe, however, that fine-tuning brings small improvements. In addition to NLI classification, we also explore the more realistic task of generating affirmative interpretations directly from negations with the T5 transformer. We conclude that the generation task remains a challenge as T5 substantially underperforms humans.
Negation can be understood as an operator that transforms the meaning of some expression into another expression whose meaning is in some way opposed to the original expression (Horn and Wansing, 2020). Typically, negated statements are less informative than affirmative statements (e.g., "Paris is not located in England" vs. "Paris is located in France"). Negated statements are also harder to process and understand by humans (Horn and Wansing, 2020). According to Horn (1989), negations carry affirmative meanings. These underlying affirmative meanings, which we refer to as affirmative interpretations, range from implicatures to entailments. For example, the negated statement (1) "Mary never drives long distances without a full tank of gas", carries at least the following affirmative interpretations: (1a) "Mary drives long
An extinct volcano is one that has not erupted in recent history.
- Did something erupt? Yes
- What erupted? An extinct volcano
- When did something erupt? In the past
Affirm. Intp: An extinct volcano erupted in the past.
It was not formed by a natural process.
- Was something formed? Yes
- What was formed? It
- What was something formed by? An artificial process
Affirm. Intp: It was formed by an artificial process.
Table 1: Sentences containing negation, questions and answers about the affirmative counterpart of the main event, and the underlying affirmative interpretation.
distances," (1b) "Mary fills the gas tank before starting a long drive," and (1c) "Mary might drive short and medium distances without a full tank of gas."
In order to empower models to comprehend negation, most previous works target scope (Vincze et al., 2008; Morante and Daelemans, 2012) and focus (Blanco and Moldovan, 2011) detection (Section 2). Scope refers to the part of the meaning that is negated and focus refers to the part of the scope that is most prominently or explicitly negated (Huddleston and Pullum, 2002). Scope and focus detection plays a crucial role to understand what part of a negated statement is actually negated. These tasks do not, however, reveal affirmative interpretations—they tag tokens as belonging or not belonging to the scope and focus of a negation.
In this paper, we present a question-answer driven approach to reveal affirmative interpretations from verbal negations (i.e., when a negation cue grammatically modifies a verb). We adapt QA-SRL (He et al., 2015; FitzGerald et al., 2018) to collect questions and answers regarding the arguments of the affirmative counterpart of a negated predicate. Then, we manipulate the questions and answers to generate an affirmative interpretation. We find that generating and answering questions is
Findings of the Association for Computational Linguistics: NAACL 2022, pages 490 - 503
intuitive to non-experts (albeit they are native English speakers). Consider the examples in Table 1. Annotators first generate and answer a question regarding whether the main predicate in the sentence occurred (with unknown arguments at this point). Then, they generate and answer questions about the arguments of the affirmative counterpart of the main predicate. Arguments may come directly from the negated statement (e.g., What erupted? An extinct volcano) or using commonsense and world knowledge after reading the negated statement (e.g., When did something erupt? In the past). After collecting questions and answers, we automatically generate an affirmative interpretation in the form of a statement (e.g., An extinct volcano erupted in the past).
1. A question-answer driven annotation schema to create AFIN, a corpus of verbal negations and their AFFirmative INterpretations (4,472 negations, 7,277 questions and answers, and 3,001 affirmative interpretations);
2. Corpus analysis indicating which predicate arguments are most often rephrased in the affirmative counterparts;
3. Casting the problem of revealing affirmative interpretations as a natural language inference task and showing that it is challenging for state-of-the-art transformers; and
4. Casting the problem of revealing affirmative interpretations as a generation task and showing that the T5 transformer substantially underperforms humans.
Revealing affirmative interpretations from negations is a challenging endeavor. In the literature, researchers primarily seek to identify scope and focus of negation. The creation of the BioScope (Szarvas et al., 2008) and ConanDoyle-Neg (Morante and Daelemans, 2012) corpora spearheaded research on scope detection (Morante and Daelemans, 2009). Proposals include using traditional machine learning (Lapponi et al., 2012), off-the-shelf semantic parsers and semantic representations (Packard et al., 2014), and neural networks (Fancellu et al., 2016, 2017). PB-FOC (Blanco and Moldovan, 2011) is the largest corpus with focus of negation annotations. Recent proposals for focus detection in
clude graph-based models with discourse information (Zou et al., 2014, 2015), neural networks with word-level and topic-level attention (Shen et al., 2019), and networks using scope information and context (Hossain et al., 2020). Scope and focus are useful to identify what is and what is not negated in a negated statement. Consider the second example in Table 1. Scope and focus do reveal that It was formed—everything but the focus (i.e., by a natural process) is affirmative—but provide no hints about how it was formed (i.e., by an artificial process, artificially, etc). The main goal of this paper is to find these affirmative counterparts to generate affirmative interpretations.
More related to the work presented here, Sarabi et al. (2019) present a corpus of negations and their underlying affirmative interpretations (they call them positive interpretations). We are inspired by them but bypass several of their limitations. First, they only work with negations from Simple Wikipedia, a site devoted to English learners. As a result, their corpus uses (relatively) unsophisticated vocabulary and grammar. Second, they impose several restrictions on the negations they work with (e.g., negation cue modifies root verb, sentences between 6 and 25 tokens and not including certain tokens (because, until, etc.)). Third, their affirmative interpretations are restricted to a rephrasing of the statement containing negation with only one change: an argument of the negated predicate. In contrast, we barely impose restrictions on the negations we work with (no questions and no auxiliary verbs). More importantly, we introduce a question-driven approach that allows us to obtain multiple affirmative interpretations with increasing degrees of complexity (see examples in Table 3).
Recently, Jiang et al. (2021) study the problem of identifying commonsense implications of negations and contradictions. More specifically, they work with if-then rules such as If  $X$  does not learn new things, then  $X$  does not gain new knowledge and If  $X$  does not leave the building, then  $X$  stays in the building. These rules capture general commonsense knowledge about what happens if an event does not occur. Unlike them, we work with naturally occurring sentences that include negated predicate-argument structures with many arguments (agent, theme, manner, time, etc.). In addition, our affirmative interpretations reveal that predicates that are grammatically negated are actually factual (but with different arguments).
<sup>1</sup>Corpus and code available at https://github.com/mosharafhossain/AFIN.
Sent.	WH	AUX	SUB	VERB	OBJ1	PREP	OBJ2
Predicate questions	(a)	Was	something	formed	?
(b)	Does	something	kill	someone	?
(c)	Will	someone	have	to	do something
Argument questions	(a)	What	was	something	formed	by	?
(b)	How often	does	something	kill	someone	?
(c)	Who	will	have	to	do something
Table 2: Predicate questions and argument questions (one per negated predicate) generated by annotators from the sentences (a) It was not formed by a natural process., (b) However, the ground shaking almost never kills people, [...] and (c) [...] he hopes Australian teams will not have to travel so much to meet first class competition.
3 A Question-Answer Driven Approach to Collect Affirmative Interpretations
This section outlines our approach to create AFIN, a corpus of verbal negations and their affirmative interpretations. We first describe the sources of naturally occurring negations in our corpus. Then, we outline the template-based approach to guide annotators in generating and answering questions about the affirmative counterpart of the negated predicate. Lastly, we describe the process to generate natural-language affirmative interpretations from the questions and answers.
3.1 Collecting Sentences Containing Negation
We start with the sentences in QA-SRL Bank 2.0 (FitzGerald et al., 2018), a corpus with 64,000 sentences across three domains: Wikipedia, Wikinews, and science textbooks (Kembhavi et al., 2017). Motivated by Fancellu et al. (2016), we select sentences containing negations checking for the following negation cues: not, n't, no, never, without, nothing, none, nobody, nowhere, and neither and nor. We only impose two restrictions: the sentences cannot be questions and the negation cues have to modify a verb that is not an auxiliary verb. We check the latter using universal dependencies as extracted by the parser in spaCy (Honnibal et al., 2020). We consider cues that directly or indirectly modify the verb, as exemplified in Figure 1. We will use target verb to refer to the negated verb in the remaining of the paper.
3.2 Generating and Answering Questions
Given a sentence and a target verb, our goal is to guide annotators to generate and answer questions about the (potential) affirmative counterpart of the target verb. First, they ask a predicate question to determine whether the affirmative counterpart of the target verb is factual (with unknown arguments). If it is, then they ask and answer argument
Figure 1: Illustration of the criteria to select negated verbs. We select all negations that modify non-auxiliary verbs either directly (top) or indirectly (bottom).
questions about the arguments of the affirmative counterpart of the target verb. Consider the following sentence: However, no children resulted from the marriage. The answer to the predicate question (Did anything result?) is No, thus no argument questions are considered. Now consider another sentence: Cloning does not happen naturally. The answer to the predicate question (Does something happen?) is Yes, thus annotators continue asking and answering argument questions: What happens? Cloning and How does it happen? Artificially (or with human intervention, for example).
Template-Based Question Generation In principle we could allow annotators to generate questions following their preferred wording. We found, however, that guiding them increases consistency and speed. To this end, we adapt the seven-slot template technique by He et al. (2015). For predicate questions (expected answer: Yes or No), we use the following combinations of slots: AUX x SUB x VERB x OBJ1 x PREP x OBJ2. For argument questions, we include an additional slot in the first position: WH. The full list of values for each slot are detailed in Appendix A. We provide below some examples for each slot.
- WH: Who, What, Whom, When, Where, etc.
- AUX: is, was, does, did, has, had, can, etc.
- SUB: something, or someone
- VERB: full conjugation of the target verb
- OBJ1: something, or someone
- PREP: by, to, for, with, about, of, or from
- OBJ2: someone, something, somewhere, do, doing, etc.
The templates allow annotators to generate a wide variety of questions. Table 2 shows several examples of predicate and argument questions generated from three target verbs. Note that humans are needed to choose values for each slot so that the resulting question is correct (right auxiliaries, conjugation, tense, number matching, etc.). Annotators generate questions in the following order of wh-words: who (or what) does/did (something) to whom (or what), when, where, how, how much, how many, how long, how often, and why. This order makes the generation of affirmative interpretations in natural language easier (Section 3.3).
Answering Questions and Assigning Confidence Scores Immediately after generating a question (i.e., before generating the next question), annotators answer it and indicate how confident they are in their answer. Note that several compatible answers are usually possible (e.g., before and in the past are usually interchangeable). Answers may come from the sentence containing the target verb and its arguments, or written by annotators using commonsense and world knowledge. Consider the following sentence: The steep sides form because the lava cannot flow too far from the vent (example (1) in Table 3). The answer to What flows? comes from the sentence: Lava. On the other hand, the answer to Where does something flow? is a rewrite of an argument of the target verb: close to vents. In the second example of Table 3, all answers come from the sentence with the target verb except When was something classified?, which is In the past.
Regardless of where answers come from, annotators assign a confidence score using a four-point Likert scale:
- 4: Extremely confident. I am certain that the answer is correct given the negated statement. For example, given "Scientists think that it will probably not erupt again," annotations answer When did something erupt? with In the past and assign a score of 4.
- 3: Very confident. My answer is very likely correct given the negated statement. For example, given “These volcanoes usually do not produce streams of lava,” an annotator generated How often does something produce? and answered Rarely with a confidence score of 3.
Text: It was not formed by a natural process.
Figure 2: Web interface to guide annotators in asking and answering questions. The screenshot shows the fillers for the VERB slot (i.e., the conjugation of form)
- 2: Moderately confident. My answer is likely correct given the negated statement. There are, however, many possible answers and my answer may be incorrect in an unlikely scenario. For example, given "It does not release carbon dioxide," an annotator assigned a confidence score of 2 to his answer to the question What does something release? Fresh air.
- 1: Slightly confident. My answer is probably correct, but there is no strong evidence in the sentence. For example, given "The second plot can not be explained using data," an annotator answered How is something explained? with Using observations and assigned a confidence score of 1. These answers often encode commonsense rather than an inference from the statement containing the target verb.
Scaling the Annotation Process Inspired by FitzGerald et al. (2018), we develop a web interface that facilitates the task of generating questions following our templates. More specifically, the interface auto-suggest to annotators the valid fillers for each slot. For example, if annotators start typing  $W$ , only fillers for the WH slot starting with  $W$  are suggested. The fillers for the next slot are suggested after the selection for the current slot is finalized. Figure 2 presents a screenshot of the interface with the auto-suggestions for the VERB slot (i.e., the conjugation of the target verb, form).
Annotation Quality Five undergraduate students who are native English speakers participated in the annotation process. They were trained in multiple sessions and conducted pilot annotations followed by discussion sessions before starting the annotations that resulted in the corpus described here. We do not calculate inter-annotator agree
WH	AUX	SUB	VERB	PREP	Answer	Affirmative Interpretation
(1)	What Where	does	something	flows flow	? ?	Lava Close to vents	Lava flows. Lava flows close to vents.
(2)	What What When	was was was	something something	classified classified classified classified	as	? ? ?	Fungi Plants In the past	Fungi were classified. Fungi were classified as plants. Fungi were classified as plants in the past.
Table 3: Examples of questions and answers generated by annotators and the resulting affirmative interpretations. The sentences containing the negated predicates are (1) The steep sides form because the lava cannot flow too far from the vent. and (2) Today, fungi are no longer classified as plants. We do not show the OBJ1 and OBJ2 slots because they are empty for all the questions in these examples.
ment since two different answers to the same questions are likely to be correct. Consider the following sentence: "Scientists never use only one piece of evidence to form a conclusion." Two valid (and yet non-overlapping) answers to the question What does someone use? are a reasonable amount of evidence and mathematical models. The limitations of current automated metrics to determine whether these two answers are correct are well known (Liu et al., 2016), so we decided to conduct a manual evaluation. More specifically, we manually validated 479 questions and answers from a random sample of 200 target verbs in the corpus. A sixth person not involved in the generation and answering of the questions validated the 479 question-answer pairs as well as graded them with the same 4-point confidence scale. The validation phase revealed that (a) only  $3\%$  of the question-answer pairs are incorrect and (b) there is a strong correlation (Spearman: 0.71, Pearson: 0.70, p-value  $< 0.005$  for both) between the scores.
3.3 Generating Affirmative Interpretations from Questions and Answers
We devise a rule-based approach in order to go from the questions generated and answered by annotators to an affirmative interpretation in natural language. Recall that annotators generate (and answer) questions in the following order: who (or what) does/did (something) to whom (or what), when, where, how, how much, how many, how long, how often, and why. Our approach is deterministic and manipulates answers depending on verb tense and number (which are obtained with part-of-speech tags and regular expressions).
We start with the answer to the first question (who (or what) does/did something?) in order to establish the subject of the affirmative counterpart of the target verb. Depending on whether the question uses the AUX slot, the affirmative interpretation
also uses an auxiliary. Consider the examples in Table 3. In the first example, the question about the subject is What flows? and the answer is Lava, resulting in the initial affirmative interpretation Lava flows. Similarly, in the second example, the question is What was classified? and the answer is Fungi, resulting in the initial affirmative interpretation Fungi were classified.
Having generated an initial affirmative interpretation, the process continues adding arguments to the predicate-argument structure. We add them sequentially to the end of the affirmative interpretation in the order in which argument questions were generated and answered. Consider again the first example in Table 3. The only argument question left is Where does something flow?, which was answered with Close to vents. The initial affirmative interpretation becomes Lava flows close to vents. Since there are no additional questions, this is the final affirmative interpretation. Let us now consider the second example again. After incorporating the answer to the second question into the initial affirmative interpretation, we have Fungi were classified as plants (after including the preposition used in the question). Incorporating the answer to the third question, we have the final affirmative interpretation: Fungi were classified as plants in the past. The Appendix B provides additional details and special cases.
The question-answer driven approach to generate and answer questions revealed that 3,001 out of the 4,472 (negated) target verbs carry an affirmative interpretation  $(67.1\%)$ . On average, annotators generated and answered 2.4 questions per target verb. Also, the average lengths of those questions and answers (in tokens) are 5.0 and 3.5, respectively. The average negated sentence is 25.8 tokens long, while its affirmative interpretation is 11.2 tokens
min. confidence	4	≥ 3	≥ 2	≥ 1
%verbs	85.50	97.77	99.87	100.0
Table 4: Percentage of target verbs depending on the minimum confidence score assigned to any of the answers regarding the affirmative counterpart. Annotators almost always (97.77%) are extremely (4/4) or very confident (3/4) about their answers.
long, indicating that affirmative interpretations are much shorter than negated sentences. Appendix C provides additional details, including the distribution of wh-words in the questions.
Percentages, shown in Table 4, indicate that a vast majority of affirmative interpretations (85.5%) are generated from questions and answers about which annotators were extremely confident (confidence score: 4). The percentage raises to 97.77% if we include questions answered about which annotators were very confident (confidence score: 3).
Similar to Sarabi et al. (2019), we manually analyze 100 random examples from our corpus to find which arguments differ in the verb-argument structure of the (negated) target verb and the affirmative counterpart. We discovered these arguments primarily have the following functions (Frequencies and examples in Table 5):
- Patient (or theme)  $(24\%)$ . The most common argument is the person or thing that is affected or acted upon by the target verb. In the first example, we go from workers had nothing to workers had only their labor.
- Manner (23%). The second most common argument is the way in which the target verb takes place (the how). In the example, we go from don't go through life with regrets to go through life with satisfaction.
- Quantity (10%). Arguments expressing specific (e.g., four, three) or abstract quantities (e.g., many, less) represent 10% of changes in arguments. For example, we go from Many mutations have no effect on the proteins to Some mutations have an effect on the proteins.
- Time (10%). Tied in frequency with quantity, we observed arguments expressing temporal information. In the example, we go from not allowed today to allowed in the past.
- Reason (or cause)  $(9\%)$ . The fifth most common argument expresses the why of the target verb. We understand why widely, including reasons, causes, justifications, and explana
tions. In the example, we go from something not existing without water to Earth has complexity and diversity because of water.