SlowGuess/NAACL_2021_OCR huggingface dataset:
Liwei Jiang<sup>1</sup> Antoine Bosselut<sup>2,3</sup> Chandra Bhagavatula<sup>2</sup> Yejin Choi<sup>1,2</sup>
{lwjiang,yejin}@cs.washington.edu,{chandrab}@allenai.org
Natural language inference requires reasoning about contradictions, negations, and their commonsense implications. Given a simple premise (e.g., "I'm mad at you"), humans can reason about the varying shades of contradictory statements ranging from straightforward negations ("I'm not mad at you") to commonsense contradictions ("I'm happy"). Moreover, these negated or contradictory statements shift the commonsense implications of the original premise in nontrivial ways. For example, while "I'm mad" implies "I'm unhappy about something," negating the premise (i.e., "I'm not mad") does not necessarily negate the corresponding commonsense implications.
In this paper, we present the first comprehensive study focusing on commonsense implications of negated statements and contradictions. We introduce ANION<sup>1</sup>, a new commonsense knowledge graph with 624K if-then rules focusing on negated and contradictory events. We then present joint generative and discriminative inference models for this new resource, providing novel empirical insights on how logical negations and commonsense contradictions reshape the commonsense implications of their original premises.
Humans reason about underlying causes and effects of events described in text. For example, in Figure 1, the event "X wears a mask" is associated with many causal inferences such as "X is seen as responsible," or "Others get protected." Hypothesizing and reasoning about commonsense inferences is used for understanding complex situations encountered in everyday life (Sap et al., 2019; Bisk et al., 2020; Bhagavatula et al., 2020; Sakaguchi et al., 2020). This ability eludes AI systems, and has motivated the design of a wealth of
Figure 1: Commonsense inferences for the event "X wears a mask," its logical negation and commonsense contradiction events, and their associated inferences.
commonsense knowledge resources, such as Cyc (Lenat, 1995), ConceptNet (Speer et al., 2017), and ATOMIC (Sap et al., 2020; Hwang et al., 2020), to provide structured reasoning capabilities to AI systems (Lin et al., 2019; Feng et al., 2020).
However, reasoning about negated observations remains a challenge (Hossain et al., 2020). While negation is often considered a poorer form of meaning than affirmation $^{2}$  (Ackrill, 1963; Horn and Wansing, 2020), negated statements can still imply expressive commonsense inferences. In Figure 1, the negated event "X doesn't wear a mask,"
<sup>1</sup>Data and code available at https://github.com/ liweijiang/anion
$^{2}$ Following Horn and Wansing (2020), we classify declarative expressions as affirmations or negations/contradictions based on whether they affirm or deny an action or object.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4380-4397 June 6-11, 2021. ©2021 Association for Computational Linguistics
is connected to rich commonsense inferences, despite describing the absence of action. However, negated observations are rarely found in commonsense knowledge resources. For example, negated examples make up only  $\sim 3\%$  of examples in the ConceptNet knowledge graph (Li et al., 2016).
This scarcity poses downstream issues for systems that must understand negated situations. Commonsense knowledge models (Bosselut et al., 2019; Hwang et al., 2020) trained on resources of largely affirmative instances struggle particularly with negation examples. Their ability to hypothesize inferences for negated events is  $35\%$  lower than for affirmative events ( $\S 4.2$ ). Furthermore, since negated statements are asymmetrically mentioned in text compared to affirmative statements (Jowett et al., 1892; Horn and Wansing, 2020), large-scale pretraining does not implicitly learn negation scoping (Kim et al., 2019). As a result, when presented with negated concepts, pretrained neural language models (PLMs) often exhibit the same associations as affirmative statements (Kassner et al., 2020). Motivated by these observations, our work focuses on improving the ability of knowledge models to make commonsense inferences about events that convey denial, rejection or contradiction of actions.
We define our contributions as follows. First, we crowdsourced a new large scale resource, Array of commonseNse Inferences for Oppositions and Negations (ANION), which contains inferences for different types of negated events. This new resource can be used to train knowledge models on commonsense inferences associated with the absence of actions. Second, we propose a new class of negation discriminators that can be applied to generated commonsense inferences. These discriminators partition inferences based on logical consistency, thereby mitigating the effects of common affirmative associations that violate negation constraints. Discriminators are trained using contrastive samples from paired affirmative and negated events in ANION. Finally, we conduct an empirical study of both of these techniques and show that using training- and discriminator-based approaches for modeling negation cuts the performance difference between affirmative and negated events by  $73\% - 85\%$  depending on the negation variety.
2 Commonsense Negation
Negation in Language In Categories and De Interpretatione, Aristotle classifies declarative state
ments into affirmation and negation, which respectively affirms or denies observations about an event (Ackrill, 1963). Despite this seeming simplicity, natural language often expresses negation in complex and subtle ways, using diverse syntactic, semantic and pragmatic formulations (Horn and Wansing, 2020). For example, syntactically, different negation determiners (i.e., negation cues) such as no, few and only result in distinct explicit and implicit negative perceptions (Xiang et al., 2016).
Despite their diversity, however, negated language expressions are much less likely to appear in text than affirmative statements (Reitan et al., 2015). Consequently, PLMs, which rely on large-scale textual corpora as training data, are prone to decreased performance when confronted with negated constructions. In machine translation, for example, the presence of negation may heavily affect the quality of produced translations (Fancellu and Webber, 2015; Hossain et al., 2020). In factual knowledge understanding tasks, PLMs memorize positive and negative sentences seen during training, but generalize more poorly to unseen negated instances (Kassner and Schütze, 2020).
Negation in Commonsense Reasoning Understanding negation and oppositional expressions is critical for reasoning about commonsense knowledge, particularly in counterfactual scenarios (Qin et al., 2019). However, negation is rarely explicitly modeled in NLP studies on commonsense reasoning. As a result, in many NLP tasks, these models experience a performance drop when presented with examples exhibiting negated characteristics.
As a case study, the ATOMIC (Sap et al., 2020) knowledge graph encodes social commonsense knowledge about event pre-conditions, event post-conditions, and static attributes in the form of natural language if-then rules. However, despite the fact that ATOMIC provides a rich set of seed events, it comprises an unbalanced set of affirmative events  $(97.9\%)$  and negated events  $(2.1\%)$ . As a result, when systems link to ATOMIC to retrieve relevant social commonsense inferences, they are likely to recover inferences of affirmative events even when searching for negated instances. Furthermore, knowledge models that use this resource (e.g., COMET; Bosselut et al., 2019) are unlikely to learn implicit differences between inferences of affirmative and negated events. When given negated events, these models often produce associations of counterpart affirmative events. For example, for
Types	Example Negation Cues	Example Sentences
Affixes	un-, ir-, non-, il-, im-, -less, etc.	X addresses an irrelevant point
X is unlikely to be a spy
X unsaddles the horse
Single-word	not, no, nothing, nobody, few, little, without, never, hardly, rarely, barely, seldomly, etc.	X does not tell the truth to the public
X never eats ice cream
X went to a movie without his friends
Multi-word	no longer, barely/hardly ever, not at all, a lack of, be deprived of, in the absence of, on no condition, by no means, not by any means, under no circumstances, make no attempt to, etc.	X no longer wants to buy a car
X is not at all impressed by Y's ideas
X under no circumstances smokes
X is by no means cheating on Y
Negative Verbs	oppose, refuse, resist, avoid, disapprove, lack, discontinue, stop, cease, halt, prohibit, forbid, prevent, reject, fail, etc.	X denies the existence of god
X restrains himself from eating with Y
X refuses to be in a relationship
Table 1: Negation cues and examples from ANION.
the negated event, "X opposes racism," COMET infers "X intends to be a racist," an association of the affirmative statement, "X supports racism."
At the heart of this problem is that inferring commonsense knowledge about negations often requires implicit reasoning. In factual knowledge reasoning, applying logical rules over statements can be effective for handling negative queries (Asai and Hajishirzi, 2020; Ren and Leskovec, 2020). However, directly manipulating affirmative forms with logic-guided rules may fail for commonsense reasoning: the boundary of commonsense inferences between affirmative and negated statements is not always wholly contrastive. Many inferences can be relevant to both forms. The events “X puts the potato in the oven” and “X doesn’t put the potato in the oven,” could both have an associated inference: “X wants to make dinner.” The affirmative event clearly implies this inference. For the negated event to be worth mentioning on its own (Grice et al., 1975), an implicit complementary event (e.g., “X puts the potato in the microwave”) would likely hold, which might validate the inference w.r.t. the negated event. To model the defeasibility of commonsense reasoning (Pratt, 1994; Rudinger et al., 2020), modeling both common and contrastive inferences of negated forms is necessary.
3 ANION:Commonsense Inferences of Oppositions and Negations
To provide a rich resource of commonsense inferences for opposition and negation events, we design ANION. Using the same schema as the ATOMIC knowledge graph (Sap et al., 2020), we initialize 22,483 negated forms paired to original ATOMIC events and crowdsourced 627,042 new inferences for these negated events. Consistent with ATOMIC,
ANION is constructed using English formulations of events and inferences. We briefly recap ATOMIC and describe the construction of ANION below.
ATOMIC Background The ATOMIC knowledge graph contains  $\sim 24\mathrm{K}$  base events (e.g., "X plays the piano") with  $877\mathrm{K}$  accompanying social commonsense inferences (e.g., "Before, X needs to buy a piano.") along nine dimensions (e.g.,  $xNeed$ ). The full description of ATOMIC relation types can be found in Table 12 in the Appendix.
3.1 Overview of ANION Construction
Our knowledge construction pipeline consists of two steps. First, we collect negated and contradictive events by deriving oppositions of events in ATOMIC. Inspired by the distinction made between negation contributed by semantic assertion (explicit negation) or non-asserted content (implicit negation) (Xiang et al., 2016), we define three varieties of negated events: logical negations, semi-logical negations, and commonsense contradictions, which we describe in detail below. Logical and semiological negations were heuristically formulated from ATOMIC events. Commonsense contradiction events were crowdsourced from Amazon Mechanical Turk (MTurk). Negated events in ANION are assigned to the same data split as the corresponding affirmative event from which they are derived (e.g., negated events for ATOMIC training set events are found in the ANION training set).
Once a list of negated events is compiled, we crowdsourced inferences of these new events on MTurk using similar annotation templates as Sap et al. (2020). We design qualifying tasks to filter out unreliable workers and screen their answers manually for quality control purposes.
Type	#Words	Total	Train	Development	Test
ATOMIC	event	4.61	25,096	20,322	2,282	2,492
inference	-	795,059	643,571	72,227	79,261
ANION - Logical (L)	event	4.47	8,285	4,175	1,903	2,207
inference	-	225,635	110,864	57,170	57,601
ANION - Semi-logical (S)	event	4.52	5,019	2,457	1,223	1,339
inference	-	138,587	66,087	33,030	39,470
ANION - Commonsense Contradiction (C)	event	4.46	9,179	3,267	2,808	3,104
inference	-	262,820	93,419	95,685	73,716
Logical Negation We define logical negation events as events with the negation cue not added to their original formulation (e.g., "X does not play the piano"). However, different positions of the not modifier in a clause can result in different negation scopes, which can alter the semantics of the event (Council et al., 2010). To be consistent, we systematically insert not after the subject of the event clause. If necessary, we change verb forms and add auxiliary words (e.g., do, does, did, is, was, can, could, would, should, may, might). For quality control, we have human workers validate each logically negated event form and exclude events that annotators identify as uninterpretable or awkwardly worded. For each created event, we then collect the same nine dimensions of inferences as defined in ATOMIC. Consequently, we collected 8,285 logically negated events with  $225\mathrm{K}$  corresponding inferences (as shown in Table 2). Appendix A.1 provides further details of the compilation of logical negation events.
Semi-logical Negation We define semi-logical negation using explicit cues other than not. We categorize these negation cues (words or phrases) into four subtypes: affixes (e.g., legal/illegal), single-word cues (e.g., never), multi-word cues (e.g., no longer), and negative verbs (e.g., refuse). See Table 1 for examples. We create semi-logical negation events by heuristically adding these cues to different positions of ATOMIC events. Similar to logically-negated events, we avoid grammatically incorrect or semantically awkward events by removing auto-generated instances of low quality. The final set of data includes 5,019 semi-logical negation events. We then crowdsourced a total of 138K inferences for these new events. Appendix A.1 provides further details of the compilation of semi-logical negation events.
Table 2: Statistics of ATOMIC and different subsets of ANION (ANION-L + ANION-S + ANION-C).
Event	Commonsense Contradiction
X buys a bicycle	X buys a car
X donates a bicycle
X walks in the door	X stops at the door
X walks out of the building
X works hard all day	X plays games all day
X puts in minimal effort all day
X finishes the story	X starts the story
X stops halfway through the story
X turns the air blue	X secretly curses
X speaks appropriately
Table 3: Contradictions of events from ATOMIC
Commonsense Contradiction We formulate commonsense contradiction as contradictory statements without negation cues. Commonsense contradiction events are not identifiable as negations on their own, but demonstrate reversed semantic or pragmatic meaning when paired with their affirmative counterparts (e.g., "X eats a hamburger" vs. "X eats a salad"). To obtain commonsense contradictions, we crowdsourced two oppositional events for each ATOMIC event, excluding events with blank placeholders representing generic objects, resulting in 40K new commonsense contradiction events. For 9,179 of these events, we crowdsourced an additional 262K commonsense inferences. Appendix A.1 provides further details of the crowdsourcing of commonsense contradiction events.
4 Knowledge Models of Negated Events
ANION can be used as training data for commonsense models to make inferences about negated events. Here, we recap COMET (Bosselut et al., 2019), a commonsense knowledge model, and evaluate how training knowledge models on ANION affects their ability to hypothesize commonsense knowledge for negated and oppositional events.
Eval Set	Train Set	PPL ↓	BL2 ↑	P@10 ↑
ATOMIC	ATOMIC	9.30	14.18	55.18
ATOMIC + ANION	9.28	14.05	*53.61
ANION-L	ATOMIC	10.87	10.86	35.84
ATOMIC + ANION	9.08	11.96	**45.42
ANION-S	ATOMIC	11.69	12.07	36.89
ATOMIC + ANION	9.80	13.22	**46.88
ANION-C	ATOMIC	12.02	14.32	46.70
ATOMIC + ANION	11.20	14.64	**50.65
Table 4: Evaluations of COMET models trained on ATOMIC and ANION KGs. Training on examples of negated events leads to large improvements in the quality of generated inferences with minimal dropoff in the quality of inferences for affirmative events. Single (*) and double asterisks  $(^{**})$  indicate significance at  $p < 0.05$  and  $p < 0.01$ , respectively.
Commonsense transformers (COMET) are generative knowledge models that learn to hypothesize commonsense inferences by training on examples from a knowledge graph. Specifically, COMET receives knowledge tuples in  $\{h,r,t\}$  form during training, where  $h$  is a head entity,  $r$  is a relation type, and  $t$  is a tail entity. The model is trained to maximize the conditional loglikelihood of predicting the tokens of the tail entity  $t$  given the tokens of the head entity  $h$  and relation  $r$ :
$$
\mathcal {L} _ {G} = - \sum \log P (t | h, r) \tag {1}
$$
In ATOMIC and ANION,  $h$  corresponds to events, such as "X has a nightmare,"  $t$  corresponds to commonsense inferences about those events, such as "X wakes up," and  $r$  corresponds to commonsense inference types, such as "As a result, X does...".
Following Bosselut et al. (2019) and Sap et al. (2020), for each event and relation type in ATOMIC, 10 candidate inferences are decoded from COMET using beam search with  $b = 10$ .
As oppositional instances remain challenging to knowledge models such as COMET, we evaluate how ANION can be used to augment the type of examples seen by COMET during training.
Evaluation Metrics Following Bosselut et al. (2019), we evaluate the quality of generated inferences using BLEU-2 (Papineni et al., 2002) as an automatic evaluation. We also compute the perplexity of models on their reference generations.
For the human evaluation, we employ human judges from MTurk to identify whether generated commonsense inferences are plausible. We randomly sample 100 events from the original ATOMIC test set along with their negated counterparts from ANION. For each event, we present every decoded inference to five crowdworkers and ask them to identify whether the inference is plausible given the event. For each model trained on a different combination of ATOMIC and ANION (i.e., ANION-L, ANION-S, ANION-C), we evaluate the same events for comparison. We calculate Precision @ 10 (P@10) across these human ratings, i.e., the average number of correct options per event-relation prompt. Specifically, we average the results from 45K ratings to compute the final human score (100 events × 9 relations × 10 options × 5 annotators). The pairwise agreement score of human evaluation is 63.6, which is on par with other similar commonsense reasoning annotation tasks (Rashkin et al., 2016).
Does negated event training improve commonsense inference for negated situations? We train a COMET model on the events from ATOMIC (i.e., COMET-ATOMIC), and another on the examples from both ATOMIC and ANION (i.e., COMET-FULL). The combined dataset is shuffled so that the original and negated examples are uniformly mixed during training.
We report our comparison of these two models in Table 4. The performance of the original COMET model trained only on the ATOMIC knowledge graph drops significantly across all types of oppositional instances. Most surprisingly, a drop in performance is also observed on commonsense contradictions (ANION-C), which have no explicit negation cues. However, commonsense contradiction events can often be richer in content (see Table 3), making them more challenging for knowledge models. Meanwhile training on all negated examples in the ANION knowledge graph produces significant improvements across all negation categories (ANION-{L,S,C}), though we do observe a slight drop in human ratings on the examples from the original ATOMIC test set.
Does negated event training deteriorate commonsense inference of affirmative situations?
We note in Table 4 that training on ATOMIC + ANION hurts inference performance on the original ATOMIC evaluation set. To analyze why COMET-
FULL does not improve on this set of examples, we perform a case study on inferences generated by COMET-ATOMIC and COMET-FULL under the same event and relation prompt, and note two qualitative patterns.
First, we observe that COMET-FULL tends to generate inferences that are less generic, but that may require additional implicit context. For example, for the event "X is really sad" and the relation  $xEffect$  (i.e., the effect of the event on X), COMET-ATOMIC generates inferences such as "cries," "gets depressed" and "takes medication." Conversely, COMET-FULL generates context-specific inferences such as "thinks about the past" and "thinks about what they did," which, while plausible in some context, may be less straightforward when evaluated broadly (not all feelings of sadness lead to reflection on the past or one's own actions).
Second, we find an overall improvement for certain compositional events in ATOMIC that contain conjunction words: "and" or "but." On these examples, COMET-FULL outperforms COMET-ATOMIC with 12.41 and 12.22 BLEU-2 scores respectively. For example, for the event "X is hot and humid" and the relation  $xEffect$ , COMET-ATOMIC's generation includes correct inferences, such as "to take a shower," "to cool down," "to drink some water," "to go outside," and incorrect inferences, such as "to turn on the heat" and "to drink a hot tea." COMET-FULL generates all of COMET-ATOMIC's correct inferences, but none of the incorrect inferences, demonstrating that training COMET jointly on ATOMIC and ANION can help avoid incorrect inferences involving commonsense mismatch in more compositional situations.
In summary, the ability to generate richer, contextual inferences for COMET-FULL is beneficial when handling complex events, but may not be necessary for many of the simple events in ATOMIC, and may backfire when subtler inferences are made.
Which variety of negated events are most crucial to include in training sets? As ablations, we train additional models using different subsets of ANION: logical negations (ATOMIC + ANION-L), semi-logical negations (ATOMIC + ANION-S), and commonsense contradictions (ATOMIC + ANION-C). These ablations evaluate whether knowledge models can adapt to certain types of negation more efficiently with additional data.
In Table 5, we show that training with examples of each negation type improves performance
Eval Set	Train Set	PPL ↓	BL2 ↑	P@10 ↑
ATOMIC	ATOMIC	9.30	14.18	55.18
+ ANION-L	9.27	14.20	**58.11
+ ANION-S	9.30	14.09	55.74
+ ANION-C	9.29	14.10	**52.22
ANION-L	ATOMIC	10.87	10.86	35.84
+ ANION-L	9.28	11.94	**44.94
+ ANION-S	9.93	11.29	**44.01
+ ANION-C	10.34	11.04	**42.33
ANION-S	ATOMIC	11.69	12.07	36.89
+ ANION-L	10.69	12.69	**42.38
+ ANION-S	10.23	12.79	**45.50
+ ANION-C	10.95	12.35	**41.76
ANION-C	ATOMIC	12.02	14.32	46.70
+ ANION-L	11.72	14.43	47.78
+ ANION-S	11.67	14.34	46.09
+ ANION-C	11.50	14.58	**48.79
Table 5: Ablation results of models trained and evaluated on different portions of ANION. The best result on each subset of ANION comes from training on similar examples. The model trained on negated events from ANION-L performs the best at generating inferences for the original ATOMIC events. Double asterisks  $(^{**})$  indicate significance at  $p < 0.01$ .