Venue: NAACL 2021

# "I'm Not Mad":

 Commonsense Implications of Negation and Contradiction

Liwei Jiang1 Antoine Bosselut2,3 Chandra Bhagavatula2 Yejin Choi1,2

###### Abstract

Natural language inference requires reasoning about contradictions, negations, and their commonsense implications. Given a simple premise (e.g., "I'm mad at you"), humans can reason about the varying shades of contradictory statements ranging from straightforward negations ("I'm not mat at you") to commonsense contradictions ("I'm happy"). Moreover, these negated or contradictory statements shift the commonsense implications of the original premise in nontrivial ways. For example, while "I'm mad" implies "I'm unhappy about something," negating the premise (_i.e._, "I'm _not_ mud") does not necessarily negate the corresponding commonsense implications.

In this paper, we present the first comprehensive study focusing on commonsense implications of negated statements and contradications. We introduce Anion1, a new commonsense knowledge graph with 624K _if-then_ rules focusing on negated and contradictory events. We then present joint generative and discriminative inference models for this new resource, providing novel empirical insights on how logical negations and commonsense contradictions reshape the commonsense implications of their original premises.

Footnote 1: Data and code available at [https://github.com/liweijiang/anion](https://github.com/liweijiang/anion)

+
Footnote †: Equal G. Allen School of Computer Science & Engineering, University of Washington

\({}^{2}\)Allen Institute for Artificial Intelligence

\({}^{3}\)Stanford University

{lwjiang,yejin}@cs.washington.edu, {chandrab}@allenai.org

## 1 Introduction

Humans reason about underlying causes and effects of events described in text. For example, in Figure 1, the event "X wears a mask" is associated with many causal inferences such as "X is seen as responsible," or "Others get protected." Hypothesizing and reasoning about commonsense inferences is used for understanding complex situations encountered in everyday life Sap et al. (2019); Bisk et al. (2020); Bhagavatula et al. (2020); Sakaguchi et al. (2020). This ability eludes AI systems, and has motivated the design of a wealth of commonsense knowledge resources, such as Cyc Lenat (1995), ConceptNet Speer et al. (2017), and AtomicSap et al. (2020); Hwang et al. (2020), to provide structured reasoning capabilities to AI systems Lin et al. (2019); Feng et al. (2020).

However, reasoning about negated observations remains a challenge Hossain et al. (2020). While negation is often considered a poorer form of meaning than affirmation2Ackrill (1963); Horn and Wansing (2020), negated statements can still imply expressive commonsense inferences. In Figure 1, the negated event "X doesn't wear a mask,"

Figure 1: Commonsense inferences for the event “X wears a mask,” its logical negation and commonsense contradiction events, and their associated inferences.



is connected to rich commonsense inferences, despite describing the _absence_ of action. However, negated observations are rarely found in common-sense knowledge resources. For example, negated examples make up only \(\sim\)3% of examples in the ConceptNet knowledge graph Li et al. (2016).

This scarcity poses downstream issues for systems that must understand negated situations. Commonsense knowledge models Bosselut et al. (2019); Hwang et al. (2020) trained on resources of largely affirmative instances struggle particularly with negation examples. Their ability to hypothesize inferences for negated events is 35% lower than for affirmative events (SS4.2). Furthermore, since negated statements are asymmetrically mentioned in text compared to affirmative statements Jowett et al. (1892); Horn and Wansing (2020), large-scale pretraining does not implicitly learn negation scoping Kim et al. (2019). As a result, when presented with negated concepts, pretrained neural language models (PLMs) often exhibit the same associations as affirmative statements Kassner et al. (2020). Motivated by these observations, our work focuses on improving the ability of knowledge models to make commonsense inferences about events that convey denial, rejection or contradiction of actions.

We define our contributions as follows. First, we crowdsource a new large scale resource, Array of commonsNe**N**e**nIferences for **O**ppositions and **Negations** (Anion), which contains inferences for different types of negated events.This new resource can be used to train knowledge models on commonsense inferences associated with the absence of actions. Second, we propose a new class of negation discriminators that can be applied to generated commonsense inferences. These discriminators partition inferences based on logical consistency, thereby mitigating the effects of common affirmative associations that violate negation constraints. Discriminators are trained using contrastive samples from paired affirmative and negated events in Anion. Finally, we conduct an empirical study of both of these techniques and show that using training- and discriminator-based approaches for modeling negation cuts the performance difference between affirmative and negated events by 73% - 85% depending on the negation variety.

## 2 Commonsense Negation

Negation in LanguageIn _Categories_ and _De Interpretation_, Aristotle classifies declarative statements into affirmation and negation, which respectively affirms or denies observations about an event Ackrill (1963). Despite this seeming simplicity, natural language often expresses negation in complex and subtle ways, using diverse syntactic, semantic and pragmatic formulations Horn and Wansing (2020). For example, syntactically, different negation determiners (_i.e._, negation cues) such as _no_, _few_ and _only_ result in distinct explicit and implicit negative perceptions Xiang et al. (2016).

Despite their diversity, however, negated language expressions are much less likely to appear in text than affirmative statements Reitan et al. (2015). Consequently, PLMs, which rely on large-scale textual corpora as training data, are prone to decreased performance when confronted with negated constructions. In machine translation, for example, the presence of negation may heavily affect the quality of produced translations Fancellu and Webber (2015); Hossain et al. (2020). In factual knowledge understanding tasks, PLMs memorize positive and negative sentences seen during training, but generalize more poorly to unseen negated instances Kassner and Schutze (2020).

Negation in Commonsense ReasoningUnderstanding negation and oppositional expressions is critical for reasoning about commonsense knowledge, particularly in counterfactual scenarios Qin et al. (2019). However, negation is rarely explicitly modeled in NLP studies on commonsense reasoning. As a result, in many NLP tasks, these models experience a performance drop when presented with examples exhibiting negated characteristics.

As a case study, the AtomicSap et al. (2020) knowledge graph encodes social commonsense knowledge about event pre-conditions, event post-conditions, and static attributes in the form of natural language _if-then_ rules. However, despite the fact that Atomic provides a rich set of seed events, it comprises an unbalanced set of affirmative events (97.9%) and negated events (2.1%). As a result, when systems link to Atomic to retrieve relevant social commonsense inferences, they are likely to recover inferences of affirmative events even when searching for negated instances. Furthermore, knowledge models that use this resource (_e.g._, COMET; Bosselut et al. (2019) are unlikely to learn implicit differences between inferences of affirmative and negated events. When given negated events, these models often produce associations of counterpart affirmative events. For example, for 

the negated event, "X opposes racism," COMET infers "X intends to be a racist," an association of the affirmative statement, "X supports racism."

At the heart of this problem is that inferring commonsense knowledge about negations often requires implicit reasoning. In factual knowledge reasoning, applying logical rules over statements can be effective for handling negative queries Asai and Hajishirzi (2020); Ren and Leskovec (2020). However, directly manipulating affirmative forms with logic-guided rules may fail for commonsense reasoning: the boundary of commonsense inferences between affirmative and negated statements is not always wholly contrastive. Many inferences can be relevant to both forms. The events "X puts the potato in the oven" and "X doesn't put the potato in the oven," could both have an associated inference: "X wants to make dinner." The affirmative event clearly implies this inference. For the negated event to be worth mentioning on its own Grice et al. (1975), an implicit complementary event (_e.g._, "X puts the potato in the microwave") would likely hold, which might validate the inference w.r.t. the negated event. To model the defeasibility of commonsense reasoning Pratt (1994); Rudinger et al. (2020), modeling both common and contrastive inferences of negated forms is necessary.

## 3 Anion: Commonsense Inferences of Oppositions and Negations

To provide a rich resource of commonsense inferences for opposition and negation events, we design Anion. Using the same schema as the Atomic knowledge graph Sap et al. (2020), we initialize 22,483 negated forms paired to original Atomic events and crowdsource 627,042 new inferences for these negated events. Consistent with Atomic, Anion is constructed using English formulations of events and inferences. We briefly recap Atomic and describe the construction of Anion below.

A atomic BackgroundThe Atomic knowledge graph contains \({\sim}24\)K base events (_e.g._, "X plays the piano") with 877K accompanying social commonsense inferences (_e.g._, "Before, X needs to buy a piano.") along nine dimensions (_e.g._, _xNeed_). The full description of Atomic relation types can be found in Table 12 in the Appendix.

### Overview of Anion Construction

Our knowledge construction pipeline consists of two steps. First, we collect negated and contradictive events by deriving oppositions of events in Atomic. Inspired by the distinction made between negation contributed by semantic assertion (explicit negation) or non-asserted content (implicit negation) Xiang et al. (2016), we define three varieties of negated events: logical negations, semi-logical negations, and commonsense contradictions, which we describe in detail below. Logical and semilogical negations were heuristically formulated from Atomic events. Commonsense contradiction events were crowdsourced from Amazon Mechanical Turk (MTurk). Negated events in Anion are assigned to the same data split as the corresponding affirmative event from which they are derived (_e.g._, negated events for Atomic training set events are found in the Anion training set).

Once a list of negated events is compiled, we crowdsource inferences of these new events on MTurk using similar annotation templates as Sap et al. (2020). We design qualifying tasks to filter out unreliable workers and screen their answers manually for quality control purposes.

\begin{table}
\begin{tabular}{l l p{142.3pt}}
**Types** & **Example Negation Cues** & **Example Sentences** \\ \hline Affixes & un-, ir-, non-, il-, im-, -less, etc. & X addresses an _irre_elevant point \\  & & X is _un_likely to be a spy \\ Single-word & not, no, nothing, nobody, few, little, without, never, hardly, rarely, barely, seldomly, etc. & X _nus_addles the horse \\ Multi-word & no longer, bareyly/hardly ever, not at all, a lack of, be deprived of, in the absence of, on no condition, by no means, not by any means, under no circumstances, make no attempt to, etc. & X does _not_ tell the truth to the public \\ Negative Verbs & oppose, refuse, resist, avoid, disapproprove, lack, discontinuue, stop, cease, halt, prohibit, forbid, prevent, reject, fail, etc. & X _no longer_ wants to buy a car \\  & & X is _not at all_ impressed by Y’s ideas \\ Negative Verbs & oppose, refuse, resist, avoid, disapproprove, lack, discontinuue, stop, cease, halt, prohibit, forbid, prevent, reject, fail, etc. & X _under no circumstances_ smokes \\ \end{tabular}
\end{table}
Table 1: Negation cues and examples from Anion.

 

Logical NegationWe define logical negation events as events with the negation cue _not_ added to their original formulation (_e.g._, "X does not play the piano"). However, different positions of the _not_ modifier in a clause can result in different _negation scopes_, which can alter the semantics of the event [10]. To be consistent, we systematically insert _not_ after the subject of the event clause. If necessary, we change verb forms and add auxiliary words (_e.g._, do, does, did, is, was, can, could, would, should, may, might). For quality control, we have human workers validate each logically negated event form and exclude events that annotators identify as uninterpretable or awkwardly worded. For each created event, we then collect the same nine dimensions of inferences as defined in Atomic. Consequently, we collected 8,285 logically negated events with 225K corresponding inferences (as shown in Table 2). Appendix A.1 provides further details of the compilation of logical negation events.

Semi-logical NegationWe define semi-logical negation using explicit cues other than _not_. We categorize these negation cues (words or phrases) into four subtypes: affixes (_e.g._, legal/illegal), single-word cues (_e.g._, never), multi-word cues (_e.g._, no longer), and negative verbs (_e.g._, refuse). See Table 1 for examples. We create semi-logical negation events by heuristically adding these cues to different positions of Atomic events. Similar to logically-negated events, we avoid grammatically incorrect or semantically awkward events by removing auto-generated instances of low quality. The final set of data includes 5,019 semi-logical negation events. We then crowdsource a total of 138K inferences for these new events. Appendix A.1 provides further details of the compilation of semi-logical negation events.

Commonsense ContradictionWe formulate commonsense contradiction as contradictory statements without negation cues. Commonsense contradiction events are not identifiable as negations on their own, but demonstrate reversed semantic or pragmatic meaning when paired with their affirmative counterparts (_e.g._, "X eats a hamburger" vs. "X eats a salad"). To obtain commonsense contradictions, we crowdsource two oppositional events for each Atomic event, excluding events with blank placeholders representing generic objects, resulting in 40K new commonsense contradiction events. For 9,179 of these events, we crowdsource an additional 262K commonsense inferences. Appendix A.1 provides further details of the crowd-sourcing of commonsense contradiction events.

## 4 Knowledge Models of Negated Events

Anion can be used as training data for commonsense models to make inferences about negated events. Here, we recap COMET [20], a commonsense knowledge model, and evaluate how training knowledge models on Anion affects their ability to hypothesize commonsense knowledge for negated and oppositional events.

\begin{table}
\begin{tabular}{l|r r r r r r} \hline \multicolumn{1}{c|}{**Type**} & \multicolumn{3}{c}{\#**Words**} & \multicolumn{1}{c}{Total} & \multicolumn{1}{c}{Train} & \multicolumn{1}{c}{Development} & \multicolumn{1}{c}{Test} \\ \hline Atomic & event & 4.61 & 25,096 & 20,322 & 2,282 & 2,492 \\  & inference & - & 795,059 & 643,571 & 72,227 & 79,261 \\ \hline \multirow{2}{*}{Anion - Logical (L)} & event & 4.47 & 8,285 & 4,175 & 1,903 & 2,207 \\  & inference & - & 225,635 & 110,864 & 57,170 & 57,601 \\ \hline \multirow{2}{*}{Anion - Semi-logical (S)} & event & 4.52 & 5,019 & 2,457 & 1,223 & 1,339 \\  & inference & - & 138,587 & 66,087 & 33,030 & 39,470 \\ \hline \multirow{2}{*}{Anion - Commonsense Contradiction (C)} & event & 4.46 & 9,179 & 3,267 & 2,808 & 3,104 \\  & inference & - & 262,820 & 93,419 & 95,685 & 73,716 \\ \hline \end{tabular}
\end{table}
Table 2: Statistics of Atomic and different subsets of Anion (Anion-L + Anion-S + Anion-C).

\begin{table}
\begin{tabular}{l l} \multicolumn{1}{p{123.8pt}}{**Event**} & \multicolumn{1}{c}{**Commonsense Contradiction**} \\ \hline X buys a bicycle & \begin{tabular}{l} X buys a car \\ X donates a bicycle \\ \end{tabular} \\ \hline X walks in the door & \begin{tabular}{l} X stops at the door \\ X walks out of the building \\ \end{tabular} \\ \hline X works hard all day & \begin{tabular}{l} X plays games all day \\ X puts in minimal effort all day \\ \end{tabular} \\ \hline X finishes the story & \begin{tabular}{l} X starts the story \\ X stops halfway through the story \\ \end{tabular} \\ \hline X turns the air blue & 
\begin{tabular}{l} X secretly curses \\ X speaks appropriately \\ \end{tabular} \\ \hline \end{tabular}
\end{table}
Table 3: Contradictions of events from Atomic 



### Setup

Commonsense transformers (COMET) are generative knowledge models that learn to hypothesize commonsense inferences by training on examples from a knowledge graph. Specifically, COMET receives knowledge tuples in \(\{h,r,t\}\) form during training, where \(h\) is a head entity, \(r\) is a relation type, and \(t\) is a tail entity. The model is trained to maximize the conditional loglikelihood of predicting the tokens of the tail entity \(t\) given the tokens of the head entity \(h\) and relation \(r\):

\[\mathcal{L}_{G}=-\sum\log P(t|h,r) \tag{1}\]

In Atomic and Anion, \(h\) corresponds to events, such as "X has a nightmare," \(t\) corresponds to commonsense inferences about those events, such as "X wakes up," and \(r\) corresponds to commonsense inference types, such as "As a result, X does...".

Following Bosselut et al. (2019) and Sap et al. (2020), for each event and relation type in Atomic, 10 candidate inferences are decoded from COMET using beam search with \(b\)=10.

### Experiments

As oppositional instances remain challenging to knowledge models such as COMET, we evaluate how Anion can be used to augment the type of examples seen by COMET during training.

Evaluation MetricsFollowing Bosselt et al. (2019), we evaluate the quality of generated inferences using BLEU-2 (Papineni et al., 2002) as an automatic evaluation. We also compute the perplexity of models on their reference generations.

For the human evaluation, we employ human judges from MTurk to identify whether generated commonsense inferences are plausible. We randomly sample 100 events from the original Atomic test set along with their negated counterparts from Anion. For each event, we present every decoded inference to five crowdworkers and ask them to identify whether the inference is plausible given the event. For each model trained on a different combination of Atomic and Anion (_i.e._, Anion-L, Anion-S, Anion-C), we evaluate the same events for comparison. We calculate Precision @ 10 (P@10) across these human ratings, _i.e._, the average number of correct options per event-relation prompt. Specifically, we average the results from 45K ratings to compute the final human score (100 events \(\times\) 9 relations \(\times\) 10 options \(\times\) 5 annotators). The pairwise agreement score of human evaluation is 63.6, which is on par with other similar commonsense reasoning annotation tasks Rashkin et al. (2016).

Does negated event training improve commonsense inference for negated situations?We train a COMET model on the events from Atomic (_i.e._, COMET-Atomic), and another on the examples from both Atomic and Anion (_i.e._, COMET-Full). The combined dataset is shuffled so that the original and negated examples are uniformly mixed during training.

We report our comparison of these two models in Table 4. The performance of the original COMET model trained only on the Atomic knowledge graph drops significantly across all types of propositional instances. Most surprisingly, a drop in performance is also observed on commonsense contradictions (Anion-C), which have no explicit negation cues. However, commonsense contradiction events can often be richer in content (see Table 3), making them more challenging for knowledge models. Meanwhile training on all negated examples in the Anion knowledge graph produces significant improvements across all negation categories (Anion-{L,S,C}), though we do observe a slight drop in human ratings on the examples from the original Atomic test set.

Does negated event training deteriorate commonsense inference of affirmative situations?We note in Table 4 that training on Atomic + Anion hurts inference performance on the original Atomic evaluation set. To analyze why COMET

\begin{table}
\begin{tabular}{l|c c r r} \hline \hline
**Eval Set** & **Train Set** & **PPL \(\downarrow\)** & **BL2 \(\uparrow\)** & **P@10 \(\uparrow\)** \\ \hline \multirow{2}{*}{atomic} & Atomic & 9.30 & **14.18** & **55.18** \\  & Atomic + Anion & **9.28** & 14.05 & *53.61 \\ \hline \multirow{2}{*}{Anion-L} & Atomic & 10.87 & 10.86 & 35.84 \\  & Atomic + Anion & **9.08** & **11.96** & ****45.42** \\ \hline \multirow{2}{*}{Anion-S} & Atomic & 11.69 & 12.07 & 36.89 \\  & Atomic + Anion & **9.80** & **13.22** & **46.88** \\ \hline \multirow{2}{*}{Anion-C} & Atomic & 12.02 & 14.32 & 46.70 \\  & Atomic + Anion & **11.20** & **14.64** & ****50.65** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Evaluations of COMET models trained on Atomic and Anion KGs. Training on examples of negated events leads to large improvements in the quality of generated inferences with minimal dropoff in the quality of inferences for affirmative events. Single (*) and double asterisks (**) indicate significance at p<0.05 and p<0.01, respectively.



Full does not improve on this set of examples, we perform a case study on inferences generated by COMET-Atomic and COMET-Full under the same event and relation prompt, and note two qualitative patterns.

First, we observe that COMET-Full tends to generate inferences that are less generic, but that may require additional implicit context. For example, for the event "X is really sad" and the relation _xEffect_ (_i.e._, the effect of the event on X), COMET-Atomic generates inferences such as "creies," "gets depressed" and "takes medication." Conversely, COMET-Full generates context-specific inferences such as "thinks about the past" and "thinks about what they did," which, while plausible in some context, may be less straightforward when evaluated broadly (not all feelings of sadness lead to reflection on the past or one's own actions).

Second, we find an overall improvement for certain compositional events in Atomic that contain conjunction words: "and" or "but." On these examples, COMET-Full outperforms COMET-Atomic with 12.41 and 12.22 BLEU-2 scores respectively. For example, for the event "X is hot and humid" and the relation _xEffect_, COMET-Atomic's generation includes correct inferences, such as "to take a shower," "too cool down," "to drink some water," "to go outside," and incorrect inferences, such as "to turn on the heat" and "to drink a hot tea." COMET-Full generates all of COMET-Atomic's correct inferences, but none of the incorrect inferences, demonstrating that training COMET jointly on Atomic and Anion can help avoid incorrect inferences involving commonsense mismatch in more compositional situations.

In summary, the ability to generate richer, contextual inferences for COMET-Full is beneficial when handling complex events, but may not be necessary for many of the simple events in Atomic, and may backfire when subtler inferences are made.

Which variety of negated events are most crucial to include in training sets?As ablations, we train additional models using different subsets of Anion: logical negations (Atomic + AnionL), semi-logical negations (Atomic + AnionS), and commonsense contradictions (Atomic + Anion-C). These ablations evaluate whether knowledge models can adapt to certain types of negation more efficiently with additional data.

In Table 5, we show that training with examples of each negation type improves performance on the evaluation set related to that negation type. Interestingly, though, training on certain types of negation examples can also yield benefits downstream on other negation types. For example, training on commonsense contradictions (Anion-C) provides a clear benefit when evaluating on semilogically negated events (Anion-S) as opposed to merely training on Atomic. Notably, the knowledge model trained with logically negated examples (Atomic + Anion-L) outperforms the model trained only on Atomic on all test sets.

## 5 Discriminating Inconsistent Inferences

While training on examples of negated events helps knowledge models generate commonsense inferences for these event types, there is still a large gap compared to their performance on affirmative events. To address this discrepancy, we introduce a discriminator-based approach for distinguishing inconsistent inferences of negated events. Our inference discriminator learns to identify plausible and invalid inferences of events by learning from contrastive samples from Atomic and Anion.

### Experimental Setup

We fine-tune the RoBERTa-base model Liu et al. (2019) as a binary classifier to identify whether a given knowledge tuple \(\{h,r,t\}\) is logically valid. The model is trained on paired original and negated

\begin{table}
\begin{tabular}{l|c c r r}
**Eval Set** & **Train Set** & **PPL \(\downarrow\)** & **BL2 \(\uparrow\)** & **P@10 \(\uparrow\)** \\ \hline \multirow{3}{*}{Atomic} & Atomic & 9.30 & 14.18 & 55.18 \\  & + Anion-L & **9.27** & **14.20** & **58.11** \\  & + Anion-S & 9.30 & 14.09 & 55.74 \\  & + Anion-C & 9.29 & 14.10 & **55.22** \\ \hline \multirow{6}{*}{Anion-L} & Atomic & 10.87 & 10.86 & 35.84 \\  & + Anion-L & **9.28** & **11.94** & **44.94** \\  & + Anion-S & 9.93 & 11.29 & **44.01 \\  & + Anion-C & 10.34 & 11.04 & **42.33** \\ \hline \multirow{6}{*}{Anion-S} & Atomic & 11.69 & 12.07 & 36.89 \\  & + Anion-L & 10.69 & 12.69 & **42.38 \\  & + Anion-S & **10.23** & **12.79** & **45.50** \\  & + Anion-C & 10.95 & 12.35 & **41.76** \\ \hline \multirow{6}{*}{Anion-C} & Atomic & 12.02 & 14.32 & 46.70 \\  & + Anion-L & 11.72 & 14.43 & 47.78 \\  & + Anion-S & 11.67 & 14.34 & 46.09 \\ \cline{1-1}  & + Anion-C & **11.50** & **14.58** & **48.79** \\ \hline \end{tabular}
\end{table}
Table 5: Ablation results of models trained and evaluated on different portions of Anion. The best result on each subset of Anion comes from training on similar examples. The model trained on negated events from Anion-L performs the best at generating inferences for the original Atomic events. Double asterisks (**) indicate significance at p<0.01.

 

events as described below. Such training examples inject implicit commonsense nuances that differ between oppositional events to teach the discriminator to identify logical pitfalls. Training details for discriminators can be found in Appendix A.3.

DataThe paired events used to train the negation discriminator are automatically constructed from the Atomic and Anion knowledge graphs. Positive examples can be constructed by sampling tuples from each knowledge graph. To construct negative training samples, we introduce the concept of _common_ and _contrast_ sets among inferences of events and their oppositions.

Common and contrast sets distinguish how commonsense inferences are not necessarily negated in the same manner as their corresponding events. While certain inferences of events are also in opposition to a negated event, some may be common. For the events "X cats a cheeseburger" and "X cats a salad," an inference such as "X is hungry" might be common to both events while inferences such as "X is unhealthy" or "X is healthy" would be viewed as contrastive.

Specifically, we assume two head events in Atomic and Anion, and their respective set of tail inferences regarding a common relation type. We define the common set of these inferences as the intersection of the two sets of tail inferences connected to each head event by applying the exact match of string forms. The contrast set is formed by distinct tail inferences connected to the two head events. Logically valid (_i.e._, positive) training examples consist of knowledge tuples from Atomic and Anion. Logically invalid (_i.e._, negative) training examples are formed by swapping the set of contrast set inferences between paired original and negated events.3

Footnote 3: We note that annotations in Atomic and Anion are finite (_i.e._, not covering the full space of possible commonsense inferences about events). As a result, it is possible that in a more expansive annotation, elements of the contrast sets would in fact be part of the common set of an event and its negation. For the purpose of this work, however, contrast sets were an efficient way of acquiring high-quality semantically negative examples for training discriminators.

To balance the training set, we sample the same number of positive and negative tuples for original and negation events. Statistics of the resulting training sets are in Table 6.

### Experiments

Using different portions of Anion for training yields four unique discriminators (_i.e._, **L**, **S**, **C** and **LSC**) that we apply to commonsense inferences generated by COMET. The discriminators classify each option as either logically _valid_ or _invalid_, partitioning the candidates into two sets, which we evaluate with human judgements. As a baseline, we also record the precision of not using a discriminator, which assumes all generated inferences are valid candidates (_i.e._, the _all_ set).

MetricsWe evaluate and compare the quality of the _all_, _valid_ and _invalid_ sets using BLEU-2 and the same human evaluation as in SS4. The _all_ set contains the full set of 10 candidates, while the _valid_ and _invalid_ sets have varying number of elements depending on how discriminators classify them, summing to 10. To compute statistical significance between valid and all sets, we use a permutation test with 100K permutations. Details are provided in Appendix A.4.

Do discriminators effectively distinguish inconsistent inferences?The results in Table 7 demonstrate that the discriminator trained on all subsets of Anion (**LSC**) can select subsets of inferences (_i.e._,

\begin{table}
\begin{tabular}{l|c|c}
**Discriminator** & **Train Set** & **Size** \\ \hline Logical Negation **(L)** & Anion-L & 324,843 \\ Semi-logical Negation **(S)** & Anion-S & 194,732 \\ Commonsense Contradiction **(C)** & Anion-C & 276,272 \\ All Oppositional Data **(LSC)** & Anion & 795,845 \\ \end{tabular}
\end{table}
Table 6: Statistics of data used to train negation discriminators.

\begin{table}
\begin{tabular}{l|l l c r}
**Eval Set** & & \multicolumn{1}{r}{\#} & **BL2\textbackslash{}**P@\textbackslash{}k\textbackslash{}\uparrow\)** \\ \hline  & all & 10.0 & 14.18 & 55.18 \\ Atomic & valid & 6.3 & **14.24** & **59.07** \\  & invalid & 3.7 & 13.93 & 44.10 \\ \hline  & all & 10.0 & 10.86 & 35.84 \\ Anion-L & valid & 5.6 & **11.33** & **45.59** \\  & invalid & 4.4 & 10.13 & 25.96 \\ \hline  & all & 10.0 & 12.07 & 36.89 \\ Anion-S & valid & 6.3 & **12.63** & **44.93** \\  & invalid & 3.7 & 11.32 & 27.83 \\ \hline  & all & 10.0 & 14.32 & 46.70 \\ Anion-C & valid & 5.9 & **14.78** & **51.45** \\  & invalid & 4.1 & 13.56 & 37.33 \\ \hline \end{tabular}
\end{table}
Table 7: The evaluation of the _all_, _valid_ and _invalid_ sets of inferences generated by COMET-Atomic as partitioned by the **LSC** discriminator. **P@\textbackslash{}k** corresponds to the human-rated precision of a set. \(k\) is the number of elements in _all_, _valid_, or _invalid_ set. For the _valid_ set, higher **P@\textbackslash{}k** is better (_i.e._, more valid inferences are being partitioned). For the _invalid_ set, lower **P@\textbackslash{}k** is better (_i.e._, fewer valid inferences are being included).

 

the _valid_ set) that are more logically consistent with their seed event. This observation holds across all evaluation subsets of Anion, as well as the original Atomic evaluation set. Table 8 shows examples of _valid_ and _invalid_ candidates for negated and contradicted events from Anion as specified by the **LSC** discriminator. The discriminator is notably good at identifying invalid inferences wrongly associated to corresponding affirmative events (_e.g._, "athletic" and "careless" for the event "X does not skate around" under the relation, _xAttr_).

However, this analysis leaves open the possibility that we are generating too many inferences for each event, but that the decoder could rank correct inferences higher among the full set of generated candidates. To evaluate this possibility, we count the number of elements in the _valid_ sets for each example and only keep the same number of the top-scoring elements from the _all_ set (s scored using generation perplexity). In Table 9, we see the average precision score for the pruned _all_ sets (P@{# valid}) still underperforms the precision of their corresponding _valid_ sets.

Which negation categories are most important to provide a discriminator for?To examine the generalization effects of each negation type, we also train discriminators on a single negation subset of Anion examples (_i.e._, **L**, **S**, **C**) and compare the P@{# valid} score of the _all_ and _valid_ sets. Results in Table 9 indicate that each discriminator is best for identifying valid inferences for the types of events on which it was trained. The **L**, **S**, and **C** discriminators all achieve improvements when partitioning events similar to their training. However, the **LSC** discriminator trained on all negation forms shows the largest _valid_ set improvement across all discriminators on Atomic, Anion-S, and Anion-C. On Anion-L, the **LSC** discrimina 

tor still yields a significantly improved _valid_ set.

## 6 Discussion

Are learning-based and discriminator-based approaches complementary?We apply our discriminators to the generations of the COMET model trained on Anion. In Table 10, we see that the **LSC** discriminator, when applied to generations of COMET trained on Anion, achieves significant improvements over all evaluation sets, including the original events. The full evaluation of the P@{# valid} and P@3 scores of applying different discriminators to generations of COMET trained on different data over all evaluation sets are shown in Table 13 and 14 in Appendix A.

Can discriminators be used to more aggressively generate inferences?While applying discriminators to generated inferences yields a _valid_ subset with higher accuracy, we are left with fewer correct inferences in total. Thus, we investigate the efficiency of using discriminators to expand the number of inferences generated. We decode inferences from COMET with beam size 25, and then apply the discriminator to this larger candidate set.

Table 11 shows that for logical negation, the _valid_ set of beam 25 has higher accuracy and more correct options than the _all_ set of beam 10. Thus, when we have a larger and potentially more noisy set of candidates, applying the negation discriminator yields a set of options that have higher quality than using all the candidates from a smaller set of initial generations.

## 7 Conclusion

We present the first comprehensive study on commonsense implications of negations and contradictions. To expand commonsense resources for the challenge of negation modeling, we introduce Anion, a large scale commonsense knowledge graph for negated and contradicted events. We use Anion to train commonsense knowledge models and demonstrate that it effectively enriches machine commonsense inference capabilities around negation. Lastly, we propose a negation discriminator capable of identifying logical flaws in commonsense inferences. By combining the model trained on Anion with the negation discriminator, we achieve a further performance boost.

### Ethical Considerations

Anion Language Choice and Implications

We select English as the base language of Anion so that our resource may be directly linked with the original Atomic knowledge graph. We acknowledge, however, that resources in English are more likely to reflect the mindsets and behaviors of English speakers. Furthermore, and in our case specifically, our annotators were primarily from the US. Consequently, this language choice biases the content of the knowledge graph toward North American perspectives, which affects what models trained on these resources would learn about social norms Acharya et al. (2021). Future works may also include other languages and cultures to make the Anion resource more culturally and ideologically inclusive.

Crowdworker Recruitment, Quality Control and Remuneration

We recruit crowdworkers from MTurk who are located within the US with HIT approval rates higher

\begin{table}
\begin{tabular}{c|r r r r r} \hline \multicolumn{1}{l|}{**EvalDisc**} & \multicolumn{1}{c}{**L**} & \multicolumn{1}{c}{**S**} & \multicolumn{1}{c}{**C**} & \multicolumn{1}{c}{**LSC**} \\ \hline \multirow{3}{*}{Atomic} & all & 54.16 & 54.49 & 55.03 & 55.68 \\  & valid & **54.20** & **54.64** & **55.71** & **+=57.58** \\  & \%iprv & 0.08 & 0.28 & 1.23 & 3.41 \\ \hline \multirow{3}{*}{Anion-L} & all & 46.54 & 46.26 & 46.15 & 46.39 \\  & valid & ***50.71** & **+=48.36** & **46.16** & **+=49.85** \\  & \%iprv & 8.98 & 4.54 & 0.03 & 7.45 \\ \hline \multirow{3}{*}{Anion-S} & all & 46.90 & 47.73 & 47.47 & 47.53 \\  & valid & **47.14** & **+=50.42** & **48.20** & **+=50.62** \\  & \%iprv & 0.51 & 5.65 & 1.55 & 6.50 \\ \hline \multirow{3}{*}{Anion-C} & all & 50.80 & 51.29 & 51.28 & 51.83 \\  & valid & **50.94** & **51.52** & **52.65** & **+=53.91** \\ \cline{1-1}  & \%iprv & 0.28 & 0.45 & 2.67 & 4.02 \\ \hline \end{tabular}
\end{table}
Table 10: P@{# valid} scores of the _all_ and _valid_ sets determined by the **L**, **S**, **C** and **LSC** discriminators. Generations are from COMET-Full . Single (*) and double asterisks (**) indicate significance at p<0.05 and p<0.01, respectively. ipvrv% is the improvement of the _valid_ over the _all_ set. Underlines indicate the highest iprv% across discriminators.

 

than 98%. To ensure high quality task completions, we post pilot batches and manually examine tens of thousands of responses to identify users who provide high quality annotations. We select 834 qualified users for the formal data collection and human evaluation tasks. Since the entire study spans multiple months, we regularly sample responses to re-examine their quality during the formal study, and remove HITs from crowdworkers who provide decreased-quality responses over time. We are particularly cautious about the human evaluation tasks, so even with qualified users, we still comprehensively examine tens of thousands of human evaluation tasks by grouping HITs per users, and look at their responses together to identify potential spamming behaviors and inconsistencies.

For the data collection and human evaluation tasks, we aimed to compensate crowdworkers with an average of $15 per hour. To ensure a fair payment, we first post a pilot task to evaluate average time cost of a specific task, and pay users at a high rate in this round to avoid underpayment during the pilot study. We then calculate new payment from the pilot task such that approximately 75% of the HITs would have been paid with more than $15 per hour at the adjusted rate in the pilot round. We then adopt this new rate for the formal study. We repeat the above procedure of determining payment periodically during the study to ensure the crowdworkers are consistently well-paid.

## Acknowledgements

The authors thank Elisa Kreiss for helpful discussions. We also thank the anonymous reviewers and meta-reviewers for their helpful feedback. This research was supported in part by DARPA under the MCS program through NIWC Pacific (N66001-19-2-4031), and the Allen Institute for AI (AI2).

## References

* Acharya et al. (2021) Anurag Acharya, Kartik Talamadupula, and Mark A Finlayson. 2021. An atlas of cultural commonsense for machine reasoning. In _AAAI_.
* Ackrill (1963) JL Ackrill. 1963. _Aristotle's Categories and De Interpretatione_. Clarendon Press.
* Asai and Hajishirzi (2020) Akari Asai and Hannaneh Hajishirzi. 2020. Logicguided data augmentation and regularization for consistent question answering. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5642-5650, Online. Association for Computational Linguistics.
* Bhagavatula et al. (2020) Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In _ICLR_.
* Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piga: Reasoning about physical commonsense in natural language. In _AAAI_.
* Bosselut et al. (2020) Antoine Bosselut, Ronan Le Bras, and Yejin Choi. 2020. Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-shot Commonsense Question Answering. _arXiv: 1911.03876_.
* Bosselt et al. (2019) Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense Transformers for Automatic Knowledge Graph Construction. In _ACL_.
* Councill et al. (2010) Isaac Council, Ryan McDonald, and Leonid Velikovich. 2010. What's great and what's not: learning to classify the scope of negation for improved sentiment analysis. In _Proceedings of the Workshop on Negation and Speculation in Natural Language Processing_, pages 51-59, Uppsala, Sweden. University of Antwerp.
* Fancellu and Webber (2015) Federico Fancellu and Bonnie Webber. 2015. Translating negation: A manual error analysis. In _Proceedings of the Second Workshop on Extra-Propositional Aspects of Meaning in Computational Semantics (ExPrOd 2015)_, pages 2-11, Denver, Colorado. Association for Computational Linguistics.
* Feng et al. (2020) Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. 2020. Scalable multi-hop relational reasoning for knowledge-aware question answering. In _EMNLP_.
* Grice et al. (1975) H Paul Grice, Peter Cole, Jerry Morgan, et al. 1975. Logic and conversation. _1975_, pages 41-58.
* Haynes (2013) Winston Haynes. 2013. _Bonferroni Correction_, pages 154-154. Springer New York, New York, NY.
* Hendrycks and Gimpel (2020) Dan Hendrycks and Kevin Gimpel. 2020. Gaussian error linear units (gelus). _arXiv: 1606.08415_.
* Horn and Wansing (2020) Laurence R. Horn and Heinrich Wansing. 2020. Negation. In Edward N. Zalta, editor, _The Stanford Encyclopedia of Philosophy_, spring 2020 edition. Metaphysics Research Lab, Stanford University.
* Hossain et al. (2020) Md Mosnarafi Hossain, Antonios Anastasopoulos, Eduardo Blanco, and Alexis Palmer. 2020. It's not a non-issue: Negation as a source of error in machine translation. _arXiv: 2010.05432_.
* Hwang et al. (2020) Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2020. Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs. _arXiv: 2010.05953_ 

* Jowett et al. (1892) Benjamin Jowett et al. 1892. _The Dialogues of Plato: Parmenides. Theaetetus. Sophist. Statesman. Philebus_, volume 4. Oxford University Press, American branch.
* Kassner et al. (2020) Nora Kassner, Benno Krojer, and Hinrich Schutze. 2020. Are pretrained language models symbolic reasoners over knowledge? In _Proceedings of the 24th Conference on Computational Natural Language Learning_, pages 552-564, Online. Association for Computational Linguistics.
* Kassner and Schutze (2020) Nora Kassner and Hinrich Schutze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7811-7818, Online. Association for Computational Linguistics.
* Kim et al. (2019) Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, and Ellie Pavlick. 2019. Probing what different NLP tasks teach machines about function word comprehension. In _Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)_, pages 235-249, Minneapolis, Minnesota. Association for Computational Linguistics.
* Kingma and Ba (2017) Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. _arXiv: 1412.6980_.
* Lenat (1995) Douglas B Lenat. 1995. Cyc: A large-scale investment in knowledge infrastructure. _Communications of the ACM_, 38(11):33-38.
* Li et al. (2016) Xiang Li, Aynaz Taheri, Liftu Tu, and Kevin Gimpel. 2016. Commonsense knowledge base completion. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1445-1455, Berlin, Germany. Association for Computational Linguistics.
* Lin et al. (2019) Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. Kagnet: Knowledge-aware graph networks for commonsense reasoning. In _EMNLP_.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv: 1907.11692_.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_, ACL '02, page 311-318, USA. Association for Computational Linguistics.
* Pratt (1994) Ian Pratt. 1994. _Defeasible Inference_, pages 85-107. Macmillan Education UK, London.
* Qin et al. (2019) Lianhui Qin, Antoine Bosseltu, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. 2019. Counterfactual story reasoning and generation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5043-5053, Hong Kong, China. Association for Computational Linguistics.
* Radford et al. (2019) A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
* Rashkin et al. (2016) Hannah Rashkin, Sameer Singh, and Yejin Choi. 2016. Connotation frames: A data-driven investigation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 311-321, Berlin, Germany. Association for Computational Linguistics.
* Reitan et al. (2015) Johan Reitan, Jorgen Faret, Bjorn Gamback, and Lars Bungum. 2015. Negation scope detection for twitter sentiment analysis. In _Proceedings of the 6th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_, pages 99-108, Lisboa, Portugal. Association for Computational Linguistics.
* Ren and Leskovec (2020) Hongyu Ren and Jure Leskovec. 2020. Beta embeddings for multi-hop logical reasoning in knowledge graphs. In _NeurIPS_.
* Rudinger et al. (2020) Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 4661-4675, Online. Association for Computational Linguistics.
* Sakaguchi et al. (2020) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogradne: An adversarial winograd schema challenge at scale. In _AAAI_.
* Sap et al. (2020) Maarten Sap, Ronan LeBras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2020. ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning. In _AAAI_.
* Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. In _EMNLP_.
* Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In _AAAI_.
* Xiang et al. (2016) Ming Xiang, Julian Grove, and Anastasia Giannakidou. 2016. Semantic and pragmatic processes in the comprehension of negation: An event related potential study of negative polarity sensitivity. _Journal of Neurolinguistics_, 38:71-88 

Appendices

### Anion Data Collection Details

Heuristic of Creating Logical and Semi-logical Negation EventsFor logical negation, with the majority of the original events being simple sentences with one predicate, our general rule of thumb is to negate the original event at the sentence level. Specifically, with respect to each original event, we first identify each tokens' part of speech (POS) tags via the NLTK toolkit4. Then, we insert the negation cue _not_ after the subject of each sentence, with majority of the case the entity "PersonX," with few exceptions of "PersonX's" and "PersonX and PersonY."

Footnote 4: [https://www.nltk.org](https://www.nltk.org)

To ensure the grammar correctness of the heuristically generated logical negation events, we add appropriate auxiliary verbs (_e.g._, do, does, did, is, was, can, could, would, should, may, might) in accordance with the tenses (_e.g._, present, past, future) of the original events. Since NLTK's POS parser fails to recognize some of the verbs that have both noun and verb usage (_e.g._, "waters" the plant, "supports" her argument), we curate a list of dual-used words and map them manually. Also, while converting the original events to their logical negation counterparts, we revise grammar mistakes from Atomic and exclude awkward expressions as much as possible. In addition, to make the negation forms sound more natural, we replace the modifier "some" by "any" during conversion (_e.g._, "PersonX buys some shoes" is converted to "PersonX doesn't buy any shoes"). For the minority of compound events with clauses or complex sentence structures, we disregard them for the purpose of ensuring the data quality.

For semi-logical negation events, we curate a list of semi-logical negation cues besides _not_ from various sources5Councill et al. (2010); Hossain et al. (2020); Kim et al. (2019) and categorize them into four types including affixes, single-word cues, multi-word cues and negative verbs (Table 1). We identify appropriate rules to insert each semi-logical negation cue in simple base events from Atomic consisting of a subject and a predicate. We apply the rules to original events from Atomic and randomly select at least 200 automatically generated semi-logical negation events per each negation cue for manual screening by the first author to avoid misplacement of negation cues and awkward expressions. In the end, we were able to identify 5,019 high quality semi-logical negation events originating from Atomic.

Footnote 5: [https://dictionary.cambridge.org/us/grammar/british-grammar/negation_2](https://dictionary.cambridge.org/us/grammar/british-grammar/negation_2)

As a final quality control step of the constructed logical and semi-logical events, after obtaining the crowdsourced inferences for each event, we remove all events that annotators comment as "unclear," "doesn't make sense" or "grammatically wrong."

Crowdsourcing of Commonsense Contradiction EventsFor collecting commonsense contradiction events, we present an original Atomic event to the annotators and ask them to formulate corresponding opposite events. We exclude Atomic events with placeholders representing generic objects) to capture semantic and pragmatic subtlety. In the MTurk task, we present annotators detailed instructions of formulating the opposite events (_e.g._, avoid using negative words as much as possible, use complete sentences, follow grammar rules) and concrete examples as references. Figure 2 shows details of the MTurk task. Although we explicitly instruct annotators to avoid using negation cues, there are still some exceptions. Therefore, after the compilation of all commonsense contradiction events, we remove ones that contain any explicit negation cues to make sure the categorization is clean.

Crowdsourcing of Anion Event InferencesFor the collection of Anion event inferences, we adopt the MTurk templates used by the original Atomic data collection6. Similarly to logical and semi-logical events, we remove all inferences of events that annotators comment as "unclear," "doesn't make sense" or "grammatically wrong."

Footnote 6: [https://homes.cs.washington.edu/~msap/atomic/mTurkFiles/](https://homes.cs.washington.edu/~msap/atomic/mTurkFiles/)

### Training Details of COMET Models

InputA knowledge tuple \(\{h,r,t\}\) is represented as a concatenated sequence with tokens of each element in the tuple: \(X=\{X^{h},X^{r},X^{t}\}\) where \(X^{h}=\{x_{0}^{h},...,x_{|h|}^{h}\}\) are the tokens comprising the event, \(X^{r}=\{x_{0}^{r},...,x_{|r|}^{r}\}\) as tokens comprising the relation, and \(X^{t}=\{x_{0}^{t},...,x_{|t|}^{t}\}\) are the tokens comprising the commonsense inference.

InitializationSimilar to Bosselut et al. (2020), we initialize the trained parameters of COMET to the 345M parameter GPT2 model (GPT2-M) from Radford et al. (2019). Special tokens that 

represent relation types (_e.g._, _xIntent_) are added to the vocabulary and initialized via sampling from the normal distribution.

HyperparametersFollowing Bosselut et al. (2019), we use a dropout rate of 0.1 and GeLU Hendrycks and Gimpel (2020) units as activation functions. During training, we use the Adam optimizer Kingma and Ba (2017) with a batch size of 64. For COMET models trained on different subsets of the Atomic and Anion datasets, we adopt a maximum learning rate of 6.25c-5 with a warmup period of 0.002 times of the total number of minibatches customized for each model, which decays linearly until finishing training.

We train different COMET models for different subsets of the full on original data (Atomic), original and logical negation data (Atomic + Anion-L), original and semi-logical negation data (Atomic + Anion-S), original and commonsense contradiction data (Atomic + Anion-C), and the overall dataset (Atomic + Anion), for 21k, 25K, 24K, 24K and 29K minibatches respectively, and apply early stopping for all models. The rest of the hyperparameters are the same as those of GPT2-M in Radford et al. (2019) implemented via the publicly available HuggingFace API7 .

Footnote 7: [https://huggingface.co/transformers/](https://huggingface.co/transformers/)

All models are fine-tuned and evaluated on a single NVIDIA QUADRO RTX 8000 GPU for six to twelve hours depending on the complexity of the experimental setup.

### Training Details of Negation Discriminator

InputAs input to the discriminator model, we design sentence patterns that express relation types in natural language and fill out the patterned sentences with events and conditions before encoding them (_e.g._, "PersonX addresses a talk. As a result, PersonX wants to convince others."). Relations and their corresponding patterned sentences are listed in Table 12. Adopting patterned sentences is found to be a more effective approach than concatenating components in knowledge tuples from the pilot study.

Loss FunctionThe negation discriminator is trained to minimized the binary cross-entropy loss:

\[{\mathcal{L}}_{D}=y\cdot\log P(y)+(1-y)\cdot\log\left(1-P(y)\right) \tag{2}\]

where \(y\) is the label for an input (_i.e._, logically valid or invalid).

HyperparametersParameters are initialized with the trained weights of the RoBERTa-base model in Liu et al. (2019). During training, we use the Adam optimizer Kingma and Ba (2017) and train the model with a batch size of 64. We adopt a maximum learning rate of 4.5e-5 with a warmup period of 10 minibatches. We trained **L**, **S**, **C**, **LSC** discriminators, for 25K, 14K, 21K and 6K miniabatches respectively, and apply early stopping for all models. We use a probability threshold of 0.7 to determine whether an input knowledge tuples to the discriminator is plausible based on pilot study on the development sets. The rest of the hyperparameters are the same as those of RoBERTa-base Liu et al. (2019) implemented via the publicly available HuggingFace API8 .

Footnote 8: [https://huggingface.co/transformers/](https://huggingface.co/transformers/)

All models are fine-tuned and evaluated on a single NVIDIA QUADRO RTX 8000 GPU for four to six hours depending on the different experimental setups.

### Statistical Significance Testing

To compare P@{# valid} for the _all_ and _valid_ sets, we use a Permutation Test9 with 1,000 permutations to test for statistical significance. For multiple comparisons, we use the Bonferroni method Haynes (2013) to correct significance thresholds.

Footnote 9: [http://rasbt.github.io/mlxtend/](http://rasbt.github.io/mlxtend/)

### Quality Check for the Human Evaluation

We conduct comprehensive pre- and post-evaluation screening on the users and the tasks being completed to ensure the objectivity and high quality of the evaluations. Besides qualifying users

\begin{table}
\begin{tabular}{l|p{284.5pt}}
**Relation** & **Patterned sentences** \\ \hline _xIntent_ & \{\(h\). Because PersonX wanted \(t\). \\ _xNeed_ & \{\(h\). Before, PersonX needed \(t\). \\ _xAttr_ & \{\(h\). PersonX is seen as \(t\)} \\ _xWant_ & \{\(h\). As a result, PersonX wants \(t\). \\ _oWant_ & \(\{h\). As a result, others want \(\{t\}\). \\ _xEffect_ & \(\{h\). As a result, PersonX then \(\{t\}\). \\ _oEffect_ & \(\{h\}\). As a result, others then \(\{t\}\). \\ _xReact_ & \(\{h\}\). As a result, PersonX feels \(\{t\}\). \\ _oReact_ & \(\{h\}\). As a result, others feel \(\{t\}\). \\ \hline \end{tabular}
\end{table}
Table 12: Patterned sentences representing relation types in Atomic, used to construct inputs for training negation discriminators.

 



\begin{table}
\begin{tabular}{c|r r|r r|r r|r r|r r|r r|r r}
**Eval** & & \multicolumn{3}{c|}{**Atomic**} & \multicolumn{3}{c|}{**Anison-L**} & \multicolumn{3}{c|}{**Anion-S**} & \multicolumn{3}{c}{**AnNon-C**} \\ \hline
**Trn** & **Dis** & all & valid & ipv\% & all & valid & ipvr\% & all & valid & ipv\% & all & valid & ipv\% \\ \hline  & L & 59.41 & **59.65** & 0.40 & 44.92 & **+49.95** & 11.20 & 39.47 & **39.94** & 1.21 & 50.77 & **50.91** & 0.27 \\ Atomic & S & 59.48 & **60.14** & 1.12 & 42.88 & **+46.24** & 7.83 & 45.27 & **+49.25** & 8.81 & 51.22 & **51.84** & 1.21 \\  & C & 59.89 & **60.89** & 1.66 & 39.20 & **40.28** & 2.75 & 40.07 & **41.40** & 3.32 & 51.77 & **52.88** & 2.15 \\  & LSC & 61.37 & **63.12** & 2.85 & 46.00 & **+50.34** & 9.44 & 46.15 & **+50.23** & 8.85 & 53.29 & **55.24** & 3.65 \\ \hline  & L & 61.33 & **61.57** & 0.39 & 51.47 & ***56.16 & 9.11 & 45.73 & **46.04** & 0.68 & 51.40 & **51.57** & 0.33 \\ Atomic+ & S & 61.13 & **62.05** & 1.51 & 50.12 & **+53.40** & 6.54 & 50.84 & **+54.09** & 6.40 & 51.89 & **52.91** & 1.96 \\ Anion-L & C & 61.48 & **62.96** & 2.40 & 48.23 & **49.06** & 1.72 & 46.42 & **+46.99** & 1.22 & 52.31 & **53.67** & 2.61 \\  & LSC & 63.66 & **+65.85** & 3.44 & 51.90 & **+56.26** & 8.40 & 51.12 & **+54.59** & 6.78 & 53.97 & **56.15** & 4.04 \\ \hline  & L & 60.25 & **60.65** & 0.67 & 48.11 & **+54.45** & 13.18 & 45.97 & **46.35** & 0.81 & 50.82 & **50.89** & 0.15 \\ Atomic+ & S & 60.23 & **60.85** & 1.03 & 46.48 & **+49.14** & 5.72 & 47.58 & **+50.29** & 5.70 & 51.11 & **52.00** & 1.74 \\ Anion-S & C & 60.43 & **61.28** & 1.40 & 44.61 & **+63.31** & 3.80 & 46.21 & **+48.78** & 5.58 & 51.72 & **53.25** & 2.95 \\  & LSC & 62.22 & **64.44** & 3.58 & 47.63 & **+51.12** & 7.32 & 48.24 & **+50.70** & 5.11 & 53.51 & **+56.04** & 4.74 \\ \hline  & L & 54.36 & **54.80** & 0.81 & 46.25 & **+51.57** & 11.51 & 42.81 & **+43.13** & 0.76 & 50.71 & **50.81** & 0.20 \\ Atomic+ & S & 54.50 & **55.75** & 2.29 & 45.59 & **+48.11** & 5.53 & 45.40 & **+48.50** & 6.83 & 51.00 & **51.78** & 1.52 \\ Anion-C & C & 54.43 & **55.50** & 1.97 & 42.61 & **+43.26** & 1.53 & 43.11 & **+44.13** & 2.38 & 51.44 & **+53.46** & 3.93 \\  & LSC & 55.68 & **+57.91** & 4.01 & 47.11 & **+51.25** & 8.80 & 45.75 & **+49.03** & 7.18 & 52.44 & **+55.68** & 6.18 \\ \hline  & L & 56.63 & **57.11** & 0.85 & 50.39 & **+54.52** & 8.20 & 47.92 & **+48.27** & 0.73 & 53.11 & **53.41** & 0.56 \\ Atomic+ & S & 56.53 & **57.42** & 1.57 & 48.92 & **+52.07** & 6.44 & 49.21 & **+52.51** & 6.72 & 53.10 & **53.67** & 1.09 \\ Anion & C & 56.40 & **57.64** & 2.21 & 47.96 & **+48.30** & 0.70 & 48.16 & **50.00** & 3.82 & 53.90 & **55.48** & 2.94 \\  & LSC & 58.27 & **60.53** & 3.87 & 50.25 & **+54.27** & 8.02 & 49.85 & **+53.09** & 6.50 & 54.50 & **+57.66** & 5.79 \\ \hline \end{tabular}
\end{table}
Table 14: For generations of COMET models trained on different subsets of Atomic and Anion, the Precision @ 3 scores of the _all_ and _valid_ sets determined by **L**, **S**, **C** and **LSC** discriminators with respect to the original and negation evaluation sets. The single (*) and double asterisks (**) indicate significance at p<0.05 and p<0.01 respectively. iprv% is the percentage improvement of the _valid_ set over the _all_ set.

\begin{table}
\begin{tabular}{c|r|r r|r r|r r|r r|r r|r r}
**Eval** & & \multicolumn{3}{c|}{**Atomic**} & \multicolumn{3}{c|}{**ANION-L**} & \multicolumn{3}{c|}{**Anion-S**} & \multicolumn{3}{c}{**Anion-C**} \\ \hline
**Trn** & **Dis** & all & valid & ipvr\% & all & valid & ipvr\% & all & valid & ipvr\% & all & valid & ipvr\% \\ \hline  & L & 59.41 & **59.65** & 0.40 & 44.92 & **+49.95** & 11.20 & 39.47 & **39.94** & 1.21 & 50.77 & **50.91** & 0.27 \\ Atomic & S & 59.48 & **60.14** & 1.12 & 42.88 & **+46.24** & 7.83 & 45.27 & **+49.25** & 8.81 & 51.22 & **51.84** & 1.21 \\  & C & 59.89 & **60.89** & 1.66 & 39.20 & **40.28** & 2.75 & 40.07 & **41.40** & 3.32 & 51.77 & **52.88** & 2.15 \\  & LSC & 61.37 & **63.12** & 2.85 & 46.00 & **+50.34** & 9.44 & 46.15 & **+50.23** & 8.85 & 53.29 & **55.24** & 3.65 \\ \hline  & L & 61.33 & **61.57** & 0.39 & 51.47 & **+56.16** & 9.11 & 45.73 & **+46.04** & 0.68 & 51.40 & **51.57** & 0.33 \\ Atomic+ & S & 61.13 & **62.05** & 1.51 & 50.12 & **+53.40** & 6.54 & 50.84 & **+54.09** & 6.40 & 51.89 & **52.91** & 1.96 \\ Anion-L & C & 61.48 & **62.96** & 2.40 & 48.23 & **+49.06** & 1.72 & 46.42 & **+46.99** & 1.22 & 52.31 & **53.67** & 2.61 \\  & LSC & 63.66 & **+65.85** &

during pilot batches, we double check to remove evaluation tasks that are not carefully conducted (_e.g._, tasks done by users that select all/no options for all hundreds of tasks that they perform). Figure 3 shows a snippet of the human evaluation MTurk task.

\begin{table}
\begin{tabular}{l|l|c c} \hline \hline
**Event + Rel** & **Generation** & **V** & **P** \\ \hline X hardly ever increases X’s knowledge & intelligent & ✗ & ✗ \\ knowledge & determined & ✗ & ✗ \\ knowledge & studious & ✗ & ✗ \\ lazy & ✓ & ✓ \\ _xAttr_ & dedicated & ✗ & ✗ \\ \hline X avoids & to have fun & ✗ & ✗ \\ skating around & to be safe & ✓ ✓ & ✓ \\ x_Intent_ & to stay home & ✓ ✓ & ✓ \\  & to stay in shape & ✗ & ✗ \\ \cline{2-4}   X not at all & learn karate & ✗ & ✗ \\ wants to learn & to not like it & ✓ ✓ & ✓ \\ karate & to avoid it & ✓ ✓ & ✓ \\ _x Need_ & to be lazy & ✓ ✓ & ✓ \\ \cline{2-4}   X does not rent & to get a job & ✓ ✓ & ✓ \\ an apartment & to pay rent & ✗ & ✗ \\ _xWant_ & to move in & ✗ & ✗ \\  & to get a new apartment & ✓ & ✓ \\ \hline X is & to hire X & ✗ & ✗ \\ not offered & to fire X & ✓ & ✓ \\ the job & to accept the job & ✗ & ✗ \\ _oWant_ & to hire them & ✗ & ✗ \\ \hline X does & satisfied & ✗ & ✗ \\ not buy & hungry & ✓ & ✓ \\ a snack & satiated & ✗ & ✗ \\ _xReact_ & full & ✗ & ✗ \\ guilty & ✓ & ✗ \\ \hline X does & relieved & ✓ & ✓ \\ not bring Y into conflict & sad & ✗ & ✗ \\ _oReact_ & satisfied & ✓ & ✓ \\  & grateful & ✓ & ✓ \\  & angry & ✗ & ✗ \\ \hline X does & gains knowledge & ✗ & ✗ \\ not learn & becomes lazy & ✓ & ✓ \\ new things & gets bored & ✓ & ✓ \\ _xEffect_ & becomes ignorant & ✗ & ✓ \\  & cries & ✓ & ✓ \\ \hline X does & becomes confused & ✗ & ✗ \\ not put Y & does not think about X & ✓ & ✓ \\ in mind & Y thinks about X & ✗ & ✗ \\ _oEffect_ & Y is not remembered & ✓ & ✓ \\  & cries & ✗ & ✓ \\ \hline \hline \end{tabular}
\end{table}
Table 15: Randomly selected generations of the original COMET model regarding logical negation events in Anion-L. The top 5 options are classified as either _valid_ or _invalid_ by the **LSC** discriminator. **V** indicates whether an option is classified as _valid_ by the **LSC** discriminator. **P** indicates whether an option is plausible judging by humans.

 



\begin{table}
\begin{tabular}{l|l|c c} \hline
**Event + Rel** & **Generation** & **V** & **P** \\ \hline X keeps & determined & ✗ & ✗ \\ the old one & careful & ✗ & ✗ \\  & loyal & ✓ & ✓ \\ _xAttr_ & tried & ✗ & ✗ \\  & caring & ✗ & ✗ \\ \hline X pays twice & to own something rare & ✓ & ✓ \\ the retail price & to buy something rare & ✓ & ✓ \\ for a rare item & to purchase something rare & ✓ & ✓ \\ _xIntent_ & to obtain something & ✗ & ✗ \\  & to acquire something & ✗ & ✗ \\ \hline X is a & to avoid people & ✓ & ✓ \\ complete & to be lonely & ✓ & ✓ \\ loner & to start a relationship & ✗ & ✗ \\ _xNeed_ & to stay quiet & ✓ & ✓ \\  & to get into trouble & ✗ & ✗ \\ \hline X is & to get better & ✓ & ✓ \\ misrable & to take a break & ✗ & ✓ \\ playing & to go home & ✗ & ✗ \\ _xWant_ & to go to sleep & ✗ & ✗ \\  & to cry & ✓ & ✓ \\ \hline X is picky about & to ignore X & ✓ & ✓ \\   everything & to avoid X & ✓ & ✓ \\  & to talk to X & ✗ & ✗ \\ _oWaunt_ & to help X & ✗ & ✗ \\  & to make X feel better & ✓ & ✗ \\ \hline X resigns & relieved & ✓ & ✓ \\ himself & relaxed & ✓ & ✗ \\  & satisfied & ✗ & ✗ \\ _xReact_ & accomplished & ✗ & ✗ \\  & sad & ✓ & ✓ \\ \hline X gives away & grateful & ✓ & ✓ \\ X’s laptop & thankful & ✓ & ✓ \\  & upset & ✗ & ✗ \\ _oReact_ & sad & ✗ & ✗ \\  & surprised & ✓ & ✓ \\ \hline X goes & relaxes & ✓ & ✓ \\ home & goes to sleep & ✓ & ✓ \\  & is greeted by family & ✗ & ✓ \\ _xEffect_ & gets rest & ✓ & ✓ \\  & gets fired & ✗ & ✗ \\ \hline X puts Y & has a better day & ✗ & ✗ \\ out of mind & becomes sad & ✓ & ✓ \\  & caries & ✓ & ✓ \\ _oEffect_ & becomes grateful towards X & ✗ & ✗ \\  & feels better & ✗ & ✗ \\ \hline \end{tabular}
\end{table}
Table 17: Randomly selected generations of the original COMET model regarding commonsense contradiction events from Anion-C. The top 5 options are classified as either _valid_ or _invalid_ by the **LSC** discriminator. **V** indicates whether an option is classified as _valid_ by the **LSC** discriminator. **P** indicates whether an option is plausible judging by humans.



* **Step 1: read a short event sentence.**
* _Note that the names of specific people have been replaced by generic words (e.g., "PersonX", "PersonY")._
* **Step 2: given this event, you are asked to formulate TWO (up to FOUR) corresponding OPPOSITE events.**
* _There might be_ **multiple** _grammically correct ways of expressing your interpretation of the OPPOSITE events. Please express them in natural language (i.e., in a way that you normally talk), and make sure your OPPOSITE events are in complete sentences._
* _Please don't_ **trivially negate. Try not to use negative words directly, including but not limited to: no, not, nothing, no one, none, nobody, nowhere, neither, nor, never, lack of, unless you feel necessary._
* _Changing one of the characters (e.g., from "PersonX" to "personY"), or one of the objects (e.g., from "mother" to "father" or from "cat" to "dog?") are not the goals of the OPPOSITE event, unless you think they are appropriate._
* _Please do not add unnecessary/unrelated additional details to the OPPOSITE event._

Examples (click to collapse/expand)

Event ${Event}

Given this event, can you formulate corresponding OPPOSITE events? Make sure they are in complete sentences.

Opposite Event 1 (REQURED)

Opposite Event 2 (REQURED)

Opposite Event 3 (OPTIONAL)

Opposite Event 4 (OPTIONAL)

Submit 



## References

* [1] S. B. Krizhevsky, A. Lerman, and D. R. Bishop. The output of a neural network for learning the shape of the input. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 237-240, Dec. 2015.
* [2] M. C. Davis, J. H. Friedman, and G. E. Hinton. Visualizing data using deep networks. _arXiv preprint arXiv:1906.08810_, 2019.
* [3] P. F. Donahue, Y. Bengio, and T. Darrell. Learning representations by jointly connected layers. _Journal of Machine Learning Research_, 10(1):1-10, Jan. 2015.
* [4] N. DeVito, V. LeCun, and I. Guyon. Deep residual learning with convolutional nets. In _Advances in Neural Information Processing Systems (NIPS'06)_, pages 133-146, 2006.

[MISSING_PAGE_POST]