SlowGuess/NAACL_2021_OCR huggingface dataset:
on the evaluation set related to that negation type. Interestingly, though, training on certain types of negation examples can also yield benefits downstream on other negation types. For example, training on commonsense contradictions (ANION-C) provides a clear benefit when evaluating on semilogically negated events (ANION-S) as opposed to merely training on ATOMIC. Notably, the knowledge model trained with logically negated examples (ATOMIC + ANION-L) outperforms the model trained only on ATOMIC on all test sets.
5 Discriminating Inconsistent Inferences
While training on examples of negated events helps knowledge models generate commonsense inferences for these event types, there is still a large gap compared to their performance on affirmative events. To address this discrepancy, we introduce a discriminator-based approach for distinguishing inconsistent inferences of negated events. Our inference discriminator learns to identify plausible and invalid inferences of events by learning from contrastive samples from ATOMIC and ANION.
We fine-tune the RoBERTa-base model (Liu et al., 2019) as a binary classifier to identify whether a given knowledge tuple  $\{h, r, t\}$  is logically valid. The model is trained on paired original and negated
events as described below. Such training examples inject implicit commonsense nuances that differ between oppositional events to teach the discriminator to identify logical pitfalls. Training details for discriminators can be found in Appendix A.3.
Data The paired events used to train the negation discriminator are automatically constructed from the ATOMIC and ANION knowledge graphs. Positive examples can be constructed by sampling tuples from each knowledge graph. To construct negative training samples, we introduce the concept of common and contrast sets among inferences of events and their oppositions.
Common and contrast sets distinguish how commonsense inferences are not necessarily negated in the same manner as their corresponding events. While certain inferences of events are also in opposition to a negated event, some may be common. For the events "X eats a cheeseburger" and "X eats a salad," an inference such as "X is hungry" might be common to both events while inferences such as "X is unhealthy" or "X is healthy" would be viewed as contrastive.
Specifically, we assume two head events in ATOMIC and ANION, and their respective set of tail inferences regarding a common relation type. We define the common set of these inferences as the intersection of the two sets of tail inferences connected to each head event by applying the exact match of string forms. The contrast set is formed by distinct tail inferences connected to the two head events. Logically valid (i.e., positive) training examples consist of knowledge tuples from ATOMIC and ANION. Logically invalid (i.e., negative) training examples are formed by swapping the set of contrast set inferences between paired original and negated events. $^3$
To balance the training set, we sample the same number of positive and negative tuples for original and negation events. Statistics of the resulting training sets are in Table 6.
Using different portions of ANION for training yields four unique discriminators (i.e.,  $\mathbf{L},\mathbf{S},\mathbf{C}$  and
Discriminator	Train Set	Size
Logical Negation (L)	ANION-L	324,843
Semi-logical Negation (S)	ANION-S	194,732
Commonsense Contradiction (C)	ANION-C	276,272
All Oppositional Data (LSC)	ANION	795,845
Table 6: Statistics of data used to train negation discriminators.
Eval Set	#	BL2↑	P@k↑
ATOMIC	all	10.0	14.18	55.18
valid	6.3	14.24	59.07
invalid	3.7	13.93	44.10
ANION-L	all	10.0	10.86	35.84
valid	5.6	11.33	45.59
invalid	4.4	10.13	25.96
ANION-S	all	10.0	12.07	36.89
valid	6.3	12.63	44.93
invalid	3.7	11.32	27.83
ANION-C	all	10.0	14.32	46.70
valid	5.9	14.78	51.45
invalid	4.1	13.56	37.33
Table 7: The evaluation of the all, valid and invalid sets of inferences generated by COMET-ATOMIC as partitioned by the LSC discriminator.  $\mathbf{P}@\pmb{k}$  corresponds to the human-rated precision of a set.  $k$  is the number of elements in all, valid, or invalid set. For the valid set, higher  $\mathbf{P}@\pmb{k}$  is better (i.e., more valid inferences are being partitioned). For the invalid set, lower  $\mathbf{P}@\pmb{k}$  is better (i.e., fewer valid inferences are being included).
LSC) that we apply to commonsense inferences generated by COMET. The discriminators classify each option as either logically valid or invalid, partitioning the candidates into two sets, which we evaluate with human judgements. As a baseline, we also record the precision of not using a discriminator, which assumes all generated inferences are valid candidates (i.e., the all set).
Metrics We evaluate and compare the quality of the all, valid and invalid sets using BLEU-2 and the same human evaluation as in §4. The all set contains the full set of 10 candidates, while the valid and invalid sets have varying number of elements depending on how discriminators classify them, summing to 10. To compute statistical significance between valid and all sets, we use a permutation test with 100K permutations. Details are provided in Appendix A.4.
Do discriminators effectively distinguish inconsistent inferences? The results in Table 7 demonstrate that the discriminator trained on all subsets of ANION (LSC) can select subsets of inferences (i.e.,
<sup>3</sup>We note that annotations in ATOMIC and ANION are finite (i.e., not covering the full space of possible commonsense inferences about events). As a result, it is possible that in a more expansive annotation, elements of the contrast sets would in fact be part of the common set of an event and its negation. For the purpose of this work, however, contrast sets were an efficient way of acquiring high-quality semantically negative examples for training discriminators.
Event + Rel	Generation	V	P
X does not skate around xAttr	athletic careless lazy uncoordinated unskilled	× × × ×	× × × ×
× ×	× ×
√	√
√	√
√	√
×	×
×	×
X does not look angry xIntent	to be alone to be left alone to avoid Y to sit to wait	√	√
√	√
√	√
×	×
×	×
X refuses to look a scary noise xWant	to calm down to watch a movie to have been provoked to not be angry to be calm	×	√ ×
√	× ×
×	× ×
√	√
√	√
X never brings Y into conflicts oWant	to run away to go to sleep to be safe to keep quiet to avoid the noise	×	×
√	√
√	√
×	×
×	×
X never brings Y into conflicts oWant	to avoid X to be left alone to thank X to fight back to avoid conflict	×	×
×	√ ×
√	√
×	× ×
×	√
X scarcely gets sunburned xReact	burned hurt sick sad satisfied	×	×
×	×
×	× ×
×	× ×
×	× ×
X under no circumstances forgets Y's wallet oReact	upset sad angry thankful grateful	×	×
×	× ×
×	× ×
√	√
X has trouble with advertising X's business xEffect	loses money loses clients gets fired gets sued cries	√	√
√	√
√	√
×	× ×
×	× ×
X puts Y out of mind oEffect	has a better day becomes sad cries is grateful towards X feels better	×	× × × × ×
√	√
√	√
×	× × × × ×
Table 8: Inferences of randomly selected ANION events by COMET-ATOMIC. The top 5 options are classified as valid or invalid by the LSC discriminator.  $\mathbf{V}$  indicates whether an option is classified as valid by the LSC discriminator.  $\mathbf{P}$  indicates whether an option is plausible judging by humans.
the valid set) that are more logically consistent with their seed event. This observation holds across all evaluation subsets of ANION, as well as the original ATOMIC evaluation set. Table 8 shows examples of valid and invalid candidates for negated and contradicted events from ANION as specified by the LSC discriminator. The discriminator is notably
Disc Eval	L	S	C	LSC
ATOMIC	all	55.69	55.93	56.94	58.30
valid	55.65	56.18	57.26	59.07
%iprv	-0.07	0.44	0.57	1.32
ANION-L	all	39.46	37.85	36.43	39.45
valid	**46.39	**41.93	37.54	**45.59
%iprv	17.55	10.78	3.03	15.57
ANION-S	all	37.13	39.29	37.72	38.55
valid	37.48	**44.58	39.03	**44.93
%iprv	0.96	13.47	3.45	16.56
ANION-C	all	46.92	47.32	48.26	48.81
valid	46.83	47.68	48.79	*51.45
%iprv	-0.20	0.75	1.09	5.40
Table 9: P@ {# valid} scores of the all and valid sets determined by the L, S, C and LSC discriminators. Generations are from COMET-ATOMIC. Asterisks  $(^{**})$  indicate significance at  $p < 0.01$ . iprv% is the improvement of the valid over the all set. Underlines show the highest iprv% across discriminators.
good at identifying invalid inferences wrongly associated to corresponding affirmative events (e.g., "athletic" and "careless" for the event "X does not skate around" under the relation,  $xAttr$ ).
However, this analysis leaves open the possibility that we are generating too many inferences for each event, but that the decoder could rank correct inferences higher among the full set of generated candidates. To evaluate this possibility, we count the number of elements in the valid sets for each example and only keep the same number of the top-scoring elements from the all set (scored using generation perplexity). In Table 9, we see the average precision score for the pruned all sets  $(\mathrm{P}@\{\# \text{valid}\})$  still underperforms the precision of their corresponding valid sets.
Which negation categories are most important to provide a discriminator for? To examine the generalization effects of each negation type, we also train discriminators on a single negation subset of ANION examples (i.e., L, S, C) and compare the P@{# valid} score of the all and valid sets. Results in Table 9 indicate that each discriminator is best for identifying valid inferences for the types of events on which it was trained. The L, S, and C discriminators all achieve improvements when partitioning events similar to their training. However, the LSC discriminator trained on all negation forms shows the largest valid set improvement across all discriminators on ATOMIC, ANION-S, and ANION-C. On ANION-L, the LSC discrimina
Disc Eval	L	S	C	LSC
ATOMIC	all valid %iprv	54.16	54.49	55.03	55.68
54.20	54.64	55.71	**57.58
0.08	0.28	1.23	3.41
ANION-L	all valid %iprv	46.54	46.26	46.15	46.39
**50.71	**48.36	46.16	**49.85
8.98	4.54	0.03	7.45
ANION-S	all valid %iprv	46.90	47.73	47.47	47.53
47.14	**50.42	48.20	**50.62
0.51	5.65	1.55	6.50
ANION-C	all valid %iprv	50.80	51.29	51.28	51.83
50.94	51.52	52.65	*53.91
0.28	0.45	2.67	4.02
Table 10: P@{# valid} scores of the all and valid sets determined by the L, S, C and LSC discriminators. Generations are from COMET-FULL. Single (*) and double asterisks  $(^{**})$  indicate significance at  $p < 0.05$  and  $p < 0.01$ , respectively. iprv% is the improvement of the valid over the all set. Underlines indicate the highest iprv% across discriminators.
Beam Size	Set	#✓	#total	P@#total
10	all	3.6	10.0	35.84
valid	2.1	4.4	45.59
25	all	8.1	25.0	32.29
valid	4.3	10.5	38.18
Table 11: Number of correct generations from applying the LSC discriminator to generations of COMET-ATOMIC for beam size of 10 and 25 for logical negation events.  $\# \checkmark$  is the number of correct options. #total is the number of options in each set.
tor still yields a significantly improved valid set.
Are learning-based and discriminator-based approaches complementary? We apply our discriminators to the generations of the COMET model trained on ANION. In Table 10, we see that the LSC discriminator, when applied to generations of COMET trained on ANION, achieves significant improvements over all evaluation sets, including the original events. The full evaluation of the  $\mathrm{P@}\{\#\text{valid}\}$  and  $\mathrm{P@3}$  scores of applying different discriminators to generations of COMET trained on different data over all evaluation sets are shown in Table 13 and 14 in Appendix A.
Can discriminators be used to more aggressively generate inferences? While applying discriminators to generated inferences yields a valid subset with higher accuracy, we are left with fewer correct inferences in total. Thus, we investigate
the efficiency of using discriminators to expand the number of inferences generated. We decode inferences from COMET with beam size 25, and then apply the discriminator to this larger candidate set.
Table 11 shows that for logical negation, the valid set of beam 25 has higher accuracy and more correct options than the all set of beam 10. Thus, when we have a larger and potentially more noisy set of candidates, applying the negation discriminator yields a set of options that have higher quality than using all the candidates from a smaller set of initial generations.
We present the first comprehensive study on commonsense implications of negations and contradictions. To expand commonsense resources for the challenge of negation modeling, we introduce ANION, a large scale commonsense knowledge graph for negated and contradicted events. We use ANION to train commonsense knowledge models and demonstrate that it effectively enriches machine commonsense inference capabilities around negation. Lastly, we propose a negation discriminator capable of identifying logical flaws in commonsense inferences. By combining the model trained on ANION with the negation discriminator, we achieve a further performance boost.
ANION Language Choice and Implications
We select English as the base language of ANION so that our resource may be directly linked with the original ATOMIC knowledge graph. We acknowledge, however, that resources in English are more likely to reflect the mindsets and behaviors of English speakers. Furthermore, and in our case specifically, our annotators were primarily from the US. Consequently, this language choice biases the content of the knowledge graph toward North American perspectives, which affects what models trained on these resources would learn about social norms (Acharya et al., 2021). Future works may also include other languages and cultures to make the ANION resource more culturally and ideologically inclusive.
Crowdworker Recruitment, Quality Control and Remuneration
We recruit crowdworkers from MTurk who are located within the US with HIT approval rates higher
than  $98\%$ . To ensure high quality task completions, we post pilot batches and manually examine tens of thousands of responses to identify users who provide high quality annotations. We select 834 qualified users for the formal data collection and human evaluation tasks. Since the entire study spans multiple months, we regularly sample responses to re-examine their quality during the formal study, and remove HITs from crowdworkers who provide decreased-quality responses over time. We are particularly cautious about the human evaluation tasks, so even with qualified users, we still comprehensively examine tens of thousands of human evaluation tasks by grouping HITs per users, and look at their responses together to identify potential spamming behaviors and inconsistencies.
For the data collection and human evaluation tasks, we aimed to compensate crowdworkers with an average of  $15 per hour. To ensure a fair payment, we first post a pilot task to evaluate average time cost of a specific task, and pay users at a high rate in this round to avoid underpayment during the pilot study. We then calculate new payment from the pilot task such that approximately \(75\%$  of the HITs would have been paid with more than \)15 per hour at the adjusted rate in the pilot round. We then adopt this new rate for the formal study. We repeat the above procedure of determining payment periodically during the study to ensure the crowdworkers are consistently well-paid.
The authors thank Elisa Kreiss for helpful discussions. We also thank the anonymous reviewers and meta-reviewers for their helpful feedback. This research was supported in part by DARPA under the MCS program through NIWC Pacific (N66001-19-2-4031), and the Allen Institute for AI (AI2).
Anurag Acharya, Kartik Talamadupula, and Mark A Finlayson. 2021. An atlas of cultural commonsense for machine reasoning. In AAAI.
JL Ackrill. 1963. Aristotle's Categories and De Interpretatione. Clarendon Press.
Akari Asai and Hannaneh Hajishirzi. 2020. Logic-guided data augmentation and regularization for consistent question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5642-5650, Online. Association for Computational Linguistics.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In ICLR.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In AAAI.
Antoine Bosselut, Ronan Le Bras, and Yejin Choi. 2020. Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-shot Commonsense Question Answering. arXiv: 1911.03876.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense Transformers for Automatic Knowledge Graph Construction. In ACL.
Isaac Councill, Ryan McDonald, and Leonid Velikovich. 2010. What's great and what's not: learning to classify the scope of negation for improved sentiment analysis. In Proceedings of the Workshop on Negation and Speculation in Natural Language Processing, pages 51-59, Uppsala, Sweden. University of Antwerp.
Federico Fancellu and Bonnie Webber. 2015. Translating negation: A manual error analysis. In Proceedings of the Second Workshop on Extra-Propositional Aspects of Meaning in Computational Semantics (ExProM 2015), pages 2–11, Denver, Colorado. Association for Computational Linguistics.
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. 2020. Scalable multi-hop relational reasoning for knowledge-aware question answering. In EMNLP.
H Paul Grice, Peter Cole, Jerry Morgan, et al. 1975. Logic and conversation. 1975, pages 41-58.
Winston Haynes. 2013. Bonferroni Correction, pages 154-154. Springer New York, New York, NY.
Dan Hendrycks and Kevin Gimpel. 2020. Gaussian error linear units (gelus). arXiv: 1606.08415.
Laurence R. Horn and Heinrich Wansing. 2020. Negation. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, spring 2020 edition. Metaphysics Research Lab, Stanford University.
Md Mosharaf Hossain, Antonios Anastasopoulos, Eduardo Blanco, and Alexis Palmer. 2020. It's not a non-issue: Negation as a source of error in machine translation. arXiv: 2010.05432.
Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2020. Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs. arXiv: 2010.05953.
Benjamin Jowett et al. 1892. The Dialogues of Plato: Parmenides, Theaetetus, Sophist. Statesman. Philebus, volume 4. Oxford University Press, American branch.
Nora Kassner, Benno Krojer, and Hinrich Schütze. 2020. Are pretrained language models symbolic reasoners over knowledge? In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 552-564, Online. Association for Computational Linguistics.
Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811-7818, Online. Association for Computational Linguistics.
Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, and Ellie Pavlick. 2019. Probing what different NLP tasks teach machines about function word comprehension. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM* 2019), pages 235-249, Minneapolis, Minnesota. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. arXiv: 1412.6980.
Douglas B Lenat. 1995. Cyc: A large-scale investment in knowledge infrastructure. Communications of the ACM, 38(11):33-38.
Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel. 2016. Commonsense knowledge base completion. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1445-1455, Berlin, Germany. Association for Computational Linguistics.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xi-ang Ren. 2019. Kagnet: Knowledge-aware graph networks for commonsense reasoning. In EMNLP.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv: 1907.11692.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311-318, USA. Association for Computational Linguistics.
Ian Pratt. 1994. Defeasible Inference, pages 85-107. Macmillan Education UK, London.
Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. 2019. Counterfactual story reasoning and generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5043-5053, Hong Kong, China. Association for Computational Linguistics.
A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Hannah Rashkin, Sameer Singh, and Yejin Choi. 2016. Connotation frames: A data-driven investigation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 311-321, Berlin, Germany. Association for Computational Linguistics.
Johan Reitan, Jörgen Faret, Björn Gambäck, and Lars Bungum. 2015. Negation scope detection for twitter sentiment analysis. In Proceedings of the 6th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 99-108, Lisboa, Portugal. Association for Computational Linguistics.
Hongyu Ren and Jure Leskovec. 2020. Beta embeddings for multi-hop logical reasoning in knowledge graphs. In NeurIPS.
Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4661-4675, Online. Association for Computational Linguistics.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In AAAI.
Maarten Sap, Ronan LeBras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2020. ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning. In AAAI.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiaq: Commonsense reasoning about social interactions. In EMNLP.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI.
Ming Xiang, Julian Grove, and Anastasia Giannakidou. 2016. Semantic and pragmatic processes in the comprehension of negation: An event related potential study of negative polarity sensitivity. Journal of Neurolinguistics, 38:71-88.
A.1 ANION Data Collection Details
Heuristic of Creating Logical and Semi-logical
Negation Events For logical negation, with the majority of the original events being simple sentences with one predicate, our general rule of thumb is to negate the original event at the sentence level. Specifically, with respect to each original event, we first identify each tokens' part of speech (POS) tags via the NLTK toolkit<sup>4</sup>. Then, we insert the negation cue not after the subject of each sentence, with majority of the case the entity "PersonX," with few exceptions of "PersonX's" and "PersonX and PersonY."