SlowGuess/NAACL_2022_OCR huggingface dataset:
- Agent (8%). The sixth most common argument is the person or thing who performs an event (i.e., the doer). In the example, we go from an ideal capacitor not dissipating energy to a resistor dissipating energy.
- Other (16%). Other functions (locations, purposes, recipients, etc.) account for 16% of arguments. Table 5 exemplifies a location change: from cannot flow too far from the vent to flows close to the vents.
AFIN consists of sentences containing verbal negations and their affirmative interpretations in natural language. We experiment casting the problem of obtaining affirmative interpretations from negation as a natural language inference task (Section 5.1) and as a generation task (Section 5.2).
5.1 Affirmative Interpretations and Natural Language Inference Classification
The sentences containing the (negated) target verb and the corresponding affirmative interpretations can be understood as the premises and hypotheses in a natural language inference (NLI) setting (Bowman et al., 2015). Very briefly, NLI is a classification task that determines whether a premise entails, is neutral with respect to, or contradicts a hypothesis. We label the premise-hypothesis pairs from AFIN as follows. If all the answers to questions used to generate an affirmative interpretation received the highest confidence score (4, Extremely confident), we label them entailment (85.5% of the target verbs). Otherwise (at least one answer received a confidence score between 1 and 3), we label them neutral. Note that contradiction examples cannot be derived from AFIN. Here we present two examples:
- Premise: A dormant volcano no longer shows signs of activity. Hypothesis: A dormant volcano showed signs of activity in the past. Premise entails hypothesis.
- Premise: Respiratory infections such as pneumonia do not appear to increase the risk of COPD, at least in adults. Hypothesis: Respiratory infections appear to increase the risk of COPD in elderly. Premise is neutral with respect to the hypothesis.
Category	%	Example
Patient	24	Many workers, who had nothing but their labour to sell, became factory workers out of necessity. → Many workers had only their labour to sell.
Manner	23	I don't go through life with regrets. → I go through life with satisfaction.
Quantity	10	Many mutations have no effect on the proteins they encode. → Some mutations have an effect on the proteins they encode.
Time	10	The use of asbestos is not allowed today. → The use of asbestos was allowed in the past.
Reason	9	Without water, life might not be able to exist on Earth and it certainly would not have the tremendous complexity and diversity that we see. → Earth has complexity and diversity because of water.
Agent	8	Unlike a resistor, an ideal capacitor does not dissipate energy. → A resistor dissipates energy.
Others	16	The steep sides form because the lava can not flow too far from the vent. → Lava flows close to vents.
Table 5: Analysis of the arguments that differ in the target verb and the corresponding affirmative counterpart. Categories refer to the function in the verb-argument structure. A wavy underline indicates the new argument in the affirmative counterpart.
Tested w/	RoBERTa	XLNet
P	R	F1	P	R	F1
MNLI	MNLI-dev	88	88	88	87	87	87
MNLI-dev*	92	87	89	91	85	88
AFIN	55	43	48	54	42	47
SNLI	SNLI-dev	92	92	92	91	91	91
SNLI-dev*	93	90	92	93	90	92
AFIN	56	37	45	57	38	46
RTE	RTE-dev	76	76	76	70	68	69
AFIN	52	53	52	53	55	54
Table 6: Precision, Recall, and F1 scores (macro average) obtained with RoBERTa and XLNet trained with MNLI, SNLI, and RTE. We provide results with the original development set in each benchmark, the subsets that only contain entailment and neutral pairs  $(^{*})$ , and the premise-hypothesis pairs derived from AFIN, our corpus. Transformers trained with any of the benchmarks perform substantially worse with AFIN.
Transformers and Existing NLI Benchmarks At first, we seek to investigate whether state-of-the-art transformers trained with existing NLI benchmark can solve the premise-hypothesis pairs derived from AFIN. Note that to do so, they would need to make inference in the presence of negation. We experiment with (a) two transformers: RoBERTa (Liu et al., 2019) and XLNet (Yang et al., 2019), and (b) three NLI benchmarks: MNLI (Williams et al., 2018), SNLI (Bowman et al., 2015), and RTE (part of the GLUE benchmark (Wang et al., 2018)). We fine-tuned the transformers with the training split of each benchmark and conduct three evaluations: with (a) the development split of each benchmark, (b) the subsets of (a) that only contain entailment and neutral pairs, and (c) all premise-hypothesis pairs derived from AFIN. Note that neither RTE nor AFIN have pairs
RoBERTa	XLNet
P	R	F1	P	R	F1
MNLI-training	55	43	48	54	40	46
+ 70% of AFIN	72	51	60	61	51	55
SNLI-training	58	36	45	60	38	47
+ 70% of AFIN	42	50	46	61	52	56
RTE-training	51	52	52	52	54	53
+ 70% of AFIN	56	53	54	61	55	58
Table 7: Results obtained training with (a) MNLI, SNLI, or RTE and (b)  $70\%$  of AFIN, and evaluating with  $30\%$  of AFIN. Fine-tuning improves results, but transformers substantially underperform the original development splits (see Table 6).
annotated contradiction. Appendix D.1 details the training procedure.
Table 6 presents the results. While both transformers obtain roughly the same results when evaluated with the three labels or only entailment and neutral pairs, we observe substantial drops in F1 score when evaluated with AFIN, around  $46\%$  with MNLI and  $51\%$  with SNLI.
We observe a similar pattern with RTE, although the drop is relatively small with XLNet (note, however, that XLNet does much worse than RoBERTa (69 vs. 76), whose performance drops  $32\%$ ). We hypothesize that RTE obtains better results because it does not contain contradiction pairs. These results show that current benchmarks are not enough to identify inferences between a negation and its affirmative interpretation.
Fine-tuning with AFIN The next experiments examine whether fine-tuning helps transformers identify inference relations in the premise-hypothesis pairs generated from AFIN. To do so, we fine-tune the transformers not only with an ex
BLEU-2	chrf++	METEOR
Negated sent.	26.5	50.5	43.5
+ target verb	33.6	57.3	51.9
isting benchmark (MNLI, SNLI, or RTE), but also  $70\%$  of the pairs derived from AFIN. Then, we evaluate with  $30\%$  of the pairs derived from AFIN.
Table 7 presents the results. Perhaps unsurprisingly, fine-tuning with AFIN allows the transformers to correctly identify few more entailment and neutral pairs (F1 scores: 45-53 vs. 46-60). We note, however, that no matter how we combine transformers and NLI benchmarks, the results are substantially below those obtained with the original development split (F1 scores: 46-60 vs. 69-92).
5.2 Generating Affirmative Interpretations
Casting the problem as a natural language inference task is worthwhile but unrealistic: the affirmative interpretations to be verified (are they entailed by the sentence with the negation?) are not readily available. In our next experiments, we investigate a realistic formulation of the problem: generate affirmative interpretations given a sentence with a negation. In order to do so, we split AFIN as follows:  $70\%$  for training,  $15\%$  for development, and the remaining  $15\%$  for test.
Experimental Setup We perform the experiments with the T5-Large transformer (Raffel et al., 2020), which can generate text through a supervised learning setup. In particular, we train T5 to generate affirmative interpretations using two inputs: (a) only the sentence containing the (negated) target verb (i.e., the negated sentence), and (b) the negated sentence concatenated with the target verb. The second setup investigates whether inputting the target verb with the negated sentence aids in generating affirmative interpretation about that target verb. Additional details on the training procedure are provided in Appendix D.2.
Results and Analysis After the training process with both setups, we obtain evaluation scores using three automatic metrics: BLEU-2 (Papineni et al., 2002), chrf++ (Popovic, 2017), and METEOR (Banerjee and Lavie, 2005). We calculate these metrics comparing the human- and T5-generated affirmative interpretations from the test split (Table 8). Evidently, the system provided with the
Table 8: Evaluation results obtained with BLEU-2, chrf++, and METEOR between human and system generated affirmative interpretations.
Confidence Scores
4	3	2	1	0
AFIN (upper bound)	86.2	11.6	2.0	0.2	n/a
T5-Large
Negated sent.	32.0	15.3	12.0	3.3	37.3
+ target verb	43.3	10.0	15.3	4.0	27.3
Table 9: Percentage of affirmative interpretations assigned each confidence score in (a) the AFIN test set and (b) those generated by T5 (not providing and providing the target verb). T5 substantially underperforms AFIN, which is a human upper bound.
target verb shows comparatively better scores than the system without the target verb (e.g., BLEU-2: 33.6 vs 26.5). Based on the scores from the best setting, T5 achieves some capability to automatically generate affirmative interpretations.
While useful, automatic metrics only provide a partial picture about the quality of affirmative interpretations, as outlined in Section 3.2. Therefore, we manually evaluate the affirmative interpretations generated by T5. In particular, the same annotator that validated a sample of AFIN validated the output of T5 with the confidence scores provided in Section 3.2. Note that this time we added a new score of 0 to indicate that an affirmative interpretation is incorrect.
Table 9 provides the results. The scores assigned to AFIN represent an upper bound. We observe that explicitly providing the (negated) target verb is beneficial as it allows T5 to generate many more extremely confident affirmative interpretations (32% vs. 43.3%). We observe, however, that T5 faces challenges generating affirmative interpretations. First, over a quarter (27.3%) are incorrect. Second, compared to AFIN (i.e., human annotators), T5 only generates about half (43.3% vs. 86.2%) of affirmative interpretations that an evaluator is extremely confident about (confidence score: 4).
Qualitative Analysis In addition to confidence scores, we also analyze when T5 faces the biggest challenges generating affirmative interpretations. To this end, we randomly selected 150 instances from the test split. Then, we manually annotated the functions of the arguments that should be replaced in the affirmative interpretations with the same categories than the ones discussed in Section 4. We present the confidence score analysis in Fig-
2The only difference is that the affirmative interpretations come from T5 instead of a human annotator.
Figure 3: Analysis of scores assigned to the affirmative interpretations generated by T5. Scores are much lower when the argument that has to be changed to generate the affirmative interpretation is an agent or patient.
ure 3. For convenience, we show scores in three groups: certainly true (score: 4), might be true (scores from 1 to 3), and not true (score: 0).
We observe that it is comparatively easy for T5 to generate certainly true affirmative interpretations when the argument to be replaced contains a quantity (certainly true vs. not true:  $60\%$  vs  $13.3\%$ ). Therefore, T5 learned some patterns to replace quantities in the affirmative interpretations. For example, from negation "Schools can not charge students more than US $5 to defray the cost of insurance," T5 correctly generates "Schools can charge students US$ 5 to cover the cost of insurance." Despite the relatively success with quantities, less than  $50\%$  of all affirmative interpretations that require replacing an argument in any other category are deemed certainly correct. Agent and patient are the categories T5 finds most challenging—these affirmative interpretations are more often deemed not true than certainly true. T5 often generates affirmative interpretations in these categories by deleting the negation cue and fixing verb tense and auxiliaries to form a grammatical—but incorrect—affirmative interpretation. For example, given "Ryanair have also sacked veteran pilot John Goss for appearing on the show, the only pilot interviewed who did not seek anonymity," T5 generates "Veteran pilot John Gosson sought anonymity."
We have proposed a question-answer driven approach to reveal affirmative interpretations from verbal negations. Annotators generate and answer questions regarding the affirmative counterpart of a negated verb, and then we generate from them an af
firmative counterpart in natural language. Through analyses, we have shown that  $67.1\%$  of verbal negations convey that the negated event is actually factual. More importantly, we observe many categories in the arguments that are replaced in the affirmative interpretations (patient, manner, quantity, time, reason, etc.). The experiments show that transformers struggle substantially when we cast the problem as NLI. Doing so, however, is an unrealistic scenario: affirmative interpretations are not readily available to be fed into a natural language inference classifier. Further, we observe very limited success generating affirmative interpretations given as input a sentence containing verbal negation. We argue that generating affirmative interpretation is the realistic scenario and propose doing so as a challenging generation task requiring a combination of language comprehension, commonsense, and world knowledge currently out of reach for state-of-the-art models.
This material is based upon work supported by the National Science Foundation under Grant No. 1845757. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF. The Titan Xp used for this research was donated by the NVIDIA Corporation. Computational resources were also provided by the UNT office of High-Performance Computing. Additionally, we utilized computational resources from the Chameleon platform (Keahey et al., 2020). We also thank the anonymous reviewers for their insightful comments.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72.
Eduardo Blanco and Dan Moldovan. 2011. Semantic representation of negation using focus detection. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 581-589, Portland, Oregon, USA. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno
tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Federico Fancellu, Adam Lopez, and Bonnie Webber. 2016. Neural networks for negation scope detection. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 495-504, Berlin, Germany. Association for Computational Linguistics.
Federico Fancellu, Adam Lopez, Bonnie Webber, and Hangfeng He. 2017. Detecting negation scope is easy, except when it isn't. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 58-63, Valencia, Spain. Association for Computational Linguistics.
Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. 2018. Large-scale QA-SRL parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2051-2060, Melbourne, Australia. Association for Computational Linguistics.
Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 643-653, Lisbon, Portugal. Association for Computational Linguistics.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spacy: Industrialstrength natural language processing in python, 2020. URL https://doi.org/10.5281/zenodo, 1212303(6).
Laurence Horn. 1989. A Natural History of Negation. University of Chicago Press.
Laurence R. Horn and Heinrich Wansing. 2020. Negation. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, Spring 2020 edition. Metaphysics Research Lab, Stanford University.
Md Mosharaf Hossain, Kathleen Hamilton, Alexis Palmer, and Eduardo Blanco. 2020. Predicting the focus of negation: Model and error analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8389-8401, Online. Association for Computational Linguistics.
Rodney D. Huddleston and Geoffrey K. Pullum. 2002. The Cambridge Grammar of the English Language. Cambridge University Press.
Liwei Jiang, Antoine Bosselut, Chandra Bhagavatula, and Yejin Choi. 2021. "I'm not mad": Commonsense implications of negation and contradiction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, pages 4380-4397, Online. Association for Computational Linguistics.
Kate Keahey, Jason Anderson, Zhuo Zhen, Pierre Riteau, Paul Ruth, Dan Stanzione, Mert Cevik, Jacob Colleran, Haryadi S. Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, François Halbach, Alex Rocha, and Joe Stubbs. 2020. Lessons learned from the chameleon testbed. In Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC '20). USENIX Association.
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, pages 4999-5007.
Emanuele Lapponi, Erik Velldal, Lilja Øvrelid, and Jonathon Read. 2012. UiO 2: Sequence-labeling negation using dependency features. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 319–327, Montréal, Canada. Association for Computational Linguistics.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122-2132, Austin, Texas. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Roser Morante and Walter Daelemans. 2009. A metalearning approach to processing the scope of negation. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 21–29, Boulder, Colorado. Association for Computational Linguistics.
Roser Morante and Walter Daelemans. 2012. ConanDoyle-neg: Annotation of negation cues and their scope in Conan Doyle stories. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012). European Language Resources Association (ELRA).
Woodley Packard, Emily M. Bender, Jonathon Read, Stephan Oepen, and Rebecca Dridan. 2014. Simple negation scope resolution through deep parsing: A semantic solution to a semantic problem. In Proceedings of the 52nd Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 69-78, Baltimore, Maryland. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Maja Popovic. 2017. chrf++: words helping character n-grams. In Proceedings of the second conference on machine translation, pages 612-618.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Zahra Sarabi, Erin Killian, Eduardo Blanco, and Alexis Palmer. 2019. A corpus of negations and their underlying positive interpretations. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM* 2019), pages 158-167, Minneapolis, Minnesota. Association for Computational Linguistics.
Longxiang Shen, Bowei Zou, Yu Hong, Guodong Zhou, Qiaoming Zhu, and AiTi Aw. 2019. Negative focus detection via contextual attention mechanism. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2251-2261, Hong Kong, China. Association for Computational Linguistics.
György Szarvas, Veronika Vincze, Richard Farkas, and János Csrik. 2008. The bioscope corpus: Annotation for negation, uncertainty and their scope in biomedical texts. In Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing, BioNLP '08, pages 38-45, Stroudsburg, PA, USA. Association for Computational Linguistics.
Veronika Vincze, György Szarvas, Richard Farkas, György Móra, and János Csrík. 2008. The BioScope corpus: biomedical texts annotated for uncertainty, negation and their scopes. BMC bioinformatics, 9(11):S9.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. *ArXiv*, abs/1910.03771.
Bowei Zou, Guodong Zhou, and Qiaoming Zhu. 2014. Negation focus identification with contextual discourse information. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 522-530, Baltimore, Maryland. Association for Computational Linguistics.
Bowei Zou, Guodong Zhou, and Qiaoming Zhu. 2015. Unsupervised negation focus identification with word-topic graph model. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1632-1636, Lisbon, Portugal. Association for Computational Linguistics.
A Additional Details on Template-based Question Generation
This section provides additional details for the slots in the template-based question generation presented in Section 3.2 of the paper.
- WH indicates wh-words to generate the argument questions. The complete set of options we use are as follows: who, what, whom, when, where, how, how much, how many, how long, how often, and why.
- AUX indicates auxiliary verbs. The predicate questions always start with an auxiliary. However, argument questions may or may not contain an auxiliary verb (See examples in Table 3 of the paper). We avail the below list of auxiliary verbs for annotators: is, was, does, did, has, had, can, could, may, might, will, would, should, and must.
- SUB refers to subjects. Similar to He et al. (2015), we only avail someone or something, indicating placeholder for the subject position.
- VERB indicates the full conjugation of the target verb.