Paraphrasing in Affirmative Terms Improves Negation Understanding (2024)

Abstract: Negation is a common linguistic phenomenon. Yet language models face challenges with negation in many natural language understanding tasks such as question answering and natural language inference. In this paper, we experiment with seamless strategies that incorporate affirmative interpretations (i.e., paraphrases without negation) to make models more robust against negation. Crucially, our affirmative interpretations are obtained automatically. We show improvements with CondaQA, a large corpus requiring reasoning with negation, and five natural language understanding tasks.

heading: 1 Introduction
text: Negation is a fundamental linguistic phenomenon present in all human languages (Horn, 1989). Language models underperform in various natural language understanding (NLU) tasks when the input includes negation. For example, Ettinger (2020) and Kassner and Schütze (2020) show that BERT (Devlin et al., 2019) fails to distinguish between negated and non-negated cloze questions. Researchers have also shown that large language models such as GPT-3 (Brown et al., 2020) and InstructGPT (Ouyang et al., 2022) are insensitive to negation and fail to reason under negation (Truong et al., 2023). Jang et al. (2022) point out that language models violate the logical negation property (p is true iff ¬p is false). Hossain et al. (2022a) analyze negation in eight popular corpora for six NLU tasks. They conclude that (a) NLU corpora have few negations compared to general-purpose texts and (b) the few negations in them are often unimportant. To our knowledge, CondaQA (Ravichander et al., 2022) is the largest benchmark (14,182 question-answer pairs from Wikipedia) requiring reasoning over the implications of negations.
In this paper, we paraphrase sentences with negation without using negation to make models for
natural language understanding more robust when negation is present in the input. We will use the term affirmative interpretation to refer to paraphrases without negation (e.g., I am not sad: I am just ok, I am happy, etc.). Appendix A provides examples of how affirmative interpretations differ from simple paraphrases.
The main contributions of this paper are (a) strategies to generate and incorporate affirmative interpretations and (b) experimental results demonstrating that doing so yields better results.1
In addition to CondaQA, we experiment with five of the eight corpora analyzed by Hossain et al. (2022a): CommonsenseQA (Talmor et al., 2019), STS-B (Cer et al., 2017), QNLI (Rajpurkar et al., 2016), WiC (Pilehvar and Camacho-Collados, 2019), and WSC (Levesque et al., 2012).2 We do not experiment with the other three corpora because they do not contain any negation (Roemmele et al., 2011, COPA), there is no difference in results when negation is present (Cer et al., 2017, QQP; 0.01 in macro F1), or has already been shown (Hossain and Blanco, 2022) to benefit from affirmative interpretations (Socher et al., 2013, SST-2). The corpora we experiment with are in English.
Related Work Early research on negation targeted detecting negating cues and generating semantic representations, usually by identifying the scope and focus (Morante et al., 2011; Morante and Daelemans, 2012; van Son et al., 2016; Khandelwal and Sawant, 2020; Truong et al., 2022).
More recent works bypass formal representations. Instead, they make neural models robust when the input contains negation. Hosseini et al. (2021) combine unlikelihood training and syntactic data augmentation to enhance the ability of BERT to understand negation with negated LAMA (Kass-
1Code available at https://github.com/mhrezaei1/ paraphrase-affirmative under Apache 2.0 license.
2See examples from these corpora in Appendix B.
602
ner and Schütze, 2020). Singh et al. (2023) present a pretraining strategy designed for negation. Unlike these works, we couple original inputs containing negation with affirmative interpretations.
The first work on affirmative interpretations was by Sarabi et al. (2019). Hossain et al. (2022b) present AFIN, a corpus of ≈3,000 sentences with negations and their affirmative interpretations. These two previous works are limited to generating affirmative interpretations from negations; they do not provide extrinsic evaluations. More recently, Hossain and Blanco (2022) present Large-AFIN, over 153,000 pairs of sentences with negation and their affirmative interpretations obtained from parallel corpora via backtranslation. In this paper, we present strategies to generate affirmative interpretations that do not require parallel corpora or a machine translation system. Moreover, we demonstrate that incorporating affirmative interpretations yields better results with CondaQA and five other natural language understanding tasks.

heading: 2 Generating Affirmative Interpretations
text: An affirmative interpretation generator is a system that takes a sentence with negation as its input and outputs an affirmative interpretation. The task is similar to paraphrase generation with an additional constraint: the output must not contain negation.
We use two approaches to generate affirmative interpretations. The first one is an off-the-shelf T5 (Raffel et al., 2020) fine-tuned by Hossain and Blanco (2022) with Large-AFIN (Section 1) to generate affirmative interpretations. We refer to this model as T5-HB, and to the affirmative interpretations generated by T5-HB as AHB.
The second approach bypasses the need for a large collection of pairs of sentences with negation and their affirmative interpretations. It is based on the work by Vorobev and Kuznetsov (2023), who fine-tuned T5 on a paraphrase dataset obtained with ChatGPT (419,197 sentences and five paraphrases per sentence). We refer to this model as T5-CG. Note that it is trained to generate paraphrases—not affirmative interpretations. We obtain affirmative interpretations with T5-CG by generating five paraphrases and selecting the first one that does not contain negation. We refer to these affirmative interpretations as ACG.3 For examples of AHB and ACG, see Appendix D.
3At the time of writing, ChatGPT cannot reliably paraphrase without negation. See an example in Appendix C
We use all negation cues in CondaQA to identify negation cues in our experiments. CondaQA contains over 200 unique cues, including single words (e.g., inaction, unassisted, unknown), affixal negations (e.g., dislike, unmyelinated, unconnected, inadequate, impartial), and multiword expressions (e.g., a lack of, in the absence of, no longer, not at all, rather than). They also include multiple part-of-speech tags such as nouns (e.g., absence, nobody, inability), adverbs (e.g., indirectly, involuntarily, unexpectedly), determiners (e.g., neither, no, none), and verbs (e.g., cannot, refuse, exclude).

heading: 3 Experimental Results
text: We use RoBERTa-Large (Liu et al., 2019) as the base model. In addition to experimenting with the original inputs for a task (e.g., passage and question from CondaQA), we couple the original input with one affirmative interpretation of the sentence with negation (if any; no change otherwise). Affirmative interpretations are concatenated to the original input after the <sep> special token. Our approach is the same regardless of the type of negation. For implementation details, see Appendix E and F.

heading: 3.1 CondaQA
text: CondaQA (Ravichander et al., 2022) is a questionanswering dataset that requires reasoning over negation. It was created by asking crowdworkers to write questions about a negated sentence within a paragraph retrieved from Wikipedia. Crowdworkers also made three edits to the original paragraph:
1. Paraphrase Edit: Paraphrase the negation. 2. Scope Edit: Change the scope of the negation. 3. Affirmative Edit: Remove the negation.
Additionally, they answered the question based on the original passage and all three edited passages. (see examples in Appendix G). Note that paraphrase edits preserve meaning thus answers remain unchanged. On the other hand, scope edits change meaning but the answer may or may not remain the same. Finally, affirmative edits reverse meaning thus answers are also reversed.
Paraphrase edits are not the same as our affirmative interpretations—crowdworkers were not asked to paraphrase without using negation. We discovered, however, that 40.5% of these edits satisfy our definition of affirmative interpretation. We believe crowdworkers simply found it intuitive to paraphrase the negation without using negation. We refer to these affirmative interpretations as AG (Gold)
Input Representation Acc. Group Consistency
# Pars. Training Testing All Par. Sco. Aff.
From Ravichander et al. (2022) RoBERTa-Large 355M P+Q P+Q 54.1 13.6 51.6 26.5 27.2 UnifiedQA-v2-Base 220M P+Q P+Q 58.0 17.5 54.6 30.4 33.0 UnifiedQA-v2-Large 770M P+Q P+Q 66.7 30.2 64.0 43.7 46.5 UnifiedQA-v2-3B 3B P+Q P+Q 73.3 42.2 72.8 55.7 57.2
and only use them for training purposes, as using them at prediction time would be unrealistic.
Our evaluation reuses the metrics proposed by the authors of CondaQA: accuracy and group consistency. Group consistency is the percentage of questions answered correctly for all the passages in a group. The groups include the original passage and either all three or one of the edited passages.
Table 1 summarizes the experimental results (see Appendix H for additional results). Our implementation of RoBERTa-Large obtains substantially better results than those by Ravichander et al. (2022, Acc.: 64.9 vs. 54.1). Reviewing the training details revealed that the difference is that they stop training after ten epochs while we use early stopping and stop after 18 epochs.
The best-performing model in terms of accuracy is UnifiedQA-v2-3B (Khashabi et al., 2022), which is a 3B-parameter T5 model pre-trained on 20 question-answering corpora data (≈1M, Appendix I). Smaller versions of UnifiedQA (220M and 770M parameters) obtain substantially lower results despite being trained with the same cor-
pora (Acc.: 58.0 and 66.7). Our implementation of RoBERTa-Large using the question and passage as input almost rivals UnifiedQA-v2-Large (64.9 vs. 66.7) despite the latter having twice the size and being fine-tuned with ≈1M question-answering pairs.
Coupling the original input (passage and question) with either the sentence that contains negation (S) or the first paraphrase obtained with T5-CG with no effort to avoid negation (SCG) brings minor improvements (64.9 vs. 65.2, 65.7). More interestingly, incorporating affirmative interpretations brings statistically significantly better results (64.9 vs. 67.1 (AHB), 66.4 (ACG) and 66.7 (AG or ACG/ACG)). We conclude the following from the results:
• The benefits of affirmative interpretations are not due to pinpointing the sentence within the passage that is most relevant to answer the question (P+Q+S vs. P+Q+SCG vs. P+Q+AHB). • Training with affirmative interpretations is always beneficial as long as they are also used at prediction time. Note that we only use automatically obtained affirmative interpretations (all but AG) at testing time. However,
using both of them together does not yield better results (Acc.: 65.6 vs. 66.4 and 67.1).
• At training time, complementing AG (available for ≈40% of paraphrase edits) with AHB or ACG is beneficial (last and second-to-last block).
Qualitative and Error Analysis Manual analysis of 100 samples from CondaQA reveals that ACG contains less negations than AHB (46% vs. 23%). ACG, however, contains less meaning-preserving paraphrases (36% vs. 17%). On the other hand, paraphrases in SCG rarely do not preserve meaning (10%) but often include negation (60%). (Table 3). Sometimes it is not natural to rewrite a sentence without negation (e.g., The inner membrane is rich in an unusual phospholipid, cardiolipin.) Out of the 23 samples where AHB contains negation, a hu-
man was able to rewrite 15 of them without negation. Combined with the results from Table 1, this analysis leads to the conclusion that affirmative interpretations are beneficial despite being noisy.
We also analyzed 50 samples of the errors made representing the input with P+Q that are fixed using affirmative interpretations from AHB. A negated adjective is replaced by its affirmative counterpart (e.g., not happy → sad) in 48% of cases. Table 2 shows the analysis and examples of negated sentences and their AHB affirmative interpretations.

heading: 3.2 Other NLU Tasks
text: We experiment with five additional NLU tasks to evaluate the benefits of affirmative interpretations. We access these corpora through the GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) benchmarks. We report results on the development set of each corpus, given that the test sets are not publicly available. In addition, we report the results for important and non-important instances as identified by Hossain et al. (2022a). They consider a negation unimportant if one can disregard it and still make the correct prediction. For example, John didn’t eat the steak with gusto (most likely) entails John ate meat even if one disregards the negation.
Table 4 presents the results. Incorporating affirmative interpretations (AHB or ACG) improves perfor-
CmnsnsQA STS-B QNLI WiC WSC
F1 Prsn Sprmn F1 F1 F1
RoBERTa 0.70 0.92 0.92 0.93 0.71 0.69 instances without negation 0.69 0.92 0.92 0.93 0.71 0.67 instances with negation 0.73 0.88 0.88 0.92 0.66 0.71
Important 0.67 0.82 0.85 0.78 n/a n/a Unimportant 0.80 0.88 0.88 0.92 0.66 0.71
mance across all corpora with instances containing important negations; the only exception is STSB with AHB (Spearman: -1.2%) and ACG (Pearson: no difference). It is worth noting that WiC and WSC have no important negations, yet either AHB or ACG yield substantial improvements with unimportant negations (4.2–6.1%). Surprisingly, we found that incorporating affirmative interpretations is beneficial for instances without negation across all corpora except WSC with AHB.
These experiments demonstrate that incorporating affirmative interpretations not only obtains higher or comparable results with instances containing important negations, but also often improves results with instances not containing negation.

heading: 4 Conclusion
text: We have presented two strategies to generate and incorporate affirmative interpretations into models for natural language understanding. The idea is simple yet effective: complement inputs that contain negation with a paraphrase that does not contain negation. Crucially, we have demonstrated that automatically obtained (noisy) affirmative interpretations yield improvements with (a) CondaQA compared with a model with twice as many parameters pre-trained with ≈1M question-answer pairs from 20 existing corpora and (b) five NLU tasks. Our methodology is architecture- and task-agnostic. In fact, the model to generate affirmative interpretations was tuned with out-of-domain corpora.
Future Work. The methods we have presented are simple and effective, but they are not the only way to incorporate or generate affirmative interpretations. For example, one might be able to use LLMs such as GPT-4 or Llama to generate affirmative interpretations. Another interesting direction is to investigate the effect of affirmative interpretations on other NLU tasks, such as sentiment analysis or text classification. Finally, it would be interesting to investigate the effect of affirmative interpretations on other languages, especially those with different word order or negation structures.
Limitations
The scope of this paper is limited to question answering (CondaQA) and natural language understanding (five tasks and corpora) in English with an emphasis on negation. We leave for future work the task of exploring whether affirmative interpretations are beneficial in other languages. We acknowledge that this strategy might not generalize to other languages.
We also acknowledge that we did not conduct experiments with the latest GPT models or spend substantial amounts of time engineering prompts. We note, however, that good faith efforts using prompts showed that ChatGPT may not be well suited for generating affirmative interpretations at this time (Appendix C).
It is worth pointing out that writing affirmative interpretations for negated sentences might not be
straightforward or even possible in some cases. In this paper, we did not focus on the task of determining whether a sentence can be paraphrased without negation. We leave this for future work.
None of the corpora that we work with include information about the scope and focus of negation. Therefore, we do not have any insight into the relation between affirmative interpretations and the scope and focus of a negation.
Ethics Statement
The work in this paper does not involve human subjects. We only use publicly available datasets and models. We do not collect any personal information. Therefore, this work does not raise any ethical concerns.

heading: Acknowledgements
text: This material is based upon work supported by the National Science Foundation under Grant No. 2310334. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.
We used computational resources available at the Chameleon testbed to run our experiments (Keahey et al., 2020). We are also grateful to the anonymous reviewers for their valuable comments.

heading: A Paraphrases vs. Affirmative Interpretations
text: Affirmative interpretations are paraphrases without negation. Table 5 shows examples of automatically generated paraphrases from a negated sentence. Not all of them are correct affirmative interpretations: some (a) contain negation or (b) do not preserve the meaning of the original sentence with negation (and thus they are not actual paraphrases to begin with). The definition of affirmative interpretation is a paraphrase (i.e., rewording that preserves meaning) not containing negation.
Note that an automatically obtained paraphrase that does not preserve the full meaning (and thus does not satisfy the definition of affirmative interpretation) does not necessarily contradict the meaning of the original sentence with negation. For example, I stayed home today is not a true paraphrase of I didn’t go shopping today but is not a contradiction either. In this example, obtaining I stayed home today, despite being only plausible and not a paraphrase of I didn’t go shopping today, could be useful to answer questions such as “Did I go shopping today?” as staying home contradicts going shopping.

heading: B NLU Corpora
text: Table 6 shows examples from the five NLU corpora that we experiment with. The examples are from the development set of each corpus. In our experiments, we append the affirmative interpretation of the negated sentence in the input to the end of the input after a special token.

heading: C Attempting to Generate Affirmative Interpretations with ChatGPT
text: At the time of writing, ChatGPT cannot reliably generate affirmative interpretations (i.e., paraphrase without using negation). In the example in Figure 1, it appears convinced to be able to do so, yet it clearly fails: unhappy and lack are negations. Perhaps surprisingly, ChatGPT appears to know that the generated output does contain negation.

heading: D Affirmative Interpretations Examples
text: Table 7 shows two negated sentences and their automatically obtained affirmative interpretations. The bottom half of the table includes errors, as the automatically generated affirmative interpretations contain negations. Table 8 contains a couple examples from CondaQA in which the process to generate affirmative interpretations made mistakes along with our observations.

heading: E Training Details with CondaQA
text: We use the RoBERTa-Large model (Liu et al., 2019) for our experiments with CondaQA. We use the implementation of RoBERTa-Large in the HuggingFace Transformers library (Wolf et al., 2020). The model is trained using early stopping with a patience of 3 epochs and batch size 16. Table 9 shows the learning rates that we used for our experiments with CondaQA. We use the default values
for the other hyperparameters.

heading: F Training Details with Additional NLU Tasks
text: We use the implementation by Phang et al. (2020) with RoBERTa-Large as the base model. We use the default values for the hyperparameters, with the exception of the learning rate, batch size and maximum number of epochs for early stopping.
Table 10 shows the learning rates and batch sizes that we used for our experiments on each corpus.

heading: G CondaQA Dataset
text: Figure 2 shows an example from CondaQA. Note that CondaQA highlights the original negated sentences from the original passages but not the edited sentences. However, we use the available information in the dataset such as the original sentence, the original passage and the edited passage to extract the edited sentences. Specifically, we identify sentence boundaries in the original passage and pinpoint the index of the sentence that contains negation. Then, we identify sentence boundaries in the edited passage and use the same index to extract the edited sentence. We use the extracted edited sentence to generate affirmative interpretations. The authors manually analyzed 100 samples of the extracted edited sentences and confirmed that in 96% of the cases, the extracted edited sentences are the same as the edited sentences in the passage.
Additionally, Table 11 shows the basic properties of the edits made by crowdworkers.

heading: H Additional Results with CondaQA
text: Table 12 shows additional results with RoBERTaLarge and CondaQA for each edit type. The results show that incorporating affirmative interpretations with RoBERTa-Large improves results not only with the entire test set, but also with each edit type individually. However, not all of the improvements are statistically significant. The only statistically significant improvements are with (1) the scope edit type when trained with P+Q+ACG or ACG and tested with P+Q+ACG, and (2) the affirmative edit type when trained with P+Q+AHB and tested with P+Q+AHB.

heading: I UnifiedQA-v2 Training Corpora
text: Table 13 shows the QA corpora that Khashabi et al. (2022) used to train UnifiedQA-v2. These corpora
span the following QA formats: extractive, abstractive, multiple-choice, and yes-no questions.
Input Representation Accuracy
# Params. Training Testing All Ori. Par. Sco. Aff.
RoBERTa-Large 355M Q Q 47.4 52.1 52.3 47.4 39.0 P P 45.4 46.5 46.1 45.2 43.9

heading: Future Work
text: Future Work. The methods we have presented are simple and effective, but they are not the only way to incorporate or generate affirmative interpretations. For example, one might be able to use LLMs such as GPT-4 or Llama to generate affirmative interpretations. Another interesting direction is to investigate the effect of affirmative interpretations on other NLU tasks, such as sentiment analysis or text classification. Finally, it would be interesting to investigate the effect of affirmative interpretations on other languages, especially those with different word order or negation structures. Limitations
The scope of this paper is limited to question answering (CondaQA) and natural language understanding (five tasks and corpora) in English with an emphasis on negation. We leave for future work the task of exploring whether affirmative interpretations are beneficial in other languages. We acknowledge that this strategy might not generalize to other languages. We also acknowledge that we did not conduct experiments with the latest GPT models or spend substantial amounts of time engineering prompts. We note, however, that good faith efforts using prompts showed that ChatGPT may not be well suited for generating affirmative interpretations at this time (Appendix C). It is worth pointing out that writing affirmative interpretations for negated sentences might not be
straightforward or even possible in some cases. In this paper, we did not focus on the task of determining whether a sentence can be paraphrased without negation. We leave this for future work. None of the corpora that we work with include information about the scope and focus of negation. Therefore, we do not have any insight into the relation between affirmative interpretations and the scope and focus of a negation