Introduction
Despite being a core linguistic phenomenon, negation remains a major stumbling block for modern NLP architectures (Kassner and Schütze, 2020 ; Hossain et al., 2022 ) .
A reason for this could be that texts containing negation are underrepresented in training data of language models, as humans tend to express themselves using affirmative rather than negative expressions (Ettinger, 2020 ) .
Regardless, negation has been shown to be challenging even for humans to correctly interpret due to the diversity of forms across domains (Truong et al., 2022a ) .
For instance, in clinical documents, many acronyms are used to denote negation such as NAD (no abnormality detected) , and implicit negation abounds, such as normal chest x-ray scan , which implies the absence of an abnormality.
Even more complex is the use of negation in combination with other linguistic phenomena such as quantifiers, gradable adjectives ( not unattractive does not imply attractive ) (Truong et al., 2022b ) ; licensing context (negative polarity items, e.g. any, either, yet , normally appear in certain negative grammatical contexts Warstadt et al. ( 2019 ) ); downward entailment ( A man owns a dog entails A man owns an animal but A man does not own a dog does not entail A man does not own an animal ) (Geiger et al., 2020 ) .

Traditionally, negation has been treated as a standalone problem, e.g. as negation detection (Chapman et al., 2001 ) . The investigation of the impact of negation in various downstream tasks (Hossain et al., 2022 ; Hossain and Blanco, 2022a ) , or through probing (Ettinger, 2020 ) has revealed several limitations of modern large language models (“LLMs”) in handling negation.
Given that LLMs are being adopted in an ever-growing range of tasks and have been shown to display emergent abilities for high-level tasks that require complex reasoning (Wei et al., 2022a ) , we are interested in exploring how the handling of negation has progressed.

In this work, we investigate the performance of auto-regressive language models on different negation-focused benchmarks.
Instead of just looking at samples containing negation in common NLP datasets, we consider datasets in which negation plays an important role in making the correct judgement.
In particular, we classify the benchmarks into three categories corresponding to the requisite negation reasoning abilities: (1) sensitivity to negation through cloze completion (fill-in-the-blank) queries of factual statements; (2) lexical semantics of negation through classification of antonym/synonym relationships; and (3) ability to reason with negation through language inference tasks.

We conduct extensive experiments using prompt-based learning to facilitate zero- and few-shot evaluation of LLMs, and find the following:

larger LMs are more insensitive to negation compared to smaller ones ( Section 3 );

LLMs lack lexical semantic knowledge about negation, yielding almost random performance for synonym/antonym classification ( Section 3 );

LLMs have limited ability to reason under negation, performing worse than random across most NLI datasets ( Section 3 ). Only with the latest instruction fine-tuned model (Ouyang et al., 2022 ; Chung et al., 2022 ) do we observe above-chance performance ( Section 3 );

For each dataset, we also experiment with prompt variations and find that in most cases, providing more information (context, instruction, simple wording) leads to a degradation in performance.