This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models
Authors: Iker García-Ferrero
Begoña Altuna
Javier Alvez
Itziar Gonzalez-Dios
German Rigau
Venue: EMNLP 2023 Main (2023)
Keywords: negation
dataset
LLM
commonsense
evaluation
foundation models
WordNet
real-word knowledge
Large Language models
TL;DR: We introduce a large semi-automatically generated dataset of ~400,000 descriptive sentences about commonsense knowledge that can be true or false in which negation is present in about 2/3 of the corpus in different forms that we use to evaluate LLMs

Abstract: Although large language models (LLMs) have apparently acquired a certain level of grammatical knowledge and the ability to make generalizations, they fail to interpret negation, a crucial step in Natural Language Processing. We try to clarify the reasons for the sub-optimal performance of LLMs understanding negation. We introduce a large semi-automatically generated dataset of circa 400,000 descriptive sentences about commonsense knowledge that can be true or false in which negation is present in about 2/3 of the corpus in different forms. We have used our dataset with the largest available open LLMs in a zero-shot approach to grasp their generalization and inference capability and we have also fine-tuned some of the models to assess whether the understanding of negation can be trained. Our findings show that, while LLMs are proficient at classifying affirmative sentences, they struggle with negative sentences and lack a deep understanding of negation, often relying on superficial cues. Although fine-tuning the models on negative sentences improves their performance, the lack of generalization in handling negation is persistent, highlighting the ongoing challenges of LLMs regarding negation understanding and generalization. The dataset and code are publicly available.

This is not a Dataset:
A Large Negation Benchmark to Challenge Large Language Models
Iker García-Fereero' Begoña Altuna' Javier Álvez?
Itziar Gonzalez-Dios German Rigau'
HiTZ Center Ixa, University of the Basque Country UPV/EHU
LoRea Group, University of the Basque Country UPV/EHU
[iker. garciaf, begona al altuna, javier. alvez)@ehu. eus eus (itziar.gonzzl german. r Aeeu. eus
Abstract
Although large language models (LLMs) have apparently acquired certain level of grammat- ical knowledge and the ability to make general- izations, they fail interpret negation, a crucial step in Natural Language Processing. We to clarify the reasons for the sub-optimal perfor- mance of LLMs understanding negation. We introduce a large semi-aute aoni-automaticalyy generated dataset of circa 400,000 descriptive sentences about commonsense knowledge that can be true or false in which negation is present in about 2/3 corpus in different forms. We have used our dataset with the largest available open LLMs in a zero-shot approach to grasp their generalization and inference capability and have also fine-tuned some of the models to as- sess whether the understanding of negation can he trained. Our findings show that, while L.LMs are proficient at classifying affirmative sen- tences, they struggle with negative sentences and lack deep understanding of negation, of- ten relying on superficial cues. Although fine- tuning the models on negative sentences im- proves their performance, the lack of general- ization in handling negation is persistent, high- lighting the ongoing challenges of LLMs re- garding negation understanding and generaliza- tion. The dataset and code are publicly avail- com/hhiz-zeetroa/
birds.
Bila are commonty of tumann bockes. X
bldss. X
Blll
Figure 1: Affirmative and negative sentences in the dataset.
The presence of negation in sentence reverts the polarity of the proposition it represents, and thus affects its truth and factuality values. See how the adverb "never" changes the truth value of the sentences in Figure As a consequence, under- standing negation correctly is crucial for all NLP tasks. Morcover, understanding negation should help LLMs to grasp how things happen in real- ity, boosting NLP tasks that involve commonsense, causality, entailment and world knowledge.
The reasons for the lower capabilities of LLMs dealing with negation remain largely unclear, al- though some point out at the under-representation of negation in corpora (Hossain et al., 2022). In this work, we present corpus in which negation is present in around two thirds of the sentences in different forms. Taking advantage of the relations in WordNet (Fellbaum, 1998), we have generated a set of patterns to create descriptive sentences that
able: https://gite This-is-not- a Dataset
1 Introduction
Large Language Models (LLMs)) currently offer state of the art performance in many Natural Lan- guage Processing (NLP) tasks. Apparently, they have acquired the ability to capture syntactic (Ba- roni, 2020) and semantic (Furrer et al., 2021) ab- stractions. However, recent experiments (Kassner and Schütze, 2020; Hossain al.. 2020; Truong et al., 2022) have proven that LLMMs fail inter- preting contexts in which understanding negation is required.
work as truth and falsity tests which are then used together with a list of prompts to measure the sen- tence understanding of the different LLMs.
The dataset has been used in a series of experi- ments to test its quality and coherence. First, we assess the quality of the sentences by human anno-

---

tators. Then, to grasp its capacity of generalization and inference, we have used our dataset test dif- ferent configurations of LLMs available zero- shot approach. We have also fine-tuned some of these models to assess whether the understanding of negation can be learnt. Our initial hypothesis is that if the dataset is coherently and robustly built we will be able to learn how LLMs deal with nega- tion.
negation licensing strategies measure neural lan- guage model ability to correctly process them.
Chen et al. (2023) assess the ability of LLMs to handle negative commonsense knowledge. Since most available information exists in a positive and affirmative form, LLMs fail at dealing with world knowledge when it is presented in a nega- tive form. They propose two-task assessment in which LLMs need to i) answer yes or no world knowledge questions and ii) generate com- monsense compelling sentences from related key- words. Some recent research has been directed to building knowledge bases in which negative com- monsense is stored (Araaout et al., 2022), in order to be reused for commonsense reasoning.
The contributions of this paper are: i) We in- troduce the largest negation probing dataset. This dataset includes affirmative and negative sentences with and without distractors, incorporating multi- ple types of relations and negations. ii) We evalu- ate comprehensive set of open LLMs using our dataset in both zero-shot and fine-tuning scenarios. iii) Our findings demonstrate that current LLMs, whether in zero-shot settings or after fine-tuning with examples from our dataset, possess a profound understanding of the truthfulness of uffirmative sen- tences. However, when confronted with negation, these models heavily rely on superficial cues in- stead of effectively generalizing negation.
2.2 Negation in English
Negation in language is the representation of the logical operation in which the truth value of a proposition is inverted. It is commonly expressed by a restricted list of negative adverbs (e.g. no, never), pronouns, determiners or prefixes, that ap- pear in different contexts the sentence. Pullum et al. (2002) offer a four axe classification of nega- tion types from which we will focus on these:
2 Background
2.1 Related Works
Verbal vs. non-verbal: the verbal negation marker is associated with the verb and directly affects it, while the non-verbal couples with objects and adjuncts.
Negation is a core operator in logic and in the struc- turing of the information in text and it has long been studied for its relevance in natural language understanding. In the last two decades, works on the analysis and processing of negation have multi- era, most works
Analytic vs. synthetic: analytic negation is represented by markers that only convey nega- tion. Synthetic negation markers, instead, may have additional syntactic function (c.g. noth- ing and none might also be subjects or ob- jects).
plied. In the pre-generative-mote centered on negation detection (Chapman et al., 2001; Vilares et al., 2015) and profiling (Morante and Daclemans, 2012), so the extracted negation information could be used in downstream tasks.
With the booming of deep-learnigg architectures that were based abstract neural representations of texts, the paradigm shifted and negation was processed the rest of the elements appearing in text. It was soon noticed that systems struggled to correctly process the information when nega- tion was involved. Such is the case of negation in machine translation (Bentivogli et al., 2016; Tang et al., 2021), information extraction (Grivas et al., 2020) and sentiment analysis (Barnes et al., 2021) among others.
Clausal vs. sub-clausal: clausal negation negates the whole clause that includes it, and sub-clausal negation only affects a part of the clause.
In Table we present the different types of nega- tions considered in our work.
Dataset
3.1 Dataset construction
Our benchmark compiles 381,300 artificially gener- ated sentences in standard English. The sentences. in a definition style (e.g. X is Y, is part of Y), have been created based on the knowledge from WordNet and related resources.
has not been long since the scholar commu- nity started analyse the reasons for the lack of capability of correctly processing negation. For example, Jumelet and Hupkes (2018) analysed the

---

Negation
WordNet (Boyd-Graber et al., 2006), which is a list of the most frequently used word senses that includes 3,299 nouns and 1,000 verbs. this way, we discard words that are less commonly used. Fur- thermore, we exclude the triples on synonymy' and hyponymy that relate Basic Level Concepts (BLCs) (Izquierdo et al., 2007), which may result too gen- eral, and we use the mapping from WordNet to WordNer Top Ontology (TCO) to ignore the triples on the member meronymy relation and the agent semantic role where the noun synsets are not referring to animals or persons.
Example
type
Verhal
Agreement is not appropriate syn- onym of disagreement in any context. In context lectures may be part of courses.
Non-verbal
Analytic
No theft is small replica of a person. Mirror is never un appropriate hy- ponym of reduction.
Synthetic
Clausal
Kissing is not commonly done by en- gineers.
Sub-clausal
Bricks made of clay no context
Table 1: Examples of the types of negation.
Since WordNet and the considered related re- sources only provide true knowledge that is, all the triples and mappings describe real relations and connections- we automatically obtain false knowl- edge from WordNet triples using distractors, which are randomly selected words that replace the word senses of a synset.² 2 That is, given a WordNet triple that relates two synsets, from Core WordNet we 682
Pattern
Relation
Templates
Triples
Sentences
#01
Synonymy
21
2,996
281,624
#02
Antonymy
21
58
8,178
#03
Synonymy
24
14
2,436
#04
Antonymy
24
58
10,092
#05
Hypernymy
24
634
60,864
#06
Part
16
199
11,940
#07
Substance
15
21
1,176
#08
Member
17
11
select distractor to replace the word senses of one 240
#09
Agent
2
60
#10
Instrument
468
of the synsets and obtain distracting triple. Apart from BLCs, for the selection of suitable distrac- tors consider the lexicographer files provided by WordNet, which are 45 syntactic category and logical groupings of word senses, and WordNet Domains (Bentivogli et al., 2004), which consist of a hierarchy of 164 labels that characterize knowl- edge areas and to which each synset is connected. In Appendix C, we provide more details about the selection of distractors.
#11
Result
27
40
3,600
Total
381,300
Table 2: Distribution of sentences by pattern.
The sentences in our dataset are obtained by means of patterns. Each of the 11 patterns (#01- #11) is designed for a particular relation and in- cludes several different templates two types: af- firmative templates, which are free of negation; negative templates, which include one of the types of negations described in Table 1. Using triples on the corresponding relation, these templates used to create sentences by instantiation. Since each synset may include more than one word form Core WordNet and the proposed templates in- clude optional and alternative parts, we obtain sev- eral sentences from each couple of template and triple. The controlled application of the different templates enables us to determine the truth-value of the resulting sentences. In Appendix D, we de- scribe each pattern in detail.
Next, we illustrate the process of construct- ing our dataset. In Pattern #06, we have in- cluded the following positive and negative tem- plates that state semantic correspondences between parts and wholes on the basis of triples of the form (part, nown1, noun2):
(noun)+(ess) [ are commonly may be part of (noun)+(e)s).
(noum)+(e)s) are never part of (noung+(e)s).
The positive template yields true sentences when instantiated with true knowledge (i.e. Word- Net triples), while we get false sentences using distracting triples. On the contrary, the nega- tive one yields sentences with the opposite truth- value. For example, given the WordNet triple
More specifically, we focus on the WordNet rela- tions synonymy, hypernyny, antonymy, meronymy (part, member and substance) and the semantic roles agent, instrument and result provided by Mor- phosemantic Links (Fellbaum al., 2009). Among the nouns and verbs compiled in WordNet, we con- centrate exclusively on the ones provided by Core
'Synonymy triples are obtained by reflexivity. PIn Patterns #01 and #02, distractors are synsets when using glosses. 'The expressions enclosed in square brackets are alterna- tive.

---

(part, b bird we select human body as distractor for bird' and get the distracting triple (part, b human body). Then, by instantiat- ing the positive template using these two triples, we get the sentences in the first row of Figure! and also:
tester
B tester
AП
A
T/F prediction Comprchensibility
90.9
87.27
96.36
91.82
100
91.82
100)
Grammaticality
83.64
96.82
83.64
96.82
Plausibility
20.45
45.91
19.55
46.82
Table 3: Human evaluation of the quality of the sen- tences in the dataset.
"Bills may be part of birds."
The two sentences about birds (i.e. resulting from the WordNet triple) labelled with True, while the sentence about human bodies (that is, obtained from the distracting triple) is labelled with False, Likewise, using the same two triples for the in- stantiation of the negative template, we get the sentences in the second row of Figure I, which are respectively labelled with False and True.
3.2 Dataset quality assessment
Human Evaluation addresses the validation of the generation process and the different templates used, that to say, whether the sentences the dataset are grammatical and that overall represent true and false knowledge as expected. To prove the linguis- tic quality of the dataset and that the predictions extracted from WordNet reflect the reality, two na- tive speakers of English were required to assess a randomly selected sample of 220 sentences from our dataset. Evaluators were required to answer four questions for each sentence: Is the sentence true or false?, ii) is the sentence grammatically cor- rect?, iii) is the sentence understandable? and iv) is the sentence plausible and might be produced by a speaker?
In Table 2, we sum up some figures about the proposed dataset. For each pattern (first column), we provide the corresponding WordNet relation and the number defined templates, applied Word- Net triples and obtained sentences respectively. It is worth noting that Patterns #01-#04 include both false positive sentences and true negative sen- tences obtained from synonymy and antonymy WordNet triples by means of a dual application of templates. Furthermore, in the case of antonymy, the truth-value of the resulting sentences does not depend on whether templates are instantiated us- ing WordNet distracting triples. As conse- quence, instantiating template using WordNet and a distracting triple yields sentences with op- posite truth-value except for Patterns #02 and #04, where all sentences resulting from the same tem- plate have the same value. For example, given the antonym triple (ant, expenditure income we select wood as distractor for expenditure", (see Appendix C for details) and obtain the distract- ing triple (ant, wood, income Using these two templates, instantiate the following negative template included in Pattern #04
The answers to these questions have been sum- marised in Table 3. For the true and false predic- tions, have compared the testers' answers with the predictions we generated from the WordNet relations. Circa 90% of the predictions match with the human testers' answers. For the quality of the test sentences, the results show that the sentences the dataset are mostly comprehensible to humans even if not all are fully acceptable in English or they are not likely to be uttered by English speak- (low plausibility). Namely, have detected problems with uncountable nouns (1) and lexical selection (2). Nonetheless, low plausibility might be an interesting asset for our experiments as em- ploying non-frequent sentences may help to reduce the effect of the reliance on lexical co-occurrences models have.
(nown)) and (noun2) are the same thing in no context.
"A letter is commonly part of mail." "Officers are not members of laws in any context."
(2)
and obtain two sentences:
"Expenditure and income are the same thing in context."
In what refers the quality of the knowledge en- coded in the dataset, we have observed that over 98% of the sentences with distractors in the human test set represent actual knowledge. We can thus consider that the distractor selection mechanism is
"Wood and income are the same thing in no context."
Both sentences are true.

---

robust enough.
Is the following statement True or False? /sentence].
4
Experimental Setup
We found that models that have undergone fine- tuning for conversation tend to generate expla- nation instead of answering True or False. We use a slightly modified prompt that improves the results: the following statement True or False? Answer only True or False. (sentence]. Models that have been fine-tuned as dialogue systems utilize different prompts to represent a conversation, such as using markers like "<hot>" and "<humanz", or custom system initial prompis. In order to commodate these models, we format the input cording to the recommendations provided by the authors. Implementation details of fine-tuning and inference are available Appendix B.
In this section, we define the evaluation protocol we use to measure the performance of Language Learning Models (LLMs) on our dataset.
4.1 Models
We evaluate diverse set of LLMs, ranging in size from 7 billion parameters up to 65 billion parame- ters. Our evaluation includes Foundation Models, along with versions that have undergone additional instruction-tune and/or have been fine-tuned for conversation. We do not consider closed models where the data used for pretraining or even the model architecture is unknown, drawing mean- ingful conclusions for such systems not possible. We evaluate the following models: the 12 billion parameter T5 (Raffel et al., 2020) encoder-decoder language model, as well as FLAN-T5 (Chung et al., 2022), an enhanced version of T5 that has been fine-tuned in a mixture of tasks; LLaMA (Touvron et al., 2023) decoder-only language models with parameter sizes ranging from 7 billion to 65 bil- lion: LLaMA models that have been fine-tuned for specific tasks, including Vicuna v1.1 (Chiang et al., 2023), which has undergone additional fine-tuning as a ahhat-assittant, and WizardLM (Xu et al., 2023), which has been fine-tuned for following instruc- tions; Pythia (Biderman et ul., 2023) decoder-only 12 billion parameter model: the instruction-tume -attiin model Dolly (Conover al., 2023); and finally evaluate Falcon (Almazrouei al., 2023) and 40 billion parameter models which are decoder- only models including the instruction-following fine-tuned versions. We also evaluate other open LLMs; the full model list can be found in Appendix A.
4.3 Metries
In our dataset, we utilize two primary metrics for evaluating LLMs:
Accuracy This metrie is computed using the for- mula acc = (T (P+TNN/(TP)) +TN++F++FNN We evaluate the overall accuracy at the sentence level for all the sentences in our dataset. Addition- ally, we analyze the overall accuracy of different sentence types: Accuracy in Affirmative sentences, Negative sentences, Affirmative sentences with a distractor and Negative sentences that include distractor.
Coherence This metric aims to decouple the real-world and commonsense knowledge of the model from the understanding of negative sen- tences. We compute two coherence scores: one for the sentences without distractors ("Bills are com- monly part of birds" and "Bills are never part of birds") and another for the sentences with distrac- tors ("Bills are commonly part of human bodies." " and "Bills are part of human bodies."). An- swers are deemed coherent the affirmative and negative sentences have opposite labels, regardless of whether the answer correct or incorrect. How- ever, if the model predicts the same label for both the affirmative and negative sentences, we consider the answer incoherent. To illustrate this metric, consider the sentence pair "Bills are commonly part of birds" and "Bills are never part of birds". Both the answers "True/False" and "False/True" are considered coherent, whereas "Truc/Truc" and "False/False" are incoherent.
4.2 Task Formulation
We evaluate each sentence the dataset individ- ually as a binary task in which the model must generate either True or False tokens. Following Scheurer al. (2023), given the prompt pł com- pute the answer A follows:
Truc if
pi Truelpt) (e)Truen)ttt(Faaeee > 0.5
A
False
otherwise
We use the following prompt as input for the models:
Moreover, we calculate the overall coherence:

---

Accuracy
Coherence
Model name
Model Type
Wo Distractor
W/ Distractor
All
All
Wio Distractor
W/ Distractor
Affimmation
Negation
Affimmtion
Negation
Random
0.5
0.9
0.8
50.0
50.2
50.1
50.0
49.9
LLaMA13B
Foundation
0.0
0.2
0.3
50.1
85.8
12.2
10.6
90.2
Foundation
0.2
84.7
30.2
1JaMA65B
Foundation
0.0
0.0
0.0
50.3
96.3
3.1
1.3
99.3
Vicuna 13B
Dialogue
0.2
8.8
0.6
57.8
83.1
85.1
78.0
2.6
WizardLM33B
Instruction
6.0
0.1
57.3
53.6
95.7
88.8
2.0
P2thia12B
Foundation
0.0
0.1
0.0
50.1
93.8
15.2
4.0
86.7
Dolly12B
Instruction
0.2
72.0
33.4
25.1
TS-xxl
Foundation
0.0
0.0
0.0
50.3
96.6
0.4
99.8
Flan-T5-xxl
Instruction
0.9
46.4
1.2
66.1
86.1
94.6
6.2
Falcon40b
Foundation
0.2
49.7
11.6
83.3
Falcon-40e-nnstruc
Instruction
1.5
0.2
54.7
64.3
76.8
71.4
16.6
Table 4: Zero-shot performance of various LLMs in dataset. that surpass the Random baseline accuracy are underlined.
The best results are highlighted in bold, and scores
this happens when all the statements with and with- out distractors are coherent and correctly or all incorrectly classified. Referring the example in Figure 1, we would deem the set of statements as overall coherent the sentences with and without distractors are coherent and all the answers are ei- ther correct or all of them are incorrect. In the case of antonymy relations (Patterns #02 and #04), both the distractor-carge carryygg and non distractor-carryinn sentences carry the same label, we evaluate the overall coherence accordingly. It important to note that, for the sake of simplicity, the example only contains two sentences, but coherence is ac- tually determined at the triple level. Triples can comprise between 2 to 27 templates. So, for triple to be deemed coherent, responses to all the templates within it must be coherent. Therefore, this is very challenging metric.
W/o Distractor
Flan-Tr T5-xxl
Affirmation
Negation
#01 Synonymy
91.19
98.04
#02 Antonymy
96.36
25.62
#03 Synonymy
49.76
98.47
#04
Antonymy
82.07
21.92
Vicuna13B
Synonymy
88.69
84.88
#02 Antonymy
71.65
4.64
#03 Synonymy
57.86
90.05
#04 Antonymy 75.8
12.81
Table 5: Accuracy of Flan-T5-xxl and Vicunal in the Synonymy and Antonymy patterns. We evaluate the models in affirmative and negative sentences without distractors. Scores that surpass the Random baseline are indicated with underline.
sentences with distractor, which are True with the exception of the Antonymy patterns, that form approximately 5% of the total sentences. How- ever, these models struggle to classify negative sen- tences and affirmative sentences with opposite la- bels. Their performance these falls significantly below the random baseline exhibiting a total lack of coherence by the models.
By examining coherence in these contexts, we gain insights into the models' ability to understand the negation, even if the models do not have the real-world knowledge correctly label the sen- tences.
5 Do LLMs understand negation?
Models that have undergone dialogue or in- struction tuning, particularly Vicuna and FlanT5, demonstrate higher accuracy, instead. These mod- els achieve very high accuracy in sentences with- out distractor. Specifically, Flan-TS T5 shows coher- answers for 46% of the triples. is be noted that this is challenging metric, as a triple may be use build up to 27 templates, and all of them must be coherent for the triple to considered coherent.
In this section, we assess the performance of the LLMs in section 4.1 in our dataset. The evaluation is conducted in zero-shot setting, meaning that evaluate the models without any fine-tuning. The results of this evaluation are presented in Ta- ble Foundation models, which are trained on large amounts of unlabeled data, demonstrate an All True behavior. They accurately label as True the majority of affirmative sentences and negative

---

Accuracy
Coherence
Model name
W/o Distractor
W/ Distractor
All
All
W/o Distractor
W/ Distractor
Affirmation
Negation
Affirmation
Negation
Flan-T5-x0
51.8
55.4
92.9
94.1
96.5
86.7
96.1 98.1
98.0
Vicuna13B
81.2
86.4
94.2
95.7
92.7
94.4
97.2
Table 6: Performance of VicunaI3B and Flan-T5 T5-xx---- xxl after fine-tuning in our dataset. The best results are highlighted in bold, and scores that surpass the Random baseline accuracy are underlined.
However, these models fail to correctly label negative sentences with distractor. We further analyze the performance of Flan-T5 and Vicuna in negative sentences, focusing on the Synonymy and Antonymy patterns. In Pattern #01 and #02, as well as Pattern #03 and #04, the templates are opposite to each other, as explained in Subsection 3.1. Table 5 presents the performance of Flan- T5 and Vicuna in handling these patterns. Interestingly, both mod- els achieve good results in negative sentences from the Synonymy patterns (labeled False) but strug- gle with the negative sentences from the Antonymy patterns (labeled as True). This, along with their poor performance in negative sentences with dis- tractor (which are expected to be True, but models predict the label False), confirms that the models are heavily biased to always predict the label False in the presence of negation, regardless of the actual meaning of the sentence. This behavior suggests that the models lack deep understanding of nega- tion, and that they tend to rely on superficial cues rather than comprehending the true meaning con- veyed by the negative sentences.
ficiently large datasets containing negative knowl- edge, this hypothesis has not been extensively explored. In contrast, our dataset is substantial enough be split into training, development, and test sets. To investigate whether LLMs can learn to reason over negative knowledge given enough negated data, we split the dataset at the triple level, ensuring that all sentences within triple are as- signed to the same split to ensure no data contami- nation. Our training dataset consists of 268,505 sen- tences from 2,876 triples, the development dataset includes 2,514 sentences from 244 triples, and the test dataset contains 90,281 sentences from 980 triples.
We finetune Flan-T3 T5 and Vicuma on our dataset; the results are listed in Table 6. The impact of fine- tuning is remarkable, as it completely transforms the models' performance compared to their zero- shot counterparts. Both Flan-T5 and Vicuma exhibit higher accuracy than human annotators and achieve notably high level of coherence. However, are the models truly learning about negation, or are they just exploiting patterns in the data? We conduct experiments to asses this.
Despite the poor performance of the models in negative sentences, it is important note that they demonstrate the ability correctly label affirma- tive sentences, both with and without distractors. This demonstrates that the models have a deep un- derstanding of truth and falsehood. Models' strug- gles primarily result from the presence of negation rather than a lack of comprehension or real-world knowledge.
First, we train Vicuna, the best performing model, using varying amounts and types of neg- ative knowledge. We conduct separate fine-tuning experiments using all the affirmative sentences and all the negative sentences from the dataset, result- ing in two distinct models. The results of this train- ing are presented in Table 7. Training the model exclusively with affirmative sentences yields a high accuracy in the affirmative test sentences, but it labels incorrectly nearly all the negative sentences. Conversely, when trained solely with negative sen- tences, the model deals successfully with the neg- ative sentences but struggles with the affirmative sentences. Despite being exposed extensive real- world knowledge from WordNet, the models ex- hibit a significant failure in comprehending nega- tion. They consistently overlook the presence of it
6 Exposure to negation does not solve the problem
Understanding whether LLMs would understand negation if sufficient number of negative sen- tences were present in the pretraining corpora is crucial for improving their reasoning capabilities and addressing the limitations associated with neg- ative knowledge. However, due to the lack suf-

---

Verbal
Non- Verbal
Analytic
syathetic
clausal
subclausal
Affirmation
All
96.0
95.7
95.8
96.0
96.0
95.7
95.5
All Affirmations
6.2
6.8
6.8
5.9
6.1
6.8
95.7
All Negated
96.1
95.8
95.8
96.3
96.1
95.8
4.5
Affirmations Verbal
95.5
79.5
81.9
95.6
95.5
79.5
95.4
Affirmations Non- Verbal
94,9
95.6
95.2
96.1
94.9
95.6
95.8
Affirmations Analytic
96.1
95.6
95.6
96.1
96.0
95.6
95.9
Affirmations synthetic
94.8
44,5
51,8
96.0
94,9
44.5
95.6
Affirmations clausal
95.8
34.6
43.9
96.0
96.1
34.6
95.8
Affirmations subclausal
95.1
95.7
95.3
96.2
95.1
95.7
96.0
Table Accuracy of Vicuna after fine-tuning with different types and amount of negative knowledge. The best results are highlighted in bold, and scores that surpass the Random baseline accuracy are indicated with underline.
and generate identical outputs for both affirmative and negative sentences. We also fine-tune mod- els using various combinations of affirmative sen- tences and different types of negations. We observe that models trained with synthetic and clausal nega- tions struggle to accurately classify non-verbal, an- alytic, and sub-clausal sentences. This suggests that while the models show proficiency in under- standing and reasoning with certain types of nega- tions, they face challenges comprehending and correctly responding to other forms of negations that they have not seen in the fine-tuning step.
to those from the remaining patterns. The fail- ure of models trained in other patterns to label the Antonymy patterns, as well as the failure of mod- els trained the Antonymy patterns to label other patterns, suggest that the models are relying on repetitive data structures that are not transferable to different patterns, rather than truly understand- ing the concept of negation. While exposure negation may contribute to achieving favorable sults within a specific dataset, does not lead a generalization on negation by the models. Nega- tion continues to pose significant challenge in the field of Natural Language Processing and remains an unsolved problem, requiring further research and development.
#01 Synonymy
78
90
a02 Antonymy
50
100
85
98
50
50
50
50
50
50
50
403 Synonymy
86
79
100
BKC
BB
80
B6
93
80
Conclusion
Antorymy
105 Hypernymy
Current LLMs are typically trained using next to- ken or mask token prediction objectives, which have proven effective for various NLP tasks. How- ever, it remains an open issue understanding how certainly a model models negation. Negation to- kens, which intermittently appear in sentences, hold little predictive importance for other tokens in the sentence. As result, there is limited negation signal during language modeling training. Previ- ous research has touched upon this issue but was limited by small manually generated datasets. contrast, our study introduces the largest dataset to date comprising negative sentences. This compre- hensive dataset includes affirmative and negative sentences with and without distractors, incorpo- rating multiple types of relations and negations, which help encode the underlying mechanisms for negation understanding. Through our analy- sis, we reveal that current L.LMs, both in zero-shot settings and when fine-tuned with examples from our dataset, exhibit a profound understanding
#06 Part
72
92
78
79
100
86
65
94
78
#07 Substance
61
76
76
87
73
86
100
73
66
86
74
#0B Member
79
76
97
100
74
67
62
Agant
63
#10 Instrument
77
94
#11 Recult
85
79
84
85
77
92
90
88
81
91
All
97
34
96
100
93
94
90
96
91
88
95
Figure Evaluation of Vicunal3B accuracy when trained on one pattern (rows) and evaluated on the oth- ers (columns).
We also fine-tune Vicunal 13B with each of the 11 patterns in our dataset independently, and we eval- uate its performance on the other patterns. Figure 2 shows the overall accuracy scores. The results reveal that training the model with one pattern does not facilitate any successful generalization across all other patterns. Notably, as discussed in Sec- tion 3.1, the labels for affirmative and negative sentences from the Antonymy patterns are opposite

---

the truthfulness of affirmative sentences. However, when it comes negation, these models heav- ily rely on superficial cues instead of generalizing negation and these superficial cues are not transfer- able across different negative sentences.
in correctly responding to affirmative sentences, in- dicating that their struggles primarily arise from the presence of negation. Additionally, we introduce a coherence metric that considers whether the model changes its prediction in the presence of negation, rather than solely focusing on the accuracy of the model's answer to the question.
Negation remains persistent and unsolved chal- lenge in the field of NLP, demanding further re- search to develop systems capable of effectively handling it. Our dataset holds the potential signif- icantly contribute towards achieving this objective. In our future work, we plan to explore advanced reasoning paradigms, such as Shrin-of-Thoughi with the aim of enhancing model performance on our dataset. However, dealing properly with nega- tion may also require novel neural architectures.
Ethics Statement
The dataset has been created through the English WordNet relations, so it reflects most of the "west- ern" knowledge and might fall short in including concepts of non-English speaking communities. The generated triples from WordNet may include offensive or biased sentences. This can be caused by inherited biases from WordNet, or can be caused unintentionally during the random sampling of synsets.
Limitations
The dataset contains limited number of low- quality sentences, which are discussed in Section 3.2. Through manual evaluation, we find that over 96% of the sentences are considered understand- able and grammatically correct by at least one hu- man annotator and their prediction of whether the sentence is true or false matches the label in the dataset. Hence, the presence of low-quality sen- tences does not have significant impact the evaluation results. On the other side, majority of sentences in the dataset are not plausible and unlikely to be spoken by English speakers. This feature provides a benefit by ensuring that the sen- tences are improbable to be found in the LLM train- ing corpus, thereby it prevents models from relying solely on memorization to generate accurate re- sponses.
Acknowledgements
We would like to thank Jeremy Barnes and Aritz Farwell for willingly offering themselves to conduct the dataset quality assessment ex- periments. Begoña Altuna is supported by the Basque Government postdoctoral grant POS 2022 0024. Iker Gareía-Ferrero is supported by doctoral grant from the Basque Government (PRE_2002__2 2_____200), This work has also been partially supported by HiTZ center and the Basque Government (Research group funding IT- IT-1100--2). We also acknowledge the funding from the follow- ing projects:
2) (i) Antidote project funded by (PCI2020-120711 TAA.EE//11.1133/5500 10001 1033 and by "ERDF A way of making Europe"
All experiments were conducted by querying the models for the probability of True and False tokens. We did not explore more complex reason- ing prompts, such as Chain of Thought. However, explained in Section 7, we believe that models should be able to comprehend negation and provide accurate answers across diverse settings. Complex reasoning paradigms may not always be feasible in real-world applications, specially when models are used by non-NLP professionals.
(ii) MOTION (PID2020-1 1200-12288GBBB (12581GB-C22) sup-
ported by the Ministry of Science and Innovation of the Spanish Government (iii) The Basque Project LoRea (UPV/EHU
GIU21/044).
(iv) DeepKnowledge (PID2021-127770B-C2]
and ERDF A way of making Europe
(v) DeepR3 TTED2021-130295B--C0 and Euro-
pean Union NextGeneration EU/PRTR.
Finally, the performance of models in our dataset is not solely determined by their capability to derstand negation. Factors such as performance in question answering and prompting tasks, as well as their understanding of real-world knowledge, play a crucial role. However, models like Vicunal3B and Flan-T5-xx showcase remarkable proficiency
References
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al- shamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Hes- low, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo.

---

2023. Falcon-408 an open large language model with state-of tfat--ffthhe-rt art performance.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction finctuned language models, CoRR, abs/2210. 11122100.1111.. 11416.
Hiba Amaout, Simon Razniewski. Gerhard Weikum, and Jeff Z. Pan. 2022. UnCommonSense: Informa- tive Negative Knowledge about Everyday Concepts. In Proceedings of the s1 ACM International Con- ference on Information & Knowledge Management, CIKM '22. pages 37-46, New York, NY, USA. Association for Computing Machinery.
restruction-finetuned
Jeremy Barnes. Erik Velldal, and Lilja Øvrelid. 2021. Improving sentiment analysis with multi-task learn- ing of negation. Natural Language Engineering. 27(2):249-266
Mike Conover. Matt Hayes, Ankit Mathur, Xiangrui Meng, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free Dolly: Introducing the World's First Truly Open Instruction- Tuned LLM.
Marco Baroni. 2020.
Linguistic generalization and compositionality in modern artificial neural networks. Philosophical Transactions of the Royal Society B: Biological Sciences, 375(1791)) (75((19)))201900))) 1020100377
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer, 2022. LLM.in800: 8-bit Ma- trix Multiplication for Transformers at Scale. CORR, ahs/2208.07339.
Luisa Bentivogli, Arianna Bisazza. Mauro Cettolo, and Marcello Federico. 2016. Neural versus Phrase- Based Machine Translation Quality: a Case Study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 257- 267, Austin, Texas. Association for Computational Linguistics.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetun- ing of Quantized LLMs. CORR, fass/2355.13314
C. Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press.
Luisa Bentivogli, Pamela Forner, Bernardo Magnini, and Emanuele Pianta. 2004. Revising the Word- net Domains Hierarchy: semantics, coverage and halancing. In Proc. of the Workshop on Multilin- gual Linguistic Resources, pages 94-101, Geneva, Switzerland. COLING.
Christiane Fellbaum, Anne Osherson, and Peter E. Clark. 2009. Putting semantics into WordNet's "Mor- phosemantic" Links. In Zygmunt Vetulani and Hans Uszkoreit, editors, Human Language Technology. Challenges of the Information Society, LNAI 5603, pages 350-358. Springer.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: Suite for Analyzing Large Language Models Across Training and Scaling. CORR, abs/2304.01377
Daniel Furrer, Marc van Zee. Nathan Scales, and Nathanael Schärli. 2021. Compositional Generaliza- tion in Semantic Parsing: Pre-training vs. Specialized Architectures.
Andreas Grivas, Beatrice Alex, Claire Grover, Richard Tobin, and William Whiteley. 2020. Not a cute stroke: Analysis of Rule- and Neural Network-based Information Extraction Systems for Brain Radiology Reports. In Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis. pages 24-37, Online. Association for Com- putational Linguistics.
Jordan
Christiane Fellbaum, Daniel Os- herson, and Robert Schapire. 2006. Adding dense. weighted connections WordNet. In Proceedings of the third international WordNet conference, pages 29-36.
Wendy W. Chapman, Will Bridewell, Paul Hanbury, Gregory F. Cooper, and Bruce G. Buchanan. 2001. A Simple Algorithm for Identifying Negated Findings and Diseases in Discharge Summaries. Journal of
Md Mosharaf Hossain, Dhivya Chinnappa, and Eduardo Blanco. 2022. An Analysis of Negation in Natu- ral Language Understanding Corpora. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 716-723, Dublin, Ireland. Association for Computational Linguistics.
Biomedical Informatics, 34(5):301-310.
Jiangjie Chen, Wei Shi, Ziquan Fu, Sijie Cheng. Lei Li. and Yanghua Xiao. 2023. Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge.
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ton Stoica, and Eric P. Xing. 2023. Vicuna: An Open- Source Chatbot Impressing GPT-4 with 90%* * Chat- GPT Quality.
Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020. An Analysis of Natural Language Inference Benchmarks through the Lens of Negation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),

---

pages 9106 9118, Online, Association for Computa- tional Linguistics.
and Efficient Foundation Language Models. CoRR, abs/2302.13971
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang. Lu Wang. and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, Open- Review.net.
Thinh Hung Truong, Yulia Otmakhova, Timothy Bald- win, Trevor Cohn, Jey Han Lau, and Karin Verspoor. 2022. Not another Negation Benchmark: The NaN- NLI Test Suite for Sub-clausal Negation. In Pro- ceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference Natural Language Processing (Volume I: Long Pupers), pages 883-894, Online only. Association for Computational Linguistics.
Rubén Izquierdo, Armando Suárez, and German Rigau. 2007. Exploring the Automatic Selection of Ba- sic Level Concepts. In Proc. of the Int. Conf. on Recent Advances on Natural Language Processing (RANLP'07), volume 7.
David Vilares, Miguel A. Alonso, and Carlos Gómez- Rodríguez. 2015. On the usefulness of lexical and syntactic processing in polarity classification of Twit- ter messages. Journal of the Association for Informa- tion Science and Technology, (6(9):1799-181
Jaap Jumelet and Dieuwke Hupkes. 2018. Do Lan- guage Models Understand Anything? On the Ability of L.STMs to Understand Negative Polarity Items. In Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Net- works for NLP, pages 222-231, Brussels, Belgium. Association for Computational Linguistics.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. WizardL.M: Empowering Large Lan- guage Models to Follow Complex Instructions. CoRR, abs/2304.12244.
Nora Kassner and Hinrich Schütze. 2020. Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811-7818, Online. Association for Computational Linguisties.
A Extended zero-shot results
Apart from the models presented in Section 5, as anticipated, we have also tested the performance in the task of the following models: Koala 4, which is a LLaMA model that has been fine-tuned for dialogue: Open-Assistant (oasst-sft-l-pythia--ll which is Pythia model fine-tuned on human generated assistant conversations; and INCITE billion foundation model along with the two mod- els that have been further fine-tuned by the authors in the estruction-tuning paradigm and chat con- versation. Table 8 shows the extended evaluation results.
Roser Morante and Walter Daelemans. 2012. ConanDoyle-neg: Annotation of negation cues and their scope in Conan Doyle stories. In Proceedings of the Eighth International Conference on Language Rexources and Evaluation 12), pages 1563-1568, Istanbul, Turkey. European Language Resources Association (ELRA).
Geoffrey K. Pullum. Rodney Huddleston, Rodney Hud- dleston, and Geoffrey K. Pullum. 2002. Negation, pages 785-850. Cambridge University Press.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Lim- its of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn Res., 21:140:1-140:67.
B Efficient inference and training
To facilitate inference of the models a single GPU, we employed 8-bit quantization (Dettmers et al., 2022) for all of them. We conducted prelimi- nary experiments with Vicuna 13 billion parameter model. Results are shown in Table While the run- ning cost of the models significantly decreases, we observed only minimal performance degradation in the quantified versions.
Jérémy Scheurer, Jon Ander Campos, Tomasz Kor- bak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2023. Training Language Models with Language Feedback at Scale. CoRR, abs/2303. 16755.
Gongbo Tang, Philipp Rönchen, Rico Sennrich, and Joakim Nivre. 2021. Revisiting Negation in Neural Machine Translation. Transactions the Associa- tion for Computational Linguistics, 9:740-755.
For the training process, we utilized Low-Rank Adaptation (LoRA) (Hu et al., 2022). This ap- proach involves freezing the weights of the pre-
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open
koala/ http:s//hwuggngeeeee..eo///sssss casst-sft-1-pythia--22
hhttps://www twwget//err.g/bbbb////aaaaaaaaea

---

Coherence
Accuracy
Model
Model Type
Distractor
Distractor
Wi Distractor
Distractor
Affirmation
Negation
Affirmation
Negation
Random
0.5
0.9
0.8
50.0
50.2
50.1
50.0
49.9
LLaMA7H
Foundation
0.0
0.2
0.0
50.4
95.7
11.8
2.3
91.0
1LMMA
Foundation
0.3
50.1
85.8
12.2
10.6
90.2
LLaMA30B
Foundation
0.2
52.4
84.7
29.5
30.2
68.8
LLaMA638
Foundation
0.0
0.0
0.0
50,3
96.3
3.1
1.3
99.3
Vícuna78
Dialogüe
18.T
0.3
Vicuna 13B
Dialogue
0.2
8.8
0.6
57.8
83.1
85.1
78.0
2.6
Koalä7B
Dialogne
00.
Koulal3B
Dialogue
0.0
0.7
0.0
52.8
20.4
97.1
98.88
0.3
M77
Instruction
M133
Instruction
0.0
4.3
0.3
$7.4
68.7
86,2
87.5
2.9
M308
Instruction
95.7
88.4
2.0
Instruction
0.0
1.0
0.1
33.7
63.4
78.4
63.3
17.5
Wizardl 3R--anennoorre
Instruction
0.0
97.0
0.3
M30I ancensored
Instruction
0.9
49.8
V7.2
99.6
0.2
PBthia12B
passt-pythinl22
Dialogue
0.0
0.0
0.0
49.8
4.7
97.1
98.9
0.3
12I
Instruction
0.3
0.2
50.2
72.0
73.3
33,4
25.1
T5-xxl
Foundation
0.0
0.0
0.0
50.3
96.6
2.8
0.4
99.8
Flan
Folcon7b
Foundation
0.0
0.0
0.0
50.3
96.6
0.4
99.7
Falcon77binstrret
Instruction
50.1
77.0
Falcon40b
Foundation
0.1
0.1
0.2
49.7
90.9
13.9
11.6
83.3
Fuloons@O-innstucc
16.6
INCITE7B Base
Foundation
0.1
50.3
82.6
17.0
16.4
84.5
INCTIE7B-Iastrus INCITE7B-CHR
Instruction
0.4
73.4
23.5
78.7
Dialogue
0.1
0.3
0,2
30.0
19.8
89.4
88.0
6.0
Table 8: Zero-shot performance of various L.LMs our dataset. The hest results are highlighted in bold, and scores that surpass the Random baseline accuracy indicated with underline.
Accuracy
Coherence
Moddll name
Precision
W/o Distractor
Distractor
All
All
W/o Distractor
Distractor
Affirmation
Negation
Affimmation
Negation
Vicunal 13B
8-Bits
0.2
8.8
0.6
$7.8
83.1
85.1
78.0
2.6
Vicuna13B
Float16
0.4
10.1
57.8
84.4
74.7
Table 9: Zero-shot performance Vicuna using 8-Bits quantification and the orifinal float16 weights. The best results highlighted in bold, and scores that surpass the Random baseline indicated with underline.
trained model and introducing trainable rank de- composition matrices into each layer. The frozen model weights are quantized into bits, while the LoRA trainable weights remain in 16 bits (Dettmers al., 2023). By adopting this efficient training paradigm,
W/o Distractor
W/ Distractor
Pattern
Affirmation Negation
Afirmation
Negation
#01
True
False
False
True
#02
False
True
Fulse
True
#03
True
False
False
True
#04
False
True
False
True
able to train LLMs with up 13 billion parameters on single GPU within a reasonable timeframe.
#05
Truc
False
False
True
#06
True
Fulse
False
True
#07
True
False
False
True
#08
True
False
False
True
We perform all our experiments using single NVIDIA A100 GPU with 80GB memory. The machine used has two AMD EPYC 7513 32-Core Processors and 1024GB of RAM.
#09
True
False
False
True
#10
Truc
Fulse
Fulse
True
#11
True
False
False
True
Table 10: Truth-value of resulting sentences pattern.
C Dataset construction: selection of distractors
tors randomly selected words that replace the word senses of synset in given triple, although the whole synset replaced when using glosses templates (Patterns #01 and #02). For each Word- Net triple, we use single distracting triple except
The automatic creation of false knowledge (dis- tracting triples) on the basis of WordNet triples requires the use distractors. In general, distrac-

---

for Patterns #02, #03 and #04, where we use two distracting triples obtained by using one distractor per synset. Apart from BLCs, for the selection of suitable distractors we consider the lexicographer files provided by WordNet, which are 45 syntactic category and logical groupings of word senses, and WordNet Domains, which consist of a hierarchy of 164 labels that characterize knowledge areas and to which each synset is connected. More concretely:
optional expressions are enclosed respectively in square brackets and parentheses.
D.1 Pattern #01: synonymy (gloss)
This pattern includes 21 templates stating semantic equivalence correspondences between a word and the gloss of synset to which the word belongs. Since WordNet does not provide triples for syn- onymy relating two synsets, we get triples relating each synset to itself by reflexivity. For each re- sulting triple, templates are also instantiated using one distracting triple that is obtained by replacing the third component of each triple with a distractor (synset).
Words (synsets in the case of Pattern #01 and #02 when using glosses) belonging to some BLCs cannot be distractors to ensure that se- lected words (synsets) are not too general.
In Figure 3, introduce positive and nega- tive template and illustrate their instantiation using the synset fligth, with gloss "a scheduled trip by plane between designated airporss" and the distrac- tor troop ("a group of soldiers").
The combined lexicographer file and Word- Net Domain annotation of any word sense of the given synset and of any synset where the distractor occurs (of the synset in the case of Pattern #01 and #02 when using glosses) have to be different.
D.2 Pattern #02: antonymy (gloss)
In general, these restrictions ensure that the result- ing false triples do not encode true knowledge. The probability of choosing a synset as distractor is di- rectly proportional to the logarithm of its frequency.
This pattern includes 21 templates stating semantic equivalence correspondences between a word and the gloss of antonym synset, where the word and the gloss are respectively taken from the second and third component of triples. Furthermore, for each WordNet antonymy triple templates are also instantiated using two distracting triples that are respectively obtained by replacing the second and third component with distractors (for the third one, the distractor is synset).
For example, wood can be used as distractor of expenditure!, because wood belongs to the lexicographer files nown.substance (nouns denot- ing cognitive processes and contents), houn.group (nouns denoting groupings of people or ob- jects) and noun.artifact (nouns denoting man- made objects) while expenditure belongs to noum.possession (nouns denoting possession and transfer of possession) and noun.ctt (nouns de- noting acts or actions). Therefore, we get the distracting triple (ant, wood, income") from (ant, eapenditure!" incomez). On the contrary, the word registration cannot be used distractor of expenditure, 1 as both words belong the lexico- graphic file noun.ctt act and the synsets expenditure, 2 and registration | belong to the economy domain.
In Figure 3, introduce positive and nega- tive template and illustrate their instantiation using the antonym synsets brother', a and sister) fe- male person who has the same parents as another person") and the distractors stream and fiction' ("a literary work based on the imagination and not necessarily on fact").
D.3 Pattern #03: synonymy
This pattern includes 24 templates stating seman- tic equivalence correspondences between words. Since WordNet does not provide triples for syn- onymy relating two synsets, we get triples relating each synset to itself by reflexivity. For each re- sulting triple, templates are also instantiated using two distracting triples that are respectively obtained by replacing the second and third component with distractors.
D Dataset Description
In Table 10, we provide the truth-value of sentences that results by instantiating affirmative and negative templates using WordNet and distracting triples ac- cording to the pattern. In the following subsections, we describe each pattern and provide some exam- ples that are used in Figures 3-8 to illustrate the instantiation of a sample of the templates. In all the templates described in Figures 3-8, alternative and
In Figure 4, we introduce a positive and nega- tive template and illustrate their instantiation using

---

the synonym words path and route and the distrac- tors engine and identity.
For each WordNet triple, templates are also instan- tiated using one distracting triple that is obtained by replacing the group with a distractor.
D.4 Pattern #04: antonymy
In Figure 7, we introduce a positive and nega- tive template and illustrate their instantiation using the synset voter which is related by member with electorale, and the distractor sport.
This pattern includes 24 templates stating semantic equivalence correspondences between words. For each WordNet antonymy triple, templates are also instantiated using two distracting triples that are respectively obtained by replacing the second and third component with distractors.
D.9 Pattern #09: semantic role (agent)
This pattern includes 2 templates stating semantic correspondences between agents and events. For each WordNet triple, templates are also instanti- ated using one distracting triple that is obtained by replacing the agent with a distractor.
In Figure 5, we introduce positive and nega- tive template and illustrate their instantiation using the antonym synsets expenditure,, and income, and the distractors wood and year.
In Figure 7, we introduce a positive and nega- tive template and illustrate their instantiation using the synset rule which is related by agent with governor and the distractor hole.
D.5 Pattern #05: hypernymy
This pattern includes 24 templates stating semantic subsumption correspondences between words. For each WordNet hypermymy triple, templates are also instantiated using one distracting triple that is ob- tained by replacing the hyponym with a distractor. In Figure 5, we introduce a positive and a nega- tive template and illustrate their instantiation using the synset auction!. which is hyponym of sale n> and the distractor breakdown.
D.10 Pattern #10: semantic role (instrument)
This pattern includes 7 templates stating semantic correspondences between instruments and events, For each WordNet triple, templates are also instan- tiated using one distracting triple that is obtained by replacing the event with a distractor.
In Figure 8, we introduce a positive and nega- tive template and illustrate their instantiation using the synset lelephone which is related by instru- ment with call., and the distractor lay,
D.6 Pattern #06: meronymy (part)
This pattern includes 16 templates stating semantic correspondences between parts and wholes. For each WordNet triple, templates are also instanti- ated using one distracting triple that is obtained by replacing the whole with distractor.
D.11 Pattern #11: semantic role (result)
This pattern includes 27 templates stating semantic correspondences between results and events. For each WordNet triple, templates are also instanti- ated using one distracting triple that obtained by replacing the event with a distractor.
In Figure 6, we introduce a positive and a nega- tive template and illustrate their instantiation using the synset week.y. x. which related by part with month",
1 n. and the distractor fence.
In Figure 8, we introduce a positive and nega- tive template and illustrate their instantiation using the synset response which is related by result with answer and the distractor dress.
D.7 Pattern #07: meronymy (substance)
This pattern includes 15 templates stating semantic correspondences between substances and things. For each WordNet triple, templates are also instan- tiated using one distracting triple that is obtained by replacing the whole with distractor.
In Figure 6, we introduce a positive and a nega- tive template and illustrate their instantiation using the synset sandy, which is related by substance with b and the distractor decade.
D.8 Pattern #08: meronymy (member)
This pattern includes 17 templates stating seman- tie correspondences between members and groups.

---

Pattern #01: synonymy (gloss)
Affirmative template:
A/An (word) is (commonly) (gloss).
Sentences:
A flight is commonly scheduled trip by plane between designated airports.
True
A flight scheduled trip by plane between designated airports.
True
A flight is commonly group of soldiers.
False
A flight is a group of soldiers.
Falsc
Negative template (verbal, analytic and clausal):
A/An (word) not (gloss).
Sentences:
A flight is not a group of soldiers.
True
A flight is not scheduled trip by plane between designated airports.
False
Pattern #02: antonymy (gloss)
Affirmative template:
(word) (commonly) stands for refers to (gloss).
Sentences:
Brother commonly stands for a female person who has the same parents as another person.
False
Brother commonly refers to female person who has the same parents as another person.
False
Brother stands for female person who has the same parents as another person.
False
Brother refers female person who has the same parents another person.
False
Stream commonly stands female person who has the same parents as another person.
False
Stream commonly refers to a female person who has the same parents as another person.
False
Stream stands for female person who has the same parents another person.
False
Stream refers female person who has the parents another person.
False
Brother commonly stands for a literary work based on the imagination not necessarily on fact.
False
Brother commonly refers to literary work based on the imagination and not necessarily on fact.
False
Brother stands for literary work based on the imagination and not necessarily on fact. False
Brother refers literary work based the imagination and necessarily fact.
False
Negative template (synthetic and subclausal):
A/An (word) is never (gloss).
Sentences:
A brother is never female person who has the same parents as another person. A stream is female person who has the parents another person. brother never literary work based the imagination and not necessarily on fact,
True
True
True
Figure 3: Description of Patterns #01 and 402.

---

Pattern #03: synonymy
Affirmative template:
(nown)+(ess) and (e)sn)+(e)s) [ [are may be always different.
Sentences:
Path and always different.
False
Path and route may be always different.
False
Engine and route are always different.
True
Engine and route may be always different.
True
Path and identity always different.
True
Path and identity may always different.
True
Negative template (verbal, analytic and subclausal):
(noun)+(e)s) and (eosm)+(e)s) are not may be synonyms in any context.
Sentences:
Path and
synonyms in any context.
False
Path and route may be synonyms any context.
False
Engine and route are not synonyms in any context.
True
Engine and route may not be synonyms in any context.
True
Path and identity not synonyms in any context.
True
Path and identity may he synonyms any context.
Thue
Figure 4: Description of Pattern #03.

---

Pattern #04: antonymy
Affirmative template:
(nown)+(e)s) and (nowng+(ess) [ are may be synonyms (in certain contexts).
Sentences:
Expenditure and income are synonyms in certain contexts.
False
Expenditure and income may be synonyms in certain contexts.
False
Expenditure and income synonyms. Expenditre and income may be synonyms.
False
False
Expenditure and year are synonyms in certain contexts. Expenditure and year may be synonyms in certain contexts. Expenditure and year synonyms. Expenditure and year may synonyms. Wood and income are synonyms in certain contexts.
False
False
False
False
False
Wood and income may be synonyms in certain contexts.
False
Wood and income synonyms. Wood and income may be synonyms.
False
False
Negative template (analytic and subclausal):
(noum)+(ess) and (noung+(e)s) are may be the same thing in context.
Sentences:
Expenditure and income the thing in context.
True
Expenditure and income may be the same thing in no context.
True
Expenditure and year are the same thing in no context. Expenditure and year may be the same thing in no context.
True
True
Wood and income the same thing in context.
True
Wood and income may the same thing in no context.
True
Pattern #05: hypernymy
Affirmative template:
A/An (hyponyym) is may be a/an (hypernym) in certain contexts.
Sentences:
An auction sale in certain contexts.
True
An auction may be sale in certain contexts.
True
A breakdwon sale certain contexts.
False
A breakdown may be a sale in certain contexts.
False
Negative template (synthetic and subclausal):
A/An (hyponym) is never a/an (hypernym).
Sentences:
An auction never sale.
False
A breakdown is never a sale.
True
Figure 5: Description of Patterns #04 and #05.

---

Pattern #06: meronymy (part)
Affirmative template:
A/An (part) is commonly | may be part of a/an (whole).
Sentences:
A week is commonly part of month.
True
A week may part month.
True
A week is commonly part of a fence.
False
A week may be part of a fence.
False
Negative template (synthetic and subclausal):
A/An (part) is never part of a/an (whole).
Sentences:
A week is never part of month.
False
A week is never part of fence.
True
Pattern #07: meronymy (substance)
Affirmative template:
(thing + (e)s) are commonly may be ] made of (substance).
Sentences:
Beaches are commonly made of sand.
True
Beaches may be made of sand.
True
Decades commonly made of sand.
False
Decades may made of sand.
False
Negative template (Analytic and subclausal):
In context (thing + (e)s) are Imay be made of (substance).
Sentences:
In context beaches made of sand.
False
In no context decades are made of sand.
True
Figure 6: Description of Patterns #06 and #07.

---

Pattern #08: meronymy (member)
Affirmative template:
(member + (e)s) are may be members of (group (e)s).
Sentences:
Voters members of electorates.
True
Voters may members electorates.
True
Voters are members of sports.
Falses
Voters may be members of sports.
False
Negative template (verbal, analytic and clausal):
(member + (e)s) are not may not be members of (group + (e)s) in any context.
Sentences:
Voters are not members of electorates in any context.
False
Voters may not be members of electorates in any context.
False
Voters members of sports any context.
True
Voters may members of sports in any context.
True
Pattern #09: semantic role (agent)
Affirmative template:
(event ing) is commonly done by (agent (e)s).
Sentences:
Ruling is commonly done by governors.
True
Ruling is commonly done by holes.
False
Negative template (Verbal, analytic and clausal):
(event + ing) is not commonly done by (agent + (e)s).
Sentences:
Ruling is not commonly done by governors.
False
Ruling is not commonly done by holes.
True
Figure 7: Description of Patterns #08 and #09.

---

Pattern #10: semantic role (instrument)
Affirmative template:
A/An (instrument) [ is commonly may be used needed for (event + ing).
Sentences:
A telephone commonly used for calling.
A telephone is commonly needed for calling.
True
A telephone may be used for calling.
True
A telephone may be needed for calling.
True
A telephone commonly used for laying.
False
A telephone is commonly needed for laying.
False
A telephone may be used for laying.
False
A telephone may be needed for laying.
Falsc
Negative template (Synthetic and subclausal):
A/An (instrument) should never be used needed ] for (event + ing).
Sentences:
A telephone should never be used for calling.
False
A telephone should never be used for laying.
True
Pattern #11: semantic role (result)
Affirmative template:
(event + ing) [ commonly leads may lead ] to a/an (result).
Sentences:
Answering commonly leads to response.
True
Answering may lead to response. Dressing commonly leads response.
True
False
Dressing may lead to a response.
False
Negative template (analytic and subclausal):
(event + ing) leads may lead to a/an (result) in no context.
Sentences:
Answering leads to response no context.
False
Answering may lead to response in no context.
False
Answering leads to response in no context.
True
Answering may lead to response in context.
True
Figure Description of Patterns #10 and #11.


Reviews:
[{"review_id": "FlwixQLprq", "reviewer": "Reviewer_rCcw", "summary": "", "questions": "A. While determining the veracity of a sentence as true or false is a step forward, it leaves open the inquiry into whether models continue to depend on surface-level cues for their responses. A more effective approach involves crafting prompts that predict the concealed token, wherein the predicted token adheres to the same distribution in cases of affirmation, and conversely, originates from a distinct distribution when negation is involved.\n\nB. As stated \"Current LLMs are typically trained using next token or mask token prediction objectives, which have proven effective for various NLP tasks. However, it remains uncertain whether this training paradigm adequately models negation\".  Have you attempted fine-tuning the model using your dataset as a means of substantiating this assertion?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for taking the time to thoroughly assess our contributions. Your questions are addressed below. We also include some additional remarks.\n\nRegarding question A, in this paper, we are primarily interested in evaluating the recent wave of instruction/chat LLMs. These models are expected to be able to answer questions, therefore, we defined our prompts as a question answering task. We find your suggestion very interesting, for it allows for a more thorough linguistic analysis of the results, and we will try to explore it in the near future. As our dataset will be publicly available, alternative paradigms can also be explored. Regarding the comment about superficial cues, we believe that our experiments prove that, in fact, what you mention is what happens in both the zero-shot and fine-tuning experiments. This is especially obvious in the comparison of accuracy in the Synonymy and Antonymy patterns in Table 5. We further elaborate on this topic in Section 5.  \n\nRegarding question B. We believe that it is important to study how LLMs can be trained to model negation effectively and that our dataset can help to study this phenomena. Nevertheless, we will rephrase this sentence, as we might have overstepped in this assertion.\n\nRegarding the number of sentences revised by the human assessors, we will clarify in the paper that Human Evaluation addresses the validation of the generation process and the different templates used, that is to say, whether the sentences in the dataset are grammatical and that overall represent true and false knowledge as expected.\n\nIn what refers to the grasping of negation by models, it is true that there might be other approaches, but these would have forced us to vary drastically the configuration of the experiment. Still, we consider them worth the effort for future work as negation seems still an unsolved issue in NLU. Nonetheless, following a linguistic intuition, we have tried to see whether different types of negations affecting different parts of the sentence and placed in different positions within the sentence affect the performance of the LLMs, which has allowed us to assess the grasping of negation in a somewhat wide range of scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. To proficiently enhance negation, a dataset containing affirmative and negative sentences that align with the input criteria of Language Models (LLMs) is essential. Unfortunately, a publicly accessible dataset possessing these specific attributes is currently absent. The paper endeavors to address this gap.\n\nB. The dataset's composition and its inherent contents are comprehensively elucidated.\n\nC. The conducted analysis of the dataset satisfactorily demonstrates the alignment with the paper's hypothesis regarding the inadequate comprehension of negation by LLMs.", "reasons_to_reject": "A. While the dataset is comprehensive, it falls short of being labeled a gold standard due to the limited proportion evaluated by human assessors.\n\nB. Incorporating various additional approaches to assess the model's grasp of negation would have been more advantageous. This would have enhanced the credibility of the dataset.", "questions_to_authors": "A. While determining the veracity of a sentence as true or false is a step forward, it leaves open the inquiry into whether models continue to depend on surface-level cues for their responses. A more effective approach involves crafting prompts that predict the concealed token, wherein the predicted token adheres to the same distribution in cases of affirmation, and conversely, originates from a distinct distribution when negation is involved.\n\nB. As stated \"Current LLMs are typically trained using next token or mask token prediction objectives, which have proven effective for various NLP tasks. However, it remains uncertain whether this training paradigm adequately models negation\".  Have you attempted fine-tuning the model using your dataset as a means of substantiating this assertion?", "paper_topic_and_main_contributions": "Topic : A dataset to benchmark negation understanding in language models.\n\nContributions:\nA. Synthetically generated largest probing negation dataset.\nB. Evaluations and findings of negation understanding in  Large Language Models (LLMs) in zero-shot setting and after fine tuning. The paper asserts that LLMs lack the capability to comprehend negation.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pc8q6BN5Um", "reviewer": "Reviewer_jGqB", "summary": "", "questions": "What are the chances of making semantically incorrect sentences if you use the distractors? If so, should those sentences be included in the experiments?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for taking the time to thoroughly assess our contributions. Your questions are addressed below. We also include some additional remarks.\n\nThe chances of making semantically incorrect sentences are very low. On one hand, WordNet only contains true knowledge which accounts for 50% of the total dataset. The random selection of distractors for building counterfactual sentences could lead to semantically incorrect assertions that encode true or false knowledge when the opposite should be expected. Thus, we have defined a set of restrictions for distractor selection to avoid these situations as it is described in Appendix C. In the human evaluation experiment, we have checked that over 96% of the sentences are considered understandable, grammatically correct by at least one human annotator and that their prediction of whether the sentence is true or false matches the label in the dataset (see Table 3 in the paper). \n\nRegarding the number of sentences revised by the human assessors. We will clarify in the paper that Human Evaluation addresses the validation of the generation process and the different templates used. \n\nWe would like to address some of the concerns about our paper:\n\n- Due to the 8 page limit, an in-depth description and examples of each template described in Section 3 is included in Appendix D. \n\n- We fully agree about the remark about testing in-context learning, as well as other prompt paradigms such as “chain of thought”. We plan to continue experimenting with our dataset and we will definitely test in-context learning in the future. However, as we explain in the Limitations section, we believe that models should be able to comprehend negation and provide accurate answers across diverse settings. \n\n- The scores for LLaMA65B correspond to a model that labels every statement, negated or not, as “True”. It achieves near 100% accuracy in the set of statements that are gold labeled as “True” and near 0% accuracy in the set of sentences gold labeled as “False”. We further elaborate on this behavior at the beginning of Section 5. \n\n- We will rewrite it to better explain the coherence metric. You can find further explanations of the metric in the answer to Reviewer rrGn.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a quite large corpus containing (mostly) negation and affirmation examples which can be useful in future research. The authors perform a thorough analysis with zero-shot and fine-tuning approaches to evaluate performance of the recent LLMs when negation is present. ", "reasons_to_reject": "Issues and improvement areas:\n\n1) The construction of the dataset (Section 3) is little ambiguous. It would be nice to have sufficient examples for the different patterns described in Table 2. \n\n2) Only 220 examples are assessed out of 400K examples! Further, Table 3 should have a elaborate caption detailing the meaning of the last two columns. \n\n3) As this paper works with the LLMs, authors should check the in-context leaning (i.e., few-shot prompting) performance as well.\n\n4) The evaluation metric \"Coherence\" is ambiguous. Explaining how it is calculated would be helpful.\n\n5) The core insights from Table 4 seem missing. For example, why does LLaMA65B achieve 99.3% accuracy with negation (w/ distractor) but only 1.3% accuracy on the affirmation?\n\n\n", "questions_to_authors": "What are the chances of making semantically incorrect sentences if you use the distractors? If so, should those sentences be included in the experiments?", "paper_topic_and_main_contributions": "The focus of this paper is to investigate whether LLMs are capable of understanding negation. The author presents a dataset of 400K examples having True/False labels where 2/3 of the dataset contains negation in various forms. With the zero-shot and fine-tuning experiments, the authors find that negation is still a challenge in LLMs. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 611: Do you mean \"Chain of Thought?\""}, "extra_text": {}}, {"review_id": "vuEWQZRL5S", "reviewer": "Reviewer_rrGn", "summary": "", "questions": "I am not sure I completely understand the definition of overall coherence. Could you please elaborate on that in your response and make the definition a bit more clear in the camera-ready version should the paper be accepted?\n\nCan you speculate a bit on LLM performance for languages which employ negative concord to express negation, i.e. using multiple negation tokens to express a single semantic negation? For example, French and Spanish have double negation and some Slavic languages easily have 4 negation tokens in a single sentence. Would this repetative pattern lead to better LLM performance or even allow LLMs to generalize and understand negation better than in English? This is not directly connected to the paper but I would be interested in a short answer whether you have considered that phenomenon or are aware of any work in that direction.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for taking the time to thoroughly assess our contributions. Your questions and suggestions are addressed below. We also include some additional remarks.\n\nIn what refers to coherence and overall coherence, take Figure 1 as an example: an LLM might correctly label the sentence \"Bills are commonly part of birds\" as True. Consequently, considering the negated \"Bills are never part of birds\", the model should predict the opposite label, False. In this scenario, we would consider the answer coherent. However, if the model predicts the same label for both the affirmative and negated sentences, we would consider the answer not coherent. Conversely, if the model lacks the correct real-world knowledge and labeled \"Bills are commonly part of birds\" as False, but labeled the negated sentence as True, we would still consider the answer as coherent. This is why we claim that coherence is a metric that seeks to distinguish the model's real-world knowledge from its comprehension of negation. \n\nWe compute two coherence scores: one for the sentences without distractors (\"Bills are commonly part of birds\" and \"Bills are never part of birds\") and another for the sentences with distractors (\"Bills are commonly part of human bodies.\" and \"Bills are never part of human bodies.\"). For overall coherence, we expect the answers for the statements, both with and without distractors, to be not only coherent individually but also all of them must be correctly or all incorrectly classified. Referring to the example in Figure 1, we would deem the set of statements as overall coherent if the sentences with and without distractors are coherent and the model predicts opposite labels for \"Bills are commonly part of birds\" versus \"Bills are commonly part of human bodies,\" and for \"Bills are never part of birds\" versus \"Bills are never part of human bodies.\" In the case of antonymy relations (Patterns #2 and #4), both the non-distracted and distracted sentences carry the same label, so we evaluate the overall coherence accordingly.\n\nIt's important to note that, for the sake of simplicity, the example only contains two sentences, but coherence is actually determined at the triple level. Triples can comprise between 2 to 27 templates, and for a triple to be deemed coherent, responses to all the templates within it must be coherent. Therefore, this is a very challenging metric. We will clarify the definition of coherence and overall coherence in the paper.\n\nAnalyzing the impact of different types of negation was one of the main goals of this paper. As you mention, English does not have negative concord, but other languages do. It is effectively out of the scope of this paper, but we are already working on a similar experiment in other languages. For this, we are currently working on the template generation and we have not tested the LLMs yet. Nonetheless, analyzing the results for English, in which the negation was placed in different parts of the sentence and was affecting different elements in it, e.g. some negations are verbal, some negate the subject, etc., we have not noticed significant differences in the performance of LLMs (see Table 7). As a consequence, our intuition is that concord negatives will be understood in the same terms as for the sentences that contain single negation cues. Nonetheless, we expect that our experiment in other languages will shed some light on the issue.\n\nRegarding the number of sentences revised by the human assessor, we will clarify in the paper that Human Evaluation addresses the validation of the generation process and the different templates used.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The negation data set is a valuable resource for further research. It is the result of sound linguistic reasoning and is representative of various aspects of negation \n- Good qualitative analysis of the data set performed by human annotators\n- Vast evaluation of LLMs of different sizes and types, with well-founded explanations for the observed performance and behaviour of the models\n- The fine-tuning experiment with Vicuna and each pattern separately (results presented in Figure 2) which clearly shows that \"While exposure to negation may contribute to achieving favorable results within a specific dataset, it does not lead to a generalization on negation by the models\"\n\nThe biggest plus of the paper is the bringing of LLMs and linguistics together. I think such methods are needed more and more, if LLMs are to become more successful.", "reasons_to_reject": "None really. The only thing I can slightly complain about is the relatively low number of examples evaluated by humans. I would have liked to see may be a thousand examples which were not randomly selected from the overall data set but randomly selected from each pattern. The number of examples from the various patterns should reflect their distribution in the data set. This would make the qualitative evaluation of the data even stronger.", "questions_to_authors": "I am not sure I completely understand the definition of overall coherence. Could you please elaborate on that in your response and make the definition a bit more clear in the camera-ready version should the paper be accepted?\n\nCan you speculate a bit on LLM performance for languages which employ negative concord to express negation, i.e. using multiple negation tokens to express a single semantic negation? For example, French and Spanish have double negation and some Slavic languages easily have 4 negation tokens in a single sentence. Would this repetative pattern lead to better LLM performance or even allow LLMs to generalize and understand negation better than in English? This is not directly connected to the paper but I would be interested in a short answer whether you have considered that phenomenon or are aware of any work in that direction.", "paper_topic_and_main_contributions": "The paper investigates the linguistic phenomenon of negation within the context of LLMs. In particular, the authors use linked lexical resources (WordNet) to create a large data set of sentences containing negation and then employ this data to explore the ability of LLMs to coherently understand negation. The authors also explore ways to improve that understanding.\n\nThis is an excellent linguistic study on the topic of negation and its handling by LLMs. I was really happy to see state-of-the-art LLM technology explored through the magnifying glass of sound linguistic experiments. We must better understand  how LLMs handle certain linguistic phenomena and for this, we need well-grounded linguistic analysis.\n\nThe authors employ well-motivated and exemplary linguistic approach to create the data set. In itself, this is already a great contribution. Yet, the authors go two steps further by i) exploring the performance of state-of-the-art LLMs of different sizes and pursposes and ii) trying standard approaches such as fine-tuning and retraining to improve the LLM performance on negation. Finally, despite achieving improved results on the data set, the authors do perform additional critical experiments, only to show that this improvement was not the result of LLMs \"understanding\" negation but rather of the increased number of negation patterns in the data set.\n\nThe paper is a result of a tremendous amount of sound work and experiments, looking at and analysing the issue at hand from various view points. The submission is examplary for the quality an EMNLP paper should have.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I think that you should have either mentioned all LLMs involved in the experiments or at least summarized the extended zero-shot results in the main paper and not in Appendix A. Currenlty, this appendix presents new results which is against submission policy. Hence I am not considering the reported results in Appendix A for my review. Also, major technical setup details from Appendix B should also be in the main paper."}}]

Author rebuttal:

Meta-review: The focus of this paper is to investigate whether LLMs are capable of understanding negation. The authors presents a dataset containing utterances with various forms of negation. It is found that negation is still a challenge for LLMs.

The reviewers found that this paper was an excellent linguistic study on the topic of negation and its handling by LLMs and that the provided resource would be very useful for future research. It also fits the EMNLP theme track particularly well. The main criticism identified by the reviewers lies in the somewhat small part of the corpus that was evaluated by human assessors.

For future work, it would also be interesting to see versions of this benchmark in different languages, especially languages that use different negation patterns (e.g. negative concord).
Decision: Accept-Main