NLMs: Augmenting Negation in Language Models
Authors: Rituraj Singh
Rahul Kumar
Vivek Sridhar
Venue: EMNLP 2023 Findings (2023)
Keywords: Language Models
Negation
TL;DR: In this work, we augment negation understanding in Language Models.

Abstract: Negation is the fundamental component in a natural language that reverses the semantic meaning of a sentence.
It plays an extremely important role across a wide range of applications, yet they are underrepresented in pre-trained language models (LMs), resulting often in wrong inferences. In this work, we try to improve the underlying understanding of the negation in the pre-trained LMs. To augment negation understanding, we propose a language model objective with a weighted cross-entropy loss and elastic weight consolidation regularization. We reduce the mean top 1 error rate for BERT-base to 1.1\%, BERT-large to 0.78\%, RoBERTA-base to 3.74\%, RoBERTA-large to 0.01\% on the negated LAMA dataset. It minimizes the BERT error rate by a margin of 8\% and also outperform the existing negation models. We also provide empirical evidences that negated augmented models outperform the classical models on original as well as negation benchmarks on natural language inference tasks.

Negation relates an expression e to another ex- pression with a meaning that is in some way con- trary to the meaning of the original sentence e. NLMs: Augmenting Negation in Language Models
Rituraj Singh, Rahul Kumar and Vivek Sridhar Samsung R&D Institute India, Bangalore (rituraj.s, rahul.k4, v.sridhar) @samsung.com
Abstract
masking tasks (Kassner and Schüitze, 2020). For example, when asked predict the [MASK] to- ken in the sentences: The capital of South Korea is [MASK]. and The capital of South Korea is not [MASK].: BERT often generates the same answer as "Seoul". The exercise gives evidence that LMs do not appropriately model the distribution of nega- tion sentences. LMs when fine-tuned NLI tasks, current pre-trained models tend to misclassify the examples which contain negations as contradic- tions when the true label is neutral or entailment.
Negation fundamental component natural language that reverses the semantic meaning of a sentence. It plays an extremely important role across a wide range of appli- cations, yet they are under-represented in pre- trained language models (LMs), resulting often in wrong inferences. In this work, we try to improve the underlying understanding of the negation in the pre-trained LMs. To augment negation understanding, we propose language model objective with a weighted cross-entropy loss and elastic weight consolidation regular- ization. For negated augmented models, reduce the mean top error rate for BERT- base 1.1%, BERT-large 0.78%, RoBERTa- base to 3. 3,74, RoBERTa-large to 0.01 % on the negated LAMA dataset that outperform the ex- isting negation models. It minimizes the mean error rate by a margin of 8% and 6% for orig- inal BERT and RoBERTa models. We also provide empirical evidences that negated aug- mented models outperforms the classical mod- els on original well as negation benchmarks natural language inference tasks.
In this work, we address the challenge of aug- menting negation understanding of language mod- els and alleviating the model bias. To enhance negation understanding, we adopt data-driven approach by leveraging large dataset extracted from Wikipedia. The dataset serves as the founda- tion for generating diverse set of negated sen- tences. Our approach incorporates dependency parsing and tense morphological structure to sys- tematically construct negated counterparts of the original sentences. By explicitly introducing nega- tion into the data, we aim to improve the model's ability to handle negated statements effectively.
1 Introduction
Negation plays a pivotal role in many natural lan- guage understanding tasks, such sentiment anal- ysis, question answering, Natural Language Infer- ence (NLI) tasks, etc. While large language models such as BERT have pushed state-of-the-art and are being used widely across various domains, it fails dramatically to understand negation.
The next step involves training the language models using paired set of affirmativ--fffimmatie and ceermmatvv-neegatvve sentence pairs. To effec- tively capture the nuances of negation, we employ weighted loss function and an elastic weight reg- ularizer during the training process. This combina- tion enables the models to focus on learning from both affirmative and negative instances, thereby reducing any inherent model bias towards affirma- tive statements. To evaluate the effectiveness of our models, conduct experiments on standard datasets, specifically focusing knowledge com- pletion and NLI tasks.
Negation is ubiquitous in English language text and comprises approximately 25% sentences with nega- tion clues or genre in some form (Hossain al., 2020). Despite this fact, understanding of nega- tion under--epresenned in the language models. The current Language Models (LMs) cannot distin- guish between negated and non-negated forms of
The work presented here addresses the shortcom- ings of negation understanding of LMs and makes the following contributions:
We provide a methodology to automatically

---

generate negated data from the original wiki corpus using dependency parsing, providing a diverse and representative source for negation understanding.
cellu et al., 2016; Morante and Daelemans, 2009; Li and Lu, 2018), attention analysis in negation scope (Zhao and Bethard, 2020) and focus detec- tion (Shen et al., 2019). Authors (Naik et al., 2018) study linguistic phenomenons such as antonyms, negation, and spelling mismatch and find that these models rely on syntactic clues to make the NLE inferences. Hossain et al. (2020) proposes a bench- mark for natural language inference tasks in form of a cott-hypothesis pair which negation plays a critical role. Also, the authors find that the current state-of-the-ar transformers struggle to understand negation which often results in wrong inferences. Noji and Takamura (2020) shows the utilities of explicit negative examples to improve the syntactic abilities of models for the NLI tasks.
2. We propose a pre-training framework with Elastic Weight Consolidation (EWC) regular- ization and weighted loss to overcome catas- trophic forgetting that aids to augment the negation knowledge in LMs.
3. We demonstrate that state-of-the-an trans- formers are not robust to negation and provide empirical evidence that negation augmented LMs outperform the classical LMs on original as well as negation datasets.
2 Related Work
The studies (Hossain et al., 2020; Laverghetta Jr. and Licato, 2022) show that these state-of-the-an transformer models struggle to understand the nega- tion and there exists the need to inject negation un- derstanding in these models. These works investi- gate the negation understanding of LMs. However, only few works try to fix this negation bias in LMs. Hosseini et al. (2021) augments negation derstanding in BERT by modelling objectives with an unlikelihood based on negated generic sentences. The proposed method successfully injects knowl- edge into BERT-base, however, fails BERT-large model.
Language models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have achieved remarkable results in various natural lan- guage understanding tasks such as question an- swering, sentiment analysis, named entity recogni- tion (Gillard et al., 2006; Naik et al., 2018), etc. Their ability to capture contextual information, leverage large scale pre-training and the incorpora- tion of novel training techniques have contributed to their impressive performance. These models also learn broader range of factual and common-sense knowledge (Akyurek et al., 2022). Despite these abilities and performance, Kassner and Schütze (2020) shows that these models fall short on under- standing the negated factual sentences.
Dataset
Various works investigate the linguistic knowl- edge captured in these LMs. The authors (Warstadt and Bowman, 2019) investigate the grammatical knowledge of the LMS and conclude that these mod- els have near human performance on simple sen- tences and fail to make fine-grained grammatical distinctions. Authors (Marvin and Linzen, 2018 8) propose dataset for evaluating the knowledge of understanding grammatically correct as well as in- correct sentences. The oLMpies work investigates BERT and RoBERTa model on the reasoning tasks which require comparison, conjunctions and com- position. Other studies include diagnosing syntac- tic heuristics in NLI tasks (McCoy et al., 2019), in- ner workings of these models on (negative) polarity items (Jumelet and Hupkes, 2018) and discovering of NLP pipelines in BERT (Tenney et al., 2019).
In order to effectively augment negation, we need a dataset for affirmative and negative sentences com- patible with LM's input requirements. However, publicly available dataset with such characteris- tics is not available. An alternative approach is undertaken to extract the wiki-raw-text corpus and negate the sentences.
3.1 Wiki-raw-text corpus
Each page within the wiki-raw-text corpus (Mer- ity et al., 2016) is analysed and the first two lines of text from each page is selected. The rationale behind the selection is that the first line typically provides an introductory overview or summary of the page's content, whereas the second line serves as a continuation or follow-up statement. By uti- lizing this approach collection of sentence pairs are obtained, where the first line represents an an- chor sentence and the second line serves as the follow up sentence.
Some recent work studies negation understand- ing in the area of negation detection (Khandelwal and Sawant, 2020), negation scope detections (Fan-

---

To prepare the dataset for training, pre- processing and cleaning steps are performed to ensure the dataset's quality and consistency. One specific step involves the removal of pronunciations and respelling keys. These keys are often found in text derived from the wikipedia corpus and provide phonetic indications for pronunciation variations. These keys are eliminated to ensure that the focus is solely on the textual content, without the inclusion of phonetic annotations or pronunciation indicators. Through this meticulous process, a comprehensive dataset of 25000 sentence pairs is generated. Next, we describe the method for negating an affirmative sentence.
each sentence, which allows for the extraction of the corresponding tense. For example, if the sen- tence contained the word/POS pair "walked/VBD", would match the pattern for past tense sentences. This approach is generalized and facilitates the au- tomated identification of the tense in sentences, enabling subsequent linguistic analysis and sup- porting the various other NLP tasks such as tem- poral information, extraction, text summarization, sentiment analysis and more.
Following the detection of the tense in each sentence, a negation transformation process is implemented to convert the sentences into their negated forms. This transformation involves the addition, replacement and lemmatization of words within the sentence to convey a negated meaning. To introduce the negation cues, specific modifi- cations are made to the sentence structure. For instance, in the case of past tense the main verb is first lemmatized and the negation cues are added before it. Consider an example "She walked to the park." " After negation transformation, it becomes "She did not walk to the park." " where "did not" serves as the negation cue inserted before the lem- matized main verb "walk". This effectively negates the original positive assertion. Additionally, we consider special keywords such as "somewhere", which are transformed into "nowhere" to express negation. We provide the list of special keywords in Appendix A. By applying these negation trans- formations systematically, the original sentences are modified to represent the negated statements, facilitating the analysis of negation and its impact on the semantic content and context of the text.
Conversion:
3.2 Negation
Consider an affirmative sentence that needs to be converted into its negated form. We generate the negated form of sentence using the dependency parsing tree and syntactical rules. The methodol- ogy for negating sentence is as follows.
Part of Speech tagging: In order to analyse the sentence structure and facilitate further linguistic processing for tense extraction, the Part-of- Spreech-Speech (POS) tags of the given sentence are obtained using POS tagging algorithm'. POS tagging assigns specific grammatical labels to each word in sen- tence which indicates its syntactic category and functions within the sentence. By labelling the words with their respective POS tags, the sentence is represented in the form of word/POS tag pairs, allowing for a structured representation of the sen- tence's linguistic patterns.
Tense Patterns: To accurately determine the tense of each sentence systematic approach is employed involving the utilization of POS tags represented in the form word/POS pairs. To achieve this, a set of predefined patterns for each tense type is specified. These patterns are designed to capture specific linguistic structures associated with dif- ferent tenses. For instance, to identify past tense verbs, pattern such as "\w+VBD" is defined, where "Aw+" represents one or more word characters and "VBD" corresponds to the POS tag for past tense verbs. Similarly, other tense patterns are devised, such \W+VBP for present tense verbs and \w+MD for modal verbs. We provide in Appendix A. the full list of patterns for each tense. Subsequently. by uti- lizing regular expressions (regex), the predefined patterns matched to the word/POS tag pairs of
We illustrate below some of the examples in Ta- ble In addition, evaluate the negation, we man- ually verify 100 random negated sentences. The evaluation made by English native speakers re- sulting in an overall precision of 96%. In general, the method can also be applied to other cases to generate negative samples.
3.3 Sample Generation
Based on the extracted two lines (first sentence, follow-up sentence) from each page of the wiki- raw-text corpus, two types of data are created. The first type retains the original format, where the sentence pairs consist of the affirmative sentence accompanied by the corresponding follow-up or continuous statement. The data serves as a control group for training the model with affirmation sen-

---

Derived Teuse ASentences
Example Affirmative Sentence
Converted Negalive Sentence
Simple Pasi 11186
The episode premiered on the Fox network on October 7.
The episede did not premiere on the Fux network on October 7.
Simple Present
1397
Brackton is located approximately 25 miles of Boston.
Brockton is not located approximately 25 miles of Boston.
Searching a specific search nccording binary
Searching a specific search rree according to binary key camnol be recursively ileratively programmned.
Modular verb
493
key rocursively or ileratively programumed.
Table 1: Summary of few derived tense category from the dataset of 25000 sentences, number of sentences along with examples of affirmative and their corresponding converted negated sentences.
tences. The second type of data involves negating the second sentence using the techniques described earlier. It allows for the examination of the effects of negation on the overall meaning and context of the sentence pairs. The first sentence remains unchanged as the affirmative statement, while the second sentence is now represented as the negated form of the original follow-up statement. The ap- proach facilitates the creation of two distinet types of datasets: one representing the original unaltered sentence pairs and the other representing the orig- inal sentence with negated follow-up statements. We present the process in Figure 1.
However, in situations where the object is not explicitly mentioned or unavailable, the subject of the sentence is masked instead. For instance, con- sider the sentence "Cat is sleeping". Since there is no explicit object in the sentence, the subject "Cat" is masked, leading to the transformed sentence "[MASK] is sleeping". The masking technique enables the examination of the effect of affirma- tion as well as negation on the subject component. Overall, masking helps understand the seman- tic meaning and interpretation of a sentence. Note that, we do not mask any token of S.. It acts as grounding (context) for the next sentence.
Next Sentence Prediction: In certain models, that incorporate the Next Sentence Prediction (NSP) task, specific labelling is required to indicate the relationship between pairs of sentences. To address this requirement, the following labelling scheme is implemented:
4
Model
In this section, we describe the training sample generation and the model details.
4.1
Training Samples
The training data consists of the following samples: 1. Sy) are unaltered sentence pairs where Sa is the affirmative sentence and Sy is the follow-up sentence.
1. For pairs of affirmative sentences (Sa, Sy). la- bel O is assigned. It indicates that the two sentences are affirmative and are contextually related.
2. For pairs consisting of an affirmative and a neg- ative sentence (S., Syn), a label of 1 is assigned. This label signifies that the two sentences have a contrast in meaning or sentiment.
2. (Sa, Syn) where S, is the affirmative sentence and Sjn is the negated sentence of Sy.
In each pair, we mask the tokens in the follow-up sentence (second sentence). The first sen- tence acts as an anchor sentence and gives context while predicting the MASK token in the second sentence.
Masking:
The labelling scheme enables the NSP task to effec- tively capture the relationship and context between pair of sentences. helps in training and evalu- ating the model's ability to capture the semantic relationship and contextual nuances between sen- tences, particularly in scenarios involving negation. We explicitly imply the model for the following learning: (a) Given an affirmative sentence Sa as context, predict the [MASK] token of the next af- firmative sentence S, such that S is the actual sentence that follows S,, (b) Given an affirmative sentence S, as context, predict the [MASK] token of S Sfn (negated sentence of Sy) such that negated sentence is not the actual sentence that follows the sentence S,-
The strategy for masking involves replacing ei- ther the object or subject of sentence with special token "[MASK]", to indicate that the entity has been masked. The technique allows for a more fo- cused analysis on the impact of negation on specific components of the sentence.
When masking the object of a sentence, the direct object is identified and replaced with the "[MASK]" token. For example, consider the origi- nal sentence "John ate apple". In this case, the object "apple" is masked resulting in the modified sentence "John ate an [MASK].". This masking allows us to isolate the effect of negation on the object component of the sentence while keeping the rest of the sentence intact.
4.2 Regularization
Fine-tuning language models the most critical part of the transfer learning. The aggressive fine-

---

Affirmative Sentence (s
Affirmative Sentence
Affirmative Sentence (s)
MASKED Affirmative Sentence
MASKED Affirmative Sentence 2, Label: 0
Sentence
NLMs Negated Language Models
Train the LM's [Initialised with pro- trained weights)
Negate Sentence
Affirmative Sentence
Sentence 2 (n)
Negative
Sentence
Mask
MASKED Negative Sentence
MASKED Negative Sentence 2, Label: 1
Figure 1: Data generation affirmative-fffrmatve affirmative-negatinn pairs followed by training models.
tuning causes catastrophic forgetting, climinating the knowledge of the information captured through previous learnings. On the other hand, too cautious fine-tuning leads slow convergence and leads overfitting. To eliminate these challenges, deploy the strategies of freezing the layers and con- trolled change in the set of biases and weights to optimize performance and learning. Rather than fine tniig all the layers, we first propose to update only the top layers in models. Secondly, we apply EWC, a regularization technique to control forget- ting specifically by constraining the model weights that are important for previous tasks. The sensitiv- ity of the model with respect its weight 0; can estimated by looking at the curvature the loss surface along the direction of change 0,. The high curvature signifies the slight 6, change, result- ing in sharp change in loss. To stop the weights of the model deviating from the original base model, we constrain the parameters using elastic weight consolidation (Kirkpatrick et al., 2017). Learning task, consists adjust the diagonal of Fisher Infor- mation matrix F, which signifies the second deriva- tive of the loss near minimum. The 1tth diagonal element in F denotes the importance of weight 0g. The intuition is to move the important weights (from the actual model) as little as possible, when the model is fine-tuned on the negation learning task. It is achieved by adding a regularization term to the loss function and is shown below.
loss functions that combines the objective of both tasks. The idea is to assign appropriate weights to each task's loss component based on their respec- tive importance and impact model performance. enables the model to capture both sentence-level representation and word-level representation lead- ing to improved language understanding and gener- ation capabilities.
Let LNSP be the loss associated with the NSP task and LALM denotes the loss associated with the MLM task. We introduce the weighting factor [0, 1] balance the contributions of each loss component. We denote the total loss with Lo and defined below.
Lg = aLMLM + (1 -aa
We augment negation in BERT and RoBERTa lan- guage models with the above defined regularization and loss function. We describe the full training pro- cess in Figure 1.
5 Experiments
5.1 Evaluation Dataset
Language Model Analysis (LAMA)): We use the LAMA probe (Petroni et al., 2019) in our experi- ments, which has emerged as a valuable dataset for evaluating factual knowledge. The evalua- tion dataset is collection of cloze-style questions about real-world facts. The statements are facts or common-sense knowledge generated from ei- ther subjecc-relatioonoojeee triples (X. rel, Y) or question-answer pairs. LAMA leverages the mask set-up, where the model presented with a statement is required to predict the MASK token with single token answer. The statements are generated by fol- lowing template for cach relation, which include place-holders X and Y. Here, X is the subject and Y is replaced by the MASK token. For example, for triples (i0s, developed, Apple), the statement is ios is developed by Apple. After masking, the statement i0S is developed by [MASK]
LEWC Lo + 2 Σ F. - 8))²
Here Le denotes the loss, O; denotes the original model weights, 0% denotes the new model weights while training and À is the importance factor.
4.3 Loss
Next Sentence Prediction (NSP) and Masked Lan- guage Modelling (MLM) are the two most popular modelling tasks. To leverage the strengths of both the NSP and MLL tasks, we propose weighted

---

and is fed into the model to predict the MASK token with the ground truth value being Apple. LAMA is composed of various datasets. Google- RE? consists of three relations, namely "place- of-birth", "date-of-birth" and "place-of-death". T- REx (Elsahar et al., 2018) 8) is a subset of wikipedia data triples which consists of 41 relations. Con- ceptNet consists of 16 relations from conceptNet knowledge base (Li et al., 2016). Squad is sub- set of 305 context-insensitive questions manually rephrased as cloze style questions (Rajpurkar et al., 2016). We summarize the LAMA dataset in Ap- pendix A.3.1.
large models. We choose a maximum length of 128 and use AdamW an optimizer with a linear rate scheduler. We apply EWC regularization, to conserve the previous learning as well as constrain the deviation from the original pre-trained weights. In addition, we use the weighted loss to train the models explained in Section 4.3. We manually experiment with different sets of hyperparameters in the experiments and refer readers to Table 17 in Appendix for the detailed final list of hyperparame- ters. All the models are trained on two 24 GB P40 GPUs.
5.3 Results
Negated LAMA: Negated LAMA (Kassner and Schütze, 2020) was created by manually inserting the negation cues in the template of the LAMA dataset.
Can the negated augmented transformers, outper- form and solve the LAMA, negated-LAMA and the text-hypothesis pairs including negation as com- pared to the original models? Yes, they can.
Natural Language Inference Dataset: We eval- uate the pre-trained models on RTE (Dagan et al., 2005; Giampiccolo et al., 2007; Bentivogli et al., 2009), SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018) datasets. As nega- tion is represened (Table 16 in Appendix for reference) in these datasets, we also consider the negation dataset of these tasks from Hossain et al. (2020).
Knowledge Base Completion: We evaluate the knowledge base completion task on LAMA and negated-LAMA datasets. In order to evaluate the proposed negation augmentation, we first evaluate the models on negated LAMA datasets. Follow- ing Hosseini et al. (2021), we report precision at k (higher is better) for original LAMA queries and mean top error rate for negated LAMA queries (lower is better). We report the results for the negation cases for BERT-base and BERT-large in Tables 3 and Table 5. NBERT and NRoBERTa denotes the negated augmented language models (NLMs). We make the following observations. The augmented negation model for Bert-base outperforms the original BERT model by a margin of 8% and Hosscini et al. (2021) by a margin of 2%. For BERT-large models, the method outperforms the original BERT-large and the state-of-the model by 8%. As there exist no previous works on the augmentation of negation in RoBERTa models, compare only the original models in Table 7 and Table 9, The proposed methodology outperforms the original Roberta-base model by 4% and the original Roberta-large model by 9%, Note that, the previous approaches BERTNOT (Hosseini et al., 2021) were not able to augment negation in larger models. However, the proposed technique effec- tively augments negation understanding in the base as well as large models. We also report the sults for the affirmation cases in BERT-base. BERT- large, RoBERTa-base and RoBERTa-large in Ta- bles 2, 4, and We find that augmenting negation understanding in these models has a subtle effect on affirmative LAMA queries and in most of the cases
5.2 Experiment Setups
We aim to investigate the effects of augmenting negation in BERT and RoBERTa models of vari- ous sizes. Prior research (Hosseini et al., 2021), indicates that incorporating negation understand- ing into models with different sizes can present challenges and may yield varying results.
In our experiments, we employed state-of-the- art transformers BERT and RoBERTa models of varying sizes, namely BERT-base, laRT-large, RoBERTa-base and RoBERTa-large.
By including models of different sizes, we aim to study how the size of the underlying large language models affects the performance and adaptability of the negation augmentation technique. Note that the training approach requires careful calibrations as the objective not only to augment the nega- tion knowledge in these language models but also to preserve the previous learnings on affirmation sentences. We do the experiments with several set- tings and conclude to train the pre-trained models for 2 epochs with 5000 examples. We update top 5 layers of base models and top 10 layers of the

---

Mode/LAMA
SQuAD
ConceptNet
TREx
GoogleRE
ModeINeeLAMA
SQUAD
ConceptNet
TREA
GoogleRE
BERT-basc
13.53
15.65
29.10
10.24
BERT-base
8.61
2:24
21.42
3.76
BERINOT-bas--KL13
13.64
15.64
29.28
10:27
BERTNOT-KLI9]
4.97
1.19
21.77
3.99
BERTNOT hase [3]
13.97
13.49
29.25
10.31
BERTNOT [3]
2.10
11.86
1.10
NBBERT
base
Table 2: Mean precision k=l(p@] for origi- nal LAMA queries (higher is better) of proposed NBERT- base model with baselines.
Table 3: Comparision of Mean top emror rate for
negated LAMA queries (lower is hetter) of proposed
NBERT-bast model with baselines.
Model /LAMA
SQuAD
ConceptNel
TREA
GoogleRE
Model INegLAMA
SQUAD
ConceptNet
TREA
GoogleRE
BaRT-large
10.93
BERT-large
7.95
22.97
4.13
BERINOT-lage/B) [3]
14.19
19.14
32.09
11.02
BERINOT-lare [3]
8.28
1.87
23.49
4.22
NBERT-large
17.04
18.33
32.37
10.33
NBERT-large
0.98
1.62
0.53
0.01
Table 4: Mean precision kpl(p@] 1) for origi- nal LAMA queries (higher is better) of proposed NBERT-large model with baselines.
Table 5: Comparison Mean top error rate for negated LAMA queries (lower is better) of proposed NBERT-large model with baselines.
Model /LAMA
SQUAD
ConceptNet
TREx
GoogleRE
Model .AA SQIAD 8.55
ConceptNet
TREx
GoogleRE:
RoBERTa- base
11.51
5.16
16.44
4.38
RoBFRTTa-aae
2.04
17.75
1.20
NRoRER ha-ase
9.86
4.78
19.74
3.51
NRoBERTa-bose
Table 6: Mean precision k= kpl(p@) 1) for origi- nal LAMA queries (higher is better) proposed NRoBERTa-bar base model with baselines.
Table 7: Comparison of Mean top error rate for negated LAMA queries (lower is better) of proposed NRoBERTa-baa model with baselines.
Mode//LMMA SQUAD
ConceptNet
TREx
GoogleRE
Model (NegLAAMA
SQuAD
ConceptNet
TREs
GongleRE
RoBERia-large 18.09
6.91
21.17
RoHFRTh-large
2.14
NRoBERT2-large
16.77
7.28
24.03
3,37
NRoBERTa base
0.0
0.04
0.01
Table 8: Mean precision at k=l 1) for origi- nal LAMA queries (higher is better) proposed NRoBERTa-large model with baselines.
Table 9: Comparison of Mean top 1 error rate for negated LAMA queries (lower better) of proposed NRoBEKTa-large model with baselines.
Sentence
Top-3 Prediction by BERT
Top-3 Prediction by NBERT
The capital of South Korea is [MASK].
Seoul, Tokyo, Korea
Seoul, South, Ha disputed, Olympics, merged Paris, Rome, Office Poverty, France, death
The capital of South Korea not [MASK].
Seoul, Known, listed
Charles Nodier died in [MASK]
Paris, Rome, Office
Charles Nodier did not dic in [MASK]
Paris, Rome, Office
Table Example showcasing prediction BERT-base and NBERT-baae. The wrong prediction by BERT is highlighted in bold.
RTE
SNLI
MNLI
Test Pairs
MB
BERT-base
NBERT-base
MB
BERT-base
NBERT-buse
MB
BERT-base
NBERT-base
Original dev
52.7
66.1
71.14
33.8
89.9
90.65
35.5
83.2
83.4
dev neg
$1.2
63.4
56.1
54.4
89.4
90.78
50.2
83
82.8
New w/ neg.
Tneg-H 80.2
65.2
72.8
32.6
38
45.8
64.2
T-Hneg 91
39.2
84.6
41
58.8
60.6
47.6
62.4
65.6
Tueg-Hnegg
65.6
68.4
56
69.8
41.8
35.8
47
63.6
61.4
All 78.9
57.6
71.14
57.6
44.4
44.8
46.8
63.9
63.8
Table 11: Comparison of state-of-the-ar models trained with the original training split for each benchmark and
evaluated with the original development split (dev), pairs in the original development split containing negation (dev neg), and the new pairs containing negation (New w/neg). MB stands for the majority baseline. BERT-ban base is the original base model and NBERT-base model the proposed negated augmented BERT-base model.

---

ALAMA
SQUAD
ConceptNet
TREX
GoogleRE RE
Model InegLAMA
SQuAD
ConceptNet
TREs
GooglRE
NBERT-base
11.14
14.27
30.02
10.44
NBERT-base
1.1
0.66
2.39
0.06
Without EWC
11:47
14.00
28.54
9:42
Without EWC
0%
0,52
1.96
0.09
1.2 Regulorzaiion
11.47
1399
20093
9:01
1.2 Regularivation
1.31
0.66
1.3
0.06
MLMM
13:99
28.12
MI
042
MLM 4 NSP
1213
13.78
28.55
9443
MLM NSP
1,77
0.79
2.78
0.09
14.79
7.07
Train data-size-
Train datasize 20000
10.16
14.77
28.95
7.38
Train datasize 20000
4.60
1.88
14.95
0.17
All layers updates
13.90
layers updates
Ablation study signifying the impact of the regularizers namely EWC and I.2, different loss functions, data size and all layers update. The left table shows the results original LAMA queries (Higher is better) and the right table shows the result on negated LAMA queries (Lower is better).
Table 12:
performs at par with the original models. We ob- serve that RoBERTa-large model is more adaptable negation understanding compared to other models. We also showcase few examples of the comparative results between BERT and NBERT for the knowledge base completion task. As show- cased in Table 10, the BERT model is not able to understand negation and predicts wrong com- pletion whereas the NBERT predicts the tokens correctly.
variations. We make the following observation. In individual cases such without using EWC reg- ularization, L2 regularization, only MLM, equal weight for MLM and NSP (same as original BERT) and updating
the layers, we find that the ac- curacy drops for LAMA queries. Whereas, using greater data sizes namely 10000 and 15000 leads to an increase in eπror for negated LAMA queries. It is intuitive, as a larger number of negation samples may bias the models towards negation and lead to forgetting of affirmation understanding. The study signifies the importance of optimization to mini- mize the deviation from the original model as well as augment negation understanding in the model.
Natural Language Inference: We also evaluate the proposed models on RTE, MNLI and SNLI tasks. We fine-tune the models on original devel- opment splits and the new splits from Hossain et al. (2020) which consists of negation for each task. We use the same set of hyper-parameters from Hossain et al. (2020) to fine-tune models for BERT-base and RoBERTa-base. We summa- rize the results for BERT-base models in compar- ison to state-of-the-ar models in Table 11. The fine-tuned NBERT-base model achieves superior re- sults on original dev splits on RTE and SNLI tasks as compared to BERT-base and majority baseline. For negative dev splits, we observe the NBERT- base model outperforms the original models on SNLI tasks and performs poorly on RTE and on- par MNLI datasets. For the new negation pairs, the NBERT-base model outperforms the original BERR-base-model on RTE and SNLI tasks and almost par on MNLI tasks. We also evaluate BaRT- RoBERTa- base and RoBERTa-large models and illustrate the results in Appendix A.4.2. We find similar observations for Roberta models. The experiment shows that the augmentation of negation understanding outperforms the original models in most of the downstream tasks.
Conclusion
Negation is ubiquitous in language and plays crucial aspect in language understanding. We in- troduce a negation aware technique that addresses the negation bias of the existing LMs. We modified the training data by augmenting it with explicitly negation data and trained the model with EWC reg- ularization with weighted loss. By incorporating. explicit negation cues during training, the proposed approach enables the model to better capture the subtle nuances of negation, thus resulting in im- proved contextual understanding and more accu- rate language completion. The results demonstrate that NLMs significantly improve the LMs perfor- mance understanding and representing negated contexts, leading to more accurate predictions knowledge completion and NLI tasks.
In the future, we plan to augment negation in cross-lingual and generative models. Augment- ing negation in these models involves several challenges such as different syntactic and lexical cues, inherent difficulty of aligning and training paradigms.
5.4 Ablation Study
To test the effectiveness of our approach, we ab- late the individual cases with fixed set of other parameters. As shown in Table 12, our training approach setting is effective as compared to other

---

Limitations
deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguissics.
While our approach yields promising results in improving negation understanding and mitigating modelling bias, we acknowledge a limitation in the case of tiny models. The tiny models are low- capacity models with limited parameters, designed to be lightweight and efficient in terms of computa- tional nuanced understanding of a language model. Due the constrained architecture, these mod- els struggle to effectively capture and generalize the negation augmented tiny model on LAMA and negated LAMA dataset summarized in Table 23 and Table 24. We find that augmenting negation not yields substantial improvements in their perfor- mance.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Lafor- est, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (L.REC 2018).
Federico Fancellu, Adam Lopez, and Bonnie Webber. 2016. Neural networks for negation scope detection. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume
Also, we limit our work to encoder- mod- els. However, there requires further exploration to understand the embedding of negation knowledge in GPT-based generative models. By addressing these limitations, future work pave the way for a more comprehensive and effective understanding of negation in language models.
1: Long Pupers), pages 495-504, Berlin, Germany. Association for Computational Linguistics.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1-9, Prague. Association for Computational Linguistics.
Ethics Statement
L. Gillard, P. Bellot, and M. El-Bèze. 2006. Ques- tion answering evaluation survey. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06), Genoa, Italy, European Language Resources Association (ELRA).
The research work complies with the EMNLP ethics policy and does not raise any ethical con- cerns.
Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020. An analysis of natural language infer- ence benchmarks through the lens of negation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9106-9118, Online. Association for Computa- tional Linguistics.
References
Ekin Akyurek, Tolga Bolukbasi, Frederick Liu, Bin- bin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. 2022. Towards tracing knowledge in language models back to the training data. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2429-2446, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Arian Hosseini. Siva Reddy, Dzmitry Bahdanau. R De- von Hjelm, Alessandro Sordoni, and Aaron Courville. 2021. Understanding by understanding not: Model- ing negation in language models. Proceedings of the 2021 Conference of the North American Chap- of the Association for Computational Linguistics: Human Language Technologies. pages 1301-1314 Online. Association for Computational Linguistics.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. In TAC. Citeseer.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lishon, Portugal. Association for Compu- tational Linguistics.
Jaap Jumelet and Dieuwke Hupkes. 2018. Do lan- guage models understand anything? on the ability of LSTMs to understand negative polarity items. In Proceedings of the 2018 EMNLP Workshop Black- boxNIP: Analyzing and Interpreting Neural Net- works for NLP, pages 222-231, Brussels, Belgium. Association for Computational Linguistics.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Machine learning challenges workshop, pages 177-190. Springer.
Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of

---

58th Annual Meeting of the Association for Compu- tational Linguistics, pages 7811-7818, Online. Asso- ciation for Computational Linguistics.
Roser Morante and Walter Daelemans. 2009, A met- alearning approach to processing the scope of nega- tion. In Proceedings of the Thirteenth Conference Computational Natural Language Learning (CoNLL- 2009), pages 21-29, Boulder, Colorado. Association for Computational Linguistics.
Aditya Khandelwal and Suraj Sawant. 2020. Neg- BERT: A transfer learning approach for negation de- tection and scope resolution. In Proceedings of the Twelfth Language Rexources and Evaluation Confer- ence, pages 9739-5748, Marseille, France. European Language Resources Association.
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340-2353, Santa Fe, New Mexico, USA. Association for Com- putational Linguisties.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milam, John Quan, Tiago Ramalho, Ag- nieszka Grabska-Bary Barwinska, et al. 2017. Over- coming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13)3321-3356.
Hiroshi Noji and Hiroya Takamura. 2020. An analysis of the utility of explicit negative examples to im- prove the syntactic abilities of neural language mod- cls. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 3375-3385 Online. Association for Computational Linguistics.
Antonio Laverghetta Jr. and John Licato. 2022. De- velopmental negation processing in transformer lan- guage models. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 545-551, Dublin, Ireland. Association for Computational Lin- guistics.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, An- ton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowl- edge bases? arXiv preprint vrXivv:909.001
Hao Li and Wei Lu. 2018. Learning with structured representations for negation scope extraction. Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 2: Short Papers). pages 533 539, Melbourne, Australia. As- sociation for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas, Association for Computational Linguisties.
Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel. 2016. Commonsense knowledge base completion. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume
Longxiang Shen, Bowei Zou, Yu Hong, Guodong Zhou, Qiaoming Zhu, and AiTi Aw. 2019. Negative focus detection via contextual attention mechanism. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- pages 2251-
1: Long Pupers), pages 1445-1455, Berlin, Germany. Association for Computational Linguistics.
guage Processing 2261, Hong Kong, China. Association for Computa- tional I Linguistics.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy. Association for Computational Linguistics.
Rebecca Marvin and Tal Linzen. 2018 Targeted syn- tactic evaluation of language models. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202, Brussels, Belgium. Association for Computational Linguistics.
Alex Warstadt and Samuel R Bowman. 2019. Linguistic analysis of pretrained sentence encoders with accept- ability judgments. arXiv preprint arXiv: 1901003900
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019, Right for the wrong reasons: Diagnosing syntactic heuris- tics natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 3248, Florence, Italy, Association for Computational Linguisties.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-covern challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long Papers), pages 11 12-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Stephen Merity, Caiming Xiong. James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod-
els.

---

Yiyun Zhao and Steven Bethard. 2020. How does BERT's s attention change when you fine-tune? an analysis methodology and case study in negation scope. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages Online. Association for Computational Linguisties.
A.3 Evaluation Dataset
A.3.1 LAMA dataset
We summarize the LAMA dataset in Table 15. LAMA consists of datasets from various sources namely Google-RE, TREx, ConceptNet and Squad.
A.3.2 Natural Language Inference dataset
A Appendix
We provide below summary of percentage of nega- tion in NLI benchmarks (Hossain et al., 2020).
We provide the details on tense patterns and the cor- responding actions to convert affirmative to nega- tive sentences in Section A. Section A.2 gives the details of hyperparameters to train the model and to fine-tune the model NLI tasks. We give more de- tails on the evaluation dataset for knowledge com- pletion and NLJ tasks in Section A.3. Section A.4 provides more examples from knowledge base com- pletion tasks and benchmarks on NLI tasks for BaRg-,arree RoBERTa-base and RoBERTa-large.
A.4 Additional Results
We provide some more examples of knowledge- base completion tasks for the BERT-large model. We also summarize the evaluation for NLI tasks on BERT-large, RoBERTa-bass and RoBERTa-large models. Note that, for the RoBERTa base model, we compare our results with stateoof-the-ae models. In the case of BERT-large and RoBERTa-large. we compare the negated augmented models with the original models.
A.1 Tense Patterns
A.4.1 More Examples Knowledge Base Completion tasks
We summarize the pre-defined tense patterns for each tense in Table 14. We also give operations for each tense type to convert an affirmative a negative sentence along with some examples. In Table 13, we provide the list of special words and their corresponding negated form. We replace the special words with the negated form to convert sentence to its negated form.
We provide below some more examples for the BERT-large model in Table 19. Observe that the original BaRT-large model predicts the same token for affirmative and negative sentences. The wrong predictions are highlighted in bold. The negated augmented NBERT-large model predicts the cor- rect tokens.
Tokens
Negated Form
Tokens Anybody
Negated Form
A.4.2 Benchmarks models on NLI tasks
Ever
Never
Nobody
Anyone
Noone
Anything
Nothing
We also benchmark the negation augmented trained models namely RoBERTa- NBERT-base and RoBERTa-large on RTE, SNLI and MNLI tasks on original splits as well as negation splits. We find that the proposed NBERT-large model outperforms the original BERT-large in all three tasks on orig- inal and negation splits and is shown in Table 20. For RoBERTa- base and large model, the negated augmented model performs as par as the original models and is shown Table 21 and Table 22.
Anywhere
Nowhere
Someone
Noone
Somebody
Nobody
Someone
None
Always
Never
Bither
Neither
Table 13: List of special tokens and their negated form. The tokens are replaced by the negated form to convert the affirmative sentence negative sentence.
A.2 Reproducibility
A.2.1
Hyper-parameters for augmenting negation
A.4.3 Benchmark of BERT-tiny on LAMA and LAMA datasets.
Table 17 lists all the haper-parameters used in our fine-tunings approach for different models and size.
We also tried out the methodology to augment nega- tion in BERT-úny We find that augmenting nega- tion in the tiny model is challenging and the mod- els augmented with negation perform poorly on LAMA well negated-LAMe queries. We summarize the results in Table 23 and Table 24 below. The work requires further investigations.
A.2.2 Hyper-parameters for NLI tasks evaluation
We list the hyperparameters to fine tune the models on NLI tasks in Table 18.

---

Seatrose Type
Parsen
Operation
Pinrrn Manch
Esangle
Modala: Vel
OiDD to MDD 1w//PPP
Adl acgaico cazs "oof" añee roediar vab
wordMD
boad diecter another zujot MCM.
la 1806. weeld abo bucch
Non-3d person Pleseat Those
Se -NPP
Mi legitin "do not befoce veth
WGVVSP
They use coreenss cansorrdes and astio acoddess condact own urvellines of public.
Trd peross Preuze Tread
Add sogatin cae" "dres nor" attee woth
IntevVIZ
Doechelle Vallyy and mlltic on Cattrens
The musn line inrresar 10 miswwy betwon
Pat Teae
tw//BD
Add aopaion cac 'ast" sllervats
warVDD
Buil in 1840, il part ol Lille Rock
les miltnry insull attilaation
Pas Purkiipe
Ww/VEN
Mi aqaatin cne 'dd aut before lemmatiid vod.
CudifedNBB
Through deir pes ceformnee in the Tweety Cap.
sualifted dee Charpoot Loguue Tweest
Var base forz
Adl aopdioon aut beisee voth
Secnal compentes maadly retalion use the company serviess to slsncae pecdasts
Proper Suggila Flod
IeHNSTPP NNNS
Add iegation 'Loes ou." dlter valb
produciceNNN saUsNNS
Beyonce and All Luter
The Ranfors Files production stars Elbe..
Table 14: Examples of pre-defined tense patterns for various sentence type. The operation signifies the different actions based sentence type convert affirmative negative sentence. The Pattern Match column indicates the matched pattern from the example column.
Statistics
Corpus
Relation
Facts
Relations
birth-place
2937
NIA Benchmarks
#Sentences % w/ negation
birth-date
1825
RTE
16389
7.16
Google-RE
death-ppace
765
SNLI MNLI
1138598
1.19
Total
5527
883436
22.63
N-1
20006
23
Table 16: The proportion of sentences contain- ing negation is lower than English language cor- pora(approximately 25%) existing natural lan- guage inference benchmarks. This is particularly true for the RTE and SNLI datasets.
T-REx
N-M
13096
Total
34039
41
ConceptNet
Total
2996
Squad
Total
305
Table 15: Statistics of LAMA dataset.
Parameters
BERT-base-casee
BERT-large-cssed
KoBERRTa-base-uuae RoBgEKlh-lagge-caee
Bert-tiny
Batch Size
64
64
Epoch
2
2
2
2
Max Tokens
256
256
256
256
256
#Training Samples
5000
5000
5000
5000
5000
#EWC Samples
30000
30000
30000
30000
30000
FWC Importance
le-16
le-16
le-16
le-16
16-16
# Layer Freeze
5
10
10
|
Learning Rate
le-4
5e-5
le-5
5e-5
Adam Epsilon
le-8
1e-8
le-8
1c-8
le-8
Clip Grad Norm
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
Table 17: Hyperparameters for fine-tuning LMs to augment negation understanding.
Hlyperparameters
RTE
SNLI
MNLI
BERT-base/
RoBERTa-bbasel
BERT-base/ BERT-large
RoBERTa-base
BERT-base/
RoBERTa-basee
BERT-large
RoBERTa-large
RoBERTa-large
BERT-large
RoBERTa-large
Batch size
8
16
32
32
32
32
Learning Rate
2c-5
2c-5
le-5
Ic-5
2c-5
2c-5
Epochs
50
10
3
3
3
3
Weight decay
0.0
0.0
0.1
0.1
0.0
0.0
Table 18: Hyperparameters for fine-tuning models on RTE, SNLI. and MNLI tasks.

---

Sentence
Top-3 prediction by BERT
Top-3 prediction by NBERT
A teacher is most likely teaching [MASK].
school, university, college
school, university, college age, school, funeral Paris, Liège, Ljubljana age, friendly, the
teacher is not most likely teaching a [MASK].
school, university, college
Marcel Oopa died in the city of [MASK].
Paris, Warsaw, Helsinki
Marcel Oopa did not die in the city of [MASK].
Paris, Warsaw, Helsinki
Bible [MASK] text.
religious, sacred, complete religious, sacred, complete
religious, sacred, Christian new, friendly, novel
Bible is not a [MASK] text.
Table 19: The prediction of BERT-large and NBERT-arge,, wrong predictions are highlighted in bold.
RTE
SNLI
MNLI
Test pairs NBERT BERT-large
BERT-larg
NBERT-large
BERT-large
NBERT-lage
Original dev
58.27
72.26
52.4
91.69
66.87
65.4
dev neg
46.34
51,21
60.2
91.24
55
70
New w/ neg-
Tneg-H
65.6
78.4
48.6
47.6
64.8
65.4
T-Hneg
40.4
80.4
62.8
60.6
70
70
Tneg-Hneg
68.8
64
45.8
42.8
65.8
64.8
All 55.2
74.2
52.4
50.3
66.9
66.7
Table 20: Comparison of BERT-large models trained with the original training split for each benchmark and evaluated with the original development split (dev), pairs in the original development split containing negation (dev neg), and the new pairs containing negation (New w/neg).
Test Pains
MB
RoHER la-se
NRoBFRTnnhhee
MR
KOHERTa-Nas
NKoBERTa-haee
MB
KoRE-KTa-base
NRoBFRTs-hase
Original
75.8
88
dev
51.2
78.1
79.1
54.4
91.7
91.3
50.2
87.2
New w/ neg.
Tneg
J-Hneg 91
51.4
63.6
65.4
47.6
70.4
69.2
Tneg-Hoog 65.6
65.4
66.2
69.8
45.8
44.2
47
63.6
64.2
Table 21: Comparison of RoBERTa-base models trained with the original training split for each benchmark and evaluated with- the original development split (dev), pairs in the original development split containing negation (dev neg), and the new pairs containing negation (New w/neg).
RTE
SNLI
MNLI
Test Puirs
RoBERTa- large
NRoBERTa-large
RoBERTa large
NRoBERTH large
RoBERTa large
NRoHERTa- large
Original
91.5
neg
73.17
80.49
64.4
92.3
60.2
89.2
New neg.
Tneg-H
88
93.4
45.4
50.4
67.8
67.4
T-Hneg
87.6
90.6
66
70.4
73.6
74.2
Theg Hoog
81.8
81.2
44.2
48.8
67
67.2
All
85.8
88.4
51.9
56.5
69.5
69.6
Table 22: Comparison RoBERTa-large models trained with the original training split for each benchmark and evaluated with the original development split (dev), pairs in the original development split containing negation (dev neg), and the new pairs containing negation (New w/neg).
Model /LAMA
SQUAD
ConceptNet
GooglRR
INegLAMA
SQUAD
ConceptNet
GongleRE 2.00
BERT-iny
1,96
2,58
9.84
3.91
BERT day
1.97
0.93
8.58
NBERT-imm tiny
1.38
2.04
1.25
NBERT-iay
1.38
0.30
0.94
0.0
Table Mean precision at k=1(p@] 1) for original LAMA queries (higher is better) of proposed BERT- tiny model with baselines.
Table 24: Comparison Mean top error rate for negated L.AMA queries (lower is better) of proposed BERT-tiny model with baselines.


Reviews:
[{"review_id": "vmTkKMKK4A", "reviewer": "Reviewer_ZAkP", "summary": "", "questions": "1. Although the proposed method can mitigate the issue of not recognizing negation in LLMs, it is still hard to explain why the LLMs understand the negation. So do you have any solution for LLMs to explain the negation understanding?\n2. The negation appears not only in LLMs but also Vision-Language Models. Can your proposed method be applied to these?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and suggestions.\n\n1. We agree that the paper do not investigates the underlying reason that why language models understand negation. The study's primary objective is to enhance negation understanding in language models. Negation is underrepresented in training data. Intuitively, it is the reason that transformer models struggle to understand negation. The study of explainability in negated language models is a novel and interesting area of research that will help to understand interpretability aspects of these NLMs. We consider this aspect to be a future research direction.\n\n2. Negation is omnipresent in both language and vision tasks. For instance, in the context of visual question answering, a language model must be able to comprehend negation when generating responses, especially when the question requires an understanding of negation in relation to the visual scene. Without negation understanding, the language model will fail to give the right responses. The study aims to enhance negation comprehension in language models. However, we propose that utilizing negated language models in cases of negated visual question answering tasks will yield better outcomes. This problem presents an interesting research opportunity, opening up new avenues for exploration.\n\n3. We provide a comprehensive list of all parameters used in both our training and evaluation experiments in appendix. To enhance the reproducibility of our work, we will provide a more detailed explanation of negation generation, computing infrastructure, and other relevant details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper proposed a new weight consolidation regularization that can help the negation understanding better.\n* The paper tested on multiple benchmarks demonstrating the consistency and power of the proposed method.", "reasons_to_reject": "* The method is not explainable although it helps negation understanding.", "questions_to_authors": "1. Although the proposed method can mitigate the issue of not recognizing negation in LLMs, it is still hard to explain why the LLMs understand the negation. So do you have any solution for LLMs to explain the negation understanding?\n2. The negation appears not only in LLMs but also Vision-Language Models. Can your proposed method be applied to these?", "paper_topic_and_main_contributions": "This work focuses on addressing the issue of negation understanding in pretrained language models (PLMs). Negation is a crucial element in natural language that reverses the semantic meaning of a sentence. However, it is often inadequately represented in PLMs, leading to incorrect inferences in various applications.\n\nTo improve the understanding of negation, the authors propose a novel approach. They introduce a language model objective that includes a weighted cross-entropy loss and elastic weight consolidation regularization. This approach is designed to augment the LMs' capability to handle negation effectively.\n\nThe experiments conducted on the negated LAMA dataset show promising results. The proposed negated augmented models achieve significant error rate reductions compared to existing negation models. Specifically, for BERT-base, BERT-large, RoBERTa-base, and RoBERTa-large, the mean top 1 error rates are reduced to 1.1%, 0.78%, 3.74%, and 0.01%, respectively. These models outperform the original BERT and RoBERTa models by 8% and 6% in terms of minimizing the mean error rate.\n\nFurthermore, the authors demonstrate that the negated augmented models also outperform classical models on both original and negation benchmarks (Tab.2 to Tab.12) for natural language inference tasks.\n\nIn summary, this work presents an effective approach to enhance the understanding of negation in pretrained language models, resulting in improved performance on negation-related tasks and natural language inference benchmarks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No", "Typos_Grammar_Style_and_Presentation_Improvements": "No"}, "extra_text": {}}, {"review_id": "ms5viQqhba", "reviewer": "Reviewer_f5qH", "summary": "", "questions": "In line 379, \"Rather than fine tuning all the layers, we first propose to update only the top layers in models. Secondly, we apply EWC,\na regularization technique to control forgetting specifically by constraining the model weights that are important for previous tasks.\" \nWhat is a rational behind this? If some comparison of this method and fine tuning all layers, I would like to know.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and suggestions.\n\n1. The NBERT model, after fine-tuning, demonstrates superior performance on downstream tasks, compared to the majority baseline. RTE consists of shorter sentences, which generally favor model's performance due to its greater ability to capture short-range dependencies as compared to long sentences. However, MNLI task features longer sentences, the model achieves nearly identical accuracy to baselines. This highlights the classical challenge of understanding larger sentences (affirmative as well as negative) in language models, which warrants further investigation.\n\n2. We believe that data generation process is efficient and can generate huge pile of dataset. In our experiments, we intentionally opted for a smaller dataset to mitigate the risk of introducing bias in the understanding of negation and affirmation, which could have led to more drastic weight changes. We refer the reviewers to Table 12 where we present the results for 10000 and 20000 samples. The greater train data size leads to an increase in error in negated LAMA queries. Nevertheless, we also manually examine the negated generated sentences (96% precision (refer section 3.2)) and the training data to marginalize any error.\n\n3. We refer to Table 12, that showcases the comparative performance of fine-tuning all layers versus employing other regularization techniques, specifically L2 regularization and without using Elastic Weight Consolidation (EWC). The results indicate that updating all layers leads to a decline in accuracy for LAMA queries. In contrast to other regularization technique, EWC effectively mitigates forgetting by constraining the model weights that are crucial in classical language models.\n\n4. We provide a comprehensive list of all parameters used in both our training and evaluation experiments in appendix. To enhance the reproducibility of our work, we will provide a more detailed explanation of negation generation, computing infrastructure, and other relevant details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposal of their pre-training framework with Elastic Weight Consolidation (EWC) regularization and weighted loss is interesting. The data generation for this pre-training framework (Figure 1) is interesting, too. \n\nTable 12 shows various comparison with different loss functions, which shows interesting experiments.", "reasons_to_reject": "Table 11 shows results where RTE is good while SNLI and MNLI are so so. \n\nData generation process in Figure 1 are quite limited since they extracted two lines (first sentence, follow-up sentence) from each page of the wiki-raw-text corpus. Thus this generates tiny corpus. All the more, if this generation yields errors, the results are instantly affected.\n\n\n", "questions_to_authors": "In line 379, \"Rather than fine tuning all the layers, we first propose to update only the top layers in models. Secondly, we apply EWC,\na regularization technique to control forgetting specifically by constraining the model weights that are important for previous tasks.\" \nWhat is a rational behind this? If some comparison of this method and fine tuning all layers, I would like to know.", "paper_topic_and_main_contributions": "Although negation plays an important role across a wide range of applications,\nthey are under-represented in pretrained language models (LMs).\nThis paper proposes a language model objective with a weighted cross-entropy loss and elastic weight consolidation regularization for the augment negation understanding. For negated augmented models, they reduce the mean top 1 error rate for BERT base to l.1%, BERT-large to 0.78%, RoBERTa base to 3.74%, RoBERTa-large to 0.01% on the negated LAMA dataset that outperform the existing negation models. It minimizes the mean error rate by a margin of 8% and 6% for original BERT and RoBERTa models. They also provide empirical evidences that negated augmented models outperforms the classical models on original as well as negation benchmarks on natural language inference tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CgZU3l3Hgv", "reviewer": "Reviewer_nWX2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and suggestions.\n\n1. We agree that our approach does require a careful balance of training techniques to achieve the desired results. We believe that this precision is what allows our method to excel in mitigating forgetting in language models. By carefully fine-tuning the model's weights and utilizing strategies such as EWC, we are able to strike a balance between preserving the knowledge gained from the pre-training task and adapting to the negation understanding at hand. While it is true that this precision can be challenging to achieve, we believe that the benefits of our approach far outweigh the difficulties. We believe that the method can also be applied to generative models such as GPT and LLaMA with some optimization in parameters.\n\n2. We will carefully address and correct all the grammatical errors suggested.\n\n3. We provide a comprehensive list of all parameters used in both our training and evaluation experiments in appendix. To enhance the reproducibility of our work, we will provide a more detailed explanation of negation generation, computing infrastructure, and other relevant details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The results seem convincing.", "reasons_to_reject": "The approach seems to be balanced on a knife edge, with very specific training techniques being necessary to make it work.", "questions_to_authors": "", "paper_topic_and_main_contributions": "They investigate ways to continue the pre-training of pre-trained BERT models so that the resulting models are better able to handle negation. Their technique continues the pre-training for two epochs on 5000 examples that feature negative statements, avoiding \"catastrophic forgetting\" by incorporating a regularization term in the loss function and by restricting parameter updates to the top 5 layers of BERT-base and the top 10 layers of BERT-large models.\n\nTheir training examples consists of the first two sentences from Wikipedia articles, where sometimes the second sentence has been negated using a custom algorithm that employs part-of-speech tags and a dependency parse. Their pre-training loss function incorporates an MLM objective and a next sentence objective (just like the original BERT loss). However, their MLM objective only ever masks the object of the second sentence (or the subject if no object exists). The model is rewarded for correctly filling in the mask for the unaltered sentence pair, and is penalized for correctly filling in the mask for the negated sentence pair.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "sec 1: \"do not appropriately models\" => \"do not appropriately model\"\nsec 1: \"LMs when finetuned on...\" => \"When finetuned on...\"\nsec 2: \"Language models such as BERT, RoBERTa\" => \"Language models such as BERT and RoBERTa\"\nsec 2: \"leverage on large scale pre-training\" => \"leverage large scale pre-training\"\nsec 2: don't preface references with \"The authors\" or \"Authors\" or \"The studies\"\nsec 2: \"grammatical correct\" => \"grammatically correct\"\nsec 2: \"on the reasoning tasks which requires\" => \"on reasoning tasks which require\"\nsec 2: \"The other studies include\" => \"Other studies include\"\nsec 2: \"Some recent work studies the negation understanding\" => \"Some recent work studies negation understanding\"\nsec 2: \"to improve the syntactic abilities of the models\" => \"to improve the syntactic abilities of models\"\nsec 3: \"public available\" => \"publicly available\"\nsec 3: \"to extract the wiki-raw-text corpus and negate the sentences\" => \"to negate the sentences of the wiki-raw-text corpus\"\nsec 3.2: \"consider an affirmative sentence, that needs\" => \"consider an affirmative sentence that needs\"\nsec 3.2: \"the dependency parsing tree and the syntactical rules\" => \"the dependency parse tree and syntactic rules\"\nsec 3.2: ' \"\\w+VBD\" is defined, where \"\\w\" represents...' => ' \"\\w+/VBD\" is defined, where \"\\w+\" represents...' \nsec 4.1: \"between pair of sentences\" => \"between pairs of sentences\"\nsec 4.1: \"we explicitly imply the model\" => ???\nsec 4.2: \"The aggressive fine-tuning\" => \"Aggressive fine-tuning\"\nsec 4.2: When EWC is first used, expand the acronym.\nsec 4.2: \"Learning a task, consists of adjust the diagonal of Fisher Information matrix F, which signifies the second derivative of the loss near minimum\" => ??? (hard to definitively know what this sentence is trying to say, but it needs to be re-expressed)\nsec 4.3: \"Mask Language Modelling\" => \"Masked Language Modelling\"\nsec 4.3: \"To leverage, the strengths\" => \"To leverage the strengths\"\nsec 4.3: \"we propose a weighted loss functions that combine\" => \"we propose a weighted loss function that combines\"\nsec 4.3: \"We introduce the weighted factor \\alpha\" => \"We introduce weighting factor \\alpha\"\nsec 4.3: \"with the above defined regularization and the loss function\" => \"with the above defined regularization and loss function\"\nsec 5.1: \"which has emerged as a valuable data\" => \"which has emerged as a valuable dataset\"\nsec 5.1: \"wikiedia\" => \"wikipedia\"\nsec 5.2: \"Note that, the training approach\" => \"Note that the training approach\"\nsec 5.2: \"to train the pre-trained models with 2 epochs\" => \"to train the pre-trained models for 2 epochs\" \nsec 5.2: \"refer to readers\" => \"refer readers to\"\nsec 5.3: \"for the affirmation cases\" => \"for the affirmative cases\"\nsec 5.3: \"has a subtle effect\" => \"has a minor effect\"\nethics: \"The research work comply with EMNLP ethics policy and do not have any ethical concerns.\" => \"The research work complies with the EMNLP ethics policy and does not raise any ethical concerns.\" "}}]

Author rebuttal:

Meta-review: The paper investigates negation understanding in LMs. It proposes fine-tuning models on data that includes negation understanding and propose a two-part loss to balance fine-tuning with avoiding catastrophic forgetting. Experiments show that this recipe improves negation understanding for MLMs without hurting performance on other tasks.

The reviewers agree that the proposed recipe is interesting and experiments support the hypotheses well. There are some concerns regarding generalization to longer sequences and whether the proposed approach would generalize beyond the experiments in this work, both of which the authors acknowledge would be interesting future work.
Decision: Accept-Findings