The Self-Contained Italian Negation Test (SCIN)
Authors: Viola Gullace, David Kletz, Thierry Poibeau, Alessandro Lenci, Pascal Amsili (CNRS, Ecole Normale Supérieure, Université Sorbonne-Nouvelle, Università di Pisa, Scuola Normale Superiore, Université Paris Cité, 2024)
Topic: Language Models

Abstract
Recent research has focused extensively on state-of-the-art pretrained language models, particularly those based on Transformer architectures, and how well they account for negation and other linguistic phenomena in various tasks. This study aims to evaluate the understanding of negation in Italian \url{bert}- and \url{robert}-based models, contrasting the predominant English-focused prior research. We develop the SCIN Set, an Italian dataset designed to model the influence of polarity constraints on models in a masked predictions task. Applying the SCIN Set reveals that these models do not adjust their behaviour based on sentences polarity, even when the resulting sentence is contradictory. We conclude that the tested models lack a clear understanding of how negation alters sentence meaning.

Introduction
Compositionality is a fundamental feature of human language, based on the principle that the meaning of a complex expression derives from its parts and their respective arrangements. One notable compositional phenomenon is negation, formally defined as a semantic operator (or function) that reverses the truth-value of a sentence AUTHOR. Given its importance, understanding how well pretrained language models (PLMs) can grasp and apply this principle is crucial. These models achieve impressive performance across a wide array of language modeling tasks. Nonetheless, they often reveal to rely on shallow heuristics or exhibit other issues in handling specific aspects of language. A prominent bias in the body of research is that the vast majority of research on language models has predominantly concentrated on English. this focus raises concerns about the generalizability of findings to other languages which may be structurally different from English. Conducting similar experiments in other languages could provide valuable context and material for comparison, potentially highlighting language-specific effects or revealing new generalization. Therefore, we decide to undertake a new experiment focusing on Italian negation. Thus, in this article, we aim to explore whether the behavior of PLMs accurately models the polarity of sentences. We will investigate how the addition of negation to a sentence can alter its overall meaning (demonstrating the models’ capability to handle shifts in meaning due to structural changes). Given the limitations explained above, our work has deliberately chosen to concentrate on Italian. This choice not only addresses the need to explore how these models perform with languages other than English but also serves as a critical test for PLMs dedicated to Italian. We suspect that these models may not be as advanced or effective as their English counterparts, highlighting the need for further developments outside English. We adapt the test set developed for English by AUTHOR to Italian, creating the Self-Contained Italian Neg Set (SCIN Set). Using the dataset to evaluate BERT- and ROBERTA-based models for Italian, we find that these models are unable to adjust their prediction in response to constraints posed by negation, often generating contradictory text.

Conclusion
In this paper, we investigated the ability of several Italian PLMs to take negation into account in their predictions. To do this, we adapted to Italian the proposed by AUTHOR, which is based on minimal pairs of aligned sentences. Applying this test to six models enabled us to show that negation modifies their predictions, but that this does not happen consistently or in a way that is always coherent with the semantic effect that we expect negation to have on sentences. These results suggest a strong need to adapt these models to make them more sensitive to negation and its semantic consequences. Nevertheless, we also noted a fairly marked difference in performance from one model to another, correlated with the different corpora used to train them. We thus suggest that a lexical and statistical study of these corpora could shed further light on the behavior of the models. Lastly, it would be interesting to compare these results with the performance of generative models, in order to study the relative importance of the number of model parameters in relation to their architecture.