FANCY: A Diagnostic Data-Set for NLI Models
Authors: Guido Rocchietti, Flavia Achena, Giuseppe Marziano, Sara Salaris, Alessandro Lenci (Università di Pisa, 2021)
Topic: Distributional Semantics

Abstract
We present here FANCY (FActivity, Negation, Common-sense, hYpernimy), a new dataset with 4000 sentence pairs concerning complex linguistic phenomena such as factivity, negation, common-sense knowledge, hypernymy and hyponymy. The analysis is developed on two levels: coarse-grained for the labels of the Natural Language Inference (NLI), that is to say the task of determining whether a hypothesis is true (entailment), false (contradiction), or undetermined (neutral) and fine-grained for the linguistic features of each phenomenon. For our experiments, we analyzed the quality of the sentence embeddings generated from two transformer-based neural models, BERT <cit.> and RoBERTa <cit.>, that were fine-tuned on MNLI and were tested on our dataset, using CBOW as a baseline. The results obtained are lower than the performance of the same models on benchmarks like GLUE <cit.> and SuperGLUE <cit.> and allow us to understand which linguistic features are the most difficult to understand.

Introduction
Nowadays it has become more and more important to understand how much neural models applied to Natural Language Processing can understand about language features. The probing task methodology is a simple but effective approach to address this issue <cit.>. A network is trained on a specific task and then the representations are passed to a classifier. The performance of the classifier is evaluated with a dataset constructed to test the understanding of specific linguistic phenomena. If the classifier performs well, then it can be deduced that the neural embeddings have stored syntactic and semantic knowledge relative to those specific linguisitc phenomena.   One of the most widely used tasks for this approach is Natural Language Inference, in which the model must decide whether a hypothesis is an entailment, a contradiction, or simply neutral with respect to the premise.   Another approach consists in using benchmarks, i.e. datasets relating to various types of tasks, which are able, on the basis of the results obtained, to provide a general judgment on the performance of the model. Although benchmarks are very useful in evaluating the average performance of models, they are less effective in representing a wide range of linguistic phenomena that the models are able to deal with.   It is in this context that the challenge sets are born, (also called adversarial sets, stress sets or diagnostic sets) such as the SNLI (Stanford Natural Language Inference) <cit.> and the MultiNLI (Multi-genre Natural Language Inference) <cit.>. This datasets provide the possibility of more specific evaluation frameworks compared to traditional benchmarks <cit.>: as in the case of the probing task, the aim is to evaluate the quality of linguistic information encoded by vector representations.   For our research we built a diagnostic dataset that addresses key aspects of the human knowledge of lexical and compositional meaning, in order to test the deep semantic abilities of the latest computational models.   In this paper, we introduce FANCY, a dataset with 4,000 different hand-annotated sentence pairs with inference relation between them. In Section <ref> we will briefly present the linguistic phenomena we decided to analyze. In Section <ref> we will present the methods of dataset construction and in Section <ref> we will discuss the results of the experiments conducted on FANCY.

Conclusion
Following a large number of recent studies <cit.>, <cit.>, <cit.>, <cit.>, we also tried to investigate whether the latest neural models were able to understand certain linguistic phenomena. On the one hand, we wanted to test the models on the real understanding of the English language, on the other hand, we wanted to build a fine-grained dataset, which allows a detailed analysis of each phenomenon. We tested two of the the most high-performance models such as BERT and RoBERTa and we observed how they struggle dealing with linguistic features that are quite simple to understand for a human being. We have shown how the models can better handle phenomena such as factivity and negation if compared with the results obtained on hierarchical relation and common-sense knowledge. More in particular, we were able to stress how the state-of-the-art models struggle in dealing with linguistic phenomena that are essential for a correct understanding of the language such as the possibility generated by a statement, temporal relations between entities, the negation when there is a presence of temporal adverbs and relative pronouns and cases of downward monotone sentences. In future developments of our work we could use FANCY in order to perform fine tuning on Transformer-based models with the aim of increasing model performance and inferential capabilities. To do this it would be useful to produce more data, possibly annotated by different people, to test the models developed on different types of natural language. At the same time, the dataset could be implemented with other languages, such as Italian.