SimilEx: the First Italian Dataset for Sentence Similarity with Natural Language Explanations
Authors: Chiara Alzetta, Felice Dell'Orletta, Chiara Fazzone, Giulia Venturi (CNR-ILC, 2024)
Topic: Language Models

Abstract
Large language models (LLMs) demonstrate great performance in natural language processing and understanding tasks. However, much work remains to enhance their interpretability. Annotated datasets with explanations could be key to addressing this issue, as they enable the development of models that provide human-like explanations for their decisions. In this paper, we introduce the SimilEx dataset, the first Italian dataset reporting human judgments of semantic similarity between pairs of sentences. For a subset of these pairs, the annotators also provided explanations in natural language for the scores assigned. The SimilEx dataset is valuable for exploring the variability in similarity perception between sentences and among human explanations of similarity judgments.

Introduction
Large language models (LLMs) display impressive linguistic skills and demonstrate outstanding performances on a variety of tasks concerning natural language processing and understanding. This is particularly true for the most recent and ground-breaking models such as GPT-3.5\4 [1], LLama-2 [2] and Gemini [3]. LLMs, however, also present risky limitations such as lack of factuality [4, 5], poor interpretability [6, 7] and hallucinations [8]. Consequently, it has become important to verify whether these models are explainable, and specifically whether they can provide human-like explanations using natural language for decisions made [9, 10]. The ability of LLMs to explain the reasoning needed to solve a given task is fundamental, particularly for tasks where there is no established or shared evaluation protocol or benchmark. Annotated datasets with explanations are key to addressing this issue, as they enable the development of models that provide human-like explanations for their decisions. Therefore, multiple datasets have been created with free-form explanations to be incorporated into the model training process and used as benchmarks at test time, mostly focusing on English [10]. Some examples are the e-SNLI dataset [11], a version of the Stanford Natural Language Inference (SNLI) dataset [12] enriched with human-annotated explanations, and the Common Sense Explanations (CoS-E) [13] and Semi-Structured Explanations for COPA (COPA-SSE) [14] datasets, which include natural language explanations for commonsense reasoning. To the best of our knowledge, the only existing dataset enriched with explanations for Italian is ‘e-RTE-3-it’ [15], an Italian version of the RTE-3 dataset for textual entailment. In this paper, we introduce the SimilEx dataset1 , as far as we are aware, the first Italian dataset of 2,112 pairs of sentences manually annotated for semantic similarity. About half of the pairs are further enriched with free-form human-written explanations that justify the similarity score. The identification of textual similarity is a natural language understanding (NLU) task that involves determining the degree of semantic equivalence between two texts [16, 17]. It is a foundational NLU problem relevant to many applications such as summarisation, question answering and conversational systems [18]. Despite its relevance, this task is highly challenging even for humans due to its subjective nature: human annotations often widely disagree on similarity scores [19] suggesting that the cues driving sentence similarity are neither well codified nor transparent and that their perceived relevance may vary among annotators. Possibly due to these challenges, and as far as we know, datasets including human explanations for the sentence similarity task are lacking. However, they are invaluable as they force annotators to reason about their choices and identify the most relevant traits influencing their annotations.

Conclusion
This paper presented SimilEx, the first Italian dataset on sentence similarity enriched with human judgments and free-form explanations. The analyses of the collected judgments confirmed that the perception of sentence similarity is inherently subjective, as evidenced by the fair agreement between the scores. Notably, annotators tend to agree less on similar sentence pairs, showing greater convergence when sentences are markedly different. The style of the paired sentences appears to influence this convergence: while most linguistic traits may not directly impact the similarity score, some of them affect the homogeneity of judgments assigned by different annotators. These features mostly concern properties of sentence structure rather than raw sentence features such as lenght, which does not play a role in homogeneity. Regarding explanations, we found a correlation between the similarity of the content of the explanations and the similarity scores assigned, indicating that annotators tend to write more similar explanations, using a similar writing style, when their scores align. The findings from this study open several prospects. Expanding SimilEx to include sentences from different textual genera could provide further insights into the factors affecting similarity judgments. Additionally, incorporating more annotators with varying linguistic backgrounds could foster a better understanding of the subjectivity in similarity perception. Lastly, our dataset could help develop automated tools to evaluate the explainability of LLMs. By leveraging SimilEx, researchers can create models that predict similarity scores and generate explanations, enhancing the interpretability of LLMs.