Unraveling Text Coherence from the Human Perspective: a Novel Dataset for Italian
Authors: Federica Papa, Luca Dini, Dominique Brunato, Felice Dell'Orletta (Università di Pisa, CNR-ILC, 2023)
Topic: Lexical and Semantic Resources and Analysis

Abstract
This paper presents a novel resource designed to study text coherence in the Italian language. The dataset aims to address existing deficiencies in coherence assessment by focusing on human perception of coherence. Recently, it has been integrated into the DiSCoTex benchmark, part of EVALITA 2023, the 8th evaluation campaign for NLP and speech tools in Italian. Our resource aims to provide a comprehensive understanding of coherence, highlighting the influence of both genre and text perturbations on perceived coherence.

Introduction
Coherence plays a central role in maintaining the overall unity of a text and is influenced by both linguistic and extra-linguistic factors. From the linguistic point of view, it primarily relies on cohesion, which encompasses various linguistic devices used in natural languages to establish connections within a text, such as anaphoric and cataphoric relationships, discourse markers, and elliptical constructions. While cohesion mainly ensures local coherence between adjacent or nearby sentences, to be fully coherent, a text needs to achieve global coherence, a property that pertains to the connection of concepts and relationships that underlie the surface text, ensuring a logical flow of ideas around an overall intent. This aspect of coherence adds a subjective component, as it also depends on the reader or listener’s familiarity with the text, language proficiency, and level of interest and attention. Modeling coherence in natural language is essential for a wide range of downstream applications. One such application is automatic essay scoring in language learning settings, where coherence assessment can provide valuable writing feedback by identifying poorly organized paragraphs and abrupt topic transitions. In clinical contexts, coherence modeling is relevant for automatic language assessment, as speech irregularities indicative of a lack of coherence can serve as markers for mental disorders like schizophrenia. Furthermore, coherence has been adopted as an intrinsic evaluation metric for assessing the quality of texts generated by Natural Language Generation (NLG) systems. Additionally, coherence modeling is gaining importance in research on the interpretability of modern deep neural networks. Indeed, while existing work has mainly focused on probing sentence-level properties, understanding how these models encode discourse and pragmatic phenomena remains a crucial aspect. In light of this interest, various attempts have been made to approach coherence assessment in the Natural Language Processing (NLP) community, especially in the ‘pre-deep learning’ era. With this respect, early computational models of discourse coherence were primarily built upon two linguistic theories: centering theory and rhetorical structure theory. Studies aligned with centering theory focused on analyzing the distribution of entity transitions over sentences as a means to predict text coherence. On the other hand, works inspired by rhetorical structure theory employed discourse parsers to generate discourse relations over sentences. With the advent of neural models, researchers have also explored their application in coherence assessment. The importance of building challenging datasets for coherence evaluation cannot be overstated. With this respect, models of discourse coherence are typically tested on tasks such as reordering, which aim to discern an original text from a corrupted one artificially created by shuffling the order of its sentences, or tasks that require systems to detect whether a document contains an intruder sentence from another document or to classify whether a target sentence is contiguous or not with a given passage. However, these approaches have come under criticism because they neglect key aspects of coherence. They fail to identify the qualities that make the shuffled text incoherent, do not pinpoint the linguistic devices responsible, and overlook the subjective component underlying coherence. Additionally, most existing benchmarks are limited to the English language. Our contribution in this paper is to address some of the existing deficiencies in coherence assessment by presenting a novel resource tailored for the Italian language, designed specifically to study text coherence from the perspective of human perception. The dataset, which to our knowledge is the first for Italian, has been recently used as part of a larger benchmark released for DiSCoTex, one of the shared-tasks presented at the 8th evaluation campaign of NLP and speech tools for the Italian language (EVALITA 2023). The results of first analyses on the resource shed light on the influence of both genre and text perturbations on perceived coherence.

Conclusion
This paper has introduced a novel resource for studying and computationally modeling text coherence in the Italian language, with a focus on human perception. The investigation into genre and text perturbations revealed a significant interplay between the two dimensions. Interestingly, text passages from Wikipedia were rated on average as more coherent than those extracted from TEDx talks, even when presented in a perturbed form. Furthermore, a deeper analysis of the perturbations revealed distinct effects on coherence perception. Modifications that disrupted coherence by altering the sentence order or introducing intruder sentences had varying impacts. Notably, coherence judgments also varied for original texts, and the syntactic structure and complexity-related features emerged as influential factors in human assessment. In the future, we would like to gain deeper insights into the underlying factors that influence coherence perception by incorporating a more diverse range of text genres and perturbations. This deeper understanding of coherence will have significant implications for the development of more sophisticated language understanding and generation systems.