AI vs. Human: Effectiveness of LLMs in Simplifying Italian Administrative Documents
Authors: Marco Russodivito, Vittorio Ganfi, Giuliana Fiorentino, Rocco Oliveto (Università di Molise, 2024)
Topic: Text Simplification

Abstract
This study investigates the effectiveness of Large Language Models (LLMs) in simplifying Italian administrative texts compared to human informants. This research evaluates the performance of several well-known LLMs, including \gptthree, \gptfour, \modelllama, and \modelphi, in simplifying a corpus of Italian administrative documents (\sitaist), a representative corpus of Italian administrative texts. To accurately compare the simplification abilities of humans and LLMs, six parallel corpora of a subsection of \itaist are collected. These parallel corpora were analyzed using both complexity and similarity metrics to assess the outcomes of LLMs and human participants. Our findings indicate that while LLMs perform comparably to humans in many aspects, there are notable differences in structural and semantic changes. The results of our study underscore the potential and limitations of using AI for administrative text simplification, highlighting areas where LLMs need improvement to achieve human-level proficiency.

Introduction
Due to the increasing popularity of generative Artificial Intelligence (AI) language tools AUTHOR, significant attention has been devoted to the use of LLMs for text simplification AUTHOR. Several studies have addressed the application of LLMs to simplify texts, particularly focusing on administrative documents, including those in Italian AUTHOR. Italian administrative texts are often notably complex and obscure AUTHOR, which restricts a large segment of the population from fully accessing the content produced by the Italian public administration AUTHOR. This work aims to (a) evaluate the quality of automatic text simplification performed by several well-known LLMs, and (b) compare LLM-based simplification with human-based simplification. To address these research questions, the following procedures were undertaken: 1) From an empirical perspective, a large corpus of Italian administrative texts was collected itaist. A parallel simplified counterpart of the corpus was created using different LLMs. Additionally, a shorter version of the administrative corpus was manually simplified by two annotators. 2) From an analytical perspective, several statistical analyses were conducted to measure the semantic and complexity closeness between human and LLM-generated data. The comparison of scores for both LLM and human datasets highlights significant differences and similarities in manual and AI-driven simplification. The results concerning readability indexes Gulpease and semantic and structural similarities (edit distance) reveal that LLMs generally perform comparably to human informants. However, AI-simplified texts are slightly less similar to the original documents than those generated by human simplifiers. LLMs tend to introduce more changes in the simplified corpora than human annotators. The empirical study indicates that texts simplified by AI exhibit more structural and lexical dissimilarities from the original documents than those simplified by humans.

Conclusion
In this study, we investigated the automatic simplification of Italian administrative documents. Our results demonstrate that LLMs can effectively simplify these texts, performing comparably to humans. Among the models examined, gptfour shows superior performance in text simplification, exhibiting significant improvements in complexity metrics. Nonetheless, it is noteworthy that humans tend to maintain a higher level of editdistance and semsim, ensuring the preservation of the original meaning and structure of the text. In other words, humans---aware of the importance of precise language for these documents---mostly preserved the original meaning and structure, whereas LLMs, while simplifying, tended to rephrase extensively. This rephrasing, although effective in reducing complexity, might inadvertently alter the legal nuances, which are critical in administrative texts. Despite this limitation, LLMs can serve as valuable support tools for text simplification, significantly accelerating a process that typically requires hours of manual work. By generating initial drafts, LLMs can reduce the workload of human experts, who would then review and refine the AI-generated drafts, ensuring the preservation of the overall meaning and legal integrity of the text. The results achieved in our study indicated that modern LLMs can simplify administrative documents almost as effectively as humans. However, the achieved findings indicate that LLMs are not fully capable of preserving the semantic meaning of the text, tending to rephrase more extensively than humans. This could introduce legal issues into the simplified text. Further study could be conducted to evaluate the juridical equivalence of automatically simplified documents. A manual investigation of our parallel corpus, supervised by expert jurists, may reveal important implications in this sensitive context. Another promising direction for future research is to investigate the impact of automatic simplification on text comprehension. An additional empirical study could be designed to evaluate whether automatically simplified documents are easier to understand than their original versions. Additionally, it would be worthwhile to explore different prompting strategies to further improve simplification quality. For instance, few-shot prompting AUTHOR with some manually simplified gold samples could better align LLMs with human style.