La non canonica l'hai studiata? Exploring LLMs and Sentence Canonicity in Italian
Authors: Claudiu Daniel Hromei, Danilo Croce, Rodolfo Delmonte, Roberto Basili (Università di Roma Tor Vergata, Università Ca' Foscari Venezia, 2024)
Topic: Language Models

Abstract
This paper investigates the ability of Large Language Models (LLMs) to differentiate between canonical and non-canonical sentences in Italian, employing advanced neural architectures like LLaMA and its adaptations. Canonical sentences adhere to the standard Subject-Verb-Object (SVO) structure. We hypothesize that recent generative LLMs are influenced heavily by the English language, where non-canonical structures are very rare. Using the in-context learning technique, we probe these models and further fine-tune them for this specific task. Initial results indicate that these models continue to struggle with this task even after fine-tuning. Additionally, we introduce a test set comprising several hundred sentences from the poetry domain, which presents significant challenges for the canonical structure task.

Introduction
Unlike contemporary English, which primarily follows a Subject-Verb-Object (SVO) structure, Italian exhibits a rich variety of non-canonical syntactic structures that deviate from this pattern AUTHOR. Italian is generally considered a configurational language with a neutral or canonical SVO sentence structure. However, it also displays characteristics of a weak non-configurational language due to several typological parameters: free subject inversion, pro-drop, and nonlexical expletives. Additionally, Italian lacks wh- in situ, preposition stranding, deletable complementizers, impersonal passives, and parasitic gaps with the same argument AUTHOR. In cognitive linguistic terms, the use of surface or syntactic constituency and word order in non-canonical sentences in Italian reflects its informational structure. As an example, the first sentence ``Sempre caro mi fu quest'ermo colle e questa siepe che da tanta parte de l'ultimo orizzonte il guardo esclude''''} of Leopardi's famous L'infinito is a typical example of a non-canonical sentence: the complement is fronted and the subject is in post-verbal position, also known as complete argument inversion, for letting the reader focus on the subject and main verb rather than the complement. The functional or relational interpretation of these syntactic structures, along with semantic processing, is essential to understanding the semantic roles associated with displaced grammatical functions. For instance, when a subject appears in an inverted position, it indicates a pragmatically motivated displacement, emphasizing focus over an otherwise topic-related function. Typically, subjects, understood as topics or ``what the sentence is about'' and constituting old information, precede the verb. This is consistent with Italian and English, both of which follow an SVO structure. Conversely, focus, defined as “the essential piece of new information carried by a sentence,” usually follows the verb in the “comment” portion of the sentence. We consider complexity measures sensitive to non-canonical structures (NCS), which are pragmatically motivated and used to encode structured meaning with high informational content, related to the FOCUS/TOPIC non-argument functions in Lexical-Functional Grammar (LFG) AUTHOR. Non-canonical structures can aid the reader or interlocutor in better understanding the pragmatically relevant meaning in context AUTHOR. Italian NCS are relatively frequent in text. In AUTHOR, the authors analyzed the VIT (Venice Italian Treebank) by manually annotating non-canonical structures and inflected propositions in Italian. The study found that Left Dislocated Complements, where a complement of the main verb according to subcategorization restrictions occurs, appear in 0.03% of cases. Dislocated Subjects, indicating any NP subject not followed by the main verb, occur in 0.28% of cases. The overall percentage of non-projectivity in written texts is 7%, based on 230,629 constituents. Compared to Latin, where the non-projectivity index is 6.65% in the Latin Dependency Treebank containing about 55,000 tokens, Italian and Latin are quite similar. In contrast, English tree projectivity in the Penn Treebank (PT), where the majority of data corresponds to the articles of Wall Street Journal (WSJ), shows much lower numbers: with 720,086 constituents, the non-projectivity index is 0.01004%. Thus, Italian speakers have high expectancies for the presence of an NCS due to processing difficulties also raised by the number of unexpressed subjects: 61% of all Inflected Propositions lack a lexically expressed subject. This does not apply to English speakers, for whom NCS are infrequent and context-specific. In this view, Italian is considered unique for its use of many of the non-canonical structures found in contemporary poetry and examined in this experiment. The richness and freedom of the language give the speakers the ability to produce such a diverse typology of non-canonical structures, which stems from its Latin heritage, with the Null Subject being one of the most well-known features. Like many other languages, including Spanish, Portuguese, and Catalan, as well as Chinese, Japanese, Slavic languages, Greek, and Hebrew, Italian is a Null Subject Language. However, this parameter alone does not fully explain the richness and complexity of syntactic structures seen in Italian poetry. While other Romance languages share similar syntactic traits, the specific linguistic legacy and poetic traditions of Italian give it a unique character in this regard. In this paper, we want to analyze the ability of recently proposed Large Language Models to detect non-canonical sentences in Italian. Our hypothesis is that, given the very large percentage of English training data (usually more than 90%) and the very low percentage of Italian training data (usually less than 1%), these models have a limited capacity to process such structures and they rely mostly on the English writing structures. On the other hand, the models that have been specifically adapted or fine-tuned on Italian data should show a better understanding of the canonicity in Italian. In the rest, Section sec:rel_work describes the related work, Section sec:method shows the approach in recognizing the canonical structures, Section sec:experiments presents and discusses the results, while Section sec:conclusion derives the conclusions.

Conclusion
In this study, we have shown the potential of Large Language Models, particularly the LLaMA architecture and its Italian adaptations, in distinguishing between canonical and non-canonical sentences in Italian. Our experiments indicate that instruction-tuned models specifically for Italian, such as Camoscio and LLaMAntino, exhibit a strong grasp of Italian syntax and can effectively handle diverse sentence structures. However, the performance for this task is still penalized by the large portion of English data they ingest during pre-training. The findings underscore the importance of tailored language models for specific languages and the benefits of incorporating extensive syntactic variations into training datasets. Future work should focus on expanding the training datasets with more diverse syntactic structures and improving model architectures to better capture the nuances of non-canonical sentences. Claudiu Daniel Hromei is a Ph.D. student enrolled in the National Ph.D. in Artificial Intelligence, XXXVII cycle, course on Health and life sciences, organized by the Universit\`a Campus Bio-Medico di Roma. We acknowledge financial support from the PNRR MUR project PE0000013-FAIR and support from Project ECS 0000024 Rome Technopole, - CUP B83C22002820006, NRP Mission 4 Component 2 Investment 1.5, Funded by the European Union - NextGenerationEU.