Assessing the Asymmetric Behaviour of Italian Large Language Models across Different Syntactic Structures
Authors: Elena Sofia Ruzzetti, Federico Ranaldi, Dario Onorati, Davide Venditti, Leonardo Ranaldi, Tommaso Caselli, Fabio Massimo Zanzotto (Università di Roma Tor Vergata, Sapienza Università di Roma, University of Edinburgh, University of Groningen, 2024)
Topic: Language Models

Abstract
While LLMs get more proficient at solving tasks and generating sentences, we aim to investigate the role that different syntactic structures have on models' performances on a battery of Natural Language Understanding tasks. We analyze the performance of five LLMs on semantically equivalent sentences that are characterized by different syntactic structures. To correctly solve the tasks, a model is implicitly required to correctly parse the sentence. We found out that LLMs struggle when there are more complex syntactic structures, with an average drop of 16.13(\pm 11.14) points in accuracy on Q\&A task. Additionally, we propose a method based on token attribution to spot which area of the LLMs encode syntactic knowledge, by identifying model heads and layers responsible for the generation of a correct answer.

Introduction
As Large Language Models (LLMs) become more proficient at generating plausible and sounding sentences, it is compelling to determine the extent to which their competence in understanding more or less complex sentences is similar to that of a human being. Large Language Models (LLMs) excel at understanding and generating text that appears human-written. Thus, it is intriguing to determine whether the models' text comprehension aligns in some way with human cognitive processes. A peculiarity of natural languages is that the same meaning can be encoded by multiple syntactic constructions. In Italian, for instance, the unmarked sentence follows a subject-verb-object (SVO) word order. However, inversions of this ordering do not necessarily lead to ungrammatical sentences. % In particular, the Subject-Verb-Object order is frequent, but inversion does not lead to ungrammatical sentences. A case in point is represented by cleft sentence, i.e., sentences where the unmarked SVO sequence is violated. This corresponds to specific communicative functions, namely emphasize a component, and it is obtained by putting one element in a separate clause. %, with the use of an empty introductory word such as "that" or "it" in English or "che" in Italian. In particular, Object Relative Clauses -- where the element that is emphasized is the object of the sentence -- are difficult to understand AUTHOR. For example the sentence ``Sono i professori che i presidi hanno elogiato alla riunione d'istituto'' is more challenging for an Italian speaker than its semantically equivalent unmarked version ``I presidi hanno elogiato i professori alla riunione d'istituto'' where the SVO order is restored. Similarly, in Nominal Copular constructions, the inversion of subject and verb clause is documented to cause difficulties in understanding the meaning of the sentence AUTHOR. Hence, syntax plays a crucial role not only in the general construction of language but also in the native speakers ability to comprehend sentences: in fact, a correct syntactic parsing of the sentences is necessary to understand their meaning, and some syntactic structures are preferred over others. To what extent this preference is replicated by LLMs needs to be further explored. Syntax plays a fundamental role not only in the construction but also in the comprehension of language, as evidenced by the preference for certain structures over others. Currently, it is unclear whether this syntactic preference is also observable in LLMs. To understand this, it is necessary to conduct an analysis of the models that considers how syntactic structures are encoded within them. Related work has primarily been based on probing method (i 3 lavori citati), whose results are rather superficial and do not allow us to understand specifically which parts of the input sentence the model focuses on during processing. If the model shows some knowledge about syntax, there should be an area of the model responsible for that. We aim to detect the area of a model responsible for its syntactic knowledge. Extensive work has been devoted to understanding how Transformer-based architectures encode information and one main objective is to localize which area of the model is responsible for a certain behavior AUTHOR. Despite its usage as an explanation mechanism being debated AUTHOR, the attention mechanism is an interesting starting point given its wide use in Transformer architecture. While the attention weights alone cannot be used as an explanation of a model's behavior AUTHOR, an analysis that includes multiple components of the attention module is shown to be beneficial to obtain an interpretation of how a model processes an input sentence AUTHOR. Probing is a common method used to detect the presence of linguistic properties of language in models AUTHOR. Probing consists of training an auxiliary classifier on top of a model’s internal representation, which could be the output of a specific layer, to determine which linguistic property the model has learned and encoded. In particular, it has been proposed to probe Transformer-based models to reconstruct syntactic representations like dependency parse trees from their hidden states AUTHOR. Probing tasks concluded that syntactic features are encoded in the middle layers AUTHOR. Correlation analysis on the weights matrices of the monolingual BERT models confirmed the localization of syntactic information in the middle layers showing that the models trained on syntactically similar languages were similar on middle layers AUTHOR. While an altered word order seems to play a crucial role in Transformer-based models' ability to process language AUTHOR, the correlation between LLMs downstream performance and the encoding of syntax needs to be further explored. In this paper, we initially examine how syntax influences the LLMs' capability of understanding language. %and generating when exposed to different syntactic structures. To achieve this, we will analyze five open weights LLMs -- trained on the Italian Language either from scratch or during a finetuning phase -- and measure their performance in question-answering (Q\&A) tasks that require an implicit parsing of the roles of words in the sentence to provide the correct answer. We use an available set of Q\&A tasks designed for Italian speakers AUTHOR and propose similar template-based questions for two other datasets of Italian sentences characterized by different syntactic structures (Section sec:task). The results show that the models are affected by the different syntactic structures in solving the proposed tasks (Section sec:qa_res): LLMs struggle when more complex syntactic structures are present, with an average drop in accuracy of 16.13(\pm 11.14) points. We then propose a method -- based on norm-based attribution AUTHOR-- to localize where syntactic knowledge is encoded by identifying the models' attention heads and layers that are responsible for the generation of a correct answer (Section sec:localizingattr). Although some differences can be observed across the five LLMs, we notice that syntactic information is more widely included in the middle and top layers of the models.

Conclusion
In this paper, we have investigated how semantically equivalent sentences are processed by LLMs in Italian when their syntax differs. We tested LLMs trained on the Italian - or with Italian data in the pre-trainig material - and measured how their capabilities in a battery of Q\&A tasks %in a Q question-answering task that %implicitly rely on parsing the correct role of words in a sentence to be solved. Our findings confirm that cleft sentences and construction with an inversion of subject and verb are difficult to understand also for LLMs - similarly to what observed for humans. Furthermore, we have identified systematically using token-to-token attribution that syntactic information tends to be encoded in the middle and top layers of LLMs. \appendix