Controllable Text Generation To Evaluate Linguistic Abilities of Italian LLMs
Authors: Cristiano Ciaccio, Felice Dell'Orletta, Alessio Miaschi, Giulia Venturi (CNR-ILC, 2024)
Topic: Language Models

Abstract
State-of-the-art Large Language Models (LLMs) demonstrate exceptional proficiency across diverse tasks, yet systematic evaluations of their linguistic abilities remain limited. This paper addresses this gap by proposing a new evaluation framework leveraging the potentialities of Controllable Text Generation. Our approach evaluates the models' capacity to generate sentences that adhere to specific linguistic constraints and their ability to recognize the linguistic properties of their own generated sentences, also in terms of consistency with the specified constraints. We tested our approach on six Italian LLMs using various linguistic constraints.

Introduction
Large-scale Language Models (LLMs) AUTHOR have exhibited extraordinary proficiency in a wide range of tasks, from text generation to complex problem-solving, by producing coherent and fluent texts AUTHOR. Their ability to understand context, generate human-like responses, and even engage in creative tasks underscores their potential in various applications. Such capabilities have been extensively evaluated against several benchmarks, as evidenced by the success of platforms such as the OpenLLM Leaderboard AUTHOR or Italian LLM-Leaderboard AUTHOR, specifically developed to evaluate Italian models. %in a task-oriented scenario covering a wide range of NLP tasks. However, despite their impressive capabilities, the evaluation of LLMs' linguistic abilities when generating sentences remains an understudied topic. In fact, while earlier works have demonstrated the implicit encoding of many linguistic phenomena within the representations of smaller models AUTHOR or by prompting LLMs to assess their linguistic competence AUTHOR, there is no guarantee that generative LLMs can comply with such properties in generating texts. Studies on Controllable Text Generation (CTG) indirectly assessed models' capabilities by examining their adherence to linguistic constraints AUTHOR. For instance, AUTHOR studied the abilities of LLMs in adhering to lexical and morpho-syntactic constraints %for the task of personalized text generation. when generating personalized texts. Nevertheless, these works are mainly focused on task-oriented scenarios (e.g. text simplification) and therefore they do not provide systematic evaluations of the linguistic abilities of these models. [t!] \centering \includesvg[width=0.48\textwidth]{frame-example.svg From a complementary perspective, in recent years, several works have proposed diverse approaches to assess the consistency of LLMs as an essential component of the models' evaluation AUTHOR, where consistency can be defined as ``the requirement that no two statements given by the system are contradictor'' AUTHOR or "the invariance of its behaviour under meaning-preserving alternations in its input" AUTHOR. Despite their differences, all these approaches aim to understand the reasoning processes that the models employ in various reasoning tasks AUTHOR while also measuring the predictability and coherence of the models' generated responses under different conditioning inputs. Among these, AUTHOR studied the consistency between generation (e.g.\ ``what is 7+8?'') and validation (e.g. "7+8=15, True or False?'') of LLMs considering 6 different tasks (e.g. arithmetic reasoning, style transfer). AUTHOR, instead, employed several consistency checks to measure models' faithfulness and to understand whether self-explanations truly reflect the model’s behaviour. Importantly, the training procedure of an LM does not explicitly target consistency AUTHOR, meaning this ability to produce non-contradictory statements eventually emerges as a byproduct of pre-training and fine-tuning. Therefore, studying models under such conditions serves as a valuable proxy for evaluating their capacity to handle different but complementary tasks, such as generation vs. validation. In this paper, we bring together the two perspectives and propose an evaluation approach to thoroughly test the linguistic abilities of several Italian LLMs. Specifically, by instructing a model to generate sentences that adhere to a set of targeted linguistic constraints (e.g.\ ``Generate a sentence with 2 adjectives'') and then asking to validate its own sentences ("How many adjectives does this sentence have: 
?"), %it's possible to gather evidence about different phenomena: we seek to answer the following research questions: i) To what extent is an Italian LLM capable of generating sentences that adhere to specific linguistic constraints? ii) How consistent are LLM's responses to the validation questions w.r.t. the specified linguistic constraints? iii) How well can Italian LLMs recognize the linguistic features present in their own generated sentences? \noindent{Contributions}. Our main contributions are: \item We propose a framework for evaluating the linguistic abilities of state-of-the-art Italian LLMs when generating text. \item We conduct extensive evaluations across different models and linguistic constraints. \item We assess models' consistency with the requested constraints and their ability to validate their own generated content.

Conclusion
In this paper, we presented the results of a new framework to extensively evaluate the linguistic abilities of Italian LLMs when generating sentences according to multiple linguistic constraints and, subsequently, when validating the linguistic properties of their own outputs. Results showed that models' architectures and dimensions of pre-training data have an impact on their ability to correctly follow the constraints, with ANITA being the best-performing model across all configurations. When validating each model against their own generated sentences, we noticed that i) LLMs tend to be more consistent with the requested constraints when they correctly followed them during the generation phase, and ii) the generation abilities do not always align with the ability of the models to recognize the linguistic properties of their generated sentences. Our findings also highlighted that the evaluation metric chosen can significantly affect the results, underscoring the complexity of evaluating LLMs and the necessity for further research in this direction. Considering that the evaluation of LLMs is an ongoing and multifaceted effort across all languages, we believe that this study opens the way for numerous further in-depth analyses focused on various aspects of evaluation. Among other aspects, we could evaluate the overall quality of the generated sentences, which we have not accounted for so far. Preliminary investigations revealed that the overall quality of the generations varies across Italian LLMs, with Italia appearing to be the most fluent.}. Thus, future research should also involve a more comprehensive evaluation that compares the linguistic abilities of LLMs with their fluency and grammaticality. %A further direction includes the analysis of the impact of multiple annotation schemas as a means to evaluate the linguistic abilities of the models. This work has been supported by: \noindent {0.1\linewidth \includegraphics[scale=0.05]{fair.jpg \hspace{0.01\linewidth {0.70\linewidth FAIR - Future AI Research (PE00000013) project under the NRRP MUR program funded by the NextGenerationEU. \hspace{0.01\linewidth {0.1\linewidth \includegraphics[scale=0.08]{eu.pdf \\ \noindent \hspace{0.01\linewidth {0.70\linewidth TEAMING-UP - Teaming up with Social Artificial Agents project under the PRIN grant no.\ 20177FX2A7 funded by the Italian Ministry of University and Research. \hspace{0.01\linewidth \clearpage \appendix