A study on the soundness of closed-ended evaluation of Large Language Models adapted to the Italian language
Authors: Elio Musacchio, Lucia Siciliani, Pierpaolo Basile, Edoardo Michielon, Marco Pasqualini, Asia Beatrice Uboldi, Giovanni Semeraro (Università di Bari Aldo Moro, Università di Pisa, Fastweb SpA, 2024)
Topic: Language Models

Abstract
With the rising interest in Large Language Models, deep architectures capable of solving a wide range of Natural Language Generation tasks, an increasing number of open weights architectures have been developed and released online. %Due to the amount of data required for training from scratch such models, the majority of them have been trained in English. In contrast with older architectures, which were aimed at solving specific linguistic assignments, Large Language Models have shown outstanding capabilities in solving several tasks at once, raising the question of whether they can truly comprehend natural language. Nevertheless, evaluating this kind of capability is far from easy. One of the proposed solutions so far is using benchmarks that combine various types of tasks. This approach is based on the premise that achieving good performance in each of these individual tasks can imply having developed a model capable of understanding language. However, while this assumption is not incorrect, it is evident that it is not sufficient, and the evaluation of Large Language Models still remains an open challenge. %However, many of these pre-trained checkpoints have been trained mainly in English, meaning they lack the same linguistic skills for other languages. Because of this, many works have adapted these pre-trained models to other languages with fewer resources and evaluated them using state-of-the-art benchmarks. %Still, these benchmarks focus on closed-ended reasoning-based tasks that test the inherent knowledge inside a model. %We argue that it is not a correct practice to evaluate adapted models and study other approaches on the same benchmarks and datasets to assess whether our hypothesis is correct. In this paper, we conduct a study aimed at highlighting the potential and limitations of current datasets and how a new evaluation setting applied to language-adapted Large Language Models may provide more insight than traditional approaches.

Introduction
Large Language Models (LLMs) are models based on the Transformer architecture capable of solving a wide variety of Natural Language Generation (NLG) tasks, even those not encountered during training, due to their extensive training and large number of parameters. Thanks to their remarkable skills, interest in LLMs is now at its climax, resulting in a proliferation of open-weight models (e.g. \textsc{LLaMA}, \textsc{Mistral}, and many others). Among the several challenges related to the development of LLMs, one of the most critical is their evaluation AUTHOR. One approach to tackle this issue has been to build benchmarks that collect different datasets, with the aim of obtaining a more comprehensive evaluation of the model's overall capabilities. Currently, there is a leaderboard} AUTHOR which keeps track of the capabilities of openly available LLMs. Specifically, the models are tested on six tasks that span different abilities a language model should have, e.g. reasoning or text completion. Regarding their reasoning abilities, the models are tested by solving closed-ended tasks. Specifically, multiple-choice question answering tasks are provided, where a question is given with a list of possible alternatives associated with an identifier (a letter, a number, and so on). Intuitively, since the model has also been pre-trained on closed-ended question-answering data, it should be able to generalize and understand the correct choice out of the available ones. Furthermore, rather than generating the output directly, the probabilities learned by the model are studied, using log-likelihood to assess which option is more likely to be correct. For the English language, this evaluation methodology has been a standard approach to assess the capabilities of LLMs. However, when adapting a model to a new language, due to the low amount of non-English data that has been used to pre-train such models, this methodology may not be as sound. The model only has to generate the correct option identifier, therefore this is not really testing the ability of the model of generating high-quality text in another language. The goal of this work is to understand whether a new evaluation setting applied to language-adapted LLMs may give more insight than the traditional approach. Therefore, our contributions are the following: \item We test two evaluation settings for language-adapted LLMs changing the structure of closed-ended question answering tasks; \item We evaluate the performance of state-of-the-art models on these settings; \item We study the sensitivity that the models have for the input prompt.

Conclusion
We have carried out a study on the effectiveness of evaluation of Italian-adapted LLMs on closed-ended tasks, multiple-choice question answering tasks specifically. We have experimented with two settings: an open-ended one and a closed-ended one without option identifiers. The results show better performance for the latter. Furthermore, they also show that, with respect to the Open Italian LLM Leaderboard, there are significant differences regarding model performance. We can conclude that the evaluation of Italian-adapted models should follow a more rigorous procedure which does not mainly rely on closed-ended tasks. We release the code that was used on GitHub}. In the future, we plan to further work on the topic and attempt to define best practices for the evaluation of these models. We acknowledge the support of the PNRR project FAIR - Future AI Research (PE00000013), Spoke 6 - Symbiotic AI (CUP H97G22000210007) under the NRRP MUR program funded by the NextGenerationEU.