ITA-Bench: Towards a More Comprehensive Evaluation for Italian LLMs
Authors: Luca Moroni, Simone Conia, Federico Martelli, Roberto Navigli (Sapienza Università di Roma, 2024)
Topic: Language Models

Abstract
Recent Large Language Models (LLMs) have shown impressive performance in addressing complex aspects of human language. These models have also demonstrated significant capabilities in processing and generating Italian text, achieving state-of-the-art results on current benchmarks for the Italian language. However, the number and quality of such benchmarks is still insufficient. A case in point is the ``Open Ita LLM Leaderboard'' which only supports three benchmarks, despite being one of the most popular evaluation suite for the evaluation of Italian-language LLMs. In this paper, we analyze the current limitations of existing evaluation suites and propose two ways of addressing this gap: i) a new suite of automatically-translated benchmarks, drawn from the most popular English benchmarks; and ii) the adaptation of existing manual datasets so that they can be used to complement the evaluation of Italian LLMs. We discuss the pros and cons of both approaches, releasing our data to foster further research on the evaluation of Italian-language LLMs.

Introduction
LLMs are becoming more and more prominent in NLP, showing impressive results on an increasing range of standard benchmarks, thanks in particular to their reasoning and in-context-learning capabilities AUTHOR. The current trend points towards increasingly larger models trained on massive amounts of data AUTHOR.\ However, despite these advancements, there remains a significant gap in the availability of high-quality benchmarks for languages other than English, including Italian, which is often considered too optimistically as a high-resource language. Benchmarks are essential for measuring progress in NLP, providing a standardized way to evaluate and compare models, and this is now especially important for Italian given the growing amount of language-specific models that are being developed for the language AUTHOR. High-quality benchmarks must be well-crafted to ensure they accurately reflect the complexities of the language and the specific challenges it presents. As of today, most of the existing Italian benchmarks are translations of English datasets, which may not fully capture the nuances and unique characteristics of the Italian language. Nevertheless, the ability to automatically translate English benchmarks into Italian is valuable and enticing for two main reasons. First, it provides a way to compare almost 1-to-1 the results obtained in English to the ones obtained in Italian, as the translation process is aimed at keeping an alignment from the source to the target text by design. Second, it provides a quick and relatively simple way of producing a benchmark in Italian, assuming that the translation tool is able to produce high-quality outputs. Unfortunately, the current evaluation suites that are based on automatic translations include only a limited number of benchmarks. For instance, the ``Open Ita LLM Leaderboard'', which is one of the most popular evaluation suites for Italian LLMs, relies on just three main benchmark translations, namely, MMLU, HellaSwag, and ARC-Challenge. This biases and hampers the assessment, and may not allow the advanced capabilities of modern LLMs to be fully analyzed, even though recent efforts are starting to address this limitation AUTHOR. Having gold LLM benchmarks natively written in Italian is also important, as their scarcity hinders the accurate evaluation of LLMs' capabilities in the Italian language, limiting our understanding of their true performance and potential areas for improvement. Indeed, the translation of English-centric benchmarks may contain instances that refer to concepts, entities, cultures, traditions, historic events, politics, and economics that are not akin to what one is more likely to find in Italian texts and/or in Italy AUTHOR. However, the creation of completely new datasets that take into account such elements is difficult, complex, and time-consuming, and requires expert knowledge. Falling in between automatic translations of existing datasets from English and the creation of brand-new datasets in Italian, there is the option of adapting existing Italian datasets that were originally created for a different purpose, to measure the capabilities of LLMs in Italian language understanding and generation. This direction has gained traction over the past few months, with efforts that focus on repurposing Italian tests (usually designed for humans) to evaluate LLMs instead AUTHOR. In this paper, we follow both directions and introduce ITA-Bench, a more comprehensive benchmark suite for the evaluation of Italian-language LLMs. First, ITA-Bench proposes a new extended suite of benchmarks created by automatically translating the most popular English benchmarks into Italian.\ Second, ITA-Bench includes existing manually curated datasets, adapted to enhance the evaluation framework for Italian LLMs. These two complementary approaches aim to bridge the evaluation gap and provide a more thorough understanding of the capabilities of Italian-language LLMs. With ITA-Bench, we hope to foster further development and refinement of evaluation techniques for Italian LLMs, ultimately contributing to the broader field of multilingual NLP. ITA-Bench is available at .

Conclusion
In this paper, we introduce a novel evaluation suite aimed at advancing the Italian community's ability to assess the competencies of LLMs on Italian data. Our approach follows two main directions. First, we define a novel pipeline called OBenTO, which involves translating state-of-the-art English benchmarks into Italian. Second, we rephrase existing Italian benchmarks to be used for prompting and testing large language models. Additionally, we conduct a comprehensive evaluation of the quality of automatically translated benchmarks, highlighting the inherent challenges of such an approach and analyzing the errors made by four LLMs. We hope that our work can provide a solid evaluation framework for evaluating the capabilities of current and future LLMs in Italian. Simone Conia gratefully acknowledges the PNRR MUR project URL, which fully funds his fellowship. Federico Martelli and Roberto Navigli acknowledge the support of the CREATIVE project (CRoss-modal understanding and gEnerATIon of Visual and tExtual content, Progetti di Interesse Nazionale - PRIN 2020). Finally, we acknowledge the work of the M.Sc. students of Prof. Navigli's multilingual NLP course for the Academic Year 2024, whose contributions provided valuable insights and ideas for the adaptation of existing Italian benchmarks. We acknowledge the CINECA award IsB28\_medit under the ISCRA initiative for the availability of high-performance computing resources and support. \appendix