ItaEval and TweetyIta: A New Extensive Benchmark and Efficiency-First Language Model for Italian
Authors: Giuseppe Attanasio, Pieter Delobelle, Moreno La Quatra, Andrea Santilli, Beatrice Savoldi (Instituto de Telecomunicações, KU Leuven, Leuven.AI, Kore University of Enna, Sapienza Università di Roma, Fondazione Bruno Kessler, 2024)
Topic: Language Models

Abstract
Current development and benchmarking efforts for modern, large-scale Italian language models (LMs) are scattered. This paper situates such efforts by introducing two new resources: \itaeval, a comprehensive evaluation suite, and \tweetyita, an efficiency-first language model for Italian. Through \itaeval, we standardize evaluation across language understanding, commonsense and factual knowledge, and social bias-related tasks. In our attempt at language modeling, we experiment with efficient, tokenization-based adaption techniques. Our \tweetyita shows encouraging results after training on as little as 5G tokens from natural Italian corpora. We benchmark an extensive list of models against \itaeval and find several interesting insights. Surprisingly, i) models trained predominantly on English data dominate the leaderboard; ii) \tweetyita is competitive against other forms of adaptation or inherently monolingual models; iii) natural language understanding tasks are especially challenging for current models. We release code and data at and host a live leaderboard at . \def\thefootnote{*}\makeatletter\def\Hy@Warning#1{}\makeatother\footnotetext{This research has been carried out by the participants of the ItaLLM and ItaEval research sprints.}%\def\thefootnote{\arabic{footnote}

Introduction
``The strength of the team is each individual member. The strength of each member is the team.'' -- Phil Jackson The increasing availability of Italian corpora and related resources has sparked new interest in advancing the state of the art for language models. Various works have prioritized different approaches. \citet{sarti-nissim-2024-it5} build a T5 model AUTHOR from scratch and use standard fine-tuning for task specialization. More recent works experiment with efficient instruction fine-tuning AUTHOR or continual-learning AUTHOR starting from autoregressive monolingual English models. Community-driven efforts or .} and multilingual models that include Italian AUTHOR among their pretraining corpora complete the picture. Despite many modeling contributions, insights on evaluation remain partial and broadly scattered. Test-beds in \citet{sarti-nissim-2024-it5} include downstream language understanding tasks (e.g., text summarization or style transfer) but lack commonsense and factual tests, which are instead commonly central components of modern language model development. or Apple's OpenELM AUTHOR.} Some works follow this line AUTHOR while others lack a systematic quantitative evaluation AUTHOR. In this landscape, we are thus left with a puzzling scenario and several open questions: What is the current state-of-the-art model? Does a new state-of-the-art exist at all? How are ``better'' or ``worse'' even measured? Which are the most critical weak spots for Italian state-of-the-art models? Which language training or adaptation technique yields better results for Italian? Leaving these paramount questions unanswered risks running computationally and environmentally expensive adaptation experiments with limited returns due to duplicated efforts or prioritization of dead ends. [!t] \centering \includegraphics[width=0.7\linewidth]{img/ita_eval.drawio-4.pdf Tasks challenge models on Natural Language Understanding (left), Commonsense and Factual Knowledge (center), and Bias and Fairness (right) datasets. Data comes from Italian sources or English corpora, which were machine-translated (robot icon). Both pre-existing and new (star icon) tasks are included. This paper introduces two community-built resources to clarify the current development and evaluation of Italian language models. First, we release a new extensive evaluation suite to address the lack of multi-faceted assessment for Italian. \itaeval (v1.0) includes i) natural language understanding tasks (for comparability with existing benchmarks), ii) commonsense- and factual knowledge-oriented tests (to align with new evaluation requirements for language models), and iii) bias, fairness and safety tests, which are often overlooked dimensions. %\itaeval's The suite includes 18 tasks, built upon both ``native'' (i.e., datasets whose data is originally collected in Italian) and machine-translated datasets. To gain a more nuanced view of the types of adaptation to Italian, we release \tweetyita, a new efficiency-oriented 7B autoregressive, monolingual language model. Based on lightweight En\rightarrowIt token replacement, \tweetyita achieves surprising results after running language adaptation on as little as 5G Italian tokens. \paragraph{Contributions. We release \itaeval v1.0, a new evaluation suite for Italian language models and run several language models against it. We release a new efficiency-oriented 7B language model and prove that token mapping is an efficient and competitive adaptation alternative for En\rightarrowIt model conversion. Code and data are released under a permissive license to foster research.

Conclusion
In this work we introduced \itaeval (v1.0), an evaluation suite for Italian language models, and \tweetyita, an efficiency-first language model tailored for Italian. \itaeval standardizes evaluations across tasks in natural language understanding, commonsense and factual knowledge, and social bias. Empirical results show that \tweetyita performs competitively, demonstrating the effectiveness of efficient adaptation techniques. Interestingly, models trained mainly on English data lead the evaluation leaderboard, indicating the strength of cross-lingual training. We believe these contributions will help clarify the evaluation landscape for Italian language models and encourage further research. Looking ahead, we plan to expand \itaeval to enhance its scope and detail of evaluation. \itaeval and \tweetyita are the result of a joint effort of members of the \includegraphics[height=2ex]{img/logo.png} ``Risorse per la Lingua Italiana'' community (): we thank every member who dedicated their time to the project. We thank CINECA for providing the computational resources (ISCRA grant: HP10C3RW9F). The Portuguese Recovery and Resilience Plan supported the work by Giuseppe Attanasio through project C645008882-00000055 (Center for Responsible AI) and by Fundação para a Ciência e Tecnologia through contract UIDB/50008/2020. Beatrice Savoldi is supported by the PNRR project FAIR - Future AI Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU. \appendix