On the impact of Language Adaptation for Large Language Models: A case study for the Italian language using only open resources
Authors: Pierpaolo Basile, Pierluigi Cassotti, Marco Polignano, Lucia Siciliani, Giovanni Semeraro (Università di Bari Aldo Moro, 2023)
Topic: Language Models

Abstract
The BLOOM Large Language Model is a cutting-edge open linguistic model developed to provide computers with natural language understanding skills. Despite its remarkable capabilities in understanding natural language by capturing intricate contextual relationships, the BLOOM model exhibits a notable limitation concerning the number of included languages. In fact, Italian is not included among the languages supported by the model, making its usage challenging in this context. Within this study, using an open science philosophy, we explore different Language Adaptation strategies for the BLOOM model and assess its zero-shot prompting performance across two different downstream classification tasks over EVALITA datasets. It has been observed that language adaptation followed by instruction-based fine-tuning is effective in correctly addressing a task never seen by the model in a new language learned on a few examples of data.

Introduction
As language diversity becomes increasingly important in the digital age, the capability of a Natural Language Understanding model to handle a wide array of languages gains significance. Large Language Models (LLMs) have emerged as excellent approaches for comprehending, generating, and manipulating human language with unprecedented accuracy and fluency. They can grasp nuances, idioms, and even ambiguous phrases, enabling more accurate sentiment analysis, question answering, and information retrieval tasks. This enhanced understanding contributes to more effective communication between humans and machines, fostering seamless interactions across various applications. LLMs possess remarkable generalization capabilities, allowing them to perform well on tasks they were not explicitly trained for, also in a multilingual fashion. Among the largest and most effective LLMs is BLOOM, a 176B-parameter open-access language model designed and built thanks to the collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model trained on a large corpus comprising hundreds of sources in 46 natural and 13 programming languages, culminating in a dataset spanning 59 languages. However, it excludes some of the world’s most widely spoken languages, including Russian, Korean, and Italian, raising the need for a more inclusive linguistic approach. Training an effective LLM focused solely on a particular language is a prohibitive challenge, given the substantial data and resources required. At the same time, tackling downstream tasks in a specific language necessitates a model with a comprehensive understanding of that language. Our hypothesis centers on the Language Adaptation methodology, which addresses the challenge of transferring knowledge from a pre-trained Language Model (LM) to a specific application language. In this context, we aim to adapt BLOOM models to work with a new language, such as Italian, using only a limited sample size of 100,000 examples. We evaluated the adapted models after instruction-based fine-tuning on two classification tasks using Italian data. Our experiments demonstrate that the Language Adaptation process improves the zero-shot ability of the model when executed in the same language as the evaluation data. A key aim of our work is to develop all methodologies using an open-science approach without relying on private data created or processed by non-open-source tools. All data and models used in this work are under an open-source license, reflecting our commitment to transparency and collaboration with the scientific community. This study proves that fostering innovation and building effective LLMs is achievable using only open resources.

Conclusion
In this paper, we explored a language adaptation strategy for the BLOOM model to address the challenge of handling languages not covered during its training. Our approach is distinguished by its reliance on open-source data, software, and models, aligning with a commitment to transparency and accessibility in research. Despite the remarkable capabilities of the BLOOM model in understanding natural language for widely spoken languages, it showed limitations when applied to languages not included in the original training set, such as Italian. To overcome this limitation, we conducted experiments using the MAD-X language adaptation approach, followed by instruction-based fine-tuning on Italian data. The outcomes of our research demonstrate the effectiveness of language adaptation in significantly improving the zero-shot ability of the BLOOM model for Italian. The combination of MAD-X language adaptation with EVALITA fine-tuning achieved the highest performance on both the AMI2020 and HASPEEDE 2 tasks, showcasing the importance of the adaptation process for downstream classification tasks in Italian. In future work, we plan to evaluate our approach on larger BLOOM models and more recent tasks for Italian. The proposed methodology can be adapted for other languages and requires few examples to achieve satisfying results. The adapted models can be easily fine-tuned on various tasks by providing proper instructions. Our future research will extend to testing this approach with other languages and exploring diverse adaptation strategies, contributing to the broader landscape of language model adaptability.