Camoscio: an Italian Instruction-tuned LLaMA
Authors: Andrea Santilli, Emanuele Rodolà (Sapienza Università di Roma, 2023)
Topic: Language Models

Abstract
In recent years, Large Language Models (LLMs) have significantly advanced the state of the art in various natural language processing tasks. However, access to these models is often restricted to paid API services, limiting researchers’ ability to conduct extensive investigations. Although some open-source models have been developed by the community, these are usually either English-centric or multilingual, lacking specific adaptation for the Italian language. To address this gap and enhance open resources for Italian, we introduce *Camoscio*, a language model fine-tuned to follow user prompts specifically in Italian. We fine-tuned the smallest variant of LLaMA (7B parameters) with Low-Rank Adaptation (LoRA) on a dataset of instruction prompts translated into Italian using ChatGPT. Our results show that *Camoscio*’s zero-shot performance on various Italian downstream tasks is competitive with existing models that have been specifically fine-tuned for similar tasks. All project artifacts, including the code, dataset, and model, are released to the community at the following URL: [URL].

Introduction
In recent years, Large Language Models (LLMs) have advanced significantly, achieving state-of-the-art performance across a range of natural language processing tasks. However, most of these models are controlled by for-profit organizations that typically limit access to their models through paid APIs. This restriction hampers researchers who require access to model weights and detailed training data for comprehensive studies, particularly in privacy-sensitive applications like the medical field, where data cannot be shared with external providers. To address this, several open-source models have emerged as alternatives to proprietary models. Nevertheless, these are primarily English-centric or multilingual, often providing subpar performance on monolingual tasks and limited support for Italian. For instance, BLOOM, the largest open multilingual model available, lacks any specific Italian data, while LLaMA contains only a minimal amount of Italian-language training data. Moreover, these models typically employ a standard language modeling objective, predicting the next token based on previous tokens, trained on raw text corpora. Yet, research indicates that instruction-tuning, a second phase of training on instructional data, is essential for improving performance on downstream tasks. A recent milestone in this direction was achieved by Taori et al., who introduced Stanford Alpaca, an instruction-tuned version of LLaMA tailored for English. Building on this approach, this paper introduces Camoscio, an Italian instruction-tuned version of LLaMA, trained on a dataset derived from the instruction-tuning dataset of Stanford Alpaca translated into Italian. Specifically, we fine-tuned the smallest version of LLaMA, with 7 billion parameters, using LoRA, an efficient parameter-tuning technique that enables training larger models on standard hardware. Our contributions are as follows: We introduce a dataset for instruction tuning in Italian, derived from a translation of the Stanford Alpaca dataset. We train Camoscio on this dataset and evaluate its zero-shot performance on multiple Italian downstream tasks, including NewsSum-IT, SQuAD-IT, and XFORMAL IT. We release all associated artifacts—code, dataset, and model checkpoints—to support ongoing research and development in Italian NLP.

Conclusion
In this paper, we introduced Camoscio, a 7-billion-parameter instruction-tuned model specifically developed for the Italian language, along with a corresponding Italian instruction-tuning dataset. Our results demonstrate that Camoscio’s zero-shot performance on various Italian-language downstream tasks is competitive with existing models that have been fine-tuned for these tasks. While these models have recognized limitations, Camoscio represents an initial step toward a versatile, generalist model capable of handling a broad array of tasks in Italian without the need for explicit fine-tuning. This capability is particularly valuable in domains where data is limited or unavailable, such as the medical field. To further support the development of open resources for Italian, we are making all associated artifacts—code, dataset, and model—publicly available to the community.