How good is NLLB for low-resource languages? A study on the Genoese Language
Authors: Davide Buscaldi, Paolo Rosso (CNRS, Université Sorbonne Paris Nord, PRHLT, Universidad Politécnica de Valencia, 2023)
Topic: Machine Translation

Abstract
In this paper, we analyze the performance of the NLLB-200 models from Meta AI on a manually built parallel corpus of Ligurian (specifically, the Genoese variant), consisting of 283 sentences and their respective Italian translations. Our experiments highlight some issues with NLLB-200, especially regarding local knowledge, stemming from certain choices made during the training process.

Introduction
NLLB (No Languages Left Behind) [1] is a collection of language models created by Meta AI to fill the void in Machine Translation (MT) for some low- and very low-resource languages. NLLB-200 is the latest model and is capable of providing MT for 200 languages, including some that had never been considered before. One of these languages is Ligurian, an endangered language spoken mainly in the Liguria region in Italy, Monaco (where it is called Monegasque), and some small islands in the Mediterranean Sea (Carloforte and Calasetta in Sardinia). The content in Ligurian on the web is very scarce. The main source is Wikipedia, which has only 11,172 articles in Ligurian, many of them being little more than drafts. In comparison, the number of articles in Welsh, which has an estimated equivalent number of native speakers (500,000), is twenty times greater than Ligurian. This difference is easily explained by the fact that Welsh is an official language, supported by the local government, while Ligurian is mostly spoken orally. The scarcity of content in Ligurian is not the only issue that may affect MT tools and methods. In particular, the syntax of Ligurian has not been completely standardized: many variants of the same word exist, even when pronounced in the same way, due to various reasons. Primarily, there are local variants of Ligurian, but also because the language has been passed down mostly orally from one generation to another. For example, in Monegasque, the word "white" is written as giancu, while in Genoese (the predominant variant) it is written as gianco. This problem has been well documented in the work of [2], which also cites the lack of regulatory bodies as a source of variations. In their study, they propose a corpus of normalized and unnormalized texts in Ligurian to train a neural model for the normalization of Ligurian texts. The example in Figure 1 illustrates the high variability of Ligurian spelling. Given this premise, it is important to evaluate whether the NLLB-200 model is capable of dealing with these issues. To this end, we conducted an evaluation by composing a test dataset in Genoese that was not used for NLLB training. The copyright-free subset of this dataset is available at the following address: https://github.com/dbuscaldi/zeneize. In the following section, we describe the NLLB-200 model and how the training was carried out. In Section 3, we present the experiments conducted on our dataset and discuss the results. Finally, in Section 4, we draw some conclusions from this analysis and propose ideas for improving the model based on this experimentation.

Conclusion
From our preliminary analysis, carried out on a dataset specific to the Genoese culture, we can affirm that currently NLLB-200 is not good enough to deal with Genoese texts or to translate text into Genoese. In particular, we found that local toponyms are difficult to translate: how good is an MT tool that is not able to correctly translate the name of the largest city where the language is spoken or the name of the language itself? Given the information provided regarding NLLB-200 models, we can identify two main elements explaining this behavior. The first one is the training data: they do not cover local information, but general English Wikipedia articles, so they lack the context in which Genoese is usually spoken. The second one is the tokenization process and the LASER-3 embeddings: given the high spelling variability of the Ligurian language, we suspect that the tokenization process may not be precise and that it may map some tokens into a position in the embedding space that does not correspond to their actual "meaning," possibly due to interference from other Western Romance languages that are very close to Ligurian. However, NLLB-200 is a big step forward in making endangered languages such as Ligurian and its variants available to everyone. From our point of view, we think that NLLB-200 could be improved in various ways, for instance, by fine-tuning the model on more "local" datasets, and possibly including knowledge regarding Out-Of-Vocabulary words, which are often named entities, for example with the methods proposed by [9], or integrating dictionaries to handle named entities.