Nesciun Lengaz Lascià Endò: Machine Translation for Fassa Ladin
Authors: Giovanni Valer, Nicolò Penzo, Jacopo Staiano (Università di Trento, Fondazione Bruno Kessler, 2024)
Topic: Machine Translation

Abstract
Despite the remarkable success recently obtained by Large Language Models, a significant gap in performance still exists when dealing with low-resource languages which are often poorly supported by off-the-shelf models. In this work we focus on Fassa Ladin, a Rhaeto-Romance linguistic variety spoken by less than ten thousand people in the Dolomitic regions, and set to build the first bidirectional Machine Translation system supporting Italian, English, and Fassa Ladin. To this end, we collected a small though representative corpus compounding 1135 parallel sentences in these three languages, and spanning five domains. We evaluated several models including the open (Meta AI's No Language Left Behind, NLLB-200) and commercial (OpenAI's gpt-4o) state-of-the-art, and indeed found that both obtain unsatisfactory performance. We therefore proceeded to fine-tune the NLLB-200 model on the data collected, using different approaches. We report a comparative analysis of the results obtained, showing that 1) jointly training for multilingual translation (Ladin-Italian and Ladin-English) significantly improves the performance, and 2) knowledge-transfer is highly effective (e.g., leveraging similarities between Ladin and Friulian), highlighting the importance of targeted data collection and model adaptation in the context of low-resource/endangered languages for which little textual data is available.

Introduction
The growing scale of Large Language Models, based on the Transformer architecture, has led to models with surprising capabilities in a number of tasks, including Machine Translation (MT). However, most of the NLP community effort is focused on high-resource standardized languages, leaving behind the vast majority of local under-resourced languages. Recent works have demonstrated the utility of creating language-specific datasets for MT AUTHOR and the effectiveness of relatively small quantities of high-quality translation data to teach a new language to pre-trained LLMs AUTHOR. To date, little work has addressed the Ladin language: even the most recent models that have included a great number of languages have not been trained with Ladin data AUTHOR, due to the scarcity of freely available parallel corpora (to our knowledge, only the OPUS corpora AUTHOR), which are also poorly curated -- e.g., wrong translations or mixed up Ladin varieties.\footnote{See Appendix. Further, previous works have mainly focused on the two South Tyrolean varieties, Gherdëina and Badiot AUTHOR: despite having a standardized written form and being officially recognized as a minority language, the Fassa variety (Fascian) has been mostly overlooked AUTHOR, while its speakers rightfully expect access to the same digital tools available for other languages AUTHOR. We introduce the first dataset of parallel Fassa Ladin-Italian-English sentences, spanning over multiple domains: literature, news, laws, brochures, and game rules. We evaluate several \emph{out-of-the-box translation systems, including the open (Meta AI's No Language Left Behind, NLLB-200) and commercial (OpenAI's gpt-4o) state-of-the-art models, and experiment with both \emph{zero-shot pivot-based and multilingual strategies to obtain satisfactory performances in bidirectional translation between Fassa Ladin and Italian/English. Figure \ref{fig:overview provides a schematic overview of our experiments, which are thoroughly described in Section. Our results show how the collection of small quantities of parallel data is very effective in `adding' support for a previously unsupported language to existing state-of-the-art models. More specifically, we find that the NLLB-200 model fine-tuned using a multilingual strategy can outperform even the most capable commercial LLMs (e.g., OpenAI gpt-4o). For reproducibility purposes, we make the dataset and code publicly available.

Conclusion
In this work, we show that it is possible to add a specific language variety to a pre-trained MT model using little amount of data for fine-tuning (fewer than 900 parallel sentences). To add Fassa Ladin, we fine-tune the model using as starting point a similar language included in NLLB-200: Friulian. This approach significantly improves the performance. Moreover, in such condition, fine-tuning with parallel sentences in more than two languages proves to help regularization and to improve translations, with respect to a \emph{zero-shot} pivot-based transfer learning approach. Future work includes extending the dataset with new resources and domains, improving the alignment quality, and including human evaluation of translation quality. Adding data from other Ladin varieties might be a viable solution to improve the low performance caused by unknown words. Moreover, experimenting with translated words from vocabulary entries could be beneficial for Fassa Ladin, a language variety that has scarce parallel data but various publicly accessible vocabularies.