EuroLLM: Multilingual Language Models for Europe
Authors: name: Pedro Henrique Martins; name: Patrick Fernandes; name: João Alves; name: Nuno M. Guerreiro; name: Ricardo Rei; name: Duarte M. Alves; name: José Pombal; name: Amin Farajian; name: Manuel Faysse; name: Mateusz Klimaszewski; name: Pierre Colombo; name: Barry Haddow; name: José G. C. de Souza; name: Alexandra Birch; name: André F. T. Martins
Published: 2024-09-24 16:51:36

Abstract: The quality of open-weight LLMs has seen significant improvement, yet they
remain predominantly focused on English. In this paper, we introduce the
EuroLLM project, aimed at developing a suite of open-weight multilingual LLMs
capable of understanding and generating text in all official European Union
languages, as well as several additional relevant languages. We outline the
progress made to date, detailing our data collection and filtering process, the
development of scaling laws, the creation of our multilingual tokenizer, and
the data mix and modeling configurations. Additionally, we release our initial
models: EuroLLM-1.7B and EuroLLM-1.7B-Instruct and report their performance on
multilingual general benchmarks and machine translation.