Judging Quality Across Languages: A Multilingual Approach to Pretraining
  Data Filtering with Language Models
Authors: name: Mehdi Ali, name: Manuel Brack, name: Max Lübbering, name: Elias Wendt, name: Abbas Goher Khan, name: Richard Rutmann, name: Alex Jude, name: Maurice Kraus, name: Alexander Arno Weber, name: Felix Stollenwerk, name: David Kaczér, name: Florian Mai, name: Lucie Flek, name: Rafet Sifa, name: Nicolas Flores-Herr, name: Joachim Köhler, name: Patrick Schramowski, name: Michael Fromm, name: Kristian Kersting

High-quality multilingual training data is essential for effectively
pretraining large language models (LLMs). Yet, the availability of suitable
open-source multilingual datasets remains limited. Existing state-of-the-art
datasets mostly rely on heuristic filtering methods, restricting both their
cross-lingual transferability and scalability. Here, we introduce JQL, a
systematic approach that efficiently curates diverse and high-quality
multilingual data at scale while significantly reducing computational demands.
JQL distills LLMs' annotation capabilities into lightweight annotators based on
pretrained multilingual embeddings. These models exhibit robust multilingual
and cross-lingual performance, even for languages and scripts unseen during
training. Evaluated empirically across 35 languages, the resulting annotation
pipeline substantially outperforms current heuristic filtering methods like
Fineweb2. JQL notably enhances downstream model training quality and increases
data retention rates. Our research provides practical insights and valuable
resources for multilingual data curation, raising the standards of multilingual
dataset development.