Title: MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining
Venue: NeurIPS 2025
Accepted: yes
Keywords: Natural Language Processing
Abstract: Data quality is a critical driver of large language model performance, yet existing model-based selection methods focus almost exclusively on English, neglecting other languages that are essential in the training mix for multilingual LLMs. We introduce MuRating, a scalable framework that transfers high-quality English data-quality signals into a multilingual autorater, capable of handling 17 languages. MuRating aggregates multiple English autoraters via pairwise comparisons to learn unified document quality scores, then projects these judgments through translation to train a multilingual evaluator on monolingual, cross-lingual, and parallel text pairs. Applied to web data, MuRating selects balanced subsets of English and multilingual content to pretrain LLaMA-architecture models of 1.2B and 7B parameters. Compared to strong baselines, including QuRater, FineWeb2-HQ, AskLLM, DCLM, our approach increases average accuracy on both English benchmarks and multilingual evaluations. Extensive analyses further validate that pairwise training provides greater stability and robustness than pointwise scoring, underscoring the effectiveness of MuRating as a general multilingual data-selection framework.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a novel method to curate high quality multilingual datasets based on transferring data-quality signals to other target languages. Multiple state-of-the-art English pairwise ratings are aggregated via a Bradley-Terry model into a single quality score. For the multilingual case the samples are translated to the respective languages and either re-rated or the rating is transferred. The data selection approach based on the aggregation of multiple quality scores shows promising performance in English-only as well as when transferred to the multilingual setup.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n- The work presents a novel method to aggregate existing data quality raters in the English and multilingual context demonstrating promising performance. As high-quality multilingual data is lagging behind English data curation this novel method is an important contribution.\n- The work and experiments are generally well structured and easy to follow.\n\n**Weaknesses:**\n- The work mentions Fineweb-HQ in the introduction, which has a significant overlap in languages. To critically discuss the method's performance in the context of existing multilingual work, I would have expected the work to compare to this baseline or at least to other multilingual baseline dataset, such as HPLT-2 and FineWeb-2.\n- The work could be more detailed on the experimental setup, e.g. how was language identification performed or what is the language mixture used (number of tokens by language) for the multilingual experiments.\n- The results of section 4.3.2. Translation Quality Assessment, could be strengthened with discussion including how translation performance impacts data selection and  thus LLM performance.\n\nQUESTIONS:\n- Could the authors please clarify the rationale behind the decision not to compare MuRatings performance with baselines such as HPLT-2, Fineweb-2 and Fineweb-HQ (for all the overlapping languages)?\n- Could the authors provide more detailed information about the multilingual corpus used for LLM training? For example, it would be beneficial to know:\n  - How was language identification performed?\n  - What is the language mixture used (number of tokens by language) for the multilingual experiments?\n  - How were the Fineweb filter adjusted for the multilingual context?\n- Given that the classifier training dataset of FineWeb-HQ reportedly includes resources like the Aya collection (an instruct-style dataset available in 101 languages), could the authors clarify the statement in line [95] regarding the method's reliance on the availability of high-quality samples of multilingual benchmarks? There seems to be a potential discrepancy that warrants further explanation.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper introduces MuRating, a scalable data selection framework that leverages high-quality English data signals to build a multilingual rater across 17 languages. The proposed method consist of multiple steps such as 1) Aggregating multiple English "raters" via pairwise comparisons to learn a unified document-quality score, 2) Transferring this judgement through translation data pairs to train a multilingual quality evaluator, then 3) applying the evaluator to web-crawled mutlingual texts to select high-quality data.
In the experiment, they pretrained a 1.2B parameter LLaMA model using the selected multilingual corpus. Results show substantial improvements over strong baselines like QuRater, Ask LLM and DCLM, particularly on knowledge-intensive tasks.

- this paper is well motivated and mostly easy to follow.
- Tables are too small to read. Please consider making it bigger in the camera-ready version.

STRENGTHS AND WEAKNESSES:
strengths 
- the authors exploits English-quality signals and translation to bootstrap evaluators for low-resourced languages. This help offer a practical path for multilingual data curation.

weaknesses
- there could be selection bias. The proposed approach relies on English-originated quality criteria, so this may not capture culturally or linguistically relevant features in non-English language texts. Have you ever conducted some analysis in this line?
- Experiments are limited to a 1.2B model. It might be still uncertain how the proposed approach scale to larger architectures or more diverse data settings. If you could run additional experiments with e.g., 7B model if the authors have enough computational resources, that'd be great.

QUESTIONS:
see the summary.

LIMITATIONS:
n/a
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
In this paper, the authors proposed Murating, a scalable framework for multilingual LLM pretraining that transfers high-quality English data-selection signals to 17 target languages. Existing model-based data selection methods are primarily English-centric, lacking robust multilingual adaptation. MuRating uses unifying English Raters, Translation-Based Multilingual transfer and Cross-lingual regularization to achieve robust multilingual adaptation.

STRENGTHS AND WEAKNESSES:
Strengths:

(1) The paper is well-written and easy to follow;

(2) MuRating extends data selection to low-resource languages without requiring language-specific annotations;

(3) Aggregating various English rates via pairwise comparisons mitigates individual rater biases.

(4) Parallel text pairs enforce semantic alignment, reducing score divergence.

(5) The experimental results are promising.

Weaknesses:

(1) The translation quality is heavily reliant on LLM translators.

(2) How do the authors guarantee the translation quality, Did the authors use human verifiers? The authors should provide details and discussions on this aspect, as LLM translators are not very skilled in low-resource languages.

(3) As MuRating utilizes multiple LLM as raters, are the comparisons unfair, the  authors should provide discussions on the fairness of the experimental comparison shown in this paper.

QUESTIONS:
Please see Weaknesses.

LIMITATIONS:
Many LLMs are included as raters in the proposed method, so the fairness of the comparisons is a concern.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03