MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining
Authors: Zhixun Chen
Ping Guo
Wenhan Han
Yifan Zhang
BINBINLIU
Haobin Lin
Fengze Liu
Yan Zhao
Bingni Zhang
Taifeng Wang
Yin Zheng
Trevor Cohn
Meng Fang
Venue: NeurIPS 2025 poster ()
Keywords: Natural Language Processing
Primary area: deep_learning
TL;DR: We proposed a new data selection method for pretraining multilingual Large Language Models

Abstract: Data quality is a critical driver of large language model performance, yet existing model-based selection methods focus almost exclusively on English, neglecting other languages that are essential in the training mix for multilingual LLMs. We introduce MuRating, a scalable framework that transfers high-quality English data-quality signals into a multilingual autorater, capable of handling 17 languages. MuRating aggregates multiple English autoraters via pairwise comparisons to learn unified document quality scores, then projects these judgments through translation to train a multilingual evaluator on monolingual, cross-lingual, and parallel text pairs. Applied to web data, MuRating selects balanced subsets of English and multilingual content to pretrain LLaMA-architecture models of 1.2B and 7B parameters. Compared to strong baselines, including QuRater, FineWeb2-HQ, AskLLM, DCLM, our approach increases average accuracy on both English benchmarks and multilingual evaluations. Extensive analyses further validate that pairwise training provides greater stability and robustness than pointwise scoring, underscoring the effectiveness of MuRating as a general multilingual data-selection framework.

MuRating: A High Quality Data Selecting Approach
to Multilingual Large Language Model Pretraining
Zhixun Chen
Ping Guo
Wenhan Han'
Yifan Zhang² Binbin Liu²
Haobin Lin
Fengze Liu
Yan Zhao?
Bingni Zhang² Taifeng Wang²
Yin Zheng*
Trevor Cohn' Meng Fangs.3.1
'Hong Kong University of Science and Technology (Guangzhou) BByteDance
Eindhoven University of Technology *University of Melbourne University of Liverpool
Abstract
Data quality is critical driver of large language model performance, yet existing model-based selection methods focus almost exclusively on English, neglecting other languages that are essential in the training mix for multilingual LLMs. We introduce MuRating, a scalable framework that transfers high-quality English data- quality signals into multilingual autorater, capable of handling 17 languages. MuRating aggregates multiple English autoraters via pairwise comparisons to learn unified document quality scores, then projects these judgments through translation to train multilingual evaluator on monolingual, lingua,, and parallel text pairs. Applied to web data, MuRating selects balanced subsets of English and multilingual content to pretrain and 7B parameters. Compared to strong baselines, including QuRater, Web2- HQ, AskLLM, DCLM, our approach increases average accuracy on both English benchmarks and multilingual evaluations. Extensive analyses further validate that pairwise training provides greater stability and robustness than pointwise scoring, underscoring the effectiveness of MuRating as general multilingual data-selection framework.
arrcitecccii models 1.2B
Introduction
Large Language Models (LLMs) have achieved remarkable performance across a wide range of tasks, and recent studies have consistently emphasized the critical role of high-qualit, pretraining data in driving these advances [5]
51 7 To improve data quality, various strategies have been adopted, such as deduplication 31 1. heuristic and rule-based filtering [1 45 and domain-aware sampling
65 (54) While effective, these methods often rely heavily on manual heuristics and domain expertise, lacking unified or principled framework for evaluating and selecting pretraining data. Moreover, they are typically applied as pre-defined or post-hoc filters, limiting their adaptability to downstream performance. In response, model-based data selection approaches have emerged, aiming to learn data quality judgments from examples or auxiliary supervision. These methods utilize different model architectures and data selection criteria. For instance, DCLM [3 trains FastText classifier [27] using high-que gigh-tulltty samples from OH2.5 and Reddit EL15 as positive supervision, while treating Common Crawl web data negatives. Other approaches such as AskLLM [52). QuRater [64), and the Fine Web-Edu (38) classifier employ prompt-hased evaluation criteria using various L.MSs to assess the quality of input samples.
*Work done during Zhixun's internship ByteDance. Yin Zheng the tech lead of multilingual LLM pretrain project at ByteDance. 1 Corresponding Authors. Emails; yzhengåxg @ gmail..mamlmmmmm...oooooooomoo.e and Meng. Fang liverpool.ac.uk
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Data selection beyond English remains an important challenge [3 30]. While model-basee data selection methods have demonstrated effectiivness in improving training quality, they have been developed almost entirely for English and are not explicitly designed or validated for non-English languages, leaving a critical gap in multilingual data quality assessment. As L.LMs are increasingly applied in diverse linguistic contexts, there is growing need for selection strategies that extend beyond English. A recent attempt Fineweb2-HQ [39] train language-specific raters using benchmark datasets as positive supervision and general pretraining corpora as negatives, following strategy similar to DCLM [33]. However, this approach uses benchmark-derie data, posing a risk test set contamination.
In this work, we introduce MuRating, two-stage, translation-anin and aud-pairwise framework for multilin- gual data-quality estimation. MuRating begins by aggregating multiple state-of- thetthe-rt English raters majority-vote pairwise comparisons, fitting Bradley-Terry model [ to learn single, unified quality scorer. Next, it translates scored English document pairs into each of 17 target languages and construct monolingual, gross-lingual, and parallel pairs- projectnng original preference labels onto translated comparisons and assigning neutral labels to parallel translations. Here, parallel pairs consist of identical content translated into two different languages, while cross-lingual pairs involve distinct texts written in different languages. This design yields one multilingual evaluator that preserves English-derived quality signals while remaining sanguage-agnoste
We apply MuRating framework to fine-tune MuRater model to annotate English and multilingual web documents and select top 10% data to pretrain LLaMA architecttu [17 models of 1.2B and 7B parameters for validation. Compared to strong baselines baselines-uniform uniform sampling with 50% more data, QuRater 64 AskLLM (52), Fine Weh2-Ho 3D -our points on twelve English benchmarks and 1.8 points on diverse multilingual suite. We further assess translation fidelity via human evaluation, examine the impact of cross-lingual and parallel data, and compare different score transfer approaches.
our selection yields an average gain of to 3.4
Our contributions are as follows
Unified English rater aggregation. We consolidate four distinct English quality raters via Bradley-Terry pairwise framework, producing a single, robust scoring model.
Translation-busee multilingual transfer. We show how to project English pairwise judgments into monolingual, cross-lingual, and parallel pairs across 17 languages, enabling language- agnostic quality evaluation.
Scalable pretraining gains. The results from both 1.2B and 7B model experiments demon- strate significant gains over state-of-the-art baselines across English and multilingual LLM benchmarks.
Related Work
Data Selection. Data selection essential in constructing high high-quality pretraining corpora for LLMs and typically falls into three main categories: deduplication, heuristic-based filtering, and LLM- guided quality evaluation. Early-stage deduplication removes exact or near-duplicate documents to minimize redundancy and enhance model generalization (31). More advanced fuzzy and semantic methods filter syntactically or semantically similar content 255 1. which crucial at scale to avoid
training instability and performance degradation [68 [1]]
Heuristic filtering uses rules or lightweight models exclude low-quality text, such short, repetitive, or toxic content (30
47 [55.. While handcrafted heuristics can be effective, they often have limited generalization and inefficiency, prompting the use of simple classifiers, perplexity scores, or importance sampling 59 66, (2 3d 35). However, these approaches may unintentionally favor simplistic or repetitive content, which can diminish the diversity and informativeness of the dataset.
62
In contrast, LLM--gudee quality scoring directly leverages language models to evaluate data along dimensions like factuality and coherence [18 52 34)
Frameworks such as QuRating [64] and FineWeb-BB Web-Edu [38] prioritize educational content using multi-criteria assessment, while Dataman
48 and FIRE 67 extend this to domain-awan aware or reliability-sensiiv filtering. Despite their
OOr code is available at: https: com/ wwt.:o/mmaaaatttttiaeeo

---

Sample English Pairs
Rater Jurgemenes and Integration
Train Large Language Model
Text A:
opportuticios ene eeellerpu. seset lenders are iefeguencing soueets prisecy, dice security, ane scuini
QuRaiing
DCLM
Sa
Sts
Select High Quality Data
SA
extentirher Thee only hantrode of this frxcta bo wes por Limeste truit werther thing?
PineWeel
S,
Edia
AGLLLM
Anzotate Web Corpus
E
Tranchate Muhtlingual Pairs
Multilinuual Pair Scores
Portugusse
Itaban
Sx
S
German
Tha
Koreon
S
MuRater
Sx
5A4
Sa
Indonesian
Arabic
Viennamese
Bussion
Manaliigul parl
Creas-ingual Park
Porehe
Overview of the MuRating pipeline: English document pairs are first annotated using various data selection methods and unified, then translated into multiple languages create diverse multilingual pairs. These are used to train the MuRater model, which scores large-scale web data. The top 10% of scored data selected to train an LLM, yielding superior performance compared to state-of-the-and sampling baselines.
Figure I:
advancement, recent approaches depend heavily on GPT-style judgments, potentially introducing model-specific biases.
Multilingual Pretraining, Efforts to construct multilingual datasets for multilingual LLM pretraining have followed similar strategies to those used for English, incorporating deduplication and heuristic- based filtering techniques. Prominent corpora such as mC4 [69]. RedPajama [62]. CulturalX [43), HPTL 12 and Web-2 [46] leverage these methods to scale multilingual resources, ranging from a few dozen to thousands of languages, significantly enhancing -oossllnngual performance in multilingual LLM pretraining. To mitigate the issue of data scarcity for low-resource languages, -ea-sWeb-FEdu [61] addresses this by translating high-quality English data into multiple languages. In addition, EMMA-X [20] introduces an E.Minspirred framework that jointly learns -ongsll semantic alignment and sentence representations from large-scale non-parallel multilingual data, while CLIMB [9] enhances multilingual capability by dynamically adjusting the data-mix ratio across languages. However, research on model-based data selection for multilingual LLM pretraining remains limited. Recently, an initial approach refine the Fine Web-2 dataset by training language-speciins classifiers, using multilingual benchmark datasets positive examples and web corpus data as negative examples. However, this method relies on the availability of high-quality samples from existing multilingual benchmarks, which may risk contaminating downstream evaluation tasks with biased data.
39] introduced model-based selection method to
3 Methodology
Our approach consists of two stages: (1) consolidating multiple English-language quality raters into a single, unified scorer via pairwise comparisons, and (2) transferring the scorer to a multilingual setting through translation-base based alignment and cross-lingual regularization. We introduce a two-step method: first, we integrate existing English corpus quality raters; second, we transfer their rating capability to a multilingual setting.
3.1 Integration of English AutoRaters
To consolidate quality judgments from multiple pre-existing English raters, we employ a pairwise comparison framework grounded in statistical preference modeling. Let (LA, (B) denotes a pair of texts randomly sampled from a large corpus, and let N be the set of raters. Each rater N assigns a scalar score to both texts, denoted S% and Sn. reflecting the rater's estimation of the quality of t and tg, respectively.
3

---

We define a binary preference for each rater: if S > Sg. we consider that rater prefers text E over B. and vice versa. the two scores are nearly identical (i.e., 1Sa S c), we treat the preference as ambiguous and discard the pair from the training dataset. Based on the remaining valid preferences from all raters, we compute an empirical confidence score PA>B B indicating how likely A is preferred over Lp:
PA>B
Σ REN
Sph,
PASB E 10, where 1S S
(I)
where I[:] is the indicator function that equals 1 when the condition is true and 0 otherwise. if This score quantifies the relative preference strength of a over t across all raters.
To construct large-scale preference dataset, we apply this scoring procedure across wide set of sampled text pairs. This process yields a judgment dataset: J {(AA. PA>B)} consisting of text pairs and the estimated probability of preference.
To convert these pairwise comparisons into continuous scalar quality scores, we employ a learning framework based on the Bradley-Terry model [4]. Let so(t) denote the learnable scalar quality score of text i, parameterized by We adopt a binary cross-entropy loss function, following the formulation proposed in [64], which is analogous to the reward model training paradigm in Reinforcement Learning from Human Feedback (RLHF) [4], but without incorporating user prompts or conditioning input queries:
((-ttt))))(-P--a))))gg////////// so(()))) (2)
Co
E (CAstn.Pm=AN) A JEO
PB-A
where (-) denotes the sigmoid function, and Pв~A = 1 PA>B is the empirical probability that t preferred over LA- This formulation encourages the model to assign higher scores to texts that are consistently preferred in the pairwise judgments.
After training, the model outputs single scalar score representing the quality of each document. These scores are treated as logits over the dataset and are used for quality-based sampling, where a subset of high-quality texts is selected based on their relative scores.
3.2
Multilingual Data Quality Rater
3.2.1
Translation-Base Alignment of Multilingual Preferences
To extend data quality scoring from English to a set of target languages M, we adopt translation- based strategy. Building the scored English text pairs introduced in the previous section, translate each document pair (L : ( into a target language m E M. For each pair, we compute confidence score PA Bun following Equation 1 and then directly transfer this preference the translated pair by assuming PA R PA-">Ben.
We also experimented with the reverse approach translating multilingual pairs into English, scoring them in English, and then projecting the scores back to the corresponding multilingual pairs. A comparison between the two setups is presented in the experiment section 4.2.1
This assumption is based on the premise that translation preserves both the semantic content and the relative quality between text pairs. Prior work QuRating [64] highlights that pairwise com- parisons offer increased stability when evaluating text quality. multilingual settings, pointwise acoring-where absolute quality scores are assigned to individual texts-is more susceptible to subtle changes in tone or phrasing introduced during translation, which can compromise the consistency of the supervision signal. In contrast, pairwise supervision is inherently more robust to such translation- induced variations. As long as the relative ranking between the texts remains consistent (i.e., tx continues to be preferred over ten after translation), the corresponding translated pair (E ( re- mains a valid training example. This robustness makes pairwise comparisons a more reliable and effective framework for training quality evaluation models in multilingual contexts.
3.3 Cross-Lingual and Language-Agnostcc Alignment
While the previous section addressed only in-language supervision-i.e. training on text pairs (t% em) where both documents are in the same language m m-this setup alone is insufficient to

---

guarantee language-agnon agnostic scoring behavior. To promote consistency in quality assessments across languages, we augment our training dataset with both cross- mooss-lingual and parallel text pairs.
For cross-lingual pair construction, we generate mixed-language pairs by randomly translating L and LB into different target languages, resulting in pairs of the form (4%, / with m o m'. The original English pairwise preference score then transferred these cross-lingual pairs by assuming P. PAm> umm Fmm
2 " > gen.
For parallel pair construction, we build semantically equivalent pairs explicitly regularize the model's behavior. Given a text l and its direct translation tx into another language m', form the pair (tT & and assign neutral preference score, i.e., PA">A"" 0.5. This reflects the expectation that both texts, despite being in different languages, convey identical semantic meaning and should be treated as equally quality.
Formally, these neutral-pair constraints act as a regularization signal that aligns the model's internal representation of quality across languages:
Cparallet B rereer)ee (e)))) [- log a (s((tT) se(eT') log (sT so (
(3)
where J' is the datasets of parallel pairs. This formulation encourages the model to minimize score divergence between translations while still preserving the ability differentiate documents of genuinely different quality in the broader training set.
3.3.1 Multilingual Rater Objective
The final loss function is combination of the original pairwise loss from same-language and cross-language comparisons, along with the parallel text regularization term:
Lrotal Cpirwise + A Cparallll
where Cpairiise means the loss calcuated by equation 2 for both monolingual and cross-lingual pairs, is a tunable hyperparameter balancing cross-lingual consistency and discrimination. This joint training approach allows to construct multilingual quality rater that is robust, consistent across languages, and sensitive to relative quality differences.
3.3.2 Training the Rater Model
To construct high-quality multilingual rater, begin with 300,000 English text pairs anno- tated using four rating methods. For GPT-4o-based annotation, we prompt the model in both (trcctions-(A,.f55 tg) and (tg, -u)mmuliple times to mitigate order bias, and compute the final con- fidence score PA>BB by averaging the predicted preference probabilities. For other raters (AskLLM (38), DCLM (33)), we collect their individual scores and derive pair-
[52] wew--eee--E---sseeeeee wise preferences following Equation] []
We then extend these English pairs into multilingual settings using GPT-4o for translation: 150,000 monolingual pairs, 150,000 cross-lingual pairs, and 75,000 parallel pairs ( en ), with language proportions balanced across all target languages. The combined dotaset-comprrsing English, monolingual, cross-lingual, and parallel examples-form the final MuRater training set. We adopt QuRater's 's training setup [64), applying a confidence margin to all but the parallel examples.
We fine-tune an encoder-basee model following the BGE-M3 architecture (6), adding linear head predict quality ratings. We choose BGE-M3 (6 for its strong multilingual representation ability and lightweight design, which make it well-suited for large-scale multilingual scoring. The resulting rater achieves over 93% accuracy on the validation set and 97% on the training set, demonstrating strong multilingual preference modeling. The effect of translation quality and implementation details, including tokenizer settings and hyprrparameters, are provided Appendix A
Experiments
4.1 Experimental Setups
Dataset construction. We build on the deduplication and heuristic-fitterng pipelines of FineWeb-2 46 to assemble large web-crawl corpus. It comprises 1.5 trillion English tokens plus 3 trillion
5

---

tokens across 17 additional languages (Arabic, Chinese, Dutch, French, German, Indonesian, Italian, Japanese, Korean, Portuguese, Russian, Spanish, Thai, Turkish, Vietnamese, Malay, Tagalog). We then apply MuRater and other baselines to assign quality scores to every document, Although scoring trillions of tokens compute-intensive. it parallelizes efficiently across GPUs, and batching strategies reduce overhead in practice. Corpus statistics are detailed in Appendix B.1
Baselines. For the English experiments, we train and evaluate the model using only English pairwise data, comparing it against several established data-quality raters: QuRater, which selects data based on educational value [64]: AskLLM, which follows the prompt design in [52] using Flan-T5-XX
[8]; the Fine Web-Edu Class ifier trained on 450K LLaMA3-7000 700--InIttteee 4 labels to identify
educational content; and DCLM fastText classifier trained on high quality dataset to differentiate
between informative and low-quality web content.
For the multilingual experiments, we extend the QuRater framework [64] to build QuRater-M, which employs GPT-4o to annotate multilingual pairs via relative preference, following the same training pipeline as its English counterpart. We further include datasets from HPTL []2] and FineWeb-2
(46] for comprehensive comparison, and evaluate against FineWeb2-HI (39]. In addition, we analyze the difference between MuRater(E) and MuRater(MM: MuRater(MM scores multilingual pairs that have been translated into English, whereas MuRater(E)) starts from rated English data and translates it into multilingual pair and cross- tross-lingual pair form. Most multilingual evaluations cover 17 languages plus English, while the experiment involving Fine FineWeb2-HQ is restricted to 13 overlapping languages (Arabic, Chinese, Dutch, French, German, Indonesian, Italian, Japanese, Portuguese, Russian, Spanish, Turkish, and Vietnamese) to ensure fair comparison.
Finally, we include Uniform baseline for both settings, which randomly samples 50% more data than the other methods, following the setup of QuRating [64]. Details are provided in Appendix B.2
Training Setup. We train randomly initialized language model based on the LLaMA architecture 17 for a single epoch over the training corpus, with data presented in a randomly shuffled order. For most experiments, the model comprises 1.2 billion parameters and employs a standard transformer architecture [60] augmented with rotary position embeddings (RoPE) [56]. To accommodate the multilingual setting, extend the tokenizer vocabulary through retraining the multilingual corpus. Comprehensive architectural and tokenizer details are provided in Appendix B.3
Building on this setup, we construct the training corpora for both English and multilingual experiments. For the English setting, we select the top-scored 200 billion tokens from the full pool of 1.5 trillion tokens across all baseline methods and MuRater. In the multilingual setup, we apply all methods to score and select the top 10% of tokens within each language, yielding roughly 300 billion tokens in total. These multilingual tokens are then combined with the 200 billion English tokens to form unified 500-billion token pretraining corpus.
To further assess robustness and scalability under varied training conditions, we additionally conduct experiments with 7B 7B-parameter model sharing the same LLaMA architecture as the 1.2B model. This larger model is pretrained on IT tokens, with 16.5% multilingual data selected by either MuRater or QuRater-M, while the remaining 83.5% comprisin English, code, and math Gata-remains identical across both setups. The multilingual portion follows the same language distribution as in the main experiments, enabling a more comprehensive evaluation of generalization across heterogeneous data sources.
Evaluation Benchmarks. We assess the performance of our pretrained models using the 1m- nn--evalaation harness framework (14). For the English-only evaluation, we consider suite of ten tasks spanning multiple linguistic competencies. These include six reading comprehension ARC-Challenge [TO], SciQ [63). LogiQA (37], TriviaAQ and BoolQ 9 four commonsense reasoning tasks HellaSwag 70). PIQA 3. OpenBookQA 40 and Wino-
Grande [53]; and two World Knowledge Daaksk-Nturrll Questions (NQ) 22 and MMLU 23]. For MMLU, we follow 2 and employ the lighteval variant to ensure more consistent and reliable
comparisons.
http:://wgggggggeeee///////////e clssciassiieer
httppp//wwggunnndeeecc/mettaetas -laaaaaaaaaaaaaaaa---aaaaaa---- Instruct

---

For the multilingual evaluation, we utilize translated versions of several English benchmarks [21 alongside multilingual- datasets. Similar to English benchmarks, we divide the task into three categories, including reading comprehension, commonsense reasoning. and world knowledge understanding. Reading comprehension evaluated using translated versions of ARC-Fasy and ARC-Challenge [TO], StoryCloze 41], and XNLI [1]. which assess contextual understanding and inference. Commonsense reasoning is tested with HellaSwag [70], XCOPA [49], and XWinograd [58]. focusing event causality, semantic plausibility, and everyday reasoning across languages. World Knowledge evaluation includes MMLU [3, BMLAMA (50), and FLORES [6], which examine factual knowledge, translation quality, and multilingual alignment.
To strengthen the evaluation of kanguage-specifne knowledge, we also incorporate localized MMLU variants- CMMLU [32], VMLU IndoMMLU (28). JMMLU and AMMLU to construct region-specific multilingual extension of MMLU, denoted as MMLU_L and add to the world knowledge category. Together, these benchmarks provide a comprehensive evaluation of cross- lingual comprehension, commonsense inference, and noowledge-grouddee reasoning, enabling holistic assessment of multilingual LLM performance. Detailed information of benchmark statistics and language coverage is provided in Appendix C
4.2
Main Results
4.2.1
Multilingual Results
Table I: Results multilingual benchmarks with different training setups. Best results within each setting are shown in bold
Reading Comprehension
Commonsense Reasoning tasks)
World Knowledge (4 tasks)
Average (11 tasls)
Selection Method
(5 tasks)
18 Languages Results
Uniform
53.16
$4.58
38.25
48.66
HPI.T-2
50.38
49.77
36.96
45.70
FineWeb-2
50.83
52.48
35.53
46.28
QuRater-M
54.58
54.87
38.12
49.19
MuRater(M)
50.02
MuRater(F)
56.05
56.42
40.40
50.96
13 Languages Results
Fin-Wob2-H
53.05
55.54
38.31
48.97
MuRater(EE
55.95
$8.30
41.17
51.81
7B Model Results
Qulater-M
61.96
63.28
43.31
56.18
62.78
44.50
57.23
As shown in Table TubleTane and Figure 2 MuRater substantially outperforms existing multilingual baselines across nearly all evaluation categories and settings. Under the 18-language configuration, MuRater(B)) achieves the highest These consistent gains highlight MuRater's capacity to identify high-quality, semantically rich, and educationally valuable text, even when faced with heterogeneous multilingual inputs. particular, the large improvements in reasoning-oriented benchmarks (e.g.. ARC and MMLU) suggest that MuRater selects examples with deeper conceptual structure and higher linguistic clarity, thereby enhancing model comprehension and reasoning generalization.
scores in all three categories, outperforming all baselines.
The performance gap between MuRater(M) and MuRater(E) further highlights the advantages of English-anchee anchored training. MuRater(E) leverages English-rated pairs and projects the resulting preferences into the multilingual space, providing more stable and transferable supervision signals. We attribute this benefit to the broader topical and stylistic diversity of English corpora, which expose the rater to richer linguistic variation and more representative pairwise patterns during training. As a result, MuRater(E) learns to generalize beyond language corpus boundaries and more effectively
https://vmlu.

---

capture shared semantic dimensions across languages, yielding stronger cross-lingus alignment and greater robustness to conslation-induce noise.
On the I.-lang subset, MuRater(E)) continues to outperform FineWeb2-HO by roughly points on average, achieving leading results across all three evaluation dimensions. When scaled to the 7B model trained on IT tokens, MuRater maintains its superiority, reaching higher performance in all three respective dengorres-ddmoostrtatti consistent gains across both model sizes and data distributions. These results confirm that MuRater generalizes effectively to diverse linguistic environments and evaluation conditions, offering more scalable and reliable framework for multilingual data quality estimation. Detailed results on each language are shown in Appendix D
eic chaloiggal PESTAAT
(a) Average performance
(b) ARC-Challenge-MM
(c) MMLU-ML
Performance of different selection methods on ARC-Challenge-ML. MMLU-ML, XWino- grad, and the overall average across all tasks during training on 200B English + 300B multilingual tokens.
Figure 2:
4.2.2 English-only Results
Table 2: Performance of different selection method over all different downstream tasks, Best results of each task category marked in black. Detailed results are performed in Appendix D
Selection Method
Reading Comprehension
Commonsense Reasoning
World Knowledge
Average (12 tasks)
(6 masks)
(4 tasks)
(2 misks)
Uniform (+550% data)
43.93
59.06
20.36
48.70
AskLLM
42.83
58.40
20.21
47.82
DCLM
46.00
$8.99
22.37
50.23
FineWeb
45.71
$7.49
22.00
QuRater
43.54
58.58
20.47
48.33
MuRater
47.13
59.95
22.53
51.23
The results in Table 2 indicate that our proposed rater successfully consolidates the strengths of existing rating methodologies, leading consistent improvements in pretrained model performance across all categories of evaluation tasks. Baseline comparisons reveal that each selection method exhibits distinct preferences for data, which translate into varying levels of effectiveness downstream tasks. For instance, as shown in Figure DCLM yields strong results on HellaSwag but underperforms on ARC-Challenge. Conversely, QuRater achieves competitive performance on ARC- ARC-Challenge but demonstrates poor results on TriviaQA. In contrast, our MuRater integrates the advantages of these methods and achieves robust performance across nearly all benchmarks. outperforming other data selection baselines by margins ranging from 1 to 3.4 percent. The model trained with our rater consistently achieves superior results on all tasks throughout the training process. This indicates more stable and efficient learning, further validating the effectiveness of our data selection approach in enhancing the quality of LLM pretraining.
different
4.3 Ablation Study
4.3.1
Effectiveness of Cross-Lingual and Parallel Pair Integration
Incorporating cross-lingual pairs and parallel translations during training significantly improves the consistency of quality scoring across languages. To validate this, we assess multilingual raters on parallel corpora-semanticn semantically equivalent texts in different languages. As shown in Figure 4 our alignment-base training produces models with lower mean squared error (MSE) and slopes closer to one, indicating stronger tross-lingual consistency. In ideal case, a language-agnostic rater would

---

(a) ARC-Challenge
(b) HellaSwag
(c) TriviaQA
Figure 3: Performance of different selection methods on ARC-Challenge, HellaSwag, TriviaQA, and the overall average across 12 tasks during training on 200B English tokens
assign identical scores to parallel texts across languages, resulting in a slope of one and minimal MSE between their score sequences,
These findings highlight the importance of modeling interlingual relationships. By leveraging cross- language comparisons and parallel data, the rater learns language-invarinn quality standards, enabling more reliable multilingual evaluation. qualitative case study in Appendix [E] further supports this, showing that high-rated texts consistently exhibit greater fluency, coherence, and instructional value across languages. We further examine how the selected data is distributed across semantic domains in different languages. Detailed results are provided in Appendix B.1
Sceen Oee Cuglth Conoee
Sevrc in inflccee Zegfin Koreen
ther intect for of
Chers Incold
(a) Arabic
(b) Chinese
(c) Korean
Figure 4: Scatter plots of scores assigned by multilingual raters 10,000 parallel documents across various languages. Green points represent ratings from raters trained with alignment using parallel and cross- lingull pairs, while blue points indicate scores from unaligned raters.
4.3.2 Comparison Between Pairwise and Pointwise Score Transfer
We examine the relative effectiveness of pairwise versus pointwise judgment methods for transferring English scoring capabilities multilingual settings. Based on the translation quality evaluations, we select two high-performing languages, Arabic and Spanish, for the study. Specifically, we translate 200 English text pairs into Arabic and 200 pairs into Spanish. Each dataset is then annotated by GPT-4o using both pairwise and pointwise scoring strategies. For pointwise annotation, GPT-4o assigns quality scores on 1-10 scale. The scoring prompts of both methods explicitly instruct GPT-4o to evaluate based content quality alone, irrespective of language and are detailed in Appendix(A.6) Each text or pair scored 20 times, and average is used as the final score. Given identical content across different languages, the ideal scenario is that a consistent model and prompt should yield nearly identical scores, regardless of the language and score strategies.
As shown in Figure pointwise scores exhibit considerable variability across languages, particularly the mid-quality range (scores between 3 and 6), despite relatively stable assessments at the high and low ends. Ideally, all points should align closely with the x (slope 1) line, which would indicate identical score for semantically equivalent content across languages. In contrast, pairwise judgments display strong cross-lingual consistency, with only minor deviations from this ideal alignment. These observations suggest that while translation quality generally adequate, subtle translation biases can still influence absolute (pointwise) ratings. The pairwise approach, however, demonstrates greater robustness to such variation, underscoring its effectiveness for reliably transferring English scoring behavior to multilingual contexts.

---

SCONI
Bryee Now
(a) Arabic (Pointwise)
(b) Spanish (Pointwise)
(c) Arabic (Pairwise)
(d) Spanish (Pairwise)
Figure 5: Scatter plots of average scores assigned by GPT-4o to Arabic and Spanish parallel data. Each point represents an average of 20 evaluations. Left: pointwise scoring. Right: pairwise scoring.
Conclusion
We introduced MuRating, scalable multilingual data selection framework that aggregates mul- tiple English raters via a Bradley-Terry pairwise model and transfers these judgments through translation to train a single multilingual MuRater over monolingual, cross-lingual, and parallel pairs. Applied to large web corpora, MuRater is used to pretrain both 1.2B- and 7B-parameter -aMM--arccitecture models and delivers consistent improvements over strong baselines (QuRater, He. AskLLM, DCLM) on multiple English and multilingual benchmarks. Ablation results further demonstrate the effectiveness of incorporating cross-lingual and parallel pairs, and confirm that pairwise supervision provides more stable multilingual scoring than pointwise methods. Analyses across translation fidelity and data composition, together with results at two model scales, indicate that MuRating is effective and scalable for large-scale multilingual data curation and yields reliable performance gains across evaluation settings.
Limitations
Our current study focuses on 17 target languages excluding English, leaving substantial room for broader linguistic coverage. The reliance on GPT-4o introduces potential biases and idiosyncrasies in- herent to proprietary large language models. Moreover, since the English raters used in our framework primarily focused on factual and informational content, our auto-rater exhibits limited performance narrative and creative domains. While the proposed approach performs well language-speciff benchmarks, further research could explore language specific rater designs or culturally aligned data selection strategies better capture the unique characteristics of each language. Future work will also aim to incorporate higher-quality translations, expand to wider range of languages, and develop adaptive data sampling techniques enrich the diversity and representativeness of the curated multilingual corpus.
References
[1] Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semd- cdup: Data-efficient learning web-scale through semantic deduplication. arXiv preprint
10

---

arXiv:2303.09540, 2023.
[2] Norah Alzahrani, Hisham Alyahya. Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yousef Almushaygih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, et al. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13787-13888 2024.
[3] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. arXiv preprint arXiv:/911.1164 2019.
[4] Ralph Allan Bradley and Milton E Tenry Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-355, 1952.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[6] Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3- embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through gelf-knowledge distillation, 2024.
[7] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1- 113, 2023.
[8] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction-finttus language models. Journal of Machine Learning Research, 25(70): 2024.
[9] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2924-2936, 2019.
[10] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1rXiv:1803.04555 2018
[11] Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross- cross-lingual sentence representations. arXiv preprint 1.K.00880000053.. 2018
[12] Ona De Gibert, Graeme Nail, Nikolay Arefyev, Marta Bañón, Jelmer Van Der Linde, Shaoxiong Bernabeu, Mikko Aulamo, Gema Ramírez-Sánchez, Andrey Kutuzov, et al. A new massive multilingual dataset for figh-performance language technologies. arXiv preprint arXiv:2403. 14009, 2024.
Ji, Jaume Zaragoza-Bernabeu.
[13] Meng Fang, Yuan Li, and Trevor Cohn. Learning how to active learn: A deep reinforcement learning approach. Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 395-605, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
[14] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
[15] Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. Un- packing tokenization: Evaluating text compression and its correlation with model performance. arXiv preprint 2rXN::2403.06265, 2024.
11

---

[16] Naman Goyal, Cymthia Gao. Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Mare' Aurelio Ranzato. Francisco Guzmán, and Angela Fan. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522-538, 2022.
[17] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint 2rXiv:240..2178. 2024.
[18] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306. /644, 2023.
[19] Ping Guo, Yubing Ren, Binbin Liu, Fengze Liu, Haobin Lin, Yifan Zhang, Bingni Zhang, Taifeng Wang, and Yin Zheng. Exploring polyglot harmony: On multilingual data allocation for large language models pretraining. arXiv preprint (rXivv2509.15555 2025.
[20] Ping Guo, Xiangpeng Wei, Yue Hu, Baosong Yang, Dayiheng Liu, Fei Huang, et al. Emma-x: an em-like multilingual pre-training algorithm for cross-lingual representation learning, Advances in Neural Information Processing Systems, 36:101 10-111414,1 2023.
[21] Wenhan Han, Yifan Zhang, Zhixun Chen, Binbin Liu, Haobin Lin, Bingni Zhang, Taifeng Wang, Mykola Pechenizkiy, Meng Fang, and Yin Zheng. Mubench: Assessment of multilingual capabilities of large language models across 61 languages, 2025.
[22] Yifei He, Alon Benhaim, Barun Patra, Praneetha Vaddamanu, Sanchit Ahuja, Parul Chopra, Vishrav Chaudhary, Han Zhao, and Xia Song. Scaling laws for multilingual language models. arXiv preprint arXiv:2410. 12883, 2024.
[23] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint vrXiv:2009.03300, 2021.
[24] Dieuwke Hupkes and Nikolay Bogoychev. Multiloko: multilingual local knowledge bench- mark for lims spanning 31 languages. arXiv preprint MXXi:25544113566 2025.
[25] Tao Jiang, Xu Yuan, Yuan Chen, Ke Cheng, Liangmin Wang. Xiaofeng Chen, and Jianfeng Ma. Fuzzydedup: Secure fuzzy deduplication for cloud storage. IEEE Transactions on Dependable and Secure Computing, 20(3):2466 2446-22333, 2022.
[26] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint 17X1v=1705.03551 2017.
[27] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint 16X0:11607.0175) 2016.
(28) Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. Large language models only pass primary school exams in Indonesia: A comprehensive test on IndoMMLU. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing (EMNLP), Singapore, 2023, Association for Computational Linguistics.
[29] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453 466, 2019.
[30] Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems, 35:31809 1809-3892222 2022.
12

---

[31] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Buree and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2
2021.
[32]
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Yichong Xu, Yujia Qin, Zihan Liu, Yiming Cui, and Yue Zhang. Cmmlu: Measuring massive multitask language understanding in chinese. In Findings of the Association for Computational Linguistics (ACL), 2024.
[33] Jeffrey Li, Alex Fang. Georgios Smyrnis, Maor Ivgi. Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems, 37:14200-14282, 2024.
[34] Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, and Tianyi Zhou. Selective reffection-tuning: Student-selected data recycling for LLM instruction-tuningg In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics ACL 2024, pages 161 1689-1621,, 1, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics.
[35] Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfitternng: Weak-to-strong data filtering for fast instruction-tuning In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14275-14273, Bangkok, Thailand, August 2024. Association for Computational Linguisties.
[36] Ming Li, Yong Zhang. Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting LLM performance with self- guided data selection for instruction tuning. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Pupers), pages 7595-7628, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
[37] Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint vrXiv:2007.08122 2020.
[38] Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024.
[39] Bettina Messmer, Vinko Sabolčec, and Martin Jaggi. Enhancing multilingual I1m pretraining with model-based data selection. arXiv preprint vrXiv22502.10361 2025.
[40] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018
[41] Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv: /604.01064000 2016.
[42] Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling cota-constr language models. Advances in Neural Information Processing Systems, 36:50358-50333 2023.
[43] Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernon- court, Ryan A Rossi, and Thien Huu Nguyen. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. arXiv preprint wwX.::2309.09400, 2023.
[44] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishiin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 05:27730-27744, 2022.
13

---

[45] Guilherme Penedo, Hynek Kydliček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:3081 1-30849, 2024.
[46] Guilherme Penedo, Hynek Kydliček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. Fineweb2: One pipeline to scale them all adapting pre- pre-training data processing to every language, 2025.
[47] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon I1m: outperforming curated corpora with web data, and web data only. arXiv preprint 2rXiv:2306.0111 2023.
[48] Ru Peng, Kexin Yang, Yawen Zeng, Junyang Lin, Dayiheng Liu, and Junbo Zhao. Dataman: Data manager for pre-training large language models. arXiv preprint urXiv:2502.19363 2025.
[49] Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen, Xcopa: A multilingual dataset for causal commonsense reasoning. arXiv preprint arXiv:2005.00333 2020.
[50] Jirui Qi, Raquel Fernández, and Arianna Bisazza. Cross-lingual consistency of factual knowl- edge in multilingual language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, 2023.
[51] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112. 1111:211..1146, 2021.
[52] Noveen Sachdeva, Benjamin Coleman, Wang- Lang-Cheng Kang, Jianmo Ni, Lichan Hong. Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient Ilms. arXiv preprint arXiv:2402. 096668 2024.
[53] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint urXiv: 10641, 2019.
[54] Zhiqiang Shen, Tianhua Tao, Liqun Ma. Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, et al. Slimpajama-de: Under- standing data combinations for IIm training. arXiv preprint arXiv:2309. 10818, 2023.
[55] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Flazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00155) 2024.
[56] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Nearocompating, 568:127063 2024.
[57] Qwen Team. Qwen3 technical report, 2025.
[58] Alexey Tikhonov and Max Ryabinin. It's all in the heads: Using attention heads as a baseline for cross-lingual transfer in commonsense reasoning. arXiv preprint arXiv:2106. 2021.
[59] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397 13971, 2023.
[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[61] Jiayi Wang, Yao Lu, Maurice Weber, Max Ryabinin, Yihong Chen, Raphael Tang, and Pontus Stenetorp. Multilingual pretraining using a large corpus machine-translated from single source language. arXiv preprint arXiv:2410.23395 2024.
14

---

[62] Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexan- drov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Trri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpajama: an open dataset for training large language models. NeurIPS Datasets and Benchmarks Track, 2024.
[63] Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Leon Derezynski, Wei Xu, Alan Ritter, and Tim Baldwin, editors, Proceedings of the 3rd Workshop Noisy User-generater Text, pages 94-106, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
[64] Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. arXiv preprint arXiv:2402.097399 2024.
[65] Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoe V Le. Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36:69798-6998 18, 2023.
[66] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing 3ystems,,6:3420111 36.34201-342277, 2023.
[67] Liangyu Xu, Xuemiao Zhang, Feiyu Duan, Sirui Wang, Jingang Wang, and Xunliang Cai. Fire: Flexible integration of data quality ratings for effective pre-training. arXiv preprint arXiv:2502.00761, 2025.
[68] Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To repeat or not to repeat: Insights from scaling 1lm under token-crisis. Advances in Neural Information Processing Systems, 36:59304 59322, 2023.
[69] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text trans- former. arXiv preprint arXiv:2010.11934, 2020.
[70] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 4791-4800, 2019.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract clearly state the main contribution of our paper: We proposed new multilingual data selection method for the pretraining of LLMs. The claims are directly supported by4 the proposed methodology and are intended to be validated through experimental results. The scope is defined as efficiently pretrain multilingual LLMs.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have included a limitation section at 5 below the conclusion. This section explicitly discuss the potential problem of using of GPT-4o and the computational cost of our experiments.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide formulas that correctly numbered and cross-referenced in Section which includes the integration of different rater signals and training objective of our rater model MuRater.
3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We discuss the experiments details in section 4.1 including datasets, training models, baselines and evaluation benchmarks we use. The hperparameter settings for our method and baselines, as well architectural details of the models, are provided in the main text and/or an appendix. to facilitate reproducibility.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
17

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We plan release the source of our prompts, code, and data used facilitate further research upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cct
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
/ //nips. //nips.cc/publl/ wwtpppbicc/guiesee.eeoeepspeeeee huabli//ggudddssodeeuubiiiiis wwweeesuubmsssoooooo...
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The paper specifies training details including the number of epochs/timesteps, and batch sizes for our method and baselines. Details on hyperparameter selection and architectural of both pretrained LLM and MuRater model provided in the experimental sections and further elaborated the appendix and supplemental materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
18

---

Experiments of error bars are included. such as Figure 6 We take LLM experiments based on previous work setup and ensure the statistical significance of our experiment results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Information regarding the computational resources used for experiments, such as the type of GPUs (c.g., NVIDIA H100) and CPUs, are provided in the appendix and supplemental materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppp://ooorrsssccppp////hhhaap
Answer: [Yes]
Justification: The research presented this paper involves the development and simulation- based evaluation of algorithms for mechanism design. For the translation qualtiy assessment, we follow the ethics guidelines are provided details in teh sppendix/supplementtay materials.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
19

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The paper discusses the potential positive societal impacts of improving the multilingual ability of LLMs. We also briefly acknowledge potential negative societal impacts in the "I "Limitations and Future Work" section by analysing the computational cost of our experiments.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e-g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer:[Yor Yes]
Justification: We ensure the data we using to pretrain large language models follow the guidelines. We follow the strategy clear risk datasets. We provided more details about this part the appendix/ supplementary materials.
Guidelines:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
20

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited existing assets through citations to their original publica- tions in the bibliography.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces new assets, including a dataset and accompanying code. Comprehensive documentation is provided alongside these assets, detailing aspects such as data collection methods and preprocessing steps. We will release the code and data used for MuRater upon publish.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer:[Yes]
Justification: The study involved translation quality assessment of human experts. All participants were provided with detailed information about the study's purpose, procedures, potential risks, and their rights, including the right to withdraw at any time. Informed consent was obtained from participants prior to their involvement. The research protocol was reviewed and approved by the Institutional Review Board (IRB) at our institution, ensuring that ethical standards were upheld throughout the study, We include the detail in the appendix/supplemennaay materials.
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The study involved translation quality assessment of human experts. All participants were provided with detailed information about the study's purpose, procedures, potential risks, and their rights, including the right withdraw any time. Informed consent was obtained from all participants prior to their involvement. The research protocol reviewed and approved by the Institutional Review Board (IRB) at institution, ensuring that ethical standards were upheld throughout the study. We include the detail in the uppendix/suppeementay materials.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: In this research, LLMs are integral to the core methodology. Specifically, we employ GPT-4o to generate data selection signals and translation data to train our data selection model, MuRater, which enhances the diversity and robustness of our dataset. Additionally, we pretrain LLMs to verify the effectiveness of our method, which is the core contribution of our paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: http:p/:e//aarrppssc////eerrrree cc/Conferencees22225 for what should or should not be described.
22

---

A Details of MuRater Model
A.1 Different Annotation Method
GPT annotation We adopt the educational value prompt criteria from QuRating 64 as our annotation prompt for GPT-4o-08-06 as detailed below. This prompt is used to annotate total 300,000 document pairs. For each pair, we randomly extract segment of n. tokens- tokenizer 3 n sampled from a uniform distribution n ~ Uniform (566, 512] in 50% of cases, and fixed 5 12 tokens otherwise. Annotation involves generating 20 predictions of either "A" or "B" per criterion and document pair (in either order). The total cost of dataset creation amounts to $9,740.
the LLaMA
Pairwise Educational Value Prompt
Compare two text excerpts and choose the text which has more educational value, e.g., includes clear explanations, step-by-step reasoning, questions and answers.
Aspects that should NOT influence your judgement:
Which language the text is written in
2. The length of the text
The order which the texts are presented Note that the texts are cut off, so you have to infer their contexts. The texts might have similar quality, but you should still make a relative judgement and choose the label of the preferred text.
[Option label ... ... [Option label b] ... text b
Now you have to choose between either label or label Respond only with single word.
AskLLM We adopt the approach from [52 and use the following prompt to query for annotation 300,000 document pairs.
Ask-LLM prompt
###
This is pretraining datapoint. ###
Does the previous paragraph demarcated within ### and ### contain informative signal for pre-training farge-langues model? An informative datapoint should be well-formatted, contain some usable knowledge of the world, and strictly NOT have any harmful, racist, sexist, etc. content.
OPTIONS: -es no
Fineweb and DCLM For these two data selection methods, directly use the open-sourced model to annotate documents to obtain the scores.
A.2 Training details of MuRater and training accuracy
We adopt the XLM-RoBERTa architecture encoder model BGE-M3 (6] as the foundation of multilingual rating model, MuRater, and fine-tune it by appending linear regression head to the transformer output to predict quality scores. The fine-tuning process employs confidence margin threshold of 50%, defined as |PA -Pal 12p A -1 for a prediction between text pairs (LA, (B) (64] Fine-tuning is conducted over epochs with a batch size of 512 and learning rate of 2 10 We set to 0.5. Performance on held-in and held-out sets is summarized in Table 3 Notably. BGE-M3 supports over 100 languages and leverages large-scale multilingual unsupervised data to learn shared semantic space, making it particularly effective for multilingual and cross-lingual retrieval and rating tasks.
A.3 Translation
The translation prompts is
23

---

Evaluation Dataset
Confidence Margin
Accuracy
50%
94.3%
Training set (held-in)
80%
97.2%
50%
90.7%
Validation set (held-out)
80%
93.1%
Table Prediction accuracy of MuRater on held-in and held-out datasets under different confidence margins.
Translation Prompt
Please translate the following (lang) text into (lang2). Your translations must convey all the content in the original text and cannot involve explanations or other unnecessary information. Please ensure that the translated text is natural for native speakers with correct grammar and proper word choices, Your translation must also use exact terminology to provide accurate information even for the experts in the related fields.
The text is [text)
We translate total of 600,000 English document pairs evenly across 17 languages using the GPT-4o- 08-06 model, with the overall translation cost amounting to $18,720.
A.4 Human translation quality evaluation
To evaluate the translation quality of GPT-4o outputs, we employed professional human translators to assess a selected subset the generated texts. evaluators possessed CEFR C1-1evel or higher proficiency in both English and the respective target language. Fach language translation was reviewed by single expert. Evaluators were compensated at a rate of $16 per hour, with each assessment session lasting approximately 4 hours. the annotation criteria for translation quality is shown below.
Annotation Criteria
points: The translation accurately reflects the meaning of the original text, fluent, and contains no errors.
4 points: The translation generally reflects the meaning of the original text, with most sentences being fluent, but there are slight inaccuracies in the use of non-key terms or non-idiomatic phrases.
3 points: The translation conveys the general idea of the original text, but contains sig- nificant errors such as improper translation of key terms, incorrect word order, omissions, mistranslations, or untranslated segments.
points: The translation is largely incomprehensible or unfaithful to the original text, with serious errors including issues of order, logic, severe grammatical mistakes.
1 point: The translation is completely incomprehensible or entirely unfaithful to the orig- inal text, it fails to convey the original meaning entirely, being obscure and difficult to understand.
Please note that all sentences excerpts from web content, the last sentence of each segment, which may be unclear, not considered the evaluation.
We ensured adherence to ethical standards in our human annotation process:
Fair Compensation: All annotators received compensation at or above the minimum wage standards of their respective regions.
Informed Consent: Annotators were provided with clear instructions and information about
the annotation tasks. Participation was voluntary, and informed consent was obtained prior to their involvement.

---

Institutional Review: Our study underwent review and received approval from the Institu- tional Review Board (IRB) at our institution, ensuring that the research met ethical standards for studies involving human participants.
Transparency: Detailed information regarding the annotation are included in the supple- mentary materials to promote transparency and reproducibility.
A.5 Translation Quality Asseement
We assess the quality of our translations through human evaluation. Expert annotators are provided with 50 pairs of source and translated texts and asked to rate translation quality on a scale mentioned above. As shown in Figure the overall translation quality of GPT- 4o high, with most languages achieving average scores above 4. Japanese and Thai is comparatively lower,
Notably, performance though still above 3.5, suggesting acceptable translation quality for these languages.
To further evaluate the robustness of our framework with respect to translation quality, conducted a supplementary experiment using the open-sourced LLM Qwen 3-8B [57]. Qwen 3-8B achieves a FLORES chrF score of 56, which is lower than GPT-4o's score of 62 for the same language set. We employed both Qwen 3-8B and GPT-4o to translate our training preference pairs and trained separate MuRater models based on each translation. Each MuRater was then applied to score 10,000 multilingual documents per language. We compared the annotation outputs between the models using Pearson correlation and Kendall's Tau.
Figure 6: Translation average
scores of various languages.
Table 4: Agreement between MuRater models trained with translations from Qwen 3-8B vs. GPT-4o, measured by Kendall's Tau and Pearson correlation.
Language ja
de
cs
ar
pt
th
fr
vi
Kendall's Tau
0.8930 0.9835
0.9005 0,9854
0.8951
0.8645 0.9793
0.8903
0.8964
0.8834
0.8924 0.9831
0.8867
Pearson Corr.
0.9843
0.9822
0.9848
0,9803
0.9825
Language
it
ko
ms
r
tr
zh
nl
Kendall's Tau
0.8851 0.9815
0.8885
0.8427
0.8749
0.8942
0.8822
0.9044
0.8991
Pearson Corr.
0.9821
0.9644
0.9755
0.9850
0.9818
0.9861
0.9843
As shown in Table both metrics indicate consistently high agreement across all evaluated languages. These findings suggest that even when relying on a weaker translation model such as Qwen 3-8B, MuRater still be effectively trained, provided that the relative preference information is preserved. This demonstrates the robustness of our approach to moderate variations in translation quality.
A.6 Pointwise Score
The pointwise scoring prompt is provided below. We instruct GPT-4o to evaluate each text 10 times, then compute the average these scores to determine the final rating. The scoring range is from srade_min to grade_man 10.
25

---

Pointwise prompt evaluation for educational value
I need rate text excerpt on scale of (grade_ming mii] to (grade_man ______max (inclusive) based its educational value, e.g., it includes clear explanations, step-by-step reasoning, or questions and answers.
Aspects that should NOT influence your judgement: 1. Which language the text is written in
The length of the text
Note that the text is cut off, so you have to infer its context.
[Text] ... [text]
Now assign number grade between (grade m to (grade max (inclusive). Respond only with single digit. The score for the quality of the text is:
B Experiment Setup Details
B.1
Dataset
We use the 16 recent snapshots from FineWeb-2 as our raw data before MuRater and other base- lines annotation, namely CC-MAIN 2022202222222 2021-43, 2021-49, 2022-05, 2022-21, 2022-27 2022-33, 2022-40, 2022-49, 2023-000 10, 2023-23, 2023-40, 2023-50, 2024-10, and 2024-18.
To analyze domain composition, we employ NVIDIA's multilingual domain classifie to annotate the domain distribution of our dataset. Figures 7 illustrate the domain shifts before and after applying the MURATER-based selection. The results show that MURATER systematically prioritizes World Knowledge domains such as People and Society, Health, and Science, which are typically well-structured and rich in informational content- particularly beneficial for large language model pretraining. However, the resulting domain distributions vary across languages, primarily reflecting intrinsic differences in the domain composition of their respective source corpora.
We adopt the open-sourced FastText language identification model 277. which supports 176 languages and is widely used in large-scale multilingual data pipelines. For multilingual data selection, we retain the top 10% of highest-scoring documents per language from the raw corpus, preserving the natural data composition of Common Crawl and maintaining a distribution similar to full set of FebeW. The initial token ratio across languages is: ru (14.20 (14.2%)) es (12.835 83%), ja 19%) de (12. 19%). zh (9226)), fr (8. (8.98%), it (7 29%). pt (4.58%). (1. (1.51%), ko
nl (4.53%), , vi (3.21%) id (2.88%), ar (2.75%) tr (2. (2.20%), (1.41%), tl (0.) ms (0.02%)
We then apply temperature-basee sampling with pretraining practices [69]. The final token ratios used training are: ru (8.08%) es (7 (7.78%), ja (7.62%) de (7.62%), , zh (7.00%), fr (6. (33),%) , it (6.50%) pt (5. 71%), ml 70%) vi (7%07%) id (4.86%), ar (4.78%), tr (4. (2%52%), th (4. 15%), ko (4. 11%), tl (2.75%), (2.62%).
3.33, following standard multilingual
Figure Domain distribution of German corpus
26

---

2179
Figure 8: Domain distribution of France corpus
Figure 9: Domain distribution of Japanese corpus
cam aisbulan
Figure 10: Domain distribution of Chinese corpus
Figure 11: Domain distribution of Thai corpus
27

---

Model configuration
Values
Attention head
16
Layers Hiddent size
24
2048
Intermediate layer dimension
5504
maximum position embedding
4096
layer normalization epsilon
1 10
Training Hyperparameters
Values
Batch size
3072
Sequence length
4096
Optimizer
AdamW
Learning rate
4.3 x 10-1
Learning rate schedule
Cosine decay to 10% of inital value
Traning steps Precision
Varied based on the total token budget
training)
Table 5: Model configuration and Training Hyperparameters for pretraining LLms
B.2 Baselines
We follow the same annotation procedure for the English datasets of QuRater, AskLLM, DCLM, and Fib-Web-Edu as described in Appendix[A [A] For QuRater-M, we apply the same prompting approach (also detailed in Appendix (A) and instruct GPT-4o to annotate 300,000 multilingual pairs, focusing exclusively on content regardless of the language. We then fine-tune the multilingual QuRater baseline using both English and multilingual data, leveraging the BGE-M3 model (6 and the identical training hyperparameters outlined in Appendix
B.3
Model Architecture
We utilize a transformer architecture based on the LLaMA-2 model [59] configured to contain approximately 1.2 billion parameters. Models are randomly initialized before pretraining. The detailed information for the model contiguration and training hyperparamters is shown in Table We preprocess our training corpus to train a custom Byte-Pair Encoding (BPE) tokenizer using the BBPE algorithm, yielding vocabulary of 250,000 tokens for use in our training experiments. The main experiments is conducted using 64 NVIDIA H100 GPUs, with an average runtime of approximately 70 hours per experiment.
C Evaluation Benchmarks
All task evaluations are conducted using the am-evallationn-hrreeee framework in-context learning tasks, the following benchmakrs:
14] For English
ARC-Easy and ARC-Challenge [10] (25-shot): Multiple-choice science questions from grade school exams, assessing models' ability to apply scientific knowledge and reasoning.
(0-shot): Crowdsourced multiple-choiee science questions covering physies, chemistry, and biology, designed to evaluate scientific understanding.
SciQ (63]
(0-shot): Logical reasoning questions derived from Chinese civil service exams, testing deductive reasoning capabilities.
LogiQA 37
TriviaQA 26 (5-shot): Reading comprehension dataset with question-answer pairs au- thored by trivia enthusiasts, accompanied by evidence documents.
BoolQ (5-shot): Yes/no questions with associated passages, evaluating models' ability to answer naturally occurring questions.
For commonsense reasoning, we evaluate on:
HellaSwag 770 (10-shot): Sentence completion tasks requiring commonsense inference to select the most plausible continuation.
28

---

PIQA B (5-shot): Physical commonsense reasoning questions, focusing on everyday tasks and interactions.
OpenBookQA [40] (10-shot): Multiple-choice questions based on elementary science facts, requiring both factual knowledge and reasoning.
WinoGrande 53 (5-shot): Pronoun resolution tasks designed to test commonsense reason- ing at scale.
Additionally, two World Knowledge tasks are evaluated:
Natural Questions (NQ) 29 (5-shot): Real user questions paired with answers from Wikipedia, assessing open-domain question answering.
MMLU 23 (5-shot): A benchmark covering 57 subjects across various domains, measuring multitask language understanding.
For evaluating translated benchmarks, we use the MuBench dataset (21 and conduct evaluations across 18 languages present in our training set. In the multilingual setting, evaluate:
ARC-Easy and ARC-Challenge (25-shot): Translated versions of the science question benchmarks, assessing cross-lingual reasoning.
HellaSwag (10-shot): Evaluating commonsense reasoning in multiple languages through sentence completion tasks.
MMLU (5-shot): Multilingual evaluation of multitask language understanding across diverse subjects.
StoryCloze 41 (O-shot): Narrative understanding task where models choose the correct
ending to four-sentence story.
BMLAMA 50 (0-shot): Multilingual factual knowledge probing dataset, assessing cross-
lingual consistency in language models.
XCOPA 49 (5-shot): Causal commonsense reasoning tasks translated into multiple lan-
guages, evaluating cross-lingual inference.
XNLI [ (5-shot): Cross-lingual natural language inference benchmark, testing entailment and contradiction detection.
XWinograd [58
(5-shot): multilingual benchmark for evaluating localized knowledge and reasoning abilities of large language models across diverse languages.
MultiLoKo 2
(5-shot): large-scale multilingual evaluation suite designed to assess factual knowledge, reasoning. and question answering across 45 languages, emphasizing both cross-lingual consistency and language-sppciffe understanding.
FLORES [6] (5-shot): Multilingual machine translation benchmark, evaluating translation
quality across diverse languages.
MMLU L (5-shot): A localized version of MMLU, focusing on both general knowledge and language-specific knowledge and reasoning tasks.
D Detailed Results
D.1 English Detailed Results
Table 6 presents the detailed performance of various selection methods across individual downstream tasks. Our method consistently outperforms others on most tasks, with notable improvements on ARC, HellaSwag, and MMLU.
D.2 Multilingual Detailed Results
Detailed results of different data-selection methods across individual downstream benchmarks. For the 7B experiments, we additionally include the MultiLoKo benchmark (24 to assess cultural and regional knowledge across languages. Since its scores were too low be meaningful under the 1.2B training setup, we do not report the results here.
We also display the detailed results of each benchmark and each language below.
29

---

Table 6: Detailed performance differnt selection method over all downstream tasks with all values in percentages and per-benchmark maximum highlighted in bold.
Selection ARC_Chiie Challenge
Eacy
Bool))
HellaSwag
LogiüA
MMLLU
NQ
OpenllookAA
PRQS
TriaOA
Averoge
don))
57.85
75.75
Asklin
36.60
67.63
59.76
63.33
26.57
32,59
7.53
35.60
76.82
36.55
57.85
62.70
47.82
DCLM
40.44
13.78
64.07
62.42
25.73
35.42
0.31
37.40
76.06
38.01
60.06
57.00
50.23
FireWWth Ndu
40.10
72: 09
6462
59
76.83
96.01
7.96
38.20
76.37
1938
58.41
16-50
49.49
QuRazer
40.27
72:14
61.93
62.38
25.89
33.26
5.63
38.60
73.63
15.24
$7.50
55.80
48.33
73.84
99.31
Table 7: Performance of different deta-selection strategies across downstream tasks when mixing
200B English and 300B multilingual tokens. MuRater(MM) denotes scoring multilingual pairs translated into English, while MuRater(E)) uses rated English data translated into multilingual pair form. Best results within each setting are shown in bold.
Selection Method
MMLU (L)
C.MLL
ARC
E.MIL
ARC
FLORRS
Hella
ML
MMLU
(T)
XCOPA
XNLI
Story Close ML
XWino
BMLAMA
Average
Languages
28:74
HPLT:2
29.24
26.94
45:95
42.06
39:87
27.38
59.67
42.14
65.09
71.77
48:28
45.39
Fine Wib--
28. 11
27.50
45.07
44.
27.75
62.57
41.00
66.77
72.50
41.51
45.56
QulRater
30.86
33.89
57.53
06.50
46.77
29. 18
62.97
$1.39
65.86
71.21
45.84
48.65
MuRater(M)
31.86
34,26
58.21
47.43
46.54
29.
64.43
42.07
67.08
72.92
50.13
49.47
MuKateril))
35.01
58.96
29.40
50.43
Langwages Subset
Pine FmeeWeh2e00 490
30.52
30.91
54.37
50.97
46.15
28.66
64.92
41.79
69.48
68:72
43.08
48.14
MuRater(F)
31.96
36.03
61.63
50.11
49.17
29.43
67.44
44.49
68.75
68.33
53.19
$1.00
Table 8: Comparison of MuRater and QuRater-M when training 7B model IT tokens with 16.5% multilingual data.
MMLUU_L
ARC C.
ARC_E_M
FLORES
HSWAG
MMLUJT
XCUPA
S.
XWino
BMLAMA
MuliLLKa
Avg.
35,93
65.20 4333
69.27 71,3
74.50
5.87
51.54 $2.61
MuKaaee
36,87
66.37
55.38
57.76
32.95
75,42
10.1
Table 9: Detailed per-language performance on across A ARC-Fasy. Bold indicates the best result for each language.
Methed
AR
DF
EN
F8
FR
ID)
17
1A
K0
MS
NI.
PT
RU
TA
TH
TR
VI
7H
Uniform
42.21
53:07
65.57
56.40
40
53.85
52.00
52 86
49.07
46.44
45.62
51.43
54,17
50.67
37.88
19.44
46.72
48.44
$5.47
QuRaie-M
52.89
62:25
72.54
65.74
61.59
61 61.33
62: 71
57.62
53.
$4.70
50.44
63.51
59
42.85
43.90
55.72
54.67
63.72
MuRaaenM)
52.19
62.79
72.85
66.51
63.85
63.30
62.58
53.00
$3.96
55.89
61.70
63.47
59.85
43.35
45.75
56.40
56.19
53.76
MoRasenTa
52.82
63.22
73.91
67.55
63.97
63.68
63.80
58.85
$4.59
$6.78
51.62
65.24
60.98
44.53
45.50
57.28
$7.03
65.11
Table 10: Detailed per-language performance on across ARC-Challenge. Bold indicates the best result for each language.
TA
Uniform
27.82
30.03
30.33
23.92
25.
24 33
23.24
QuRaterM
MuRazenMi
30.20
36,95
41.13
39.25
35.75
35.07
35. 49
34.39
33.36
32.51
34.56
37.71
35.84
27.94
29.78
33.45
33,70
36.35
MoRasend)
31.91
36.09
42.06
39.08
37.29
36.18
38.57
35.67
33.45
36.01
3181
39.25
37.20
27.13
30.20
35,07
31.48
35.84
Table 11: Detailed per-language performance on across HellaSwag. Bold indicates the best result for each language.
Mathed
Uniform
QuRates
MuRtsenMi
MaRator(E)
40.48
42.62
Table 12: Detailed per-language performance across MMLU. Bold indicates the best result for
each language.
Methool
AR
DE
12
18
PR
10
1T
1A
KD
M8
M1.
PF
RU1
11
11.
12
V1
2H
Unifeem
0.1628
0.2065
0.3510
0.141
0.2208
Mi
0,3727
0.3557
0.3235
0.290
0.3018
6.300
0.2644
1.29/9
0.3019
0.2877
0.2665
0.2997
(12937
0.3757
0 0.2914
012541
01291:4
Maltarer:::
0,2765
8.3033
8.3066
8.2983
8.3018
022999
6.2M55
0.2936
0.1571
6.2925
0.2976
0.2088
0.1586
0.2813
0.2850
0.2968
0.2967
0.2049
30

---

Table 13: Detailed per per-language performance on across StoryCloze Bold indicates the best result for each language.
Methoa
Alt
EN
NL.
Unifoca
06366
0:7237
0.5M0
6,6958
06618
0.6463
6.7074
0.9967 0,5797
0.6865
0.7136 0.6811
QaRater M
06914
06912
0.729
0.627
0.7005
06703
96725
05633
0.5983
0.677]
06790
0,5912
0.6757
0,5259
0,5875
B.501
66607
6.989
63314
0.001
0.7089
0,5099
86901
04649
06345
0.855
06643
0.709
0,9944
8.393
0.5830
B8029
8.6811
Ba88s
MaltwardB
64231
67307
0583
Table 14: Detailed per- per-language performance on across BMLAMA. Bold indicates the best result for
each language.
Method AR
INN
PR
801
Unitorm
03128
04530
0.5148
0.531
0.4563
0.1890
0/4221
04402
0.2764
0.3835
0.403
0.4035
0.3391
0. 2999
0.5202
B. 1.505
B.5204
0.4587
GeRanor
0.3800
0.558)
0.533
0.5075
0.4860
04757
0.5076
04317
0.3585
0.605
0.186
0.3846
0 0.3431
0.9080
0.5971
0.393]
Maloaar
Table 15: Detailed per-language performance on across XCOPA. Bold indicates the best result for each language.
Method
IT
TH
Uniform
68.20
57.20
58,80
70.60
66.20
QuRater-M
65.20
65.00
56.00
58.40
67. :40
65.80
MuRater(M))
67.80
67.20
58.20
58.80
69.00
65.60
MuRater(EB)
69.00
68.20
57.20
60.20
70.20
69.60
Table 16: Detailed per-language performance on across XNLI. Bold indicates the best result for each language.
Method
AR
DE
EN
1:5
FR
RU
TH
13
V1
ZH
Unifonn
35.30
46.47
47.67
45.74
46.14
43.29
38.35
39.00
39.56
40.60
QuRater-M
37.11
47.61
49.40
47.71
49.33
46.99
37.87
43.78
41.93
41.93
MoRaterMM
35.74
44.34
46.79
44 50
47.15
44.14
38.39
39.60
38.80
41.24
MuRater())
34,86
48.84
5149
46.55
47.39
37.27
4399
41.77
43.13
Table 17: Detailed per-language performance on XWinograd. Bold indicates the best result for each language.
Method
EN
FR
JP
PT
RU
7H
Uniform
83,70
69.88
67.78
69.96
62.86
72.02
QuRater-M
77.12
66.27
66.21
66.54
60.32
63.49
MuRater(M)
78.54
65.06
67.47
67.30
86
67.86
MuRatertE)
80.22
69.88
66.32
71.48
65.71
68.25
Translation from English (EN TO ML)
Method
DE
TH
TL
ZH
Uniform
35.03
53.27
48.27
57.70
$7.95
47.76
53.94
48.94
58.22
44.17
28. 60
40.04
48.9 98.97
19.82
QuRater-M M
36.60
52.70
48.28
58.74
59.62
48.17
23.59
19.52
54,23
48.20
59.03
45.09
29.94
42.05
39.73
48.65
19.97
MuRater(M)
53.65
60.17
48.85
49.51
30.78
50.89
19.98
MuRater(E)
37.80
53.87
48.30
58.85
60.20
49.39
23.73
20.99
54.03
49.52
60.05
46.14
29.84
42.49
40.64
50.79
20.40
(b) Translation English (MI TO EN)
Method
AR
DE
ES
FR
ID
IT
JA
KO
MS
NL
PT
RU
TH
T1.
TR
VI
ZH
Uniform
52,14 61.41
54.46
61.71
57.93
55.28
42.48
41.73
56.63
54.41
64.62
54.96
45.81
49.27
45 40
$2.24
46.10
QuRater-M
51.14
60.40
53.63
61.47
57.81
55.68
42.13
41,04
55.82
53.77
64.34
53.30
44.44
47.39
46.58
51.23
44.62
MuRater(M)
52.39
60.86
54.26
62.64
57.77
56.21
42.47
42.47
56.71
54.26
64,80
55.00
45.65
48.84
46.17
52.53
45.40
MuRater(E)
52.63 60.93 54.03
61.72
57.98
56.00
42.25
41.48
56.12
54.23
64.59
54.55
46.01
47.97
47.03
$2.04
45.31
Table 18: Detailed per-language performance on FLORES. Bold indicates the best result for each
language.
Method
AMMLU
CMMLU
INDOMMLU
JMMLU
VLMU
Unikorm
0.2594
0.3175
0.3235
0.3079
0.2009
QuRaser-M
0.2659
D.339M
0.3197
0.2898
MuRaterM)
MuRatertE)
0.2714
0.3404
0.3489
0.3323
0.3048
Table 19: Detailed per- lar-lang performance on acTOSS MMLU-L, Bold indicates the best result per column.
31

---

D.3 Impact of Translation and Data Quality on Multilingual Performance
To further investigate translation and data quality impact on multilingual performance, we analyze two representative language pairs with comparable token ratios but differing translation quality: Japanese vs. Spanish and Thai vs. Korean. Human evaluation results reveal that Japanese and Thai translations receive approximately 0.5 lower average quality scores than their respective counterparts, Spanish and Korean. As detailed in Tables above, this discrepancy reflected in downstream performance, where Japanese and Thai consistently underperform across most multilingual benchmarks.
Table 20 presents the normalized top-10% 10% selection scores (relative to Chinese). These results show that Japanese and Korean data exhibit notably lower selection scores than Spanish and Thai, aligning with observed translation-quality trends.
Table 20: Normalized top- top-10% 10% document scores across languages (relative to Chinese).
Language
ja 0.743
de
id
pt
th
vi
Score
0.862
0.922
0.877
0.996
0.896
0.922
0,806
0.967
Language
ms 0.817
ko
zh
Score
0.690
0.877
0.843
0.940
0.941
1.000
0.862
These findings suggest that translation quality partially accounts for the observed performance gaps, yet it is not the sole determinant. Additional iectors-including the intrinsic quality of the source corpora, language-specifee tokenizer compression effects [15], and cross-lingual knowledge transfer dynamics likely contribute to the variation in multilingual model performance. Collectively, the results highlight that improving translation fidelity and ensuring balanced corpus quality are both critical for enhancing multilingual LLM training.
32

---

E
Case Study
We present examples from various languages exhibiting a range of quality scores. The results demonstrate that texts with higher scores tend to be more fluent and contain richer educational content, particularly in domains such health and science. Moreover, for texts with comparable scores, the quality remains consistent across different languages. This suggests that our MuRater model evaluates text quality in a language-agnosti manner, relying solely on the content rather than the language in which it is written.
Figure 12: Sampled training examples of Japanese with quality ratings at different score range
Seaen -1.00
-0.7
Score:1200
Figure 13: Sampled training examples of German with quality ratings at different score range
Figure 14: Sampled training examples of French with quality ratings at different score range
$core--3.00
Score--0.00
Score/1.72 1.72
Sporeca 18
Figure 15: Sampled training examples of Chinese with quality ratings at different score range
3core:3.0
Figure 16: Sampled training examples of Thai with quality ratings at different score range
33


Reviews:
[{"review_id": "oHjEkr7stM", "reviewer": "Reviewer_xazu", "summary": "In this paper, the authors proposed Murating, a scalable framework for multilingual LLM pretraining that transfers high-quality English data-selection signals to 17 target languages. Existing model-based data selection methods are primarily English-centric, lacking robust multilingual adaptation. MuRating uses unifying English Raters, Translation-Based Multilingual transfer and Cross-lingual regularization to achieve robust multilingual adaptation.", "questions": "Please see Weaknesses.", "limitations": "Many LLMs are included as raters in the proposed method, so the fairness of the comparisons is a concern.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n(1) The paper is well-written and easy to follow;\n\n(2) MuRating extends data selection to low-resource languages without requiring language-specific annotations;\n\n(3) Aggregating various English rates via pairwise comparisons mitigates individual rater biases.\n\n(4) Parallel text pairs enforce semantic alignment, reducing score divergence.\n\n(5) The experimental results are promising.\n\nWeaknesses:\n\n(1) The translation quality is heavily reliant on LLM translators.\n\n(2) How do the authors guarantee the translation quality, Did the authors use human verifiers? The authors should provide details and discussions on this aspect, as LLM translators are not very skilled in low-resource languages.\n\n(3) As MuRating utilizes multiple LLM as raters, are the comparisons unfair, the  authors should provide discussions on the fairness of the experimental comparison shown in this paper.", "was_revised": true, "final_justification": "All of my concers have been addressed, I hence decide to raise my score.", "rebuttal": "We appreciate the reviewers' positive feedback on the methodology and writing of our paper. In response to the provided comments, we have made clarifications and implemented several new improvements that further strengthen our work.\n\n## Weakness\n> **Q1.** The translation quality is heavily reliant on LLM translators\n\nThank you for the observation. While our method relies on LLMs for translation, we conducted a human evaluation to ensure translation quality, as detailed in Lines 292–304. For each language, we randomly sampled 50 source-translation pairs and had them rated on a 1–5 scale by human experts with CEFR C1-level proficiency or above in both the target language and English. A score of 3 indicates acceptable translation quality. The evaluation cost approximately $1200 (Appendix C.2, page 19). As shown in Figure 5, most languages received average scores above 4, while Japanese and Thai scored above 3.5, indicating consistently good translation quality.\n\nWe acknowledge the importance of maintaining high-quality translations and will continue to invest in human translation and evaluation to further improve translation quality. Additionally, we plan to open-source our annotated training dataset to encourage community participation and advance multilingual data selection for LLM pretraining.\n\n\n> **Q2.** How do the authors guarantee the translation quality, Did the authors use human verifiers?\n\nThank you for raising these important questions. As detailed in Lines 292–304, we conducted a human evaluation to assess translation quality. For each language, we randomly sampled 50 source-translation pairs and asked human experts to rate them on a 1–5 scale, where a score of 3 indicates acceptable quality. All evaluators held CEFR C1-level proficiency or above in both the target language and English. The evaluation cost approximately $1200, as noted in Appendix C.2 (page 19). As shown in Figure 5, most languages achieved average scores above 4, while Japanese and Thai scored above 3.5, indicating consistently good translation quality across all languages.\n\nWe are committed to further improving translation quality by increasing human expert involvement and plan to open-source our annotated training dataset. By engaging the research community in dataset construction, we aim to advance multilingual data selection practices for LLM pretraining.\n\nIn addition, we conducted a supplementary experiment using the open-sourced LLM Qwen 3-8B, which has a FLORES chrF score of 56—lower than GPT-4o’s score of 62 for the same language. We used both Qwen 3-8B and GPT-4o to translate our training preference pairs and trained separate MuRater models accordingly. Each MuRater was then used to score 10,000 multilingual documents per language. We compared the annotated scores between the two models using Pearson correlation and Kendall’s Tau. As shown below, both metrics indicate high agreement between the two annotation sets across all languages.\n\n| Language | ja     | de     | es     | ar     | id     | pt     | th     | fr     | vi     | it     | ko     | ms     | tl     | ru     | tr     | zh     | nl     |\n|----------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n| **Kendall’s Tau**        | 0.8930 | 0.9005 | 0.8951 | 0.8645 | 0.8903 | 0.8964 | 0.8834 | 0.8924 | 0.8867 | 0.8851 | 0.8885 | 0.8427 | 0.8749 | 0.8942 | 0.8822 | 0.9044 | 0.8991 |\n| **Pearson Correlation**  | 0.9835 | 0.9854 | 0.9843 | 0.9793 | 0.9822 | 0.9848 | 0.9803 | 0.9831 | 0.9825 | 0.9815 | 0.9821 | 0.9644 | 0.9755 | 0.9850 | 0.9818 | 0.9861 | 0.9843 |\n\nThese results suggest that even with a weaker translation model like Qwen 3-8B, as long as the relative preference between the training pairs are preserved, MuRater can still be effectively trained. This highlights the robustness of our approach to moderate variations in translation quality.\n\n\n> **Q3.** As MuRating utilizes multiple LLM as raters, are the comparisons unfair, the authors should provide discussions on the fairness of the experimental comparison shown in this paper.\n\nThank you for raising this important concern. We consider there may be a misunderstanding, and we would like to clarify that MuRating does **not** use multiple LLMs as raters during inference or data selection. As described in Lines 110–114 of Section 3.1, we use multiple LLMs **only during training data construction**, where they are asked to score and compare preference pairs. This results in a training dataset used to fine-tune a **single encoder-based LLM with a linear head**, referred to as **MuRater** (Lines 186–189).\n\nIn all experiments, the **single MuRater**—like other baseline raters—is used to score the same corpus and select the top 10% of documents. The selected data is then used to train a 1.2B LLM from scratch. This ensures a consistent evaluation setup across methods and a fair comparison. As shown in Tables 1 and 2, MuRater performs effectively for both multilingual and English data selection under these controlled and fair comparison settings.\n\nWe appreciate the reviewer’s comment and revised the experimental setup section to clearly emphasize that only one rater is used during selection to avoid further confusion."}, {"review_id": "1uMEs3Pu4K", "reviewer": "Reviewer_Vx9U", "summary": "This paper introduces MuRating, a scalable data selection framework that leverages high-quality English data signals to build a multilingual rater across 17 languages. The proposed method consist of multiple steps such as 1) Aggregating multiple English \"raters\" via pairwise comparisons to learn a unified document-quality score, 2) Transferring this judgement through translation data pairs to train a multilingual quality evaluator, then 3) applying the evaluator to web-crawled mutlingual texts to select high-quality data.\nIn the experiment, they pretrained a 1.2B parameter LLaMA model using the selected multilingual corpus. Results show substantial improvements over strong baselines like QuRater, Ask LLM and DCLM, particularly on knowledge-intensive tasks.\n\n- this paper is well motivated and mostly easy to follow.\n- Tables are too small to read. Please consider making it bigger in the camera-ready version.", "questions": "see the summary.", "limitations": "n/a", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strengths \n- the authors exploits English-quality signals and translation to bootstrap evaluators for low-resourced languages. This help offer a practical path for multilingual data curation.\n\nweaknesses\n- there could be selection bias. The proposed approach relies on English-originated quality criteria, so this may not capture culturally or linguistically relevant features in non-English language texts. Have you ever conducted some analysis in this line?\n- Experiments are limited to a 1.2B model. It might be still uncertain how the proposed approach scale to larger architectures or more diverse data settings. If you could run additional experiments with e.g., 7B model if the authors have enough computational resources, that'd be great.", "was_revised": true, "final_justification": "In response from the author, my questions/concerns are addressed. I increased my score to 4.", "rebuttal": "Thanks for your positive comments on our method and paper-writing. We have provided clarifications to your comments and made several updates that can improve the paper based on your suggestions.\n## Weakness\n> **Q1.** there could be selection bias. The proposed approach relies on English-originated quality criteria, so this may not capture culturally or linguistically relevant features in non-English language texts. Have you ever conducted some analysis in this line?\n\nThank you for highlighting this important concern. We would like to clarify that the motivation behind our method is to transfer general quality criteria—such as educational value, coherence, and fluency, which have been shown to be effective in English data selection [1,2], but are not inherently English-specific. Given the limited work on multilingual data selection, our goal is to establish a strong, general-purpose foundation as a first step toward broader multilingual data filtering.\n\nTo further address potential cultural or linguistic bias, we included a baseline called QuRater-M (Lines 205–206), which leverages a strong multilingual model, GPT-4o, to directly score preference pairs in their original languages. We consider this approach can capture culturally and linguistically specific features inherent to each language by leveraging the multilingual ability of GPT-4o. As shown in Table 1, our method outperforms QuRater-M on tasks such as MMLU-L, which includes culturally specific local questions, and FLORES, which tests linguistic knowledge through translation. In our additional 7B experiments (shown below), we further evaluate on the MultiLoKo [3] benchmark, which includes tasks that are locally relevant to specific languages as well as translation-based tasks. MuRater continues to perform strongly, suggesting that the general quality criteria we apply are also effective in capturing culturally and linguistically relevant content.\n\nWe fully agree that language-specific selection is important and consider it a key direction for future work, following our establishment of a solid general-purpose data selection criterion for multilingual data. We plan to explore language-specific criteria to better identify culturally and linguistically tailored content, aiming to advance multilingual language model pretraining in the future.\n\n> **Q2.** Experiments are limited to a 1.2B model. It might be still uncertain how the proposed approach scale to larger architectures or more diverse data settings. If you could run additional experiments with e.g., 7B model if the authors have enough computational resources, that'd be great.\n\nThank you for the valuable suggestion. To evaluate the robustness of our approach, we conducted additional experiments using a 7B model with an architecture similar to the 1.2B model discussed in the main paper. In this setting, we compared the performance of MuRater and baseline QuRater-M.\n\nThe model was trained on 1T tokens, with 16.5% multilingual data selected by each method. The remaining 83.5%, consisting of English, code, and math data, was kept identical across both setups. The language mix ratio for the multilingual portion follows the same distribution as in our main experiments.\n\nWe chose this setup—with a larger model architecture, a lower multilingual data ratio, and a more diverse mix of data types—to evaluate the robustness of our method under varied training conditions. As shown in the results below, MuRater consistently outperforms QuRater-M across most tasks, demonstrating its effectiveness in multilingual data selection when scaled to larger models and more diverse data.\n\nNotably, the strong performance on MMLU_L, FLORES, and MultiLoKo further validates that our general quality criteria are capable of capturing culturally and linguistically relevant content, thereby enhancing multilingual LLM performance.\n\nWe have included these results in our paper to strengthen the empirical support for MuRater. Additionally, we plan to open-source MuRater to support the research of multilingual data selection and encourage further evaluation and improvement by the broader research community.\n\n|              | MMLU_L| ARC_C| ARC_E| FLORES| HELLASWAG| MMLU_T| XCOPA | XNLI | STORYCLOZE | XWINOGRAD |BMLAMA| MultiLoKo |Average|\n|------------|-------|----------|----------|--------|------------|--------|--------|------|------------|----------|--------|---------|---------|\n| QuRater-M|35.93| 42.36  | 65.20  | 55.15  | 57.29      | 32.54  | 69.27  | **45.13** | 74.50   | 82.60 | 49.61  | 8.87       | 51.54   |\n| MuRater| **36.87**| **43.38**| **66.87**| **55.38**| **57.76**    | **32.93**| **71.03**| 45.06 | **75.42**      | **83.19**    | **52.82**| **10.61**  | **52.61** |\n\n> **Suggestion** Tables are too small to read. Please consider making it bigger in the camera-ready version.\n\nThank you for the helpful suggestion. We will adjust the table formatting and enlarge the content to improve readability in the camera-ready version.\n\n[1] Wettig, Alexander, et al. \"QuRating: Selecting High-Quality Data for Training Language Models.\" International Conference on Machine Learning. PMLR, 2024.\n[2] Li, Jeffrey, et al. \"Datacomp-lm: In search of the next generation of training sets for language models.\" Advances in Neural Information Processing Systems 37 (2024): 14200-14282.\n[3] Hupkes, Dieuwke, and Nikolay Bogoychev. \"MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages.\" arXiv preprint arXiv:2504.10356 (2025)."}, {"review_id": "CSjcBeDtS5", "reviewer": "Reviewer_erR2", "summary": "This work presents a novel method to curate high quality multilingual datasets based on transferring data-quality signals to other target languages. Multiple state-of-the-art English pairwise ratings are aggregated via a Bradley-Terry model into a single quality score. For the multilingual case the samples are translated to the respective languages and either re-rated or the rating is transferred. The data selection approach based on the aggregation of multiple quality scores shows promising performance in English-only as well as when transferred to the multilingual setup.", "questions": "- Could the authors please clarify the rationale behind the decision not to compare MuRatings performance with baselines such as HPLT-2, Fineweb-2 and Fineweb-HQ (for all the overlapping languages)?\n- Could the authors provide more detailed information about the multilingual corpus used for LLM training? For example, it would be beneficial to know:\n  - How was language identification performed?\n  - What is the language mixture used (number of tokens by language) for the multilingual experiments?\n  - How were the Fineweb filter adjusted for the multilingual context?\n- Given that the classifier training dataset of FineWeb-HQ reportedly includes resources like the Aya collection (an instruct-style dataset available in 101 languages), could the authors clarify the statement in line [95] regarding the method's reliance on the availability of high-quality samples of multilingual benchmarks? There seems to be a potential discrepancy that warrants further explanation.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n- The work presents a novel method to aggregate existing data quality raters in the English and multilingual context demonstrating promising performance. As high-quality multilingual data is lagging behind English data curation this novel method is an important contribution.\n- The work and experiments are generally well structured and easy to follow.\n\n**Weaknesses:**\n- The work mentions Fineweb-HQ in the introduction, which has a significant overlap in languages. To critically discuss the method's performance in the context of existing multilingual work, I would have expected the work to compare to this baseline or at least to other multilingual baseline dataset, such as HPLT-2 and FineWeb-2.\n- The work could be more detailed on the experimental setup, e.g. how was language identification performed or what is the language mixture used (number of tokens by language) for the multilingual experiments.\n- The results of section 4.3.2. Translation Quality Assessment, could be strengthened with discussion including how translation performance impacts data selection and  thus LLM performance.", "was_revised": true, "final_justification": "I thank the authors for the detailed rebuttal. They have addressed my concerns.", "rebuttal": "We appreciate the reviewers' positive feedback on both the proposed methodology and paper writing. We have provided clarifications to comments and made several updates that have improved the paper based on your suggestions.\n\n## Weaknesses and Questions\n\n> **Q1.** The work mentions Fineweb-HQ in the introduction, which has a significant overlap in languages. To critically discuss the method's performance in the context of existing multilingual work, I would have expected the work to compare to this baseline or at least to other multilingual baseline dataset, such as HPLT-2 and FineWeb-2.\n\nThank you for the insightful suggestion. We have conducted additional experiments comparing our method with other multilingual baseline datasets, including FineWeb-HQ, FineWeb-2, and HPLT-2. Following the primary experimental setup in our paper, we train a 1.2B LLM on 200B English tokens and 300B multilingual tokens, keeping the language mix ratios consistent with our main experiment.\n\nSince FineWeb-HQ does not cover several languages in our setting, we limit the comparison to the 13 overlapping languages. For the remaining languages, we use the same data across the experiment to ensure a fair comparison.\n\nAs shown in the results below, our method consistently outperforms all three baselines in multilingual data selection. Compared to FineWeb-2 and HPLT-2, our method achieves higher scores across all benchmarks and demonstrates stronger overall performance. Against FineWeb-HQ, our method outperforms it on nearly all benchmarks, with slightly lower results on FLORES and StoryCloze, which are narrative and translation tasks. We identify these as important future directions for improvement.\n\nThese additional comparisons have been included in the paper to further validate the effectiveness of our approach in selecting high-quality multilingual data for LLM pretraining.\n\n|                  |MMLU_L|ARC_C|ARC_E|FLORES|HELLASWAG|MMLU_T|XCOPA|XNLI|StoryCloze|XWINOGRAD|BMLAMA|AVERAGE|\n|------------------|------|--------|--------|------|----------|------|-----|----|-----------|----------|------|--------|\n|MuRater|**31.96**|**35.01**|**58.98**|**47.27**|**47.11**|**29.40**|**65.73**|**44.46**|**67.67**|**74.13**|**52.97**|**50.43**|\n|HPLT-2|29.24|26.94|45.95|42.66|39.87|27.58|59.67|42.14|65.09|71.77|48.38|45.39|\n|FineWeb-2 |28.33|27.50|45.97|44.52|42.39|27.75|62.57|41.00|66.77|72.89|41.51|45.56|\n|----------------------|\n|MuRater(13lang) |**31.96**|**36.03**|**61.63**|50.11|**49.17**|**29.43**|**67.44**|**44.49**|68.75|**68.83**|**53.19**|**51.00**|\n|FineWeb-HQ(13lang)|30.52|30.91|54.37|**50.97**|46.15|28.66|64.92|41.79|**69.48**|68.72|43.08|48.14|\n\n> **Q2.** The work could be more detailed on the experimental setup, e.g. how was language identification performed or what is the language mixture used (number of tokens by language) for the multilingual experiments.\n(1)How was language identification performed?\n(2)What is the language mixture used (number of tokens by language) for the multilingual experiments?\n(3)How were the Fineweb filter adjusted for the multilingual context?\n\nThank you for the helpful suggestion. We have added further details to clarify the experimental setup in the revised paper:\n\n**(1) Language Identification:**  We use the open-sourced FastText language identification model [1], which supports the detection of 176 languages and is widely adopted in large-scale multilingual data processing pipelines.\n\n**(2) Language Mixture:**  For multilingual data selection, we first select the top 10% highest-scoring documents for each language from our raw dataset. This results in a distribution closely aligned with the natural composition of Common Crawl and similar to that of FineWeb-2. The initial token distribution is:\n- ru (14.29%), es (12.83%), ja (12.19%), de (12.19%), zh (9.26%), fr (8.98%), it (7.29%), pt (4.58%), nl (4.53%), vi (3.21%), id (2.88%), ar (2.75%), tr (2.20%), th (1.51%), ko (1.41%), tl (0.04%), ms (0.02%)\n\nWe then apply temperature-based sampling with $\\tau=3.33$, following the common practice in multilingual LLM training [2]. The final token distribution used in our experiments is:\n- ru (8.08%), es (7.78%), ja (7.62%), de (7.62%), zh (7.00%), fr (6.93%), it (6.50%), pt (5.71%), nl (5.70%), vi (5.07%), id (4.86%), ar (4.78%), tr (4.52%), th (4.15%), ko (4.11%), tl (2.75%), ms (2.62%)\n\n**(3) FineWeb Filter Adjustments:**  \nWe adopt the filtering configuration of FineWeb-2, which applies heuristics such as thresholds on non alpha words ratio, minimum/maximum/average character lengths, and other quality signals. While the full configuration is available in the FineWeb-2 GitHub repository (not linked here due to NeurIPS 2025 anonymity policy), we adapt this setup to our own dataset for cost-efficiency reasons. This allows us to achieve comparable quality while keeping data processing and storage costs manageable at scale.\n\n> **Q3.** The results of section 4.3.2. Translation Quality Assessment, could be strengthened with discussion including how translation performance impacts data selection and thus LLM performance.\n\nThank you for the suggestion. To explore how translation quality impacts data selection and downstream LLM performance, we conducted a supplementary experiment using the open-sourced LLM Qwen 3-8B, which has a FLORES chrF score of 56 and lower than GPT-4o’s score, 62. We used both models to translate our preference training pairs and trained separate MuRater models.\n\nEach MuRater was used to score 10,000 multilingual documents per language. We then compared their annotation results using Pearson correlation and Kendall’s Tau. As shown below, the two models produce highly consistent annotations across all languages. This suggests that even with moderately degraded translation quality, MuRater can still be effectively trained—as long as the relative preference score of the training pairs is preserved—highlighting the robustness of our approach.\n\n|Language|ja|de|es|ar|id|pt|th|fr|vi|it|ko|ms|tl|ru|tr|zh|nl|\n|--------|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|\n|**Kendall’s Tau**|0.8930|0.9005|0.8951|0.8645|0.8903|0.8964|0.8834|0.8924|0.8867|0.8851|0.8885|0.8427|0.8749|0.8942|0.8822|0.9044|0.8991|\n|**Pearson Correlation**|0.9835|0.9854|0.9843|0.9793|0.9822|0.9848|0.9803|0.9831|0.9825|0.9815|0.9821|0.9644|0.9755|0.9850|0.9818|0.9861|0.9843|\n\n\nTo further analyze language-level variation, we focus on two language pairs with similar token ratios but differing translation quality: Japanese vs Spanish and Thai vs Korean. Human evaluations show that Japanese and Thai have ~0.5 lower translation scores than Spanish and Korean. As shown in Tables 6–13 (Appendix D.2), this is reflected in LLM performance—Japanese and Thai consistently underperform their counterparts across most benchmarks.\n\nWe also present the normalized score of selected data assigned by MuRater (relative to zh):\n|Language| ja| de | es | ar | id| pt | th|fr|vi|ms|tl|it|ko|ru|tr|zh|nl|\n|----------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|\n|Score|0.743|0.862|0.922|0.877|0.996|0.896|0.922|0.806|0.967|0.817|0.690|0.877|0.843|0.940|0.941|1.000|0.862|\n\nInterestingly, these results indicate that Japanese data tend to be of lower quality than Spanish, but Thai data have higher quality than Korean, which is supported by the uniform selection results in Tables 6–13 (Appendix D.2). The reverse performance between Thai and Korean suggest that while translation quality may influence data selection and LLM performance, other factors-such as corpus original quality, tokenizer compression ratio of different languages [3], and cross-lingual knowledge transfer [4]—also play significant roles.\n\nWe view this as an important direction for future work. We plan to further explore the reason of language performance imbalance from translation quality, tokenizer compression ratio and cross-lingual knowledge transfer. We will also continue to invest in human translation and evaluation to enhance the quality of multilingual data selection.\n\n> **Q4.** Given that the classifier training dataset of FineWeb-HQ reportedly includes resources like the Aya collection (an instruct-style dataset available in 101 languages), could the authors clarify the statement in line [95] regarding the method's reliance on the availability of high-quality samples of multilingual benchmarks? There seems to be a potential discrepancy that warrants further explanation.\n\nThank you for pointing this out. Our original statement was based on the observation that some multilingual benchmarks like MMMLU [5] cover only a limited set of languages (e.g., 14), leading us to consider that certain high-quality benchmarks were unavailable for many others. Upon further consideration, we agree that this limitation can be mitigated through high-quality translation. To avoid confusion, we have removed the statement from the paper.\n\nWe also recognize FineWeb-HQ as a valuable contribution to multilingual pretraining data selection. As demonstrated in our comparative experiments above, it shows strengths in narrative task such as StoryCloze and translate  task Flores. We appreciate its contributions and plan to explore how the strengths of both approaches can be combined in future work to advance multilingual data selection and support more effective LLM pretraining across languages.\n\n[1] Joulin, A., et al. \"FastText.zip: Compressing Text Classification Models.\" arXiv:1612.03651, 2016.\n[2] Xue, L., et al. \"mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer.\" NAACL 2021.\n[3] Goldman, O., et al. \"Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance.\" ACL Findings, 2024.\n[4] He, Y., et al. \"Scaling Laws for Multilingual Language Models.\" arXiv:2410.12883, 2024.\n[5] OpenAI. MMMLU: Multilingual Massive Multitask Language Understanding [Dataset]. Hugging Face. Accessed 31 July 2025."}]

Author rebuttal:

Meta-review: The paper proposes MuRating, a multilingual data selection framework that transfers quality signals from English raters into a unified multilingual evaluator across multiple languages. The method enables effective data curation for pretraining large language models, as validated by the empirical results. Reviewers highlighted the clarity of writing, the practical significance of addressing multilingual data curation, and the robustness of results to translation quality and model scale. Weaknesses include potential bias from English-originated quality signals and the limited coverage of cultural/linguistic features. During rebuttal, the authors provided new baselines, translation robustness studies, and new experiments, which satisfied most reviewer concerns.

Overall, this paper makes a meaningful and well-substantiated contribution to multilingual LLM pretraining. I recommend acceptance.
Decision: Accept (poster)