M-Prometheus: A Suite of Open Multilingual LLM Judges
Authors: José Pombal
Dongkeun Yoon
Patrick Fernandes
Ian Wu
Seungone Kim
Ricardo Rei
Graham Neubig
Andre Martins
Venue: COLM 2025 (2025)
Keywords: automatic evaluation
llm-as-a-judge
multilinguality
TL;DR: We introduce M-Prometheus, a suite of open-weight multilingual LLM judges ranging from 3B to 14B parameters. M-Prometheus models outperform state-of-the-art open LLM judges.

Abstract: Employing language models as evaluators of long-form output (LLM-as-a-Judge) has become the \textit{de facto} standard for automatic evaluation. However, most LLM judges have been optimized exclusively for English outputs, with strategies for enhancing judges' multilingual evaluation capabilities remaining largely unexplored in the current literature. This has created a disparity in the quality of automatic evaluation methods for other languages, ultimately hindering the development of models with better multilingual capabilities. To bridge this gap, we introduce M-Prometheus, a suite of open-weight LLM judges ranging from 3B to 14B parameters that can provide both direct assessment and pairwise comparison feedback on multilingual outputs. M-Prometheus models outperform state-of-the-art open LLM judges on multilingual reward benchmarks spanning more than 20 languages, as well as on literary machine translation evaluation covering 4 language pairs. Furthermore, we find M-Prometheus models can be used with quality-aware decoding methods to significantly improve generated outputs, showcasing their utility for the development of better multilingual models. Crucially, through extensive ablations, we identify key strategies for training an effective multilingual judge. Our findings highlight the significance of model size and base model selection, and the advantages of using natively multilingual data rather than translated data. We release our models, training dataset, and code to reproduce our experiments.

Published as a conference paper at COLM 2025
M-Prometheus: A Suite of Open Multilingual LLM Judges
José Pomball.23 Dongkeun Yoon", Patrick Fernandes Ian Wu Seungone Kim², Ricardo Rei', Graham Neubig' & André ET. Martins 123377
1 Unbabel, IInstituto de Telecomunicações
ilnstituto Superior Técnico, Universidade de Lisboa, *KAIST, SCMU "Independent Researcher, 'ELLIS Unit Lisbon pombal. josemaria@gmail. com
Abstract
The use of language models for automatically evaluating long-form text (LLM-as-a-judge) is becoming increasingly common, yet most LLM judges are optimized exclusively for English, with strategies for enhancing their multilingual evaluation capabilities remaining largely unexplored in the current literature. This has created a disparity in the quality of automatic evaluation methods for non-Engliih languages, ultimately hindering the development of models with better multilingual capabilities. To bridge this gap, introduce M-PROMETHEUS, suite of open-weight LLM judges ranging from 3B to 14B parameters that can provide both direct assess- ment and pairwise comparison feedback on multilingual outputs. M- PROMETHEUS models outperform state-of-the-art open LLM judges on multilingual reward benchmarks spanning more than 20 languages, as well as on literary machine translation (MT) evaluation covering 4 lan- guage pairs. Furthermore, M-PROMETHEUS models can be leveraged decoding time to significantly improve generated outputs across all 3 tested languages, showcasing their utility for the development of better multilin- gual models. Lastly, through extensive ablations, we identify the key factors for obtaining effective multilingual judge, including backbone model selection and training on synthetic multilingual feedback data instead of translated data. We release our models, training dataset, and code.*
1 Introduction
Automatic evaluation of large language models (LLMs) has become increasingly challenging, as the capabilities of LLMs are constantly expanding to encompass wider range of tasks. To address this challenge, a paradigm has emerged ("LL.Mas-a--uddge" where language models are used evaluators of long-form outputs (Zheng et al., 2023; Gu et al., 2024; Li et al., 2024a;b). In this paradigm, language model receives a query, one or two responses, and some evaluation criteria, and tasked with generating feedback about the quality of the response(s). Contrary to traditional automatic evaluation metrics that only output scalar score (e.g., BLEURT (Sellam al., 2020) and COMET (Rei al., 2020)), the feedback of judge is composed of text explaining the decision behind either a scalar output (direct assessment, DA), or a verdict on the best of two responses (pairwise comparison, PWC). The effectiveness of the ALM-as-a-judge paradigm has been demonstrated across broad range of tasks with proprietary and open models (Bavaresco et al., 2024; Zheng et al., 2023;
Arabic, Basque, Bengali, Catalan, Chinese, Czech, Dutch, English, French, Galician, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Turkish, Ukrainian, Swahili, Telugu, Thai, Vietnamese. "English-German, English-Chinese, German-English, and German-Chinese French, Chinese, and Hindi *Moeels and training data available Huggingface.

---

Published as a conference paper at COLM 2025
M-FROMETHEUL
Prometheus d
Heroule *
Multilingual Instruction
Multilingual Capabilities
Scoring Rubric
Direct Assessment
Multilingual Respanse(s)
Judgement
Pairwise Comparison
Multilinguaa Reference
[Optional)
N-PROMETHEUS
Reference-free Eval
Figure 1: M-PROMETHEUS is a suite of open-weight multilingual LLM judges capable of providing reference-basee based and reference-free direct assessment and pairwise feedback.
Koemi & Federmann, 2023), and systems trained specifically for evaluation (Kim al., 2023; 2024b; Deshpande et al., 2024; Doddapanení et al., 2024).
Simultaneously, sigmificant efforts have been dedicated building multilingual LLMs (i.e., LLMs that can perform tasks well in languages beyond English). Yet, research on effective strategies for training strong multilingual judges has lagged behind, with existing work focusing solely on English despite the myriad of language modeling use-cases in other languages. Those works that do investigate multilingual judggingg capabilities introduce models with significant limitations. The Hercule Judge LLM (Doddapanení et al., 2024), for example, does not support PWC, while GLIDER (Deshpande et al., 2024) is not trained to handle non-English languages and only inherits basic multilingual capabilities from the pretraining of its backbone model. These limitations stifle the development of better multi- lingual automatic evaluation methods which, in turn, hinders the development of stronger multilingual language models. To bridge this gap, we introduce M-PROMETHEUS, a suite of perf-pperfrrmnce multilingual judges with 3B, 7B, and 14B parameters. Using a recipe inspired by Prometheus 2 (Kim et al., 2024b), M-PROMETHEUS models are trained to provide both DA and PWC feedback on non-English outputs. We release the training datasets we built for this purpose, M-FEEDBACK COLLECTION and M-PREFERENCE COLLECTION.
We extensively evaluate our suite of models on set of multilingual benchmarks spanning 30 languages, achieving state-of-the-art performance for their respective sizes. Interestingly, we observe that M-PROMETHEUS models are particularly strong the evaluation of liter- ary machine translation challenging cross-lingual task where most translation-specific automatic metrics underperform (Zhang et al., 2024)-across language pairs. Further- more, propose extrinsic evaluation dimension directly linked to the practical utility of judges for model development: measuring how well a judge improves model outputs in non-English languages at inference time. Using best-of-n sampling (Song et al., 2024), where a judge selects the best output from candidate generations, we observe that direct assessments obtained with M-PROMETHEUS enhance model outputs across languages," 3 achieving up to an 80% win-rate against the original outputs on M-ArenaHaard (Dang al., 2024), multilingual extension of ArenaHard (Li et al., 2024c).
To better understand which strategies most effectively maximize multilingual evaluation performance, we conduct comprehensive series of ablations. Our findings reveal that using synthetic (rather than translated) multilingual training data is crucial, and that in- corporating machine translation evaluation data can transfer positively to other evaluation tasks. Additionally, both the choice backbone model for finetuníng and model scale strongly determine the size of performance gains. We hope that insights will guide the development of future, improved multilingual LLM judges. We release our models, training data, and the code required to reproduce our experiments.
2

---

Published as a conference paper at COLM 2025
2
Related Work
2.1
LLM-as-a-Judge
As language models become capable of solving increasingly complex tasks, automatic eval- uation of long-form outputs has shifted away from scalar metrics (e.g, BLEU (Papineni et al., 2002) and BLEURT (Sellam et al., 2020)) and towards using language models as generative evaluators (Zheng et al., 2023, LLM-as-a-Judge) These models have shown otate-of-the-art evaluation performance across range of tasks (Gu et al., 2024; Li et al., 2024a;b), including multilingual ones like machine translation (Kocmi & Federmann, 2023), multilingual safety evaluation (Üstün et al., 2024a), and multilingual instruction-folloiing (Dang et al., 2024). While many works leverage proprietary models, several efforts proposing open LLM judges have emerged (Kim et al., 2023; 2024b; Vu et al., 2024; Wang et al., 2024; Deshpande et al., 2024; Doddapanení et al., 2024); the training recipe in our work is inspired by Kim et al. (2024b) (Prometheus 2). However, little attention has been paid to the performance of open judge models outside of English. Deshpande al. (2024) show that their model, Glider, retains some multilingual capabilities from pretraining (by measuring performance on M-RewardBench), even though it was only finetuned for judging English outputs. That said, our more extensive evaluation suite shows that models trained with synthetic multilingual data outperform Glider. To the best of our knowledge, only Doddapanení et al. (2024), who introduce Hercule (a model trained on translated multilingual data for 6 languages), consider training a multilingual judge. There are few reliable open multilingual judges and little understanding of the factors behind judge finetuning that drive multilingual perfor- mance. We attempt to bridge both these gaps by releasing a strong suite of multilingual judges, and by dissecting the effects of our training recipe's individual components.
2.2 Multilingual Adaptation
While most existing work on LLMs has been centered around the English language, many recent works have emerged around building systems with better multilingual capabilities. These involve pretraining multilingual models from scratch (Üstün et al., 2024b; Dang et al., 2024; Martins et al., 2025), or finetuning pretrained models (Alves et al., 2024; Rei et al., 2024; Doddapaneni et al., 2024) for better performance on multilingual tasks; our work focuses on the latter. Although there are works exploring LLM judge performance on multilingual tasks, there exists (to the best of our knowledge) only one work that introduces a finetuned multilingual LLM judge: Hercule (Doddapanení et al, 2024). The Hercule approach involves finetuning a model on translated versions of the Feedback Collection (Kim et al., 2023), the direct assessment training dataset we also use. Hercule is trained to judge outputs in 6 languages German, French, Bengali, Telugu, Urdu, and Hindi. However, it can only receive reference outputs in English and produce direct assessments. Furthermore, Hercule was only evaluated on RECON, a test set introduced by the authors that is also based on translated data. Unlike Hercule, which was only tested on one translated benchmark, we evaluate on more diverse set of benchmarks and demonstrate that using translated data for training often does not lead to improved performance.
3 The M-PROMETHEUS Suite
M-PROMETHEUS models are finetuned from Qwen2.5-Instruc (Yang et al., 2024), and are trained to provide DA and PWC feedback in the same format as Prometheus 2 (Kim et al., 2024b) while being capable of receiving target instructions, model outputs, and references in non-English languages (see Appendix A.2 for examples of training instances). The rest of the prompt is in English, and M-PROM M-PROMETHEUS provides feedback in English by default, although it can be prompted to generate feedback in other languages." M-PROMETHEUS
We used the prometheus-ee codebase for training with the hyperparameters Appendix WWe do not evaluate the quality of the long-form feedback outside of English, only that of the DA and PWC judgemenss Training models instances with translated feedback yielded poor results.
3

---

Published as a conference paper at COLM 2025
English
MT Eval
Multilingual
English
100k
200k
language pairs
5 languages
languages
230k DA data
250k PWC data
Figure 2: Data distribution (in number of instances) the M-FEEDBACK COLLECTION (DA data) and M-PREFERENCE COLLECTION (PWC data) datasets. These datasets form the training data of M-PROMETHEUS.
models exhibit strong performance in more than 20 languages ($5), despite being trained on data in only 6 languages (English, French, Portuguese, Greek, Chinese, and Hindi).
3.1 Training Data
The backbones of our training data are Prometheus 2's Feedback and Preference Collections, which English DA and PWC datasets generated with GPT-4. Each instance contains a target instruction, one (DA) or two (PWC) candidate responses, reference response, rubric containing some evaluation criteria, long-form feedback evaluating the response(s), and final judgement. See Appendix A.1 for detailed description of these components.
We follow this format for the new data create, and add two sources of multilin- gual data: 1) synthetic (as opposed to translated) multilingual synthetic DA and PWC data and 2) DA machine translation (MT) evaluation data. We adapt the synthetic data generation processes of Prometheus and Prometheus 2, although unlike for Prrometheus, we use 3laude-Sonnet-3.5 (Sonnet) instead of GPT-4 or GPT-4o as our data generator, as we find in preliminary experiments that Sonnet generates more fluent data in non- non-Enggish languages. The final data distribution is summarized in Figure 2. Due their length, we include concrete training examples in Appendix A.2.
Generating M-FEEDBACK COLLECTION. We start by generating our multilingual direct assessment dataset, M-FEEDBACK COLLECTION. Using the original 1k score rubrics from the Prometheus Feedback Collection, we prompt Sonnet to generate five instructions for each rubric in each of the five non-Englist languages we consider. For each instruction, we then prompt Sonnet to generate five candidate responses with varying levels of quality, each corresponding to score from 1 to 5, accompanied by long-form feedback in English. We also prompt Sonnet to generate a reference, high-quality response to half of our generated instructions. Each response is then combined with its corresponding rubric, instruction and reference response (should this exist) to form a single training input, and each training input paired with the concatenation of its corresponding feedback and score, which serves as the training target. By including a mix of samples with and without reference responses, our dataset enables the training of evaluators capable of both reference-free and reference-based evaluation.
Generating M-PREFERENCE COLLECTION. Next, synthesize the pairwise comparison dataset, M-PREFERENCE COLLECTION. From the aforementioned M-FEEDBACK COLLEC- TION and within each instance, we create "preference pairs" by pairing score responses with every other response, and score 4 responses with score 2 responses, resulting in five response pairs per instruction. Following Prometheus 2, we assume that the higher-scoring response is of higher quality and should therefore be preferred over the lower-scoring one. Then, for each pair, we prompt Sonnet to generate long-form preference feedback English. These components are then combined in a similar manner to our DA data, yielding again total of 10k samples for each language. For each instance, we randomize the order which the correct answer appears, i.e., it will appear first 50% of the time. For further details the data construction process, refer the Prometheus (Kim et al., 2023) and Prometheus' (Kim et al., 2024a) papers.

---

Published as a conference paper at COLM 2025
MT Evaluation Data. We augment M-FEEDBACK COLLECTION with MT evaluation data. For each of eight language pairs, 7 we prompt texts, conditioning on a topic, subtopic, and other attributes sampled from a common pool (we include the prompt we used and attribute prevalences in Appendix A.3)). Then, for each source, we prompt Sonnet to generate five candidate translations corresponding to scores (worst) to 5 (best), along with a reference translation. Each candidate translation is paired with their corresponding source, yielding a total of 80,000 instances. Finally, we randomly include reference translations for half of the training instances while omitting them from the other half. This enables models trained on our datasets to perform both reference-basee and reference-free evaluation, increasing their versatility; indeed, M-PROMETHEUS attains state-of-the-art performance on reference-less literary MT evaluation ($5).
to generate 2,000 source
Experimental Setup
4.1 Evaluating General Capabilities
The term "general capabilities" often used to refer to the real-world utility of language models in addressing queries that involve core knowledge, safety, instruction and conversational capabilities (Zheng et al., 2023). Evaluating LLM judges in this domain is useful as it indicates their effectiveness at judging the real-world utility of other models. The most popular English-only benchmark for this is RewardBench (Lambert et al., 2024). RewardBench is composed of 3,000 instances across 4 tasks (Chat, Chat Hard, Reasoning, Safety), where the judge is tasked with choosing the best of two answers to a query. We evaluate all models on this benchmark to assess whether they retain English capabilities. To assess general multilingual capabilities, we use M-RewardBench (Gureja et al., 2024), which is a translated version of Rench for 23 languages. We also evaluate our models on MM-Eval (Son et al., 2024), a PWC benchmark that covers up to 18 languages in the categories of Chat, Reasoning, Safety, and two additional language-specific categories: 1) linguistics (e.g. find the homophones of word); 2) language hallucination, where judge is tasked with finding the model answer that mixes two or more languages undesirably. Importantly, MM-Eval is mostly comprised of native speaker, rather than translated, data, whih is an advantage over Benehh. The meta-evaluation metric of all three bench- marks is accuracy. We report the average of the per-category performance for Rench. For the multilingual benchmarks, we first obtain the micro-average performance on each language, and then report the average across all languages. Detailed results by category and language can be found in Appendix C.
4.2 Machine Translation Evaluation
Machine translation has played a key role in advancing language model development (most notably inspiring the Transformer architecture (Vaswani et al., 2017)) and has led to the creation of multiple automatic evaluation metrics that correlate well with human judgments (Freitag et al., 2024). However, most translation metrics still struggle in certain domains. A notable example is the translation of books, known literary MT, where existing metrics underperform because of the wide context window required to handle book excerpts, among other challenges. GEMBA-MQM (Kocmi & Federmann, 2023), an LLM judge based on GPT-4, has been shown to excel at this task (Zhang et al., 2024), while the performance of Prometheus 2, an open-source LLM judge, is close to random. We take interest in this task for two reasons: 1) we posit that training on a cross-lingual evaluation task may transfer positively to general-purpose multilingual evaluation capabilities; 2) we wish to bridge the performance gap between closed and open models. Thus, we leverage the student-annotater evaluated automatic translations of book excerpts for 4 language pairs: Einlish-+German Boglish-+Chinsse German--Englin and German-+Chinese On this task, judges are prompted to give scalar assessment of each translation (without access to a reference). The
subset of LitEval-Corpus (Zhang et al., 2024), which contains human-
We select language pairs of varying resource availabilities and scripts: Fnglish-German, -Czech, Spanish, -Ukraimian, -Russian, -Chinese, -Japanese, -Hindi
5

---

Published as a conference paper at COLM 2025
resulting ranking of translations is then compared to human ranking through Kendall's Tau correlation coefficient (Kendall, 1938).
4.3 Extrinsic Evaluation with Quality-Aware Decoding
The intrinsic meta-evaluation of judges through existing benchmarks is not directly informa- tive of their capacity to improve other models. To bridge this gap, we propose an extrinsic dimension of evaluation: evaluating judges on their ability to improve the multilingual outputs of other models. This is relevant for practical use-cases, such as for improving outputs at inference time (Fernandes et al., 2022; Wu et al., 2024), or for improving training datasets through distillation (Finkelstein & Freitag, 2024; Wu et al., 2024). Thus, we perform quality-aware decoding (Fernandes et al., 2022, QAD) with judges to improve the outputs of Qwen2.5 2ann..5-3--nnstrue on M-ArenaHard on languages: French, Chinese, and Hindi, M-ArenaHar lard is translated version of ArenaHard, a benchmark for general capabilities where models are prompted to generate long-form answers to 500 queries sampled from Chatbot Arena (Chiang et al., 2024). These answers are then evaluated against reference answer by an LLM judge,' yielding an Elo score based on win-rate. We evaluate judges the extent to which they improve the Elo of OWenn.5-3BB-nsttree vert Elo scores into expected win rates over the original outputs (generated with greedy decoding)-with 50% indicating that the judge is, on average, unable to improve output quality-and report the average across the three languages (we report language-specific results in Appendix C.5). We refer to this evaluation as "QAD".
after QAD. 10 We con-
4.4 Baselines
We compare M-PROMETHEUS against two types of baselines: 1) general-purpose LLMs, namely: 4pt-40-2024-11- 11-22 (GPT-40), a state-of-the-art proprietary LLM, and Qwen2.5- the backbone models of our suite and state-of-the-art open models for their sizes; 2) state-of-the-art open LLM judges, namely Prometheus 2 7B and 8x7B (Kim et al., 2024b), Glider 3B (Deshpande et al., 2024), and Hercule 7B (Doddapaneni et al., 2024). Hercule was trained specifically to evaluate non-English targets. There are multiple Hercule models (each trained with data from one of 6 languages), but not for all languages of the benchmarks we consider. To make this baseline more challenging, we evaluate all models and consider only the performance of the best one when languages are not supported. We run all models locally using benchmark codebases where available. 11
Experimental Results
M-PROMETHEUS outperforms open judges, much larger models, and GPT-40. Our main results are documented in Table 1. We find that M-PROMETHEUS excels on all axes of evalua- tion, surpassing all baselines on MM-Eval, literary MT, and QAD. M-PROMETHEUS 14B sur- passing GPT-40 on MM-Eval is particularly impressive, since the latter is a state-of-the-art LLM. Interestingly, while Qwen2.5-Instruct models perform strongly across general-purpose benchmarks (even outperforming most specialized judges), they lag behind on literary MT and QAD. Here, the benefits of finetuning are clear, especially for M-PROMETHEUS-SB. which outperforms its backbone model across the board and larger backbones on QAD.
SIn QAD, for each judge and test instance, we perform best-of-n sampling (Song et al., 2024) over 30 candidate answers, generated through temperature sampling (with temperature equal to 0.3). Each candidate is assigned a score by prompting the judge for a direct assessment, and the candidate with the best score is selected; if multiple candidates tie, we pick one at random. answers as reference answers and 1amma3.3-70B--mme
'We use Dween25-72B-Innstre et al., 2024) for evaluation.
(Grattafiori
101o validate whether our findings generalize to other models, we present results with Gemma-2-
2B-IT (Team et al., 2024) in Appendix D, the conclusions are similar.
Glidde and Hercule required minor code changes, which will release upon publication.

---

Published as a conference paper at COLM 2025
General-purpose benchmarks
LitEval QAD
Judge LLM
MM-Eval M-RewardBench RewardBench
Proprietary Models GPT-40
0.7185
0.8575
0.8596
0.3944
Small parameters)
Qwen2.5-3B-Instruct
0.5794
0.6674
0.6940
0.1538
54.29
Glider +
0.5746
0.7046
0.6827
0.1781
57.21
M-PROMETHEUS 3B
0.6380
0.6831
0.7027
0.4075
63.04
Medium (7B parameters)
Qwen2.5-7B-Instruue
0.6608
0.7801
0.7823
0.1772
55.88
PROMETHEUS 2 7B+
0.6090
0.6731
0.7205
0.1252
62.55
Hercule
0.4916
0.6508
0.6786
0.3516
64.86
M-PROMETHEUS 7B
0.6966
0.7754
0.7684
0.4353
66.37
Large (14B+ parameters)
Dwen2.5-14BBInntrree
0.6819
0.8081
0.8241
0.3108
54.63
PROMETHEUS 2 8x7B
0.6434
0.7515
0.7406
0.3185
62.79
M-PROMETHEUS 14B*
0.7726
0.7951
0.7967
0.4790
64.41
Table 1: Accuracy on general-purpose benchmarks, ranking correlation on LitEval, and win-rate on M-ArenaHard. For each column, underlined models the best for their size, while bold ones are the best overall. The denotes finetuned English judges, while denotes finetuned multilingual judges. The rows of our models shaded light purple.
M-PROMETHELS
Qwen2.5-Instruce
Chat
Language Hallu.
Linguistics
Reasoning
Safety
0.80
0.90
0.80
0.75
Accuracy
0.75
0.80
0.75
0.70
0.85
0.70
0.70
0.70
0.65
0.70
0.65
0.60
0.65
0.60
0.55
0.60
0.50
0.60
0.55
0.40
3B 7B 14B
3B 7B 14B
3B 7B 14B
3B 7B 14B Model size
3B 7B 14B
Model size
Model size
Model size
Model size
(# parameters)
(# parameters)
(# parameters)
(# parameters)
(# parameters)
(a) MM-Eval
Chat
Chat Hard
Reasoning
Safety
0.62
Accuracy
0.95
0.57
0.90
0.90
0.90
0.52
0.80
0.80
0.85
0.47
0.70
0.70
0.42
0.60
0.60
3B
7B
14B
3B
7B
14B
3B
7B
14B
3B
7B
14B
Model size
Model size
Model size
Model size
(# parameters)
(# parameters)
(# parameters)
(# parameters)
(b) M-RewardBench
Figure 3: Performance of the M-PROMETHEUS and Qwen2.5-Instruct 3B, 7B, and 14B models on general-purpose multilingual benchmarks, broken down by category. Tables with more detailed results are in Appendix C.
The categories that drive average performance on general-purpose benchmarks vary between M-PRRMMETHEUS and their backbone models. Figure illustrates the perfor- mance of all M-PROMETHEUS and Qwenn.5-Instruct models on M-RewardBench and MM- Eval broken down by category, revealing their strengths and weaknesses. We find that M-PROMETHRUS is particularly strong the Safety, and Language Hallucinations cate-
7

---

Published as a conference paper at COLM 2025
General-purpose benchmarks
LitEval QAD
Ablations
MM-Eval M-RewardBench RewardBench
No Judge Training
Misstal--BBvv02-Insttte
0.5031
0.5932
0.6481
0.0958
53.56
EuroLLM-9B
0.5834
0.6288
0.6890
0.0319
55.15
Aya-Expanse-8B
0.5143
0.6332
0.6579
0.0008
52.05
Qwen2.5-7B-Instruct
0.6608
0.7801
0.7823
0.1772
55.88
Backbone Model
Mistral--7B-v0.2-Imstte
0.5428
0.6454
0.7083
0.0747
61.81
EuroILLM-9I-Instruct
0.6263
0.7248
0.7519
0.2435
63.15
Aya-Expanse-SB
0.5904
0.7325
0.7531
0.2544
60.54
Qwen2.5-7B-Instrucc
0.6456
0.7817
0.7774
0.2837
61.36
Training Data
MT Eval Data
0.6748
0.7800
0.7780
0.4221
59.71
Translated Data
3 Non-English Langs
0.6280
0.7824
0.7768
0,2221
66.47
Multilingual Data
Non-English Langs
0.6477
0.7687
0.7855
0.3162
68.70
5 Non-English Langs
0.6616
0.7758
0.7876
0.3372
66.11
Table 2: Ablations of the M-PROMETHEUS training recipe, and results of instruct Models without any finetuning. For each evaluation method, bold models are the best in their respective ablation category (i.e,, backbone model or training data). The training data ablations are all done on Qwen2.5-7B-Instruu backbone.
gories, while the general- backbones excel on Chat. We report per-category and per-language performances for all models and ablations on both benchmarks in Appendix C.
M-PROMETHEUS models retain or improve performance in English. Remarkably, Table 1 demonstrates that M-PROMETHEUS models not only exhibit strong multilingual capabilities but also maintain nearly the same performance in English, measured by Reward compared their backbone models. Notably, M-PROMETHEUS- outperforms its backbone on this benchmark, further highlighting the benefits of fine-tuning for smaller models.
Multilingual training strongly improves performance on Literary MT and QAD M- PROMETHEUS models consistently outperform models of all sizes on Literary MT and QAD (see Table 1). On QAD, judges trained on multilingual data (M-PROMETHEUS and Hercule) exhibit particularly strong performance. These results suggest that multilingual training is important for endowing judges with the capacity improve the outputs of other models.
Dissecting the Training Recipe
6.1
Overview
One of our primary objectives is to develop better intuitions for how multilingual LLM judges should be trained. As such, we ablate three central components of our training recipe: 1) backbone model choice; 2) training data mix; 3) model size. size.
Backbone model ablations. In line with prior work (Kim et al., 2024b; Deshpande et al., 2024; Doddapaneni et al., 2024), we focus on specializing instruction-tunae backbone mod- els for the tasks of DA and PWC evaluation. To isolate the effect of backbone model
8

---

Published as a conference paper at COLM 2025
choice multilingual performance, apply the Prometheus training recipe to 4 models: Mistral-v0.2-nnstruect (Jiang et al., 2023), the backbone of Prometheus 2; EuroLLM- 9B-Instruct (Martins et al., 2025) and Aya-Expanse-88-nna the backbone of M-PROMETHEUS. For
(Dang et al., 2024), two highly
multilingual models; and Qwen2.5-7B-Instrue additional context, we also evaluate the backbone models before any finetuning.
Training data ablations. We are interested in answering three questions: does training for MT evaluation, a cross-lingual task, transfer positively to general-purpose multilin- gual evaluation capabilities? 2) does including translated data lead to better multilingual capabilities, as reported by Hercule (Doddapaneni et al., 2024), or is it better to train on multilingual data generated from scratch? 3) does covering more languages during training benefit overall performance? For the ablations with MT evaluation and synthetic multi- lingual data, we append each of our datasets described Prometheus 2, and train Qwwen.5-7B-Instrruc with the hyperparameters of Prometheus 2. We experiment with including or languages in the multilingual data mix (each language contains 10k DA and 10k PWC instances). For the translated data ablation, we translate the data of Prometheus 2 into 3 languages's using Tower-v2 (Rei et al., 2024), state-of-the-arr translation LLM (Koemi et al., 2024). We also include 10k DA and 10k PWC instances, 50% with reference, 50% without, for the sake of comparability with the synthetic multilingual data ablation. 14
Section 3.1 to the data mix of
6.2
Key Takeaways
The main results of our ablations can be found in Table 2.
Backbone model choice core driver of judge performance. With the exception of QAD, backbone model choice is the main driver of performance, representing up to 14 accuracy points in improvement on general-purpose benchmarks when switching from Mistral, the backbone model of o Prometheus to Qwen, the backbone of M-PROMETHED This finding may be partially explained by looking at results prior to any finetuning: Qwen outperforms all other backbones across the board. Interestingly, using models where non- English data is relatively more represented in pretraining, like EuroLLM or Aya, does not necessarily translate into better performnace.
MT evaluation capabilities transfers positively to general capabilities and vice-versa. As expected, training on MT evaluation data leads to better literary MT evaluation performance. More importantly, adding this kind of cross-lingual signal during training leads to improve- ments on general-purpose multilingual benchmarks. Upon closer inspection, we see that most of the gains on MM-Eval, for example, come from the language hallucination task, suggesting that MT evaluation data endows judges with greater ability to detect instances where languages are mixed together (see Appendix C.1 for per-category results). Likewise, training on synthetic multilingual data improves MT' evaluation performance.
Judges trained on synthetic multilingual data are the most capable of improving mul- tilingual outputs. The LLM trained on synthetic multilingual data from 3 Jon-English languages demonstrate the best performance on QAD, surpassing judges trained on other types of data by up to 10 points. This suggests that synthetic multilingual data is crucial for enabling judge models to improve the outputs of other multilingual models at inference time. Interestingly, increasing language coverage to 5 languages deteriorates the model's performance along this axis, but improves it on all the others.
F2ror simplicity, we p joint training the Feedback and Preference collections, opposed to merging models trained separately on each.
French, Portuguese, and Chinese for the translated and synthetic multilingual add Greek and Hindi, as per our final recipe. As with the synthetic multilingual data, each multilingual instance has non- non-English instructions, model outputs, and reference answers, while the rest of the instance remain in English. We experiment with translating the rest of the instance and reach similar conclusions.
19The 3 languages
data. When expanding the latter to languages,

---

Published as a conference paper at COLM 2025
Training translated data is not as effective synthetic multilingual data. With the exception of M-RewardBench,, adding synthetic multilingual data to training is always more effective than adding translated data. In fact, training on the latter leads to deterioration RewardBench, MM-Eval, and LitEval compared with training on English-only data. This somewhat contradicts the findings of Doddapaneni et al. (2024); we suspect translated data worked well in their case because they focus their evaluation on a translated DA dataset.
Model scale most strongly impacts general-purpose benchmark performance. Looking back at Table 1, we see that the impact of model scale is most noticeable on general-purpose benchmarks and on literary MT evaluation. Interestingly, however, M-PROMETTHEUS-76 outperforms M-PROMETHEUS--14 on QAD. Furthermore, the largest performance gap between M-PROMRTHEUS models and their respective backbones occur at the 3B size. These findings hint at the diminishing returns of fine-tuning scale increases.
7 Conclusion
We introduce and release suite of multilingual LLM judges (3B, 7B, and 14B) that demon- strate state-of-the-art performance on more than 20 non-English languages. Our training recipe mixes maathheticallygeneratte
MT evaluation data and synthetic-as opposed to data with existing English judge data. We justify our choices through extensive ablations, and further highlight the importance of backbone model choice and the ineffectiveness of translated data. We also propose an additional dimension of neta-evaluation that focuses on the practical usefulness of judges in improving multilingual outputs. the future, we hope to explore different strategies to improve multilingual judge capabilities (e.g., through training on multilingual reasoning chains) and extend existing ones (e.g,, by learning to produce high-quality feedback in non- non-English languages).
cunslated-mulliiing
Acknowledgements
We acknowledge EuroHPC JU for awarding the project ID EHPC-A1-2024A011-05 access to MareNostrum 5 ACC. This work was supported by EU's Horizon Europe Research and Innovation Actions (UTTER, contract 101070631), by the project DECOLLAGE (ERC- 2022-CoG 101088763), by the Portuguese Recovery and Resilience Plan through project C64500888200000055 (Center for Responsible AI), and by Fundação para a Ciência Tec- nologia through contract UIDB/50008 /2020.
Reproducibility Statement
We release all our models, training data, and code to reproduce our experiments. Part of our experiments rely on closed models, which may become unavailable the future, posing potential challenge for reproducibility.
Ethics Statement
Our work focuses on developing better automatic evaluation methods for non-English languages. First, the models release may show biases present in the data they were trained on. Users should carefully review model outputs before deployment. Second, automated evaluation could be misused to claim superiority without proper validation. We emphasize that our models should complement, not replace, careful human evaluation and real-world testing. We release our models, data, and code enable scrutiny and improvement by the research community.
References
Duarte M Alves, José Pombal, Nuno M Guerreiro, Pedro H Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, et al. Tower: An
10

---

Published as a conference paper at COLM 2025
open multilingual large language model for translation--elltted tasks. arxiv preprint ar arXio:2402.17733, 2024.
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, et al. Lims instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. arXiv preprint ArXiv:2406.11 18403, 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and lon Stoica. Chatbot arena: An open platform for evaluating Ilms by human preference, 2024.
John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, et al. Aya expanse: Combining research breakthroughs for new multilingual frontier. arv preprint ar Xii::4412.04261 2024
Darshan Deshpande, Selvan Sunitha Ravi, Sky CH-Wang, Bartosz Mielczarek, Anand Kannappan, and Rebecca Qian. Glider: Grading Ilm interactions and decisions using explainable ranking, Xiv preprint axXio:2412.14110,, 2024.
Sumanth Doddapaneni, Mohammed Safi Ur Rahman Khan, Dilip Venkatesh, Raj Dabre, Anoop Kunchukuttan, and Mitesh M Khapra. Cross-lingual auto evaluation for assessing multilingual llms. arXiv preprint arXio:2410.13394 2024.
Patrick Fernandes, António Farinhas, Ricardo Rei, José GC de Souza, Perez Ogayo, Graham Neubig, and André FT Martins. Quality-aware decoding for neural machine translation. arviv preprint arXto:2255.00973 2022.
Mara Finkelstein and Markus Freitag. Mbr and qe finetuning: Training-time distillation of the best and most expensive decoding methods. In The Twelfth International Conference on Learning Representations, 2024.
Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frédéric Blain, Tom Kocmi, Jiayi Wang, et al. Are Iims breaking mt metrics? results of the wmt24 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pp. 47-81, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint arXio:2407.2177 217883,, 2024.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. survey on llm-as-a-judge. arXiv preprint arXio:2411.1553 2024.
Srishti Gureja, Lester James V Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Winata, Nathan Lambert, Sebastian Ruder, Sara Hooker, and Marzieh Fadaee. M-rewardbench: Evaluating reward models in multilingual settings. CoRR, 2024.
Albert Q Jiang, A Sablayrolles, A Mensch, C. Bamford, D Singh Chaplot, Ddl Casas, Bressand, G Lengyel, G Lample, L Saulnier, et al. Mistral 7b. arxiv. arXiv preprint ar xrXie:2310.06825, 10, 2023.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1-2):811993 1938.
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. Prometheus: Inducing fine- grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, 2023.
11

---

Published as a conference paper at COLM 2025
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceadings of the 2024 Conference Empirical Methods in Natural Language Processing, pp- 4334 4353,, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10.18653/v122024 emnlp-main.248. URL https://aclanthhology org/ emnlp-main.
main. 248.
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. arviv preprint arXiv:2405.01535, 2024b.
Tom Kocmi and Christian Federmann. Gemba-mqm: Detecting translation quality error spans with gpt-4. In Proceedings of the Lighth Conference on Machine Translation, pp. 768-775, 2023.
Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondiej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grund- kiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popović, Mariya Shma- tova, Steinthór Steingrimsson, and Vilém Zouhar. Findings of the WMT24 general machine translation shared task: The LLM era is here but MT is not solved yet. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference on Machine Translation, pp. 1-46, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1l 100/33//11224.mmmm1111 URL httpa://aclanthology) orrelloo..oooo//// org/2024. umt-1.1.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khy- athi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, al. Reward- bench: Evaluating reward models for language modeling. CoRR, 2024.
Dawei Li, Bohan Jiang, LiangJie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge- arxiv preprint ar arXi::2411.16594 2024a.
Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Lims-as-judges: comprehensive survey on llm-based evaluation methods. arXiv preprint ar arXie:2412.05579 2024b.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena- hard and benchbuilder pipeline. arXiv preprint ar Xrv::24406.1993 11939, 2024c.
Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M Guerreiro, Ricardo Rei, Duarte M Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, et al. Eurollm: Multilingual language models for europe. Procedia Computer Science, 255:
53-62, 2025.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie Comet: A neural framework for mt evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2685-2702, 2020.
Ricardo Rei, José Pombal, Nuno M Guerreiro, João Alves, Pedro Henrique Martins, Patrick Fernandes, Helena WU, Tania Vaz, Duarte Alves, Amin Farajian, et al. Tower v2: Unbabel- ist 2024 submission for the general mt shared task. In Proceedings of the Ninth Conference on Machine Translation, pp. 185-204 2024.
12

---

Published as a conference paper at COLM 2025
Thibault Sellam, Dipanjan Das, and Ankur Parikh. Bleurt: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892 2020.
Guijin Son, Dongkeun Yoon, Juyoung Suk, Javier Aula-Blasco, Mano Aslan, Vu Trong Kim, Shayekh Bin Islam, Jaume Prats-Cristic Lucía eval: A multilingual meta-evaluatton benchmark for llm-as- -in-as---judge and reward models. ar Xiv preprint ar Xi::2410..1557 2024.
and Seungone Kim. Mm-
Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The good, the bad, and the greedy: Evaluation of Ilms should not ignore non-determinism. ar Xiv preprint ar trXiv2407.1045// 2024.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arviv preprint ar arXiv:2408.0011. 2024.
Ahmet Üstün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. Aya model: An instruction finetuned open-access multilingual language model. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 15894- 15939, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18653
URL https:://acclaaatoolegg http:/ooooooo.../2220.... org/2024. acl acl-long. 845.
Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbe- mileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. Aya model: An instruction finetuned open-access multilingual language model. ar arXiv preprint 0rXiv:2402.07827, 2024b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural informa- fion processing systems, 30, 2017.
Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. Foundational autoraters: Taming large language models for better automatic evaluation. ariiv preprint arXiv:2407.10817 2024.
Peifeng Wang, Austin Xu, Yilun Zhou, Caiming Xiong, and Shafiq Joty. Direct judgement preference optimization. arXiv preprint ar Xiv:2400. 1ene24449.1664, 2024.
Ian Wu, Patrick Fernandes, Amanda Bertsch, Seungone Kim, Sina Pakazad, and Gra- ham Neubig. Better instruction-folloiing through minimum bayes risk. ar Xiv preprint ar axXi::2410002900 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arviv preprint ar Xrit:22112.55115 2024.
Ran Zhang, Wei Zhao, and Steffen Eger. How good are Ilms for literary translation, really? literary translation evaluation with humans and 1lms. arXiv preprint arXio:2410.18697, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595- 46623, 2023.
13

---

Published as a conference paper at COLM 2025
A Training Data Details
A.1 Input and Output Components
We denote English-language components with (E) and non-Enggish-languuae components with (M).
Each input is composed of the following components:
Instruction (M): user instruction.
2. Response (M): A response to the instruction to evaluate, This evaluation is done based on the score rubric.
Rubric (E): The custom scoring criteria defined by the user. For both DA and PWC, the rubric consists of description of the evaluation criteria. For DA, this is complemented with a description of the features responses receiving a certain score (from 1 to sinclusive) should possess.
Reference Answer (M): Optional. A reference answer represents an ideal response to the user instruction. When used in reference-based mode, the evaluator's decisions made relative to the reference, which is assumed to be the gold-standard response. When used in reference-free model, no reference is provided and the evaluator tasked with making decisions independently.
For PWC, we include two responses instead of one, and the evaluator is tasked with choosing the better of the two responses. Each output, meanwhile, is composed of the following two components:
1. Feedback (E): A long-form explanation of the decision of the evaluator.
Judgement: An integer score (DA) or binary indicator (PWC) representing the evaluator's final decision.
14

---

Published as a conference paper at COLM 2025
A.2 M-FEEDBACK COLLECTION and M-PREFERENCE COLLECTION Training Examples
###Task Description:
An instruction (might include an Input inside it), response to evaluate, reference answer that gets a score of 5, and score rubric representing evaluation criteria given. Write a detailed feedback that assess the quality of the response strictly on the given score rubric, not evaluating in general.
2. After writing feedback, write score that is an integer between 1 and 5. You should refer to the score rubric.
The output format should look as follows: "Feedback: (write feedback for criteria) [RESULT]
(an integer number between 1 and 5)"
4. Please do not generate any other opening, closing, and explanations.
###The instruction evaluate:
Dans le cadre de I'organisation d' d'un festival culturel municipal, proposer une programmation d' l'activvtés et d' d'événements qui permettrait d' d'attirer et d' d'engager tous les membres de la commu- nauté locale. La proposition doit tenir compte des différents groupes d'áge, origines culturelles, capacités physiques situations cocio-économiquss. Le budget alloué est de 50 000 euros, et festival doit se dérouler sur trois jours.
###Reppnsse to evaluate:
Notre festival culturel municipal propose une programmation variée sur trois jours, avec une scène principale présentant des spectacles de musique et danse représentant différentes traditions, ainsi que des espaces dédiés aux arts culinaires du monde. Des ateliers créatifs et éducatifs sont prévus pour tous les åges, incluant des activités adaptées pour les enfants, les adolescents et les seniors. Le site est aménagé avec des zones de repos, des espaces familiaux des stands d' d'artisanat local, complétés par une aire de restauration diversifiće. Des navettes gratuites desservent les différents quartiers de la ville, et des tarifs préférentiels sont proposés pour les étudiants, les familles nombreuses et les semiors. La signalisation sur le site est claire et multilingue, avec des bénévoles disponibles pour guider et assister les visiteurs. Les activités sont réparties entre animations plein air et espaces couverts pour d'adapter aux conditions météorologiques et aux préférences de chacun.
Figure 4: Multilingual example of M-FEEDBACK COLLECTION part 1.
15

---

Published as a conference paper at COLM 2025
###eefrence Answer (Score 5):
Notre programmation du festival culturel s'articule autour de plusieurs axes pour garantir une inclusion maximale:
Accessibilité physique et financière:
Entrée gratuite pour les moins de 12 ans et tarifs réduits pour les étudiants, seniors et personnes mobilité réduite
Site entièrement accessible aux fauteuils roulants avec des rampes d' d'accès
Service de navettes gratuites depuis les quartiers périphériques Interprétation en langue des signes pour les événements principaux
Diversité culturelle et artistique: :
Scène principale alternant musiques traditionnelles locales et internationales Espace dédié aux arts culinaires avec des stands représentant différentes cultures Exposition d' a'art contemporain incluant des artistes handicapés valides Ateliers de danse traditionnelle adaptés tous les niveaux de mobilité
3. Programmation intergénérationnelll::
Zone enfants avec activités sensorielles et éducatives multilingues Espace ados avec musique moderne et ateliers création numérique
Coin détente pour seniors avec animations douces et espaces de discussion Activités familiales favorisant l'échange intergénérationnel
4. Inclusion sociale et éducative
Ateliers de création artistique animés par des artistes de différentes origines
Espace d' d'expression libre pour les associations locales Coins lecture avec livres en braille et luvie-livres
Médiateurs culturels multilingues disponibles sur site
Aménagements spécifiques:
Zones calmes pour les personnes neurodivergentes
Signalétique inclusive (pictogrammes, braille, plusieurs langues)
Espaces de prière de méditation
Aires de repos ombragées avec sièges adaptés
Communication inclusive:
Documents promotionnels en plusieurs langues et formats
Site web accessible aux malvoyants
Communication les réseaux sociaux adaptée aux différentes communautés
Ligne téléphonique d'information en plusieurs langues
Budget réparti équitablement entre les différentes activités pour garantir une qualité homogène des prestations pour tous les publics.
###Score Rubrics:
[This criteria is subcategory of overall Harmlessness. Does the model's response account for various abilities, backgrounds, and experiences? This ensures that the content inadvertently exclusive or alienating for certain user groups.]
Score 1: The response is exclusive, potentially alienating certain groups perspectives. Score 2: The response occasionally includes various perspectives but still lacks comprehensive inclusivity.
Score 3: The response attempts inclusivity but misses certain groups or perspectives.
Score 4: The response is mostly inclusive, only missing minor nuances.
Score 5: The response perfectly considers a broad spectrum of backgrounds, experiences, and perspectives.
Figure 5: Multilingual example of M-FEEDBACK COLLECTION part 2.
16

---

Published as a conference paper at COLM 2025
###Feedback:
The response demonstrates a strong commitment to creating an inclusive festival environment by addressing various aspects of accessibility and engagement. It effectively covers physical accessi- bility through shuttle services, different activity zones for various age groups, and multilingual signage. The programming considers different cultural expressions through diverse music, dance, and culinary offerings. The response also addresses economic accessibility through preferential pricing for certain groups and free transportation. However, it slightly falls short of perfect score as it could have been more explicit about accommodations for people with disabilities and neurodivergent individuals. While the response mentions volunteer assistance and clear signage, could have detailed specific provisions for sensory needs or specialized equipment for different abilities. The cultural programming, while diverse, could have included more specific mentions of local community involvement in planning and execution. The response demonstrates thoughtful consideration of most accessibility and inclusion aspects, with only minor gaps in comprehensive coverage. So the overall score is 4. |REUUTT] 4
Figure 6: Multilingual example of M-FEEDBACK COLLECTION part 3.
###Task Description:
An instruction (might include Input inside it), two responses evaluate (denoted as Response and Response B), reference answer, and an evaluation criteria are given.
1. Write a detailed feedback that assess the quality of the two responses strictly based on the given evaluation criteria, not evaluating in general.
2. Make comparisons between Response A, Response B, and the Reference Answer. Instead of examining Response and Response B separately, go straight to the point and mention about the commonalities and differences between them.
3. After writing the feedback, indicate the better response, either "A" or "B".
4. The output format should look as follows: "Feedback: (write feedback for criteria) [RESULT] (Bither "A" or "B")"
5. Please do not generate any other opening, closing, and explanations.
## instruction to evaluate:
Necessito uma explicação detalhada sobre o processo de implementação de um sistema de gestāo de residuos industriais numa fábrica textil, considerando as regulamentações ambientais portuguesas atuais. fundamental incluir aspetos técnicos, logísticos administrativos do processo.
R#sponssse A to evaluate:
Na implementação do sistema de gestāo de residuos têxteis, começa-se com recolha e depois passa direto para armazenamento temporário dos materiais diversos produzidos na fábrica téxtil, que devem seguir várias etapas do processo industrial conforme determinado pelos regulamentos. Os operadores precisam realizar separação adequada considerando diferentes tipos existentes para tratamento posterior adequado segundo normas vigentes atualmente Portugal sobre resíduos industriais especificos deste setor. Ô processo envolve também documentação necessária registos conforme exigido pela legislação ambiental portuguesa atual que regula este tipo de atividade industrial textil em especifico. Na parte de monitorização é preciso fazer acompanhamento regular das atividades relacionadas sistema implementado na fábrica para garantir conformidade. Quanto à formação dos colaboradores envolvidos diretamente haver treinamento inicial completo sobre procedimentos. controle de qualidade precisa ser constante em todas as etapas do processo industrial textil para evitar problemas futuros com fiscalização. Em relação aos contratos com empresas terceirizadas para coleta e destinação final, é importante estabelecer parcerias adequadas. Para finalizar, melhoria continua do sistema depende de análises regulares dos resultados obtidos ao longo do tempo. A documentação completa deve estar organizada num manual técnico específico para consulta.
operações do sistema, deve
Figure 7: Multilingual example of M-PREFERENCE COLLECTION part 1.
17

---

Published as a conference paper at COLM 2025
###Response to evaluate:
Para implementar um sistema eficaz gestão de resíduos industriais numa fábrica téxtil em Portugal, necessário seguir um processo estruturado e abrangente:
1. Planeamento Análise Inicial
Mapeamento detalhado dos fluxos de residuos
Avaliação das práticas atuais
Identificação das necessidades especificas
Definição de objetivos metas
2. Estruturação do Sistema
Desenvolvimento do manual operacional
Estabelecimento de procedimentos
Definição de responsabilidades Criação de registos documentação
3. Implementação Prática Infraestrutura
Áreas de armazenamento adequadas Equipamentos necessários
Sistemas de controlo
Sinalização apropriada
Gestão Operacional
Procedimentos segregação
Rotinas de recolha
Registos documentação Monitorização continua
4. Aspetos Administrativos
Licenciamentos necessários
Contratos com operadores
Registos obrigatórios Relatórios periódicos
Formação Controlo
Capacitação
Treino inicial
Atualizações periódicas
Material informativo
Avaliações regulares
Monitorização
Indicadores de desempenho
Auditorias internas
Ações corretivas
Melhorias contínuas
Figure 8: Multilingual example of M-PREFERENCE COLLECTION part 2.
18

---

Published as a conference paper at COLM 2025
###Refrrecee Answer:
processo implementação de um sistema de gestão resíduos industriais numa fábrica textil pode ser estruturado nas seguintes fases principais:
Fase de Diagnóstico Inicial
Levantamento completo dos tipos de resíduos gerados
Quantificação dos volumes de produção de residuos
Análise dos processos existentes de gestão Identificação de pontos críticos no fluxo de resíduos
2. Enquadramento Legal
Decreto-Lein n. 102-D/2020 (Regime Geral de Gestão de Resíduos)
Portaria n. 289/2015 (Sistema Integrado de Registo Eletrónico de Resíduos) Regulamentações especificas para indústria têxtil
Requisitos municipais aplicáveis
3. Implementação Operacional Infraestrutura Física
Criação de áreas dedicadas para armazenamento temporário
Instalação de contentores específicos por tipo de resíduo Implementação de sistemas de pesagem registo Definição de rotas de circulação internas
Processos Procedimentos
Desenvolvimento de procedimentos operacionais padrão Estabelecimento de fluxos de trabalho e responsabilidades Criação de documentação registos necessários Implementação de sistema de rastreabilidade
Formação Sensibilização
Programa de formação inicial para todos colaboradores
Treino específico para operadores do sistema
Campanhas de sensibilização continua
Materiais informativos sinalética adequada
Monitorização Controlo
Indicadores de Desempenho
Taxa de segregação correta dos resíduos
Volume de resíduos por categoria
Custos de gestão e tratamento
Conformidade legal
Sistema de Gestão
Auditorias internas regulares
Revisão periódica dos procedimentos
Ações corretivas preventivas
Relatórios de desempenho
Parcerias Contratos
Seleção de operadores licenciados de gestão de resíduos Estabelecimento de contratos de recolha tratamento Parcerias para valorização de residuos
Acordos com entidades recicladoras
7. Melhoria Continua
Análise regular dos resultados
Identificação de oportunidades de melhoria
Atualização de procedimentos práticas
Incorporação de novas tecnologias métodos
Este sistema deve ser documentado num Manual de Gestão de Resíduos, que servirá como re- ferência principal para todas operações procedimentos relacionados gestão de resíduos na instalação.
Figure 9: Multilingual example of M-PREFERENCE COLLECTION part 3.
19

---

Published as a conference paper at COLM 2025
##Evaluation Criteria:
the response structured to promote readability and coherence? Does the response exhibit excellent organization?
####ceedbcck: When examining the organizational structure, Response presents information in a continuous paragraph format, making it challenging to navigate through different aspects of waste management implementation. The ideas flow without clear segmentation, which reduces the readability and makes it difficult to locate specific information about technical, logistical, and administrative aspects.
In contrast, Response employs hierarchical structure with clear numbered sections and bullet points. The information is organized into five main categories: Planning and Initial Analysis, System Structuring, Practical Implementation, Administrative Aspects, and Training and Control. Each category is further broken down into relevant subcategories with specific bullet points, creating a logical flow information.
The use of indentation and consistent formatting in Response B helps readers distinguish between different levels of information. For instance, under "Practical Implementation, " the subcategories Infrastructure and Operational Management are clearly delineated, making it easy to understand the relationship between different components of the system.
While both responses cover similar content, Response B's structured approach with clear headings, subheadings, and bullet points makes it significantly more effective in presenting the information in a coherent and accessible manner. The systematic organization allows readers to quickly locate specific information and understand the relationships between different aspects of the waste management system. Therefore, Response B is superior in terms of structure and organization. [RESULT]
Figure 10: Multilingual example of M-PREFERENCE COLLECTION part 4.
20

---

Published as a conference paper at COLM 2025
A.3 MT Evaluation Data Details
A.3.1 Prompts For Generating Source Texts and Translations
You multilingual content creator and translation expert. Your task is to generate comprehen- sive translation exercise package based on the given attributes. Follow these instructions carefully:
1. Review the following input variables:
Source language: (/soureelanguagee
'Target language: $ (target language]
- -Topc: $(topic}
SSuttppc: {(ubbtopic]
Source Length: {ssurce length)
Audience: $ ((audience)
Style: S(style]
Generate source text:
Create an original text in the source language, adhering to the specified topic, subtopic, and length. The text should be coherent, informative, and suitable for translation.
Create a translation instruction: Formulate clear and specific instruction for translating the source text, taking into account the given attributes. The instruction should guide the translator how to approach the translation task.
4. Generate reference translation:
Produce high-quality, fluent translation of the source text in the target language. This translation should serve reference for evaluating other translations.
5. Develop scoring rubrics: Create one three scoring factors to evaluate translations. These rubrics should be in English, clear, specific, and relevant the translation task.
6. Generate descriptions of scores, ranging from score 1 (worst) to score 5 (best), which will later be used guidelines to score translations. Give description in English of what each score represents.
Format your output as follows:
<START OF SOURCE> [INSERT THE SOURCE TEXT HERE
<END OF SOURCE>
<START TRANSLATION INSTRUCTION>
JINSERT THE TRANSLATION INSTRUCTION HERE
<END OF TRANSLATION INSTRUCTION>
<START OF REFERENCE TRANSLATION> [INSERT THE REFERENCE TRANSLATION HERE]
<END OF REFERENCE TRANSLATION>>
<START OF SCORING RUBRICS>
INSERT SCORING RUBRICS IN ENGLISH SEPARATED BY A <END OF SCORING RUBRICS>
Figure 11: Prompt for generating MT Eval source texts and references part 1.
21

---

Published as a conference paper at COLM 2025
<START OF SCORE 1 DESCRIPTION>
[INSER SCORE DESCRIPTION IN ENGLISH HERF)
END OF SCORE DESCRIPTION>
<START OF SCORE 2 DESCRIPTION
JINSERT' SCORE 2 DESCRIPTION IN ENGLISH HERE]
<END OF SCORE DESCRIPTION
<START SCORE 3 DESCRIPTION>
[INSERT SCORE DESCRIPTION IN ENGLISH HERE)
<END OF SCORE >
<START OF SCORE 4 DESCRIPTION>
[INSERT SCORE 4 DESCRIPTION IN ENGLISH HERE]
<END OF SCORE DESCRIPTION>
<START OF SCORE 5 DESCRIPTION>
[INSERT SCORE DESCRIPTION IN ENGLISH HERE)
<END OF SCORE DESCRIPTION>
Ensure that your response is comprehensive, coherent, and follows all the instructions provided
above.
IMPORTANT: ABIDE STRICTLY BY THE REQUESTED FORMAT AND KEEP GENERATING UNTIL THE END OF THE REQUESTED OUTPUT.
Figure 12: Prompt for generating MT Eval source texts and references part
Generate example translation of score {N} for the given translation instruction, source, and
scoring rubrics:
<START OF SOURCE>
$(source)
<END OF SOURCE>
<START OF TRANSLATION INSTRUCTION>
<END OF TRANSLATIÓN INSTRUCTION>
<START OF SCORING RUBRICS>
(scoring rubrics)
<END OF SCORING RUBRICS>
<START OF SCORE (N} TRANSLATION> [INEET TRANSLATION HERE
<END OF SCORE (N) TRANSLATION>
IMPORTANT: ABIDE STRICTLY THE REQUESTED FORMAT AND KEEP GENERATING UNTIL THE END OF THE REQUESTED OUTPUT.
Figure 13: Prompt for generating MT Eval score example.
A.3.2 Data Generation Prompt Variable Counts
We list the number of training instances for each value of each variable in our MT evaluation data generation prompt.
Topic and Subtopic. Gaming & Software: 296 (Virtual Reality: 48, Software Development:
56, Mobile Games: 16, Cloud Gaming: 32, Game Development: 32, Gaming Communi- ties: 64, Gaming Hardware: 48); Sports Industry: 288 (Sports Management: 16, Athletic Training: 56, Athletic Equipment: 64, Sports Technology: 32, Sports Medicine: 48, E-sports: 24, Professional Leagues: 48); Financial Services: 280 (Digital Banking: 40, Insurance: 88,
Wealth Management: 40, Payment Systems: 32, Financial Technology: 32, Risk Management:
22

---

Published as a conference paper at COLM 2025
16, Investment Management: 32); Mumbai: 272 (Cultural Heritage: 40, Entertainment: 40,
Fashion: 40, Film Industry: 48, Business Center: 48, Food Culture: 40, Urban Development:
16); China: 264 (Cultural Heritage: 24, Business Culture: 56, Urban Development: 48,
Technology Industry: 48, Traditional Customs: 32, Food Culture: 40, Innovation Hub: 16);
Music Industry: 264 (Music Technology: 40, Music Production: 40, Industry Trends: 32,
Live Events: 32, Music Publishing: 48, Digital Distribution: 24, Artist Management: 48);
Food & Agriculture: 256 (Food Technology: 32, Food Safety: 56, Urban Farming: 16, Agri-
cultural Trade: 32, Agricultural Policy: 40, Organic Production: 40, Sustainable Farming: 40);
Manufacturing & Safety: 256 (Production Processes: 24, Workplace Standards: 40, Equip-
ment Safety: 24, Safety Regulations: 32, Risk Assessment: 48, Industrial Safety: 24, Quality
Control: 64); Brazil: 248 (Cultural Festivals: 24, Urban Life: 24, Business Environment 48,
Tourism Industry: 48, Food & Cuisine: 56, Music Scene: 16, Sports Culture: 32); Fitness &
Wellness: 248 (Nutrition: 32, Mental Health: 48, Health Tracking: 24, Exercise Programs: 24, Wellness Technology: 40, Wellness Education: 56, Personal Training: 24); Architecture &
Design: 248 (Sustainable Design: 16, Digital Architecture: 40, Interior Design: 24, Design
Innovation: 48, Building Technology: 24, Architectural Heritage: 56, Urban Architecture: 40);
India: 232 (Culinary Traditions: 24, Cultural Diversity: 40, Technology Sector: 40, Festival
Culture: 48, Business Hub: 32, Film Industry: 32, Traditional Arts: 16); Social Media: 232
(Social Commerce: 24, User Engagement: 64, Influencer Marketing: 32, Social Analytics:
32, Digital Communities: 32, Content Creation: 24, Platform Development: 24); Seoul: 232
(Fashion Trends: 24, Tech Industry: 24, Urban Innovation: 40, Food Scene: 48, Business
Hub: 24, Pop Culture: 24, Entertainment: 48); Books Literature: 224 (Publishing Industry:
24, Book Marketing: 40, Author Platform: 16, Digital Publishing: 72, Literary Events: 32,
Reading Technology: 40); São Paulo: 224 (Sports Culture: 32, Business Hub: 24, Cultural
Scene: 48, Urban Life: 24, Entertainment: 48, Food & Dining: 32, Fashion Industry: 16);
Spain: 224 (Tourism Industry: 40, Sports Culture: 16, Cultural Traditions: 32, Business
Environment: 56, Urban Life: 24, Culinary Arts: 56); Portugal: 224 (Cultural Heritage: 40, Business Innovation: 40, Food & Wine: 32, Arts Scene: 24, Urban Development: 16, Tourism
Industry: 48, Maritime Culture: 24); Tokyo: 224 (Entertainment Districts: 32, Cuisine: 48,
Fashion: 16, Traditional Culture: 32, Urban Innovation: 16, Technology Industry: 56, Pop
Culture: 24); Workplace Transformation: 224 (Office Technology: 48, HR Innovation: 48,
Workplace Safety: 24, Remote Work: 16, Corporate Culture: 16, Professional Development:
24, Employee Wellness: 48); Dubai: 224 (Cultural Traditions: 48, Luxury Lifestyle: 40,
International Trade: 40, Tourism Industry: 32, Business Center: 32, Urban Development:
24, Technology Innovation: 8); Berlin: 224 (Startup Scene: 24, Alternative Culture: 40,
Cultural History: 24, Tech Industry: 40, Art Community: 40, Nightlife: 24, Urban Planning:
32); Poetry: 224 (Haiku: 24, Asian Poetry: 32, Theme identification: 32, Modernism: 88,
Contemporary: 16, European Poetry: 32); London: 224 (Food Scene: 56, Theatre & Arts:
40, Urban Transport: 56, Financial Services: 32, Royal Traditions: 24, Cultural Heritage:
16); Lisbon: 216 (Urban Innovation: 24, Maritime Heritage: 48, Tourism Industry: 48
Arts Scene: 32, Startup Ecosystem: 8, Food & Wine: 40, Cultural History: 16); New York
City: 216 (Entertainment: 24, Tourism: 32, Urban Development: 56, Sports Teams: 24,
Business & Finance: 32, Arts & Culture: 24, Food & Dining: 24); Global Markets: 216 (Stock
Exchanges: 40, Foreign Investment: 56, Emerging Markets: 40, International Trade: 32,
Foreign Exchange: 16, Market Regulations: 24, Commodity Markets: 8); Weather & Climaaee:
216 (Climate Technology: 40, Climate Change: 24, Climate Science: 16, Environmental
Impact: 72, Weather Forecasting: 24, Atmospheric Research: 16, Weather Systems: 24);
Urban Development: 208 (Infrastructure: 48, Smart Cities: 32, Green Spaces: 16, Public
Transportation: 16, Urban Planning: 56, Housing Projects: 32, Sustainable Development:
8); Arts & Culture: 208 (Art Market: 32, Performance Art: 40, Art Education: 24, Cultural
Events: 40, Cultural Heritage: 32, Visual Arts: 32, Digital Art: 8); Germany: 200 (Technology
Sector: 72, Sports Culture: 48, Education System: 16, Automotive Industry: 24, Cultural
Traditions: 8, Business Innovation: 16, Urban Development: 16); Japan: 200 (Business
Practices: 32, Traditional Culture: 32, Arts & Crafts: 32, Technology Industry: 48, Popular
Culture: 24, Social Customs: 24, Food & Cuisine: 8); Insurance & Risk Management: 200
(Risk Assessment: 32, Underwriting: 40, Claims Processing: 8, Risk Mitigation: 32, Insurance
Technology: 32, Regulatory Compliance: 48, Insurance Products: 8); Italy: 200 (Fashion Industry: 48, Design Industry: 24, Arts Scene: 32, Business Culture: 32, Cultural Heritage:
32, Food & Wine: 32); Pharmaceutical Industry: 200 (Manufacturing: 40, Patient Safety: 32,
23

---

Published as a conference paper at COLM 2025
Drug Development: 32, Clinical Trials: 32, Regulatory Approval: 24, Market Access: 40);
Beauty & Cosmetics: 200 (Makeup Products: 40, Beauty Technology: 32, Sustainability: 8,
Product Development: 40, Natural Cosmetics: 56, Skincare: 8, Marketing: 16); Interaatoonal
Relations: 200 (Global Security: 72, Trade Agreements: 32, Cultural Exchange: 24, Regional
Alliances: 40, Diplomatic Missions: 16, International Aid: 16); Medical & Healthcare: 200
(Healthcare IT: 16, Medical Insurance: 24, Phharmaceutical Research: 40, Patient Care: 48,
Clinical Trials: 40, Telemedicine: 16, Medical Devices: 16); Automotive Industry: 200
(Safety Systems: 40, Auto Design: 16, Autonomous Technology: 56, Vehicle Manufacturing:
24, Market Trends: 32, Car Technology: 24, Electric Vehicles: 8); Environmental Policy:
200 (Climate Agreements: 32, Marine Conservation: 32, Carbon Trading: 24, Renewable
Energy Initiatives: 16, Urban Planning: 56, Waste Management: 16, Wildlife Protection:
24); Marketing & Advertising: 200 (Advertising Technology: 32, Content Marketing: 56,
Social Media Marketing: 24, Market Research: 16, Digital Marketing: 32, Brand Strategy:
32, Campaign Management: 8); France: 192 (Arts & Literature: 32, Business Culture: 24,
Wine Industry: 24, Tourism: 32, Cultural Heritage: 32, Culinary Arts: 24, Fashion Industry:
24); Home & Living: 192 (Smart Home: 40, Furniture: 56, Sustainable Living: 8, Home
Improvement: 40, Decorative Arts: 16, Interior Design: 16, Home Technology: 16); Paris:
192 (Culinary Arts: 24, Art Scene: 48, Luxury Brands: 72, Cultural Landmarks: 16, Fashion
Industry: 16, Urban Life: 8, Tourism Industry: 8); Parenting & Family: 192 (Family Dynam-
ics: 40, Education: 32, Family Health: 40, Parenting Resources: 32, Child Safety: 16, Child
Development: 32); Patents& & Intellectual Property: 192 (Patent Applications: 32, Trade Se- crets: 40, IP Litigation: 32, Trademark Registration: 24, International Patents: 40, Copyright
Protection: 16, IP Strategy: 8); Amsterdam: 192 (Art Scene: 40, Business Innovation: 32,
Cycling Culture: 16, Tourism: 56, Tech Industry: 24, Urban Planning: 16, Cultural Heritage:
8); Economic Policy: 184 (Trade Regulations: 40, Fiscal Measures: 16, Economic Stimulus:
40, Employment Policy: 32, Tax Reform: 16, Banking Regulations: 24, Monetary Policy: 16);
Public Health: 184 (Vaccination Programs: 32, Epidemiology: 32, Mental Health Services:
24, Healthcare Systems: 48, Disease Prevention: 24, Health Technology: 16, Maternal Health:
8); Tech Innovation: 184 (Green Tech: 32, Cybersecurity: 40, Quantum Computing: 48, Robotics: 16, Biotechnology: 24, Edge Computing: 8, Artificial Intelligence: 16); Singapore:
184 (Cultural Diversity: 24, Urban Planning: 56, Education: 24, Food Culture: 24, Business
Innovation: 16, Financial Hub: 24, Smart City Initiatives: 16); Film & Cinema: 184 (Film
Marketing: 32, Digital Effects: 8, Film Industry: 48, Film Technology: 24, Distribution: 32,
Cinema Innovation: 24, Film Production: 16); Cultural Trends: 184 (Fashion Movements: 8,
Entertainment Trends: 48, Digital Culture: 24, Social Media Influence: 24, Pop Culture: 16,
Art Movements: 40, Cultural Festivals: 24); Religious & Cultural Studies: 184 (Sacred Texts:
56, Interfaith Dialogue: 16, Cultural Anthropology: 40, Religious Education: 24, Religious
Practices: 24, Religious Traditions: 16, Cultural Heritage: 8); Politics & Governance: 184
(Political Communication: 24, Government Innovation: 16, Electoral Processes: 24, Political
Systems: 24, Public Policy: 16, Governance Reform: 48, Civic Technology: 32); Consumer
Electronics: 184 (Mobile Devices: 16, Audio Equipment: 24, Display Technology: 24, Gam-
ing Hardware: 32, Smart Home: 24, Personal Computing: 48, Wearable Technology: 16);
Madrid: 176 (Business Hub: 40, Tourism: 24, Sports Culture: 16, Food & Wine: 16, Arts
Scene: 40, Urban Life: 32, Cultural Heritage: 8); NGOs & Nonprofits: 176 (International
Development: 24, Social Innovation: 32, Fundraising: 32, Community Development: 48,
Humanitarian Aid: 8, Social Impact: 32); Wildlife & Nature: 176 (Environmental Protection:
32, Conservation: 16, Species Preservation: 16, Wildlife Research: 56, Biodiversity: 24,
Natural Habitats: 16, Ecosystem Management: 16); E-commerce & Retail: 176 (Customer
Experience: 16, Online Marketplaces: 48, Mobile Commerce: 24, Retail Technology: 24,
Digital Payment: 32, Supply Chain: 24, Retail Analytics: 8); Stockholm: 176 (Business
Hub: 40, Cultural Scene: 48, Urban Planning: 24, Food & Lifestyle: 32, Sustainability: 16,
Design Culture: 8, Tech Innovation: 8); Legal & Compliance: 168 (Legal Technology: 40,
International Law: 24, Regulatory Compliance: 24, Corporate Law: 24, Consumer Rights:
16, Intellectual Property: 24, Data Protection: 16); Dating & Relationships: 168 (Personal
Growth: 16, Relationship Psychology: 40, Relationship Counseling: 48, Dating Culture:
8, Online Dating: 24, Social Connection: 16, Dating Apps: 16); Academic Research: 168
(Peer Review: 48, Scientific Publications: 16, Research Ethics: 40, Academic Collaboration:
8, Data Analysis: 40, Research Methodology: 16); Media & Entertainment 168 (Digital
Media: 32, Content Creation: 48, Streaming Services: 16, Publishing: 16, Film Production:
24

---

Published as a conference paper at COLM 2025
24, Broadcasting: 8, Gaming Industry: 24); Telecommunications: 168 (Communication Services: 16, Industry Standards: 32, Digital Networks: 24, Telecom Innovation: 40, Mobile
Technology: 32, Wireless Technology: 24); Scientific Discoveries: 168 (Marine Biology: 32,
Genetic Research: 32, Physics Advances: 40, Archaeological Finds: 8, Space Exploration:
24, Climate Science: 24, Medical Breakthroughs: 8); Tourism & Hospitality: 160 (Hotel
Management: 32, Eco-Tourism: 16, Tourism Marketing: 24, Event Planningg 16, Travel
Services: 16, Cultural Tourism: 24, Customer Service: 32); Education Reform: 160 (Digital
Learning: 24, Curriculum Changes: 16, STEM Initiatives: 24, Assessment Methods: 16,
Higher Education: 32, Teacher Training: 32, Special Education: 16); Space Exploration: 160
(Space Technology: 40, Space Industry: 32, Astronomical Discovery: 32, Space Policy: 16,
Satellite Systems: 8, Space Research: 16, Space Travel: 16); Fashion Apparel: 152 (Textile
Industry: 24, Fashion Technology: 48, Sustainable Fashion: 24, Retail Fashion: 24, Luxury
Brands: 24, Fashion Design: 8); Real Estate: 144 (Investment: 24, Sustainable Building: 32,
Property Management: 8, Real Estate Technology: 24, Commercial Real Estate: 32, Market
Analysis: 24); Mental Health: 144 (Mental Health Technology: 48, Mental Health Education:
40, Support Programs: 24, Therapy Services: 24, Youth Mental Health: 8); Transportation &
Mobility: 144 (Aviation: 8, Electric Vehicles: 32, Autonomous Driving: 24, Maritime Trans-
port: 56, Ride Sharing: 8, Public Transit: 16); Government Documentation: 144 (Regulatory
Guidelines: 32, Administrative Procedures: 32, Official Forms: 40, Policy Documents: 8,
Legislative Documents: 32); Food & Cuisine: 136 (Food Technology: 16, Culinary Arts:
16, Food Innovation: 40, Dietary Trends: 40, Culinary Education: 8, Food Culture: 16);
Sydney: 136 (Tourism: 32, Urban Development: 16, Sports Events: 32, Business District:
8, Lifestyle & Culture: 24, Food Scene: 16, Entertainment: 8); Sports & Recreation: 136
(Fitness Training: 24, Sports Technology: 16, Sports Management: 16, Equipment Innovation:
32, Recreational Activities: 16, Sports Medicine: 16, Professional Sports: 16); Renewable
Energy: 120 (Sustainable Development: 8, Solar Power: 48, Energy Storage: 16, Clean
Energy Innovation: 16, Green Technology: 8, Energy Policy: 8, Wind Energy: 16); History &
Heritage: 120 (Archaeological Studies: 32, Cultural Preservation: 32, Heritage Conservation:
8, Cultural Memory: 24, Digital Archives: 16, Historical Research: 8); United Kingdom: 112
(Arts & Entertainment: 8, Sports Culture: 16, Business Innovation: 16, Financial Services: 16,
Education System: 40, Urban Life: 8, Cultural Heritage: 8).
Style. journalistic: 1024; creative: 1016; analytical: 968; formal: 960; poetic: 960; min-
imalist: 952; humorous: 936; academic: 912; elaborate: 912; narrative: 880; rushed: 864;
technical: 840; neutral: 832; informal: 824; descriptive: 792; casual: 792; concise: 776;
persuasive: 760.
Audience. seniors: 1616; parents: 1488; college students: 1408; experts: 1408; general public: 1400; professionals: 1352; teenagers: 1304; middle-aged adults: 1288; educators: 1256; beginners: 1216; children: 1160; young adults: 1104.
Source Length. short: 4168; very long: 4112; long: 3912; medium: 3808.
B Training Hyperparameters
We train all our models on 1 epoch of our training dataset, with cosine learning rate scheduler with a warmup of 10% the total steps, and initial learning rate of 1 decaying to 0. We use a batch size of 32 with sequences of up to 4096 tokens.
25

---

Published as a conference paper at COLM 2025
C Results by Language, Category, and Language Pair
C.1 MM-Eval
MMEval (English)
Model
Chat
Language Hallucinations
Linguistics
Reasoning Safety
Proprietary Models
GPT-40
0.7216
0.9800
0.7913
0.5761
Small (3B parameters)
Qwen2.5-3D-Instruct
0.6598
0.6000
0.5913
0.5978
Glider 3B +
0.6495
0.6000
0.6174
0.7174
M-PROMETHEUS 3B
0.5876
0.6800
0.5652
0.8696
Medium (7B parameters)
Qwwn2..-77-Instruct
0.6907
0.6800
0.6783
0.8478
PROMETHEUS 2 7B
0.6340
0.6667
0.4826
0.7554
Hercule 7B
0.6289
0.4933
0.5000
0.2446
M-PROMETHEUS
0.6804
0.7600
0.6087
0.8261
Large (14B+ parameters)
Qwen2.55144BInnsrrue
0.7423
0.8933
0.7217
0.5652
PROMETHEUS 2 8x7B
0.6495
0.6467
0.5478
0.8804
M-PROMETHEUS 14B*
0.6186
0.8800
0.6435
0.9022
Backbone Model Ablations
Mistral-7B-v0.2IInsttee
0.6392
0.6267
0.4783
0.5543
FuroILLM-9B-Imsrruct
0.6289
0.6667
0.5826
0.9022
Aya-Expanse-8B Qwenn25-7B-Instruct
0.6598
0.6933
0.5957
0.6957
0.7010
0.7067
0.6000
0.6848
Training Data Ablations
MT Eval Data
0.6082
0.7733
0.6174
0.7717
Translated Data
3 Non-English Langs
0.6082
0.8133
0.5826
0.6957
Multilingual Data
3 Non-English Langs
0.5979
0.7200
0.5652
0.8152
5 Non-English Langs
0.6186
0.6933
0.5565
0.8478
Table 3: Accuracy on MMEval (English) broken down by category.
26

---

Published as a conference paper at COLM 2025
MMEval (German)
Model
Chat
Language Hallucinations
Linguistics Reasoning Safety
Proprietary Models
GPT-40
0.7069
0.6933
0.7966
Small (3B parameters)
Qwen2.5-3B-nnstruct
0.6724
0.6400
0.5876
Glider 3B +
0.6379
0.5200
0.5819
M-PROMETTEUS 3B
0.5517
0.6267
0.7232
Medium (7B parameters)
Qwen2.5-7B-Instruct
0.7586
0.6800
0.6554
PROMETHEUS 2 7B+
0.7069
0.6800
0.5819
Hercule 7B
0.7155
0.6133
0.5678
M-PROMETHEUS 7B
0.6552
0.5467
0.6271
Large (14B+ parameters)
Qwen2.5-14B-Instrue
0.6897
0.6667
0.7288
PROMETHEUS 2 8x7B
0.6897
0.6133
0.6554
M-PROMETHEUS 14B
0.7241
0.6133
0.7006
Backbone Model Ablations
Mistral-7B--v..2Innttru
0.6207
0.4133
0.5480
EuroLLM-9B-Instruue
0,5862
0.6667
0.6130
Aya-Expanse-8B Qwen2..-77BIInsrruct
0.6724
0.6533
0.6356
0.6207
0.6400
0.6215
Training Data Ablations
MT Eval Data
0.6724
0.6267
0.6610
Translated Data
Non-English Langs
0.6379
0.5733
0.6695
Multilingual Data
3 Non-English Langs
0.6379
0.4933
0.6215
5 Non-English Langs
0.6034
0.5733
0.5395
Table 4: Accuracy MMEval (German) broken down by category.
27

---

Published as a conference paper at COLM 2025
MMEval (French)
Model
Chat
Language Hallucinations
Linguistics Reasoning Safety
Proprietary Models
GPT-40
0.7333
0.7847
Small (3B parameters)
Qwen2.5-3B-nnstruct
0,6000
0.6181
Glider 3B +
0.6889
0.6771
M-PROMETTEUS 3B
0.7111
0.6875
Medium (7B parameters)
Qwen2.5-7B-Instruct
0.7111
0.6319
PROMETHEUS 2 7B+
0.5000
0.5347
Hercule 7B
0.6111
0.5556
M-PROMETHEUS 7B
0.6667
0.6319
Large (14B+ parameters)
Qwen2.5-14B-Instrus
0.7556
0.7222
PROMETHEUS 2 8x7B
0.7222
0.5833
M-PROMETHEUS 14B
0.6444
0.7014
Backbone Model Ablations
Mistrall7B-v0..-IInsrruu
0.5556
0.5903
EuroLLM:9B-Instrue
0.6222
0.6007
Aya-Expanse-8B Qwen2.5-77BIInsrruct
0.6222
0.6458
0.6222
0.6910
Training Data Ablations
MT Eval Data
0.6000
0.6875
Translated Data
Non-English Langs
0.6000
0.7257
Multilingual Data
3 Non-English Langs
0.6222
0.6007
5 Non-English Langs
0.5778
0.5521
Table 5: Accuracy MMEval (French) broken down by category.
28

---

Published as a conference paper at COLM 2025
MMEval (Spanish)
Model
Chat
Language Hallucinations
Linguistics Reasoning Safety
Proprietary Models
GPT-40
0.6413
0.7240
0.9533
0.7928
Small (3B parameters)
Qwen2.5-3B-Instruct
0,6630
0.5729
0.5600
0.6184
Glider 3B +
0.5870
0.6432
0.5600
0.6020
M-PROMETTEUS 3B
0.6087
0.5625
0.6400
0.6053
Medium (7B parameters)
Qwen2.5-7B-Instruct
0.7826
0.5677
0.5867
0.6513
PROMETHEUS 2 7B+
0.6322
0.5625
0.6667
0.5362
Hercule 7B
0.5543
0.5208
0.5067
0.5559
M-PROMETHEUS 7B
0.5978
0.7135
0.6267
0.6053
Large (14B+ parameters)
Qwen2.5-14B-Instrus
0.7174
0.8333
0.7867
0.7368
PROMETHEUS 2 8x7B
0.6522
0.7057
0.6933
0.5888
M-PROMETHEUS 14B
0.6413
0.8646
0.6667
0.6908
Backbone Model Ablations
Mistral-7B--v..-Innttru
0.5978
0.4818
0.6533
0.5395
EuroLLM:9B-Instruc
0.5435
0,5885
0,5867
0.5691
Aya-Expanse-8B Qwen2.5-77BIInsrruct
0.5978
0.5938
0.5200
0.5658
0.6848
0.6771
0.6400
0.6546
Training Data Ablations
MT Eval Data
0.6196
0.7135
0.6400
0.6513
Translated Data
Non-English Langs
0.6522
0.6094
0.6133
0.6711
Multilingual Data
3 Non-English Langs
0,5435
0.6198
0.6267
0.6184
5 Non-English Langs
0.6413
0.6250
0.6533
0.5066
Table 6: Accuracy MMEval (Spanish) broken down by category.
29

---

Published as a conference paper at COLM 2025
MMEval (Catalan)
Model
Chat
Language Hallucinations Linguistics Reasoning Safety
Proprietary Models
GPT-40
0.7500
0.7423
0.9267
Small (3B parameters)
Qwen2.5-3B-Instruct
0.7250
0.4691
0.6867
Glider 3B +
0.6875
0.5928
0.5933
M-PROMETTEUS 3B
0.6000
0.5135
0.6533
Medium (7B parameters)
Qwen2.5-7B-Instruct
0.7250
0.5206
0.6133
PROMETHEUS 2 7B+
0.6500
0.4433
0.6933
Hercule 7B
0.6250
0.5258
0.4733
M-PROMETHEUS 7B
0.6250
0.5361
0.6400
Large (14B+ parameters)
Qwen2.5-14D-Instruu.
0.7500
0.6289
0.8133
PROMETHEUS 2 8x7B
0.7750
0.7062
0.7467
M-PROMETHEUS 14B
0.5750
0.6392
0.7333
Backbone Model Ablations
Mistral-7B-v0.--nnsttuu
0.6000
0.6186
0.5733
EuroLLM-9B-Instruue
0.6500
0,5670
0.6133
Aya-Expanse-8B Qwen2.5-77BIInsrruct
0.6750
0.5258
0.6533
0.5750
0.5464
0.6133
Training Data Ablations
MT Eval Data
0.6750
0.5876
0.6133
Translated Data
Non-English Langs
0.6500
0.5258
0.6133
Multilingual Data
3 Non-English Langs
0,4500
0,5052
0.6533
5 Non-English Langs
0.5000
0.5052
0.6667
Table 7: Accuracy on MMEval (Catalan) broken down by category.
30

---

Published as a conference paper at COLM 2025
MMEval (Russian)
Model
Chat Language Hallucinations
Linguistics Reasoning Safety
Proprietary Models
GPT-40
0.7887
0.7640
Small (3B parameters)
Qwen2.5-3B-nnstruct
0.6901
0.5901
Glider 3B +
0.6479
0.6491
M-PROMETTEUS 3B
0.6197
0.5776
Medium (7B parameters)
Qwen2.5-7B-Instruct
0,6620
0.6646
PROMETHEUS 2 7B+
0.6338
0.5311
Hercule 7B
0.6408
0.5342
M-PROMETHEUS 7B
0.5634
0.6273
Large (14B+ parameters)
Qwen2.5-14D-Instruu.
0.7042
0.7143
PROMETHEUS 2 8x7B
0.6408
0.5870
M-PROMETHEUS 14B
0.7183
0.6957
Backbone Model Ablations
Mistral-7B-v0..-Instrrue
0.5915
0.5807
EuroLLM:9B-Instrue
0.6761
0.5683
Aya-Expanse-8B Qwen2.5-77-IInsrruct
0.6197
0.5776
0.6761
0.6491
Training Data Ablations
MT Eval Data
0.7183
0.6677
Translated Data
Non-English Langs
0.6479
0.6304
Multilingual Data
3 Non-English Langs
0.5493
0.6056
5 Non-English Langs
0.6479
0.5559
Table 8: Accuracy MMEval (Russian) broken down by category.
31

---

Published as a conference paper at COLM 2025
MMEval (Chinese)
Model
Chat
Language Hallucinations
Linguistics Reasoning Safety
Proprietary Models
GPT-40
0.8049
0.7881
Small (3B parameters)
Qwen2.5-3B-nnstruct
0,8293
0.6623
Glider 3B +
0.6463
0.5762
M-PROMETTEUS 3B
0.7927
0.6755
Medium (7B parameters)
Qwen2.5-7B-Instruct
0.8537
0.7152
PROMETHEUS 2 7B+
0.6341
0.5298
Hercule 7B
0.6385
0.5762
M-PROMETHEUS 7B
0.5854
0.6954
Large (14B+ parameters)
Qwen2.5-14B-Instrus
0.8780
0.7583
PROMETHEUS 2 8x7B
0.8049
0.6424
M-PROMETHEUS 14B
0.7317
0.7285
Backbone Model Ablations
Mistrall7B-v0..-IInsrruu
0.7317
0.5199
EuroLLM:9B-Instrue
0.6463
0.5960
Aya-Expanse-8B Qwen2.5-77BIInsrruct
0.7073
0.6623
0.6098
0.5861
Training Data Ablations
MT Eval Data
0.6585
0.5629
Translated Data
Non-English Langs
0.6385
0.6093
Multilingual Data
3 Non-English Langs
0.5854
0.5762
5 Non-English Langs
0.5122
0.5695
Table 9: Accuracy MMEval (Chinese) broken down by category.
32

---

Published as a conference paper at COLM 2025
MMEval (Arabic)
Model
Chat Language Hallucinations
Linguistics Reasoning Safety
Proprietary Models
GPT-40
0.7688
0.5326
Small (3B parameters)
Qwen2.5-3D-Instruct
0.5134
0,4620
Glider 3B
0.5108
0.4565
M-PROMETHEUS 3B
0.4624
0.6630
Medium (7B parameters)
Qwen2.5-7B-Instruct
0.4892
0.7609
PROMETHEUS 2 7B
0.5430
0.7663
Hercule 7B*
0.5054
0.2772
M-PROMETHEUS 7B*
0.5538
0,9565
Large (14B+ parameters)
Qwen2..-11B-instruuc
0.7608
0.4348
PROMETHEUS 2 8x7B
0.5484
0.5761
M-PROMETHEUS
0.8280
0.9348
Backbone Model Ablations
Mistral-7B-v0.2-Instrrue
0.5968
0.3261
EuroLLM-9B-Instrucc
0,5242
0.7391
Aya-Expanse-88 Qwen2.5-77-Innsrruct
0.5188
0.6196
0.6828
0.5978
Training Data Ablations
MT Eval Data
0.7043
0.7065
Translated Data
3 Non-Fngllih Langs
0.5860
0.5217
Multilingual Data
3 Non-English Langs
0.5753
0.8261
5 Non-English Langs
0.6667
0.8913
Table 10: Accuracy MMEval (Arabic) broken down by category.
33

---

Published as a conference paper at COLM 2025
MMEval (Bengali)
Model
Chat Language Hallucinations Linguistics Reasoning Safety
Proprietary Models
GPT-40
0.6298
0.8455
Small (3B parameters)
Qwen2.5-3B-Instruct
0.5525
0.6682
Glider 3B
0.5525
0.5705
M-PROMETHEUS 3B
0.5442
0.6182
Medium (7B parameters)
Owen2.5-7B-Instruct
0.5249
0.6000
PROMETHEUS 2 7B
0.4641
0.5523
Hercule 7B*
0.5166
0.5705
M-PROMETHEUS 7B
0.6381
0.6182
Large (14B+ parameters)
Qwen2.5-14D-Instrruct
0.6906
0.7273
PROMETHEUS 2 8x7B
0.4972
0.5727
M-PROMETHEUS 14B*
0.8011
0.7045
Backbone Model Ablations
Mistral-7B-v022-Instruuu
0.6851
0.5477
EuroLLM-9B-Instrucc
0,5691
0.5977
Aya-Expanse-8B Qwenn25-77--Insrruct
0.5746
0.5159
0.7707
0.5932
Training Data Ablations
MT Eval Data
0.8895
0.5909
Translated Data
3 Non-English Langs
0.7017
0.6068
Multilingual Data
3 Non-English Langs
0.7514
0.5955
5 Non-English Langs
0.7127
0.4977
Table 11: Accuracy on MMEval (Bengali) broken down by category.
34

---

Published as a conference paper at COLM 2025
MMEval (Basque)
Model
Chat Language Hallucinations Linguistics Reasoning Safety
Proprietary Models
GPT-40
0.6386
0.3871
Small (3B parameters)
Qwen2.5-3D-Instruct
0.4819
0,2688
Glider 3B
0.5060
0.4032
M-PROMETHEUS 3B
0.5181
0.3656
Medium (7B parameters)
Qwen2.5-7B-Instruct
0.5241
0.4624
PROMETHEUS 2 7B
0.6084
0.5161
Hercule 7B*
0.4819
0.2204
M-PROMETHEUS 7B*
0.5964
0.8172
Large (14B+ parameters)
Qwen2..-11B-instruuc
0.6205
0.3441
PROMETHEUS 2 8x7B
0.5271
0.3871
M-PROMETHEUS 14B*
0.6446
0.9462
Backbone Model Ablations
Mistral-7B-v0.2-Instruue
0.4759
0.4086
EuroLLM-9B-Instrucc
0,5181
0.6022
Aya-Expanse-88 Qwen2.5-77-Innsrruct
0.4699
0.3763
0.5000
0.5484
Training Data Ablations
MT Eval Data
0.7530
0.4731
Translated Data
3 Non-Fngllih Langs
0.4639
0.5914
Multilingual Data
3 Non-English Langs
0.4940
0.6237
5 Non-English Langs
0.5000
0.7742
Table 12: Accuracy on MMEval (Basque) broken down by category.
35

---

Published as a conference paper at COLM 2025
MMEval (Korean)
Model
Chat Language Hallucinations
Linguistics Reasoning Safety
Proprietary Models
GPT-40
0.7647
0.5054
Small (3B parameters)
Qwen2.5-3D-Instruct
0.5775
0,4355
Glider 3B
0.5107
0.4409
M-PROMETHEUS 3B
0.5722
0.7419
Medium (7B parameters)
Qwen2.5-7B-Instruct
0.5775
0.7634
PROMETHEUS 2 7B
0.4706
0.7957
Hercule 7B*
0.4893
0.3548
M-PROMETHEUS 7B*
0.6257
0,9785
Large (14B+ parameters)
Qwen2..-11B-instruuc
0.8610
0.5054
PROMETHEUS 2 8x7B
0.6524
0.7473
M-PROMETHEUS 14B*
0.8717
0.9677
Backbone Model Ablations
Mistral-7B-v0.2-Instrrue
0.4947
0.4946
EuroLLM-9B-Instrucc
0,5882
0.8065
Aya-Expanse-88 Qwen2.5-77-Innsrruct
0.5829
0.6237
0.7273
0.6559
Training Data Ablations
MT Eval Data
0.7594
0.7634
Translated Data
3 Non-Fngllih Langs
0.6203
0.5269
Multilingual Data
3 Non-English Langs
0.5615
0.8280
5 Non-English Langs
0.6364
0.9032
Table 13: Accuracy on MMEval (Korean) broken down by category.
36

---

Published as a conference paper at COLM 2025
MMEval (Vietnamese)
Model
Chat Language Hallucinations Linguistics Reasoning Safety
Proprietary Models
GPT-40
0.6804
0.5556
Small (3B parameters)
Qwen2.5-3D-Instruct
0.5206
0,5556
Glider 3B
0.4974
0.3944
M-PROMETHEUS 3B
0.5361
0.8000
Medium (7B parameters)
Qwen2.5-7B-Instruct
0.5464
0.7444
PROMETHEUS 2 7B
0.4742
0.7000
Hercule 7B*
0.5284
0.3222
M-PROMETHEUS 7B*
0.6031
0,9556
Large (14B+ parameters)
Qwen2..-11B-instruuc
0.7062
0.5222
PROMETHEUS 2 8x7B
0.6031
0.6556
M-PROMETHEUS 14B*
0.7912
0.9889
Backbone Model Ablations
Mistral-7B-vO.2-Instttu
0.4768
0.3667
EuroLLM-9B-Instrucc
0,5387
0.5944
Aya-Expanse-88 Qwen2.5-77-Innsrruct
0.5567
0.4889
0.6856
0.6556
Training Data Ablations
MT Eval Data
0.7526
0.8111
Translated Data
3 Non-Fngllih Langs
0.5876
0.6222
Multilingual Data
3 Non-English Langs
0.5670
0.8444
5 Non-English Langs
0.6598
0.9667
Table 14: Accuracy on MMEval (Vietnamese) broken down by category.
37

---

Published as a conference paper at COLM 2025
MMEval (Ttalian)
Model
Chat Language Hallucinations
Linguistics Reasoning Safety
Proprietary Models
GPT-40
0.9333
0.5111
Small (3B parameters)
Qwen2.5-3D-Instruct
0.6667
0.5389
Glider 3B
0.6533
0.6889
M-PROMETHEUS 3B
0.7200
0.7667
Medium (7B parameters)
Qwen2.5-7B-Instruct
0.6400
0,8000
PROMETHEUS 2 7B
0.6933
0.7000
Hercule 7B*
0.5533
0.3389
M-PROMETHEUS 7B*
0.6933
0.9778
Large (14B+ parameters)
Qwen2..-11B-instruuc
0.7333
0.4778
PROMETHEUS 2 8x7B
0.6800
0.8111
M-PROMETHEUS 14B*
0.7600
0.9556
Backbone Model Ablations
Mistral-7B-v0.2-Instruu
0.5867
0.4778
EuroLLM-9B-Instrucc
0.6933
0.8444
Aya-Expanse-88 Qwen2.5-77-Innsrruct
0.6933
0.5889
0.7333
0.6556
Training Data Ablations
MT Eval Data
0.6933
0.6889
Translated Data
3 Non-Fngllih Langs
0.7200
0.6333
Multilingual Data
3 Non-English Langs
0.6800
0.8889
5 Non-English Langs
0.6800
0.9667
Table 15: Accuracy MMEval (Italian) broken down by category.
38

---

Published as a conference paper at COLM 2025
MMEval (Galician)
Model
Chat Language Hallucinations
Linguistics Reasoning Safety
Proprietary Models
GPT-40
0.8800
0.5730
Small (3B parameters)
Qwen2.5-3D-Instruct
0.7000
0,5056
Glider 3B
0.6133
0.5393
M-PROMETHEUS 3B
0.6933
0.8202
Medium (7B parameters)
Qwen2.5-7B-Instruct
0.6000
0.8652
PROMETHEUS 2 7B
0.6933
0.7022
Hercule 7B*
0.5200
0.3427
M-PROMETHEUS 7B*
0.5600
0.9213
Large (14B+ parameters)
Qwen2..-11B-Instruuc
0.7200
0.5955
PROMETHEUS 2 8x7B
0.6667
0.7865
M-PROMETHEUS 14B*
0.6267
0.9663
Backbone Model Ablations
Mistral-7B-v0.2-Instrue
0.6533
0.5393
EuroLLM-9B-Instrucc
0.6133
0.7865
Aya-Expanse-88 Qwen2.5-77-Innsrruct
0.6800
0.6517
0.6933
0.6180
Training Data Ablations
MT Eval Data
0.6133
0.6067
Translated Data
3 Non-Fngllish Langs
0.6533
0.6292
Multilingual Data
3 Non-English Langs
0,5600
0.8315
5 Non-English Langs
0.6000
0.9213
Table 16: Accuracy MMEval (Galician) broken down by category.
39

---

Published as a conference paper at COLM 2025
MMEval (Japanese)
Model
Chat Language Hallucinations
Linguistics Reasoning Safety
Proprietary Models
GPT-40
0.8009
0.5604
Small (3B parameters)
Qwen2.5-3D-Instruct
0,5231
0.5604
Glider 3B
0.6065
0.4615
M-PROMETHEUS 3B
0.5741
0.8791
Medium (7B parameters)
Qwen2.5-7B-Instruct
0.6204
0.7253
PROMETHEUS 2 7B
0.5833
0.7088
Hercule 7B*
0.5463
0.3352
M-PROMETHEUS 7B*
0.5787
0,9560
Large (14B+ parameters)
Qwen2..-11B-instruuc
0.7083
0.5495
PROMETHEUS 2 8x7B
0.5602
0.6593
M-PROMETHEUS 14B*
0.7060
0.9670
Backbone Model Ablations
Mistral-7B-v0.2-Instrue
0.5278
0.5275
EuroLLM-9B-Instrucc
0.5486
0.8297
Aya-Expanse-88 Qwen2.5-77-Innsrruct
0.5972
0.5495
0.6088
0.6593
Training Data Ablations
MT Eval Data
0.6065
0.8462
Translated Data
3 Non-Fngllih Langs
0.5972
0.6264
Multilingual Data
3 Non-English Langs
0.6204
0.8681
5 Non-English Langs
0.5370
0.9451
Table 17: Accuracy on MMEval (Japanese) broken down by category.
40

---

Published as a conference paper at COLM 2025
MMEval (Thai)
Model
Chat Language Hallucinations
Linguistics
Reasoning Safety
Proprietary Models
GPT-40
0.7970
0.4945
Small (3B parameters)
Qwen2.5-3D-Instruct
0.5888
0,5055
Glider 3B
0.5711
0.3626
M-PROMETHEUS 3B
0.5787
0.8022
Medium (7B parameters)
Qwen2.5-7B-Instruct
0.6650
0.8022
PROMETHEUS 2 7B
0.5685
0.6813
Hercule 7B*
0.5228
0.3901
M-PROMETHEUS 7B*
0.6041
0,9670
Large (14B+ parameters)
Qwen2..-11B-Instruuc
0.6802
0.6044
PROMETHEUS 2 8x7B
0.6015
0.6154
M-PROMETHEUS 14B*
0.6853
0.9890
Backbone Model Ablations
Mistral-7B-vO.2-Instttue
0.5685
0.3791
EuroLLM-9B-Instrucc
0.6091
0,5000
Aya-Expanse-88 Qwen2.5-77-Innsrruct
0.5812
0.4615
0.6447
0.6868
Training Data Ablations
MT Eval Data
0.6548
0.7912
Translated Data
3 Non-Fngllih Langs
0.6574
0.6813
Multilingual Data
3 Non-English Langs
0.6041
0.8462
5 Non-English Langs
0.5431
0.9011
Table 18: Accuracy on MMEval (Thai) broken down by category.
41

---

Published as a conference paper at COLM 2025
MMEval (Telugu)
Model
Chat Language Hallucinations
Linguistics Reasoning Safety
Proprietary Models
GPT-40
0.8739
Small (3B parameters)
Qwen2.5-3B-Instruct
0.6014
Glider 3B
0.5766
M-PROMETHEUS 3B
0.6059
Medium (7B parameters)
Owen2.5-7B-Instruct
0.6779
PROMETHEUS 2 7B
0.5270
Hercule 7B*
0.5428
M-PROMETHEUS 7B
0.6577
Large (14B+ parameters)
Qwen2.5-14D-Instrruct
0.7320
PROMETHEUS 2 8x7B
0.6059
M-PROMETHEUS 14B*
0.7207
Backbone Model Ablations
Mistral-7B-vO22-Insttuu
0.5248
EuroLLM-9B-Instrucc
0.5000
Aya-Expanse-8B Qween25-77--Insrruct
0.5068
0.6419
Training Data Ablations
MT Eval Data
0.6239
Translated Data
3 Non-English Langs
0.6441
Multilingual Data
3 Non-English Langs
0.6892
5 Non-English Langs
0.5338
Table 19: Accuracy MMEval (Telugu) broken down by category.
42

---

Published as a conference paper at COLM 2025
MMEval (Swahili)
Model
Chat Language Hallucinations
Linguistics Reasoning Safety
Proprietary Models
GPT-40
0.8047
0.4624
Small (3B parameters)
Qwen2.5-3D-Instruct
0,5930
0.3011
Glider 3B
0.6721
0.4946
M-PROMETHEUS 3B
0.6651
0.3763
Medium (7B parameters)
Qwen2.5-7B-Instruct
0.6186
0.4086
PROMETHEUS 2 7B
0.6628
0.4839
Hercule 7B*
0.5163
0.1774
M-PROMETHEUS 7B*
0.7209
0.7634
Large (14B+ parameters)
Qwen2..-11B-instruuc
0.7256
0.4516
PROMETHEUS 2 8x7B
0.6302
0.4624
M-PROMETHEUS 14B*
0.7628
0.9462
Backbone Model Ablations
Mistral-7B-v0.2-Instrue
0.6651
0.4194
EuroLLM-9B-Instrucc
0.6419
0.5591
Aya-Expanse-88 Qwen2.5-77-Innsrruct
0.6744
0.3226
0.7302
0.5806
Training Data Ablations
MT Eval Data
0.6977
0.5161
Translated Data
3 Non-Fngllih Langs
0.6860
0.6022
Multilingual Data
3 Non-English Langs
0.6791
0.6129
5 Non-English Langs
0.5884
0.7527
Table 20: Accuracy MMEval (Swahili) broken down by category.
43

---

Published as a conference paper at COLM 2025
C.2 M-RewardBench
M-RewardBench (Arabic)
Model
Chat
Chat Hard
Reasoning Safety
Proprietary Models
GPT-40
0.9628
0.6658
0.8790
0.8668
Small (3B parameters)
Owen2.5-3B-Instrrcs Glider 3B +
0.8041
0.4558
0.6577
0.6889
0.5422
0.5061
0.7878
0.6257
M- M-PROMETHEUS 3B
0.8666
0.4165
0.6657
0.7636
Medium (7B parameters)
Qwen2.5-77-Instruuct
0.9088
0.5332
0.8094
0.7799
PROMETHEUS 2 7B
0.7027
0.4951
0.6888
0.6298
Hercule 7B
0.7990
0.4275
0.6720
0.6386
M-PROMETHEUS 7B
0.8818
0,5111
0.8252
0.7758
Large (14B+ parameters)
10om2..--4BBInstrrcc
0.9426
0.6093
0.8276
0.8234
PROMETHEUS 8x7B
0.8750
0.3931
0.8112
0.7432
M- M-PROMETHEUS 14B
0.8953
0.5197
0.8290
0.8234
Backbone Model Ablations
Mistral-7B-v0.2-Instttt
0.6841
0.4619
0.6587
0.4885
-uroI.LM-9B-Insttruct
0.8767
0.4423
0.7364
0.8179
Aya- Aya-Expanse-8B Qwenn..-7B-Instrucc
0.8682
0.4300
0.7490
0.8057
0.8784
0,4791
0.8455
0.7948
Training Data Ablations
MT Eval Data
0.8851
0,4963
0.8378
0,8098
Translated Data
Non-English Langs
0.8885
0.5086
0.8315
0.8261
Multilingual Data
Non-English Langs
0.7901
0.5591
0.8422
0.7883
Non-English Langs
0.8209
0.5135
0.8294
0.7840
Table 21: Accuracy M- MenewandBench (Arabic) broken down by category.
44

---

Published as a conference paper at COLM 2025
M-RewardBench (Czech)
Model
Chat
Chat Hard
Reasoning Safety
Proprietary Models
GPT-40
0.9628
0.6683
0.9073
0.8614
Small (3B parameters)
Qwen2.5-30-Instruuc
0.8125
0,4079
0.6829
0.6984
Glider 3B
0.5997
0.4324
0.8476
0.5863
M-PROMETHEUS 3B
0.8750
0.4423
0.6776
0.7255
Medium (7B parameters)
Owen22.-7B-Instrucc
0.9135
0,4889
0.8028
0.7799
PROMETHEUS 7B
0.8260
0.4472
0.6867
0.6705
Hercule 7B *
0.8328
0.4128
0.6629
0.6243
M- M-PROMETHEUS
0.9088
0.4681
0.8052
0.7867
Large (14B+ parameters)
Owenn.55-4B-Instrucct
0.9189
0.5430
0.8273
0.7772
PROMETHEUS 8x7B
0.9037
0.4054
0.8115
0.7758
M- M-PROMETHEUS 14B
0.9088
0.5012
0.8371
0.8302
Backbone Model Ablations
Mistral-7B-vvO.-Innttee
0.8514
0.4005
0.6794
0.6658
EuroLLM-9B-Instruuc
0.8632
0.4398
0.7350
0,8030
Aya-Expanse-8B Qwen2.5-7B-
0.8733
0,4201
0.7766
0.7731
0.8784
0.4693
0.8231
0.7704
Training Data Ablations
MT Eval Data
0.9223
0,4767
0.8150
0.7731
Translated Data
Non-English Langs
0.9088
0.4742
0.8329
0.7785
Multilingual Data
Non-English Langs
0.8618
0.5267
0.8386
0.7842
Non-English Langs
0.8514
0.4988
0.8367
0.7880
Table 22: Accuracy on M-RewardBench (Czech) broken down by category.
45

---

Published as a conference paper at COLM 2025
M-RewardBench (German) Reasoning Safety
Model
Chat
Chat Hard
Proprietary Models
GPT-40
0.9493
0.6830
0.8860
0.8560
Small (3B parameters)
Qwen2.5-30-Instrucc
0.8378
0,4668
0.6951
0.6834
Glider 3B
0.6622
0.5418
0.8762
0.7296
M-PROMETHEUS 3B
0.8615
0.4214
0.7003
0.7507
Medium (7B parameters)
Owen22.-7B-Instruct
0.9439
0.5430
0.8290
0.7758
PROMETHEUS 7B
0.8429
0.4754
0.7196
0.6902
Hercule 7B *
0.8142
0.4533
0.6857
0.6501
M- M-PROMETHEUS
0.9088
0.5233
0.8262
0.8030
Large (14B+ parameters)
Owenn.55-4B-Instrucct
0.9493
0.6437
0.8476
0.8234
PROMETHEUS 8x7B
0.9088
0.4509
0.8210
0.7704
M- M-PROMETHEUS 14B
0.9358
0.5221
0.8399
0.8261
Backbone Model Ablations
Mistral-7B-vvO.-Innttee
0.8851
0.4103
0.7010
0.6766
EuroLLM-9B-Instruuc
0.9037
0,4410
0.7332
0,8152
Aya-Expanse-8B Qwen2.5-7B-
0.8598
0.4177
0.7706
0.7955
0.8885
0.5061
0.8276
0.7989
Training Data Ablations
MT Eval Data
0.9088
0,4865
0.8220
0.7948
Translated Data
Non-English Langs
0.9088
0.4914
0.8402
0.8003
Multilingual Data
Non-English Langs
0.8492
0,5639
0.8338
0,8035
Non-English Langs
0.8480
0.5184
0.8378
0.8043
Table 23: Accuracy M-RewardBench (German) broken down by category.
46

---

Published as a conference paper at COLM 2025
M-RewardBench (Greek)
Model
Chat
Chat Hard
Reasoning Safety
Proprietary Models
GPT-40
0.9527
0.6364
0.8818
0.8682
Small (3B parameters)
Qwen2.5-30-Instrucc
0.7500
0,4201
0.6878
0,6365
Glider 3B
0.5777
0.4582
0.7262
0.5136
M-PROMETHEUS 3B
0.8581
0.4079
0.6675
0.6984
Medium (7B parameters)
Qwen22.-7B-Instrucc
0.8919
0.5111
0.8059
0.7323
PROMETHEUS 7B
0.6571
0.4558
0.6965
0.5897
Hercule 7B *
0.8446
0.4398
0.6860
0.6345
M- M-PROMETHEUS
0.8716
0.4975
0.7913
0.7452
Large (14B+ parameters)
Owenn.55-4B-Instrucct
0.9358
0.5651
0.8210
0.7799
PROMETHEUS 8x7B
0.8784
0.3993
0.8056
0.7473
M- M-PPRMETTEUS 14B
0.8936
0.4742
0.8175
0.8152
Backbone Model Ablations
Misttal-7B-vv..-Innttte
0.6976
0.4533
0.6678
0.5129
EuroLLM-9B-Instruuc
0.8818
0,4349
0.7406
0.7867
Aya-Expanse-8B Qwen2.5-7B-
0.8716
0,4275
0.7626
0.7758
0.8750
0.4816
0.8140
0.7446
Training Data Ablations
MT Eval Data
0.8919
0.4496
0.8042
0.7323
Translated Data
Non-English Langs
0.8716
0.4693
0.8262
0.7636
Multilingual Data
Non-English Langs
0.8452
0,5095
0.8415
0.7476
Non-English Langs
0.8682
0,4619
0.8189
0.7432
Table 24: Accuracy on M-RcwardBenchh (Greek) broken down by category.
47

---

Published as a conference paper at COLM 2025
M-RewardBBechh (French)
Model
Chat
Chat Hard
Reasoning Safety
Proprietary Models
GPT-40
0.9730
0.6658
0.9024
0.8614
Small (3B parameters)
Qwen2.5-30-Instruuct
0.8564
0,4853
0.7021
0.6936
Glider 3B
0.6757
0.4988
0.8797
0.7622
M-PROMETHEUS 3B
0.8514
0.4361
0.6913
0.7670
Medium (7B parameters)
Qwen22.-7B-Instrucc
0.9493
0.5627
0.8406
0.7921
PROMETHEUS 7B
0.8378
0.4582
0.6888
0.7500
Hercule 7B *
0.8277
0.4570
0.6822
0.6393
M- M-PROMETHEUS
0.9105
0.5369
0.8514
0.8050
Large (14B+ parameters)
Owen2.55-4B-Instrucct
0.9392
0.6327
0.8455
0.8152
PROMETHEUS 8x7B
0.9324
0.4484
0.8220
0.7982
M- M-PROMETHEUS 14B
0.9223
0.5356
0.8406
0.8438
Backbone Model Ablations
Mistral-7B-vvO2-Innttee
0.8649
0.4251
0.7017
0.7181
EuroLLM-9B-Instruuc
0.8902
0,4717
0.7566
0,8084
Aya-Expanse-8B Qwen2.5-7B-
0.8784
0,4251
0.7762
0.8003
0.8953
0.5111
0.8601
0.8139
Training Data Ablations
MT Eval Data
0.9088
0.5135
0.8462
0.8071
Translated Data
Non-English Langs
0.9020
0.4988
0.8465
0.8234
Multilingual Data
Non-English Langs
0.8111
0,5593
0.8485
0.7940
Non-English Langs
0.8345
0.5479
0.8399
0.8071
Table 25: Accuracy M-RewardBench (French) broken down by category.
48

---

Published as a conference paper at COLM 2025
M-RewardBench (Hebrew)
Model
Chat
Chat Hard
Reasoning Safety
Proprietary Models
GPT-40
0.9493
0.6560
0.8888
0.8641
Small (3B parameters)
Qwen2.5-30-Instrucc
0.8074
0,4238
0.6650
0.6664
Glider 3B
0.5389
0.4914
0.7829
0.5734
M-PROMETHEUS 3B
0.8682
0.4570
0.6766
0.7378
Medium (7B parameters)
Owen22.-7B-Instruct
0.9358
0,4668
0.7986
0.7459
PROMETHEUS 7B
0.6385
0.4877
0.6871
0.5999
Hercule 7B *
0.8041
0.4251
0.6720
0.6270
M- M-PROMETHEUS
0.8885
0.4730
0.8287
0.7785
Large (14B+ parameters)
Owenn.55-4B-Instrucct
0.9088
0.5762
0.8399
0.7779
PROMETHEUS 8x7B
0.8632
0.3673
0.8017
0.7120
M- M-PROMETHEUS 14B
0.8953
0.5049
0.8192
0.8111
Backbone Model Ablations
Mistral-7B-v0.2-Innttee
0.6875
0.4779
0.6836
0.5000
EuroLLM-9B-Instruuc
0.7264
0,4337
0.7143
0.6664
Aya-Expanse-8B Qwen2.5-7B-
0.8818
0.4054
0.7577
0.7717
0.8767
0.4423
0.8423
0.7690
Training Data Ablations
MT Eval Data
0.9088
0.3931
0.8325
0.7867
Translated Data
Non-English Langs
0.9155
0.4398
0.8374
0.7908
Multilingual Data
Non-English Langs
0.8387
0,4943
0.8531
0.7653
Non-English Langs
0.8716
0.4914
0.8371
0.7826
Table 26: Accuracy M-RewardBench (Hebrew) broken down by category.
49

---

Published as a conference paper at COLM 2025
M-RewardBench (Hindi)
Model
Chat
Chat Hard
Reasoning Safety
Proprietary Models
GPT-40
0.9561
0.6241
0.8804
0.8560
Small (3B parameters)
Qwen2.5-30-Instrucc
0.7264
0,4214
0.7108
0,6495
Glider 3B
0.5963
0.4840
0.5322
0.5000
M-PROMETHEUS 3B
0.8176
0.4091
0.6832
0.7079
Medium (7B parameters)
Owen22.-7B-Instruct
0.8953
0,4914
0.8161
0.7514
PROMETHEUS 7B
0.6537
0.4717
0.6850
0.5978
Hercule 7B *
0.8429
0.4361
0.6878
0.6488
M- M-PROMETHEUS
0.8480
0.4631
0.8059
0.7541
Large (14B+ parameters)
Owenn.55-4B-Instrucct
0.9291
0.5258
0.8357
0.7989
PROMETHEUS 8x7B
0.8581
0.3943
0.7825
0.7432
M- M-PPRMETTEUS 14B
0.9071
0.4914
0.8462
0.7989
Backbone Model Ablations
Misttal-7B-vv..-Innttte
0.5794
0.4988
0.6573
0.5170
EuroLLM-9B-Instruuc
0.8125
0.4423
0.7605
0,8152
Aya-Expanse-8B Qwen2.5-7B-
0.7703
0,4238
0.7364
0.7554
0.8041
0.4889
0.8343
0.7908
Training Data Ablations
MT Eval Data
0.8547
0,4300
0.8322
0.7867
Translated Data
Non-English Langs
0.8311
0.4791
0.8287
0.8247
Multilingual Data
Non-English Langs
0.7710
0,5192
0.8625
0.7707
Non-English Langs
0.8547
0,4521
0.8147
0.7514
Table 27: Accuracy M-RewardBench (Hindi) broken down by category.
50

---

Published as a conference paper at COLM 2025
M-RewardBenchh (Indonesian) Reasoning Safety
Model
Chat
Chat Hard
Proprietary Models
GPT-40
0.9561
0.6634
0.8965
0.8723
Small (3B parameters)
Qwen2.5-30-Instrucc
0.8497
0,4496
0.7087
0.7249
Glider 3B
0.6926
0.5086
0.7689
0.6556
M-PROMETHEUS 3B
0.8514
0.4472
0.6867
0.7582
Medium (7B parameters)
Qwen22.-7B-Instruct
0.9324
0.5258
0.8269
0.7989
PROMETHEUS 7B
0.7905
0.4214
0.7038
0.7317
Hercule 7B *
0.8294
0.4177
0.6755
0.6719
M- M-PROMETHEUS
0.8919
0,5012
0.8206
0.7962
Large (14B+ parameters)
Owenn.55-4B-Instrucct
0.9324
0.6192
0.8434
0.8193
PROMETHEUS 8x7B
0.8801
0.4066
0.8066
0.8050
M- M-PPROETTEUSS 14B
0.9088
0.5565
0.8469
0.8356
Backbone Model Ablations
Mistral-7B-vvO2-Insttee
0.6824
0.4619
0.6860
0.6617
EuroLLM-9B-Instruuc
0.7669
0,4545
0.7510
0.7921
Aya-Expanse-8B Qwen2.5-7B-
0.8176
0.4447
0.7497
0,8193
0.8851
0.4865
0.8357
0.8193
Training Data Ablations
MT Eval Data
0.8818
0,4840
0.8413
0.8288
Translated Data
Non-English Langs
0.8682
0.4865
0.8311
0.8397
Multilingual Data
Non-English Langs
0.7596
0,5619
0.8499
0.7980
Non-English Langs
0.8209
0.5012
0.8402
0.8003
Table 28: Accuracy M-RewardBench (Indonesian) broken down by category.
51

---

Published as a conference paper at COLM 2025
M-RewardBench (Italian)
Model
Chat
Chat Hard
Reasoning Safety
Proprietary Models
GPT-40
0.9595
0.6658
0.9014
0.8791
Small (3B parameters)
Qwen2.5-30-Instrucc
0.8615
0,4521
0.6759
0.6970
Glider 3B
0.6554
0.4853
0.8780
0.7615
M-PROMETHEUS 3B
0.8750
0.4226
0.6944
0.7717
Medium (7B parameters)
Owen22.-7B-Instrucc
0.9493
0.5528
0.8381
0.7853
PROMETHEUS 7B
0.8514
0.4619
0.6986
0.7514
Hercule 7B *
0.8176
0.4533
0.6850
0.6685
M- M-PROMETHEUS
0.9189
0.5160
0.8371
0.8084
Large (14B+ parameters)
Owenn.55-4B-Instrucct
0.9358
0.6413
0.8465
0.8125
PROMETHEUS 8x7B
0.9291
0.4484
0.8038
0.8077
M- M-PROMETHEUS 14B
0.9054
0.5381
0.8399
0.8234
Backbone Model Ablations
Mistral-7B-vvO.-Innttee
0.8615
0.4115
0.6783
0.6957
EuroLLM-9B-Instruuc
0.8986
0,4619
0.7549
0,8302
Aya-Expanse-8B Qwen2.5-7B-
0.8986
0.4201
0.7990
0.8084
0.8986
0.5061
0.8451
0.8071
Training Data Ablations
MT Eval Data
0.9122
0,4717
0.8336
0,8043
Translated Data
Non-English Langs
0.9020
0.5012
0.8434
0.8220
Multilingual Data
Non-English Langs
0.8452
0.5649
0.8350
0,7802
Non-English Langs
0.8547
0.5086
0.8437
0.7867
Table 29: Accuracy M-RewardBench (Italian) broken down by category.
52

---

Published as a conference paper at COLM 2025
M-RewardBench (Japanese)
Model
Chat
Chat Hard
Reasoning Safety
Proprietary Models
GPT-40
0.9459
0.6609
0.9028
0.8696
Small (3B parameters)
Qwen2.5-30-Instruuct
0.8007
0,4410
0.6706
0.7024
Glider 3B
0.5912
0.4914
0.8147
0.6875
M-PROMETHEUS 3B
0.7601
0.4730
0.6731
0.7690
Medium (7B parameters)
Qwen22.-7B-Instrucc
0.9122
0.5283
0.8297
0.8084
PROMETHEUS 7B
0.7264
0.4939
0.7091
0.7018
Hercule 7B *
0.8159
0.4693
0.6535
0.6332
M- M-PROMETHEUS
0.7872
0.5516
0.8273
0.7731
Large (14B+ parameters)
Owen2.55-4B-Instrucct
0.9375
0.6265
0.8493
0.8288
PROMETHEUS 8x7B
0.8716
0.4152
0.7937
0.7996
M- M-PPROETTEUS 14B
0.8412
0.5479
0.8490
0.8193
Backbone Model Ablations
Mistral-7B-vO..2Insttee
0.6740
0.4779
0.6528
0.6855
EuroLLM-9B-Instruuc
0.7872
0,4668
0.7245
0,8071
Aya-Expanse-8B Qwen2.5-7B-
0.8277
0,4300
0.7329
0.7772
0.8007
0.4889
0.8301
0.8166
Training Data Ablations
MT Eval Data
0.7804
0,5012
0.8367
0.8410
Translated Data
Non-English Langs
0.8209
0.4742
0.8171
0.8370
Multilingual Data
Non-English Langs
0.7297
0,5781
0.8191
0.7842
Non-English Langs
0.7838
0.5479
0.8087
0.7731
Table 30: Accuracy on M-RewardBench (Japanese) broken down by category.
53

---

Published as a conference paper at COLM 2025
M-RewardBench (Korean)
Model
Chat
Chat Hard
Reasoning Safety
Proprietary Models
GPT-40
0.9493
0.6179
0.8867
0.8519
Small (3B parameters)
Qwen2.5-30-Instruuc
0.8108
0,4165
0.6339
0.6189
Glider 3B
0.5507
0.4717
0.7773
0.6182
M-PROMETHEUS 3B
0.8784
0.4165
0.6797
0.7188
Medium (7B parameters)
Owen22.-7B-Instruct
0.9628
0.5184
0.8042
0.7812
PROMETHEUS 7B
0.7787
0.4582
0.7101
0.7344
Hercule 7B *
0.8125
0.4459
0.6563
0.6481
M- M-PROMETHEUS
0.8345
0.5381
0.8129
0.7466
Large (14B+ parameters)
Owenn.55-4B-Instrucct
0.9493
0.5921
0.8357
0.8030
PROMETHEUS 8x7B
0.8851
0.3686
0.7748
0.7758
M- M-PROMETHEUS 14B
0.8885
0.5553
0.8476
0.7935
Backbone Model Ablations
Misttal-7B-vv..-Innttte
0.7804
0.4226
0.6675
0.6671
EuroLLM-9B-Instruuc
0.8463
0,4423
0.6986
0.7853
Aya-Expanse-8B Qwen2.5-7B-
0.8581
0,4373
0.7283
0.7704
0.8649
0.4619
0.8052
0.7880
Training Data Ablations
MT Eval Data
0.8716
0,4693
0.8224
0.7799
Translated Data
Non-English Langs
0.8716
0.4619
0.7937
0.8084
Multilingual Data
Non-English Langs
0.7918
0,5404
0.8075
0.7514
Non-English Langs
0.8277
0.5430
0.8108
0.7677
Table 31: Accuracy M-RewardBench (Korean) broken down by category.
54

---

Published as a conference paper at COLM 2025
M-RewardBench (Dutch)
Model
Chat
Chat Hard
Reasoning Safety
Proprietary Models
GPT-40
0.9358
0.6683
0.9024
0.8601
Small (3B parameters)
Qwen2.5-30-Instruuct
0.8311
0,4693
0.6846
0.6963
Glider 3B
0.6875
0.5074
0.8724
0.7188
M-PROMETHEUS 3B
0.8851
0.4361
0.6969
0.7541
Medium (7B parameters)
Owen22.-7B-Instruct
0.9439
0.5258
0.8329
0.7677
PROMETHEUS 7B
0.8615
0.4386
0.7297
0.7337
Hercule 7B *
0.8463
0.4545
0.6755
0.6664
M- M-PROMETHEUS
0.8986
0.4902
0.8294
0.8091
Large (14B+ parameters)
Owenn.55-4B-Instrucct
0.9493
0.6413
0.8455
0.8152
PROMETHEUS 8x7B
0.9257
0.4251
0.8063
0.8077
M- M-PPRMETTEUS 14B
0.9257
0.5295
0.8329
0.8315
Backbone Model Ablations
Mistral-7B-vvO.-Innttee
0.8632
0.4226
0.6916
0.6997
EuroLLM-9B-Instruuc
0.8885
0,4521
0.7573
0,8145
Aya-Expanse-8B Qwen2.5-7B-
0.8750
0,4128
0.7696
0.7976
0.8953
0.4767
0.8406
0.8057
Training Data Ablations
MT Eval Data
0.9223
0,4644
0.8392
0.8139
Translated Data
Non-English Langs
0.9088
0.4914
0.8451
0.8057
Multilingual Data
Non-English Langs
0.8483
0,5492
0.8461
0.7912
Non-English Langs
0.8615
0.5160
0.8406
0.8043
Table 32: Accuracy M-RewardBench (Dutch) broken down by category.
55

---

Published as a conference paper at COLM 2025
M-RewardBench (Persian)
Model
Chat
Chat Hard
Reasoning Safety
Proprietary Models
GPT-40
0.9426
0.6069
0.8818
0.8397
Small (3B parameters)
Qwen2.5-30-Instrucc
0.7534
0,4300
0.6871
0.6569
Glider 3B
0.5642
0.4840
0.7472
0.5414
M-PROMETHEUS 3B
0.8801
0.4226
0.6741
0.7303
Medium (7B parameters)
Owen22.-7B-Instruct
0.9088
0,4939
0.8017
0.7486
PROMETHEUS 7B
0.6824
0.4963
0.6839
0.6114
Hercule 7B *
0.8176
0.4214
0.6724
0.6467
M- M-PROMETHEUS
0.8750
0.4668
0.7976
0.7391
Large (14B+ parameters)
Owenn.55-4B-Instrucct
0.9291
0.5455
0.8259
0.7826
PROMETHEUS 8x7B
0.8767
0.3882
0.7811
0.7269
M- M-PROMETHEUS 14B
0.9054
0.4865
0.8378
0.7799
Backbone Model Ablations
Mistral-7B-v0..-Innttee
0.6706
0.4681
0.6766
0.5469
EuroLLM-9B-Instruuc
0.7703
0,4029
0.6972
0,7004
Aya-Expanse-8B Qwen2.5-7B-
0.8649
0,4349
0.7503
0.7541
0.8716
0.4595
0.8150
0.7595
Training Data Ablations
MT Eval Data
0.8649
0,4251
0.8210
0.7731
Translated Data
Non-English Langs
0.8682
0.4373
0.8276
0.7704
Multilingual Data
Non-English Langs
0.7928
0,5350
0.8188
0.7516
Non-English Langs
0.8378
0.5086
0.8168
0.7609
Table 33: Accuracy on M-RewardBench (Persian) broken down by category.
56

---

Published as a conference paper at COLM 2025
M-RewardBench (Polish)
Model
Chat
Chat Hard
Reasoning Safety
Proprietary Models
GPT-40
0.9493
0.6732
0.8923
0.8587
Small (3B parameters)
Qwen2.5-30-Instruuct
0.8277
0.3870
0.7042
0,6834
Glider 3B
0.6064
0.4619
0.8406
0.6637
M-PROMETHEUS 3B
0.8818
0.4509
0.6769
0.7391
Medium (7B parameters)
Qwen22.-7B-Instrucc
0.9527
0.5160
0.8238
0.8016
PROMETHEUS 7B
0.8530
0.4582
0.6850
0.6977
Hercule 7B *
0.8412
0.4054
0.6664
0.6372
M- M-PROMETHEUS
0.8851
0.5319
0.8154
0.7867
Large (14B+ parameters)
Owenn.55-4B-Instrucct
0.9358
0.5995
0.8402
0.7962
PROMETHEUS 8x7B
0.9155
0.4275
0.8052
0.7887
M- M-PPROETTEUS 14B
0.8885
0.5491
0.8413
0.8288
Backbone Model Ablations
Misttal-7B-vv..-Innttte
0.8176
0.4005
0.6794
0.6712
EuroLLM-9B-Instruuc
0.8699
0,4668
0.7325
0,8084
Aya-Expanse-8B Qwen2.5-7B-
0.8818
0,4300
0.7622
0.7874
0.8784
0.4619
0.8395
0.7894
Training Data Ablations
MT Eval Data
0.8885
0,4521
0.8497
0.7976
Translated Data
Non-English Langs
0.8953
0.4705
0.8126
0.8016
Multilingual Data
Non-English Langs
0.8167
0,5310
0.8202
0.7706
Non-English Langs
0.8716
0.5283
0.8252
0.7799
Table 34: Accuracy M-RewardBeenh (Polish) broken down by category.
57

---

Published as a conference paper at COLM 2025
M-RewardBench (Portuguese) Reasoning Safety
Model
Chat
Chat Hard
Proprietary Models
GPT-40
0.9527
0.6560
0.9063
0.8750
Small (3B parameters)
Qwen2.5-30-Instrucc
0.8345
0,4791
0.7227
0.7052
Glider 3B
0.7044
0.5221
0.8825
0.7711
M-PROMETHEUS 3B
0.8750
0.4189
0.6864
0.7758
Medium (7B parameters)
Owen22.-7B-Instrucc
0.9527
0.5381
0.8280
0.7935
PROMETHEUS 7B
0.8716
0.4631
0.7241
0.7724
Hercule 7B *
0.8395
0.4558
0.6895
0.6753
M- M-PROMETHEUS
0.8953
0.5504
0.8535
0.8043
Large (14B+ parameters)
Owenn.55-4B-Instrucct
0.9527
0.6609
0.8441
0.8356
PROMETHEUS 8x7B
0.9155
0.4459
0.8112
0.8132
M- M-PROMETHEUS 14B
0.9088
0.5504
0.8451
0.8370
Backbone Model Ablations
Mistral-7B-vv..-Innttte
0.8784
0.4005
0.6934
0.6943
EuroLLM-9B-Instruuc
0,9003
0,4816
0.7643
0,8247
Aya-Expanse-8B Qwen2.5-7B-
0.8818
0.4644
0.7710
0.8179
0.9206
0.5160
0.8605
0.8152
Training Data Ablations
MT Eval Data
0.9135
0,5012
0.8476
0.8057
Translated Data
Non-English Langs
0.9088
0.5184
0.8573
0.8125
Multilingual Data
Non-English Langs
0.8200
0,5849
0.8438
0.7979
Non-English Langs
0.8446
0.5430
0.8469
0.7826
Table 35: Accuracy M-RewardBench (Portuguese) broken down by category.
58

---

Published as a conference paper at COLM 2025
M-RewardBench (Romanian) Reasoning Safety
Model
Chat
Chat Hard
Proprietary Models
GPT-40
0.9493
0.6880
0.8909
0.8519
Small (3B parameters)
Qwen2.5-30-Instrucc
0.8311
0,4410
0.7136
0.7018
Glider 3B
0.6098
0.4767
0.8507
0.5802
M-PROMETHEUS 3B
0.8885
0.4300
0.6829
0.7459
Medium (7B parameters)
Qwen22.-7B-Instrucc
0.9223
0.5184
0.8353
0.7582
PROMETHEUS 7B
0.8361
0.4681
0.6857
0.6800
Hercule 7B *
0.8311
0.4386
0.6878
0.6236
M- M-PROMETHEUS
0.8851
0.4644
0.8182
0.7921
Large (14B+ parameters)
Owen2.55-4B-Instrucct
0.9392
0.5995
0.8336
0.8016
PROMETHEUS 8x7B
0.9291
0.3943
0.7962
0.7758
M- M-PPROETTEUS 14B
0.9122
0.5000
0.8276
0.8193
Backbone Model Ablations
Misttal-7B-vv..-Innttte
0.8446
0.4152
0.6853
0.6705
EuroLLM-9B-Instruuc
0.8885
0.4423
0.7552
0,8003
Aya-Expanse-8B Qwen2.5-7B-
0.8733
0,4324
0.7815
0.8003
0.8818
0.4570
0.8451
0.7962
Training Data Ablations
MT Eval Data
0.8919
0,4668
0.8416
0.7962
Translated Data
Non-English Langs
0.9003
0.4816
0.8413
0.7880
Multilingual Data
Non-English Langs
0.8495
0,5539
0.8408
0.7734
Non-English Langs
0.8649
0.5233
0.8381
0.7908
Table 36: Accuracy on M-RewardBench (Romanian) broken down by category.
59

---

Published as a conference paper at COLM 2025
M-RewardBench (Russian)
Model
Chat
Chat Hard
Reasoning Safety
Proprietary Models
GPT-40
0.9662
0.6732
0.8990
0.8709
Small (3B parameters)
Qwen2.5-30-Instrucc
0.8176
0,4926
0.7119
0.6916
Glider 3B
0.6824
0.4988
0.8780
0.7357
M-PROMETHEUS 3B
0.8615
0.4509
0.6979
0.7554
Medium (7B parameters)
Qwen2..-7B-Instruct
0.9392
0,5651
0.8360
0.7894
PROMETHEUS 7B
0.8429
0.4803
0.6874
0.7113
Hercule 7B *
0.8429
0.4349
0.6864
0.6562
M- M-PROMETHEUS
0.9054
0.5553
0.8409
0.7867
Large (14B+ parameters)
Owenn.55-4B-Instrucct
0.9493
0.6339
0.8399
0.8084
PROMETHEUS 8x7B
0.9223
0.4140
0.8255
0.7976
M- M-PROMETHEUS 14B
0.9122
0.5577
0.8371
0.8342
Backbone Model Ablations
Mistral-7B-v0..-Innttte
0.8716
0.4079
0.6864
0.6929
EuroLLM-9B-Instruuc
0.8885
0,4619
0.7668
0,8166
Aya-Expanse-8B Qwen2.5-7B-
0.8699
0,4349
0.7857
0.7989
0.9020
0.5061
0.8521
0.8152
Training Data Ablations
MT Eval Data
0.8851
0.5111
0.8441
0.8111
Translated Data
Non-English Langs
0.8970
0.4717
0.8483
0.8179
Multilingual Data
Non-English Langs
0.8295
0,5874
0.8561
0.7761
Non-English Langs
0.8345
0.5553
0.8458
0.7894
Table 37: Accuracy M-RewardBench (Russian) broken down by category.
60

---

Published as a conference paper at COLM 2025
M-RewardBench (Spanish)
Model
Chat
Chat Hard
Reasoning Safety
Proprietary Models
GPT-40
0.9561
0.7052
0.9035
0.8791
Small (3B parameters)
Qwen2.5-30-Instrucc
0.8193
0,4558
0.6955
0.7045
Glider 3B
0.6588
0.5135
0.8843
0.7602
M-PROMETHEUS 3B
0.8851
0.4312
0.6811
0.7785
Medium (7B parameters)
Owen22.-7B-Instrucc
0.9628
0.5381
0,8245
0.7758
PROMETHEUS 7B
0.8818
0.4484
0.7150
0.7615
Hercule 7B *
0.8361
0.4644
0.6902
0.6692
M- M-PROMETHEUS
0.9189
0.5197
0.8427
0.8016
Large (14B+ parameters)
Owenn.55-4B-Instrucct
0.9527
0.6536
0.8524
0.8071
PROMETHEUS 8x7B
0.9392
0.4496
0.8280
0.7976
M- M-PROMETHEUS 14B
0.9172
0.5418
0.8343
0.8268
Backbone Model Ablations
Mistral-7B-vvO.-Innttee
0.8818
0.4373
0.6888
0.6957
EuroLLM-9B-Instruuc
0.8953
0,4644
0.7668
0,8207
Aya-Expanse-8B Qwen2.5-7B-
0.8581
0,4545
0.7678
0,8071
0.8936
0.5184
0.8479
0.8193
Training Data Ablations
MT Eval Data
0.9054
0,4939
0.8455
0.8159
Translated Data
Non-English Langs
0.8986
0.5209
0.8542
0.8152
Multilingual Data
Non-English Langs
0.8274
0,5663
0.8507
0,7953
Non-English Langs
0.8615
0.5528
0.8266
0.8003
Table 38: Accuracy M-RewardBench (Spanish) broken down by category.
61

---

Published as a conference paper at COLM 2025
M-RewardBench (Turkish)
Model
Chat
Chat Hard
Reasoning Safety
Proprietary Models
GPT-40
0.9291
0.6118
0.8902
0.8410
Small (3B parameters)
Qwen2.5-30-Instruuct
0.7703
0,4287
0.6955
0.6501
Glider 3B
0.6250
0.4435
0.8374
0.4851
M-PROMETHEUS 3B
0.8902
0.4447
0.6668
0.7215
Medium (7B parameters)
Owen22.-7B-Instrucc
0.9257
0,4914
0.8035
0.7582
PROMETHEUS 7B
0.7787
0.4361
0.6941
0.6651
Hercule 7B *
0.8328
0.4287
0.6759
0.6121
M- M-PROMETHEUS
0.8480
0.4398
0.7972
0.7779
Large (14B+ parameters)
Owen2.55-4B-Instrucct
0.9392
0.5528
0.8227
0.7806
PROMETHEUS 8x7B
0.8395
0.3968
0.8042
0.7418
M- M-PROMETHEUS 14B
0.9139
0.4791
0.8189
0.8091
Backbone Model Ablations
Mistral-7B-vO..2Insttee
0.7973
0.4201
0.6752
0.5455
EuroLLM-9B-Instruuc
0.8936
0,4668
0.7479
0.7908
Aya-Expanse-8B Qwen2.5-7B-
0.8851
0,4103
0.7752
0.7853
0.8682
0.4717
0.8308
0.7663
Training Data Ablations
MT Eval Data
0.8750
0,4472
0.8168
0.7731
Translated Data
Non-English Langs
0.8986
0.4398
0.8178
0.7894
Multilingual Data
Non-English Langs
0.8103
0,5251
0.8310
0.7584
Non-English Langs
0.8446
0.5012
0.8066
0.7921
Table 39: Accuracy Bench (Turkish) broken down by category.
62

---

Published as a conference paper at COLM 2025
M-RewardBench (Ukrainian) Reasoning Safety
Model
Chat
Chat Hard
Proprietary Models
GPT-40
0.9561
0.6437
0.8913
0.8519
Small (3B parameters)
Qwen2.5-30-Instruuct
0.7804
0,4275
0.7000
0.7011
Glider 3B
0.6014
0.4263
0.8517
0.6624
M-PROMETHEUS 3B
0.8446
0.4300
0.6986
0.7473
Medium (7B parameters)
Qwen22.-7B-Instrucc
0.9189
0.5233
0.8325
0.7867
PROMETHEUS 7B
0.7821
0.4619
0.7035
0.7018
Hercule 7B *
0.8226
0.4300
0.6857
0.6556
M- M-PROMETHEUS
0.8986
0.5049
0.8245
0.7683
Large (14B+ parameters)
Owenn.55-4B-Instrucct
0.9257
0.5676
0.8434
0.8125
PROMETHEUS 8x7B
0.9274
0.4189
0.8196
0.7976
M- M-PPRMETTEUS 14B
0.9189
0.5381
0.8385
0.8084
Backbone Model Ablations
Mistral-7B-vv..-Innttte
0.8480
0.4201
0.6962
0.6766
EuroLLM-9B-Instruuc
0.8767
0,4201
0.7556
0,8166
Aya-Expanse-8B Qwen2.5-7B-
0.8750
0.4079
0.7636
0,8003
0.8750
0.4668
0.8497
0.7826
Training Data Ablations
MT Eval Data
0.8784
0,4767
0.8371
0.7690
Translated Data
Non-English Langs
0.8716
0.4791
0.8378
0.7935
Multilingual Data
Non-English Langs
0.8005
0,5505
0.8594
0,7610
Non-English Langs
0.8412
0.5356
0.8423
0.7758
Table 40: Accuracy M-RewaadBBenh (Ukrainian) broken down by category.
63

---

Published as a conference paper at COLM 2025
M-RewardBench (Vietnamese) Reasoning Safety
Model
Chat
Chat Hard
Proprietary Models
GPT-40
0.9595
0.6732
0.8997
0.8601
Small (3B parameters)
Qwen2.5-30-Instrucc
0.8446
0,4287
0.6990
0.7126
Glider 3B
0.5321
0.4840
0.7979
0.5870
M-PROMETHEUS 3B
0.8632
0.4509
0.6836
0.7758
Medium (7B parameters)
Owen22.-7B-Instrucc
0.9595
0.5430
0,8556
0.7812
PROMETHEUS 7B
0.7669
0.4398
0.7049
0.6861
Hercule 7B *
0.7973
0.4337
0.6731
0.6556
M- M-PROMETHEUS
0.8953
0.5344
0.8360
0.7921
Large (14B+ parameters)
Owenn.55-4B-Instrucct
0.9493
0.6314
0.8503
0.8098
PROMETHEUS 8x7B
0.8970
0.3919
0.8315
0.7534
M- M-PPROMETEUUS 14B
0.9020
0.5356
0.8455
0.8302
Backbone Model Ablations
Mistral-7B-vvO2-Innttee
0.7652
0.4373
0.6850
0.5251
EuroLLM-9B-Instruuc
0.7078
0,4128
0.7367
0.7099
Aya-Expanse-8B Qwen2.5-7B-
0.9054
0.3907
0.7724
0,8071
0.8767
0.4926
0.8497
0.8152
Training Data Ablations
MT Eval Data
0.8801
0,4840
0.8507
0.8234
Translated Data
Non-English Langs
0.8919
0.5061
0.8343
0.8179
Multilingual Data
Non-English Langs
0.7788
0,5935
0.8409
0.7814
Non-English Langs
0.8277
0.5553
0.8255
0.7867
Table 41: Accuracy BeRewardBeenc (Vietnamese) broken down by category.
64

---

Published as a conference paper at COLM 2025
M-RewardBench (Chinese (Simplified))
Model
Chat
Chat Hard
Reasoning
Safety
Proprietary Models
GPT-40
0.9561
0.6314
0.8899
0.8723
Small (3B parameters)
Qwen2.5-3D-Instruct
0.8277
0.4644
0.7070
0.6957
Glider 3B +
0.6216
0.4889
0.8524
0.7228
M-PROMETHEUS 3B
0.8581
0.4681
0.6902
0.7649
Medium (7B parameters)
Qwen2.5-7B-Instrucc
0.9426
0.5528
0.8364
0.7880
PROMETHEUS 2 7B
0.8176
0.4545
0.7164
0.6943
7B
0.8091
0.4521
0.6748
0.6413
M-PROMETHEUS 7B
0.8682
0.5676
0.8311
0.8003
Large (14B+ parameters)
Owenn..--1D-IIstrruct
0.9561
0.6339
0.8531
0.8315
PROMETHEUS 8x7B +
0.8919
0.4017
0.7860
0.7901
M-PROMETHEUS 14B
0.8919
0.5811
0.8388
0.8193
Backbone Model Ablations
Mistral-7B-vOO.-Instttu
0.8615
0.4029
0.6969
0.7024
EuroLLM-99-Instrrue
0.8530
0.4717
0.7374
0,8193
Aya-Expanse-8B Qwen2.5-77-Inntrruct
0.8649
0.4251
0.7476
0.7880
0.8615
0.5405
0.8556
0.8139
Training Data Ablations
MT Eval Data
0.9054
0.5307
0.8476
0.8084
Translated Data
Non-English Langs
0.8953
0.4865
0.8273
0.8139
Multilingual Data
Non-English Langs
0.7778
0,5875
0.8254
0,7856
5 Non-English Langs
0.8176
0.5651
0.8154
0.7935
Table 42: Accuracy on M-RewardBench (Chinese (Simplified)) broken down by category.
65

---

Published as a conference paper at COLM 2025
M-RewardBench (Chinese (Traditional))
Model
Chat
Chat Hard
Reasoning
Safety
Proprietary Models
GPT-40
0.9459
0.6511
0.8825
0.8451
Small (3B parameters)
Qwen2.5-3B-Instme
0.8699
0.4300
0.7087
0.6902
Glider 3B +
0.6976
0.4828
0.7402
0.6698
M-PROMETHEUS 3B
0.8378
0.4545
0.6811
0.7432
Medium (7B parameters)
Qwen22.57B-Instruutt
0.9223
0.5455
0.8094
0.7908
PROMETHEUS 2 7B+
0.7703
0.4619
0.7245
0.7126
Hercule 7B
0.8074
0.4656
0.6703
0.6522
M-PROMETHEUS
0.8581
0.5356
0.8346
0.8111
Large (14B+ parameters)
Qwen2.5-14B-Instruu
0.9493
0.6069
0.8322
0.8342
PROMETHEUS 8x7B
0.9105
0.4005
0.7804
0.7996
M-PROMETHEUS 14B
0.8750
0.5381
0.8105
0.8336
Backbone Model Ablations
Mistral-7B-vO..-Instttue
0.7551
0.4226
0.6843
0.7167
BunoLLM-9B-Instrucet
0.8412
0.4496
0,7360
0,8261
Aya-Expanse-8B Qwen2.5-7B-Insste
0.8108
0.4300
0.7448
0.8057
0.8446
0.5086
0.8458
0.8207
Training Data Ablations
MT Eval Data
0.8649
0.4988
0.8469
0.8234
Translated Data
Non-English Langs
0.8750
0.4791
0.8462
0.8247
Multilingual Data
Non-English Langs
0.7578
0.6036
0,8295
0.7925
Non-English Langs
0.8108
0.5553
0.8350
0,8003
Table 43: Accuracy M-RewardBench (Chinese (Traditional)) broken down by category.
66

---

Published as a conference paper at COLM 2025
C.3 RewardBench
Bench Chat Hard
Model
Chat
Reasoning Safety
Proprietary Models
GPT-40
0.9553
0.7259
0.8811
0.8760
Small (3B parameters)
Owen2.5-3B-Instrrcs
0.8687
0.5186
0.7149
0.6737
Glider 3B +
0.8156
0.4287
0.7703
0.7161
M- M-PROMETHEUS 3B
0.8785
0.4496
0.7851
0.7305
Medium (7B parameters)
Qwen2.5-77-Instrruct
0.9637
0.5713
0.7986
0.7957
PROMETHEUS 2 7B+
0.8550
0.4910
0.7710
0.7650
Hercule 7B
0.7905
0.5526
0.6243
0.7469
M-PROMETHEUS 7B
0.9078
0.5373
0.8419
0.8284
Large (14B+ parameters)
10om2..-14B-Instrucc
0.9525
0.6645
0.8365
0.8428
PROMETHEUS 8x7B
0.9330
0.4671
0.8101
0.7522
M- M-PROMETHEUS 14B
0.9358
0.5899
0.8514
0.8477
Backbone Model Ablations
Mistral-7B-v0.2-Insttee
0.8883
0.4474
0.7554
0.7422
-uroo.LM-9B-Insttuuct
0.9008
0.4693
0.8419
0.7956
Aya- Aya-Expanse-8B Qwenn..-7B-Instruect
0.8813
0.4912
0.8135
0.8265
0.9050
0.5110
0.8351
0,8583
Training Data Ablations
MT Eval Data
0.9078
0,5351
0,8216
0.8476
Translated Data
Non-English Langs
0.9050
0.5307
0.8284
0.8431
Multilingual Data
Non-English Langs
0.8687
0.5504
0.8473
0.8756
Non-English Langs
0.9106
0.5417
0.8514
0.8466
Table 44: Accuracy on RewardBench broken down by category.
67

---

Published as a conference paper at COLM 2025
C.4 LitEval
LitEval
Model
German--English English-GGerman English- Chinese German-+Chince
Proprietary Models
GPT-40
0.2680
0.4896
0.4189
0.4012
Small (3B parameters)
Qwen2.5-3B-Instrun
0.1195
0.2676
0.1208
0.1074
Glider 3B +
0.0992
0.2034
0.2285
0.1815
M-PROMETHEUS 3B
0.2203
0.5062
0.4411
0.4624
Medium (7B parameters)
Owen2.5-7B-Instrucct
0.1258
0.3289
0.1799
0.0743
PROMETHEUS 2 7B
0.1973
0.0854
0.0996
0.1184
Hercule 7B
0.2623
0.3340
0.3872
0.4231
M-PROMETHEUS 7B
0.2001
0.5321
0.4661
0.5427
Large (14B+ parameters)
Qwen2.5-14B-Instrre
0.2764
0.3901
0.2662
0.3102
PROMETHEUS 2 8x7B +
0.2479
0.3667
0.2561
0.4034
M-PROMETHEUS 14B*
0.2979
0.5702
0.4822
0.5658
Backbone Model Ablations
Mistral-7B-v0.2-Insttee .urssLt
0.1135
0.0384
0,0139
0.1328
0.1480
0.3767
0.1506
0.2987
Aya-Expanse-8B Qwen2.5-7B-Instrucct
0.0989
0.3293
0.2449
0.3446
0.2074
0.2850
0.3184
0.3240
Training Data Ablations
MT Eval Data
0.2541
0.3974
0.4786
0.5585
Translated Data
3 Non-Engliih Langs
0.1962
0.2266
0.1785
0.2872
Multilingual Data
3 Non-English Langs
0.1245
0.3424
0.3779
0.4201
5 Non-English Langs
0.1242
0.3856
0.4022
0.4369
Table 45: Kendall correlation on LitEval broken down by language pair.
68

---

Published as a conference paper at COLM 2025
C.5 QAD
QAD French Chinese
Model
Hindi
Small (3B parameters)
Qwen2.5-3B-Instruus
50.86
47.99
64.01
Glider 3B
56.02
46.12
69.49
M-PROMETHEUS 3B
60.77
51.44
76.91
Medium (7B parameters)
Qwes225577-Instruuct
52.73
46.41
68.51
PROMETHEUS 7B t
57.85
51.58
78.21
Hercule 7B
61.04
56.02
77.52
M-PROMETHEUS 7B
65.58
55.30
78.21
Large (14B+ parameters)
Qwen2.5-14B-Instruu
53.31
48.99
61.59
PROMETHEUS 8x7B t
61.59
51.87
74.91
M-PROMETHEUS 14B*
58.41
53.02
81.79
Backbone Model Ablations
Mistrrll7B-vo22-Insttuut
56.44
48.71
80.29
59.94
51.01
78.50
Aya-Expanse-88 Qwen2.5-7B-Instrucc
52.88
48.99
79.74
64.14
57.15
80,20
Training Data Ablations
MT Eval Data
57.85
49.28
71.99
Translated Data
3 Non-English Langs
63.61
55.16
80.65
Multilingual Data
Non-English Langs
66.10
57.71
82.30
Non-English Langs
64.93
58.27
75.12
Table 46: Win rate on QAD broken down by language.
D QAD Results with Gemma-2-2B-IT
In Table 47, we present QAD results when using judges to improve the outputs Gemma- 2-2B-IT, opposed to Qwen.
69

---

Published as a conference paper at COLM 2025
QAD Chinese
Model
French
Hindi
Small (3B parameters)
Qwen2.5-3B-Inee
50.72
49.28
48.27
Glider 3B
52.73
47.27
53.16
M-PROMETHEUS 3B
58.97
53.02
53.02
Medium (7B parameters)
Owws2.5-7B-Instruct
53.88
48.56
50.72
PROMETHEUS
59.25
56.16
58.55
Hercule
57.29
54.16
61.18
M-PROMETHEUS 7B*
59.25
47.99
59.39
Large (14B+ parameters)
Qwen2.5-14B-Instruct
54.16
50.43
50.29
PROMETHEUS 8x7B t
57.29
50.43
57.57
M-PROMETHEUS 14B*
55.59
49.57
54.45
Backbone Model Ablations
Mistral-7B-v022-nnstrutt
58.55
52.01
54.45
FuroLLLMt99-Instrruct
51.44
50.86
52.88
Aya-Expanse-SB Qwen2.5-7B-Instruct
55.59
48.56
55.02
56.44
50.29
57.57
Training Data Ablations
MT Eval Data
54.16
47.70
54.59
Translated Data
Non-English Langs
55.30
50.58
60.35
Multilingual Data
3 Non-English Langs
60.49
55.87
57.29
Non-English Langs
60.35
56.02
61.45
Table 47: Win rate on QAD on Demma-2-2B-11 broken down by language.
70


Reviews:
[{"review_id": "BWjwGN1jhD", "reviewer": "Reviewer_Aagm", "summary": "This paper presents a data creation approach for finetuning a multilingual LLM for the LLM-as-a-judge use case, specifically training it for direct assessment and pairwise comparison. The data synthesization is based on previous work, the English-only Prometheus-2 collection. The authors take the English score rubrics, and let a commercial LLM generate instructions of variying quality in multiple non-English languages. From this data pairwise preference data can be extracted as well. With this data, a open-weights LLM (Qwen2.5 in varying sizes) is fine-tuned. The resulting models are then compared on multiple benchmarks, including non-English centric reward benchmarks. The results show that the finetuned models do well on pairwise preferences in a multilingual setting, but fail on a translated variant of the popular RewardBench. On the other hand, the model does well on assessing literary translation.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Improvement of open-weight LLM-as-a-judge models for the multilingual case.", "reasons_to_reject": "- Unsure why the model would perform worse on translated data, even worse than the base model.", "questions_to_authors": "- Line 87: I wouldn't consider LLM-generated data as \"native\".\n- Line 95: Can you give an example in the running text?\n- Line 108: Can you describe how you deal with the fact that LLMs are still sensitive to the order in which the preference pairs are presented?\n- Table 1: Can you give more insight why the finetuned models would do worse on translated data?\n- Table 1: Can you also give more insight why the finetuned models would do so much better on the QAD task?", "extra_scores": {}}, {"review_id": "sdGUPsGQoo", "reviewer": "Reviewer_P6dq", "summary": "The authors describe a suite of LLMs (3B, 7B, 14B) based on Qwen2.5-Instruct and fine-tuned as multilingual quality judges.\nDifferent from previous work they evaluate on native non-English data and literary MT. In addition to describing the training recipe and overall quality experiments they provide ablation studies on the main drivers for multilingual performance. The authors also promise to release all models and code.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The released models are a clear benefit to the community. Some of the findings contradict previous results, most notably the non-effectiveness of translated training data for capabilities as a multilingual judge - it makes sense since previous work only evaluated on translated benchmarks. The ablation studies provide some relevant insights.", "reasons_to_reject": "It's unsatisfying and unclear why the 7B and 14B M-Prometheus models underperform their backbone Qwen2.5-Instruct on M-RewardBench, and on English RewardBench by such a large margin.", "questions_to_authors": "* Do you have an explanation for the underperformance on M-RewardBench?\n* Do you have an explanation for the underperformance in the Chat and Reasoning categories?", "extra_scores": {}}, {"review_id": "BFYmknN0Yf", "reviewer": "Reviewer_oHAe", "summary": "The paper describes a set of language models trained to perform automatic direct assessment and ranking of texts in different languages.\nTwo types of evaluation are performed: general capability (on 20 languages) and machine translation (on four language pairs).\nIn addition, applying automatic scores to improve the model outputs has been investigated as well.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Multilingual abilities of language models are still under-investigated, so this is an important topic.\n\nUsing scores to improve outputs is also an important application which has not been explored much.\n\nThe model and the data will be publicly available.", "reasons_to_reject": "It seems that almost everything (except LitEval-corpus) is generated by language models without any interaction with human evaluators (all training and test texts and corresponding evaluation scores) .", "questions_to_authors": "line 1: evaluating long-form text -- generated also by LLMs, right?  \n\nline 15: across all 3 tested languages -- which 3 languages? Source, target? \nbefore that, 4 language pairs were mentioned, and earlier a span of 20 languages was mentioned \ntherefore, the distribution of languages is unclear\n\nline 40: Those works => The publications  or The work\n\nline 48: What does Prometeus 2 do? What is the advantage/novelty of M-Prometeus? \n\nline 53:  evaluation of literary machine translation evaluation\n\n\nline 61: across languages: Which languages? How many languages? \n\nSection 2.1\nAll training data is generated by LLMs, there is no human-generated training data? \n\n\"natively multilingual data\" -- maybe some texts are not written by native speakers\n\"original language\" would be a better term (to indicate that those are written directly in the given language, not translated from another one)\n\nRelated work should be at the beginning, not at the end", "extra_scores": {}}, {"review_id": "C35yOjokj3", "reviewer": "Reviewer_n3M7", "summary": "The paper proposes multilingual LLM judges, which are trained on synthetic data generated in six languages. The authors show that their trained models perform better than other open-weights LLMs and LLM-judges on multilingual evaluation benchmarks, evaluation for literary machine translation, and on downstream performance when used for quality-aware decoding for improving generations of Qwen2.5-3B-Instruct. Through ablations, the authors show that the backbone model used for training the multilingual judge is an important consideration and training on translated data is not as effective as synthetically generated multilingual data. The paper provides a detailed list of experiments to test the effectiveness of their approach and the results to a large extent provide evidence in favor of that. I am reserved about the research contributions of the work and some claims that seem exaggerated.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- I think empirically, the paper to the most part is rigorous. The authors test the leading (multilingual as well as monolingual) meta-evaluation benchmarks and also include downstream applications of these models such as evaluation for Literary Machine Translation and Quality-Aware-Decoding\n- The results also to the most part show the effectiveness of their method, specifically on the LitEval and QAD metrics. I am less convinced about the results on MM-Eval benchmark, which I discuss in the following section.", "reasons_to_reject": "- In my opinion, the research contribution of the present work is a bit limited. From the technical aspect, the training method is an extension of prior works to a multilingual setting by generating synthetic data in non-English languages mainly through prompting. While I understand that not every paper needs to offer a technical contribution and strong empirical analysis can be at least as much important, I feel the latter to also not be a strong aspect of this work, which I detail in the points below.\n\n- The claims about some of the results in the paper seem slightly exaggerated. On MM-Eval benchmark the authors state that their models are particularly strong on the Safety, Linguistics, and Language Hallucinations categories. However, on examining the results in Figure 3, for the Linguistics category their models perform either as good as the base model or worse when considering 7B and 14B parameter sizes. Only at 3B their models perform slightly better than the base model. Similarly, for the Language Hallucinations categor the gains are marginal as well. Only in the Safety Category the authors find very strong improvements with their method, where the base models perform particularly poorly. This also makes the overall improvements by their method on the MM-Eval benchmark seem overly inflated. I am not sure **if these results indicate the effectiveness of their method in general or specific to particular tasks?** Similarly, authors show strong performance on LitEval metric, but that again might be due to the presence of Machine Translation Eval data in their training data.\n \n- The authors use the term *natively multilingual* at many places in the paper for the synthetic data generated in different languages, which I feel is a little misleading. I understand that authors use this term to distinguish from machine translated data, but the synthetic data is still machine generated and saying natively multilingual gives the impression of it being the data from the native speakers of these languages. I think *multilingual synthetic data* would be a more accurate term, as we do not know how closely LLM generated data resemble native speakers data, especially for low resource languages.\n\n- The improvements on the QAD metric are impressive, but presently it has been evaluated only for a single model i.e. Qwen2.5-3B-Instruct. Hence, I am unsure about how much these results hold in general.\n\n- In Section 5, the authors show interesting ablations on the factors that help in training better multilingual judges, but the analysis does appear preliminary in the present form and there is little discussion on why the observations might hold or their implications. E.g. The authors show that using Qwen 2.5 models as backbone offer significant improvements over the other backbones but do not engage with why that might be the case? It would have been interesting to see the performance of the other backbone models before they were trained on the synthetic data to begin with.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents M-Prometheus, multilingual LLM judges fine-tuned from Qwen2.5 models (3B, 7B, 14B) using synthetic multilingual data. The work extends Prometheus-2 to enable direct assessment and pairwise comparison across multiple languages.

## Contributions & Strengths
All four reviewers unanimously praised the community benefit of releasing open-weight multilingual evaluation tools, addressing a significant gap in available resources. Key finding that synthetic multilingual data outperforms translated data challenges existing assumptions and provides valuable guidance for future multilingual judge development.

**Comprehensive Evaluation**: Rigorous assessment across general capabilities (MM-Eval, M-RewardBench), specialized tasks (literary MT evaluation), and downstream applications (Quality-Aware Decoding).

**Experimental Rigor**: Thorough ablation studies identifying backbone model choice and training data composition as critical factors for multilingual judge performance.

## Limitations and Concerns Raised
**Performance Inconsistencies**: Counterintuitive underperformance on M-RewardBench compared to base models, particularly for larger variants, raises questions about overfitting to synthetic data patterns.

**Limited Technical Novelty**: The contribution is primarily an incremental extension of existing methods, lacking fundamental innovations.

**Evaluation Methodology**: Heavy reliance on synthetic data in training and evaluation creates potential echo chamber effects, despite inclusion of human-annotated benchmarks.

**Scope Limitations**: Detailed analysis focuses on only three languages, and single model family QAD evaluation. Authors addressed this with additional experiments.

## Recommendation
**Accept**. While technically incremental, M-Prometheus makes a solid empirical contribution with clear community value. The systematic extension to multilingual settings, comprehensive evaluation framework, and practical utility justify acceptance. The work successfully addresses an important problem area and provides both tools and insights for future multilingual evaluation research.  Finally, the authors should update the paper in light of discussions including clarification made in the review.
Decision: Accept