Déjà Vu: Multilingual LLM Evaluation through the Lens of Machine Translation Evaluation
Authors: Julia Kreutzer
Eleftheria Briakou
Sweta Agrawal
Marzieh Fadaee
Tom Kocmi
Venue: COLM 2025 (2025)
Keywords: multilingual
evaluation
meta-evaluation
machine translation evaluation
TL;DR: What can multilingual LLM evaluation learn from MT evaluation?

Abstract: Generation capabilities and language coverage of multilingual large language models (mLLMs) are advancing rapidly. However, evaluation practices for generative abilities of mLLMs are still lacking comprehensiveness, scientific rigor, and consistent adoption across research labs, which undermines their potential to meaningfully guide mLLM development. We draw parallels with machine translation (MT) evaluation, a field that faced similar challenges and has, over decades, developed transparent reporting standards and reliable evaluations for multilingual generative models.
Through targeted experiments across key stages of the generative evaluation pipeline, we demonstrate how best practices from MT evaluation can deepen the understanding of quality differences between models. Additionally, we identify essential components for robust meta-evaluation of mLLMs, ensuring the evaluation methods themselves are rigorously assessed. We distill these insights into a checklist of actionable recommendations for mLLM research and development.

Published as a conference paper at COLM 2025
Déjà Vu: Multilingual LLM Evaluation through the Lens of Machine Translation Evaluation
Julia Kreutzer' Eleftheria Briakou² Sweta Agrawal?
Marzieh Fadaee Kocmi Tom
Cohere Labs Google 'Cohere
Corresponding author: jullakreutzer@ccherr..co
Abstract
Generation capabilities and language coverage of multilinggul large lan- guage models (mLLMs) are advancing rapidly. However, evaluation prac- tices for generative abilities of mLLMs are still lacking comprehensiveness, scientific rigor, and consistent adoption across research labs, which un- dermines their potential to meaningfully guide MLLM development. We draw parallels with machine translation (MT) evaluation, a field that faced similar challenges and has, over decades, developed transparent report- ing standards and reliable evaluations for multilingual generative models. Through targeted experiments across key stages of the generative evalua- tion pipeline, we demonstrate how best practices from MT evaluation can deepen the understanding of quality differences between models. Addition- ally, we identify essential components for robust meta-evaluation of mLLMs, ensuring the evaluation methods themselves are rigorously assessed. We distill these insights into checklist of actionable recommendations for mLLM research and development.
Introduction
Evaluating LLMs in a multilingual context involves testing their capabilities across different languages and tasks, with particular attention to less-studied and lower-resoureed non- English languages (Huang et al., 2024). Naturally, it inherits challenges from monolingual L.M evaluation, such as benchmark contamination (Yang et al., 2023; Deng et al., 2024; Dong et al., 2024; Li et al., 2024b; Ni et al., 2025), label noise (Vendrow et al., 2025), costs vs coverage trade-offs (Zhang et al., 2024a), standardization, reliability, diversity (McIntosh et al., 2024) and reproducibility issues (Biderman et al., 2024). These challenges become more evident when drawing conclusions about model progress across multiple languages.
Prior classification benchmarks from cross/ matis/mmltiiinaaaa studies that pre-date the decoder- only-LLM era can be re-used to gain performance insights for MLLMS (Hu et al., 2020; Ruder et al., 2021; Liang et al., 2020; Ahuja et al., 2023; Asai et al., 2024). However, many of these benchmarks have reached saturation (Kiela et al., 2021; 2023) and are not separating models sufficiently (Zhang et al., 2024c). They are unreliable predictors of generative abilities of MLLMS (Üstün et al., 2024), as they serve primarily for knowledge testing. Generative abilities are key in real-world applications (Tamkin et al., 2024; Wu et al., 2025), and have thus moved into the spotlight of LLM evaluations (Dubois et al., 2023; Chiang et al., 2024; Lin et al., 2024). Multilingual models shine especially in these generative tasks, outperforming monolingual models across the bench (evidence in App. E). However, particularly this area of evaluation is still in the early stages.
Current generative evaluation approaches for multilingual models lack nuances in reporting, reproducibility, standardization, robustness and reliability, and most notably, meta-evaluation. These challenges, albeit new in the mLLM evaluation field, are familiar problems in sister field, the evaluation of machine translations. In this paper, we thus establish a connection to machine translation (MT) evaluation research, linking new questions in mLLM evaluation research to known solutions MT evaluation research.
MT has had a headstart on navigating these complexities in multilingual generation evalua- tion. As one of the core tasks in the NI.P field, it has rich research history of evaluations
1

---

Published as a conference paper at COLM 2025
Judge?
Source
#Lagg
Tranel
Judge?
Source
#tangs
TransL
Benchmark
Task
Rank
Size
Benchmark
Task
Rank
Size
BLORIS-200
Tb
*knk
5: 1,0000
+
200
M+
MCSM
**
250
0
10
H
NTREX-128
RJ 2,000
AfrIMGSM
250
WM724 + +
2: 2,000
0
55
H
SeaBench
300
5.
3
General MT
Sea-M Hench
MAFAND-MT
1,000
0
21
MTG
3,000
M+
XiSum
34
500-11,0000
0
45
OMGGival
/
804
[08]
5
M
CrossSum-In
29
marenalladd
500
SEA-IFEval
* Cnnt
105
0
[0]
H
Delly translated
*
200
101
M+
MIFEval MultilF
96
10
human-ann.
250
7
454-909
[0]]
M
Poly
A:
155
240
M
MuiliQ
200
05/
137
M
Table 1: Public generative benchmarks for downstream text-based evaluation of mLLMs. They are sourced from the web crowds experts (ao), machine generated (#E). Brackets indicate extension of previous benchmarks. The Size column counts the number of prompts per language. Translations of prompts are denoted as H(uman) and M(achine), with M+ indicating human post-edits. We mark LLM judged benchmarks (D), and rank them by popularity Aht in model releases. The details for these benchmarks can be found in tab. 5 and the popularity rating is grounded in survey of model releases in app. B.
with automatic metrics (Papineni et al., 2002; Koehn & Monz, 2006a; Lavie & Agarwal, 2007; Stanojević & Sima' an, 2014; Popović, 2015; Rei et al., 2020, inter alia) and human judgments (Vilar et al., 2007; Birch & Osborne, 2010; Lopez, 2012; Graham et al., 2013; Freitag et al., 2021; Kocmi et al., 2024b, inter alia), spurred by venues like the annual Conference Machine Translation (WMT). The development of evaluation went hand in hand with gradual improvement of model abilities and language coverage: Evaluation metrics that once worked sufficiently for statistical models became ineffective for neural models with superior translation quality (Freitag et al., 2022), or for newly added languages (Bapna et al., 2022). Meta-evaluation (Callison-Burche et al., 2007; 2008; Macháček & Bojar, 2013; Post, 2018; Mathur et al., 2020; Amrhein et al., 2022; Deutsch et al., 2023, inter alia), i.e., the evaluation of evaluations, led to the development of evaluation and transparency standards and built framework for metric development.
Elements of this progress have yet to be seen in mLLM evaluation, due to traditionally disjoint research streams, and the rapid speed of mLLM development. To bridge this gap, we first identify challenges in generative MLLM evaluation through an assessment of current benchmarks and their adoption in model releases (S 2). We then highlight five concrete evaluation principles that are lacking in MLLLM evaluations but established in MT (S 3). Finally, we establish which prerequisites are necessary for meta-evaluations ($ 4).
We distill our findings into an actionable checklist for mLLM research (App. J), to help steering MLLM development towards more reliable, expressive, and rigorous evaluations.
The Status Quo of MLLM Generation Evaluation
We compile a non-exhaustive list of open multilingual generative benchmarks in Table 1 to survey the mLLM landscape. 2 We sunmarize trends as follows, and link (++)) them to proposed strategies from MI evaluation research in $ 3 and meta-evaluation in $ 4.
Multilinguality via translation Most tasks rely on the translation of the original English benchmark for multilingual expansion. Only XLSSum (Hasan et al., 2021), Aya human- annotated (Singh et al., 2024c), SeaBench (Zhang et al., 2024b), MAFAND-NS 2022) are directly curated in the target languages. Automatic prompt translations might not be universally applicable or high-quality (Zhang al., 2023; Plaza et al., 2024; Agrawal et al., 2024a; Thellmann et al., 2024), which some benchmarks address with post-editing or localization (e.g. SEA-IFExal (Ong Limkonchotiwat, 2023)). While translation achieves broad coverage of languages, it limits the cultural representativeness and might propagate Western-centric and Anglo-centric biases (Singh et al., 2024b; Guo et al., 2024) (PH 3.1).
(Adelani et al.,
1https:/ htas::///hhhhl com//ooereLaas//nee ti tilinguall waiiggaa-llmmeeeeee wwl-eeallatiinn---oaaa coiiiioooncccisstttte. We exclude classification benchmarks such MCQA problems, see discussion in App. D.
2

---

Published as a conference paper at COLM 2025
Small and not mighty The majority of test sets contain less than 500 prompts per lan- guage, with MT benchmarks as outliers with over 1,000 samples. While prompt sourcing is a challenging task, especially with experts, such small sets raise questions of statistical power ( S 3.2). When included, human evaluations tend to cover even fewer instances (Gehrmann et al., 2023). Most benchmarks provide only a test split, lacking development sets for tun- ing, which increases the risk of overfitting and diminishes the significance of reported improvements over time (van der Goot, 2021; Ott al., 2022). Qualitative insights beyond aggregated task metrics are rarely included in evaluation reports (D#1 5 3.4).
Divergences in benchmark adoption and reporting Only few generative benchmarks are well-established, i.e, multiple labs use them for reporting results in open MLLM releases (App. B). Flores-200 (Costa-jussà et al., 2022), MGSM (Shi et al., 2023), and XLSum (Hasan et al., 2021) are the most popularly used benchmarks, as indicated by the rank in tab, 1. These are closed generative evaluation tasks that have the advantage of having relatively well-defined evaluation paradigms. Open generation tasks like chat and open-ended QA have less standardized evaluations and tend to rely on L.Mjudges, which introduces more ambiguities. What complicates cross-pap poos--pperr comparisons even when using the same bench- mark, is the lack of transparency and standardization in evaluation reporting. This goes from the choice of automatic metric (or LLM judge), over prompting conditions and formula- tions (+5 $ 3.5), to the selection and aggregation across languages for comparison (> 3.3). For instance, performance on Flores-200 is measured with different metrics ((spLEEU (Goyal et al., 2022), ChrF (Popović, 2015), COMET-22 (Rei et al., 2022)), and for MGSM model reports vary the number of shots, or even define new criteria (Barcelona Supercomputing Center, 2024). Sometimes, it is not even stated which languages of a benchmark are chosen for evaluation, and rarely do they cover all of the supported languages of a model (tab. 6).
Generative models are becoming the metric The emergence of new generative tasks, such as chat and open-ended generation, do not come with decades of task-specific metrics research, especially not across languages. Thus, (m)LLM judges are used to express prefer- ences through pairwise comparisons of model outputs, both in training (Lee et al., 2024), and in evaluation (Zheng et al., 2023a; Gu et al., 2025). However, this in itself is a generative evaluation task measuring how good MLLMS are judging multilingual generations (Gureja et al., 2024; Doddapanení et al., 2024) raising questions about judge biases (Ye et al., 2024) and gameability (Zheng et al., 2025; Eisenstein et al., 2024) (>NS $4.1.
Evaluation with a rapidly moving target All of today's leading MLLMS and most of the generative benchmarks are less than year old. Benchmarks quickly "expire", due to score saturation as a consequence 2024), or they simply lose relevance for LLM user or broader research needs (Zheng et al., 2023b; Tamkin et al., 2024; Wu et al., 2025). New model releases often test on newly intro- duced benchmarks to highlight new strengths, but these benchmarks are rarely adopted by consequent releases of other labs. Open leaderboards attempt to close this gap, tracking progress on selection of tasks and languages across models (see App. C), but they are also prone to expiry, might lack utility (Ethayarajh & Jurafsky, 2020) and heavily rely on aggrega- tions for interpretation, which requires particular care for multilingual models (Hulagadri et al., 2025) ( S 3.3). This calls for larger are of evaluation, namely the evaluation of evaluations themselves, including automatic ( S 4.1) and human evaluation ( S 4.2).
overfitting, evolved capacities or contamination (Ahuja et al.,
3 Adopting Evaluation Practices from MT Evaluation
Based on the challenges outlined above, we identify five central questions in the MLLM evaluation pipeline, and relate them to insights and practices from MT. The guiding question is: What knowledge would we gain about MLLMs, if we supplemented their ecaluations with MT-style evaluation techniques?
3.1 Where Does the Data Come From? Treating Synthetic Data with Care
Machine translated datasets are commonly used in MLLM training (Dang et al., 2024a) and evaluation (Lai et al., 2023), with the intention to reduce data scarcity across lan-
3

---

Published as a conference paper at COLM 2025
Anerage:
XCOMET 1
WR à
0.4
Transfitore
Languggea
NULB
Changa ra
0.3
AVA
GT
on
CT
90.57
93.65
+0.02
+0.14
CMD
CMD A
92.80
+0.05
0.2
0.1
Table Average roundtrip translation quality of trans- lation models tested on Aya human annotated prompts across five languages (ar, en, pt, tr, zh), and the change in win rate (WR) A when comparing AYA EXPANSE 8B and GEMMA 2 9B. Ideally, translation should not affect the win rate.
0.0
0.1
85.0
quality prompis (XComell)
Figure 1: The effect of prompt translation quality on win rates differences between AYA EXPANSE 8B vs GEMMA2 9B: Win rate differences (A) mostly increase compared to the ones under original prompts (y 0). Transparent points reflect non-significant win rate As (at 95% CI).
guages (Muennighoff al., 2023; Holmström Doostmohammadi, 2023; Üstün et al., 2024). However, synthetic, model model-generated data is prone to systematic biases (Ahn et al., 2022; Lukasik et al., 2022; Shimabucoro et al., 2024), including biases induced by translated train- ing data for the generating models (Li et al., 2025). In particular, machine-translate may contain translation artifacts affecting evaluation outcomes (Chen et al., 2024; Guo et al., 2024; Agrawal et al., 2024a). In MT research, studies have shown that grammar, structure, or word choice of the source text can systematically influence human and machine translations a phenomenon known translationese (Gellerstam, 1986; Laviosa, 2011). In evaluations, the presence of translationese in the sources has been found to decrease the difficulty of the task (Zhang & Toral, 2019), even leading to false claims of human parity (Hassan et al., 2018; Toral et al., 2018; Graham et al., 2020). As result, maintaining source authenticity has become a critical principle in the creation of test sets (Barrault et al., 2019).
prompts
A To illustrate the effects of prompt translation in multilingual generative evaluation, we conduct experiment using 250 Aya human annotated prompts for Arabic, Chinese, En- glish, Portuguese and Turkish. These are crowdsourced prompts for open-ended tasks written originally in the target language. We round-trip tripttd-tttt them automatically via pivot language (Portuguese for English prompts, and English for all other languages) to cre- ate comparison between original prompts and translated prompts (Chen al., 2024). For translation, we use Google Translate (CT), NLLB NLLB-200-3.33 (NLLB Team, 2022), AYA EXPANSE 32B (Dang et al., 2024b) and COMMAND A (Cohere et al., 2025). We measure how GPT-4o- as-a-judge win rates change when comparing mLLM generations for original prompts to those for translated prompts. We focus on comparison of GFMMA2 9B (Gemma Team, 2024) and AYA EXPANSE 8B, selected from wider range of models documented in App. H.
Q We find that win-rate differences in pairwise evaluations are affected by translation, with magnitudes that vary across languages and translation models (fig. 1), depending on translation quality (tab. 2). We can see that the majority of translations tilt the scale in favor of AYA EXPANSE 8B, increasing the win rate delta over GEMMA2 9B from 0.18 to 0.32 on average across languages (especially for Chinese and Turkish). Why hypothesize that AYA EXPANSE 8B is more robust to translation artifacts in the prompts due to exposure during training (Artetxe et al., 2020a). We also note that MLLMS used for translation (AyA EXPANSE 32B, COMMAND A) appear to have proportionally larger downstream effects. Overall, this simulation demonstrates that win rates computed on translated evaluation prompts might systematically favor particular models that are more robust to translation artifacts, leading to inflated win rates. A more detailed analysis can be found in app. H.
& Recommendation 1: For evaluation, prefer target-language original prompts over trans- lated alternatives (silver standards, coined by Holtermann et al. (2024)). If translations are unavoidable, ensure that their quality is optimized without assuming off-the-shelf adequacy for any task (e.g. choosing best MT, adding post-edits, localization). Measure and document translation quality on representative subset for each task.
4

---

Published as a conference paper at COLM 2025
3.2 What do Score Differences Mean? Measuring Significance, Power and Effect Size
Although platforms like Chatbot Arena report confidence intervals using bootstrapping, significance testing is not yet a standard part of the LLM development pipeline (Vaugrante et al., 2024; Ackerman et al., 2025). To this aim, Miller (2024) proposed best LLM evaluation practices, emphasizing the importance of reporting sample size, confidence intervals, and standard errors, particularly for clustered and paired tests. In MT research, such reporting and significance tests have a long history (Koehn, 2004; Riezler & Maxwell, 2005; Graham et al., 2014b; 2020) and have found moderate adoption (Marie et al., 2021), also enabled by ease of use in tools like sacrebleu (Post, 2018) or comet compare (Rei et al., 2020). Statistical power analyses can further help determine the sample size required for reliable evalua- tions (Card et al., 2020), e.g. for human preference evaluation. In MT, for instance, statistical power usually sufficient (> 0.8) to rank even close models with x 1..K sources (Graham et al., 2020), yet smaller sample sizes may be insufficient (Wei et al., 2022). Based on test sizes of the benchmarks reviewed in $ 2, it is likely that especially under metrics with high variance such pairwise LLM judgments, many mLLM evaluations might be underpowered.
There are some pitfalls to be aware of: first, different metrics for the same task may have varying sensitivity (Riezler & Maxwell, 2005), which could lead to differences in one metric being significant but insignificant in another. Second, the more statistical tests are done, the more likely false positives will be encountered. This becomes particularly relevant for testing multiple models on multiple languages on multiple benchmarks. Correction (Zerva et al., 2022; Ulmer et al., 2022) can prevent this inflation, e.g. by increasing the threshold of significance for individual tests (Bonferroni correction), implemented at WMT.
is important to recognize that statistical significance does not necessarily imply that a difference is noticeable meaningful to humans (Mathur et al., 2020; Agrawal et al., 2024b). With sufficiently large sample sizes, even very small differences in metric scores can become statistically significant, despite being too subtle to notice in practice. This issue is specifically known in MT, where the magnitude of the effect size plays a crucial role in determining whether system improvements are genuinely meaningful (Kocmi et al., 2024c).
0.8
0.75
0.50
0.6
Win rate d
0.25
0.4
0.00
0.2
-0.25
0.0
-0.00
en
he
de
2h
-0.2
-1.00
en fr ar zh cs el he hi id ja ko fa pl pt ro ru tr uk
100
200
300
400
500
Figure 2: Win rates deltas for AYA EXPANSE 8B vs. 5WEN2.5 7B on HarenaHard prompts (23 languages), with GPT-4o as a judge. Error bars denote std. dev. across 5 samples for each prompt.
Figure 3: Win rate deltas in relation to sample size. Differences are signif- icant when the 95% confidence inter- val (shaded) lies above/ above/below
below zero.
A To illustrate the benefits of statistical significance testing, we inspect pairwise com- parisons of AYA EXPANSE SB and QWEN2.5 7B INSTRUCT on the 500 prompts of the HarenaHard benchmark for 23 languages (hard user-submitted prompts from the Chatbot Arena (" A'Areaa-Har--Autto (Li et al., 2024a)) that were machine-translate
with GPT-4o as judge. This comparison was previously reported (Dang et al., 2024b), but without considering significance tests or sample sizes. We compare win rates across languages and compute significance based on 95% confidence intervals, as recommended by Miller (2024). Experimental details are in App. F.
Q Across five runs, AYA EXPANSE 8B wins on average across languages with delta of 0.49, nominally outperforming 5WEN2.5 7B INSTRUCT in 21/23 languages. However, individual win rate deltas vary widely between languages, from -0.21 (loss) for English (en) to 0.93 for
5

---

Published as a conference paper at COLM 2025
All
High
Medium
Low
Models
Aug MMLU
Arg
MMLU
Aug
GSMSK
MMLU
Avg
MMLUU
QWEN2-78
2
2
3
MMSTAAI-NBMO-BAS--22BB-88
3
3
[3]
[3]
3
2
[2]
2
2
(2)
MINTRAL-SS7B-VO.1
2
33
3
[4]
GEMMA-78
5
2
5
5
5
4
15
1
3
Table Effect of different aggregation strategies on model ranking of top-5 pretrained systems as generated by the European Leaderboard on GSM8K and MMLU datasets.
Hebrew (he) (Figure 2). Moreover, the significance of the wins is dependent on sample sizes and languages: Figure 3 illustrates how these win rate differences behave under different sample sizes (sub-sampled from the full 500 samples), At 95% confidence level, a few Hebrew samples already reveal significant win rate differences, whereas even 300 Chinese samples are insufficient. This analysis highlights that we cannot reliably determine with this dataset whether the models qualitatively differ in Chinese text generation capabili- ties. For smaller test sets, in most generative benchmarks 2, such analysis is essential to avoid overconfidence in low-power evaluations. Finally, there are a few languages where win rate deltas below 0.2 even if these are significant, it is unclear if they indicative humanly recognizable difference between the models.
& Recommendation 2: Test the statistical significance of evaluation results rather than relying on metric differences alone, following task-specific recommendations (Dror et al., 2018; Miller, 2024; Ackerman et al., 2025). Estimate statistical power, particularly when working with small sample sizes. Additionally, consider the magnitude of the effect size to determine whether observed differences are also meaningful in practice.
3.3 What Gets Lost In Averages? Aggregating Responsibly
With MLLMs, we are modeling multiple languages and tasks at once. How we aggregate results thus naturally informs the interpretation of model comparisons. The go-to approach to report uniformly weighted averages across languages and tasks. This is not necessarily fair or adequate evaluation (Colombo et al., 2022) due to differences in training distribu- tions and metrics nor is it expressive enough as outliers (e.g,, by unseen languages) can disproportionately affect system rankings (Hulagadrí et al., 2025). Languages and tasks also differ in their expressive power, as seen in $3.2.
In multilingual MT, several aggregation formats were explored beyond reporting plain averages across languages. For instance, counting the number of wins (Zouhar al., 2024), grouping by language resourcedness, e.g. to study language-specific routing (Zhang et al., 2021); by directionality (Zhang et al., 2020); by unseen mnseen/seen seen languages (Aharoni et al., 2019) to isolate zhro-shot generalization. Additionally, WMT offers constrained track to isolate model improvements from data gains.
A Table 3 shows the ranking of the top 5 systems obtained using the European Leaderboard under different configurations: a) by language and b) by task. We categorize languages based on number of speakers into high (> 50M+; en, es, pt, de, fr, it, pl), medium (< 50 and 10M; nl, el, hu, sv, cz, ro) and low (< 10M; dk, fi, sk, sl, bg, It, lv, et) resource.
is the second best system after QWEN2-7B, whereas when looking at task-specific aggregates, we find consistently outperforms QWEN2-7B on MMLU. For medium and low-resource languages, however, its performance for GSMSk drops, leaving the second rank to others. This shows, that system rankings shift based on task and language focus. Optimal model selection for a specific task and language group can thereby deviate from the average best system.
Q Based on average scores, we would conclude that
v Recommendation 3: When comparing models across multiple languages, consider differ- ences in language support and aggregate results according to languages being seen by the multilingual models in question. Report task- and language-specific scores, and number of wins across languages as a supplement to averages. When you discuss averages, take language coverage into account, and make sure task metrics are comparable.
Phttps://en http:://wwwwkkiiiiiaaaara/////// of languages by number of native speakers

---

Published as a conference paper at COLM 2025
Swen2.5 78 Aga Expanse an Tl
0.5
Wins
2.6
0.4
0.2
500
1000
1500
hualess
Nuck
Prompt Length
Prompt Category
Figure 4: Win-rates for AYA EXPANSE 8B vs. QWEN2.5 7B on mArenaHard prompts bucketed by a) prompt length (left) and b) prompt categories (right).
3.4 Where Do Models Differ? Conducting Richer Analyses
Aggregate benchmark metrics do not provide insights into what differentiates the outputs of two models - yet identifying these distinctions is often the first step in human preference evaluation. In MT, specialized tools were developed to facilitate pairwise comparisons on specific examples, such as MT Compare Eval (Klejch et al., 2015) and compare-m mt (Neubig & 2018). In parallel, there has been a steady effort to create challenge sets and test suites designed to probe particular capabilities and phenomena of MT (Stanovsky et al., 2019; Bawden & Sagot, 2023; Manakhimova et al., 2024). In contrast, LLM evaluations typically rely user preferences in arena setting automatic judges with limited explainability. Before investing in human evaluation, automatic metrics can already offer insights into quality differences between model outputs across languages. Auxiliary metrics such as diversity scores, length statistics, bias detectors, language confusion statistics, and edit distance can highlight key trends. While not as comprehensive as human feedback, they can reveal biases that could influence both human and automatic pairwise evaluations.
and
A We illustrate this by comparing the win rates of AYA EXPANSE 8B and QWEN2.5 7B INSTRUCT on a subset of languages (en, de, fr, zh) from the marenaHard benchmark bucketed by a) the prompt length and b) manually annotated prompt category in Figure 4.
Q The first plot shows clear trend: QWEN2.5 7B INSTRUCT tends to win on longer prompts, while AYA EXPANSE 8B performs better on shorter prompts, suggesting that QWEN2.5 7B INSTRUCT can handle detailed and long queries better. On the other hand, from the second plot, AYA EXPANSE 8B emerges victorious in all categories except for "Math / ML. Puzzles" problems, where QWEN2.5 7B INSTRUCT has a clear advantage. These results provide valuable insights: while the average win rates in Figure 2 suggest general preference for AYA EXPANSE 8B, they obscure QWEN2.5 7B INSTRUCT's clear advantage specific prompt types. Such findings can guide targeted test set design, inform human evaluation sampling, and steer future model development.
model
Recommendation 4: Complement automatic metric analyses with qualitative error anal- ysis to better understand systematic patterns. Use visualization and systematic category breakdowns to contextualize metric results, ensuring that observed differences align with meaningful distinctions rather than incidental artifacts.
3.5 What Do We Need to Share? Advancing Reproducibility Through Transparency
Reproducing evaluation results in the LLM era has become increasingly challenging, if not impossible (Vaugrante et al., 2024; Biderman et al., 2024). Not only are many evaluations stochastic, but they are also dependent on configurations that are rarely fully disclosed, such as preambles or system prompts, task formatting, decoding strategies, temperature, or answer parsing. similar challenge arose in MT, where even straightforward metric like BLEU was implemented differently across frameworks, leading to discrepancies in reported scores. The introduction of SacreBLEU (Post, 2018) marked a turning point by standardizing the evaluation pipeline into single toolkit, with each evaluation assigned unique sig- nature containing all relevant parameters, ensuring comparability across papers. Efforts like simple-evall and the LM Evaluation Harness (Gao et al., 2024; Biderman et al., 2024)
httpps: //ts thubb wtt.hb..ccmooppaai/iimmeeeeeaaaa

---

Published as a conference paper at COLM 2025
0.8
Prompt
EN 1
EN 4
EN
Prompt
Aya Expanse 8B
46.0%
48.9%
42.4%
Llama 3.1 8B
50.4%
48.2%
51.1%
0.2
Gemma 2 9B
48.2%
49.6%
51.1%
0.0
Score range
4.4%
1.4%
8.7%
-0.2
C5
VI
70
de
05
en
0.8
0.8
0.6
Aya first
0.6
Gemini
Win rate
0.4
Qwen first
0.4
0.2
0.2
0.0
-0.2
-0.2
Figure Accuracy on German MCQA (Include 44) with three instruction variants (top-left). Win-rates of Aya Expanse 8B vs. Qwen 2.5 7B on mArenaHard prompts showing prompt (top-right), positional (bottom-left), and judge bias (bottom-right).
aim to standardize task formulations and output parsing for LLM evaluations. However, true transparency requires open evaluation releases that contain publicly available code with exact versioning (e.g,, commit hashes), full release of all prompts (including instruction text, exact wording, punctuation, and formatting), and disclosure of task formulations in each language. For example, Briakou et al. (2024) demonstrate that minor variations prompt wording for translation tasks can lead to drastically different outcomes, such models refusing to translate or producing overly verbose responses. Such findings are enabled by releasing model outputs, championed in the annual WMT shared task competitions (Koehn & Monz, 2006b), and has kindled metrics and meta-evaluation research by allowing retroac- tive comparisons and enabling longitudinal studies (Graham et al., 2014a). To hopefully start trend, we release the pairwise evaluation artifacts from this paper here.
A We illustrate configuration's impact on accuracy results for German MCQA (INCLUDE 44 (Romanou et al., 2024b)) (3 prompts) and mArenaHard LLM-as-a-judge win rates varying a) the prompt, b) the compared systems' order, and c) the judge (GPT-4o Gemini 1.5Pro).
Q Figure 9 shows that system accuracy MMLU-like evaluations changes significantly with different instruction wordings, undermining the robustness of benchmarking (Alzahrani et al., 2024). The of L.IMS judges further complicates reproducibility. Variability in model choice, decoding strategies (App. F.1), various biases (Ye et al., 2024; Shimabucoro et al., 2024; Zhang et al., 2025), and prompt phrasing adds layers of complexity. Figure illustrates how evaluations can be manipulated through positional biases (system presenta- tion order) and prompt formulation differences, yielding significantly divergent outcomes. Finally, LLM version obsolescence prevents reliable comparisons of results over time. Evalu- ations are non-transitive (Xu et al., 2025), making optimization a moving target.
V Recommendation 5: Use standardized pipelines, publish the exact prompt wording, and release the evaluation code, model outputs, and evaluation scores with versioning.
0.5
GPT4
GPT-4-Turbo GPT-4o
0.4
Command
0.28
Kdlpandorr'ss
Experoe 328
Command
0.26
0.3
Aya Expanse 32B
0.20
# Best
0.2
GPT-4-Turbo
CPT-40 Command Aya Expanse 32B
0.1
0.0
1a
ko
Table 4: Summary across languages.
Figure 10: LLM-as-a-judge agreement with humans on arena.
8

---

Published as a conference paper at COLM 2025
4
Evaluating mLLM Evaluation: Towards Meta Evaluation
The field of MT has been consistently involved in meta-evaluation of machine translation evaluation methods for the last twenty years (Callison-Burde et al., 2007). This process, whether implicit or explicit, has driven progress MT systems by redefining which metrics best correlate with human judgments throughout various milestones of model improve- ments. Given this progress in MT evaluation, a natural question arises: why has not similar progress been observed in multilingual evaluation? To better understand this disparity, we revisit the prerequisites of meta-evaluation. Meta-evaluation fundamentally requires three components: system outputs, human judgments (of those outputs), and automatic evalua- tions of those same outputs. In the following sections, we identify the missing components in the multilingual setting and outline the steps necessary to overcome these challenges.
Need for (More) Metrics: Beyond One-size-fits All 4.1
Lesson from MT While L.M--a--a-judge offers the convenience of using a single model for multilingual assessments, MT meta-evaluation shows there no universally best met- ric (Marie et al., 2021; Anugraha et al., 2024). Central to this issue was the widespread use pretrained language models as backbones for developing learned metrics (Lo, 2020), with varying degrees of language representation coverage. Although learned metrics often outperform string-based ones like BLEU (Papineni et al., 2002) - the choice between a string- baseg- and learned metric is heavily language and domain dependent. This has led to the informal convention of employing different types of metrics for targeted evaluations and reporting multiple metrics.
Application to multilingual evaluation Zheng et al. (2023b) used human evaluations from Chatbot Arena (Chiang et al., 2024) to study the reliability of LLM judges. We extend their predominantly English analysis to non- mon-English "battles" (pairwise comparisons) from the released sample of battles (Imarena-ai aaraaaarrnaaaaann arrea--humn--rrefeeeee--- 100k), focusing the 18 languages with 200 prompts. We score a subset of 200 generation pairs for each language with two open and two closed MI.LMS as judges. We measure agreement with human preferences with Krippendorff's a (interval measurements) shown in Figure 10. Overall agreement, and the choice of the best judge varies across languages. Just like in MT, the optimal choice of an LLM judge (i.e., metric) for multilingual evaluation is language dependent (Tab. 4).
4.2
Need for Nuanced Human Evaluation: Towards Richer Assessments
Lesson from MT Human evaluation of translation quality is a multifaceted challenge, rooted in the fundamental questions of what to measure and how to elicit accurate, consistent human assessments. These questions have long been central focus in machine translation. Below, we highlight key areas and insights from this extensive body of work.
Detailed work has explored different evaluation protocols (Vilar et al., 2007; Graham et al., 2013; 201 4a) and quality dimensions, including fluency versus adequacy (Koehn Monz, 2006a; Bojar et al., 2016). The trend then went from monolithic assessment scores of pair- wise assessments towards more fine-grained protocols, egg. highlighting and annotating errors using established taxonomies, such as the Multidimensional Quality Metrics (MQM) framework (Burchardt, 2013; Freitag et al., 2021). These taxonomies are getting refined to balance cognitive load and annotation effectiveness (Ge et al., 2024), and adapted for non-professional annotators (Graham et al., 2015; Castilho et al., 2017; Wang et al., 2024). Annotation efforts have also expanded to target specific use cases, such critical error detection (Specia et al., 2021; Zerva et al., 2022), detection of errors grounded in high-stake scenarios (Mehandru et al., 2023), and to contextualize evaluations within user-centric frameworks (Briakou et al., 2023; Savoldi et al., 2025).
Application to multilingual evaluation Chatbot Arena, where users compare two models in a chat and choose a winner, is the primary source of public human mLLM evaluations. Samples of data are released periodically,' forming the largest public collections of multilin-
Chhttps://githhs http:///////....aaaaaaa////a//aa release.md

---

Published as a conference paper at COLM 2025
gual human preferences. In the data from 2024, 27-43% of battles per language end in tie (analysis in App. D), suggesting that human evaluation in the arena format lacks sensitivity to fine-grained differences inherently includes significant uncertainty. The scarcity of publicly available multilingual preference data limits research in this area. Emerging efforts such as MM-EVAL (Son et al., 2024b) introduce multilingual meta-evaluation benchmarks, highlighting that LLM-as-a-judge lacks fairness and consistency across languages.
4.3
Need for Meta-evaluation Research: Closing the Loop on Evaluation
Lesson from MT Meta-evaluation research has been formally conducted within the WMT Metrics shared task since 2007 Gallison-Burchh et al., 2007). This research aims to improve MT evaluation by identifying best performing metrics, addressing weaknesses in correlation- based metrics, e.g., proper handling of ties (Deutsch et al., 2023), meta-evaluation techniques (Kocmi et al., 2021; Thompson et al., 2024), and challenges of conducting reliable human evaluations, such as ensuring replicable human evaluations (Riley et al., 2024) and studying inter-annotator agreement (Popović, 2021; Popović & Belz, 2022).
Application to multilingual evaluation Evaluation solely based on correlation with human pairwise preferences on prompt-level is not sustainable, human agreement decreases as the quality differences between the contrasted systems shrink (Zheng et al., 2023b). We get glimpse of this loss of signal in arena battles: from 2023 to 2024, the ratio of ties has grown significantly, from an average of 29% to 40% across the six most dominant languages (App. 1). For long-term progress in MLLM modeling and evaluation, we need to iterate the meta-evaluation loop. As a first step, we need to answer the questions which differences between models matter humans and how to capture these. Then, we can adapt automatic metrics accordingly. Finally, we can measure modeling progress automatically and reliably, which results models with enhanced qualities, bringing us back the first step.
5
Conclusion
MT has long grappled with the complexities of multilingual generative evaluations, from constructing datasets to benchmarking evaluation metrics. We demonstrated that estab- lished practices from MT can enhance the understanding and reliability comparisons among mLLMs, and outlined which elements are necessary for establishing meta-evaluations. Our recommendations are distilled into a practical checklist (App.)) J).
Limitations Our experiments are focused on open-ended generative tasks, which could be still considered more ambiguous than typical MT tasks, since MT evaluation criteria are more defined. However, with more advances in quality, mLLMM evaluations are trending towards more tightly defined benchmarks that require in-depth expert knowledge (e.g. coding, math), which brings these tasks closer to the conditions of MT evaluations. Furthermore, recommendations and best practices from other sub-fields of NLP that are now sharing multilingual benchmarks should also be considered, see for example the recommendations by Gehrmann al. (2023) for evaluating text generation, or those by Iskender et al. (2021) for human evaluation of text summarization.
Outlook Since mLLMs are now also competing with non-LLM MT models (Koemi et al., 2023; 2024a; Zhu et al., 2024b), and MT benchmarks have become established evaluation tasks for MLLMs (Zhu et al., 2024a), the sharing of knowledge and insights across both disciplines becomes even more important to drive meaningful progress. Our checklist with practical recommendations for mLLM evaluations is the first step towards the aim of bringing research communities closer.
Acknowledgments
We thank the anonymous reviewers and our colleagues for their helpful feedback on the paper: Shivalika Singh, John Dang, Colin Cherry.
10

---

Published as a conference paper at COLM 2025
References
Hugo Abonizio, Thales Sales Almeida, Thiago Laitz, Roseval Malaquias Junior, Gio- vana Kerche Bonás, Rodrigo Nogueira, and Ramon Pires. Sabiá-3 technical report, 2025. URL https://arxiv.orr/aasst.
12049.
Samuel Ackerman, Eitan Farchi, Oma Raz, and Assaf Toledo. Statistical multi-metric evaluation and visualization of Ilm system predictive performance, 2025, URL https: wwarxiv.orggabs/22500 18243.
David Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Sham- suddeen Muhammad, Guyo Jarso, Oreen Yousuf, Andre Niyongabo Rubungo, Gilles Hacheme, Eric Peter Wairagala, Muhammad Umair Nasir, Benjamin Ajibade, Tunde Ajayi, Yvonne Gitau, Jade Abbott, Mohamed Ahmed, Millicent Ochieng, Anuoluwapo Aremu, Perez Ogayo, Jonathan Mukiibi, Fatoumata Ouoba Kabore, Godson Kalipe, Der- guene Mbaye, Allahsera Auguste Tapo, Victoire Memdjokam Koagne, Edwin Munkoh- Buabeng, Valencia Wagner, Idris Abdulmumin, Ayodele Awokoya, Happy Buzaaba, Blessing Sibanda, Andiswa Bukula, and Sam Manthalu. A few thousand translations go a long way! leveraging pre-trained models for African news translation. In Marine Carpuat, Catie-Ca de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3053-3070, Seattle, United States, July 2022. Association for Computational Linguistics. doi: https: / haapas///cllnthoooogy.. org/ naacl -main. 223.
maw.mmaaallmmm222222222 URL
David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba O. Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun An- nie Lee, Chiamaka Chukwuneke, Happy Buzaaba, Blessing Sibanda, Godson Kalipe, Jonathan Mukiibi, Salomon Kabongo, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuya, Shamsuddeen Hassan Muhammad, Sa- lomey Osei, Sokhar Samb, Tadesse Kebede Guge, and Pontus Stenetorp. Irokobench: A new benchmark for african languages in the age of large language models, 2024. URL https://arxiv. wwwg/aa/////aae 03368.
Hapky
Ashish Agrawal, Barah Fazili, and Preethi Jyothi. Translation errors significantly impact low-resource languages in cross-lingual learning. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Com- putational Linguistics (Volume 2: Short Papers), pp. 319-329, St. Julian's, Malta, March 2024a. Association for Computational Linguistics. URL https:: //wpa/ahaaaaaaoyy....... org/ org/2024. eacl-short. 28.
Sweta Agrawal, António Farinhas, Ricardo Rei, and Andre Martins. Can automatic metrics assess high-quality translations? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 14491-14502, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10. 1018653/v122222 2tt/3//1122244.mmmmmmmmmmm 2024. emnlp-main. 802.
URL https: h/tp///aaaaaaaoogyyyoo org/
Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3874-3884, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653 /vl/ N19-. URL https::t//// htttts://aclatttihteee http://aaaaaaana...aaaca oor.
Jaimeen Ahn, Hwaran Lee, Jinhwa Kim, and Alice Oh. Why knowledge distillation amplifies gender bias and how to mitigate from the perspective of DistilBERT. In Christian Hard- meier, Christine Basta, Marta R. Costa-jusså, Gabriel Stanovsky, and Hila Gonen (eds.), Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pp.
11

---

Published as a conference paper at COLM 2025
266-272 Seattle, Washington,,Jll 2022. Association for Computational Linguistics. 10.18653/ //2////vv/1
Linguuttiionaaalniniississs doi:
ÚRL https: hatps///oloaantooooooo.ooooo http:///alaaattooooorr///////// wttp////...g.eeeeeeoee.. genzep-ebblpp.1.. 1. 27.
Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. MEGA: Multilingual evaluation of generative AL. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4232-4261, Singapore, December 2023, Association wttp:33/v1222..mmm////
for Computational Linguistics. doi: //aclanthology. org org/2023. http22223...1ppppmmmmmmmmmm emnlp-maaaaa mann-mmi..... 258.
258. URL https:
Sanchit Ahuja, Varun Gumma, and Sunayana Sitaram. Contamination report for multilin- gual benchmarks, 2024. URL htw.:: / /ttp://awxvvvvvaa http::////aaii.oo///////////// 16186.
Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan AlRashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, M Saiful Bari, and Haidar Khan. When benchmarks are targets: Revealing the sensitiv- ity of large language model leaderboards. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13787-13805, Bangkok, Thailand, August 2024. /ttp://1///222lll////
Association for Computational Linguistics. doi: nttps://aclanthollogh org/ acl sel-longg long.. 744.
URL
Chantal Amrhein, Nikita Moghe, and Liane Guillou. ACES: Translation accuracy challenge sets for evaluating machine translation metrics. In Philipp Koehn, Loic Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference Machine Translation (WMT), pp. ap.4779-133 Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL https: / www.anhooonnttoooooomooooooooooa wrg./022...... 1 44.
David Anugraha, Garry Kuwanto, Lucky Susanto, Derry Tanti Wijaya, and Genta Winata. MetaMetrics-MII Tuning meta-metrics for machine translation via human preference calibration. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference on Machine Translation, pp. 459-469, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10886533v1/222/ 1m3-1.32. URL httpss:/aclaattolll..oo
org/ w w.t-1.32. 32.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Translation artifacts in cross-lingual trans- fer learning. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp- 7674-7684, Online, November 2020a. Association for Computational Linguistics. doi: 10.18653 / v1 /222220.emnllmmaii..66 URL https:/ / h/tp:///aaaaaaooo//aooo org/2020. www.0.eeenp00mm.. 618.
emnlp-mail main.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolingual representations. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4633--637, Online, July 2020b. Association for Computational Linguistics. URL https: / //aclaahhlln org/
doi: 10.18653 / v1/2222.222lll// acaimin..221 421.
org/2020.00.. 2020.000..o acl-main
Akari Asai, Sneha Kudugunta, Xinyan Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. BUFFET: Benchmarking large language models for fhw-shot cross- cross-lingual transfer. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1771 Mexico City, Mexico, June 2024. Association for Computational
12

---

Published as a conference paper at COLM 2025
Linguistics. doi: 111186633/1122244aaa////// 2024. naacl-long..... 100.
URL httpss//acclanthooogg.orr
Nishant Balepur, Rachel Rudinger, and Jordan Lee Boyd-Graber. Which of these best describes multiple choice evaluation with Ilms? a) forced b) flawed c) fixable d) all of the above, 2025. URL https://arxil: http:////aaxivv.og/bbbb.s/e/
org/abs/2502. 14127.
Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Axelrod, Jason Riesa, Yuan Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apurva Shah, Yanping Huang, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. Building machine translation systems for the next thousand languages, 2022. URL htttttttt///iiiii http:///awwxwww./oaabbba org/abs/22955 03983.
Barcelona Supercomputing Center. Salamandra.
https://huggingtt http:/aaaa.cccccc--aeee co/BSC-LT/
sentaadra--biinstrree 2024.
Loic Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 conference on machine translation (WMT19). In Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor (eds.), Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pp. 1-61, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/1/10 /v/55//v//W/ /W119555301 URL
org/W13 9-5301.
Rachel Bawden and Benoît Sagot. RoCS-MT: Robustness challenge set for machine transla- tion. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Translation, pp. 198-216, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1// wttp://wl1223vm///////m https://aclanthology.. hrr.aaannthoooyooog//22220 wmt-1.22
URL
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, François Yvon, and Andy Zou. Lessons from the trenches on reproducible evaluation of language models, 2024. URL httpss://arks wwwpxiiv.rg/aaaee arg/abs/2485. 14782.
Alexandra Birch and Miles Osborne. LRscore for evaluating lexical and reordering quality in MT. In Chris Burlch, Philipp Koehn, Christof Monz, Kay Peterson, and Omar Zaidan (eds.), Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pp. 327-332, Uppsala, Sweden, July 2010. Association for Computational Linguistics. URL https: / / lanthology org/W10-17493
Ondite Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 conference on machine translation. In Ondfej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Liane Guillou, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Aurélie Névéol, Mariana Neves, Pavel Pecina, Martin Popel, Philipp Koehn, Christof Monz, Matteo Negri, Matt Post, Lucia Specia, Karin Ver- spoor, Jörg Tiedemann, and Marco Turchi (eds.), Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, Pp. 131-198, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653 111653//11W16622/5// URL https://aclanttoolgy..oom wwwp://woooooo.oo////o////// 2301.
13

---

Published as a conference paper at COLM 2025
Eleftheria Briakou, Navita Goyal, and Marine Carpuat. Explaining with contrastive phrasal highlighting: case study in assisting humans to detect translation differences. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 11220-11237, Singapore, December 2023. Association for Computational Linguistics. doi: URL http:://aclannhhloogy org 22tg:/203... emmlp-main 690.
Eleftheria Briakou, Zhongtao Liu, Colin Cherry, and Markus Freitag. On the implications of verbose Ilm outputs: A case study in translation evaluation. ArXiv, abs/ 0abs/240000063,, 2024.
URL https://api. semanticschon iesshhllarcrrrrcc cerpollaa.rrr/oopooeee org//oorpusD::273221133
Aljoscha Burchardt. Multidimensional quality metrics: a flexible system for assessing trans- lation quality. In Proceedings of Translating and the Computer 35, London, UK, November 28-29 2013. Aslib. URL https: / hacan/hollln www.llogy.googoggoo org/2013. 1.
Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. (meta-) evaluation of machine translation. In Chris Callison-Burchh Philipp Koehn, Cameron Shaw Fordyce, and Christof Monz (eds.), Proceedings of the Second Workshop on Statistical Machine Translation, PP. 136-158, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https: http::/:///aaaahhollooooooo or org/W07-0718.
Chris Callison-Burde Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. Further meta-evaluation of machine translation. In Chris Callison-Burees Philipp Koehn, Christof Monz, Josh Schroeder, and Cameron Shaw Fordyce (eds.), Proceedings of the Third Workshop on Statistical Machine Translation, pp. 70-106, Columbus, Ohio, June 2008. Association for Computational Linguistics. URL https::/aclantthtt
org/w08- -3099.
Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. With little power comes great responsibility. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9263-92267777 9274, Online, November 2020. Association for Computational Linguistics. doi: 10.18653. 10111833/v1/200 http///1/2200mmmm// 200mm/000mmmlmmmmmi............. URL https: //aclantholes . org/2020. emn.2-mmnn.-.... 745.
Sheila Castilho, Joss Moorkens, Federico Gaspari, Rico Sennrich, Vilelmini Sosoni, Panayota Georgakopoulou, Pintu Lohar, Andy Way, Antonio Valerio Hiceli-Barone, and Maria Gialama. A comparative quality evaluation of PBSMT and NMT using professional translators. In Sadao Kurohashi and Pascale Fung (eds.), Proceedings of Machine Translation Summit XVI: Research Track, pp. 116-131,, Nagoya Japan, September 18 - September 22 2017. URL https: httpps://acaanthologg hww.hhollooo.oo/g2111/ sumummit papeers.
Tyler A. Chang, Catherine Amettt Zhuowen Tu, and Benjamin K. Bergen. When is multilin- guality a curse? language modeling for 250 high- and low-resource languages, 2023. URL
http:::/aaxiv..or/ab///111..11
Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Benjamin Bergen. Goldfish: Monolin- gual language models for 350 languages, 2024. URI. https:://rriiss org/ /bs///0s....// 10441.
laroslav Chelombitko and Aleksey Komissarov. Specialized monolingual BPE tokenizers for Uralic languages representation in large language models. In Mika Hämäläinen, Flammie Pirinen, Melany Macias, and Mario Crespo Avila (eds.), Proceedings of the 9th International Workshop on Computational Linguistics for Uralic Languages, pp. 89-95, Helsinki, Finland, /acltthhooo...........
November 2024. Association for Computational Linguistics. URL org/2024. wwclul-1.1.
Pinzhen Chen, Simon Yu, Zhicheng Guo, and Barry Haddow. Is it good data for multilingual instruction tuning or just bad multilingual evaluation for large language models? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 9706-9726, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 1..18633/v1/200 emnlp-main.544 542. URL https:://allanthoolggy org/ 2024. emnlp-main.55 542.
14

---

Published as a conference paper at COLM 2025
Yiran Chen, Zhenqiao Song, Xianze Wu, Danqing Wang, Jingjing Xu, Jiaze Chen, Hao Zhou, and Lei Li. MTG: A benchmark suite for multilingual text generation. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Findings of the Association for Computational Linguistics: NAACL 2022, pp. 2508-2527, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18655 /1///6633////// / findings-naacl.199 URL https::/aclannthlleee org/ 2222.2222..22 rindings-naacl... 192.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and lon Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024.
URL https://arxii...oote wwwpexiii..og//b/b. wttp httlb///2434444
Team Cohere, Aakanksha, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Yazeed Alnu- may, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis A Aumiller, Raphaël Avalos, Zahara Aviy, Sammie Bae, Saurabh Baji, Alexandre Barbet, Max Bartolo, Björn Bebensee, Neeral Beladia, Walter Beller-Morales, Alexandre Bérard, Andrew Berne- shawi, Anna Bialas, Phil Blunsom, Matt Bobkin, Adi Bongale, Sam Braun, Maxime Brunet, Samuel Cahyawijaya, David Cairuz, Jon Ander Campos, Cassie Cao, Kris Cao, Roman Castagné, Julián Cendrero, Leila Chan Currie, Yash Chandak, Diane Chang, Giannis Chatziveroglou, Hongyu Chen, Claire Cheng, Alexis Chevalier, Justin T. Chiu, Eugene Cho, Eugene Choi, Eujeong Choi, Tim Chung, Volkan Cirik, Ana Cismaru, Pierre Clavier, Henry Conklin, Lucas Crawhall-Stein, Devon Crouse, Andres Felipe Cruz-Salinas, Ben Cyrus, Daniel D'souza, Hugo Dalla-Torre, John Dang, William Darling, Omar Darwiche Domingues, Saurabh Dash, Antoine Debugne, Théo Dehaze, Shaan Desai, Joan Devassy, Rishit Dholakia, Kyle Duffy, Ali Edalati, Ace Eldeib, Abdullah Elkady, Sarah Elsharkawy, Irem Ergün, Beyza Ermis, Marzieh Fadaee, Boyu Fan, Lucas Fayoux, Yannis Flet-Berliac, Nick Frosst, Matthias Gallé, Wojciech Galuba, Utsav Garg, Matthieu Geist, Moham- mad Gheshlaghi Azar, Seraphina Goldfarb-Tarrant, Tomas Goldsack, Aidan Gomez, Victor Machado Gonzaga, Nithya Govindarajan, Manoj Govindassamy, Nathan Grinsz- tajn, Nikolas Gritsch, Patrick Gu, Shangmin Guo, Kilian Haefeli, Rod Hajjar, Tim Hawes, Jingyi He, Sebastian Hofstätter, Sungjin Hong, Sara Hooker, Tom Hosking, Stephanie Howe, Eric Hu, Renjie Huang, Hemant Jain, Ritika Jain, Nick Jakobi, Madeline Jenkins, JJ Jordan, Dhruti Joshi, Jason Jung, Trushant Kalyanpur, Siddhartha Rao Kamalakara, Julia Kedrzycki, Gokce Keskin, Edward Kim, Joon Kim, Wei-Yin Ko, Tom Kocmi, Michael Kozakov, Wojciech Kryšciński, Arnav Kumar Jain, Komal Kumar Teru, Sander Land, Michael Lasby, Olivia Lasche, Justin Lee, Patrick Lewis, Jeffrey Li, Jonathan Li, Hangyu Lin, Acyr Locatelli, Kevin Luong, Raymond Ma, Lukas Mach, Marina Machado, Joanne Magbitang, Brenda Malacara Lopez, Aryan Mann, Kelly Marchisio, Olivia Markham, Alexandre Matton, Alex McKinney, Dominic McLoughlin, Jozef Mokry, Adrien Morisot, Autumn Moulder, Harry Moynehan, Maximilian Mozes, Vivek Muppalla, Lidiya Mu- rakhovska, Hemangani Nagarajan, Alekhya Nandula, Hisham Nasir, Shauna Nehra, Josh Netto-Rosen, Daniel Ohashi, James Owers-Bardsley, Jason Ozuzu, Dennis Padilla, Gloria Park, Sam Passaglia, Jeremy Pekmez, Laura Penstone, Aleksandra Piktus, Case Ploeg, Andrew Poulton, Youran Qi, Shubha Raghvendra, Miguel Ramos, Ekagra Ranjan, Pierre Richemond, Cécile Robert-Michon, Aurélien Rodriguez, Sudip Roy, Laura Ruis, Louise Rust, Anubhav Sachan, Alejandro Salamanca, Kailash Karthik Saravanakumar, Isha Satyakam, Alice Schoenauer Sebag, Priyanka Sen, Sholeh Sepehri, Preethi Seshadri, Ye Shen, Tom Sherborne, Sylvie Chang Shi, Sanal Shivaprasad, Vladyslav Shmyhlo, Anirudh Shrinivason, Inna Shteinbuk, Amir Shukayev, Mathieu Simard, Ella Snyder, Ava Spataru, Victoria Spooner, Trisha Starostina, Florian Strub, Yixuan Su, Jimin Sun, Dwarak Talupuru, Eugene Tarassov, Elena Tommasone, Jennifer Tracey, Billy Trend, Evren Tumer, Ahmet Üstün, Bharat Venkitesh, David Venuto, Pat Verga, Maxime Voisin, Alex Wang, Donglu Wang, Shijian Wang, Edmond Wen, Naomi White, Jesse Willman, Marysia Winkels, Chen Jessica Xie, Minjie Xu, Bowen Yang, Tan Yi-Chern, Ivan Zhang, Zhenyu Zhao, and Zhoujie Zhao Command a: An enterprise-ready large language model, 2025.
Pierre Colombo, Nathan Noiry, Ekhine Irurozki, and Stephan Clémençon. What are the best systems? new perspectives on nlp benchmarking. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in
15

---

Published as a conference paper at COLM 2025
Neural Information Processing Systems, volume 35, Pp. 26915-26932. Curran Asso- ciates, Inc., 2022. URL https://proceeedingg neurip 045FF//5555555223313445545453333
neurips cwwpapp.cc/pppprrraa
pdf.
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. Emerging cross-lingual structure in pretrained language models. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceddings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6022-6034, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1 http://11220.0mmmammmi. acl-main.555 536.
URL
org/ org/2020.
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arviv preprint ar arXiv:2007 04672, 2022.
John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. RLHF can speak many languages: Unlocking multilingual preference optimiza- tion for LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 13134-13156, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: URL https: htttps://aclanthoologyy /224g/ emnlp-maill main...
729.
John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Koemi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Acyr Locatelli, Hangyu Lin, Dwarak Talupuru, Bharat Venkitesh, David Cairuz, Bowen Yang, Tim Chung, Wei-Yin Ko, Sylvie Shang Shi, Amir Shukayev, Sammie Bae, Aleksandra Piktus, Roman Castagné, Felipe Cruz-Salinas, Eddie Kim, Lucas Crawhall-Stein, Adrien Morisot, Sudip Roy, Phil Blunsom, Ivan Zhang, Aidan Gomez, Nick Frosst, Marzieh Fadace, Beyza Ermis, Ahmet Üstün, and Sara Hooker. Aya expanse: Combining research breakthroughs for new multilingual frontier, 2024b. URL
/2t..//41....22 04261.
Leandro Rodrigues de Souza, Rodrigo Nogueira, and Roberto Lotufo. On the ability of monolingual models to learn language-agnostic representations, 2021. URL https: wwtrxiv.org/abs//// 01942.
Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investi- gating data contamination in modern benchmarks for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapler of the Associalion for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3706-8719, 19, Mexico City, Mexico, June 2024. As- sociation for Computational Linguistics. doi: orgo 2024. naacl-long 482.
201/2024.naacl-oom 482. URL
Daniel Deutsch, George Foster, and Markus Freitag, Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12914-12929, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653. 1tt.1853/v1/2223.comm wttp:00....mmmmmmaam...... URL https: pectps://acee hatps/////clanhooogyy org/ 2023. emnlp-main. 798.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653 10/553/v1/N/ www/////-111111 URL https: / hattaa///aclathhology.. Drg/N19-1423.
16

---

Published as a conference paper at COLM 2025
Sumanth Doddapaneni, Mohammed Safi Ur Rahman Khan, Dilip Venkatesh, Raj Dabre, Anoop Kunchukuttan, and Mitesh M. Khapra. Cross-lingual auto evaluation for assessing multilingual Iims, 2024. URL https: //atsi///axiv...
http:/:///axvvoo///aaa//a www.w/s..//...t..t.oo..oo..... 2410. 13394.
Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generaliza- tion or memorization: Data contamination and trustworthy evaluation for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Associa- fion for Computational Linguistics: ACL. 2024, pp. 12039-12054 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653 ht////1112222////// findi..iniigg--a..776 URL httpps://acla hwtp://ooocllttoooooyoooooo org/2024. findings-ach acl. 716.
Longxu Dou, Qian Liu, Fan Zhou, Changyu Chen, Zili Wang, Ziqi Jin, Zichen Liu, Tongyao Zhu, Cunxiao Du, Penghui Yang, Haonan Wang, Jiaheng Liu, Yongchi Zhao, Xiachong Feng, Xin Mao, Man Tsung Yeung, Kunat Pipatanakul, Fajri Koto, Min Si Thu, Hynek Kridtaphad Sae-Khow, Nirattisai Thongchim, Taechawat Konkaew, Narong Borijidaagoon, Anh Dao, Mati- chon Maneegard, Phakphum Artkaew, Zheng-Xin Yong, Quan Nguyen, Wannaphong Phatthiyaphaibun, Hoang H. Tran, Mike Zhang, Shiqi Chen, lianyu P'agg Chao Du, Xinyi Wan, Wei Lu, and Min Lin. Sailor2: Sailing in south-east asia with inclusive multilingual llm. Xiv preprint ar Xix:2502.11982 2025.
Kydlíček, Zeyi Liu, Qunshu Lin, Sittipong Sripaisarnmongkor
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. The hitchhiker's guide to testing statistical significance in natural language processing. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 1383-1392, Melbourne, Australia, July 2018. Association for Computational Linguistics.
doi: 10.18653/v1 /1t////1//P/111/// URL
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback. In Thirty-seventh Conference on Neural Infor- mation Processing Systems, 2023. URL https://apenreiil
http:/oormmiid-hhhcm cKXX.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alexander Nicholas D'Amour, Krishnamurthy Dj Dvijotham, Adam Fisch, Katherine A Heller, Stephen Robert Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. In First Conference on Language Modeling, 2024. URL https: /ttp::////peeeeeeeeeeee www.eewwwwwwwwweeeeeeeeeeeeeeeee peetfforum?e wowooomdddduuupueeneoio
Kawin Ethayarajh and Dan Jurafsky. Utility is in the eye of the user: A critique of NLP leaderboards. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp- 4846 4853, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ / v1 v112222.emnllpppp main.393. URL https: hatp:///aaaaaaooo///oo org/2020.00022 emn1p-main. 393.
Christian Federmann, Tom Kocmi, and Ying Xin. NTREX-128 news test references for MT evaluation of 128 languages. In Kabir Ahuja, Antonios Anastasopoulos, Barun Patra, Graham Neubig, Monojit Choudhury, Sandipan Dandapat, Sunayana Sitaram, and Vishrav Chaudhary (eds.), Proceedings of the First Workshop on Scaling Up Multilingual Evaluation, pp. 21-24, Online, November 2022. Association for Computational Linguistics. URL https: http:::s//cclathhloo/ooo org/ 22gg. sumeval-1.
doi: 10.18653 v1 s2022..mmm 4.
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460- 1474, 2021. doi: 10.1162/tar tacl 00437. URL. https::/aclanthhll.ge org/202) wtt....ac1......... tacl-1.87
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. Martins. Results of WMT22 metrics shared task: Stop using BLEU neural metrics are better and more
17

---

Published as a conference paper at COLM 2025
robust. In Philipp Koehn, Loic Barrault, Ondiej Bojar, Fethi Bougares, Rajen Chatter- jee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), Pp. 46-68, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL https://aclanthology. www.:gg....r..//e// wmt-1... 1.2.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https://zenodo. http://rrrrrrerrr/////////0/ 2608860..
Jinchao Ge, Zeyu Zhang, Minh Hieu Phan, Bowen Zhang, Akide Liu, and Yang Zhao. Esa: Annotation-efficient active learning for semantic segmentation. ArXiv, abs/2400.13444 2024. URL https: /appi. semanticscholar wrg/CorppsID. :ttp//orPpuID::211777566
Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. Repairing the cracked founda- tion: A survey of obstacles in evaluation practices for generated text. Journal of Artificial Intelligence Research, 77:103-166, 2023.
Martin Gellerstam. Translationese in swedish novels translated from english. Translation studies in Scandinavia, -188-95, 1986.
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini. Are we done with mmlu?, 2024. URL
https://arxiv.orgg/ bbargg/bs//40e/ 04127.
Gemma Team. Gemma 2: Improving open language models at a practical size, 2024. URL http:///arxxv..ogabbbbs///ss 00144008.01100
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Compulattional Linguistics, 10:522-538, 2022. doi: 10.1162/ tac4.a_00444. URL https: / / hatph//oaacnttooooooooooooooo org/2022. t. 30.
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. Continuous measure- ment scales in human evaluation of machine translation. In Antonio Pareja-Lora, Maria Liakata, and Stefanie Dipper (eds.), Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pp. 33-41, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL ht : http///aaaaaaaaogaaaoooocooo org/
wrg//133 /W1 3-2355.
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. Is machine translation getting better over time? In Shuly Wintner, Sharon Goldwater, and Stefan Riezler (eds.), Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pp. 443-451, Gothenburg, Sweden, April 201 Association for Computational URL httph:tt:// htttt::://aclanthooog.. org//14-100////
Linguistics. doi:
Yvette Graham, Nitika Mathur, and Timothy Baldwin. Randomized significance tests in machine translation. In Ondfej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, and Lucia Specia (eds.), Proceedings of the Ninth Workshop on Statistical Machine Translation, pp. 266-274, Baltimore, Maryland, USA, June 2014b. Association for Computational Linguistics. doi: 10.3115/11/114-00 URL https: //tc///cclccaa wnw.hhoolggyyyoooooooooooooooooe org //1g//144 /1////14----11 3333.
18

---

Published as a conference paper at COLM 2025
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, 23:3 - 30, 2015. URL https://api. stw.apiiissmmniiccooolraaa wwweeorrg//ooooo htte//CppusI-:.... 1892872.
Yvette Graham, Barry Haddow, and Philipp Koehn. Statistical power and translationese in machine translation evaluation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 72-81, Online, November 2020. Association for Computational Linguistics. doi: 10.18653 htt./2220..mmppp. 2020. emnlp-main.6.
URL https://aclanttooll org/
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey on llm-as-a-judge, 2025. URL https: //arxiv www.v.ogg/aaoe aaaaa//s/44/4 2211/2111111 15594.
Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979-995, 2024. doi: 10.1162 tac.lalll000000 URL https: / http:://aaaaathhoogeoo org/2024. tac1-1.54.
Yanzhu Guo, Simone Conia, Zelin Zhou, Min Li, Saloni Potdar, and Henry Xiao. Do large language models have an english accent? evaluating and improving the naturalness of 15956.
multilingual Ilms, 2024. URL https: //tp://///////.///a http:/iw..rg//b//////////
Srishti Gureja, Lester James V. Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Winata, Nathan Lambert, Sebastian Ruder, Sara Hooker, and Marzieh Fadaee, M-rewardbench: Evaluating reward models in multilingual settings, 2024. URL 15522.
https://arxivv
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. XL-sum: Large-scale multilin- gual abstractive summarization for 44 languages. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4693-4703, Online, August 2021. Association for Computational Linguistics. doi: 10.18653 /v11 2022 1001..indings--cl.41 URL https: https: haaps///oo/annhooooy..ooom org/ 2021. findings-anll..... 413.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dommun William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. Achieving human parity on automatic chinese to english news translation, 2018. URL https://arxiv https://axxiv.orgabb//////// wwwo/bs//////eeeeo ats//bb//100..... 05567.
Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, Shruti Bhosale, Chenguang Zhu, Karthik Abinav Sankararaman, Eryk Helenowski, Melanie Kambadur, Aditya Tayade, Hao Ma, Han Fang, and Sinong Wang. Multi-if: Benchmarking lims on multi-turn and multilingual instructions following, 2024. URL https://arxis:
abs///b/...../ 15553.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Oskar Holmström and Ehsan Doostmohammadi Making instruction finetuning accessible to non-English languages: A case study on Swedish models. In Tanel Alumäe and Mark Fishel (eds.), Proceedings of the 24th Nordic Conference Computational Linguistics (NoDaLiDa), pp. 634-642, Tórshavn, Faroe Islands, May 2023, University of Tartu Library. URL httpss//acllatthnt orgg/ nodal ida-1.62. 62.
19

---

Published as a conference paper at COLM 2025
Carolin Holtermann, Paul Röttger, Timm Dill, and Anne Lauscher. Evaluating the ele- mentary multilingual capabilities of large language models with MultiQ. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Compu- tational Linguistics: ACL 2024, pp. 4476-4494, Bangkok, Thailand, August 2024. Asso- URL
ciation for Computational Linguisties. doi: 10. 06653/v112224.fimm// https://aclancholooy..rrrg
findings-ac1.288
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross- lingual generalisation. In International conference on machine learning, pp. 4411-4421. PMLR, 2020.
Kaiyu Huang, Fengran Mo, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, Jinan Xu, Jian-Yun Yun Nie, and Yang Liu. A survey on large language models with multilingualism: Recent advances and new frontiers, 2024. URL https://arae htrggabb/24409 10936.
Adithya Venkatadri Hulagadri, Julia Kreutzer, Jian Gang Ngui, comprehensive multilin-
and Xian Bin Yong. Towards fair and gual Ilm benchmarking, 2025. -aw
URL https://cohere: /ttp:///cooerre.coo/bo/g///
towards-faiid
Neslihan Iskender, Tim Polzehl, and Sebastian Möller. Reliability of human evaluation for text summarization: Lessons learned and challenges ahead. In Anya Belz, Shubham Agarwal, Yvette Graham, Ehud Reiter, and Anastasia Shimorina (eds.), Proceedings of the Workshop Human Evaluation of NLP Systems (HumEval),, pp. 86-96, Online, April 2021. Association for Computational Linguistics. URL haap::///aclatthlooyy/ orgooog//22 2021. humeval-1, 10.
Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyán O'Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, and Barry Haddow. Emma- 500: Enhancing massively multilingual adaptation of large language models, 2024. URL https://arxiv.org// wtt.l//.ogg//////e/eeee/oo awrg/abs/2409. 17892.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in NLP. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, lz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4110-4124, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v12 //aclanthology. org/2021 gtt///2..naaclmmaaaa naacl-main. 324.
324. URL https:
Douwe Kiela, Tristan Thrush, Kawin Ethayarajh, and Amanpreet Singh. Plotting progress in ai. Contextual Al Blog, 2023. https:/ contextual.ai aa//boag//////// Blllog/poottnggppeee plotting-progress.
Ondrej Klejch, Eleftherios
Avramidis, Aljoscha Burchardt, and Martin Popel. Mt-compareval- Graphical evaluation interface for machine translation development. The Prague Bulletin of Mathematical Linguistics, 104:63 74, 2015. URL https: //ppi. semanticscholar org/ CorpusID: 1 1488228.
Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Lunzzys-Dommunt Hi- tokazu Matsushita, and Arul Menezes. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Tom Koemi, Andre Martins, Makoto Morishita, and Christof Monz (eds.), Proceedings of the Sixth Conference Machine Translation, pp. 478-494, Online, November 2021, Association for Computational Linguistics. URL httpt::/ssc/aaattiiitt www.aaaanaooooyyy...om org//021. 1.t-1.57.
20

---

Published as a conference paper at COLM 2025
Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Chris- tian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Ken- ton Murray, Makoto Nagata, Toshiaki Nakazawa, Martin Popel, Maja Popović, and Mariya Shmatova. Findings of the 2023 conference on machine translation (WMT23): LLMs are here but not quite there yet. In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Translation, pp. 1-42, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1 1tt5:////11222.mmmmm1/ URL https: htttps://aclanthoooogy wrg//023.... 2 wmt-1.1.
Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondfej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grund- kiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popović, Mariya Shma- tova, Steinthór Steingrimsson, and Vilém Zouhar. Findings of the WMT24 general machine translation shared task: The LLM era is here but MT is not solved yet. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference on Machine Translation, pp. 1-46, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 20066/////2224mmmm--11.m URL https://aclanthologgs org/2024. wmt-1.1
Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpin- ska, Maja Popović, Mrinmaya Sachan, and Maríya Shmatova. Error span annotation: A balanced approach for human evaluation of machine translation. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Con- ference on Machine Translation, pp. 1440-1453, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10.18653/ wtt.///2222t...... https://aclantholopg orgph///0/oyyyo//2224444 wmt-1.133
URL
Tom Kocmi, Vilém Zouhar, Christian Federmann, and Matt Post. Navigating the metrics maze: Reconciling score magnitudes and accuracies. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1999-2014, Bangkok, Thailand, August 2024c. Association for Computational Linguistics. doi: 10.18653/ ///////.....alooooggg. 110. URL https: orgpo/ooooo.r///2244 org/ acl-long. 110.
Philipp Koehn. Statistical significance tests for machine translation evaluation. In Dekang Lin and Dekai Wu (eds.), Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp- 388-395, Barcelona, Spain, July 2004. Association for Computa- tional Linguistics. URL https: http://aaaannnoolgoooooooooo hrrg/we4-3255
Philipp Koehn and Christof Monz. Manual and automatic evaluation of machine translation between European languages. In I'hilipp Koehn and Christof Monz (eds.), Proceedings on the Workshop on Statistical Machine Translation, pp. 102-121, New York City, June 2006a. Association for Computational Linguistics. URL /aalanthooy hwwpollooyyyygggggeeeeeeoo org/W06-3311
Philipp Koehn and Christof Monz (eds.). Proceedings on the Workshop Statistical Machine Translation, New York City, June 2006b. Association for Computational Linguistics. URL https://aclanthollgg.
Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. Okapi: Instruction-tuned large language models in multiple lan- guages with reinforcement learning from human feedback. In Yansong Feng and Els Lefever (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing: System Denionstrations, pp. 318-327, Singapore, December 2023. Asso- ciation for Computational Linguistics. doi: 10.18653/vil https://aclanthoology org. 2023. emnlp-demo 28.
URL
Alon Lavie and Abhaya Agarwal. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Chris -Burih, Philipp Koehn, Cameron Shaw Fordyce and Christof Monz (eds.), Proceedings of the Second Workshop on
21

---

Published as a conference paper at COLM 2025
Statistical Machine Translation, pp. 228-231, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https: http://aaaaatttoo......oo or org/W07-0734.
Sara Laviosa. Corpus linguistics and translation studies. In Perspectives corpus linguistics, Pp. 131-154. John Benjamins Publishing Company, 2011.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. RLAIF: Scaling reinforcement learning from human feedback with AT feedback, 2024. URL https://
openreview.net//eeee wwt..e/tooormmmidxx1ls500002
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024a. URL https:///mmyss:: htggbllg////oooo/ooo//ggggooggg/ 2eeg//224--0000 04-1000011111 19 arena-hard/.
Yafu Li, Ronghao Zhang, Zhilin Wang, Huajian Zhang, Leyang Cui, Yonging Yin, Tong Xiao, and Yue Zhang. Lost in literalism: How supervised training shapes translationese in llms. Xiviv preprint ar
2025.
Yucheng Li, Yunhao Guo, Frank Guerin, and Chenghua Lin. An open-source data con- tamination report for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 528-541, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 006663/v1/222.fimd// //aclanthology org org/2024.0. findings-emmlp.... 30.
URL https:
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Pp. 6008 8008-6018, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1. /vl/ 2020.emnlp-main-466 484. URL https: hattps://aclattholoog org/2020. 22200.00022 emnlp-main 1pppamn...- 484.
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichan- der, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking lims with challenging tasks from real users in the wild, 2024. URL 04770.
https://aaxxv.oog/bbbssss
Chaoqun Liu, Wenxuan Zhang, Jiahao Ying, Mahani Aljunied, Anh Tuan Luu, and Lidong Bing. SeaExam and SeaBench: Benchmarking LLMs with local multilingual questions in Southeast Asia. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp. 6119-6136, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-195- URL https: https://aclanthoologyy. org/2025.
7. doi: 10.18653/ httl//11/22///////// 211/222..indiiii 2tt.//2...iniing--aaeeeee naal.3. findings-naacl. 341/.
Yang Liu, Meng Xu, Shuo Wang, Liner Yang, Haoyu Wang, Zhenghao Liu, Cunliang Kong, Yun Chen, Yang Liu, Maosong Sun, and Erhong Yang. Omgeval: An open multilingual generative evaluation benchmark for large language models, 2024. URL https://arxiv org/abs/2402 13524.
Llama Team. The llama 3 herd of models, 2024. URL
Chi-kiu Lo. Extended study on using pretrained language models and YiSi-1 for machine translation evaluation. In Loic Barrault, Ondiej Bojar, Fethi Bougares, Rajen Chatter- jee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki
22

---

Published as a conference paper at COLM 2025
Nakazawa, and Matteo Negri (eds.), Proceedings of the Fifth Conference Machine Trans- lation, pp. 895-902, Online, November 2020. Association for Computational Linguistics. URL httt::://accannhooogg.o org.orgg///// /2rgg/2220..o 1mt-1.99.
Adam Lopez. Putting human assessments of machine translation systems in order. In Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia (eds.), Proceedings of the Seventh Workshop on Stafisfical Machine Translation, pp. 1-9, Montréal, Canada, June 2012. Association for Computational Linguistics. URL https://aclanthollo. org/w1 2-3101.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Teacher's pet: understanding and mitigating biases in distillation. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https: http:s:///pppeeveeeeeee http::////ppeeeieeeeeeee/oorom/ witppmmmddddpappppeeeeeee
Matouš Machátek and Ondłej Bojar. Results of the WMT13 metrics shared task. In Ondrej Bojar, Christian Buck, Chris Callison-Burch, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Herve Saint-Amand, Radu Soricut, and Lucia Specia (eds.), Proceedings of the Eighth Workshop on Statistical Machine Translation, pp. 45-51, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL
org/W13 org/W13-2202.
Shushen Manakhimova, Vivien Macketanz, Eleftherios Avramidis, Ekaterina Lapshinova- Koltunski, Sergei Bagdasarov, and Sebastian Möller. Investigating the linguistic per- formance of large language models in machine translation. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference on Machine Translation, pp. 355-371, Miami, Florida, USA, November 2024. Associa- tion for Computational Linguistics. doi: 10.18653/ 200033/v122244/ wmt-1.28. URL https: //aclanthology org o22/2024 182.
Benjamin Marie, Atsushi Fujita, and Raphael Rubino. Scientific credibility of machine translation research: A meta-evaluation of 769 papers. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), PP. 7297-7306, Online, August 2021. As- sociation for Computational Linguistics. doi:
aclogoo. URL
https: httpss//aclanthologg.o 2021. acl- moll-loog.560
Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F, T. Martins. Eurollm: Multilingual language models for europe, 2024. URL https://arxiv. org/abs/2409.. 16235.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4984-4997, Online, July 2020. Association for Computational Linguistics. doi: 2t..8653//1/2220.ad/e https://aclanthology.oo org/2222 moei.mmain.45
URL
Timothy R. Mcintosh, Teo Susnjak, Nalin Arachchilage, Tong Liu, Paul Watters, and Malka N. Halgamuge- Inadequacies of large language model benchmarks in the era of generative artificial intelligence, 2024. URL https::/arxxv..or org.as///////////////////../ss 24022, 09880.
Nikita Mehandru, Sweta Agrawal, Yimin Xiao, Ge Gao, Elaine Khoong, Marine Carpuat, and Niloufar Salehi. Physician detection of clinical harm in machine translation: Qual- ity estimation aids in reliance and backtranslation identifies critical errors. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 11633-11644 Singapore, December 2023. As- sociation for Computational Linguistics. doi: 10.18653 /v/l///vvv/// v// /1//222..e// 1002..mmll-mmin... URL https://aclanthonn org/2023. emnlp-main 712.
23

---

Published as a conference paper at COLM 2025
Evan Miller. Adding error bars to evals: A statistical approach to language model evalua- tions, 2024. URL htttts:///t/iiiii http://aaavvvvoggbbb/////s// wttp.b///////11////////////// .00400.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xi- angru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generaliza- tion through multitask finetuning. In Anna Rogers, Jordan Boyd- BoadeGra and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15991-16111, Toronto, Canada, July 2023, As- sociation for Computational Linguistics. doi: 10.18653 / / 2221/222.aaaa acl33..c-- https://aclanthologgy org wwg. aclaaclllooooo -lng.. 891.
URL
Graham Neubig and Junjie Hu. Rapid adaptation of neural machine translation to new languages. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, PP. 875-880, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 2211653/v1/D188
/
1103.
Shiwen Ni, Xiangtao Kong, Chengming Li, Xiping Hu, Ruifeng Xu, Jia Zhu, and Min Yang. Training on the benchmark is not all you need, 2025. / http://ww...ggb/////...../// 01790.
NLLB Team. No language left behind: Scaling human-centered machine translation, 2022. URL
24672
David Ong and Peerat Limkonchotiwat. SEA-LION (Southeast Asian languages in one family of Southeast Asian language models. In Liling Tan, Dmitrijs Milajevs, Geeticka Chauhan, Jeremy Gwinnup, and Elijah Rippeth (eds.), Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pp. 245- 245, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ org/o0g//22 1lposs-1.22
network):
v1/2023.nlposs-1.. URL
Simon Ott, Adriano Barbosa-Sill Silva, Kathrin Blagec, Jan Brauner, and Matthias Samwald. Mapping global dynamics of benchmark creation and saturation in artificial intelli- gence. Nature Communications, 13(1), November 2022. ISSN 2041-1723, doi: 10.1038/ /t////441477022-3455511
0414467-02--35591-0. URL http: dx. org/1/.r//1....
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Anmual Meeting of the Association for Compulational Linguistics, Pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1073003.10733333 URL https: htattp://aclaanhoooogg org/P02- 1040.
Irene Plaza, Nina Melero, Cristina del Pozo, Javier Conde, Pedro Reviriego, Marina Mayor- Rocher, and María Grandury. Spanish and 11m benchmarks: is mmlu lost in translation?, 2024. URL https: http///wwww....///bbob//
wttp:/vw.orr//////400000 17789.
Maja Popović. chrF:: character n-gram F-score for automatic MT evaluation. In Ondřej Bojar, Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina (eds.), Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 392-395, Lisbon, Portugal, September 2015, Association for Computational Linguistics. doi: 10.18653 00663//11/11500000 URL https:://aclanttoo..o org/W15-3049 3049.
Maja Popović. Agree to disagree: Analysis of inter-annotator disagreements in human evaluation of machine translation output. In Arianna Bisazza and Omri Abend (eds.), Proceedings of the 25th Conference on Computational Natural Language Learning, pp. 243, Online, November 2021. Association for Computational Linguistics. doi: 10.18653 /vll 2021.conll-1.n1 1.18. URL https: //acl wwwp/olooocoooooooooooooooooooo org/2021 con11-1.11
24

---

Published as a conference paper at COLM 2025
Maja Popović and Anya Belz. On reporting scores and agreement for error annotation tasks. In Antoine Bosselut, Khyathi Chandu, Kaustubh Dhole, Varun Gangal, Se- bastian Gehrmann, Yacine Jernite, Jekaterina Novikova, and Laura Perez-Beltrachiní (eds.), Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), Pp- 306-315, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. doi: (...1663/112222..g622 org//022..o... gem-1.66.
URL
Matt Post. A call for clarity in reporting BLEU scores. In Ondłej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Anto- nío Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor (eds.), Proceedings of the Third Conference on Machine Translation: Research Papers, PP. 186-191, Brussels, Belgium, October 2018, Association for Computational Linguistics. doi: 10.18653/ 10.18533/11WW18-333
URL https::/aclaatthooote wrrpl/ooooooooooooooe org/wrg///// (errg//18-6333 6319.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. Online, November 2020. Association for Computational Linguistics. doi: 10.18653 / /v1 /1/2020.emmlpmmaii 213.
URL htttp:://aclantholooggoo org/2020. emnlp-man main.
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Philipp Koehn, Loic Barrault, Ondłej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 578-585, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL
wrg./022//20 2022. 1.t-1.52.
Ricardo Rei, Nuno M. Guerreiro, JosÅ@ Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, and André Martins. Scaling up CometKiwi: Unbabel--ST 2023 submission for the quality estimation shared task. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Translation, pp. 841-848, Singapore, December 2023. Association for Computational Linguistics. doi: URL https: www.aaaoltttoyyo..o/ooo org.olllooooooooooooooooooooooot .rg.ogg////// 202g. wmt-1.73.
Stefan Riezler and John T. Maxwell. On some pitfalls in automatic evaluation and signifi- cance testing for MT. In Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss (eds.), Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 57-64, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL httpss////claaahooooooo org/W05-0900
Parker Riley, Daniel Deutsch, George Foster, Viresh Ratnakar, Ali Dabirmoghaddam, and Markus Freitag. Finding replicable human evaluations via stable ranking probability. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4908-4919, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1. wrw.antoooooooooo/222244444 naacl 275.
URL
Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A. Haggag, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Viraat Aryabumi, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad Duwal,
25

---

Published as a conference paper at COLM 2025
Daniil Dzenhaliou, Daniel Fernando Erazo Florez, Fabian Farestam, Joseph Marvin Im- perial, Shayekh Bin Islam, Perttu Isotalo, Maral Jabbarishiviari, Börje F. Karlsson, Eldar Khalilov, Christopher Klamm, Fajri Koto, Dominik Krzemiński,, Gabriel Adriano de Melo, Syrielle Montariol, Yiyang Nan, Joel Mikkaus,Jekkterrei Novikova, Johan Samir Obando Ceron, Debjit Paul, Esther Ploeger, Jebish Purbey, Swati Rajwal, Selvan Sunitha Ravi, Sara Rydell, Roshan Santhosh, Drishti Sharma, Marjana Prifti Skenduli, Arshia Soltani Moakhar, Bardia Soltani Moakhar, Ran Tamir, Ayush Kumar Tarun, Azmine Toushik Wasi, Thenuka Ovin Weerasinghe, Serhan Yilmaz, Mike Zhang, Imanol Schlag, Marzieh Fadaee, Sara Hooker, and Antoine Bosselut. Include: Evaluating multilingual language under- standing with regional knowledge, 2024a. URL https: / awvvvv.vvvvvvv//vv///..//./aa h/gg/b///2111// 241.11999..
Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A Haggag, Alfonso Amayuelas, et al. Include: Evaluating multilingual language understanding with regional knowledge, arviv preprint arXfv:2411.19799, 2024b.
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. XTREME-R: Towards more challenging and nuanced multilingual evaluation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Pp. 10215- 10245, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: //aclanthology 282/ mamlp-maiin. 802.
URL https:
Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. How good is your tokenizer? on the monolingual performance of multilingual language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3118-3135, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/200 /tt/:///11122...oooogo 243. URL https: ac/ wewlchnnnnoooyyyyyooooooooololll org/ wttgg/221..2a2 acl acl-long. 243.
Beatrice Savoldi, Alan Ramponi, Matteo Negri, and Luisa Bentivogli. Translation in the hands of many: Centering lay users in machine translation interactions. arXiv preprint d#Xiv:2502.13780, 2025.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual haii-offthhough reasoners. In The Eleventh International Conference on Learning Representations, 2023. URL https://opeerrvieewnee net/ forum?id=fRRWCCk IXp.
Luisa Shimabucoro, Sebastian Ruder, Julia Kreutzer, Marzieh Fadaee, and Sara Hooker. LLM see, LLM do: Leveraging active inheritance to target non-differentiable objectives. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 9243-9267, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.18653/ v1 v1/2024.emnlp-main.222 URL https: http:::///aaaatholloo/oo org/2024. emnlp-main. 521.
AI Singapore. Sea-lion (southeast asian languages in one network): A family of large language models for southeast asia. https: htt.tub...oomiiiiggopr//ssseeeae
2024.
Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, and Partha Taluk- dar. IndicGenBench: A multilingual benchmark to evaluate generation capabilities of LLMs on Indic languages. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), Pp. 11047-11073, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18653/ v1 /2222.22clllaccc acl-log.555 URL https://aclanttollgy..rrr// org.//gg..rr//22244 acl ael-llle long.-9.....
26

---

Published as a conference paper at COLM 2025
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei-Yin Ko, Madeline Smith, Antoine Bosselut, Alice Oh, Andre F. T. Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, and Sara Hooker. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation, 2024b. URL https:/ /taps://aaxxiie org/ abs/2412. 03304.
Shivalika Singh, Freddie Vargus, Daniel D'souza, Börje Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O'Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Moura, Dominik Krzemiñiski, Hakimeh Fadaei, Irem Ergun, Ifeoma Okoh, Aisha Alaagib, Oshan Mu- dannayake, Zaid Alyafeai, Vu Chien, Sebastian Ruder, Surya Guthikonda, Emad Al- ghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), PP. 11521-11567, Bangkok, Thailand, August 2024c. Association for Computational Linguistics. doi: 2088553v11202444 https://aclanthol..0g org/2024 acl aal--lome long..222
URL
Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. Kmmlu: Measuring massive multitask language understanding in korean, 2024a. https://///ar http://wwww..rg//////////// 11548.
Guijin Son, Dongkeun Yoon, Juyoung Suk, Javier Aula-Blasco, Mano Aslan, Vu Trong Kim, Shayekh Bin Islam, Jaume Trats-Cristià, Lucía Tormo-Bane Hemoo-Bauuelos, and Seungone Kim. Mm- eval: A multilingual meta-evaluation benchmark for llm-as-a-judge and reward models. ar Xiv preprint ar XXxv:2410.17578, 2024b.
Lucia Specia, Frédéric Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishrav Chaudhary, and André F. T. Martins. Findings of the WMT 2021 shared task on quality estimation. In Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, An- tonio Jimeno Yepes, Philipp Koehn, Tom Kocmi, Andre Martins, Makoto Morishita, and Christof Monz (eds.), Proceedings of the Sixth Conference on Machine Translation, pp. 684-725, Online, November 2021. Association for Computational Linguistics. URL https://aclanthology org/2021....2 111-1.71.
Miloš Stanojević and Khalil Sima'an. BEER: BEtter evaluation as ranking. In Ondłej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, and Lucia Specia (eds.), Proceedings of the Ninth Workshop on Statistical Machine Translation, Pp. 414-419, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. doi: 10.3115/11W1143354. URL https: / /aclanthool.. org/w14-3354.
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. Evaluating gender bias in machine translation. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Pro- ceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp- 1679-1684, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653 v1/P19-1164. URL
Hui Su, Xiao Zhou, Houjin Yu, Xiaoyu Shen, Yuwen Chen, Zilin Zhu, Yang Yu, and Jie Zhou. Welm: A well-read pre-trained language model for chinese, 2023. URL https: //arxiv.org/abb/2222 220.2210000 10372.
Haoran Sun, Renren Jin, Shaoyang Xu, Leiyu Pan, Supryadi, Menglong Cui, Jiangeun Du, Yikun Lei, Lei Yang, Ling Shi, Juesi Xiao, Shaolin Zhu, and Deyi Xiong. FuxiTranyu: A mul- tilingual large language model trained with balanced data. In Franck Dernoncourt, Daniel
27

---

Published as a conference paper at COLM 2025
Preotiuc-Pietro, and Anastasia Shimorina (eds.), Proceedings of the 2024 Conference Empir- ical Methods in Natural Language Processing; Industry Track, pp. 1499-1522, Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10.18653/v1 /20244 emnlp-industry.110 URL https: http///aaaaaaooooooooooooo org/ org/2024. 2024. emnlp- inppinnddtrr.... 110.
Alex Tamkin, Miles McCain, Kunal Handa, Esin Durmus, Liane Lovitt, Ankur Rathi, Saffron Huang, Alfred Mountfield, Jerry Hong, Stuart Ritchie, Michael Stern, Brian Clarke, Landon Goldberg, Theodore R. Sumers, Jared Mueller, William McEachen, Wes Mitchell, Shan Carter, Jack Clark, Jared Kaplan, and Deep Ganguli. Clio: Privacy-preserving insights into real-world ai use, 2024. URL https: //axii //11 13678.
Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, Fabio Barth, Johannes Leveling, Nicolas Flores-Hers Joachim Köhler, René Jäkel, and Mehdi Ali. Towards multilingual Iim evaluation for european languages, 2024. URL
Brian Thompson, Nitika Mathur, Daniel Deutsch, and Huda Khayrallah. Improving statisti- cal significance in human evaluation of automatic metrics via soft pairwise accuracy. In Barry Haddow, Tom Kocmi,Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference on Machine Translation, pp. 1222-1234, Miami, Florida, USA, November URL
2024. Association for Computational Linguistics. doi: 11116533v122024.wwa https://aclanthollogg. 0024 1.
Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. Attaining the unattainable? re- assessing claims of human parity in neural machine translation. In Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor (eds.), Proceedings of the Third Conference Machine Translation: Research Papers, pp. 113- 123, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 0166553/v1/1155
URL htwp:: htspttts///cllanthollo//// wwtp:///ww.../////1122222
Dennis Ulmer, Christian Hardmeier, and Jes Frellsen. deep-significance Easy and meaning- ful signifcance testing in the age of neural networks. In ML Evaluation Standards Workshop at the Tenth International Conference Learning Representations, 2022.
Ahmet Üstün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. Aya model: An instruction finetuned open-access multilingual language model. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15894- 15939, Bangkok, Thailland, August 2024. Association for Computational Linguistics. doi: 10.18653/ / 222202..allooogg.5ee
URL /tttp:://aclanthhoogg. org ptrg/2224..o llng. 845.
Rob van der Goot. We need to talk about train-dev-test splits. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4485-4494, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2000 /ttp//022..mmmmmma.....66.66 URL httpps//acclast http://aaaaaatooo....oooooo org/ 2021. emnlp-main. 368.
Laurène Vaugrante, Mathias Niepert, and Thilo Hagendorff. A looming replication crisis in evaluating behavior in language models? evidence and solutions, 2024. URL https: w//axxi..org/abs//// 20303.
Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry, Do large language model benchmarks test reliability?, 2025. URL https://aarix: wwwpivv..rrr/bb//eeeee 03461.
David Vilar, Gregor Leusch, Hermann Ney, and Rafael E. Banchs. Human evaluation of machine translation through binary system comparisons. In Chris Callison-Burche Philipp Koehn, Cameron Shaw Fordyce, and Christof Monz (eds.), Proceedings of the Second
28

---

Published as a conference paper at COLM 2025
Workshop Statistical Machine Translation, pp. 9--10,, Prague, Czech Republic, June 2007. org/W07-0713.
Association for Computational Linguistics. URL aatpss:/aclanttiiie
Jiayi Wang, David Adelani, Sweta Agrawal, Marek Masiak, Ricardo Rei, Eleftheria Briakou, Marine Carpuat, Xuanli He, Sofia Bourhim, Andiswa Bukula, Muhidin Mohamed, Temi- tayo Olatoye, Tosin Adewumi, Hamam Mokayed, Christine Mwase, Wangui Kimotho, Foutse Yuehgoh, Anuoluwapo Aremu, Jessica Ojo, Shamsuddeen Muhammad, Salomey Osei, Abdul-Hakeem Omotayo, Chiamaka Chukwuneke, Perez Ogayo, Oumaima Hour- rane, Salma El Anigri, Lolwethu Ndolela, Thabiso Mangwana, Shafie Mohamed, Hassan Ayinde, Oluwabusayo Awoyomi, Lama Alkhaled, Sana Al-azzawi, Naome Etori, Millicent Ochieng, Clemencia Siro, Njoroge Kiragu, Eric Muchiri, Wangari Kimotho, Toadoum Sari Sakayo, Lyse Naomi Wamba, Daud Abolade, Simbiat Ajao, Iyanuoluwa Shode, Ricky Macharm, Ruqayya Iro, Saheed Abdullahi, Stephen Moore, Bernard Opoku, Zainab Ak- injobi, Abeeb Afolabi, Nnaemeka Obiefuna, Onyekachi Ogbu, Sam Ochieng', Verrah Otiende, Chinedu Mbonu, Yao Lu, and Pontus Stenetorp. AfriMTE and AfriCOMET: Enhancing COMET to embrace under-resourced African languages. In Kevin Duh, He- lena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 1: Long Papers), pp. 5997-6023, Mexico City, Mexico, June 2024. As- sociation for Computational Linguistics. doi: 10. 11865///12224..aaccll https://aclanthologg wwwp:nhhhoooggooo///22222/2
URL
maacl-long 334.
Fangyun Wei, Xi Chen, and Lin Luo. Rethinking generative large language model evaluation for semantic comprehension. In Forty-first International Conference on Machine Learning, 2024. URL https: https: //ppsr////peeeeiieww www.:/ee///eeeeee
Johnny Wei, Tom Koemi, and Christian Federmann. Searching for a higher power in the human evaluation of MT. In Philipp Koehn, Lofc Barrault, Ondfej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussä, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 129-139, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL https::/aclaatthoote http:///laaantoogg....o.o wrgg//2222 1m7
Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, Tianxiang Hu, Shangjie Li, Binyuan Hui, Bowen Yu, Dayiheng Liu, Baosong Yang, Fei Huang, and Jun Xie. Polylm: An open source polyglot large language model, 2023. URL https: attps://arxivz http://axxxvv..oaab/..s wtwg/bs///////////// 2tt.//3370000000
Minghao Wu, Weixuan Wang, Sinuo Liu, Huifeng Yin, Xintong Wang, Yu Zhao, Chenyang Longyue Wang, Weihua Luo, and Kaifu Zhang. The bitter lesson learned from 2,000+ multilingual benchmarks, 2025. URL
Yi Xu, Laura Ruis, Tim Rocktäschel, and Robert Kirk. Investigating non-transitivity in llm-as-a-judge, 2025. URL
14074.
Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Yun Xing, Junjue Wang, Huitao Li, Xin Li, Kunyu Yu, Nan Liu, Qingyu Chen, Douglas Teodoro, Edison Marrese- Taylor, Shijian Lu, Yusuke Iwasawa, Yutaka Matsuo, and Irene Li. Mmlu-prox: A multilingual benchmark for advanced large language model evaluation, 2025. URL 10497.
https://arxiv. http:///aaxxv.og//bb////////
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao
29

---

Published as a conference paper at COLM 2025
Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https: //te:///axxxx http///axivv..r.oo www../orgabb//22.... 10671.
Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. Rethinking benchmark and contamination for language models with rephrased samples, 2023. URL httpss//aaxxvv.or/aabbaa
04850.
Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. Justice or prejudice? quantifying biases in llm-as-a-judge. Xiviv preprint ar arXiv:241.002736, 2024.
Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neubig. Pangea: A fully open multilingual multimodal I1m for 39 languages. arXiv wwwpii..rrgaab//441/1 16153.
preprint arXiv:2410.16153, 2024. URL
Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvittayakumjor, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Orăsan, Marina Fomicheva, André T. Martins, and Lucia Specia. Findings of the WMT 2022 shared task quality estimation. In Philipp Koehn, Loic Barrault, Ondfej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 69-99, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL org/222 1m.-1.3.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively mul- tilingual neural machine translation and zero-shot translation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 1628-1639, Online, July 2020. As- sociation for Computational Linguistics. doi: 10. 00.18633/v1/2220aad m48.. URL https://acllanthol..0 wwg. acl-main. 148.
Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. Share or not? learning to schedule language-specific capacity for multilingual translation. In International Conference on Learning Representations, 2021. URL /t
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024a. URL https://arxiv.. 2bss2200..1272.
Mike Zhang and Antonio Toral. The effect of translationese in machine translation test sets. In Ondfej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Gra- ham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor (eds.), Proceedings of the Fourth Conference Ma- chine Translation (Volume 1: Research Papers), pp. 81, Florence, Italy, August 2019. URL https:
Association for Computational Linguistics. doi: //aclanthology. org/W19- Mt//w/9550 52288
Wenxuan Zhang, Hou Pong Chan, Yiran Zhao, Mahani Aljunied, Jianyu Wang, Chaoqun Liu, Yue Deng, Zhiqiang Hu, Weiwen Xu, Yew Ken Chia, Xin Li, and Lidong Bing. Seallms 3: Open foundation and chat multilingual large language models for southeast asian 19672.
languages, 2024b. URL
30

---

Published as a conference paper at COLM 2025
Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. Don't trust ChatGPT when your question is not in English: A study of multilingual abilities and types of LLMs. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7915-7927, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v/ 2023. mo0.eemllp-ain.-99 URL https: h/tp://///aaaahhloooyyy 0023.023.... emnlp-main. m 491.
Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, and Tong Zhang. From lists to emojis: How format bias affects model alignment. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 26940-26961, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-99176-251-0 URL https://aclanthologg 2025 acl 1308/.
Yidan Zhang, Boyi Deng, Yu Wan, Baosong Yang, Haoran Wei, Fei Huang, Bowen Yu, Junyang Lin, Fei Huang, and Jingren Zhou. P-mmeval: A parallel multilingual multitask benchmark for consistent evaluation of llms, 2024c. URL http::::/aaxii..rrg/////// 09116.
Yiran Zhao, Chaoqun Liu, Yue Deng, Jiahao Ying, Mahani Aljunied, Zhaodonghui Li, Lidong Bing, Hou Pong Chan, Yu Rong, Deli Zhao, and Wenxuan Zhang. Babel: Open multilingual large language models serving over 90 URL httpp://arxiv..rrg/abaaa 00865.
Zhixue Zhao and Nikolaos Aletras. Comparing explanation faithfulness between multilin- gual and monolingual fine-tuned language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3226-3244, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653 /v1 2222 naa.l.naal--oong... 178. //aclanthology. .orgayo......... org/2024. wrg/2224.naacl1oom. naanl-long..... 178.
URL https:
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonza- lez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 46595-46623. Curran As- sociates, Inc., 2023a. URL https: https//ee/cccooooooeeeeccoo neurips. cc wwtppcpppppaeeeeeeeeeeeee wtweerfiilssppe//////// file/91 f18
Paaaaaaee--eeee-eaatettttttt and Benchmarks. pdf.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023b. wwwpeee.eeeffffriieeeeee net//forumiiduucccHGl...
URL
Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Jing Jiang, and Min Lin. Cheating automatic LLM benchmarks: Null models achieve high win rates. In The Thirteenth International Conference Learning Representations, 2025. URL forum?id=syThii TmWWm.
Shaolin Zhu, Supryadi, Shaoyang Xu, Haoran Sun, Leiyu Pan, Menglong Cui, Jiangcun Du, Renren Jin, António Branco, and Deyi Xiong. Multilingual large language models: A systematic survey, 2024a. URL https::/arxiv..or/aabssa
11072.
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Multilingual machine translation with large language models: Empirical results and analysis. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2762-27811, Mexico City, Mexico, June 2024b. Association for Computational Linguistics. doi: 10.18653 /vl/ 2024.findings 0024.innding--macc..
htt.httss///ccattteeet wtwpchhollgyyorr//2222444/ findings-naacl. 176.
31

---

Published as a conference paper at COLM 2025
Vilém Zouhar, Pinzhen Chen, Tsz Kin Lam, Nikita Moghe, and Barry Haddow. Pitfalls and outlooks in using COMET. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference on Machine Translation, pp. 1272-1288, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: wmt 1.121. URL https: / http://ccllnnooolloor////////// wmt-1. 121.
32

---

Published as a conference paper at COLM 2025
A Inspected Multilingual Generative Benchmarks
Benchmark
Test Size
Metricts)
Source
+Lngg
Translated?
Thandation
Plore-3000 (Coetarjaesi al, 2022)
#ik
Comet-22, ChrF++, sPBLRU
Wikinews, Wilk Wikijonior, Wikivoy- agc
200
Human
NTRDX-128 (Federmann al., 2022)
=2k
Comet-22, 22, SpBLUU
News from 2019 Neves, literary. e-commerce, 50- cial, speech
128
Uuman Uuuman
WMT General MT (Kocmi et al, 2021h, inter alia)
=2000
Comet-22
MT (Adelari al, 2012)
1000
ChrF
Online news sources
Summariizttoo
2001)
500-11k
ROUGE
BBC banslated XLSum
CrossSum-In In (Singh 20244)
ChrF
Human
Math
MCSM (Shi al, 2003)
250
Accuracy
CSMBK transltedd MC2M
10
Fhuman
AIrMESM (Adelani et al, 2024)
250
Accuracy
16
Human
Open-cndef goirnatior MTG (Chen al, 2022)
3000
derived from ROUGE
tranelated English taoks with hu- man post-cdits
5
Google Translate API
OMGEval (Liu et 2024)
301
win rate
selected prompts from Alpocali- val, translated, localized, vertfied LMARINN prompts
CFT-4
myArenal lard (Dang et al, 2020b)
500
with rate
23
Google Transiate AP NLLPP
Dolly transiated (hingh et al., 21244)
200
win cale
mixed prompts from Dutabricks
101
employees
Aya human annotated (Singh et al,, 2024c)
250
with rale
community sourced Aya dataset
Wiite (f et al., 2024)
wil- BLFU
Wiiting taoks, genertted by Clui-
240
Google Transiaate API
GPT
MultiQ (Hodkermann et al,, 2024)
200
11 Ludged accuracy
selected from LMSSS and GPT-4
137
Google Transiate API
generated queshons
Clul
SeaBench (Liu al, 2025)
score against reterence sgainst
human writien and localized
Men-hBBench (Sippppor, 2024)
Foraut Follesdigg
IHvvll (Singaponu, 2024)
translated IFEval
Human unspecified L.M
MIFFvall (7hang al,, 2004c)
96
Accuracy
tranelated and pout-editedd local- ined, fibered IFFvaa translated and localized IFEval, vecilfied, and expandeed with ad- ditional turns
10
MuhilF (He wt 2024)
454-909
Accuracy
7 Ulama 3.1 4058
Table 5: Public generative benchmarks for downstream text-based evaluation of multilingual LLMs. Note that WMT annually releases benchmarks for varying languages and domains that we summarize here under a single item. "Test size" counts the number of prompts in the test split per language.
Tab. 5 gives an overview of the multilingual generative benchmarks that we inspected for this paper. Tab. 1 summarizes these more concisely.
B Benchmark Adoption in Model Releases
Rank
Benchmark
Model Releases (BenchmarkeddSupppatee Languages)
Flores-200
Aya101 (99/101), Aya Expanse (22/23), Qwen2 (7/=30), EMMA (199/546), Eu (34/35), PangeaLLM SeaLLM (12/12), SEA-LION (4/13), Salamandra (3/35), Babel (25/25), Sailor2 (15/15)
MGSM
Aya Expanse (7/23), Llama3 (7/8), Qwen2 (107/=30), EMMA (10/546), PangeaL (107/39), SeaLLM (6/ 12). Salamandra (5/35), Babel (107/25)
XLSum
Aya101 (45/101), EMMA (44/546), FuxiTranyu (15/43), SEA-LION (4/13), Salamandra (2/35)
4
WMT Dolly translated
Eurol (16/35), auitiaay (3/43), PolyL M (4/8+) Aya101 (3/101), Aya
(23/23), EMMA (119/546)
mArenaHard
Aya Expanse (23/23)
Aya human-translater
Aya101 (5/101)
PolyWeite
EMMA (240/ 5546)
SeaBench
S (3/12)
SeaMT Bench
SEA-LIO LON (6/13)
SEA-IFEval
SEA-LION (6/13)
MTG
P (5/8)
Table 6: We rank open benchmarks from Table 1 on their popularity in model release reports. For each model we indicate in how many of its supported languages the model evaluated. For WMT General Benchmarks, we report the union of all subsets.
Table 6 indicates which of the benchmarks in 5 were included in recent (state March 2025) open (explicitly) multilingual model releases, 8 including Aya-101 (Üstün et al., 2024), Aya
Otthe models such as Gemma2 (Gemma Team, 2024) might have multilingual capabilities but are not explicitly stating that they do.
33

---

Published as a conference paper at COLM 2025
Expanse (Dang et al., 2024b), Llama3 (Llama Team, 2024), Qwen2 (Yang et al., 2024), EMMA- 500 (ji et al., 2024) (base model), EuroLLM (Martins et al., 2024), PangeaLLM (Yue et al., 2024) (multi-modal), Fuxi (Sun et al., 2024), PolyLM (Wei et al., 2023), SeaLLMss (Zhang et al., 2024b), SEA-LION (Ong & Limkonchotiwat, 2023), Salamandra (Barcelona Supercom- puting Center, 2024), Babel (Zhao et al., 2025), Sailor2 (Dou et al., 2025).² 9
C Multilingual Leaderboards
Lamggngs
Language
Evalusted Open MiLM
Langagesal
Europsan LLM Loaderboard
21
European
Llama3, EuroLLM, Qwena, 2y123
Alican Languages LLM Eval Leadetbourd
15
Ailkan
Liam3,, Ayc10)
n0
South-East Asian
1lama3, Qwee2, SM SEA-LION, Aya Expanse, Aya23
yea
Landebbord
Japanese Leaderboand
Japanese
Llam3 Quree2, Apa Expanse
Open Ko LLM Loaderboard
Koeean
Owen2. SeaLLM
Open Foetuguan Leaderboard
Open Pencan Landerhound
Partugurss
Pencam
Qwar2, Llema,, Ayu Expures, Aya23, SeaLIM
Quaen2 Asa Expumee, Llama3
no
yea
Arahic Looderboard
Atabic
Owen2. LJama,, Aya Expanse, Seal LM Aya23, Puro LM
CaschBench Leadefooard
Czach
Llama3
Fehhew M. Lnderrbond
Hebrew
Seal M. Aga Pepanse, Quenn, Aya75, I Lames
Open 1IM sedertoase
Pubish
Iaams,, Queee, Asa Poparne, Fnrel Ape
yeur
OpenLEM Turkids
Expans, Aya EureLLM, LlamaB, Qiven Owend Llam3, EureLLM
yos
LLM Leadetboard
French
Table 7: Non-English leaderboards evaluating multilingual models with their focus lan- guages and evaluated open MLLMs from Table 6. Based on the average ranking on the respective leaderboards, we measure if LLMs for the respective focus languages win over the more massively multilingual ones, restricted to models below 13B parameters. For leaderboards that involve multiple languages, we aggregate wins via majority votes. This table reflects the state of 10 February 2025, 7 March 2025 for the French leaderboard.
Table lists open, non-English leaderboards and the models from our overview in Section that they evaluate. We also report whether as of the current state, multilingual models or specialized target language models are in the lead in the size of up to 14B parameters.
D Generative Evaluation In Disguise: MMLU
Even though MMLU (Hendrycks et al., 2021) is by design a discriminative task (MCQA), it deserves to be discussed here the most popular benchmark for multilingual models to-date because of the seemingly ease of evaluation (one of four options is correct). The original MMLU data has been translated automatically and with humans in various efforts with various translation tools (X-MMLU in Okapi work, MMMLU (openAI), LLama3 report uses Google Translate for translation, GlobalMMLU (Singh al., 2024b)), replicated in other languages (Son et al., 2024a; Xuan et al., 2025), analyzed and corrected (MMLU- Redux (Gema al., 2024)), sub-categorized (GlobalMMMLU) extended (INCLUDE (Romanou et al., 2024a)), and critizices (Balepur et al., 2025). However, since LLMs are generators by design, evaluation is not straightforward, therefore multiple approaches exist, 10 such as based on likelihood rankings of answers, or exact string matching. These details are rarely specified in multilingual MMLU evaluations, but may make the difference for system ranking (Wei et al., 2024). MCQA tasks (the majority of them) can also be turned into a generative benchmark by stripping the answer options from the prompt and having LLM judge decide whether the model's generation matches the correct answer option, possibly in comparison with another model's generation (Wei et al., 2024). The generative form of evaluation has so far not been explored multilingually.
'e excluded models that do not report any generative evaluations, such as Ministral and Mixtral, or those that are only in-house (Qwen). Models might additionally have been benchmarked by external parties or competing model releases (egg. EuroLLM benchmarks Gemma on translation tasks).
htttps: htttep:://huggghee htw.ggiin.......oooo/////eeeeeee 11m- http:/ppennnmmmmmeeeeererrrrrrd leaderboard-mmlu
34

---

Published as a conference paper at COLM 2025
E
Multilingual vs Monolingual Models and Benchmarks
Individual language benchmarks and models often receive little recognition in multilingual LLM development. While extensive work on English monolingual LLMs is widely respected and adapted for other languages, monolingual or less massively multilingual models tend to be overlooked. One challenge in evaluation arises when moving beyond monolingual to multilingual settings, the language coverage of individual benchmarks and models often does not fully align. This mismatch can lead to benchmarks being deemed incomplete for certain models or, being overly extensive, unfairly penalizing models for languages they do not support.
One argument against specialization is the potential for sharing of information in multilingual settings where knowledge learned from one language can benefit others (Conneau et al., 2020; Artetxe et al., 2020b; de Souza et al., 2021), or general reasoning abilities transfer (Chang et al., 2024), especially with increased model sizes (Chang et al., 2023). Building monolingual models, however, provides the opportunity to specialize the model (Chang et al., 2024), including optimizing tokenization strategies (Chelombitko & Komissarov, 2024; Zhao & Aletras, 2024) and tailoring the training data to the specific linguistic characteristics of the target language (Su et al., 2023; Abonizio et al., 2025).
Experiment: Monolingual vs Multilingual Model Performance We compare a multilin- gual model, AYA EXPANSE 8B, with individual monolingual models on two open-ended generation tasks: general knowledge (Singh et al., 2024c), and more challenging set of math, code, and reasoning questions (Dang et al., 2024b). We cover diverse set of monolin- gual models, including those pretrained from scratch exclusively on single language, as well as models obtained by specialized finetuning of multilingual model. Our selection also spans models with language-specific tokenizers versus general tokenizers, and models specialized for domains such as code or math in contrast to general-purpose language models. The languages include French' H2 Chinese Arabic and Japanese GPT4-o is used as judge to evaluate the quality of generations in pairwise comparison setting.
Figure 11 shows that, in almost all cases, the multilingual model outperforms the monolin- gual counterparts in both evaluation sets. Prreviously on smaller scale Rust al. (2021) compared mBERT (Devlin et al., 2019) and pretrained monolingual BERT models across selected languages and showed that languages adequately represented in the multilin- gual model's vocabulary exhibit little to no performance degradation compared to their monolingual counterparts. Now at a larger scale, ranging from 3B to 9B parameters, we observe an even stronger pattern of multilingual models outperforming their monolingual counterparts.
Aside from the challenges of properly configuring an entirely new model to generate co- herent text in each new language, multilingual models also appear more powerful for open-ended generation tasks, consistently producing stronger outputs across languages. However, this advantage may also stem from factors such as a higher number of experimen- tal iterations, broader (rather than specialized) evaluation objectives, and the continuous updating and maintenance of multilingual models benefits that monolingual models, often developed in more "one-and-done" style might lack.
Unttpss:/huuginggaae htttep///iiiiiiieeec////rrrroott http:////cccccctatteeeeeeeeetttt Instruct-DPO-v1.2 12httpss//hhugiiggfee
/httss/s//nnggninooooooooddddddd co//dicte-i1//iittmmmeee---e o-nnsit
1Pnttps://hugiinggaae http:///hhaa.aa..0.//a/--11-----
#hntpps:/luggiigffaee.oo
-2-2222
phttpprr/tugginggaaa....///////p instructt
35

---

Published as a conference paper at COLM 2025
m--ArenaHard
dolly
PolyWrite
Chrese
Fiench
0.
0.2
C.4
6.6
0.5
3.0
0.0
0.2
0.4
0.
0.8
1.0
0.0
0.2
044
0.8
0.8
1.0
Win Probebilty
Win Probabiiity
Win Procebility
Monel ngall LLM
Tiee
Inghing LUM
Figure 11: Comparing performance of a multilingual model (Aya Expanse 8B) with language- expert Monolingual models on general open-ended questions (dolly-translated-200 (Singh et al., 2024c) and creative writing prompts from PolyWrite (Ji et al., 2024)) and a more challenging set of math, code, and reasoning questions (m-ArenaHard,, (Dang et al., 2024b))
F
Win Rate Comparisons
F1
Sampling
For our win rate comparisons $ 3.2, we sample 5 generations from each model with ancestral sampling (temperature=1.0) implemented vLLM. We chose this setup because we did not want to tune temperatures individually for each model, nor was there any guide from either model provider how to set it the best way. In hindsight, we noticed that lower temperature would have been beneficial for QWEN2.5, which explains why our AYA EXPANSE 8B win rates are more inflated than those in the Aya Expanse tech report (Dang et al., 2024b), especially for languages that had already lower quality. Upon communication with the authors, we found out that their evaluations were run with temperature=0.75, and were able to confirm with spot checks of few languages (from varying win rate buckets) that QWEN2.5 generations were of higher quality under that setup, see Table Win rates differences under different temperatures vary heavily, up to around 50 points in the most extreme case. For Japanese and Portuguese, even the directionality of the wins change: under temperatures 0.0 or 0.75, QWEN2.5 wins overall, while under temperature=1.0, AYA EXPANSE wins overall. Generally, this highlights how essential the documentation of decoding parameters for replication.
Language Temperature
WR AYA EXPANSE
WR QWEN2.5
WR
0.0
74.4
25.0
49.4
0.75
76.8
22.2
54.6
1.0
89.2
10.2
79.0
fa
0.0
59.0
40.6
18.4
0.75
71.0
28.2
42.8
1.0
90.4
9.2
81.2
ja
0.0
44.8
54.8
-10.0
0.75
47.8
51.1
-3.3*
1.0
70.8
28.2
42.6
pt
0.0
43.6
55.6
-12.0
0.75
41.8
57.0
-15.2
1.0
58,8
39.8
19.0
Table 8: The effect of temperature settings win rates (WR, in %) on marenaHard for pairwise comparisons between AYA EXPANSE 8B and QWEN2.5 7B INSTRUCT. Win rate differences notably higher under temperature=1.0. Non-significant differences (95% confidence interval) are marked with asterisk.
36

---

Published as a conference paper at COLM 2025
System
You are helpful assistant whose goal is select the preferred (least wrong) response for given instruction in language name.. Which of the following responses is the best one for the given instruction in language name? A good response should follow these rules: 1) It should be language name, 2) should complete the request in the instruction, 3) It should be factually correct and semantically comprehensible, 4) It should be grammatically correct and fluent.
Judge
Instruction: instruction
Response (A): completion
Response (B): completion b
FIRST provide concise comparison of the two responses. If one Response is better, explain which you prefer and why. If both responses are identical or equally good or bad, explain why.
SECOND, on new line, state exactly one of 'Response (A)' or 'Response (BY or 'TIE' to indicate your choice of preferred response.
Your response should use the format: Comparison: <concise comparison and explanation> Preferred: < <"Response (A) or 'Response (BY or 'TIE>>
Table 9: Prompts for LLM-as-a-judge evaluations
F2
LLM-as-a-Judge Prompting
For LLM-as-a-judge evaluations, use the prompts listed in tab. 9 and randomize the order of model generations to prevent position bias. When using GPT4o as judge, we use version 2024-11-20.
E3 Statistical Significance
Preliminary experiments with GPT4o-mini (2024-07-18) for the setup described in 3.2 revealed that standard errors for win-rates were much higher than for GPT4o, so that even 500 examples are not enough for the differences to be significant in Chinese
G Instruction Wording
In this experiment, we use the German questions from Include 44 (Romanou et al., 2024b) test set containing localized multiple-choice questions. MCQA test sets are usually evaluated with log-likelihood probability, however, when that is not possible, especially when com- paring against models behind API, researchers reformulate the questions into instruction following.
In this experiment, we show how much the instruction can change final system ranking, thus opening room for metric hacking. We design six different instructions in English (EN 1-6) and also translate them into German (DE 1-6), all listed in fig. 12. The model outputs is then automatically parsed with regular expressions to select the proper answer. The final prompt contains the instruction followed with the question and list of all four answers and only the instruction is changed between experiments.
Tab. 10 shows how different wording changes the final model accuracy for three different models.
H Translation Effects
H.1 Experimental Setup
The decoding setup is the same as for the win rate experiments described in Appendix F. We choose the set of languages because they are in the common of supported languages for our models of interest. We translate the original prompts via a pivot language back into the original language simulate translation effects on the prompts. The pivot language is English for all languages except English, and Portuguese for English. We translate with Google Translate, NLL.B-3,3, AYA EXPANSE 32B and COMMAND A have a diverse mix of
37

---

Published as a conference paper at COLM 2025
Aya Expanse 8B
Llama 3.1
Gemma 9B
EN 1
46.0%
50.4%
48.2%
EN 2
44.6%
48.2%
47.5%
EN 3
46.8%
48.2%
47.5%
EN 4
48.9%
48.2%
49.6%
EN 5
48.9%
50.4%
48,9%
EN 6
42.4%
51.1%
51.1%
DE
45.3%
51.8%
50.4%
DE 2
48.2%
46.8%
51.1%
DE 3
46.8%
48.9%
48.9%
DE 4
46.8%
48.2%
48.2%
5
45.3%
51.8%
48.9%
DE
43.9%
47.5%
48.2%
Table 10: Comparison of performance on German MCQA testset from Include 44 when using different instructions.
EN 1:
"Here's a multiple-choice question with answer options. Please respond with only the letter of the correct choice. Do not include any additional information in your answer.",
EN 2:
"Please examine the following multiple-choice question carefully and reply with just the letter corresponding to the correct answer. No additional text should be included in your response.",
EN 3: "Select the correct option and reply with its corresponding letter only.
Nothing else.",
En 4: "Here's question for you! Just type the letter of the correct answer
not provide any explanations or extra words.",
EN 5: "Respond using only the letter of the correct answer. Do not add anything else.".
EN 6: "Whatever you do, don't send anything besides the letter of the correct answer. No explanations, no extra words!",
DE 1: "Hier ist eine Multiple-Choice-Frags mit Antwortoptionent Bitte antworte nur mit dem Buchstaben der richtigen Auswahl. Füge deiner Antwort keine zusätzlichen Informationen hinzu."
DE "Bitte überprüfe die folgende Multiple-Choic--raage sorgfältig und antworte nur mit dem Buchstaben der richtigen Antwort. Füge deiner Antwort keinen zusätzlichen Text hinzu.".
DE 3: "Wähle die richtige Option aus und antworte nur mit dem entsprechenden Buchstaben. Sonst nichts.",
DE 4: "Hier ist eine Frage für dich! Gib nur den Buchstaben der richtigen Antwort ein { keine Erklärungen oder zusätzlichen Wörter DE 5: "Antworte
mit dem Buchstaben der richtigen Antwort. Füge nichts Weiteres hinzu.",
DE 6: "Was auch immer du tust, sende nichts außer dem Buchstaben der richtigen Antwort. Keine Erklärungen, keine überflüssigen Worte!"
Figure 12: English and German instructions for multiple choice question answering.
MT and m...M translators. For NLLB, we split the prompt into individual sentences with the sentence.split tter library, before translation, and concatenate the translations. We do not post-process the translations in any way, but we notice that the translations contain <unk>s, which can throw off generation models.
The translation template for AYA EXPANSE 32B and COMMAND A is the following: "You are professional translator. Translate from src language into target language. Return nothing but the translation." We did not do extensive prompt tuning, but noticed that AyA
penttps://githuie httpttss///ttii...c////aaaaa///s http:///ww....aaaaaa//aaaaaaeeea
38

---

Published as a conference paper at COLM 2025
EXPANSE 32B often answered prompts rather than translating them if we did not include an explicit instruction to only return the translation.
We evaluate outputs from LLAMA3.1 8B Instruct (Llama Team, 2024), GEMMA2 9B (Gemma Team, 2024), AYA EXPANSE 8B, and QWEN2.5 7B INSTRUCT (Yang et al., 2024) models.
H.2 Translation Quality
Table compares the corpus ChrF (Popović, 2015) and XCOMET-XL (Guerreiro et al., 2024) scores of roundtrip translations, and reference-free wmt23-<ometkiwit da-xl (Rei et al., 2023) scores for translation models on aya-human-annotater prompts. 17 According the roundtrip evaluation against the original prompt, Google Translate delivers the highest quality translations with a small margin over COMMAND A, followed by NLLB 3.3B and AYA EXPANSE 32B. NLLB translates notably better into Arabic and English than AYA EXPANSE 32B, while AyA translates better into Turkish and Chinese.
Model
Language
ChrF
XComet
CometKiwi
Google Translate
69.49
88,88
70.76
en
86.52
96.54
80.87
pt
81.77
96.95
80.83
tr
75,94
97.33
76.09
zh
32.15
88.52
70.68
Azg 69.17
93.65
75.84
NLLB 3.3B
61.05
84.52
68.83
en
79.88
95.52
81.12
pt 75.29
95.71
78.77
tr 60.66
92.60
71.01
zh
18.29
81.80
65.80
Aug
59.03
90.03
73.11
Aya Expanse 32B
ar
35.11
80.83
66.01
en
77.11
96.55
81.15
pt
75.70
95.22
79.08
tr
62.88
95.38
74.99
zh
22.26
84.89
68.64
Aug 54.61
90.57
73.97
Command A
ar
62.80
86.60
69.47
en
82.72
96.32
78.25
pt 67.06
96.00
79.63
81.90
97.12
75.88
zh 37.12
87.99
67.09
Aug
66.31
92.80
74.07
Table 11: Translation quality of prompt roundtrip translations of the Aya human annotated benchmark. ChrF and XČOMET are reference-based metrics computed for translations from pivot language target language, and quality estimated without references for the translation into the pivot language with COMET-KIWI.
H.3 Changes in Generation
We want to measure how translation affects the generations. For that purpose, we compute Spearman correlation between translation quality and the generation quality, relative to the untranslated version. Both quantities are computed with sentence-level ChrF. In Table 12 we report these correlations for various mLLMS and translation models. Overall, we find that correlations the generation to the generation for the untranslated prompt. For translations from NLLB and AYA this correlation is stronger than for Google Translate, as they also have lower
always positive, meaning the better the prompt translated, the closer
Sacrebleu signature: nrefs: case: mixed eff: | n space:no 2ersion:2.5.11
39

---

Published as a conference paper at COLM 2025
12000
10000
Total
8000
Ties
6000
4000
2000
ru
zh
ja
vi
de
ko
es
fr
pt
Cz
it
fa
ar
pl
uk
nl
Figure 13: Number of total and tied Chatbot Arena battles (total 100k) for non-English languages with more than 200 prompts from 2024.
quality and thereby cause more changes to the prompts. Across languages and translations, QWEN is the most susceptible to changes in the prompt.
Model
NLLB 3.3B
Aya Expanse 32B
Command A
Google Translate
Avg
Owen 2.5 Instruct
0.31
0.58
0,23
0.57
0.34
Gemma2 9B
0.41
0.43
0.34
0.28
0.29
Aya Expanse 8B Llama3.1 7B Instruct Aug
0.37
0.35
0.28
0.26
0.25
0.23
0.21
0.20
0.15
0.16
0.34
0.33
0.23
0.30
0.24
Table 12: Pearson correlation between translation quality and generation quality, for several translation and generation models, averaged across languages. NLLB translations correlate the strongest with changes in generations, and QWEN seems most susceptible to translation artifacts in prompts.
H.4 Changes in Win Rate
tab. 13 lists the win rates for all languages in the comparison of AYA EXPANSE 8B vs GEMMA
9B under GPT-4o as judge (2024-05-13). We can see that the translation of prompts affects win rates across the bench, with a magnitude depending on the language and translation model.
I Chatbot Arena Analysis
L1 Multilinguality
The recently released 100k conversations and preferences collected between June and August 2024 (lmarena-ai/ the previously released 33k data (Imsys/chatbot crrtottaarnaaacccaae wow.ooaaaiooonnsssseeeeeesssssse from April to June 2023 that was 88% English. The set of common languages with more than 200 prompts in each is: English, German, Spanish, French, Portuguese, Russian.
more multilingual (54%) than
I.2 Ties
Figure 13 shows the ratio of ties in human pairwise ratings for the five most prominent non- English languages from total of 100k Chatbot Arena battles that were collected between June and August 2024 (lmarena-al htt//areaaauuune wwwpraaneeeeeeeeeeeeeeeeee stats for the released battles from April to June 2023 (Lmsys/chatbot chatbot arena conversations)
and fig. 14 the same
40

---

Published as a conference paper at COLM 2025
OG
Translator
NLLB
GT
AYA
CMD A
ar
0.71
0.68
0.58
0.84
0.74
0,01
-0.05*
0.04*
0.15
0.06*
pt 0.08*
0.09*
0.13
0.02*
0.00*
0.29
0.43
0.42
0.40
0.42
zh -0.21*
0.16
-0.19*
0.20
-0.08*
Aug
0.18
0.26
0.20
0.32
0.23
Table 13: Win-rate differences (AYA EXPANSE 8B-GEMMA2 9B, "non-significant) on original prompts (OG) vs translated prompts from various translation models. Positive values mean that AYA EXPANSE wins, negative values mean that GEMMA wins.
Total
500
400
300
200
28.7 28.7%
28.8%
27.6%
32. 92.1%
29.8%
de
st
Figure 14: Number of total and tied Chatbot Arena battles (total 33k) for non-English languages with more than 200 prompts from 2023.
41

---

Published as a conference paper at COLM 2025
J Checklist for mLLM Evaluation
J.1 Evaluation Prompts
O Are evaluation prompts representative samples of all languages included in the evaluation?
Are evaluation prompts human-curated, localized or edited?
If using model generated prompts: Have you analyzed the data for potential biases?
If using translations: Have you estimated, reported, and attempted to optimize translation quality on this particular set of prompts?
J.2 Choice of Metrics
Are metrics adequate for all evaluated languages?
J.3
Statistical Testing
Does the evaluation include adequate statistical significance tests for all included languages?
Does the evaluation include an estimate of statistical power?
If using stochastic decoding, does it include estimates of sampling induced variance?
J.4 Aggregating Results Across Languages
Are metrics comparable across languages?
Is the aggregation of results disproportionately influenced by any outliers?
Are language support differences taken into consideration when aggregating results across languages?
Are language support differences documented with the results?
Are task- and language-specific scores reported?
J.5
Qualitative Insights
Are quantitative metrics accompanied by qualitative error analyses?
Are differences in metrics due to meaningful distinctions rather than incidental artifacts?
J.6
Reproducibility
Are the results calculated with standardized pipelines?
Is the evaluation code released?
Are exact evaluation prompts and format published?
Are model outputs released?
Are prompt-level evaluation scores released?
Are metric hyperparameters documented?
Are model versions documented?
J.7 Enabling Meta-Evaluation
Are any human evaluations (consensually) released?
42


Reviews:
[{"review_id": "pSMbtgtlF4", "reviewer": "Reviewer_YSt1", "summary": "The authors give recommendations for the evaluation of (L)LMs, taking inspiration from learnings in machine translation evaluation from the past ~20 years. Issues with current (multilingual) evaluation methodology are laid out as well, like using machine translation for translating prompts or using small data sets for evaluation. Some specific issues that are discussed:\n- Translationese: The authors show with round-trip translation via a pivot language that the worse the MT system is that's used to produce the translations, the easier the task becomes.\n- Significance testing: In MT evaluation, sigtests are widespread.\n- Averaging over languages: The authors show that categories like high- or low-resource languages are also useful for LM evaluation.\n- Call for more in-depth evaluation: Performance on long vs. short promps is the example here.\n- Reproducibility: The authors call for reproducible benchmarks, as exemplified by sacreBLEU which outputs identifiers for how it was run.\n- The authors point out that when doing LM-as-a-judge comparisons, the results may largely depend on the input language and evaluation LM.\n- Lastly, the authors call for meta-evaluation, to come up with evaluation protocols or quality dimensions (like MQM).", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Insightful analysis of LM evaluation in contrast to work in MT.\n- Clear recommendations for LM evaluation.", "reasons_to_reject": "None", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1SVoFSaRPX", "reviewer": "Reviewer_ieqc", "summary": "The paper addresses evaluation of multilingual large language models on different languages and different tasks.\nSince this type of evaluation is very new and the standards are not yet established, the authors propose to use the 20-year experience in machine translation evaluation. \nThe authors describe each of the current problems/challenges and present the recommended solution learnt from machine translation evaluation.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The addressed problem is important and not investigated yet.\n\nThe well-established principles in machine translation evaluation can really be helpful, and the authors do a good analysis and useful recommendations which will be useful for the community.\n\nThe paper is well-written (only some definitions/explanations are missing, see \"drawbacks\" below)", "reasons_to_reject": "not really a reason to reject, but it should be improved: \n\nnamely, the data sets ('benchmarks') should be shortly described, in order to better understand what is going on.\n\nfor example, in machine translation test sets there is the source text, a human-translated text as a correct solution, and machine-translated text to be evaluated. Then, the machine-translated text is compared with the human-translated text. \n\nIt is, however, not explained, what the generative data sets consist of and how the evaluation is carried out, which makes a number of the described details unclear.", "questions_to_authors": "line 67: automatic prompt translations -- what the benchmarks are consisting of? Of prompts and the expected responses? Or of something else? That should be clarified briefly.\n\nTable 1: how the popularity ranks are defined and calculated? What do the yellow stars mean? (orange stars represent the rank, right?)\n\nWhat is the unit of \"Size\"? Number of documents? \n\nWhat do the task symbols mean? \n\nLLM-judged benchmarks are the contrast to what? To human-judged ones? \n\nAll that can be explained/defined in table caption.\n\nline 74: what is \"prompt sourcing\"?\n\n\nline 76: what is \"test split\"? Maybe it is a \"test set\", or \"test part\"? (in contrast to development/validation set/part)\n\n\nline 101: heavily rely on aggregation -> what does that mean exactly? \n\n\n\nline 124: what are \"human annotated prompts\"?\n\nThat should also be explained in section 2 (what the data sets consist of)\n\nline 125: Why via a pivot language and not directly? Which pivot language?\nOK, round-trip translation of original prompt into a pivot language and back -- but what was the pivot language? English? Was the same language used for all five languages? That is very important. \n\n\nTable 3 appears way before the text which describes it -- it would be better to appear closer (and after the text is better than before the text)\n\nAlso, it is not clear what \"all\", \"high\", \"medium\" and \"low\" mean from the table caption (the description should be added)\n\nline 334: periodically releases => periodically release\n\nline 347: on on correlation", "extra_scores": {}}, {"review_id": "0rRgSkmYPD", "reviewer": "Reviewer_gL8s", "summary": "This position+empirical paper argues and showcases several current problems in the multilingual evaluation (of LLMs). Many issues have appeared in machine translation research (hence Déjà Vu): translated data sources, a lack of trustworthy and standardised reporting, the use of neural/LLM metrics, and the need for meta-evaluation from different aspects.  Authors ran small-scale experiments to demonstrate the current limitations in evaluating LLMs and recommended better practices or standards for future work.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors of this paper identified several drawbacks in the current multilingual LLM evaluation practice. These cover almost every aspect of the evaluation process, like data sourcing, prompts, metrics, reporting, and meta-evaluation. I consider the work to be quite comprehensive.\n- Their arguments are well-supported by reasoning, experiments, and visualisations (mostly Sec 3) as well as related past experience (mostly Sec 4).\n- I think the work can have a good impact in the field. The \"lessons\" can be consulted by practitioners to help the community conduct evaluations that are more transparent, fair, and meaningful.", "reasons_to_reject": "I vote to *accept* this paper. \n\nHere I wish to point out that the organisation of the paper was causing some confusion:\n  - Sec 3 questions (section titles) do not correspond to problems in Sec 2, so I had to go back and forth to check which recommendation addresses which drawback.\n  - Sec 3 suggests adopting MT evaluation practices, and Sec 4 calls for meta-evaluation, but most citations still go to MT papers. To me, the difference is that Sec 4 is mostly \"position\" w/o experiments, but both sections advocate for adopting MT practices.", "questions_to_authors": "In 3.3 (score aggregation): perhaps reporting the number of wins is also meaningful, since it accounts for language coverage without making assumptions about the score distribution [1].\n\nIn 4.1 (metrics) and 4.2 (human eval): it may be helpful to add some discussions on i) how LLMs can be over-optimised towards neural metrics [2] and ii) even human preference may be gamed (via emoji, formatting) and does not reflect utility [3].\n\n[1] Pitfalls and Outlooks in Using COMET\\\n[2] Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking\\\n[3] From Lists to Emojis: How Format Bias Affects Model Alignment", "extra_scores": {}}, {"review_id": "XCIwKOe7vN", "reviewer": "Reviewer_5tiH", "summary": "This position paper aims to raise awareness of the flaws of procedures that are currently used to evaluate multilingual language models. For this, it draws a comparison with the long history of evaluation and meta-evaluation best practices in machine translation, which have been developed notably in the course of the WMT workshops since 2006.  Five recommandations are then made regarding evaluation protocols and the best ways to report their findings, in the view of making them more transparent and reproducible. These recommandations are illustrated with focused experiments with contemporary mLLMs. A final section is devoted to meta-evaluation.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n\n- an actionable list of recommandation for evaluating mLLMs - focused more on the comparison of mLLMs using multilingual benchmarks, than on a proper evaluation of the actual multilingual abilities of models;\n- a call for cautiousness when meta-evaluating the evaluation of LLMs with crowd-sourced judgments (ELO);\n- a very extensive review of existing work on the evaluation of mLLMs, as well as a thorough historical coverage of (meta-)evaluation in the statistical and neural MT eras.", "reasons_to_reject": "Weaknesses:\n- such calls for more scientific evaluations are not entirely new (eg. (BigScience, 2022 \"Bloom: a 175b multilingual model\"; Biderman et al 2024 \"Lessons from the trenches\", inter alia - neither is cited)). These missing reference are minor issues, but they hint at the fact that the community does not just need more recommandations, but also a discussion on how to enforce them - eg. via checklists that the evaluation of mLLMs should adhere to.\n- recommandation #1 (against using translated test sets) could be made a bit stronger - we see a general proliferation of translated test sets, that can only increase as the choice of candidate MT systems widens, all this adding an extra layer of confusion when comparing papers and systems . One lesson of all these MT evaluation exercices is that test sets must be prepared with all the necessary care (eg. avoiding translationese effects) and human intervention, and yes, redone every year. \n- I also tend to view recommandation #3.3 as a bit problematic and ill-motivated. There are strong arguments (Colombo et al, \"What are the best NLP systems\" proc. NIPS 2023) against averaging across languages, and even more so across languages. Grouping languages representativeness in the data by vague categories such as high-resource / low-resource is not always possible, and does not solve the problem of aggregating numbers that are not directly comparable; this cannot be a lesson learned from the history of MT where averaging BLEU scores across languages has never been regarded as making much sense;\n- using LLMs-as-judge in a multilingual setting is also becoming a serious issue, worth perhaps a recommandation on its own. \n- Finally, there are two more lessons from the MT community that I did not see mentionned: (a) human evaluation are more reliable than automatic metrics. Compiliing human scores also enables to progress on the meta-evaluation of automatic metric; (b) do not evaluate systems whose training data is not public - especially for many commercial mLLMs for which the training language mix is unknown.\n\n\nMinor issues:\n\n- What is the colour code in table 1? Tasks, as the small tasks icons tend to suggest. Could be explicit in the caption.\n- In the reference, make sure to upcase acronyms and capitalise language names.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper tries to transfer lessons learnt from decades of MT evaluation to the evaluation of multilingual LLMs (mLLMs), highlighting for every area in that problem possible ("like data sourcing, prompts, metrics, reporting, and meta-evaluation", reviewer gL8s) insights from the MT literature, and performing small-scale experiments to illustrate claims and ideas. One reviewer in particular appreciated the thorough review, but raised concerns about some recommendations like averaging across languages.

Overall a useful review and conversation paper, reviewers seem to all only highlight minor issues if any at all. Personally, I appreciated the thoroughness with which concepts were evaluated, as shown for example by the inclusion of Figure 2, and agree that this could become a well-cited paper with influence in the field of mLLMs.
Decision: Accept