# RoCS-MT v2 at WMT 2025: Robust Challenge Set for Machine Translation

Rachel Bawden, Benoît Sagot

<figure><img src="image_1.png" /></figure>

To cite this version:

Rachel Bawden, Benoît Sagot. RoCS-MT v2 at WMT 2025: Robust Challenge Set for Machine Translation. WMT 2025 - Tenth Conference on Machine Translation, Nov 2025, Suzhou, China. pp.834-849. hal-05344725

HAL Id: hal-05344725

https://hal.science/hal-05344725v1

Submitted on 3 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
RoCS-MT v2 at WMT 2025:
Robust Challenge Set for Machine Translation

Rachel Bawden Benoît Sagot

Inria, Paris, France

firstname.lastname@inria.fr

Abstract

RoCS-MT (Robust Challenge Set for Machine Translation) was initially proposed at the test suites track of WMT 2023. Designed to challenge MT systems' translation performance on user-generated content (UGC), it contains examples sourced from English Reddit, with manually normalised versions, aligned labelled annotation spans and reference translations in five languages. In this article, we describe version 2 of RoCS-MT in the context of the 2025 WMT test suites track. This new version contains several improvements on the initial version including (i) minor corrections of normalisation, (ii) corrections to reference translations and addition of alternative references to accommodate for different possible genders (e.g. of speakers) and (iii) a redesign and re-annotation of normalisation spans for further analysis of different non-standard UGC phenomena. We describe these changes and provide results and preliminary analysis of the MT submissions to the 2025 general translation task.

# 1 Introduction

Large language models (LLMs) have truly arrived in the field of Machine Translation (MT); their performance often rivals that of dedicated MT systems across various domains (Kocmi et al., 2023; Xu et al., 2024; Cui et al., 2025). However, while they have opened up new possibilities for translation, enabling fine-grained control of output formats, styles and formalities, they are also characterised by new types of errors that were less present with dedicated models, such as translating in the wrong language, inappropriate copying of the source text, generation of outputs that are not translations of the source text, etc. Evaluation continues to be a highly important aspect of the field, and as MT technologies progress, so does the way in which we evaluate. The WMT shared tasks are a good example of this, with an evolution a few years ago to general translation instead of news translation

(Kocmi et al., 2022), in order to challenge systems to translate a wider range of domains. One of the selected domains is social media content, known for covering a wide range of topics and containing non-standard language typical of user-generated content (UGC) (Foster, 2010; Seddah et al., 2012; Eisenstein, 2013; Baldwin and Li, 2015; van der Goot et al., 2018).

The translation of UGC has been a topic for a number of years (Belinkov and Bisk, 2018; Michel and Neubig, 2018; Vaibhav et al., 2019; Park et al., 2020; Nishimwe et al., 2024; Peters and Martins, 2025). In particular, a shared task was organised on the matter at WMT in 2023 (Kocmi et al., 2023), designed to target non-standard language from Reddit forums. Several parallel test sets of UGC texts exist (Ling et al., 2013; Vicente et al., 2016; Sluyter-Gäthje et al., 2018; Michel and Neubig, 2018; Rosales Núñez et al., 2019; Mubarak et al., 2020; Fujii et al., 2020; McNamee and Duh, 2022), for a range of languages, although they differ according to the language pairs covered and the degree of non-standardness present.

In the 2023 edition of the test suite track at WMT, we proposed the RoCS-MT test suite (Robust Challenge Set for MT) (Bawden and Sagot, 2023), designed to contain particularly challenging sentences with respect to their non-standard nature (e.g. with spelling mistakes, use of acronyms, marks of expressiveness, devowelling, contractions, etc.). Sourced from English Reddit, we aimed to cover a range of these phenomena in the texts selected, which we manually segmented into sentences, normalised into standard English (according to guidelines that normalised as reasonably possible whilst preserving fluency and meaning) and then translated by professional translators into French, German, Czech, Ukrainian and Russian.

For this 2025 edition of WMT (Kocmi et al., 2025), we resubmit RoCS-MT in an improved version (v2), after (i) some minor corrections to the
source-side normalisations, (ii) corrections to the existing references and addition of multiple references to account for different genders and (iii) improvements to the annotations of the non-standard phenomena for additional analysis. We release this version in Huggingface's Datasets (Lhoest et al., 2021).<fnref n="1" /> In this paper we describe those changes and provide results and analysis for the WMT2025 MT systems, with a major difference being that all systems this year were applied at the document level (at the level of Reddit post text in out case). We compare different segmentations of the texts and the performance of systems when applied to the original and normalised source texts.

## 2 The Test Suite

**Composition** The main composition of the challenge set is described in the article presenting the first version (Bawden and Sagot, 2023). The English source texts are taken from Reddit (all varieties of English including some non-native language, although we avoided code-switching). Candidate posts were identified using keyword searches on the Reddit API and chunks of text were manually selected from within those posts. The selected texts were manually segmented into sentences (non-trivial since many texts did not contain standard punctuation and sometimes contained newlines within sentences) and manually normalised. The normalisation guidelines we drew up aimed to balance (i) normalising as much as possible and at the same time (ii) rendering the output natural and realistic and (iii) not over-normalising to avoid losing the original text's style. For example, we did not use normalised variants that could be spontaneously and naturally used (e.g. we kept *lol* instead of *laughing out loud*. Finally, translations of the normalised texts were professionally produced in five languages: French, German, Czech, Ukrainian and Russian. Although not all these language directions are represented in this year's shared task, these references remain relevant for future use of the challenge set for these five languages.

**Changes with respect to the First Version** Several changes were carried out in this second version of the test suite, namely:

* Minor corrections to the normalisations of the source-side texts; we corrected a few

wrong normalisations and typos and reverted some hypercorrections to make sure that certain grammatical variations due to dialectal differences were conserved (i.e. not over-normalising)

* Corrections to some references after a manual check, including making sure that emojis and emoticons were always included in the references (this was not the case previously, notably for the Russian translations). For certain target languages, we also complete the reference translations with alternations for multiple possible genders where appropriate (namely where the gender is underspecified given the available context of the Reddit post).

* Re-annotation of the normalisation spans according to a new annotation scheme that is organised hierarchically and structures the types differently.

The new annotation types can be found below. Detailed descriptions including examples can be found in Appendix A.

* **Punctuation, typographic conventions, symbols, etc.:** punct:diff, punct:norm, caps, slash_to_or, slash_to_and, slash_distribution, word_to_symbol, symbol-placement

* **Spacing:** spacing, spacing:camelcase

* **Phonetically similar spellings (including imitation of speech):** phon, phon:char, phon:digits, phon:cute, phon:hesitate, phon:sound, phon:interjection

* **Other spelling variations (ergographic, expressiveness):** elongation, devowelling, contraction, truncation, acronym, abbreviation

* **Spelling mistakes:** spell, spell:charswap

* **Misc:** digit_letter_sim, letter_to digit, suffix

* **Added and dropped words:** word_drop, word_drop:pronoun, word_drop:det, word_add, word_add:det, symbol_drop, symbol_add

* **Grammar:** inflection, grammar, grammar:v, grammar:v:inflect

* **Lexical changes:** lexChoice, surrounding Emphasis, emoticon, censored

<footnote n="1">¹https://huggingface.co/datasets/rbawden/RoCS-MT-v2</footnote>
# 3 WMT 2025 submissions

There were 56 submissions to the 2025 general task (including variants of the same systems and systems run by the general MT task organisers) that translated the test suite. A range of architectures were used, with a majority using LLMs. In a bid to encourage document-level translation, one of the important factors to be taken into account in MT evaluation (Läubli et al., 2018), this year's general MT task focused on document-level MT, where documents were typically paragraphs of text. For the test suites, individual segments as provided in the test suite were concatenated to form source documents to be translated in one go. RoCS-MT was provided in two different formats: (i) manual segmentation and (ii) segmentation purely based on newlines within posts.

The language directions of the 2025 shared task overlap somewhat with the 2023 language directions (e.g. English to Czech, Chinese, Japanese, Ukrainian and Russian), although not all target languages for which we have reference translations are present (e.g. French and German), and many of the language directions are new and therefore do not have references (e.g. Arabic, Bhojpuri, Estonian, Icelandic, Italian, Korean, Massai, Serbian). We therefore choose in our initial results and analysis (see Section 4) to concentrate on quality estimation (QE) (i.e. without using the references for automatic analysis). We use two different metrics to calculate the scores (at the document level) used for the rankings: (i) CometKiwi (Rei et al., 2022),² and (ii) MetricX (Juraska et al., 2024).³ This is to avoid bias towards a single metric, especially as many systems optimise for a particular QE metric. We acknowledge however that (i) these metrics may well have issues handling certain languages, particularly those not explicitly included in the training data of the underlying models, and (ii) the scores may well favour models that optimise for QE in general, even if the same QE model is not used.

# 4 Results and Analysis

In this section, we present results for the submitted systems along with several brief analyses. In order to compute rankings, we took into account the fact that not all systems took part in every language pair, and adopted the ranking algorithm commonly

used for nations in the Olympic Games, which addresses a comparable situation. We observed that if systems are ranked based on their absolute scores for each language pair, a few systems are consistently ranked first, meaning that most systems never achieve a rank of 1 or 2, even if they have relatively high overall scores. The result of this is that an overall weaker system that has low scores in most languages but happens to be ranked highly for a single language pair can be ranked higher than a system that has high but not the best scores across all language pairs. We therefore choose to apply the “gold first” algorithm on quartile-based rankings instead of raw rankings; for a given language pair, all systems within the top 10% of scores are assigned to rank 1 and treated as such by the “gold first” algorithm, the next 10% to rank 2, and so on. In other words, systems are then ordered according to the number of language pairs in which they achieve rank 1 (i.e. belong to the top 10%). In the event of a tie, the number of rank 2 placements (top 10–20%) is considered, followed, if necessary, by the number of rank 3 placements (top 20–30%), and so on.

We applied this approach to get rankings from each of the two metrics used. These rankings are based on the original inputs (i.e. before normalisation) that have been manually segmented into sentences. We then computed overall rankings based on the average of the two rankings. Table 1 displays both metric-specific rankings and the overall ranking, the number of first-, second- and third-ranks for each system and each metric and the absolute gap between each system’s CometKiwi-based and MetricX-based ranks, to get an idea of how consistent they are and the confidence we can place in the resulting rankings. Appendix B provides raw CometKiwi and MetricX scores per language pair, first comparing scores on original and normalised inputs, then comparing scores on manual and new-line segmentations applied to original inputs.

Results highlight a small group of systems that dominate performance. Yolu occupies the top position with nine first-place results across all twelve language pairs on both metrics, followed closely by Shy-hunyuan-MT and CommandA-WMT. Laniqo and SalamandraTA follow, despite the fact that Laniqo participated in only seven pairs. Among the organiser-run systems, GPT-4.1 ranks 7th, closely followed by ONLINE-B and both TowerPlus models. Several larger LLM-based baselines, such as Claude-4 (18th) and both Gemini models (also

²WMT22-COMETKIWI-DA model. Higher is better.
³METRICX-24-HYBRID-XL-V2P6 model. Lower is better.
<table><caption>Table 1: Main ranking table. For each system and for both the CometKiwi and MetricX metrics, we provide their rank according to the metric, computed using the “gold first” scoring algorithm (Rank), the number of language pairs for which the system ranked first, second and third (“Medals”). For each system we also provide the number of language pairs it participated in (#lp), an overall rank based on the average between the CometKiwi- and the MetricX-based ranks, and the difference between the two original ranks, which shows for each system how consistent the two metrics are. Systems run by the task organisers are marked with an asterisk, while systems fine-tuned to optimise a Qe metric, such as CometKiwi and MetricX, are indicated with a °.</caption><thead><tr><th rowspan="2">System</th><th rowspan="2">#lp</th><th colspan="2">CometKiwi</th><th colspan="2">MetricX</th><th rowspan="2">Overall rank</th><th rowspan="2">Δrank</th></tr><tr><th>Rank</th><th>"Medals"</th><th>Rank</th><th>"Medals"</th></tr></thead><tbody><tr><td>Yolu</td><td>12</td><td>1</td><td>9, 1, 0</td><td>3</td><td>9, 1, 1</td><td>1</td><td>2</td></tr><tr><td>Shy-hunyuan-MT</td><td>12</td><td>4</td><td>6, 4, 2</td><td>1</td><td>11, 0, 1</td><td>2</td><td>3</td></tr><tr><td>CommandA-WMT</td><td>12</td><td>2</td><td>7, 3, 1</td><td>4</td><td>8, 2, 0</td><td>3</td><td>2</td></tr><tr><td>Laniqo°</td><td>7</td><td>5</td><td>6, 1, 0</td><td>5</td><td>4, 3, 0</td><td>4</td><td>0</td></tr><tr><td>SalamandraTA</td><td>11</td><td>3</td><td>7, 2, 1</td><td>8</td><td>1, 0, 1</td><td>5</td><td>5</td></tr><tr><td>GemTrans</td><td>12</td><td>12</td><td>1, 2, 1</td><td>2</td><td>10, 1, 0</td><td>6</td><td>10</td></tr><tr><td>UvA-MT</td><td>12</td><td>7</td><td>2, 4, 4</td><td>16</td><td>0, 5, 3</td><td>7</td><td>9</td></tr><tr><td>*GPT-4.1</td><td>12</td><td>17</td><td>0, 4, 1</td><td>6</td><td>1, 8, 1</td><td>7</td><td>11</td></tr><tr><td>*ONLINE-B</td><td>11</td><td>8</td><td>2, 1, 4</td><td>20</td><td>0, 2, 2</td><td>9</td><td>12</td></tr><tr><td>*TowerPlus 9B</td><td>12</td><td>9</td><td>1, 5, 2</td><td>21</td><td>0, 2, 0</td><td>10</td><td>12</td></tr><tr><td>*TowerPlus 72B</td><td>12</td><td>11</td><td>1, 2, 3</td><td>22</td><td>0, 2, 0</td><td>11</td><td>11</td></tr><tr><td>SRPOL</td><td>7</td><td>6</td><td>3, 4, 0</td><td>28</td><td>0, 1, 0</td><td>12</td><td>22</td></tr><tr><td>IR-MultiagentMT</td><td>12</td><td>23</td><td>0, 1, 1</td><td>17</td><td>0, 4, 4</td><td>13</td><td>6</td></tr><tr><td>TranssionTranslate</td><td>12</td><td>10</td><td>1, 2, 4</td><td>33</td><td>0, 0, 2</td><td>14</td><td>23</td></tr><tr><td>*CommandA</td><td>12</td><td>18</td><td>0, 2, 4</td><td>26</td><td>0, 1, 1</td><td>15</td><td>8</td></tr><tr><td>NNTSU</td><td>1</td><td>32</td><td>0, 0, 1</td><td>12</td><td>1, 0, 0</td><td>15</td><td>20</td></tr><tr><td>Erlendur</td><td>1</td><td>32</td><td>0, 0, 1</td><td>12</td><td>1, 0, 0</td><td>15</td><td>20</td></tr><tr><td>In2x</td><td>1</td><td>15</td><td>1, 0, 0</td><td>30</td><td>0, 1, 0</td><td>18</td><td>15</td></tr><tr><td>*Claude4</td><td>12</td><td>21</td><td>0, 1, 2</td><td>24</td><td>0, 1, 2</td><td>18</td><td>3</td></tr><tr><td>*DeepSeek V3</td><td>12</td><td>22</td><td>0, 1, 2</td><td>23</td><td>0, 1, 7</td><td>18</td><td>1</td></tr><tr><td>Algharb°</td><td>12</td><td>26</td><td>0, 1, 0</td><td>19</td><td>0, 2, 3</td><td>18</td><td>7</td></tr><tr><td>*Gemini 2.5 Pro</td><td>12</td><td>38</td><td>0, 0, 0</td><td>7</td><td>1, 1, 7</td><td>18</td><td>31</td></tr><tr><td>*Gemma 3 27B</td><td>12</td><td>28</td><td>0, 0, 1</td><td>18</td><td>0, 3, 4</td><td>23</td><td>10</td></tr><tr><td>*AyaExpanse 32B</td><td>12</td><td>13</td><td>1, 0, 2</td><td>35</td><td>0, 0, 1</td><td>24</td><td>22</td></tr><tr><td>*AyaExpanse 8B</td><td>12</td><td>20</td><td>0, 2, 0</td><td>29</td><td>0, 1, 0</td><td>25</td><td>9</td></tr><tr><td>KIKIS</td><td>1</td><td>39</td><td>0, 0, 0</td><td>12</td><td>1, 0, 0</td><td>26</td><td>27</td></tr><tr><td>*EuroLLM 22B</td><td>12</td><td>19</td><td>0, 2, 0</td><td>36</td><td>0, 0, 1</td><td>27</td><td>17</td></tr><tr><td>*Gemma 3 12B</td><td>12</td><td>25</td><td>0, 1, 0</td><td>32</td><td>0, 0, 2</td><td>28</td><td>7</td></tr><tr><td>Yandex</td><td>1</td><td>46</td><td>0, 0, 0</td><td>12</td><td>1, 0, 0</td><td>29</td><td>34</td></tr><tr><td>Systran°</td><td>1</td><td>15</td><td>1, 0, 0</td><td>44</td><td>0, 0, 0</td><td>30</td><td>29</td></tr><tr><td>*Llama 3.1 8B</td><td>12</td><td>14</td><td>1, 0, 0</td><td>47</td><td>0, 0, 0</td><td>31</td><td>33</td></tr><tr><td>Kaze-MT°</td><td>12</td><td>52</td><td>0, 0, 0</td><td>9</td><td>1, 0, 0</td><td>31</td><td>43</td></tr><tr><td>KYUoM°</td><td>12</td><td>52</td><td>0, 0, 0</td><td>10</td><td>1, 0, 0</td><td>33</td><td>42</td></tr><tr><td>ctpc_nlp</td><td>12</td><td>52</td><td>0, 0, 0</td><td>11</td><td>1, 0, 0</td><td>34</td><td>41</td></tr><tr><td>Wenyiil°</td><td>12</td><td>30</td><td>0, 0, 1</td><td>34</td><td>0, 0, 2</td><td>35</td><td>4</td></tr><tr><td>*Mistral-Medium</td><td>9</td><td>40</td><td>0, 0, 0</td><td>25</td><td>0, 1, 1</td><td>36</td><td>15</td></tr><tr><td>*CommandR</td><td>12</td><td>24</td><td>0, 1, 1</td><td>43</td><td>0, 0, 0</td><td>37</td><td>19</td></tr><tr><td>*Qwen3 235B</td><td>12</td><td>41</td><td>0, 0, 0</td><td>27</td><td>0, 1, 1</td><td>38</td><td>14</td></tr><tr><td>*ONLINE-W</td><td>8</td><td>27</td><td>0, 1, 0</td><td>42</td><td>0, 0, 0</td><td>39</td><td>15</td></tr><tr><td>AMI°</td><td>1</td><td>32</td><td>0, 0, 1</td><td>37</td><td>0, 0, 1</td><td>39</td><td>5</td></tr><tr><td>*EuroLLM 9B</td><td>12</td><td>29</td><td>0, 0, 1</td><td>41</td><td>0, 0, 0</td><td>41</td><td>12</td></tr><tr><td>IRB-MT</td><td>12</td><td>42</td><td>0, 0, 0</td><td>31</td><td>0, 0, 3</td><td>42</td><td>11</td></tr><tr><td>*Llama-4-Maverick</td><td>12</td><td>37</td><td>0, 0, 0</td><td>38</td><td>0, 0, 0</td><td>43</td><td>1</td></tr><tr><td>CUNI-MH-v2</td><td>1</td><td>32</td><td>0, 0, 1</td><td>46</td><td>0, 0, 0</td><td>44</td><td>14</td></tr><tr><td>bb88</td><td>1</td><td>32</td><td>0, 0, 1</td><td>49</td><td>0, 0, 0</td><td>45</td><td>17</td></tr><tr><td>*NLLB</td><td>12</td><td>44</td><td>0, 0, 0</td><td>39</td><td>0, 0, 0</td><td>46</td><td>5</td></tr><tr><td>*Mistral 7B</td><td>12</td><td>31</td><td>0, 0, 1</td><td>53</td><td>0, 0, 0</td><td>47</td><td>22</td></tr><tr><td>DLUT_GTCOM</td><td>2</td><td>45</td><td>0, 0, 0</td><td>40</td><td>0, 0, 0</td><td>48</td><td>5</td></tr><tr><td>CUNI-SFT</td><td>3</td><td>48</td><td>0, 0, 0</td><td>45</td><td>0, 0, 0</td><td>49</td><td>3</td></tr><tr><td>TranssionMT</td><td>8</td><td>43</td><td>0, 0, 0</td><td>51</td><td>0, 0, 0</td><td>50</td><td>8</td></tr><tr><td>*Qwen 2.5</td><td>12</td><td>47</td><td>0, 0, 0</td><td>48</td><td>0, 0, 0</td><td>51</td><td>1</td></tr><tr><td>CGFOKUS</td><td>1</td><td>51</td><td>0, 0, 0</td><td>49</td><td>0, 0, 0</td><td>52</td><td>2</td></tr><tr><td>*ONLINE-G</td><td>10</td><td>49</td><td>0, 0, 0</td><td>52</td><td>0, 0, 0</td><td>53</td><td>3</td></tr><tr><td>SH</td><td>1</td><td>50</td><td>0, 0, 0</td><td>54</td><td>0, 0, 0</td><td>54</td><td>4</td></tr><tr><td>CUNI-DocTransformer</td><td>1</td><td>55</td><td>0, 0, 0</td><td>55</td><td>0, 0, 0</td><td>55</td><td>0</td></tr><tr><td>COILD-BHO</td><td>1</td><td>55</td><td>0, 0, 0</td><td>55</td><td>0, 0, 0</td><td>55</td><td>0</td></tr></tbody></table>
18th and 23rd, respectively), appear in the middle of the table, while Llama-4-Maverick, Mistral 7B, Qwen 2.5 and ONLINE-G fall into the lower ranks. NLLB, a widely used dedicated MT model, ranks only 46th, with consistently low ranks in both metric-specific rankings (44th and 39th). Conversely, several single-pair submissions are quite successful, achieving first place according to one of the metrics, and sometimes second or third according to the other one (NNTSU, Erelendur and In2x reach a joint overall 15th place). Overall, the results indicate that dedicated MT systems continue to outperform many general-purpose LLMs when evaluated using CometKiwi and MetricX.

These results are somewhat surprising, particularly the relatively low performance of several large LLMs—Qwen3 235B ranks only 38th overall, and GPT-4.1 is ranked 17th according to the CometKiwi-based ranking—and popular reference models such as NLLB, despite generally being evaluated quite highly in MT tasks, whether by automatic metrics or human evaluation. Three main factors may account for this outcome. First, strong performance on edited data does not necessarily translate into equally strong performance on non-standard data; success on the former is not a guarantee of robustness. Secondly, automatic evaluation metrics—especially CometKiwi, on which our ranking is based—may perform poorly when applied to translations of non-standard text. Thirdly, several systems used QE metrics for optimisation and may therefore have gained higher rankings than their actual quality would otherwise justify.<fnref n="4" /> We leave these questions open for future investigation, and invite the reader to take our results and conclusions with a pinch of salt.

Another surprising observation is that several systems are positioned very differently in our two metric-specific rankings. The most extreme case is KazeMT, ranked 52nd using CometKiwi but 9th using MetricX. Another example of a large Δrank is Gemini 2.5 Pro, ranked 38th using CometKiwi but 7th using MetricX. Several single-pair submissions also display large discrepancies, such as Yandex and Systran, which both achieve first place according to one metric (MetricX for Yandex and CometKiwi for Systran), but do not perform as well according to the other metric. Such discrepancies could be explained, at least in some cases, in the

way these models were trained or fine-tuned, for instance by optimising for QE, as mentioned above.

## 4.1 Original versus Normalised Texts

We first compare the impact of translating the original inputs (containing non-standard language) against the normalised inputs (both with manual segmentation). The full results are given in Appendix B (Tables 3 and 4). The scores for original texts are in general lower for CometKiwi and higher for MetricX than for the normalised ones. This is somewhat unsurprising for several reasons: (i) it is expected that more standard texts are easier to translate, as the majority of the texts that the models were trained on was standard, and the UGC texts are characterised by high levels of variation, (ii) metric scores are likely to penalise translations that are less standard. Concerning (i), there is some indication that is going on. For example, the difference between translations from normalised and original texts is very large for the lowest-resource language directions, at least for CometKiwi (English to Icelandic and to Maasai), showing that the models are struggling more with the non-standard texts. Concerning (ii), some further investigation is necessary here to ascertain whether the difference in scores are a property of the metrics themselves or whether they translate into real differences in translation quality. Our observations in Section 4.3 indicate that there is more going on than these basic scores and that we should not trust the metric scores alone.

## 4.2 Manual Segmentation versus Newline Segmentation

We then compare the impact of the text segmentation by looking at the scores based on inputs with manual segmentation and those separated on newlines (both with raw inputs). Results are shown in the same appendix section (Tables 5 and 6). The differences between the two segmentation types appears less than the differences previously observed between original and normalised inputs. In reality, given that the posts were given as complete documents, the segmentation has less of an impact than if the systems had been translating on the sentence level, as was the case for most systems in previous years.

## 4.3 A First Qualitative discussion

Table 2 shows the results of all systems (apart from those whose output was obviously the result of a

<footnote n="4">⁴We indicate systems that self-declare as using QE in some way with a °.</footnote>
<table><caption>Table 2: Example of character repetition linked to a mark of expressivity for en–cs (same source text as in (Bawden and Sagot, 2023) to illustrate 2023 en–de results). For each system we provide the CometKiwi score (multiplied by 100; higher is better) and the MetricX score (lower is better) for the corresponding document, as scores were computed at the document level. Systems are ordered by increasing CometKiwi score. The two last columns provide a manual assessment of how much of the input sentence was translated into Czech—or at least not kept in English—("tgt lang") and of how well the elongation phenomenon was transferred to the output sentence ("elong tr"; non-translated tokens are ignored). Systems whose outputs obviously result from an error are not included.</caption><thead><tr><th>System</th><th>Text</th><th>CometKiwi</th><th>MetricX</th><th>tgt lang</th><th>elong tr</th></tr></thead><tbody><tr><td colspan="6"><i>Source</i> "politics used to be mooooorrreeee poolliitte we neeeeed a return to The Discouuurrsrse"</td></tr><tr><td>Qwen 2.5</td><td>politika byla dávno většinou politická, potřebujeme návrat k The Discouuurrsrse.</td><td>45.4</td><td>17.4</td><td>+</td><td>---</td></tr><tr><td>Llama 3.1 8B</td><td>politika byla kdysi mnohem menší, potřebujeme návrat k diskuzi.</td><td>46.2</td><td>15.9</td><td>+++</td><td>---</td></tr><tr><td>ctpc_nlp</td><td>„politics used to be mooooorrreeee poolliitte we neeeeed a return to The Discouuurrsrse.”</td><td>46.3</td><td>17.4</td><td>---</td><td>---</td></tr><tr><td>Mistral 7B</td><td>politika byla dávno více politická, potřebujeme se vrátí do The Discouuurrsrse.</td><td>47.7</td><td>16.9</td><td>+</td><td>---</td></tr><tr><td>GPT-4.1</td><td>„politika býýývala mnohem slušnějšíří, potřebujeme návrat k diskuuuzii.”</td><td>49.6</td><td>14.6</td><td>+++</td><td>++</td></tr><tr><td>CUNI-SFT</td><td>politika bývala mooooorrreeee poolliitte, potřebujeme návrat k diskusi.</td><td>49.6</td><td>16.3</td><td>---</td><td>---</td></tr><tr><td>Gemma 3 27B</td><td>„politika bývala moooooc víč slušná, potřebujeme návrat k Diissskkuizi.”</td><td>51.3</td><td>10.9</td><td>+++</td><td>++</td></tr><tr><td>EuroLLM 9B</td><td>„politika bývala mooooorrreeee poolliitte, potřebujeme návrat k Diskouuurrsrse.”</td><td>51.4</td><td>18.0</td><td>---</td><td>+</td></tr><tr><td>CUNI-DocTransformer</td><td>„politika bývala mooooorrreeee poolliitte my jsme potřebovali návrat do The Discourrrsrse.”</td><td>51.8</td><td>17.9</td><td>---</td><td>---</td></tr><tr><td>Gemini 2.5 Pro</td><td>politics used to be mooooorrreeee poolliitte we neeeeed a return to The Discouuurrsrse.</td><td>52.8</td><td>11.5</td><td>++</td><td>+</td></tr><tr><td>ONLINE-W</td><td>„politika bejzávala slušně-č-ě-jšíří, po-tře-bu-je-mee se vrátí k Diskurzuuuu.”</td><td>52.8</td><td>17.7</td><td>---</td><td>---</td></tr><tr><td>Wenyiil</td><td>politika bývala mooooorrreeee poolliitte, potřebujeme návrat k Diskurzu.</td><td>53.6</td><td>10.7</td><td>++(!)</td><td>+++</td></tr><tr><td>CUNI-MH-v2</td><td>politika bývala mooooorrreeee poolliitte, potřebujeme návrat k Diskurzu.</td><td>53.6</td><td>18.4</td><td>---</td><td>---</td></tr><tr><td>Algharb</td><td>„politika bejzáááála kdysi slušněččějššší, musíme se vrááátit k Diiskuuuurrzu”</td><td>53.7</td><td>10.0</td><td>+++(!)</td><td>+++</td></tr><tr><td>AyaExpanse-32B</td><td>politika bývala víceee čistááá, potřebujeme se vrátí k Diiskusi.</td><td>54.5</td><td>14.0</td><td>+++</td><td>++</td></tr><tr><td>SRPOL</td><td>„politika bývala mooooorrreeee poolliitte, potřebujeme návrat k diskuusii.”</td><td>54.6</td><td>16.2</td><td>---</td><td>+</td></tr><tr><td>GemTrans</td><td>„politika bývala kdysi mnohem civilnější a potřebujeme návrat k seriózní debatě.”</td><td>54.7</td><td>4.6</td><td>+++</td><td>---</td></tr><tr><td>Tower Plus 72B</td><td>„politika bývala více politická, potřebujeme návrat k Diskurzu.”</td><td>55.1</td><td>10.8</td><td>+++</td><td>---</td></tr><tr><td>IRB-MT</td><td>politika bývala dřív mnohem uhlazenější, potřebujeme návrat k slušné konverzaci.</td><td>55.1</td><td>7.8</td><td>+++</td><td>---</td></tr><tr><td>Qwen3 235B</td><td>„dřív byla politika víč vstřícná, potřebujeme se vrátí ke civilizované debatě.”</td><td>55.3</td><td>9.6</td><td>+++</td><td>---</td></tr><tr><td>Gemma 3 12B</td><td>politika bývala dřív mnohem, mnohem političejší, potřebujeme návrat k Diskuuzi.</td><td>55.8</td><td>10.8</td><td>+++</td><td>---</td></tr><tr><td>CommandA</td><td>„politika bývala moooooc poolliitická, potřebujeme návrat k Diissccouuurrsrse.”</td><td>56.2</td><td>12.2</td><td>---</td><td>+</td></tr><tr><td>Yolu</td><td>politika dřív bývala mnohem... mnohem... polemickyjší a my potřebujeme návrat k společné debatě.</td><td>56.3</td><td>9.0</td><td>+++</td><td>+(!)</td></tr><tr><td>Shy-hunyuan-MT</td><td>„politika dřív byla mnohem slušnější, potřebujeme návrat k racionálnímu diskurzu”</td><td>57.3</td><td>7.6</td><td>+++</td><td>---</td></tr><tr><td>Claude-4</td><td>politika bývala víříc sluuušnáá, potřebujeme se vrátí k Diskuuurzu.</td><td>57.5</td><td>12.1</td><td>+++</td><td>+</td></tr><tr><td>DeepSeek-V3</td><td>„politika bývala slušnějšíří, musíme se vrátí k diskurzuuuu”</td><td>57.5</td><td>9.8</td><td>+++</td><td>+</td></tr><tr><td>IR-MultiagentMT</td><td>politika bývala mnohem více otevřená, potřebujeme se vrátí k diskurzu.</td><td>57.7</td><td>10.1</td><td>+++</td><td>---</td></tr><tr><td>Mistral-Medium</td><td>politika bývala kdyššiiii moooooc civilizovanější, my potřebujeme návrat k Diskuuuuursu.</td><td>58.0</td><td>9.0</td><td>+++</td><td>++</td></tr><tr><td>Laníqo</td><td>politika se stala blbá a potřebujeme návrat ke klasickému diskurzu.”</td><td>58.3</td><td>9.0</td><td>+++</td><td>---</td></tr><tr><td>UvA-MT</td><td>politika bývala mnohem političejší, potřebujeme návrat k Diskuuzi.</td><td>58.5</td><td>8.9</td><td>+++</td><td>+</td></tr><tr><td>TowerPlus 9B</td><td>„politika bývala moooooc lepší, potřebujeme návrat k debatám.”</td><td>58.7</td><td>9.7</td><td>+++</td><td>+</td></tr><tr><td>Llama-4-Maverick</td><td>politika bývala mnohem uhlazenější, potřebujeme se vrátí k diskusi.</td><td>58.7</td><td>10.1</td><td>+++</td><td>---</td></tr><tr><td>AyaExpanse 8B</td><td>politika byla kdysi zábavnější, potřebujeme se vrátí k diskusi.</td><td>60.1</td><td>12.1</td><td>+++</td><td>---</td></tr><tr><td>CommandA-WMT</td><td>„politika bývala víč politická, potřebujeme se vrátí k diskurzu”</td><td>61.9</td><td>6.8</td><td>+++</td><td>---</td></tr><tr><td>SalamandraTA</td><td>„politika bývala mnohem menší, takže jsme potřebovali návrat k The Discouuurrsrse”</td><td>62.5</td><td>11.8</td><td>++</td><td>---</td></tr><tr><td>TransionTranslate</td><td>„politika bývala moooooočná, že potřebujeme návrat k The Discouuurrsrse.”</td><td>64.8</td><td>21.6</td><td>++</td><td>+</td></tr><tr><td>ONLINE-B</td><td>„politika bývala moooooodně dobrá, potřebujeme návrat k The Discouuurrsrse.”</td><td>66.7</td><td>11.0</td><td>++</td><td>+</td></tr><tr><td>NLLB</td><td>Politika bývala mooooorrreeee poolliitte, potřebujeme návrat do The Discouuurrsrse</td><td>68.0</td><td>15.3</td><td>---</td><td>---</td></tr></tbody></table>

bug) on the example already used in (Bawden and Sagot, 2023) to illustrate the behaviour of MT systems in the presence of several instances of the elongation phenomenon, by which one or more characters are repeated to express emphasis. A first glance at the results shows that there is not necessarily a convincing correlation between perceived translation quality and the automatic evaluation provided by the CometKiwi metric, whereas MetricX results look slightly more correlated. Looking more closely at the translations, two main observations can be made:

* Firstly, a number of systems tend to keep unchanged original English tokens that have undergone elongation, and sometimes even the whole input. The fact that the two last tokens are capitalised in the input sentence makes it even more difficult for most systems to actually try to translate them.

* Secondly, not all systems attempt to transfer the elongation phenomenon into their output. Some seem to (try to) produce standard Czech rather than preserving the non standard expressivity mark. Some even try to render the same expressivity using another non standard phenomenon.

To better understand what is at play here, we decided to manually annotate these translations for two features: how much of the input sentence was (tentatively) translated into Czech, and how much of the elongation phenomenon was transferred into the output sentence (ignoring tokens kept in their original English form). Comparing these annotations with system types is interesting. Although a single example is in no way sufficient to allow for any generalisations, it seems that generic LLMs are more liable to preserving elongation and, more generally, to produce better translations, whereas
dedicated MT models seem to produce more stan-
dard outputs and/or not to translate significant parts
of the input. Interestingly, this is not reflected in
the CometKiwi scores, but it is more visible in the
MetricX scores. For instance, the best CometKiwi-
scored translation contains two segments that are
still in English, a situation that invariably leads to
bad (high) MetricX scores. However, CometKiwi
ad MetricX seem consistently bad at penalising the
absence of elongations in the produced translation.
The best CometKiwi-scored translation does not
contain any elongation in genuinely Czech tokens,
and the best Metric-X-scored translation, which
is perfect Czech, does not include any elongation
whatsoever. On the contrary, the output of GPT-4.1
is good in both regards—it is entirely in Czech and
does contain elongations—, and is a good trans-
lation, but it is scored poorly by both CometKiwi
and MetricX. This shows that modern metrics such
as CometKiwi and MetricX might not be reliable
when it comes to assessing translation quality of
non-standard content. We leave a more quantitative
and systematic exploration of these questions and
their implications for MT evaluation in general to
future work.

Although the test suite this year presents new an-
notations for the non-standard phenomena present
in the test suite that are more consistent and inter-
esting for analysis, we also leave the analysis on a
per-phenomenon basis to future work, in which we
will go into more detail and length.

5 Conclusion

We have presented a new version (v2) of the RoCS-
MT challenge set, first presented at the WMT 2023
test suites shared task track. This 2025 edition has
several improvements, with minor corrections to
source texts, some corrections to references and
improved categorisation of non-standard phenom-
ena. We describe these changes and also use the
challenge set to compare systems submitted to this
year's shared task, comparing translation from the
original UGC inputs and their manually normalised
versions. A major difference with previous years
of the shared task is a switch to document-level
MT, so whole chunks of posts were submitted
to systems for translation. We nevertheless com-
pare two different segmentation types (to see if ini-
tially manually segmenting into sentences and then
concatenating the sentences with newlines could
help translation) and discuss preliminary insights

into the shortcomings of popular metrics such as
CometKiwi and MetricX when applied on non-
standard text MT.

Acknowledgments

This work was partly funded by both authors’
chairs in the PRAIRIE institute, now PRAIRIE-
PSAI, funded by the French national agency
ANR, respectively as part of the “Investissements
d’avenir” programme under the reference ANR-
19-P3IA-0001 and as part of the “France 2030”
strategy under the reference ANR-23-IACL-0008.

References

Tyler Baldwin and Yunyao Li. 2015. An in-depth analysis of the effect of text normalization in social media. In *Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 420–429, Denver, Colorado. Association for Computational Linguistics.

Rachel Bawden and Benoît Sagot. 2023. RoCS-MT: Robustness challenge set for machine translation. In *Proceedings of the Eighth Conference on Machine Translation*, pages 198–216, Singapore. Association for Computational Linguistics.

Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In *International Conference on Learning Representations*.

Menglong Cui, Pengzhi Gao, Wei Liu, Jian Luan, and Bin Wang. 2025. Multilingual machine translation with open large language models at practical scale: An empirical study. In *Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pages 5420–5443, Albuquerque, New Mexico. Association for Computational Linguistics.

Jacob Eisenstein. 2013. What to do about bad language on the internet. In *Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 359–369, Atlanta, Georgia. Association for Computational Linguistics.

Jennifer Foster. 2010. “cba to check the spelling”: Investigating Parser Performance on Discussion Forum Posts. In *Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics*, pages 381–384, Los Angeles, California. Association for Computational Linguistics.

Ryo Fujii, Masato Mita, Kaori Abe, Kazuaki Hanawa, Makoto Morishita, Jun Suzuki, and Kentaro Inui.
2020. PheMT: A phenomenon-wise dataset for machine translation robustness on user-generated contents. In *Proceedings of the 28th International Conference on Computational Linguistics*, pages 5929–5943, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In *Proceedings of the Ninth Conference on Machine Translation*, pages 492–504, Miami, Florida, USA. Association for Computational Linguistics.

Tom Kocmi, Ekaterina Artemova, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Konstantin Dranch, Anton Dvorkovich, Sergey Dukanov, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Howard Lakougna, Jessica M. Lundin, Christof Monz, Kenton Murray, and 10 others. 2025. Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets. In *Proceedings of the Tenth Conference on Machine Translation*, China. Association for Computational Linguistics.

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Masaaki Nagata, Toshiaki Nakazawa, Martin Popel, and 3 others. 2023. Findings of the 2023 conference on machine translation (WMT23): LLMs are here but not quite there yet. In *Proceedings of the Eighth Conference on Machine Translation*, pages 1–42, Singapore. Association for Computational Linguistics.

Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. 2022. Findings of the 2022 conference on machine translation (WMT22). In *Proceedings of the Seventh Conference on Machine Translation (WMT)*, pages 1–45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has machine translation achieved human parity? a case for document-level evaluation. In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 4791–4796, Brussels, Belgium. Association for Computational Linguistics.

Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan

Chhablani, Bhavitva Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, and 13 others. 2021. Datasets: A community library for natural language processing. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pages 175–184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Wang Ling, Guang Xiang, Chris Dyer, Alan Black, and Isabel Trancoso. 2013. Microblogs as parallel corpora. In *Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 176–186, Sofia, Bulgaria. Association for Computational Linguistics.

Paul McNamee and Kevin Duh. 2022. The multilingual microblog translation corpus: Improving and evaluating translation of user-generated text. In *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 910–918, Marseille, France. European Language Resources Association.

Paul Michel and Graham Neubig. 2018. MTNT: A testbed for machine translation of noisy text. In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 543–553, Brussels, Belgium. Association for Computational Linguistics.

Hamdy Mubarak, Sabit Hassan, and Ahmed Abdelali. 2020. Constructing a bilingual corpus of parallel tweets. In *Proceedings of the 13th Workshop on Building and Using Comparable Corpora*, pages 14–21, Marseille, France. European Language Resources Association.

Lydia Nishimwe, Benoît Sagot, and Rachel Bawden. 2024. Making sentence embeddings robust to user-generated content. In *Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)*, pages 10984–10998, Torino, Italia. ELRA and ICCL.

Jungsoo Park, Mujeen Sung, Jinhyuk Lee, and Jaewoo Kang. 2020. Adversarial subword regularization for robust neural machine translation. In *Findings of the Association for Computational Linguistics: EMNLP 2020*, pages 1945–1953, Online. Association for Computational Linguistics.

Ben Peters and Andre Martins. 2025. Did translation models get more robust without anyone Even noticing? In *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 2445–2458, Vienna, Austria. Association for Computational Linguistics.

Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbelabel-IST 2022 submission for the metrics shared task. In *Proceedings of the Seventh Conference on Machine Translation (WMT)*,
pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

José Carlos Rosales Núñez, Djamé Seddah, and Guil-laume Wisniewski. 2019. Comparison between NMT and PBSMT Performance for Translating Noisy User-Generated Content. In *Proceedings of the 22nd Nordic Conference on Computational Linguistics*, pages 2–14, Turku, Finland. Linköping University Electronic Press.

Djamé Seddah, Benoit Sagot, Marie Candito, Virginie Mouilleron, and Vanessa Combet. 2012. *The French Social Media Bank: a Treebank of Noisy User Generated Content*. In *Proceedings of COLING 2012*, pages 2441–2458, Mumbai, India. The COLING 2012 Organizing Committee.

Henny Sluyter-Gäthje, Pintu Lohar, Haithem Afli, and Andy Way. 2018. *FooTweets: A bilingual parallel corpus of world cup tweets*. In *Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)*, Miyazaki, Japan. European Language Resources Association (ELRA).

Vaibhav Vaibhav, Sumeet Singh, Craig Stewart, and Graham Neubig. 2019. Improving robustness of machine translation with synthetic noise. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 1916–1920, Minneapolis, Minnesota. Association for Computational Linguistics.

Rob van der Goot, Rik van Noord, and Gertjan van Noord. 2018. A taxonomy for in-depth evaluation of normalization for user generated content. In *Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)*, pages 684–688, Miyazaki, Japan. European Language Resources Association (ELRA).

Iñaki San Vicente, Iñaki Alegría, Cristina Españ-Bonet, Pablo Gamallo, Hugo Gonçalo Oliveira, Eva Martínez Garcia, Antonio Toral, Arkaitz Zubigaaga, and Nora Aranberri. 2016. *TweetMT: A parallel microblog corpus*. In *Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)*, pages 2936–2941, Portorož, Slovenia. European Language Resources Association (ELRA).

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024. Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation. In *Proceedings of the Forty-First International Conference on Machine Learning*.

## A Normalisation Span Classification

The original and normalised texts were aligned and different non-standard phenomena classified.

Below is the list of normalisation categories with examples.

### Punctuation, typographic conventions, symbols, etc.

* **punct:diff**: extra punctuation is included or necessary punctuation is removed (e.g. missing final punctuation, missing apostrophes, commas, etc.).
  e.g. `im→I'm`

* **punct:norm**: punctuation to be normalised according to certain conventions (e.g. same apostrophes and quotes).
  e.g. that's→that's

* **caps**: capitalisation differs from what is considered standard (e.g. lowercase initial characters, all uppercase, etc.).
  e.g. `IM SO HAPPY→I'm so happy`

* **slash_to_or**: a slash is used, where in normalised speech an “or” would be used to represent a list of items. This applies to the whole list, including where etc. is included. Not that this does not include cases where the items are alternatives in the discourse
  e.g. `cat/exhaust/etc→cat or exhaust, etc.`
  e.g. `truth/dare→truth or dare`
  e.g. [counter-example] `AW WELL MY DOG/CHILD IS VERY FRIENDLY SO LET ME APPROACH→aw, well my dog/child is very friendly, so let me approach`

* **slash_to_and**: a slash is used, where in normalised speech an “and” would be used. This applies to the whole list, including where etc. is included.
  e.g. `work/paint→work and paint`

* **slash_distribution**: the use of a slash to separate two items where the slash does not separate two complete items (i.e. part of one element is distributed to both items thanks to the slash). An example makes this easier to understand:
  e.g. just disrespects any / everyone→just disrespects any / everyone

* **word_to_symbol**: the use of a symbol to represent a word
  e.g. `+→and, &→and`
  e.g. →around
  e.g. `$$$→money`
* **symbol placement**: non-standard placement of a symbol with respects to English norms. e.g. 100$→$100

## Spacing

* **spacing**: missing or added spacing in the original text
e.g. aswell→as well
e.g. over thinking→overthinking

* **spacing:camelcase**: the use camelcase (capital letters at the beginning of words) instead of using spaces
e.g. surroundedUs→surrounded us
e.g. sawThat→saw that

## Phonetically similar spellings (including imitation of speech)

* **phon**: the word uses a variant of spelling based on the phonetic similarity of the sequence of characters. This also includes the use of individual letters to represent words or syllables because of an equivalence in their pronunciation (u→you, b→be, c→see).
e.g. saturday sesh→Saturday session (also a case of truncation)
e.g. sup→What's up (also truncation)
e.g. bcos -> because
e.g. n→and
e.g. tho→though
e.g. speakin→speaking

* **phon:char**: a character is used in the place of a word or syllable because of its phonetic similarity with the word or syllable
e.g. b→be
e.g. c→see
e.g. u→you

* **phon:digits**: a digit is used in the place of a word or part of a word.
e.g. m8→mate
e.g. 2→to
e.g. as1 that will play→as one that will play (in a context where 1 could be incorrect, otherwise this should not be normalised)

* **phon:cute**: the spelling of a word to indicate "cute" or babyish pronunciation, e.g. using 'w' to replace an initial letter
e.g. weRecommended→recommended

* **phon:hesitate**: words that are written in a way to imitate hesitation
e.g. terribl-....yyy→terribly
e.g. Y-y-yyy-yes→Yes

* **phon:sound**: the case of words that are used to indicate a sound (very rare)
e.g. bRRrrRRrrRRrr→brr rrr rrr

* **phon:interjection**: interjections that are normalised to single (and more standard) variations
e.g. bla→blah
e.g. URGHHH→ugh
e.g. Nawh→no

## Other spelling variations (ergographic, expressiveness)

* **elongation**: characters are repeated, usually as a mark of expressiveness.
e.g. *meeeeeelltiiinggg*→melting
e.g. sooo→so

* **devowelling**: a word with the vowels removed (initial vowels are often kept however). This can often result in double characters being reduced to single ones (messages→s) In this category are also words where part of the word has been devowelled.
e.g. wt→what, ovr→over, ppl→people
e.g. askd→asked (initial vowel kept)
e.g. travllr→traveller

* **contraction**: when several words are contracted into a single one. This has some overlap with the characteristics of phonetic distance, in that it is due to the pronunciation of the words that the contraction occurs.
e.g. gonna→going to
e.g. innit→isn't it?

* **truncation**: a word is shortened, either at the end (traditional truncation) or sometimes at the beginning, often by removing a syllable or a suffix. Note the difference with acronymisation, which involves keeping initial characters.
e.g. sesh→session
e.g. cuz→because, till→until
e.g. ofc→of course -> CHANGED, NOW ACRONYM
e.g. w→with -> CHANGED, NOW ACRONYM
* **acronym:** a word or sequence of word is represented as an acronym, i.e. the initial characters of the word (or syllables) are retained and the others are elided.

e.g. RN→right now

e.g. gf→girlfriend

e.g. never mind→nvm

e.g. w→with

We also include in this category words that are partially acronymised (i.e. where one syllable is represented by its initial but the rest is not).is acronymised but the rest is not.

e.g. oline→offensive line

e.g. gmeet -> Google meet

e.g. bday→Birthday

e.g. ofc→of course

Note that sometimes slashes are included in the acronym

e.g. b/c→because,

e.g. w/o→without.

* **abbreviation:** abbreviations for units of measurement and other standard cases

e.g. ft→feet, 2k→2000, hrs→hours

e.g. Ex→for example

## Spelling mistakes (distinguished from spelling variation identified as being intentional)

* **spell:** the word contains a spelling error that is not clearly intentional (covered by the other phenomena such as truncation, devowelling, etc.) and not covered by the other more specific categories.

* **spell:charswap:** the characters in the word are present but not in the right order (most often consecutive characters being swapped)

e.g. nobel→noble

e.g. furhter→futher

## Misc

* **digit_letter_sim:** Very rare, but where a digit is used in the place of a letter due to the typographic similarity (see in 3ver→ever).

* **letter_to digit:** Very rare, but where a digit is used in place of a letter not because of their typographic similarity, but because as a sort of tautology (seen in 1nce→Once).

* **suffix:** the addition of a suffix to a word, either as a diminutive or other

e.g. lolsky→lol

e.g. meanie→mean

e.g. doggy→dog

## Added and dropped words

* **word_drop:** a word is not present in the original text and is present in the normalised version

e.g. It also confusing... → It's also confusing

e.g. u wanna see?→Do you want to see?

* **word_drop:pronoun:** the original text omits a pronoun (often the case of subject pronouns at the beginning of sentences) that is included in the normalised version.

e.g. Was gunna try distortion... → I was going

to try distortion...

* **word_drop:det:** the original text omits an article (e.g. the or a) that is included in the normalised version.

e.g. Pretty creative way... → A pretty creative way... **word_add:** a word is present in the original text and is removed in the normalised version

e.g. ... in ten days ago→... ten days ago

e.g. also for uses of word “like” as a filler

* **word_add:det:** the original text includes an article where the normalised version removes it

e.g. ... adds an 12kg of salt→... adds 12kg of salt symbol_drop: the original text omits a symbol that is included in the normalised version.

e.g. 32c→32°C

* **symbol_add:** the original text includes a symbol that is removed in the normalised version.

... no issue w being over 12+ ft... → ... no issue with being over 12 feet...

## Grammar

* **inflection:** a word is not correctly inflected (e.g. with respect to number, tense, etc.)

e.g. ... wondering what ppl thought are→... wondering what people's thoughts are

* **grammar:** inflection-related errors

e.g. ... wondering what ppl thought are→... wondering what people's thoughts are

e.g. if your good→if you're good
* grammar:v:

* grammar:v:inflect

## Lexical changes

* lexanstelle: a use of a non-standard lexical choice, including dialectisms (e.g. cannae, ain't), malapropisms (e.g. genuinely), foreign words and generally wrong choices of words (e.g. wrong part of speech, wrong semantic choice of words, lacking punctuation, use of an antonym by accident, etc.)
  e.g. I am confusion→I am confused
  e.g. genuinely→genuinely
  e.g. pish→piss
  e.g. ain't→aren't
  e.g. cannae→cannot
  e.g. y'all→everyone/all/all your (depending on the context)
  e.g. sans guac→without guacamole

* surroundingemphasis: emphasis added to certain words typographically (removed in the normalised variants).
  e.g. *without*→without
  e.g. ~find~→find

* emoticon: emoticon that is a variant on the common emoticons :-), :-D, :-(, :-/ and >:-)
  e.g. :-//////→:-/
  e.g. (:→:-)
  e.g. :J→:-)

* censored: the word contains symbols in an effort to censor the word
  e.g. upv*te→upvote
  e.g. s**t→shit, sh**→shit

## B Raw Automatic Scores per Language pair

The raw scores (calculated at the document level) can be found in this appendix section. In each of the tables, the systems are ordered by the ranking across all languages for that particular metric (as described in Section 4). Note that higher CometKiwi scores are better and lower MetricX scores are better.

Tables 3 and 6 provide the CometKiwi and MetricX scores respectively for manually segmented

texts, with a comparison of original and normalised input texts.

Tables 5 and 6 provide the CometKiwi and MetricX scores respectively for original inputs, with a comparison of manually segmented and newline-segmented texts.
<table><caption>Table 3: CometKiwi (higher is better) scores across language directions for original and normalised inputs (manual segmentation in both instances). System are sorted by their CometKiwi-based ranking (over all language pairs). Systems run by the task organisers are marked as an asterisk.</caption><thead><tr><th rowspan="2">System</th><th rowspan="2">Rank</th><th colspan="2">en-at_EG</th><th colspan="2">en-bho_IN</th><th colspan="2">en-es_CZ</th><th colspan="2">en-et_EE</th><th colspan="2">en-is_IS</th><th colspan="2">en-ja_IP</th><th colspan="2">en-ko_KR</th><th colspan="2">en-mas_KE</th><th colspan="2">en-ru_RU</th><th colspan="2">en-st_Lanti_RS</th><th colspan="2">en-uk_UA</th><th colspan="2">en-zh_CN</th></tr><tr><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th></tr></thead><tbody><tr><td>Yolu</td><td>1</td><td>78.5</td><td>73.4</td><td>71.9</td><td>68.9</td><td>82.1</td><td>76.3</td><td>84.8</td><td>79.6</td><td>35.9</td><td>22.7</td><td>83.7</td><td>79.7</td><td>83.3</td><td>79.3</td><td>35.9</td><td>22.7</td><td>81.1</td><td>76.0</td><td>83.8</td><td>78.5</td><td>80.7</td><td>75.5</td><td>80.7</td><td>76.1</td></tr><tr><td>CommandA-WMT</td><td>2</td><td>71.4</td><td>65.6</td><td>75.5</td><td>72.8</td><td>80.9</td><td>75.1</td><td>83.0</td><td>77.8</td><td>75.0</td><td>70.0</td><td>82.7</td><td>78.5</td><td>82.5</td><td>78.1</td><td>65.5</td><td>80.0</td><td>74.7</td><td>80.6</td><td>75.0</td><td>80.0</td><td>75.3</td><td>79.6</td><td>74.7</td></tr><tr><td>SalamandraTA</td><td>3</td><td>72.1</td><td>66.8</td><td>60.7</td><td>57.4</td><td>81.9</td><td>76.0</td><td>84.7</td><td>78.9</td><td>77.6</td><td>72.2</td><td>82.1</td><td>78.4</td><td>81.1</td><td>77.1</td><td>59.9</td><td>81.1</td><td>75.5</td><td>83.7</td><td>78.3</td><td>80.5</td><td>75.1</td><td>80.1</td><td>75.4</td></tr><tr><td>Shy-hunyan-MT</td><td>4</td><td>75.8</td><td>70.7</td><td>80.5</td><td>77.6</td><td>80.6</td><td>74.8</td><td>83.2</td><td>78.9</td><td>77.5</td><td>71.4</td><td>82.1</td><td>77.7</td><td>81.5</td><td>77.7</td><td>55.9</td><td>79.4</td><td>73.9</td><td>82.8</td><td>77.0</td><td>79.3</td><td>73.7</td><td>79.2</td><td>73.9</td></tr><tr><td>Lanqo</td><td>5</td><td>76.8</td><td>71.2</td><td>—</td><td>—</td><td>80.8</td><td>74.7</td><td>83.9</td><td>78.0</td><td>—</td><td>—</td><td>82.4</td><td>78.3</td><td>82.2</td><td>78.1</td><td>—</td><td>—</td><td>80.3</td><td>74.7</td><td>—</td><td>—</td><td>79.9</td><td>74.6</td><td>79.7</td><td>74.7</td></tr><tr><td>SRPOL</td><td>6</td><td>76.8</td><td>—</td><td>—</td><td>—</td><td>81.0</td><td>74.4</td><td>83.1</td><td>78.0</td><td>—</td><td>—</td><td>82.5</td><td>78.5</td><td>82.1</td><td>77.6</td><td>—</td><td>—</td><td>79.7</td><td>74.1</td><td>81.9</td><td>76.7</td><td>78.5</td><td>73.5</td><td>79.2</td><td>74.2</td></tr><tr><td>UvA-MT</td><td>7</td><td>71.6</td><td>66.4</td><td>73.0</td><td>68.2</td><td>79.9</td><td>73.8</td><td>81.3</td><td>75.7</td><td>71.9</td><td>67.9</td><td>82.3</td><td>78.2</td><td>82.1</td><td>77.6</td><td>37.3</td><td>37.1</td><td>79.5</td><td>73.8</td><td>81.9</td><td>76.7</td><td>78.5</td><td>73.8</td><td>79.3</td><td>79.3</td></tr><tr><td>*ONLINE-B</td><td>8</td><td>76.4</td><td>71.4</td><td>60.0</td><td>54.5</td><td>79.5</td><td>73.1</td><td>82.1</td><td>75.8</td><td>76.7</td><td>71.4</td><td>82.3</td><td>78.3</td><td>81.2</td><td>76.6</td><td>—</td><td>—</td><td>78.7</td><td>73.3</td><td>80.3</td><td>74.5</td><td>78.3</td><td>72.1</td><td>79.1</td><td>73.8</td></tr><tr><td>*TowerPlus 9B</td><td>9</td><td>48.9</td><td>46.0</td><td>79.1</td><td>75.8</td><td>79.6</td><td>72.9</td><td>62.3</td><td>57.9</td><td>76.2</td><td>70.8</td><td>81.7</td><td>77.5</td><td>81.6</td><td>77.2</td><td>43.6</td><td>41.3</td><td>79.0</td><td>73.7</td><td>63.8</td><td>60.2</td><td>78.3</td><td>72.8</td><td>78.9</td><td>73.6</td></tr><tr><td>TransitionTranslate</td><td>10</td><td>67.5</td><td>61.7</td><td>59.9</td><td>54.9</td><td>79.5</td><td>72.9</td><td>82.2</td><td>76.0</td><td>76.5</td><td>71.1</td><td>82.8</td><td>78.5</td><td>81.9</td><td>77.1</td><td>35.9</td><td>22.7</td><td>79.4</td><td>73.4</td><td>76.6</td><td>65.3</td><td>78.1</td><td>71.7</td><td>79.1</td><td>73.8</td></tr><tr><td>*TowerPlus 72B</td><td>11</td><td>66.4</td><td>60.4</td><td>79.7</td><td>76.2</td><td>78.8</td><td>72.7</td><td>72.6</td><td>67.4</td><td>75.5</td><td>70.4</td><td>81.6</td><td>77.1</td><td>81.4</td><td>77.0</td><td>44.0</td><td>38.7</td><td>79.1</td><td>73.4</td><td>73.6</td><td>68.8</td><td>78.2</td><td>72.5</td><td>79.0</td><td>74.1</td></tr><tr><td>GenTrans</td><td>12</td><td>75.7</td><td>70.3</td><td>80.4</td><td>76.9</td><td>78.9</td><td>72.3</td><td>81.3</td><td>74.8</td><td>72.7</td><td>67.5</td><td>80.7</td><td>76.2</td><td>80.4</td><td>75.7</td><td>36.5</td><td>34.3</td><td>78.3</td><td>72.4</td><td>81.2</td><td>75.1</td><td>78.0</td><td>72.7</td><td>78.0</td><td>73.1</td></tr><tr><td>*AyaExpance 32B</td><td>13</td><td>66.0</td><td>60.5</td><td>65.2</td><td>62.8</td><td>78.2</td><td>72.7</td><td>54.1</td><td>50.9</td><td>50.9</td><td>47.8</td><td>81.6</td><td>77.0</td><td>81.3</td><td>76.4</td><td>46.5</td><td>43.2</td><td>78.2</td><td>73.1</td><td>72.9</td><td>68.3</td><td>77.6</td><td>72.4</td><td>77.9</td><td>72.6</td></tr><tr><td>*Llama 3.1 8B</td><td>14</td><td>57.2</td><td>52.0</td><td>66.3</td><td>62.1</td><td>72.9</td><td>66.7</td><td>67.2</td><td>62.6</td><td>56.4</td><td>53.4</td><td>75.7</td><td>72.2</td><td>76.8</td><td>72.0</td><td>50.9</td><td>48.8</td><td>74.8</td><td>69.2</td><td>73.6</td><td>68.1</td><td>73.3</td><td>68.2</td><td>76.4</td><td>70.9</td></tr><tr><td>Systran</td><td>15</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>84.3</td><td>81.0</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>In2x</td><td>15</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>82.8</td><td>78.7</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*GPT-4.1</td><td>17</td><td>62.7</td><td>57.9</td><td>62.4</td><td>59.6</td><td>79.8</td><td>73.2</td><td>82.5</td><td>76.5</td><td>76.2</td><td>70.5</td><td>81.6</td><td>77.2</td><td>81.5</td><td>76.8</td><td>34.6</td><td>32.2</td><td>78.6</td><td>72.6</td><td>82.0</td><td>76.2</td><td>78.2</td><td>72.5</td><td>78.9</td><td>73.1</td></tr><tr><td>*CommandA</td><td>18</td><td>64.1</td><td>59.3</td><td>63.0</td><td>60.2</td><td>79.5</td><td>73.5</td><td>77.3</td><td>71.3</td><td>67.9</td><td>63.7</td><td>82.1</td><td>77.7</td><td>81.9</td><td>77.4</td><td>42.1</td><td>39.9</td><td>78.7</td><td>73.1</td><td>79.5</td><td>74.5</td><td>78.2</td><td>72.5</td><td>78.9</td><td>73.8</td></tr><tr><td>*EuroLLM 22B</td><td>19</td><td>72.1</td><td>66.0</td><td>77.2</td><td>73.6</td><td>79.0</td><td>72.5</td><td>81.9</td><td>76.1</td><td>45.7</td><td>43.1</td><td>81.3</td><td>76.7</td><td>81.2</td><td>75.9</td><td>39.7</td><td>36.7</td><td>78.5</td><td>73.0</td><td>79.3</td><td>74.2</td><td>77.7</td><td>71.9</td><td>77.6</td><td>73.1</td></tr><tr><td>*AyaExpance 8B</td><td>20</td><td>74.6</td><td>69.7</td><td>74.1</td><td>71.1</td><td>77.0</td><td>77.7</td><td>71.4</td><td>79.3</td><td>39.6</td><td>39.6</td><td>81.0</td><td>76.7</td><td>80.9</td><td>76.2</td><td>39.8</td><td>37.6</td><td>78.7</td><td>72.3</td><td>81.1</td><td>75.4</td><td>77.0</td><td>71.7</td><td>76.5</td><td>73.0</td></tr><tr><td>*Claud4 V3</td><td>21</td><td>64.8</td><td>59.4</td><td>63.1</td><td>59.5</td><td>79.7</td><td>72.7</td><td>81.6</td><td>74.9</td><td>75.9</td><td>69.4</td><td>82.2</td><td>77.5</td><td>81.7</td><td>76.2</td><td>38.5</td><td>35.4</td><td>78.8</td><td>72.2</td><td>81.0</td><td>75.2</td><td>77.7</td><td>72.5</td><td>77.9</td><td>72.9</td></tr><tr><td>IR-MultigentMT</td><td>22</td><td>64.6</td><td>58.9</td><td>65.1</td><td>62.5</td><td>78.9</td><td>72.4</td><td>81.5</td><td>75.3</td><td>74.5</td><td>68.8</td><td>81.2</td><td>76.4</td><td>80.8</td><td>75.7</td><td>38.4</td><td>36.2</td><td>78.0</td><td>73.0</td><td>80.9</td><td>75.2</td><td>77.7</td><td>72.5</td><td>66.9</td><td>76.7</td><td>71.4</td></tr><tr><td>IR-MultigentMT</td><td>23</td><td>68.2</td><td>63.7</td><td>63.4</td><td>61.5</td><td>78.3</td><td>72.5</td><td>67.8</td><td>42.4</td><td>39.5</td><td>75.0</td><td>69.6</td><td>81.3</td><td>76.4</td><td>80.7</td><td>76.5</td><td>44.6</td><td>41.2</td><td>72.4</td><td>66.1</td><td>61.6</td><td>57.2</td><td>72.5</td><td>76.6</td><td>74.0</td><td>72.4</td><td>72.5</td></tr><tr><td>*CommandR</td><td>24</td><td>72.8</td><td>66.5</td><td>66.0</td><td>62.2</td><td>74.2</td><td>71.5</td><td>67.8</td><td>42.5</td><td>39.5</td><td>41.1</td><td>38.7</td><td>79.9</td><td>75.3</td><td>79.2</td><td>74.6</td><td>43.9</td><td>42.1</td><td>77.7</td><td>72.6</td><td>79.6</td><td>74.0</td><td>77.4</td><td>72.4</td><td>77.6</td><td>72.5</td><td>73.5</td></tr><tr><td>*Gemma 3.12B</td><td>25</td><td>62.3</td><td>57.5</td><td>57.2</td><td>57.1</td><td>78.9</td><td>72.7</td><td>69.2</td><td>64.3</td><td>64.3</td><td>69.2</td><td>80.3</td><td>75.5</td><td>79.5</td><td>74.8</td><td>43.7</td><td>42.1</td><td>77.7</td><td>72.6</td><td>79.0</td><td>72.5</td><td>76.8</td><td>78.1</td><td>72.1</td><td>78.3</td><td>73.0</td></tr><tr><td>Agharb</td><td>26</td><td>74.8</td><td>69.2</td><td>59.1</td><td>56.7</td><td>78.1</td><td>71.6</td><td>81.4</td><td>74.9</td><td>35.9</td><td>22.7</td><td>80.5</td><td>75.8</td><td>80.0</td><td>75.1</td><td>35.9</td><td>22.7</td><td>76.9</td><td>70.6</td><td>80.9</td><td>74.7</td><td>76.8</td><td>70.9</td><td>77.2</td><td>77.2</td><td>77.2</td></tr><tr><td>*ONLINE-W</td><td>27</td><td>74.6</td><td>68.7</td><td>62.4</td><td>59.8</td><td>78.5</td><td>72.5</td><td>80.5</td><td>68.0</td><td>73.2</td><td>68.1</td><td>79.6</td><td>73.8</td><td>81.4</td><td>75.4</td><td>35.3</td><td>32.6</td><td>78.1</td><td>72.4</td><td>80.7</td><td>75.0</td><td>77.7</td><td>72.5</td><td>77.8</td><td>72.5</td><td>72.5</td></tr><tr><td>*Gemma 3.27B</td><td>28</td><td>63.9</td><td>59.1</td><td>62.4</td><td>59.8</td><td>78.5</td><td>72.5</td><td>80.9</td><td>75.1</td><td>73.2</td><td>68.1</td><td>80.3</td><td>75.9</td><td>80.3</td><td>75.4</td><td>35.3</td><td>32.6</td><td>78.1</td><td>72.4</td><td>80.7</td><td>75.0</td><td>77.7</td><td>72.6</td><td>77.8</td><td>72.6</td><td>72.5</td></tr><tr><td>*EuroLLM 9B</td><td>29</td><td>70.3</td><td>64.7</td><td>69.9</td><td>65.4</td><td>78.7</td><td>82.3</td><td>81.2</td><td>75.0</td><td>43.2</td><td>41.5</td><td>80.4</td><td>75.4</td><td>80.2</td><td>75.4</td><td>32.5</td><td>32.6</td><td>77.8</td><td>72.3</td><td>77.6</td><td>70.6</td><td>77.2</td><td>71.6</td><td>77.4</td><td>72.0</td><td>70.4</td></tr><tr><td>Wenyüil</td><td>30</td><td>74.0</td><td>67.6</td><td>58.2</td><td>55.6</td><td>76.8</td><td>68.7</td><td>79.8</td><td>72.4</td><td>35.9</td><td>22.7</td><td>79.4</td><td>74.3</td><td>78.9</td><td>73.3</td><td>35.9</td><td>22.7</td><td>76.0</td><td>69.0</td><td>79.9</td><td>72.8</td><td>76.2</td><td>69.8</td><td>76.6</td><td>70.4</td><td>65.9</td></tr><tr><td>*Mistral 7B</td><td>31</td><td>48.6</td><td>45.0</td><td>55.6</td><td>52.8</td><td>64.7</td><td>59.4</td><td>41.5</td><td>39.1</td><td>42.5</td><td>40.4</td><td>71.2</td><td>67.9</td><td>71.5</td><td>67.6</td><td>41.2</td><td>40.1</td><td>71.2</td><td>66.2</td><td>68.9</td><td>64.6</td><td>70.1</td><td>65.4</td><td>71.2</td><td>65.9</td><td>65.9</td></tr><tr><td>CUNI-MH-V2</td><td>32</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>bb88</td><td>32</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>AMI</td><td>32</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Erlendur</td><td>32</td><td>62.8</td><td>57.7</td><td>60.2</td><td>57.3</td><td>78.5</td><td>71.8</td><td>81.3</td><td>74.4</td><td>75.0</td><td>69.1</td><td>80.4</td><td>75.8</td><td>80.1</td><td>75.4</td><td>37.2</td><td>34.8</td><td>78.4</td><td>72.6</td><td>80.6</td><td>74.7</td><td>78.0</td><td>72.3</td><td>78.1</td><td>72.4</td></tr><tr><td>*Llama-4-Maverick</td><td>37</td><td>58.9</td><td>54.8</td><td>59.4</td><td>56.5</td><td>77.8</td><td>70.7</td><td>80.7</td><td>74.4</td><td>74.6</td><td>68.8</td><td>80.1</td><td>75.1</td><td>79.3</td><td>74.0</td><td>37.6</td><td>35.2</td><td>76.3</td><td>69.7</td><td>80.4</td><td>74.3</td><td>76.1</td><td>70.2</td><td>76.4</td><td>70.2</td></tr><tr><td>*Gemini 2.5 Pro</td><td>38</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>KIKIS</td><td>39</td><td>65.6</td><td>60.3</td><td>63.3</td><td>60.6</td><td>79.2</td><td>72.5</td><td>79.9</td><td>73.3</td><td>72.2</td><td>66.9</td><td>81.6</td><td>76.7</td><td>81.2</td><td>76.2</td><td>38.9</td><td>36.4</td><td>78.3</td><td>72.2</td><td>78.6</td><td>72.5</td><td>76.8</td><td>71.2</td><td>78.6</td><td>73.0</td></tr><tr><td>*Mistral-Medium</td><td>40</td><td>64.7</td><td>59.0</td><td>62.8</td><td>60.4</td><td>77.7</td><td>71.3</td><td>76.2</td><td>69.7</td><td>68.4</td><td>63.5</td><td>81.2</td><td>76.9</td><td>80.4</td><td>76.0</td><td>38.5</td><td>36.6</td><td>76.6</td><td>71.1</td><td>79.0</td><td>73.4</td><td>76.2</td><td>70.6</td><td>76.1</td><td>70.1</td></tr><tr><td>*Qwen3 235B</td><td>41</td><td>61.1</td><td>56.3</td><td>59.5</td><td>56.3</td><td>76.5</td><td>70.5</td><td>77.8</td><td>72.0</td><td>68.9</td><td>64.6</td><td>78.8</td><td>74.2</td><td>78.2</td><td>72.9</td><td>38.5</td><td>36.6</td><td>76.6</td><td>71.9</td><td>64.1</td><td>78.5</td><td>71.3</td><td>72.0</td><td>65.3</td><td>70.0</td></tr><tr><td>IRB-MT</td><td>42</td><td>67.5</td><td>60.6</td><td>59.4</td><td>57.4</td><td>68.9</td><td>58.2</td><td>71.9</td><td>63.0</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>TransitionMT</td><td>43</td><td>70.0</td><td>63.3</td><td>62.0</td><td>58.4</td><td>77.2</td><td>68.9</td><td>79.0</td><td>71.6</td><td>68.8</td><td>74.0</td><td>68.1</td><td>77.6</td><td>71.4</td><td>23.9</td><td>22.5</td><td>76.2</td><td>69.1</td><td>23.9</td><td>22.5</td><td>75.2</td><td>67.6</td><td>64.6</td><td>59.6</td><td>59.6</td></tr><tr><td>*NLLB</td><td>44</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>DLUT_GTCOM</td><td>45</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Yandex</td><td>46</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*Qwen 2.5</td><td>47</td><td>54.7</td><td>50.2</td><td>61.3</td><td>58.9</td><td>65.5</td><td>59.4</td><td>49.8</td><td>46.8</td><td>42.9</td><td>40.4</td><td>76.9</td><td>72.5</td><td>71.8</td><td>67.2</td><td>36.7</td><td>33.8</td><td>33.8</td><td>67.1</td><td>67.0</td><td>61.3</td><td>56.8</td><td>63.6</td><td>59.5</td><td>76.8</td><td>71.2</td></tr><tr><td>CUNI-SFT</td><td>48</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*ONLINE-G</td><td>49</td><td>62.9</td><td>53.5</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>SH</td><td>50</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>CGFOKUS</td><td>51</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Kaze-MT</td><td>52</td><td>32.1</td><td>27.5</td><td>32.0</td><td>27.6</td><td>32.5</td><td>27.9</td><td>32.5</td><td>28.1</td><td>32.3</td><td>28.1</td><td>32.3</td><td>32.3</td><td>28.1</td><td>32.3</td><td>32.3</td><td>28.1</td><td>32.3</td><td>32.3</td><td>32.3</td><td>32.3</td><td>32.3</td><td>32.3</td><td>32.3</td><td>32.3</td><td>32.3</td></tr><tr><td>cpc_nlp</td><td>52</td><td>32.1</td><td>27.5</td><td>32.0</td><td>27.6</td><td>62.4</td><td>53.5</td><td>32.5</td><td>28.1</td><td>32.3</td><td>28.1</td><td>32.3</td><td>32.3</td><td>28.1</td><td>32.3</td><td>32.3</td><td>28.1</td><td>32.3</td><td>32.3</td><td>32.3</td><td>32.3</td><td>32.3</td><td>32.3</td><td>32.3</td><td>32.3</td><td>32.3</td></tr><tr><td>KYGoM</td><td>52</td><td>32.1</td><td>27.5</td><td>32.0</td><td>27.6</td><td>32.5</td><td>27.9</td><td>32.5</td><td>28.1</td><td>32.3</td><td>27.7</td><td>27.7</td><td>27.7</td><td>27.7</td><td>27.7</td><td>27.7</td><td>27.7</td><td>27.7</td><td>27.7</td><td>27.7</td><td>27.7</td><td>27.7</td><td>27.7</td><td>27.7</td><td>27.7</td><td>27.7</td></tr><tr><td>CUNI-Doc-Transformer</td><td>55</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>COILD-BHO</td><td>55</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr></tbody></table>
<table><caption>Table 4: MetricX scores (lower is better) across language directions for original and normalised inputs (manual segmentation in both instances). Systems are sorted by their MetricX-based ranking (over all language pairs). Systems run by the task organisers are marked with an asterisk.</caption><thead><tr><th rowspan="2">System</th><th rowspan="2">Rank</th><th colspan="2">en-ar_EG</th><th colspan="2">en-bho_IN</th><th colspan="2">en-es_CZ</th><th colspan="2">en-et_EE</th><th colspan="2">en-is_IS</th><th colspan="2">en-ja_JP</th><th colspan="2">en-ko_KR</th><th colspan="2">en-mas_KE</th><th colspan="2">en-ru_RU</th><th colspan="2">en-st_Latn_RS</th><th colspan="2">en-uk_UA</th><th colspan="2">en-zh_CN</th></tr><tr><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th><th>norm</th><th>orig</th></tr></thead><tbody><tr><td>Shy-hunyanu-MT</td><td>1</td><td>3.4</td><td>3.8</td><td>3.9</td><td>4.2</td><td>4.7</td><td>5.4</td><td>5.7</td><td>6.4</td><td>5.2</td><td>6.1</td><td>4.0</td><td>4.5</td><td>3.7</td><td>4.1</td><td>11.8</td><td>11.6</td><td>3.1</td><td>3.8</td><td>2.2</td><td>2.6</td><td>3.9</td><td>4.6</td><td>2.3</td><td>2.8</td></tr><tr><td>GemTrans</td><td>2</td><td>3.5</td><td>3.9</td><td>4.0</td><td>4.3</td><td>4.8</td><td>5.5</td><td>6.1</td><td>6.9</td><td>7.3</td><td>4.0</td><td>3.8</td><td>4.3</td><td>4.7</td><td>15.0</td><td>15.0</td><td>3.4</td><td>3.8</td><td>2.1</td><td>2.7</td><td>4.1</td><td>4.7</td><td>2.5</td><td>2.9</td></tr><tr><td>Yolu</td><td>3</td><td>3.6</td><td>4.2</td><td>5.5</td><td>5.8</td><td>5.0</td><td>5.9</td><td>5.7</td><td>6.7</td><td>9.3</td><td>11.4</td><td>4.2</td><td>4.6</td><td>4.0</td><td>4.5</td><td>9.3</td><td>11.4</td><td>3.7</td><td>4.6</td><td>2.1</td><td>2.7</td><td>4.4</td><td>5.3</td><td>2.7</td><td>3.2</td></tr><tr><td>CommandA-WMT</td><td>4</td><td>4.1</td><td>4.7</td><td>4.4</td><td>4.8</td><td>4.5</td><td>5.1</td><td>6.6</td><td>7.3</td><td>7.9</td><td>8.9</td><td>4.1</td><td>3.8</td><td>4.1</td><td>4.7</td><td>10.8</td><td>4.1</td><td>4.7</td><td>3.9</td><td>4.6</td><td>3.9</td><td>4.6</td><td>2.8</td><td>3.2</td></tr><tr><td>Lanqi</td><td>5</td><td>5.5</td><td>6.0</td><td>6.4</td><td>6.5</td><td>6.5</td><td>5.9</td><td>5.4</td><td>6.5</td><td>6.1</td><td>7.0</td><td>4.4</td><td>4.8</td><td>4.0</td><td>4.5</td><td>15.0</td><td>15.0</td><td>3.5</td><td>4.5</td><td>2.6</td><td>—</td><td>4.2</td><td>4.9</td><td>2.7</td><td>3.3</td></tr><tr><td>*GPT-4.1</td><td>6</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*Gemini 2.5 Pro</td><td>7</td><td>6.2</td><td>6.5</td><td>6.2</td><td>6.5</td><td>5.9</td><td>6.8</td><td>6.7</td><td>7.6</td><td>6.0</td><td>6.9</td><td>4.4</td><td>4.9</td><td>4.5</td><td>5.0</td><td>15.7</td><td>16.0</td><td>4.6</td><td>5.4</td><td>3.0</td><td>3.3</td><td>5.2</td><td>6.1</td><td>3.0</td><td>3.5</td></tr><tr><td>SalamandraTA</td><td>8</td><td>7.6</td><td>8.6</td><td>9.7</td><td>10.3</td><td>5.7</td><td>7.2</td><td>6.7</td><td>8.4</td><td>6.9</td><td>8.4</td><td>6.0</td><td>6.9</td><td>6.0</td><td>6.9</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Kaze-MT</td><td>9</td><td>8.9</td><td>9.3</td><td>8.4</td><td>8.8</td><td>9.4</td><td>9.7</td><td>8.7</td><td>9.2</td><td>8.6</td><td>9.0</td><td>8.7</td><td>9.0</td><td>8.8</td><td>9.1</td><td>9.3</td><td>9.7</td><td>9.1</td><td>9.5</td><td>8.4</td><td>8.7</td><td>9.2</td><td>9.5</td><td>8.8</td><td>9.1</td></tr><tr><td>KYLiOM</td><td>10</td><td>8.9</td><td>9.3</td><td>8.4</td><td>8.8</td><td>9.4</td><td>9.7</td><td>8.7</td><td>9.2</td><td>8.6</td><td>9.0</td><td>8.7</td><td>9.0</td><td>8.8</td><td>9.1</td><td>9.3</td><td>9.7</td><td>9.1</td><td>9.5</td><td>8.4</td><td>8.7</td><td>12.5</td><td>15.3</td><td>8.8</td><td>9.1</td></tr><tr><td>ctpc_nlp</td><td>11</td><td>8.9</td><td>9.3</td><td>8.4</td><td>8.8</td><td>8.8</td><td>12.7</td><td>15.2</td><td>8.7</td><td>9.2</td><td>8.6</td><td>9.0</td><td>8.7</td><td>9.0</td><td>8.8</td><td>9.1</td><td>9.3</td><td>9.7</td><td>9.1</td><td>9.5</td><td>8.4</td><td>8.7</td><td>9.2</td><td>9.5</td><td>8.8</td><td>9.1</td></tr><tr><td>Yandex</td><td>12</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>NNTSU</td><td>12</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>KIKIS</td><td>12</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Erlendur</td><td>12</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>UvA-MT</td><td>16</td><td>5.0</td><td>5.6</td><td>6.5</td><td>7.1</td><td>5.9</td><td>6.6</td><td>8.2</td><td>8.9</td><td>9.4</td><td>9.9</td><td>4.7</td><td>5.2</td><td>4.3</td><td>4.8</td><td>13.0</td><td>13.0</td><td>4.2</td><td>5.0</td><td>2.6</td><td>3.1</td><td>4.9</td><td>5.5</td><td>3.2</td><td>3.5</td></tr><tr><td>IR-MultiagentMT</td><td>17</td><td>5.5</td><td>5.8</td><td>6.7</td><td>6.9</td><td>6.2</td><td>6.8</td><td>7.7</td><td>8.3</td><td>7.3</td><td>8.0</td><td>4.8</td><td>5.3</td><td>4.6</td><td>4.9</td><td>14.7</td><td>14.9</td><td>4.6</td><td>5.2</td><td>2.8</td><td>3.2</td><td>5.3</td><td>5.8</td><td>3.2</td><td>3.6</td></tr><tr><td>*Gemma 3 27B</td><td>18</td><td>5.5</td><td>5.9</td><td>6.7</td><td>6.8</td><td>6.1</td><td>6.7</td><td>7.7</td><td>8.6</td><td>8.2</td><td>8.8</td><td>4.6</td><td>5.0</td><td>4.4</td><td>5.0</td><td>15.4</td><td>15.7</td><td>4.4</td><td>5.2</td><td>2.7</td><td>3.2</td><td>5.2</td><td>6.0</td><td>3.1</td><td>3.5</td></tr><tr><td>Agharb</td><td>19</td><td>4.0</td><td>4.7</td><td>6.3</td><td>6.5</td><td>6.2</td><td>7.0</td><td>6.9</td><td>8.0</td><td>9.3</td><td>11.4</td><td>4.7</td><td>5.0</td><td>4.6</td><td>5.1</td><td>9.3</td><td>11.4</td><td>4.9</td><td>5.8</td><td>2.9</td><td>3.5</td><td>5.5</td><td>6.4</td><td>3.2</td><td>3.8</td></tr><tr><td>*ONLINE-B</td><td>20</td><td>4.0</td><td>4.9</td><td>7.2</td><td>9.8</td><td>5.9</td><td>7.3</td><td>6.9</td><td>8.3</td><td>6.1</td><td>7.2</td><td>4.3</td><td>4.9</td><td>4.4</td><td>5.0</td><td>5.0</td><td>4.6</td><td>5.3</td><td>4.7</td><td>4.7</td><td>5.2</td><td>5.3</td><td>6.3</td><td>3.3</td><td>4.0</td></tr><tr><td>*TowerPlus 9B</td><td>21</td><td>12.9</td><td>13.3</td><td>5.3</td><td>5.8</td><td>5.8</td><td>6.4</td><td>7.7</td><td>15.8</td><td>16.4</td><td>6.2</td><td>7.2</td><td>4.8</td><td>5.3</td><td>4.7</td><td>5.2</td><td>18.1</td><td>17.7</td><td>4.8</td><td>5.7</td><td>4.6</td><td>5.3</td><td>5.2</td><td>6.6</td><td>3.4</td><td>4.1</td></tr><tr><td>*TowerPlus 72B</td><td>22</td><td>7.1</td><td>7.7</td><td>5.3</td><td>5.8</td><td>6.6</td><td>7.8</td><td>13.0</td><td>13.5</td><td>6.5</td><td>7.6</td><td>4.6</td><td>5.3</td><td>4.7</td><td>5.3</td><td>18.3</td><td>18.1</td><td>4.8</td><td>5.8</td><td>4.0</td><td>4.9</td><td>4.2</td><td>5.6</td><td>6.0</td><td>3.0</td><td>3.5</td></tr><tr><td>*DeepSeek V3</td><td>23</td><td>5.8</td><td>6.4</td><td>6.1</td><td>6.1</td><td>6.8</td><td>6.1</td><td>7.0</td><td>7.7</td><td>8.5</td><td>7.6</td><td>8.5</td><td>4.6</td><td>5.0</td><td>4.4</td><td>5.0</td><td>15.3</td><td>15.5</td><td>4.5</td><td>5.2</td><td>2.7</td><td>3.3</td><td>5.3</td><td>6.5</td><td>3.0</td><td>3.7</td></tr><tr><td>*Claude4</td><td>24</td><td>5.7</td><td>6.2</td><td>5.9</td><td>6.4</td><td>6.3</td><td>7.6</td><td>7.9</td><td>9.1</td><td>7.1</td><td>8.3</td><td>4.5</td><td>5.1</td><td>4.4</td><td>4.9</td><td>15.3</td><td>15.5</td><td>4.9</td><td>6.0</td><td>2.9</td><td>3.6</td><td>5.5</td><td>6.5</td><td>3.0</td><td>3.7</td><td>3.4</td></tr><tr><td>*Mistral-Medium</td><td>25</td><td>6.1</td><td>6.6</td><td>6.6</td><td>7.0</td><td>6.3</td><td>6.3</td><td>7.6</td><td>9.1</td><td>10.1</td><td>5.0</td><td>4.5</td><td>4.5</td><td>4.4</td><td>4.9</td><td>15.9</td><td>15.7</td><td>5.0</td><td>5.9</td><td>3.0</td><td>3.6</td><td>5.5</td><td>6.4</td><td>3.3</td><td>3.7</td><td>3.7</td></tr><tr><td>*CommandA</td><td>26</td><td>5.6</td><td>6.1</td><td>6.6</td><td>7.0</td><td>6.1</td><td>7.0</td><td>7.9</td><td>11.0</td><td>11.9</td><td>12.3</td><td>12.7</td><td>4.7</td><td>5.1</td><td>4.9</td><td>15.8</td><td>16.1</td><td>4.8</td><td>5.8</td><td>3.5</td><td>4.3</td><td>5.8</td><td>6.7</td><td>2.9</td><td>3.5</td><td>3.5</td></tr><tr><td>*Qowen 235B</td><td>27</td><td>7.4</td><td>7.9</td><td>8.0</td><td>7.6</td><td>7.0</td><td>7.7</td><td>7.0</td><td>7.0</td><td>7.3</td><td>11.3</td><td>12.4</td><td>4.7</td><td>5.3</td><td>4.5</td><td>4.9</td><td>15.8</td><td>16.7</td><td>4.8</td><td>5.8</td><td>3.5</td><td>4.3</td><td>5.3</td><td>6.4</td><td>3.3</td><td>4.0</td></tr><tr><td>SRPOL</td><td>28</td><td>4.5</td><td>5.4</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*AyaExpense 8B</td><td>29</td><td>4.8</td><td>5.3</td><td>6.9</td><td>7.4</td><td>7.0</td><td>7.7</td><td>7.0</td><td>7.3</td><td>23.9</td><td>23.2</td><td>23.2</td><td>5.0</td><td>5.4</td><td>4.7</td><td>5.2</td><td>21.6</td><td>21.2</td><td>5.9</td><td>6.5</td><td>5.5</td><td>6.1</td><td>6.0</td><td>6.7</td><td>3.7</td><td>4.0</td></tr><tr><td>InZX</td><td>30</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>IRB-MT</td><td>31</td><td>6.3</td><td>6.7</td><td>7.9</td><td>8.1</td><td>6.7</td><td>7.2</td><td>8.9</td><td>9.7</td><td>10.4</td><td>10.4</td><td>4.8</td><td>5.3</td><td>4.8</td><td>5.4</td><td>14.3</td><td>14.6</td><td>4.7</td><td>5.4</td><td>2.8</td><td>3.4</td><td>5.3</td><td>6.1</td><td>3.1</td><td>3.6</td></tr><tr><td>*Gemma 3 12B</td><td>32</td><td>6.5</td><td>6.9</td><td>8.1</td><td>8.3</td><td>6.6</td><td>7.5</td><td>9.3</td><td>10.0</td><td>10.9</td><td>11.3</td><td>5.2</td><td>5.7</td><td>5.1</td><td>5.6</td><td>14.1</td><td>14.1</td><td>5.0</td><td>5.6</td><td>2.9</td><td>3.4</td><td>5.3</td><td>6.1</td><td>3.4</td><td>3.9</td></tr><tr><td>Transition Translate</td><td>33</td><td>6.7</td><td>8.1</td><td>7.2</td><td>9.5</td><td>6.3</td><td>8.0</td><td>7.2</td><td>8.9</td><td>6.4</td><td>7.8</td><td>4.7</td><td>5.5</td><td>4.5</td><td>5.6</td><td>9.3</td><td>11.4</td><td>4.9</td><td>6.6</td><td>4.2</td><td>7.0</td><td>5.7</td><td>7.2</td><td>3.7</td><td>4.7</td><td>4.7</td></tr><tr><td>Wenyiul</td><td>34</td><td>4.5</td><td>5.5</td><td>6.7</td><td>7.1</td><td>7.0</td><td>8.2</td><td>8.0</td><td>9.5</td><td>9.3</td><td>11.4</td><td>5.1</td><td>5.7</td><td>5.1</td><td>5.9</td><td>9.3</td><td>11.4</td><td>5.5</td><td>6.8</td><td>3.1</td><td>3.8</td><td>6.0</td><td>7.1</td><td>3.4</td><td>4.1</td><td>4.1</td></tr><tr><td>*AyaExpense 32B</td><td>35</td><td>5.7</td><td>6.1</td><td>8.1</td><td>6.4</td><td>7.0</td><td>20.8</td><td>20.7</td><td>20.6</td><td>20.4</td><td>4.8</td><td>4.1</td><td>4.6</td><td>5.0</td><td>19.3</td><td>19.1</td><td>5.2</td><td>5.8</td><td>3.8</td><td>4.3</td><td>5.6</td><td>6.2</td><td>3.4</td><td>3.4</td><td>3.9</td></tr><tr><td>*EuroLLM 22B</td><td>36</td><td>5.5</td><td>6.3</td><td>5.9</td><td>6.6</td><td>6.3</td><td>7.5</td><td>7.7</td><td>8.9</td><td>21.9</td><td>21.7</td><td>4.9</td><td>5.7</td><td>4.8</td><td>5.6</td><td>19.6</td><td>18.5</td><td>5.3</td><td>6.3</td><td>2.9</td><td>3.7</td><td>5.8</td><td>6.9</td><td>3.3</td><td>4.0</td><td>3.6</td></tr><tr><td>AMI</td><td>37</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*Llama-4-Merrick</td><td>38</td><td>6.0</td><td>6.6</td><td>6.7</td><td>7.2</td><td>6.3</td><td>7.6</td><td>7.8</td><td>9.1</td><td>8.2</td><td>9.2</td><td>4.7</td><td>5.1</td><td>4.5</td><td>5.0</td><td>15.1</td><td>15.3</td><td>4.9</td><td>6.0</td><td>2.8</td><td>3.5</td><td>5.4</td><td>6.4</td><td>3.0</td><td>3.6</td></tr><tr><td>*NLLIB</td><td>39</td><td>7.3</td><td>8.6</td><td>6.7</td><td>7.6</td><td>7.9</td><td>10.1</td><td>10.0</td><td>11.8</td><td>9.2</td><td>10.8</td><td>8.1</td><td>9.0</td><td>6.5</td><td>7.8</td><td>12.9</td><td>13.8</td><td>7.7</td><td>9.7</td><td>12.9</td><td>13.8</td><td>7.9</td><td>9.9</td><td>7.9</td><td>8.8</td></tr><tr><td>DLUT_GTCOM</td><td>40</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*EuroLLM 9B</td><td>41</td><td>6.3</td><td>7.0</td><td>7.6</td><td>8.4</td><td>6.7</td><td>8.1</td><td>8.1</td><td>9.5</td><td>22.3</td><td>21.8</td><td>5.3</td><td>5.8</td><td>5.1</td><td>5.8</td><td>15.8</td><td>15.1</td><td>5.7</td><td>7.0</td><td>3.2</td><td>4.4</td><td>6.1</td><td>7.3</td><td>3.6</td><td>4.4</td></tr><tr><td>*ONLINE-W</td><td>42</td><td>5.1</td><td>6.4</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*CommandR</td><td>43</td><td>5.4</td><td>6.2</td><td>10.1</td><td>10.5</td><td>8.6</td><td>10.0</td><td>9.1</td><td>13.5</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Systan</td><td>44</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>CUNI-SFT</td><td>45</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>CUNI-MH-v2</td><td>46</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*Llama 3.1 8B</td><td>47</td><td>10.3</td><td>10.7</td><td>8.9</td><td>9.2</td><td>9.3</td><td>10.4</td><td>15.5</td><td>15.7</td><td>17.8</td><td>17.8</td><td>6.4</td><td>6.7</td><td>6.4</td><td>6.9</td><td>16.6</td><td>16.4</td><td>7.7</td><td>8.7</td><td>4.0</td><td>4.6</td><td>8.3</td><td>9.1</td><td>3.8</td><td>4.4</td></tr><tr><td>*Qowen 2.5</td><td>48</td><td>11.0</td><td>11.5</td><td>11.7</td><td>11.8</td><td>11.9</td><td>12.8</td><td>22.0</td><td>22.0</td><td>22.8</td><td>6.8</td><td>7.4</td><td>7.5</td><td>7.9</td><td>20.6</td><td>20.0</td><td>7.6</td><td>8.6</td><td>5.8</td><td>6.7</td><td>12.0</td><td>12.7</td><td>3.6</td><td>4.2</td></tr><tr><td>CGFOKUS</td><td>49</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>b888</td><td>49</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>TransitionMT</td><td>51</td><td>6.8</td><td>8.9</td><td>7.2</td><td>8.5</td><td>12.3</td><td>16.0</td><td>14.2</td><td>17.0</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*ONLINE-G</td><td>52</td><td>10.3</td><td>13.4</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*Mistral 7B</td><td>53</td><td>14.9</td><td>15.3</td><td>14.1</td><td>14.1</td><td>12.9</td><td>13.5</td><td>23.6</td><td>23.7</td><td>22.6</td><td>22.6</td><td>22.6</td><td>22.6</td><td>22.6</td><td>22.6</td><td>22.6</td><td>22.6</td><td>22.6</td><td>22.6</td><td>22.6</td><td>22.6</td><td>22.6</td><td>22.6</td><td>22.6</td><td>22.6</td></tr><tr><td>SH</td><td>54</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>CUNI-DocTransformer</td><td>55</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>COILD-BHO</td><td>55</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr></tbody></table>
<table><caption>Table 5: CometKiwi scores (higher is better) across language directions for manual (man) segmentation versus newline (nl) segmentation on original outputs. System are sorted by their CometKiwi-based ranking (over all language pairs). Systems run by the task organisers are marked with an asterisk.</caption><thead><tr><th>System</th><th>Rank</th><th>en-ar_EG</th><th>en-bio_IN</th><th>en-es_CZ</th><th>en-et_EE</th><th>en-is_IS</th><th>en-ja_JP</th><th>en-ko_KR</th><th>en-mus_KE</th><th>en-ru_RU</th><th>en-sr_Lat_RS</th><th>en-uk_UA</th><th>en-zh_CN</th></tr><tr><th></th><th></th><th>man</th><th>nl</th><th>man</th><th>nl</th><th>man</th><th>man</th><th>man</th><th>man</th><th>man</th><th>man</th><th>man</th><th>man</th><th>man</th></tr></thead><tbody><tr><td>Yohu</td><td>1</td><td>73.4</td><td>72.7</td><td>68.9</td><td>66.2</td><td>76.3</td><td>75.6</td><td>79.7</td><td>78.3</td><td>78.2</td><td>23.0</td><td>76.0</td><td>74.6</td><td>76.1</td><td>75.8</td></tr><tr><td>CommandA-WMT</td><td>2</td><td>65.6</td><td>66.2</td><td>72.8</td><td>73.6</td><td>75.1</td><td>74.8</td><td>78.7</td><td>78.2</td><td>78.1</td><td>77.8</td><td>62.6</td><td>62.0</td><td>74.7</td><td>75.0</td></tr><tr><td>SilamandraTX</td><td>3</td><td>66.8</td><td>65.4</td><td>57.4</td><td>56.9</td><td>76.0</td><td>76.5</td><td>78.9</td><td>79.2</td><td>78.4</td><td>77.1</td><td>76.1</td><td>75.5</td><td>74.4</td><td>75.3</td></tr><tr><td>Shy-hunyuan-MT</td><td>4</td><td>70.7</td><td>70.2</td><td>76.6</td><td>74.8</td><td>74.4</td><td>77.6</td><td>77.2</td><td>71.4</td><td>71.8</td><td>77.7</td><td>76.9</td><td>73.9</td><td>75.7</td><td>78.3</td></tr><tr><td>Lanqo</td><td>5</td><td>71.2</td><td>71.6</td><td>68.6</td><td>73.8</td><td>73.7</td><td>75.7</td><td>75.5</td><td>67.9</td><td>67.6</td><td>78.2</td><td>78.0</td><td>77.2</td><td>76.9</td><td>77.0</td></tr><tr><td>SRPOL</td><td>6</td><td>66.4</td><td>66.4</td><td>68.2</td><td>68.6</td><td>73.8</td><td>73.7</td><td>75.7</td><td>75.9</td><td>71.4</td><td>72.8</td><td>78.3</td><td>78.2</td><td>76.6</td><td>74.4</td></tr><tr><td>UVA-MT</td><td>7</td><td>71.4</td><td>71.1</td><td>54.5</td><td>54.3</td><td>73.1</td><td>72.8</td><td>75.8</td><td>75.6</td><td>71.4</td><td>72.8</td><td>78.2</td><td>78.0</td><td>77.1</td><td>77.2</td></tr><tr><td>*ONLINE-B</td><td>8</td><td>46.0</td><td>45.2</td><td>75.8</td><td>75.7</td><td>72.9</td><td>73.2</td><td>57.9</td><td>59.2</td><td>70.8</td><td>70.8</td><td>77.5</td><td>77.4</td><td>77.2</td><td>73.9</td></tr><tr><td>*TowerPlus 9B</td><td>9</td><td>61.7</td><td>60.2</td><td>54.9</td><td>54.6</td><td>72.9</td><td>72.9</td><td>76.0</td><td>75.7</td><td>71.1</td><td>71.1</td><td>78.5</td><td>78.3</td><td>77.1</td><td>73.6</td></tr><tr><td>TransitionTranslate</td><td>10</td><td>60.4</td><td>60.7</td><td>76.2</td><td>76.0</td><td>72.7</td><td>72.8</td><td>67.4</td><td>67.5</td><td>70.4</td><td>70.3</td><td>77.1</td><td>77.0</td><td>76.9</td><td>75.3</td></tr><tr><td>TowerPlus 72B</td><td>11</td><td>60.5</td><td>60.3</td><td>62.8</td><td>62.5</td><td>72.7</td><td>72.2</td><td>50.9</td><td>50.9</td><td>47.8</td><td>47.8</td><td>77.0</td><td>76.9</td><td>76.4</td><td>72.5</td></tr><tr><td>GemTrans</td><td>12</td><td>70.3</td><td>69.9</td><td>76.9</td><td>76.6</td><td>72.3</td><td>72.2</td><td>74.8</td><td>74.9</td><td>67.5</td><td>67.3</td><td>76.2</td><td>75.9</td><td>75.7</td><td>73.1</td></tr><tr><td>*AyaExpanse 32B</td><td>13</td><td>52.0</td><td>51.8</td><td>62.1</td><td>62.1</td><td>66.7</td><td>66.7</td><td>62.6</td><td>62.6</td><td>53.4</td><td>53.2</td><td>72.2</td><td>71.9</td><td>72.0</td><td>68.1</td></tr><tr><td>*Llama 3.1 8B</td><td>14</td><td>52.0</td><td>51.8</td><td>62.1</td><td>62.1</td><td>66.7</td><td>66.7</td><td>62.6</td><td>62.6</td><td>53.2</td><td>53.2</td><td>72.2</td><td>71.9</td><td>72.0</td><td>68.1</td></tr><tr><td>Systran</td><td>15</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>In2x</td><td>15</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>In2x</td><td>17</td><td>57.9</td><td>57.8</td><td>59.6</td><td>59.3</td><td>73.2</td><td>73.4</td><td>76.3</td><td>70.5</td><td>70.5</td><td>77.2</td><td>77.0</td><td>76.8</td><td>76.5</td><td>72.6</td></tr><tr><td>*GPT-4.1</td><td>18</td><td>59.3</td><td>59.1</td><td>60.2</td><td>60.2</td><td>73.5</td><td>73.4</td><td>71.3</td><td>71.0</td><td>63.7</td><td>63.9</td><td>77.7</td><td>77.8</td><td>77.4</td><td>77.2</td></tr><tr><td>*CommandA</td><td>19</td><td>66.0</td><td>65.0</td><td>73.6</td><td>73.2</td><td>72.5</td><td>72.3</td><td>76.1</td><td>76.0</td><td>43.1</td><td>42.2</td><td>76.7</td><td>76.4</td><td>75.9</td><td>75.5</td></tr><tr><td>*EuroLLM 22B</td><td>20</td><td>69.7</td><td>69.5</td><td>71.1</td><td>71.2</td><td>71.4</td><td>71.4</td><td>73.5</td><td>73.4</td><td>37.4</td><td>37.4</td><td>76.7</td><td>76.2</td><td>75.8</td><td>75.8</td></tr><tr><td>*AyaExpanse 8B</td><td>21</td><td>59.4</td><td>59.3</td><td>59.5</td><td>59.4</td><td>72.8</td><td>72.9</td><td>74.9</td><td>74.8</td><td>69.4</td><td>69.3</td><td>77.5</td><td>77.4</td><td>76.9</td><td>76.6</td></tr><tr><td>*Claud4</td><td>22</td><td>58.9</td><td>59.3</td><td>62.5</td><td>62.3</td><td>72.4</td><td>72.4</td><td>75.3</td><td>75.5</td><td>88.8</td><td>68.7</td><td>69.6</td><td>76.4</td><td>76.2</td><td>75.4</td></tr><tr><td>*DeepSeek V3</td><td>23</td><td>63.7</td><td>63.4</td><td>61.5</td><td>62.7</td><td>76.8</td><td>77.8</td><td>72.4</td><td>74.9</td><td>74.9</td><td>69.6</td><td>76.2</td><td>76.4</td><td>76.6</td><td>75.6</td></tr><tr><td>IR-MultigenMT</td><td>24</td><td>66.5</td><td>66.7</td><td>62.2</td><td>62.7</td><td>67.8</td><td>67.8</td><td>39.5</td><td>40.4</td><td>38.7</td><td>38.7</td><td>75.5</td><td>75.2</td><td>74.6</td><td>75.3</td></tr><tr><td>*CommandR</td><td>25</td><td>57.5</td><td>57.3</td><td>57.2</td><td>57.5</td><td>71.8</td><td>72.7</td><td>72.6</td><td>64.3</td><td>64.2</td><td>64.3</td><td>75.5</td><td>75.4</td><td>74.8</td><td>75.3</td></tr><tr><td>Algharb</td><td>26</td><td>69.2</td><td>69.0</td><td>56.2</td><td>56.2</td><td>71.6</td><td>71.2</td><td>74.9</td><td>74.9</td><td>22.7</td><td>23.0</td><td>75.8</td><td>75.4</td><td>75.1</td><td>74.6</td></tr><tr><td>*ONLINE-W</td><td>27</td><td>68.7</td><td>68.3</td><td>—</td><td>—</td><td>68.9</td><td>67.3</td><td>68.0</td><td>66.2</td><td>—</td><td>—</td><td>73.8</td><td>69.6</td><td>76.1</td><td>75.4</td></tr><tr><td>*Gemma 3 27B</td><td>28</td><td>59.1</td><td>59.2</td><td>59.8</td><td>59.4</td><td>72.5</td><td>72.3</td><td>75.1</td><td>75.1</td><td>68.1</td><td>68.0</td><td>76.5</td><td>76.1</td><td>75.4</td><td>75.3</td></tr><tr><td>*EuroLLM 9B</td><td>29</td><td>64.7</td><td>64.8</td><td>65.4</td><td>66.9</td><td>72.4</td><td>72.0</td><td>75.0</td><td>74.8</td><td>41.5</td><td>41.3</td><td>75.9</td><td>75.9</td><td>75.6</td><td>75.3</td></tr><tr><td>Wenyiil</td><td>30</td><td>67.6</td><td>68.3</td><td>55.6</td><td>55.6</td><td>68.7</td><td>69.2</td><td>72.4</td><td>73.2</td><td>22.7</td><td>23.0</td><td>74.3</td><td>74.6</td><td>73.3</td><td>73.7</td></tr><tr><td>*Mistral 7B</td><td>31</td><td>45.0</td><td>44.3</td><td>52.8</td><td>52.4</td><td>59.4</td><td>59.2</td><td>67.9</td><td>67.4</td><td>40.5</td><td>67.9</td><td>67.7</td><td>67.6</td><td>67.2</td><td>67.1</td></tr><tr><td>CUNI-MH-v2</td><td>32</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>NNTSU</td><td>32</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>bb88</td><td>32</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>AMI</td><td>32</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Erlendur</td><td>32</td><td>57.7</td><td>57.5</td><td>57.3</td><td>57.3</td><td>71.8</td><td>71.6</td><td>71.6</td><td>74.4</td><td>74.7</td><td>67.0</td><td>66.6</td><td>75.8</td><td>75.5</td><td>75.4</td></tr><tr><td>*Llama-4-Maverick</td><td>37</td><td>54.8</td><td>54.5</td><td>56.5</td><td>56.6</td><td>70.7</td><td>70.6</td><td>74.4</td><td>74.5</td><td>68.8</td><td>68.7</td><td>75.1</td><td>75.1</td><td>74.0</td><td>73.6</td></tr><tr><td>*Gemini 2.5 Pro</td><td>38</td><td>60.3</td><td>59.8</td><td>60.6</td><td>60.5</td><td>72.5</td><td>72.4</td><td>73.3</td><td>73.1</td><td>66.9</td><td>66.8</td><td>76.7</td><td>76.7</td><td>76.2</td><td>75.8</td></tr><tr><td>KIKIS</td><td>39</td><td>59.0</td><td>59.6</td><td>60.4</td><td>60.2</td><td>71.3</td><td>71.3</td><td>69.7</td><td>69.9</td><td>63.5</td><td>63.3</td><td>76.9</td><td>76.7</td><td>76.0</td><td>75.5</td></tr><tr><td>*Mistral-Medium</td><td>40</td><td>56.3</td><td>56.0</td><td>56.3</td><td>56.7</td><td>70.5</td><td>69.9</td><td>72.0</td><td>72.0</td><td>64.6</td><td>63.9</td><td>74.2</td><td>74.1</td><td>72.9</td><td>72.6</td></tr><tr><td>IRB-MT</td><td>41</td><td>60.6</td><td>60.3</td><td>57.4</td><td>57.1</td><td>58.2</td><td>57.5</td><td>63.0</td><td>62.0</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>TransitionMT</td><td>42</td><td>60.3</td><td>60.3</td><td>58.4</td><td>58.5</td><td>68.9</td><td>69.2</td><td>58.4</td><td>59.2</td><td>39.1</td><td>40.4</td><td>67.9</td><td>67.7</td><td>67.6</td><td>67.2</td></tr><tr><td>*NLLB</td><td>43</td><td>63.3</td><td>60.7</td><td>—</td><td>—</td><td>68.9</td><td>69.2</td><td>59.2</td><td>39.1</td><td>39.1</td><td>40.5</td><td>67.9</td><td>67.7</td><td>67.6</td><td>67.2</td></tr><tr><td>DLUT_GTCOM</td><td>44</td><td>45</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Yandex</td><td>46</td><td>50.2</td><td>51.0</td><td>58.9</td><td>58.5</td><td>59.4</td><td>59.2</td><td>46.8</td><td>47.4</td><td>40.4</td><td>40.4</td><td>72.5</td><td>72.3</td><td>67.2</td><td>67.2</td></tr><tr><td>*Qwen 2.5</td><td>47</td><td>27.5</td><td>25.0</td><td>27.6</td><td>25.0</td><td>27.9</td><td>25.3</td><td>28.1</td><td>25.5</td><td>27.7</td><td>25.2</td><td>28.7</td><td>26.4</td><td>29.0</td><td>26.6</td></tr><tr><td>CUNI-SFT</td><td>48</td><td>27.5</td><td>25.0</td><td>27.6</td><td>25.0</td><td>53.5</td><td>55.1</td><td>28.1</td><td>25.5</td><td>27.7</td><td>25.2</td><td>28.7</td><td>26.4</td><td>26.6</td><td>26.0</td></tr><tr><td>*ONLINE-G</td><td>49</td><td>53.5</td><td>53.1</td><td>—</td><td>—</td><td>59.9</td><td>59.4</td><td>61.6</td><td>59.7</td><td>59.7</td><td>61.6</td><td>60.0</td><td>57.3</td><td>56.1</td><td>—</td></tr><tr><td>SH</td><td>50</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>CGFOKUS</td><td>51</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Kaze-MT</td><td>52</td><td>27.5</td><td>25.0</td><td>27.6</td><td>25.0</td><td>27.9</td><td>25.3</td><td>28.1</td><td>25.5</td><td>27.7</td><td>25.2</td><td>28.7</td><td>26.4</td><td>26.0</td><td>28.2</td></tr><tr><td>cpc.nlp</td><td>52</td><td>27.5</td><td>25.0</td><td>27.6</td><td>25.0</td><td>27.9</td><td>25.3</td><td>28.1</td><td>25.5</td><td>27.7</td><td>25.2</td><td>28.7</td><td>26.4</td><td>26.0</td><td>28.2</td></tr><tr><td>KYUoM</td><td>52</td><td>27.5</td><td>25.0</td><td>27.6</td><td>25.0</td><td>27.9</td><td>25.3</td><td>28.1</td><td>25.5</td><td>27.7</td><td>25.2</td><td>28.7</td><td>26.4</td><td>26.0</td><td>28.2</td></tr><tr><td>CUNI-DocTransformer</td><td>55</td><td>—</td><td>—</td><td>—</td><td>—</td><td>58.6</td><td>57.4</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>COILD-BHO</td><td>55</td><td>—</td><td>—</td><td>—</td><td>—</td><td>47.0</td><td>47.8</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr></tbody></table>
<table><caption>Table 6: MetricX scores (lower is better) across languages directions for manual (man) segmentation versus newline (nl) segmentation on original inputs. System are sorted by their MetricX-based ranking (over all language pairs). Systems run by the task organisers are marked with an asterisk.</caption><thead><tr><th rowspan="2">System</th><th rowspan="2">Rank</th><th colspan="2">en-ar_EG</th><th colspan="2">en-bho_IN</th><th colspan="2">en-cs_CZ</th><th colspan="2">en-et_EE</th><th colspan="2">en-is_IS</th><th colspan="2">en-ja_JP</th><th colspan="2">en-ko_KR</th><th colspan="2">en-mas_KE</th><th colspan="2">en-ru_RU</th><th colspan="2">en-rs_Lat_RS</th><th colspan="2">en-uk_UA</th><th colspan="2">en-zh_CN</th></tr><tr><th>man</th><th>nl</th><th>man</th><th>nl</th><th>man</th><th>nl</th><th>man</th><th>nl</th><th>man</th><th>nl</th><th>man</th><th>nl</th><th>man</th><th>nl</th><th>man</th><th>nl</th><th>man</th><th>nl</th><th>man</th><th>nl</th><th>man</th><th>nl</th><th>man</th><th>nl</th></tr></thead><tbody><tr><td>Sly-hunyuan-MT</td><td>1</td><td>3.8</td><td>4.2</td><td>4.3</td><td>5.4</td><td>6.0</td><td>6.4</td><td>6.1</td><td>6.0</td><td>4.5</td><td>4.1</td><td>4.6</td><td>4.8</td><td>4.9</td><td>4.7</td><td>4.8</td><td>4.9</td><td>4.7</td><td>4.6</td><td>4.6</td><td>4.9</td><td>4.9</td><td>3.4</td><td>3.5</td><td>2.9</td></tr><tr><td>Gem Trans</td><td>2</td><td>3.9</td><td>4.0</td><td>4.3</td><td>4.4</td><td>5.5</td><td>5.5</td><td>6.9</td><td>6.8</td><td>7.3</td><td>7.3</td><td>4.6</td><td>4.7</td><td>4.9</td><td>5.0</td><td>5.1</td><td>5.2</td><td>5.3</td><td>5.4</td><td>5.5</td><td>5.3</td><td>5.6</td><td>5.5</td><td>5.6</td><td>3.5</td><td>3.6</td></tr><tr><td>Yolu</td><td>3</td><td>4.2</td><td>4.2</td><td>5.8</td><td>6.2</td><td>5.9</td><td>5.9</td><td>6.7</td><td>6.8</td><td>11.4</td><td>11.4</td><td>4.5</td><td>4.5</td><td>5.1</td><td>5.2</td><td>5.3</td><td>5.4</td><td>5.5</td><td>5.3</td><td>5.8</td><td>5.8</td><td>6.1</td><td>5.8</td><td>5.8</td><td>3.6</td><td>3.7</td></tr><tr><td>CommandA-WMT</td><td>4</td><td>4.7</td><td>4.6</td><td>4.8</td><td>4.8</td><td>5.1</td><td>5.2</td><td>7.3</td><td>7.2</td><td>8.9</td><td>8.9</td><td>4.6</td><td>4.4</td><td>10.8</td><td>10.9</td><td>4.7</td><td>4.8</td><td>4.6</td><td>4.6</td><td>4.6</td><td>4.6</td><td>4.6</td><td>4.6</td><td>4.6</td><td>3.3</td></tr><tr><td>Lanqip</td><td>5</td><td>—</td><td>—</td><td>—</td><td>—</td><td>5.9</td><td>5.7</td><td>6.5</td><td>6.5</td><td>—</td><td>—</td><td>4.5</td><td>4.6</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*GPT-4.1</td><td>6</td><td>6.0</td><td>6.1</td><td>6.6</td><td>6.8</td><td>6.6</td><td>6.6</td><td>7.4</td><td>7.4</td><td>7.0</td><td>7.0</td><td>4.8</td><td>4.9</td><td>4.7</td><td>4.8</td><td>4.9</td><td>4.7</td><td>4.8</td><td>4.9</td><td>4.9</td><td>4.9</td><td>4.9</td><td>4.9</td><td>3.2</td><td>3.2</td></tr><tr><td>*Gemini 2.5 Pro</td><td>7</td><td>6.5</td><td>6.5</td><td>6.5</td><td>6.5</td><td>6.8</td><td>6.8</td><td>7.6</td><td>7.6</td><td>6.9</td><td>6.9</td><td>4.9</td><td>4.9</td><td>4.9</td><td>5.1</td><td>5.1</td><td>5.6</td><td>5.2</td><td>5.3</td><td>5.3</td><td>5.3</td><td>5.3</td><td>5.3</td><td>5.3</td><td>3.1</td></tr><tr><td>SalamandraTA</td><td>8</td><td>8.6</td><td>9.0</td><td>10.3</td><td>10.4</td><td>7.2</td><td>6.9</td><td>8.4</td><td>8.1</td><td>8.3</td><td>6.9</td><td>7.1</td><td>7.1</td><td>7.1</td><td>7.1</td><td>7.1</td><td>7.1</td><td>7.1</td><td>7.1</td><td>7.1</td><td>7.1</td><td>7.1</td><td>7.1</td><td>7.1</td><td>3.1</td></tr><tr><td>Kaze-MT</td><td>9</td><td>9.3</td><td>10.6</td><td>8.8</td><td>10.0</td><td>9.7</td><td>11.0</td><td>9.2</td><td>10.5</td><td>9.0</td><td>10.2</td><td>9.0</td><td>10.2</td><td>9.1</td><td>10.4</td><td>9.7</td><td>11.0</td><td>9.5</td><td>10.7</td><td>8.7</td><td>10.1</td><td>9.5</td><td>10.8</td><td>9.1</td><td>10.3</td></tr><tr><td>KYUoM</td><td>10</td><td>9.3</td><td>10.6</td><td>8.8</td><td>10.0</td><td>9.7</td><td>11.0</td><td>9.2</td><td>10.5</td><td>9.0</td><td>10.2</td><td>9.0</td><td>10.2</td><td>9.1</td><td>10.4</td><td>9.7</td><td>11.0</td><td>9.5</td><td>10.7</td><td>8.7</td><td>10.1</td><td>15.3</td><td>15.4</td><td>9.1</td><td>10.3</td></tr><tr><td>ctpc_nlp</td><td>11</td><td>9.3</td><td>10.6</td><td>8.8</td><td>10.0</td><td>15.2</td><td>14.2</td><td>9.2</td><td>10.5</td><td>9.0</td><td>10.2</td><td>9.0</td><td>10.2</td><td>9.1</td><td>10.4</td><td>9.7</td><td>11.0</td><td>9.5</td><td>10.7</td><td>8.7</td><td>10.1</td><td>9.5</td><td>10.8</td><td>9.1</td><td>10.3</td></tr><tr><td>Yandex</td><td>12</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>NNTSU</td><td>12</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>KIKIS</td><td>12</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Erlendur</td><td>12</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>UvA-MT</td><td>16</td><td>5.6</td><td>5.5</td><td>7.1</td><td>7.1</td><td>6.6</td><td>6.6</td><td>8.9</td><td>8.8</td><td>9.9</td><td>10.0</td><td>5.2</td><td>5.2</td><td>4.8</td><td>4.9</td><td>13.0</td><td>13.0</td><td>5.0</td><td>5.0</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>IR-MultigentMT</td><td>17</td><td>5.8</td><td>6.0</td><td>6.9</td><td>7.0</td><td>6.8</td><td>6.9</td><td>8.3</td><td>8.4</td><td>8.0</td><td>8.2</td><td>5.3</td><td>5.3</td><td>4.9</td><td>5.0</td><td>14.9</td><td>14.9</td><td>5.2</td><td>5.4</td><td>3.2</td><td>3.3</td><td>5.8</td><td>5.8</td><td>3.6</td><td>3.7</td></tr><tr><td>*Gemma 3 27B</td><td>18</td><td>5.9</td><td>5.9</td><td>6.8</td><td>6.7</td><td>6.8</td><td>8.6</td><td>8.5</td><td>8.8</td><td>8.9</td><td>5.0</td><td>5.1</td><td>5.0</td><td>5.1</td><td>15.7</td><td>15.8</td><td>5.2</td><td>5.3</td><td>3.2</td><td>3.3</td><td>6.0</td><td>6.1</td><td>3.5</td><td>3.6</td><td>3.6</td></tr><tr><td>Algharb</td><td>19</td><td>4.7</td><td>4.6</td><td>6.5</td><td>6.6</td><td>7.0</td><td>6.8</td><td>8.0</td><td>7.6</td><td>11.4</td><td>11.4</td><td>5.0</td><td>5.0</td><td>5.1</td><td>5.0</td><td>11.4</td><td>11.4</td><td>5.8</td><td>5.5</td><td>3.5</td><td>3.5</td><td>6.4</td><td>6.2</td><td>3.8</td><td>3.7</td></tr><tr><td>*ONLINE-B</td><td>20</td><td>4.9</td><td>5.0</td><td>5.8</td><td>5.9</td><td>7.7</td><td>7.6</td><td>16.0</td><td>7.2</td><td>7.2</td><td>5.3</td><td>5.3</td><td>5.2</td><td>5.3</td><td>17.7</td><td>17.8</td><td>5.7</td><td>5.9</td><td>5.3</td><td>5.4</td><td>6.3</td><td>6.3</td><td>4.0</td><td>4.1</td><td>4.1</td></tr><tr><td>*TowerPlus 9B</td><td>21</td><td>13.3</td><td>13.4</td><td>5.8</td><td>5.9</td><td>7.7</td><td>7.6</td><td>16.4</td><td>7.2</td><td>7.2</td><td>5.3</td><td>5.3</td><td>5.3</td><td>5.3</td><td>18.1</td><td>17.9</td><td>5.8</td><td>5.9</td><td>5.3</td><td>5.4</td><td>6.2</td><td>6.6</td><td>4.1</td><td>4.1</td><td>3.6</td></tr><tr><td>*TowerPlus 72B</td><td>22</td><td>7.7</td><td>7.8</td><td>5.8</td><td>5.8</td><td>7.8</td><td>7.7</td><td>13.3</td><td>13.3</td><td>7.6</td><td>5.3</td><td>5.3</td><td>5.3</td><td>5.3</td><td>18.1</td><td>17.8</td><td>5.8</td><td>5.9</td><td>5.2</td><td>4.2</td><td>6.6</td><td>6.6</td><td>4.0</td><td>4.0</td><td>3.7</td></tr><tr><td>*DeepSeek V3</td><td>23</td><td>6.4</td><td>6.3</td><td>6.1</td><td>6.2</td><td>7.0</td><td>6.9</td><td>8.5</td><td>8.5</td><td>8.5</td><td>5.0</td><td>5.1</td><td>5.0</td><td>5.1</td><td>15.5</td><td>15.4</td><td>6.0</td><td>6.0</td><td>3.6</td><td>3.6</td><td>6.5</td><td>6.5</td><td>3.7</td><td>3.7</td><td>3.7</td></tr><tr><td>*Clauded</td><td>24</td><td>6.2</td><td>6.2</td><td>6.4</td><td>6.4</td><td>7.6</td><td>7.4</td><td>9.1</td><td>9.0</td><td>8.3</td><td>8.3</td><td>5.1</td><td>5.2</td><td>4.9</td><td>5.0</td><td>15.5</td><td>15.4</td><td>6.0</td><td>6.0</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*Miskral-Medium</td><td>25</td><td>6.6</td><td>6.7</td><td>7.0</td><td>6.9</td><td>7.2</td><td>7.1</td><td>10.1</td><td>10.1</td><td>10.2</td><td>5.1</td><td>5.1</td><td>5.0</td><td>5.1</td><td>4.9</td><td>4.9</td><td>15.7</td><td>15.7</td><td>5.9</td><td>5.9</td><td>3.6</td><td>3.7</td><td>6.4</td><td>6.4</td><td>3.4</td></tr><tr><td>*CommandA</td><td>26</td><td>6.1</td><td>6.2</td><td>7.2</td><td>7.1</td><td>7.0</td><td>6.9</td><td>11.9</td><td>11.8</td><td>12.7</td><td>12.5</td><td>5.1</td><td>5.1</td><td>4.9</td><td>4.9</td><td>16.1</td><td>16.1</td><td>5.8</td><td>5.9</td><td>4.3</td><td>4.2</td><td>6.7</td><td>6.7</td><td>3.7</td><td>3.8</td></tr><tr><td>*Qwen3 235B</td><td>27</td><td>7.9</td><td>7.9</td><td>8.0</td><td>7.9</td><td>7.7</td><td>7.7</td><td>11.9</td><td>11.7</td><td>12.4</td><td>12.1</td><td>5.3</td><td>4.9</td><td>5.0</td><td>5.3</td><td>18.1</td><td>17.9</td><td>5.8</td><td>5.9</td><td>4.3</td><td>4.2</td><td>6.7</td><td>6.7</td><td>3.5</td><td>3.5</td></tr><tr><td>SRPOL</td><td>28</td><td>5.4</td><td>5.3</td><td>—</td><td>—</td><td>7.1</td><td>6.9</td><td>8.5</td><td>8.2</td><td>—</td><td>—</td><td>5.6</td><td>5.5</td><td>5.2</td><td>21.2</td><td>20.9</td><td>6.5</td><td>6.5</td><td>6.1</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*AyaExpanse 8B</td><td>29</td><td>5.3</td><td>5.3</td><td>—</td><td>—</td><td>7.4</td><td>7.7</td><td>7.8</td><td>23.9</td><td>23.7</td><td>23.2</td><td>—</td><td>—</td><td>5.4</td><td>5.6</td><td>5.2</td><td>5.2</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>In2x</td><td>30</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>IRB-MT</td><td>31</td><td>6.7</td><td>6.7</td><td>8.1</td><td>8.0</td><td>7.2</td><td>7.3</td><td>9.7</td><td>9.6</td><td>10.4</td><td>10.6</td><td>5.3</td><td>5.4</td><td>5.4</td><td>5.5</td><td>14.6</td><td>14.5</td><td>5.4</td><td>5.4</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*Gemma 3 12B</td><td>32</td><td>6.9</td><td>7.0</td><td>8.3</td><td>8.3</td><td>7.5</td><td>7.4</td><td>10.0</td><td>11.3</td><td>11.3</td><td>5.7</td><td>5.6</td><td>5.6</td><td>5.7</td><td>14.1</td><td>14.1</td><td>5.6</td><td>5.7</td><td>3.4</td><td>3.5</td><td>6.1</td><td>6.1</td><td>3.6</td><td>3.7</td></tr><tr><td>TransionTranslate</td><td>33</td><td>8.1</td><td>8.1</td><td>9.5</td><td>9.7</td><td>8.0</td><td>7.6</td><td>8.9</td><td>8.8</td><td>7.8</td><td>5.5</td><td>5.5</td><td>5.5</td><td>5.5</td><td>11.4</td><td>11.4</td><td>6.6</td><td>6.4</td><td>7.0</td><td>7.0</td><td>6.3</td><td>6.3</td><td>3.9</td><td>4.0</td><td>4.0</td></tr><tr><td>Wenyiil</td><td>34</td><td>5.5</td><td>5.0</td><td>7.1</td><td>7.1</td><td>8.2</td><td>7.7</td><td>9.5</td><td>8.6</td><td>11.4</td><td>5.7</td><td>5.2</td><td>5.2</td><td>5.2</td><td>5.5</td><td>11.4</td><td>11.4</td><td>6.8</td><td>6.0</td><td>3.8</td><td>3.6</td><td>7.1</td><td>6.7</td><td>4.1</td><td>3.8</td></tr><tr><td>*AyaExpanse 32B</td><td>35</td><td>6.1</td><td>6.1</td><td>8.1</td><td>8.1</td><td>7.0</td><td>7.1</td><td>20.6</td><td>20.4</td><td>20.5</td><td>5.1</td><td>5.0</td><td>5.0</td><td>5.0</td><td>19.1</td><td>19.0</td><td>5.8</td><td>5.8</td><td>4.3</td><td>4.3</td><td>6.2</td><td>6.2</td><td>3.9</td><td>3.9</td><td>3.9</td></tr><tr><td>*EuroLLM 22B</td><td>36</td><td>6.3</td><td>6.4</td><td>6.6</td><td>6.6</td><td>7.5</td><td>7.3</td><td>8.9</td><td>8.8</td><td>21.7</td><td>21.7</td><td>5.7</td><td>5.8</td><td>5.6</td><td>5.7</td><td>18.5</td><td>18.2</td><td>6.3</td><td>6.4</td><td>3.7</td><td>3.7</td><td>6.9</td><td>6.8</td><td>4.0</td><td>4.1</td></tr><tr><td>AMI</td><td>37</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*Llama-4-Maverick</td><td>38</td><td>6.6</td><td>6.4</td><td>7.2</td><td>7.1</td><td>7.6</td><td>7.5</td><td>9.1</td><td>8.8</td><td>9.2</td><td>9.0</td><td>5.1</td><td>5.2</td><td>5.0</td><td>15.3</td><td>15.4</td><td>6.0</td><td>5.9</td><td>3.5</td><td>3.5</td><td>6.4</td><td>6.4</td><td>3.6</td><td>3.7</td></tr><tr><td>*NLLB</td><td>39</td><td>8.6</td><td>9.2</td><td>7.6</td><td>7.7</td><td>10.1</td><td>11.2</td><td>11.8</td><td>13.3</td><td>10.8</td><td>9.0</td><td>9.7</td><td>7.8</td><td>8.6</td><td>13.8</td><td>12.0</td><td>9.7</td><td>11.6</td><td>13.8</td><td>12.0</td><td>9.9</td><td>11.0</td><td>8.8</td><td>9.2</td></tr><tr><td>DLIT_GTCOM</td><td>40</td><td>7.0</td><td>7.1</td><td>8.4</td><td>8.0</td><td>8.1</td><td>8.1</td><td>9.5</td><td>9.5</td><td>21.8</td><td>21.7</td><td>5.8</td><td>5.9</td><td>5.8</td><td>5.9</td><td>15.1</td><td>14.2</td><td>7.0</td><td>7.1</td><td>4.4</td><td>4.4</td><td>7.3</td><td>7.3</td><td>4.4</td><td>4.4</td></tr><tr><td>*EuroLLM 9B</td><td>41</td><td>6.4</td><td>6.5</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*ONLINE-W</td><td>42</td><td>6.2</td><td>6.2</td><td>10.5</td><td>10.1</td><td>9.8</td><td>9.7</td><td>23.2</td><td>22.8</td><td>21.9</td><td>21.3</td><td>6.0</td><td>6.0</td><td>6.0</td><td>6.1</td><td>19.6</td><td>18.4</td><td>9.8</td><td>9.7</td><td>6.7</td><td>6.7</td><td>9.7</td><td>9.5</td><td>4.5</td><td>4.5</td></tr><tr><td>*CommandR</td><td>43</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Systran</td><td>44</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>CUNI-SFT</td><td>45</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>CUNI-MH-v2</td><td>46</td><td>10.7</td><td>10.6</td><td>9.2</td><td>9.0</td><td>10.4</td><td>10.3</td><td>15.7</td><td>15.6</td><td>17.8</td><td>17.6</td><td>6.7</td><td>6.6</td><td>6.9</td><td>7.0</td><td>16.4</td><td>15.8</td><td>8.7</td><td>8.6</td><td>4.6</td><td>4.5</td><td>9.1</td><td>8.8</td><td>4.4</td><td>4.4</td></tr><tr><td>*Llama 3.1 8B</td><td>47</td><td>11.5</td><td>11.2</td><td>11.8</td><td>11.9</td><td>12.8</td><td>12.8</td><td>22.0</td><td>21.7</td><td>22.8</td><td>22.6</td><td>7.4</td><td>7.5</td><td>7.9</td><td>7.8</td><td>20.0</td><td>19.5</td><td>8.6</td><td>8.4</td><td>6.7</td><td>6.6</td><td>12.7</td><td>12.5</td><td>4.2</td><td>4.3</td></tr><tr><td>*Qwen 2.5</td><td>48</td><td>53</td><td>15.1</td><td>14.1</td><td>14.4</td><td>13.5</td><td>13.2</td><td>23.7</td><td>23.7</td><td>22.6</td><td>22.5</td><td>6.1</td><td>6.1</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>CGFOKUS</td><td>54</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>bb88</td><td>49</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>TransionMT</td><td>51</td><td>8.9</td><td>8.9</td><td>8.5</td><td>8.7</td><td>16.0</td><td>16.1</td><td>17.0</td><td>17.0</td><td>17.6</td><td>6.7</td><td>6.6</td><td>6.9</td><td>7.0</td><td>16.4</td><td>15.8</td><td>8.7</td><td>8.6</td><td>4.6</td><td>4.5</td><td>9.1</td><td>8.8</td><td>4.4</td><td>4.4</td><td>3.6</td></tr><tr><td>*ONLINE-G</td><td>52</td><td>13.4</td><td>13.5</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>*Mistral 7B</td><td>53</td><td>15.3</td><td>15.1</td><td>14.1</td><td>14.4</td><td>13.5</td><td>13.2</td><td>23.7</td><td>23.7</td><td>22.6</td><td>22.5</td><td>6.1</td><td>6.1</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>SH</td><td>54</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>CUNI-DocTransformer</td><td>55</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>COILD-BHO</td><td>55</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr></tbody></table>