# Findings of the 2023 Conference on Machine Translation
(WMT23): LLMs Are Here But Not Quite There Yet

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton

Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme

Gowda, Roman Grundkiewicz, et al.

<figure><img src="image_1.png" /></figure>

## To cite this version:

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, et al. Findings of the 2023 Conference on Machine Translation (WMT23): LLMs Are Here But Not Quite There Yet. WMT23 - Eighth Conference on Machine Translation, Dec 2023, Singapore, Singapore. pp.198–216. [hal-04300702](https://hal.science/hal-04300702v1)

HAL Id: hal-04300702

https://hal.science/hal-04300702v1

Submitted on 22 Nov 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Findings of the 2023 Conference on Machine Translation (WMT23):
## LLMs Are Here But Not Quite There Yet

**Tom Kocmi**

Microsoft

**Eleftherios Avramidis**

DFKI

**Rachel Bawden**

Inria, Paris

**Ondřej Bojar**

Charles University

**Anton Dvorkovich**

Dubformer

**Christian Federmann**

Microsoft

**Mark Fishel**

University of Tartu

**Markus Freitag**

Google

**Thamme Gowda**

Microsoft

**Roman Grundkiewicz**

Microsoft

**Barry Haddow**

University of Edinburgh

**Philipp Koehn**

Johns Hopkins University

**Benjamin Marie**

4i.ai

**Christof Monz**

University of Amsterdam

**Makoto Morishita**

NTT

**Kenton Murray**

Johns Hopkins University

**Masaaki Nagata**

NTT

**Toshiaki Nakazawa**

University of Tokyo

**Martin Popel**

Charles University

**Maja Popović**

Dublin City University

**Mariya Shmatova**

Dubformer

**Jun Suzuki**

Tohoku University

## Abstract

This paper presents the results of the General Machine Translation Task organised as part of the 2023 Conference on Machine Translation (WMT). In the general MT task, participants were asked to build machine translation systems for any of 8 language pairs (covering 14 translation directions), to be evaluated on test sets consisting of up to four different domains. We evaluate system outputs with professional human annotators using a combination of source-based Direct Assessment and scalar quality metric (DA+SQM).

## 1 Introduction

The Eighth Conference on Machine Translation (WMT23)<fnref n="1" /> was held at EMNLP 2023 and hosted a number of shared tasks on various aspects of machine translation (MT). This conference built on 17 previous editions of WMT as a workshop or a conference (Koehn and Monz, 2006; Callison-Burch et al., 2007, 2008, 2009, 2010, 2011, 2012; Bojar et al., 2013, 2014, 2015, 2016, 2017, 2018; Barrault et al., 2019, 2020; Akhbardeh et al., 2021; Kocmi et al., 2022).

Following last year's shift from focusing mainly on the news domain, we have continued to explore the capabilities of "General Machine Translation".

While the news domain provided a clear and familiar benchmark, we responded to the need to test MT in more diverse settings. Our goal is to assess MT systems' ability to handle a broader range of language use. How to test general MT performance is a research question in itself. Countless phenomena could be evaluated, the most important being:

* various domains (news, medicine, IT, patents, legal, social, gaming, etc.)

* style of text (formal or spoken language, fiction, technical reports, etc.)

* robustness to non-standard (or noisy) user-generated content (grammatical errors, code-switching, abbreviations, etc.)

Evaluating all phenomena is nearly impossible and creates numerous unforeseen problems. Therefore, we decided to simplify the problem and start with an evaluation of different domains. We selected the following domains: news, e-commerce, social/user-generated content (UGC), speech, and manuals. They were chosen to represent topics with different content styles and to be understandable for humans without special in-domain knowledge, thus not requiring specialized translators or human raters for evaluation. Due to limited access

<footnote n="1">¹http://www2.statmt.org/wmt23/</footnote>
to monolingual data across all languages, each lan-
guage direction contains only a subset of up to four
domains.

In addition to language pairs evaluated last year:

Czech→Ukrainian,
English↔Chinese,
English→Czech,
English↔German,
English↔Japanese,
English↔Russian,
Ukrainian→English,

we introduce a new language pair to WMT, namely:

English↔Hebrew.

Other than language pairs, there are several dif-
ferences with respect to last year's task. All lan-
guage pairs are provided with the sentence bound-
aries marked except for English↔German, where
we decided to experiment with paragraph-level
translation. Another significant change for this
year is the unification of our human evaluation
protocol. We no longer rely on reference-based
MTurk evaluation and move the evaluation towards
source-based DA+SQM evaluation (introduced last
year) with professional annotators. Finally, this
year's shared task included an increased number
of test suites (Section 6), allowing the evaluation
of MT outputs from different perspectives, includ-
ing a range of linguistic phenomena, purposely
difficult sentences, specialist domains, gendered
translations and non-standard UGC translation.

All General MT task submissions, sources, references and human judgements are available at Github.<fnref n="2" /> The interactive visualization and comparison of differences between systems can be browsed online on an interactive leaderboard³ using MT-ComparEval (Klejch et al., 2015; Sudarikov et al., 2016).

The structure of the paper is as follows. We
describe the process of collecting, cleaning and
translating the test sets in Section 2 followed by
a summary of the permitted training data for the
constrained track Section 3. We list all submit-
ted systems in Section 4. The human evaluation
approach of DA+SQM is described in Section 5.
Finally, Section 6 describes the test suites and sum-
marises their conclusions.

**Summary of the WMT2023 General MT task**

The main findings are as follows:

* Large Language Models (LLMs) exhibit strong performance across the majority of language pairs, although this is based only on two LLM-based system submissions. Test suite analysis revealed that although GPT4 excelled in some areas (e.g. UGC translation) struggled with other aspects such as speaker gender translation and specific domains (e.g. legal), whereas it ranked lower than encoder-decoder systems when translating from English into less-represented languages (e.g. Czech and Russian)

* We have observed a decline in the number of submissions into the constrained track. Consequently, we plan to re-evaluate the definition and the incentives of the constrained track and consider incorporating open-source LLMs in future constrained evaluations.

* We demonstrate the feasibility of paragraph-level German↔English tasks, although more investigation would be required before generalising to all language pairs.

* Professional human translations do not always guarantee high quality. For Hebrew↔English, our references are likely to be post-edited MT, while for Chinese→English, the reference translation is worse than the majority of automatic translations.

* The manual evaluation results obtained from DA+SQM and MQM methods yield comparable cluster rankings.

## 2 Test Data

In this section, we describe the process of collect-
ing data in Section 2.1, followed by the explanation
of preprocessing steps in Section 2.2. Producing
human references is summarized in Section 2.3 and
lastly test set analysis is conducted in Section 2.4.

2.1 Collecting test data

As in the previous years, the test sets consist of
unseen translations collected especially for the task.
The guaranteed novelty of the test sets has become
even more important with the rise of LLMs trained
on unspecified training data. To prevent possible
contamination, we focused on collecting as recent

<footnote n="2">²https://github.com/wmt-conference/wmt23-news-systems
³http://wmt.ufal.cz</footnote>
<table><caption>**Table 1:** Test set statistics per direction and domain (rows marked * are over all domains). Note that en→de shares source test data with the other from-English directions, but as translation and evaluation for both en→de and de→en were carried out on the paragraph level (a segment therefore being a paragraph rather than a sentence), this results in a lower number of segments per document. The domain name is as indicated in the released test sets and domain type indicates the broader domain category.</caption><thead><tr><th>Lang. pair</th><th>Domain name</th><th>Domain type</th><th>#docs</th><th>#segs</th><th>#segs/#docs</th></tr></thead><tbody><tr><td rowspan="5">cs→uk</td><td>*</td><td>*</td><td>156</td><td>2017</td><td>12.93</td></tr><tr><td>games</td><td>News</td><td>17</td><td>180</td><td>10.59</td></tr><tr><td>news</td><td>News</td><td>35</td><td>567</td><td>16.20</td></tr><tr><td>official</td><td>Social/UGC</td><td>26</td><td>347</td><td>13.35</td></tr><tr><td>personal</td><td>Social/UGC</td><td>31</td><td>390</td><td>12.58</td></tr><tr><td rowspan="5">de→en</td><td>*</td><td>*</td><td>210</td><td>549</td><td>2.61</td></tr><tr><td>manuals</td><td>Manuals</td><td>15</td><td>74</td><td>4.93</td></tr><tr><td>mastodon</td><td>Social/UGC</td><td>95</td><td>103</td><td>1.08</td></tr><tr><td>news</td><td>News</td><td>47</td><td>277</td><td>5.89</td></tr><tr><td>user_review</td><td>E-commerce</td><td>53</td><td>95</td><td>1.79</td></tr><tr><td rowspan="5">en→{cs,he,ja,ru,uk,zh}</td><td>*</td><td>*</td><td>192</td><td>2074</td><td>10.80</td></tr><tr><td>mastodon</td><td>Social/UGC</td><td>79</td><td>504</td><td>6.38</td></tr><tr><td>news</td><td>News</td><td>30</td><td>516</td><td>17.20</td></tr><tr><td>speech</td><td>Meeting notes</td><td>25</td><td>547</td><td>21.88</td></tr><tr><td>user_review</td><td>E-commerce</td><td>58</td><td>507</td><td>8.74</td></tr><tr><td rowspan="5">en→de</td><td>*</td><td>*</td><td>192</td><td>557</td><td>2.90</td></tr><tr><td>mastodon</td><td>Social/UGC</td><td>79</td><td>212</td><td>2.68</td></tr><tr><td>news</td><td>News</td><td>30</td><td>139</td><td>4.63</td></tr><tr><td>speech</td><td>Meeting notes</td><td>25</td><td>113</td><td>4.52</td></tr><tr><td>user_review</td><td>E-commerce</td><td>58</td><td>93</td><td>1.60</td></tr><tr><td rowspan="3">he→en</td><td>*</td><td>*</td><td>94</td><td>1910</td><td>20.32</td></tr><tr><td>news</td><td>News</td><td>68</td><td>1558</td><td>22.91</td></tr><tr><td>reviews</td><td>Social/UGC</td><td>26</td><td>352</td><td>13.54</td></tr><tr><td rowspan="5">ja→en</td><td>*</td><td>*</td><td>282</td><td>1992</td><td>7.06</td></tr><tr><td>ad</td><td>Social/UGC</td><td>53</td><td>245</td><td>4.62</td></tr><tr><td>ec</td><td>Social/UGC</td><td>25</td><td>255</td><td>10.20</td></tr><tr><td>news</td><td>News</td><td>37</td><td>495</td><td>13.38</td></tr><tr><td>qa</td><td>Conversational</td><td>118</td><td>497</td><td>4.21</td></tr><tr><td rowspan="5">ru→en</td><td>*</td><td>*</td><td>162</td><td>1723</td><td>10.64</td></tr><tr><td>manuals</td><td>Manuals</td><td>15</td><td>505</td><td>33.67</td></tr><tr><td>news</td><td>News</td><td>54</td><td>676</td><td>12.52</td></tr><tr><td>reviews</td><td>Social/UGC</td><td>93</td><td>542</td><td>5.83</td></tr><tr><td>user_review</td><td>E-commerce</td><td>49</td><td>500</td><td>10.20</td></tr><tr><td rowspan="5">uk→en</td><td>*</td><td>*</td><td>132</td><td>1826</td><td>13.83</td></tr><tr><td>clipboard</td><td>Social/UGC</td><td>30</td><td>504</td><td>16.80</td></tr><tr><td>news</td><td>News</td><td>26</td><td>514</td><td>19.77</td></tr><tr><td>other</td><td>Social/UGC</td><td>27</td><td>538</td><td>19.93</td></tr><tr><td>voice</td><td>Speech</td><td>49</td><td>270</td><td>5.51</td></tr><tr><td rowspan="4">zh→en</td><td>*</td><td>*</td><td>179</td><td>1976</td><td>11.04</td></tr><tr><td>manuals</td><td>Manuals</td><td>14</td><td>487</td><td>34.79</td></tr><tr><td>news</td><td>News</td><td>38</td><td>763</td><td>20.08</td></tr><tr><td>user_review</td><td>E-commerce</td><td>127</td><td>726</td><td>5.72</td></tr></tbody></table>

data as possible across various domains. This task is incredibly difficult and needs further investigation in future years. There are three main limitations:

* Finding sources with different domains.

* Finding data that are in the public domain or under open licenses.

* Finding recently created data to minimize the risk of them being part of the training pipelines.

The test sets are publicly released to be used as translation benchmarks. Here we describe the test sets' production and composition.

We decided to collect data from 5 domains (news, social/user-generated, e-commerce, manuals, and speech). For all language pairs, we aimed for a test set size of 2,000 sentences and to ensure that the test sets were "source-original", namely
that the source text was first written in the source language, and then the target text is the human translation. This is to avoid “translationese” effects in the source language, which can have a detrimental impact on the accuracy of evaluation (Toral et al., 2018; Freitag et al., 2019; Läubli et al., 2020; Graham et al., 2020). We collected roughly the same number of sentences for each domain. For some languages, we could not locate high quality data and therefore we selected more sentences from other domains. Note that descriptions in this section refer to source monolingual data when mentioning a language.

**News domain** For most languages this domain contains data prepared in the same way as in previous years (Akhbardeh et al., 2021). We collected news articles from February 2023 extracted from online news sites, preserving document boundaries. We expect that news domain text will generally be of high quality. The news in Hebrew was kindly provided by the Israeli Association of Human Language Technologies (IAHLT).<fnref n="4" /> These are samples of originally Hebrew texts from news published in Israel Hayom<fnref n="5" /> in 2022.

**E-commerce domain (product reviews)** This domain consists of user reviews of different Amazon products selected from the publicly available multilingual corpus (Keung et al., 2020). This corpus was designed for multilingual text classification and consists of reviews written in English, Japanese, German, French, Spanish, and Chinese, between 2015 and 2019. We used the test parts of the English, German, Japanese and Chinese corpus for extracting the source part of the WMT test set. The reviews were selected so that the resulting corpus covers each product, all rating scores for the product, and the lexical diversity is maximized. The lexical diversity was estimated as a simple ratio between the number of distinct words/characters (vocabulary) divided by the total number of words/characters.

**Social/user-generated domain** For English and German, we relied on the Mastodon Social API.<fnref n="6" /> Mastodon is a federated social network that is compatible with the W3C standard ActivityPub (Webber et al., 2018). Users publish short-form content

similar to tweets that are referred to as “toots” for historical reasons. As this is a decentralized social media network, different servers have very different data, policies, communities, and uses. We decided to use mastodon.social, the original server, as it has a large community as well as publicly available toots. We collected data in early May of 2023. We used the reported language ID label, but were only able to collect enough data in German and English. We only collected toots with more than 150 characters in length in order to allow for data that was more likely to be semantically interesting for evaluating translation systems.

For Hebrew, we used comments on news articles from the Israel Hayom site mentioned above. This data was also provided by IAHLT.

For Russian, we used data from the Geo Reviews Dataset containing reviews about organizations published on Yandex Maps and open for academic and research purposes.<fnref n="7" />

For Japanese, we used product descriptions of a b2b e-commerce site and search advertising text ads for the social and user-generated domain, because we could not obtain high-quality data for this domain type. MonotaRo Co., Ltd. provided product descriptions of their private label brands listed on their b2b e-commerce site.<fnref n="8" /> We defined a document for a product description as a combination of a title, product description, and cautionary note. CyberAgent, Inc.<fnref n="9" /> provided search advertising text ads with their client’s consent. We defined a document for an ad as the longest possible combination of multiple titles and descriptions.

**Manuals** For this domain, we primarily sourced scanned versions of different mostly gaming manuals provided by Centific<fnref n="10" />. These were then converted to digital text format using Optical Character Recognition (OCR) technology. Given the inaccuracies of OCR, the digitized content underwent a subsequent post-editing phase, where humans reviewed and corrected any errors. The selection of manuals ranged across various sources, and none of them were older than five years.

**Speech** The exact data types used in the “conversational” or “speech” domain vary across language pairs.

<footnote n="4">⁴https://www.iahlt.org</footnote>

<footnote n="5">⁵https://www.israelhayom.co.il</footnote>

<footnote n="6">⁶https://mastodon.social/api/v1/timelines/public</footnote>

<footnote n="7">⁷https://github.com/yandex/geo-reviews-dataset-2023</footnote>

<footnote n="8">⁸https://www.monotaro.com/</footnote>

<footnote n="9">⁹https://www.cyberagent.co.jp</footnote>

<footnote n="10">¹⁰https://www.centific.com</footnote>
For English→Czech, the data comes from the test set which was created for the 2023 instance of AutoMin 2023 (Ghosal et al., 2022).<fnref n="11" /> The texts are manually curated transcripts of project meetings, same in style as released in ELITR Minuting Corpus (Nedoluzhko et al., 2022). The meetings were held mostly remotely or in a hybrid form, all meeting participants were non-native speakers of English and the meetings were always on rather technical and in-depth topics. Our manual curation corrected ASR errors (but not errors in English grammar or vocabulary) and de-identified the transcripts, replacing names with placeholders (“PERSONxy”, “PROJECTxy” and similar). For person names, round brackets are used at the beginnings of lines to indicate the speaker and square brackets are used in the text when the person was mentioned. The data contain also some markup, e.g. `<unintelligible/>`. These conventions are likely to be distorted by translation systems and we also noticed that they were distorted in the reference translation (the style of the brackets was ignored). This tiny detail can influence both manual and automatic scoring on this domain.

For Japanese, we used question-answer pairs from a community question-answering service. NTT Resonant Inc., which recently merged with NTT DOCOMO, INC., provided question-answer pairs from their website, *Oshiete! goo*.¹² For every question-answer pair, we defined a document as a combination of a question and its best answer marked by the user.

**Czech and Ukrainian source texts** Source texts for Czech→Ukrainian and Ukrainian→English translation included the News domain as described above and texts collected through the Charles Translator for Ukraine.¹³ With users' consent, the service can log their inputs for the purpose of creating a dataset of real use cases. The datasets are extracted from the inputs collected from May 2022 to April 2023.

The Charles Translator mobile app supports voice input, which is converted to text using Google ASR (automatic speech recognition). The texts collected this way were marked as the voice domain. For Ukrainian→English, the remaining Ukrainian inputs were classified either as clipboard (texts inserted to the Charles Trans-

lator using the *Paste from clipboard* button) and other. The clipboard texts are more likely to include formal communication copied from web sites, but we noticed it includes personal communication (copied from chat applications) as well. Thus for Czech→Ukrainian, we decided to classify the remaining Czech inputs either as **official** (formal communication) or **personal** (personal communication), ignoring whether they were inserted from a clipboard or written using a keyboard.

The texts were filtered and pseudonymized in the same way as last year (Kocmi et al., 2022), so for example we asked the annotators not to delete or fix noisy inputs as long as they are comprehensible. There was one exception from this rule this year: the Czech voice domain data was post-edited to fix ASR errors, including missing punctuation and casing.

The source texts were translated by professional translators principally following the brief in Appendix C. Last year, parts of the Ukrainian→Czech test set was detected to be post-edited MT. Therefore this year, we decided to hire two professional translators directly without the mediation of a translation agency, we emphasised the rule that the translations must be done from scratch (without MT postediting and without translation memories). We could not detect any MT postediting in the resulting translations.

## 2.2 Human preprocessing of test data

Although testing of robustness of MT is an important task, the noisy data introduces problems for human translators and annotators. Therefore, we decided to discard data considered too noisy. Furthermore, publicly available data often contains inappropriate content, which can stress either human translators or human annotators, leading to a decrease in the quality (for example, translators refuse to translate political content considered censored in their countries).

Therefore, we asked humans to check collected data and carry out minor corrections (mainly checking sentence splits and discarding similar or repeated content). This was sufficient for the news domain because it was often clean and without serious problems. However, with the expansion towards general MT, we find ourselves running into an issue of source data being noisier and less well formatted and that therefore needs to be handled before translation. Furthermore, we asked them to

<footnote n="11">¹¹https://ufal.github.io/automin-2023/
¹²https://oshiete.goo.ne.jp/
¹³http://translator.cuni.cz</footnote>
remove shortest documents to keep longer context. The source data for test sets therefore goes through human validation checks involving linguists discarding inappropriate content altogether and carrying out minor textual corrections to the data. You can find the linguistic brief for prepossessing in Appendix B.

## 2.3 Test set translation

The translation of the test sets was performed by professional translation agencies, according to the brief in Appendix C. Different partners sponsored each language pair and various translation agencies were therefore used, which may affect the quality of the translation.

Regrettably, upon reviewing translations procured from one of the agencies (the one responsible for English to Hebrew and Hebrew to English translations), it appeared that the translations might have been post-edited from publicly available online translation systems. This observation contradicts the initial instruction provided for agency that precluded the use of any automated translation platforms. While the agency has asserted that their professional translations conducted translations from scratch, our evaluation suggested otherwise. Moving forward, we propose to build a step-by-step verification system to avoid such discrepancies.

Human translations would not be possible without the sponsorship of our partners: Microsoft, Toloka AI, Google, Charles University, NTT, and Dubformer.

## 2.4 Test set analysis

As described previously, the chosen domains, sources for the data and the number of sentences per domain was subject to the availability of high quality data in each language direction. For example, while the news domain was available for all language directions, social media data was only available for English, German (both from Mastodon) and Hebrew (from comments on news articles). The number of documents, segments, average document length and type-token ratio (of the source side of the test sets) are given in Table 1.

**Document context** Document context is available for all language directions, although the average document length varies both by domain and language direction. Manuals tend to represent the longest domains, followed by the news domain. The social media domain tends to represent the

shortest documents. along with reviews. Note that this year, we piloted translation and evaluation of en→de and de→en at the paragraph level (with each segment therefore containing several sentences), with the aim of avoiding the constraint of having a one-to-one mapping at the level of the sentence between source texts and their translations. This is visible in the statistics in Table 1 as the number of segments is lower for these two directions, as is the average document length.

**Lexical diversity** We can compare the type-token ratio (TTR) to get an idea of the relative lexical diversity of (i) domains and (ii) original vs. translated sentences.<fnref n="14" />,<fnref n="15" /> Raw TTRs for each language pair and domain are shown in Table 11 in Appendix D. Regarding domains, the TTR appears highest for texts mastodon, perhaps illustrating the diversity of conversational topics and also of the potentially non-standard nature of the texts. User reviews appear to have the lowest TTR, most likely due to the fact that similar vocabulary is used across reviews. The TTR of course differs according to the language in question, according to the differing morphological properties.

**Anonymisation and markup** A particularity of the ‘speech’ domain is the presence of placeholders for anonymised elements and markup (in the form of tags). For example, there are 35 placeholders surrounded either by square or rounded brackets to indicate different people, organisations and projects (e.g. (PERSON1), [PERSON9], [ORGANIZATION4], [PROJECT8], etc.). The ‘person’ tags are used both in-text to replace the names of people and at the beginning of lines to indicate who is talking. Markup is added to indicate speakers talking at the same time (<parallel_talk>), unintelligible passages (<unintelligible/>), laughter (<laugh/>) and other noise (<other.noise/>).

## 2.5 Test suites

In addition to the test sets of the regular domains, the test sets given to the system participants were augmented with several *test suites*, i.e. custom-made test sets focusing on particular aspects of

<footnote n="14">¹⁴The TTR is the ratio of unique tokens to total tokens, and it is higher the diverse the vocabulary of a text is. It is dependent on the morphological complexity of a language, but can also vary due to other factors.</footnote>

<footnote n="15">¹⁵Texts are tokenised using the language-specific Spacy models (Honnibal and Montani, 2017) where available. For Hebrew, we took the multilingual Spacy model, since a language-specific one was not available.</footnote>
MT translation. The test suites were contributed and evaluated by test suite providers as part of a decentralized sub-task, which will be detailed in Section 6.

## 3 Training Data

Similar to the previous years, we provide a selection of parallel and monolingual corpora for model training. The provenance and statistics of the selected parallel datasets are provided in Appendix in Table 9 and Table 10. Specifically, our parallel data selection include large multilingual corpora such as Europarl-v10 (Koehn, 2005), Paracrawl-v9 (Bañón et al., 2020), CommonCrawl, NewsCommentary-v18, WikiTitles-v3, WikiMatrix (Schwenk et al., 2021), TildeCorpus (Rozis and Skadinš, 2017), OPUS (Tiedemann, 2012), UN Parallel Corpus (Ziemski et al., 2016), and language-specific corpora such as CzEng-v2.0 (Kocmi et al., 2020), YandexCorpus,<fnref n="16" /> ELRC EU Acts, JParaCrawl (Morishita et al., 2020), Japanese-English Subtitle Corpus (Pryzant et al., 2018), KFTT (Neubig, 2011), TED (Cettolo et al., 2012), CCMT, and back-translated news. Links for downloading these datasets were provided on the task web page;¹⁷ in addition, we automated the data preparation pipeline using MTDATA (Gowda et al., 2021).¹⁸ MTDATA downloads all the mentioned datasets, except CCMT and CzEng-v2.0, which required user authentication. This year's monolingual data include the following: News Crawl, News Discussions, News Commentary, CommonCrawl, Europarl-v10 (Koehn, 2005), Extended CommonCrawl (Conneau et al., 2020), Leipzig Corpora (Goldhahn et al., 2012), UberText and Legal Ukrainian.

## 4 System submissions

This year, we received a total of 72 primary submissions from 17 participants. In addition, we collected translations from online MT systems across all language pairs. Online system outputs come from 6 public MT services and were anonymized as ONLINE-{A,B,G,M,W,Y}, which added additional 77 system outputs. The participating systems are listed in Table 2 and detailed in the rest of this section.

Finally, we added translations by three contrastive systems. Two of them are based on the NLLB translation model (NLLB Team et al., 2022) modified by (Freitag et al., 2023) to have a suboptimal performance, using (i) greedy search (NLLB_Greedy) and (ii) following minimum Bayes risk decoding (MBR) optimizing the BLEU metric (NLLB_MBR_BLEU). Neither of them is the official (and better performing) NLLB model. The third contrastive translation is produced by the large language model GPT4 using 5-shot prompting with fixed random translation examples, using the exact prompt by Hendy et al. (2023) together with their predefined few-shot examples. For languages not evaluated in their study, we took examples from the last WMT test sets.

Appendix E provides details of the submitted systems if the authors provided such details.

### 4.1 Constrained and unconstrained tracks

For presentation of the results, systems are treated as either constrained or unconstrained. A system is classified as constrained if the authors reported training only on the provided data and adhering to the rules describing the use of publicly available pre-trained models. The constrained track imposes restrictions on training data, metrics, and pretrained models, while the unconstrained track provides unrestrained flexibility.

The constrained track limitations are mainly around the training and testing data, together with the limitation on pretrained models:

* **Training data:** Only data specified for the current year are permissible, see Section 3. Multilingual systems can be used as long as they only use WMT23 data.

* **Metrics:** The training pipeline can use pretrained metrics evaluated in previous WMT Metrics shared tasks, e.g., COMET (Rei et al., 2022), Bleurt (Yan et al., 2023).

* **Pretrained models:** only the following list of models is allowed together with all their public sizes: mBART (Liu et al., 2020), BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), XLM-RoBERTa (Conneau et al., 2020), sBERT (Reimers and Gurevych, 2019), and LaBSE (Feng et al., 2022).

* **Linguistic tools:** Basic tools like taggers, parsers, and morphology analyzers are allowed.

<footnote n="16">¹⁶https://github.com/mashashma/WMT2022-data
¹⁷https://statmt.org/wmt23/translation-task.html
¹⁸http://www2.statmt.org/wmt23/mtdata</footnote>
<table><caption>**Table 2:** Participants in the General MT shared task. Online system translations were not submitted by their respective companies but were obtained by us, and are therefore anonymized in a fashion consistent with previous editions of the task.</caption><thead><tr><th>Submission Name</th><th>Language Pairs</th><th>System Description</th></tr></thead><tbody><tr><td>AIRC</td><td>de-en, en-ja, ja-en, en-de</td><td>(Rikters and Miwa, 2023)</td></tr><tr><td>ANVITA</td><td>ja-en, zh-en, en-ja, en-zh</td><td>(no associated paper)</td></tr><tr><td>CUNI-DOCTRANSFORMER</td><td>en-cs</td><td>(Popel, 2020)</td></tr><tr><td>CUNI-GA</td><td>en-cs, cs-uk</td><td>(Jon et al., 2023)</td></tr><tr><td>CUNI-TRANSFORMER</td><td>en-cs, cs-uk</td><td>(Popel, 2020)</td></tr><tr><td>GPT4-5SHOT</td><td><strong>All language pairs</strong></td><td>(Hendy et al., 2023)</td></tr><tr><td>GTCOM</td><td>de-en, ja-en, he-en, en-cs, en-he, cs-uk, en-uk, uk-en</td><td>(Zong, 2023)</td></tr><tr><td>HW-TSC</td><td>de-en, en-zh, zh-en</td><td>(Wu et al., 2023b)</td></tr><tr><td>IOL-RESEARCH</td><td>zh-en, en-zh</td><td>(Zhang, 2023)</td></tr><tr><td>TEAMKYB</td><td>ja-en, en-ja</td><td>(LI et al., 2023)</td></tr><tr><td>LAN-BRIDGE MT</td><td><strong>All language pairs</strong></td><td>(Wu and Hu, 2023)</td></tr><tr><td>MUNI-NLP</td><td>cs-uk</td><td>(Rychlý and Teslia, 2023)</td></tr><tr><td>NAIST-NICT</td><td>en-ja, ja-en</td><td>(Deguchi et al., 2023)</td></tr><tr><td>NLLB_GREEDY</td><td><strong>All language pairs</strong></td><td>(Freitag et al., 2023)</td></tr><tr><td>NLLB_MBR_BLEU</td><td><strong>All language pairs</strong></td><td>(Freitag et al., 2023)</td></tr><tr><td>ONLINE-A</td><td><strong>All language pairs</strong></td><td>-</td></tr><tr><td>ONLINE-B</td><td><strong>All language pairs</strong></td><td>-</td></tr><tr><td>ONLINE-G</td><td><strong>All language pairs</strong></td><td>-</td></tr><tr><td>ONLINE-M</td><td>en-ru, zh-en, en-zh, de-en, en-cs, ja-en, en-de, en-ja, ru-en</td><td>-</td></tr><tr><td>ONLINE-W</td><td>en-uk, ja-en, de-en, en-ja, ru-en, en-de, uk-en, en-ru, zh-en, en-cs, en-zh, cs-uk</td><td>-</td></tr><tr><td>ONLINE-Y</td><td><strong>All language pairs</strong></td><td>-</td></tr><tr><td>PROMT</td><td>en-ru, ru-en</td><td>(Molchanov and Kovalenko, 2023)</td></tr><tr><td>SRPH</td><td>he-en, en-he</td><td>(Cruz, 2023)</td></tr><tr><td>SKIM</td><td>en-ja, ja-en</td><td>(Kudo et al., 2023)</td></tr><tr><td>UPCITE-CLILLF</td><td>fr-en, en-fr</td><td>(no associated paper)</td></tr><tr><td>UVA-LTL</td><td>he-en, en-he</td><td>(Wu et al., 2023a)</td></tr><tr><td>YISHU</td><td>zh-en, en-zh</td><td>(Min et al., 2023)</td></tr><tr><td>LANGUAGE X</td><td>en-zh, en-uk, ru-en, uk-en, en-de, he-en, ja-en, zh-en, en-he, de-en, en-cs, en-ja, en-ru</td><td>(Zeng, 2023)</td></tr></tbody></table>
The online systems and contrastive systems are treated as unconstrained during the automatic and human evaluation.

## 4.2 OCELoT

We used the open-source OCELoT platform<fnref n="19" /> to collect system submissions again this year. The platform provides anonymized public leader-boards²⁰ and was also used for two other WMT23 shared tasks: Biomedical (Neves et al., 2023) and Sign Language Translation (Müller et al., 2023). As in previous years, only registered and verified teams with correct contact information were allowed to submit their system outputs and each verified team was limited to 7 submissions per test set. Submissions on leaderboards with BLEU (Papineni et al., 2002) and CHRF (Popović, 2015) scores from SacreBLEU (Post, 2018) were displayed anonymously to avoid publishing rankings based on automatic scores during the submission period. Until one week after the submission period, teams could select a single primary submission per test set, specify if the primary submission followed a constrained or unconstrained setting, and submit a system description paper abstract. These were mandatory for a system submission to be included in the human evaluation campaign.

# 5 Human Evaluation

Human evaluation for all language translation directions is performed with source-based (“bilingual”) Direct Assessment (DA, Graham et al., 2013) of individual segments in document context with Scalar Quality Metrics (SQM) guidelines, mostly following the setup established at WMT22 (DA+SQM, Kocmi et al., 2022). DA+SQM asks the annotators to provide a score between 0 and 100 on a sliding scale, but the slider is presented with seven labelled tick marks, as demonstrated in Figure 1.

Two different annotation platforms and four distinct pools of annotators (Table 3) are used for annotation of different language pairs. We use the open-source framework Appraise (Federmann, 2018) for the evaluation of English→Czech, English↔{Chinese, German, Japanese}, and Czech→Ukrainian. Toloka AI²¹ hosts the evaluation of English↔{Hebrew, Russian, Ukrainian} using their own implementation of the source-based

document-level DA+SQM task, which is as close as possible to the Appraise user interface.

We keep the selection process of documents for annotation mostly the same as in the previous year. The only change made in order to align closer with the MQM-based evaluation run at the Metrics shared task (Freitag et al., 2023) is to present the first 10 segments from a document instead of random 10 consecutive segments.

We again collect both segment-level scores and document-level scores, but compute rankings based on segment scores only.

## 5.1 Human annotators

Annotations for different language pairs are provided by four different parties with their pool of annotators of distinct profiles as presented in Table 3. We shift towards more professional or semi-professional annotators’ pools and decide not to use MTurk annotations as in past years for reference-based DA evaluation for into-English language directions.

Assessments for English↔{Chinese, German, Japanese} are provided by Microsoft and their pool of bilingual target-language native speakers, professional translators or linguists, highly experienced in MT evaluation. Microsoft monitors the annotators’ performance over time and permanently removes from the pool those who fail quality control, which increases the overall quality of the human assessment.

Charles University provides annotators for language pairs involving the Czech language, i.e., English→Czech and Czech→Ukrainian. Their annotators are linguists, translators, researchers and students who are native speakers of the target language with high proficiency in the source language.

DA scores for English↔{Hebrew, Russian, Ukrainian} are collected by Toloka AI using their paid crowd of bilingual target-language native speakers. Toloka AI tests proficiency of their annotator crowd across different NLP annotation tasks and allowed only annotators who deemed reliable according to their quality control measures.

## 5.2 Document selection and quality control

The document selection process remains the same as in the previous year with minor changes. We first randomly sample a subset of document snippets from each of the domains for annotations, sampling the domains with approximately the same number of segments per domain. This ensures that

<footnote n="19">¹⁹https://github.com/AppraiseDev/OCELoT
²⁰https://ocelot-wmt23.mteval.org
²¹https://toloka.ai</footnote>
<figure><img src="image_3.png" /></figure>

(a) Top part of the screen with segment-level scoring.

(b) Bottom part of the screen with document-level scoring.

<caption>**Figure 1:** Screenshot of the document-level DA+SQM configuration in the Appraise interface for an example assessment from the human evaluation campaign for out of English language pairs. The annotator is presented with the entire translated document snippet randomly selected from competing systems (anonymized) with additional static contexts, and is asked to rate the translation of individual segments and then the entire document on sliding scales between 0 and 100.</caption>

all systems in the given language pairs are evaluated on the same subset of the test set, allowing fair comparison between them. As in previous years, we aim to collect approximately 1,500 assessments per system per language pair. Due to concerns about having sufficient annotations, we create two batches of HITs, each providing half of the required assessments, such that at least all segments in the first batch could be covered for all systems, with the second campaign completed if possible.

For HIT generation for English↔German, which feature paragraph-level test sets (documents consist of paragraphs instead of sentences), we simply consider a whole paragraph as a “segment”, collecting paragraph-level assessments. In that regard, we collect fewer DA scores per system comparing to other language pairs, but the human evaluation covers a larger subset of the test sets.

Last year, we used snippets of at most 10 randomly selected consecutive segments from a document as “documents” for document-level annotation. This year, we use 10 first segments from a document instead, in order to align with the MQM-based evaluation used at the Metrics shared task

(Freitag et al., 2023).

All HITs consist of exactly 100 segments and are generated as in the past:

1. Snippet-system pairs are randomly sampled (from the restricted set of pre-sampled snippets) to create up to 80 segments;

2. Random snippets for the remaining 20 (or more) segments are duplicated from the first 80 to serve as quality control items;

3. BAD references are introduced to the random segments in the duplicated snippets to have about 12-14% of quality control segments per HIT.

BAD translations are created by replacing an embedded sequences of tokens in the segment with a random phrase of the same length from a different reference segment.<fnref n="22" />

We perform quality control by measuring an annotator's ability to reliably score BAD translations

<footnote n="22">²²For full details, see the HIT and batch generation code: https://github.com/wmt-conference/wmt23-news-systems</footnote>
<table><caption>**Table 3:** Annotators' profiles and annotation tools for each language pair in human evaluation.</caption><thead><tr><th>Language pairs</th><th>Annotators' profile</th><th>Tool</th></tr></thead><tbody><tr><td>English↔Chinese/German/Japanese</td><td>Microsoft annotators: bilingual target-language native speakers, professional translators or linguists, experienced in MT evaluation</td><td>Appraise</td></tr><tr><td>Czech→Ukrainian</td><td>Paid translators and target-language native speakers</td><td>Appraise</td></tr><tr><td>English→Czech</td><td>Czech paid linguists, annotators, researchers, students with high proficiency in English</td><td>Appraise</td></tr><tr><td>English↔Hebrew/Russian/Ukrainian</td><td>Toloka AI paid crowd: bilingual target-language native speakers high-performing in other task types</td><td>Toloka.ai</td></tr></tbody></table>

<table><caption>**Table 4:** Amount of segments evaluated in the WMT23 manual evaluation campaign; including human references as systems; after excluding quality control items and document-level scores.</caption><thead><tr><th>Language Pair</th><th>Sys.</th><th>Assess.</th><th>Assess/Sys</th></tr></thead><tbody><tr><td>Chinese→English</td><td>16</td><td>20,535</td><td>1283.4</td></tr><tr><td>Czech→Ukrainian</td><td>14</td><td>23,191</td><td>1656.5</td></tr><tr><td>German→English</td><td>14</td><td>13,573</td><td>969.5</td></tr><tr><td>English→Chinese</td><td>16</td><td>24,551</td><td>1534.4</td></tr><tr><td>English→Czech</td><td>16</td><td>25,527</td><td>1595.4</td></tr><tr><td>English→German</td><td>13</td><td>14,267</td><td>1097.5</td></tr><tr><td>English→Japanese</td><td>17</td><td>26,115</td><td>1536.2</td></tr><tr><td>Japanese→English</td><td>18</td><td>27,858</td><td>1547.7</td></tr></tbody></table>

<table><caption>**Table 5:** Numbers of individual annotators taking part in the WMT23 human evaluation campaign and the average number of HITs collected per annotator.</caption><thead><tr><th>Language Pair</th><th>Ann.</th><th>HITs</th><th>HITs/Ann.</th></tr></thead><tbody><tr><td>Chinese→English</td><td>13</td><td>128</td><td>9.8</td></tr><tr><td>Czech→Ukrainian</td><td>9</td><td>146</td><td>16.2</td></tr><tr><td>German→English</td><td>21</td><td>82</td><td>3.9</td></tr><tr><td>English→Czech</td><td>36</td><td>162</td><td>4.5</td></tr><tr><td>English→German</td><td>22</td><td>87</td><td>4.0</td></tr><tr><td>English→Japanese</td><td>21</td><td>164</td><td>7.8</td></tr><tr><td>English→Chinese</td><td>13</td><td>154</td><td>11.8</td></tr><tr><td>Japanese→English</td><td>20</td><td>174</td><td>8.7</td></tr></tbody></table>

significantly lower than corresponding original system outputs using a paired significance test with $p < 0.05$. We pair two HITs into a single annotation task with about 24-28 quality control segments to ensure a sufficient sample size for the statistical test. In campaigns hosted on Appraise, if an annotator is not able to demonstrate reliability on BAD references, they are excluded from further annotations, the HITs are reset and annotated from scratch by another annotator if possible.

The total number of assessments collected for each language pair and the average number of assessments per system in WMT23 manual evaluation are presented in Table 4.

## 5.3 Calibration HITs

Last year we introduced calibration HITs, which this year we collect for all language pairs. A calibration HIT is a HIT with 100 randomly selected segments, which is identical for and completed by all annotators, in addition to their regular annotation HITs. We release these alongside the other annotations and the anonymized mapping between annotators and HITs in order to enable additional analysis. With a small set of sentences annotated by all annotators, we are better able to examine questions about inter-annotator consistency and provide data for future research in this area.

Table 5 shows the number of unique annotators per language pair along with the total number of HITs and average number of HITs per annotator. We leave more detailed analysis of collected calibration data to future work.

## 5.4 Human ranking computation

The official rankings shown in Table 6 are generated on the basis of the segment-level raw DA+SQM scores that are collected within document context for all language pairs.<fnref n="23" /> Whole documents with at least one quality control segment (i.e., BAD references) and HITs that failed to pass quality control are removed prior to computing the rankings.<fnref n="24" />

In this year's evaluation, we have chosen not to normalize scores by discontinuing the use of z-scores, given their potential to exacerbate system comparisons (Knowles, 2021). While utilizing raw scores is not flawless—considering each annotator employs distinct annotation strategies — we have sought to counteract this by distributing

<footnote n="23">²³The code used to generate the rankings in Table 6 can be found here: https://github.com/AppraiseDev/Appraise/blob/main/Campaign/management/commands/ComputeWMT23Results.py</footnote>

<footnote n="24">²⁴Two HITs for Czech→Ukrainian and one HIT for English→Czech.</footnote>
<table><caption>**Table 6:** Official results of WMT23 General Translation Task. Systems ordered by DA score; systems within a cluster are considered tied; lines indicate clusters according to Wilcoxon rank-sum test *p* < 0.05; rank ranges indicate the number of systems a system significantly underperforms or outperforms; grayed entry indicates resources that fall outside the constraints provided. All language pairs used document-level evaluation.</caption><thead><tr><th colspan="3">German→English</th><th colspan="3">Czech→Ukrainian</th><th colspan="3">Japanese→English</th><th colspan="3">English→Japanese</th></tr><tr><th>Rank</th><th>Ave.</th><th>System</th><th>Rank</th><th>Ave.</th><th>System</th><th>Rank</th><th>Ave.</th><th>System</th><th>Rank</th><th>Ave.</th><th>System</th></tr></thead><tbody><tr><td>1-3</td><td>90.3</td><td>GPT4-5shot</td><td>1-3</td><td>83.7</td><td>ONLINE-B</td><td>1-3</td><td>83.6</td><td>GPT4-5shot</td><td>1-3</td><td>81.3</td><td>GPT4-5shot</td></tr><tr><td>1-3</td><td>89.9</td><td>Human-refA</td><td>1-3</td><td>83.2</td><td>Human-refA</td><td>4-8</td><td>82.8</td><td>ONLINE-W</td><td>2-4</td><td>80.6</td><td>SKIM</td></tr><tr><td>1-5</td><td>89.6</td><td>ONLINE-A</td><td>4-8</td><td>82.4</td><td>CUNI-GA</td><td>4-8</td><td>81.8</td><td>CUNI-Transformer</td><td>3-8</td><td>80.4</td><td>Human-refA</td></tr><tr><td>3-6</td><td>89.1</td><td>ONLINE-B</td><td>4-8</td><td>81.3</td><td>GTCOM_DLUT</td><td>4-8</td><td>80.6</td><td>ONLINE-A</td><td>3-8</td><td>79.5</td><td>ONLINE-Y</td></tr><tr><td>3-6</td><td>88.8</td><td>ONLINE-W</td><td>9-11</td><td>79.5</td><td>ONLINE-G</td><td>9-13</td><td>78.7</td><td>ONLINE-Y</td><td>2-8</td><td>79.4</td><td>ONLINE-B</td></tr><tr><td>4-7</td><td>88.0</td><td>ONLINE-Y</td><td>9-13</td><td>78.7</td><td>MUNI-NLP</td><td>10-13</td><td>77.4</td><td>Lan-BridgeMT</td><td>3-9</td><td>79.2</td><td>ONLINE-A</td></tr><tr><td>6-8</td><td>87.7</td><td>ONLINE-G</td><td>10-13</td><td>76.9</td><td>NLLB_MBR_BLEU</td><td>14</td><td>76.7</td><td>NLLB_Greedy</td><td>2-8</td><td>78.8</td><td>ONLINE-W</td></tr><tr><td>8-9</td><td>86.5</td><td>GTCOM_DLUT</td><td></td><td></td><td></td><td></td><td></td><td></td><td>3-8</td><td>78.4</td><td>NAIST-NICT</td></tr><tr><td>7-9</td><td>85.3</td><td>ONLINE-M</td><td></td><td></td><td></td><td></td><td></td><td></td><td>8-9</td><td>76.9</td><td>GTCOM_DLUT</td></tr><tr><td>10-11</td><td>81.8</td><td>LanguageX</td><td></td><td></td><td></td><td></td><td></td><td></td><td>10-13</td><td>76.4</td><td>Lan-BridgeMT</td></tr><tr><td>10-13</td><td>80.0</td><td>Lan-BridgeMT</td><td></td><td></td><td></td><td></td><td></td><td></td><td>10-13</td><td>75.8</td><td>ANVITA</td></tr><tr><td>11-14</td><td>79.6</td><td>NLLB_MBR_BLEU</td><td></td><td></td><td></td><td></td><td></td><td></td><td>10-13</td><td>74.8</td><td>ONLINE-G</td></tr><tr><td>12-14</td><td>78.8</td><td>AIRC</td><td></td><td></td><td></td><td></td><td></td><td></td><td>10-13</td><td>74.6</td><td>LanguageX</td></tr><tr><td>11-14</td><td>77.9</td><td>NLLB_Greedy</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><th colspan="3">English→German</th><th colspan="3">Chinese→English</th><th colspan="3">English→Japanese</th><th colspan="3">English→Japanese</th></tr><tr><td>1-5</td><td>89.0</td><td>GPT4-5shot</td><td>1-2</td><td>82.9</td><td>Lan-BridgeMT</td><td>1-2</td><td>80.9</td><td>GPT4-5shot</td><td>1-2</td><td>80.7</td><td>Human-refA</td></tr><tr><td>1-5</td><td>88.8</td><td>ONLINE-B</td><td>3-8</td><td>80.3</td><td>Yishu</td><td>3-7</td><td>80.2</td><td>ONLINE-W</td><td>2-6</td><td>79.5</td><td>GPT4-5shot</td></tr><tr><td>1-4</td><td>88.3</td><td>ONLINE-W</td><td>5-10</td><td>80.0</td><td>ONLINE-G</td><td>14-15</td><td>72.9</td><td>ONLINE-M</td><td>1-5</td><td>78.8</td><td>ONLINE-B</td></tr><tr><td>2-6</td><td>88.1</td><td>ONLINE-A</td><td>3-7</td><td>79.8</td><td>ONLINE-B</td><td>14-15</td><td>72.4</td><td>KYB</td><td>2-6</td><td>78.6</td><td>ONLINE-Y</td></tr><tr><td>4-6</td><td>88.0</td><td>ONLINE-Y</td><td>4-9</td><td>79.7</td><td>ONLINE-Y</td><td>16</td><td>68.9</td><td>AIRC</td><td>2-5</td><td>78.5</td><td>SKIM</td></tr><tr><td>1-6</td><td>87.7</td><td>Human-refA</td><td>3-8</td><td>79.1</td><td>HW-TSC</td><td>17-18</td><td>66.7</td><td>NLLB_MBR_BLEU</td><td>4-6</td><td>78.4</td><td>ONLINE-W</td></tr><tr><td>7-8</td><td>86.7</td><td>ONLINE-M</td><td>6-10</td><td>77.8</td><td>ONLINE-A</td><td>17-18</td><td>66.1</td><td>NLLB_Greedy</td><td>7-10</td><td>76.6</td><td>LanguageX</td></tr><tr><td>7-8</td><td>85.5</td><td>ONLINE-G</td><td>10-11</td><td>77.7</td><td>IOL_Research</td><td></td><td></td><td></td><td>7-10</td><td>76.2</td><td>ONLINE-A</td></tr><tr><td>9</td><td>84.0</td><td>Lan-BridgeMT</td><td>8-11</td><td>77.2</td><td>LanguageX</td><td></td><td></td><td></td><td>7-10</td><td>76.1</td><td>NAIST-NICT</td></tr><tr><td>10</td><td>82.7</td><td>LanguageX</td><td>12-13</td><td>76.9</td><td>ONLINE-M</td><td></td><td></td><td></td><td>7-10</td><td>75.2</td><td>Lan-BridgeMT</td></tr><tr><td>11-12</td><td>76.8</td><td>NLLB_MBR_BLEU</td><td>13-16</td><td>76.2</td><td>NLLB_MBR_BLEU</td><td></td><td></td><td></td><td>11-12</td><td>73.1</td><td>ANVITA</td></tr><tr><td>11-12</td><td>75.7</td><td>NLLB_Greedy</td><td>12-15</td><td>76.1</td><td>Human-refA</td><td></td><td></td><td></td><td>11-12</td><td>72.6</td><td>ONLINE-M</td></tr><tr><td>13</td><td>73.6</td><td>AIRC</td><td>14-16</td><td>74.0</td><td>NLLB_Greedy</td><td></td><td></td><td></td><td>13-15</td><td>70.8</td><td>KYB</td></tr><tr><td></td><td></td><td></td><td>13-16</td><td>72.6</td><td>ANVITA</td><td></td><td></td><td></td><td>13-15</td><td>69.6</td><td>AIRC</td></tr><tr><th colspan="3">English→Czech</th><th colspan="3">English→Chinese</th><th colspan="3">English→Japanese</th><th colspan="3">English→Japanese</th></tr><tr><td>1</td><td>85.4</td><td>Human-refA</td><td>1-5</td><td>82.2</td><td>Yishu</td><td>1-5</td><td>82.1</td><td>Human-refA</td><td>1-5</td><td>81.3</td><td>Human-refA</td></tr><tr><td>2</td><td>84.1</td><td>ONLINE-W</td><td>1-5</td><td>82.1</td><td>Human-refA</td><td>1-7</td><td>82.1</td><td>GPT4-5shot</td><td>3-8</td><td>82.0</td><td>Lan-BridgeMT</td></tr><tr><td>3-5</td><td>81.8</td><td>GPT4-5shot</td><td>1-6</td><td>81.8</td><td>ONLINE-B</td><td>4-8</td><td>81.4</td><td>ONLINE-W</td><td>1-8</td><td>81.5</td><td>HW-TSC</td></tr><tr><td>3-4</td><td>80.4</td><td>CUNI-GA</td><td>1-8</td><td>81.5</td><td>HW-TSC</td><td>5-8</td><td>80.2</td><td>ONLINE-Y</td><td>4-8</td><td>81.4</td><td>ONLINE-W</td></tr><tr><td>5-8</td><td>80.3</td><td>ONLINE-A</td><td></td><td></td><td></td><td></td><td></td><td></td><td>13-15</td><td>69.6</td><td>ONLINE-G</td></tr><tr><td>5-8</td><td>79.4</td><td>CUNI-DocTransformer</td><td></td><td></td><td></td><td></td><td></td><td></td><td>16</td><td>64.5</td><td>NLLB_Greedy</td></tr><tr><td>4-7</td><td>78.8</td><td>ONLINE-B</td><td></td><td></td><td></td><td></td><td></td><td></td><td>17</td><td>61.3</td><td>NLLB_MBR_BLEU</td></tr><tr><td>8-14</td><td>78.6</td><td>NLLB_MBR_BLEU</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>6-11</td><td>78.4</td><td>GTCOM_DLUT</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>8-12</td><td>77.4</td><td>CUNI-Transformer</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>10-14</td><td>76.8</td><td>NLLB_Greedy</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>9-14</td><td>75.7</td><td>ONLINE-M</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>10-15</td><td>75.2</td><td>ONLINE-G</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>13-15</td><td>75.0</td><td>ONLINE-Y</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>8-15</td><td>75.0</td><td>Lan-BridgeMT</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>16</td><td>74.1</td><td>LanguageX</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>
systems evenly across annotators. This approach
aims to minimize the potential bias of a particularly
stringent annotator disproportionately penalizing a
single system. Ideally, every annotator would as-
sess documents translated by all systems; however,
this could introduce task repetitiveness concerns.
For future considerations, employing calibration
HITS (see Section 5.3) to normalize each annota-
tor's behaviour could offer a promising solution.

All segment-level scores are averaged per system to compute the system-level scores. The clusters are computed using the Wilcoxon rank-sum test with *p* < 0.05. Rank ranges indicate the number of systems a particular system underperforms or outperforms: the top end of the rank range is *l* + 1 where *l* is the number of losses, while the bottom is *n* − *w* where *n* is the total number of systems and *w* is the number of systems that the system in questions significantly wins against.

Tables with head-to-head comparisons between
all systems are included in Appendix G.

At the time of preparation of the camera-ready
version of the paper, we have not been able to col-
lect the required number of high-quality assess-
ments for language pairs run through Toloka AI
that would meet WMT standards for human eval-
uation. In that regard, we decided not to publish
official rankings based on manual evaluation for
English↔{Hebrew, Russian, Ukrainian} until the
conference, we are planning to address it later.

**5.5 Comparison of human evaluation methods**

In collaboration with the metrics shared task (Fre-
itag et al., 2023), human annotation data for the
Chinese→English and English→German direction
was collected using two different approaches: the
source-based DA+SQM approach, and the Multi-
dimensional Quality Metrics (MQM) framework
(Freitag et al., 2021). We present the rankings pro-
duced by the two approaches in Table 7.

Upon examining the system rankings and in-
dividual clusters produced by both techniques, it
is evident that DA+SQM produces fewer clusters.
This suggests that it might not be sufficiently robust
to differentiate smaller system differences, whereas
MQM creates more detailed clusters. One potential
explanation is that DA+SQM, constrained by bud-
getary restrictions, might be under-powered. As
highlighted by Wei et al. (2022), the 1500 segments
we gather per system might not suffice to segregate
systems in a more detailed manner.

Conversely, the largest difference in the evalua-
tion techniques is the cost. While MQM manages
to establish more refined clusters, its deployment is
significantly more costly and complex, especially
when training professionals. An interesting ques-
tion would be determining the number of MQM
labels that could be procured within the budget
allocated for DA+SQM.

It is also important to note that the set of data
over which each of these rankings was produced
may have differed slightly due to the sampling (e.g.,
the distribution over topic domains or the amount
of coverage of the full test set), making it difficult
to determine whether these differences in rankings
represent differences due to data or due to different
annotation methods.

6 Test Suites

As can be seen in the general MT task, the improve-
ment of translation quality has made it difficult to
discriminate MT output from human translation
with the current evaluation methods. Nevertheless,
there are still cases where MT has difficulties, de-
livering outputs which despite seeming fluent and
being surrounded by other seemingly perfect trans-
lations, entail serious flaws. In general evaluation
methods, such flaws can get “hidden in the aver-
age” or simply get missed altogether. In an effort to
shed light to these cases, evaluation via test suites
is embedded in the shared task.

6.1 Setup of the sub-task

Test suites are custom extensions to standard test
sets, constructed so that they can focus on particular
aspects of the MT output. Here, the evaluation
of the MT outputs takes place in a decentralized
manner as a part of a sub-task, where test suite
providers were invited to submit their customized
test sets, following the setting introduced at the
Third Conference on Machine Translation (Bojar
et al., 2018).

Every test suite provider submitted a source-side
test set, which the shared task organizers appended
to the standard test sets of the shared task. The
corresponding outputs from the MT systems of the
shared task were returned to the test suite providers,
who were responsible for running the evaluation,
based on their own custom evaluation methods.
The results of each test suite evaluation, together
with the relevant analysis, appear in separate de-
scription papers.
<table><thead><tr><th>Rank</th><th>Ave. ↑</th><th>System (En-De)</th></tr></thead><tbody><tr><td>1-5</td><td>89.0</td><td>GPT4-5shot</td></tr><tr><td>1-5</td><td>88.8</td><td>ONLINE-B</td></tr><tr><td>1-4</td><td>88.3</td><td>ONLINE-W</td></tr><tr><td>2-6</td><td>88.1</td><td>ONLINE-A</td></tr><tr><td>4-6</td><td>88.0</td><td>ONLINE-Y</td></tr><tr><td>1-6</td><td>87.7</td><td>Human-refA</td></tr><tr><td>7-8</td><td>86.7</td><td>ONLINE-M</td></tr><tr><td>7-8</td><td>85.5</td><td>ONLINE-G</td></tr><tr><td>9</td><td>84.0</td><td>Lan-BridgeMT</td></tr><tr><td>10</td><td>82.7</td><td>LanguageX</td></tr><tr><td>11-12</td><td>76.8</td><td>NLLB_MBR_BLEU</td></tr><tr><td>11-12</td><td>75.7</td><td>NLLB_Greedy</td></tr><tr><td>13</td><td>73.6</td><td>AIRC</td></tr></tbody></table>

<table><thead><tr><th>Rank</th><th>Ave. ↑</th><th>System (Zh-En)</th></tr></thead><tbody><tr><td>1-2</td><td>82.9</td><td>Lan-BridgeMT</td></tr><tr><td>1-2</td><td>80.9</td><td>GPT4-5shot</td></tr><tr><td>3-8</td><td>80.3</td><td>Yishu</td></tr><tr><td>3-7</td><td>80.2</td><td>ONLINE-W</td></tr><tr><td>5-10</td><td>80.0</td><td>ONLINE-G</td></tr><tr><td>3-7</td><td>79.8</td><td>ONLINE-B</td></tr><tr><td>4-9</td><td>79.7</td><td>ONLINE-Y</td></tr><tr><td>3-8</td><td>79.1</td><td>HW-TSC</td></tr><tr><td>6-10</td><td>77.8</td><td>ONLINE-A</td></tr><tr><td>10-11</td><td>77.7</td><td>IOL_Research</td></tr><tr><td>8-11</td><td>77.2</td><td>LanguageX</td></tr><tr><td>12-13</td><td>76.9</td><td>ONLINE-M</td></tr><tr><td>13-16</td><td>76.2</td><td>NLLB_MBR_BLEU</td></tr><tr><td>12-15</td><td>76.1</td><td>Human-refA</td></tr><tr><td>14-16</td><td>74.0</td><td>NLLB_Greedy</td></tr><tr><td>13-16</td><td>72.6</td><td>ANVITA</td></tr></tbody></table>

<table><thead><tr><th>System (En-De)</th><th>MQM ↓</th></tr></thead><tbody><tr><td>refA</td><td>2.96</td></tr><tr><td>GPT4-5shot</td><td>3.72</td></tr><tr><td>ONLINE-W</td><td>3.95</td></tr><tr><td>ONLINE-B</td><td>4.71</td></tr><tr><td>ONLINE-Y</td><td>5.64</td></tr><tr><td>ONLINE-A</td><td>5.67</td></tr><tr><td>ONLINE-G</td><td>6.57</td></tr><tr><td>ONLINE-M</td><td>6.94</td></tr><tr><td>Lan-BridgeMT</td><td>8.67</td></tr><tr><td>LanguageX</td><td>9.25</td></tr><tr><td>NLLB_Greedy</td><td>9.54</td></tr><tr><td>NLLB_MBR_BLEU</td><td>10.79</td></tr><tr><td>AIRC</td><td>14.23</td></tr></tbody></table>

<table><caption>**Table 7:** Comparison of system clustering as done by DA+SQM and MQM technique. Top two tables are for English to German, while bottom two are for Chinese to German.</caption><thead><tr><th>System (Zh-En)</th><th>MQM ↓</th></tr></thead><tbody><tr><td>Lan-BridgeMT</td><td>2.10</td></tr><tr><td>GPT4-5shot</td><td>2.31</td></tr><tr><td>Yishu</td><td>3.23</td></tr><tr><td>ONLINE-B</td><td>3.39</td></tr><tr><td>HW-TSC</td><td>3.40</td></tr><tr><td>ONLINE-A</td><td>3.79</td></tr><tr><td>ONLINE-Y</td><td>3.79</td></tr><tr><td>ONLINE-G</td><td>3.86</td></tr><tr><td>ONLINE-W</td><td>4.06</td></tr><tr><td>LanguageX</td><td>4.23</td></tr><tr><td>IOL_Research</td><td>4.59</td></tr><tr><td>refA</td><td>4.83</td></tr><tr><td>ONLINE-M</td><td>5.43</td></tr><tr><td>ANVITA</td><td>6.08</td></tr><tr><td>NLLB_MBR_BLEU</td><td>6.36</td></tr><tr><td>NLLB_Greedy</td><td>6.57</td></tr></tbody></table>

## 6.2 Submissions

The test suite sub-task received 5 submissions with 6 test suites, whose overview can be seen in Table 8. The descriptions of each submission and their main findings are given below.

**DFKI (Manakhimova et al., 2023)** test suite offers a fine-grained linguistically motivated analysis of the shared task MT outputs, based on more than 11,500 manually devised test items, which cover up to 110 phenomena in 14 categories per language direction. Extending their previous test suite efforts (e.g. Avramidis et al., 2018; Macketanz et al., 2022), the submission of this year includes an updated test set featuring new linguistic phenomena and focuses additionally on the participating LLMs. The evaluation spans German→English, English→German, and English→Russian language directions.

Some of the phenomena with the lowest accuracies for German→English are *idioms* and *resultative predicates*. For English→German, these include *mediopassive voice*, and *noun formation(er)*. As for English→Russian, these include *idioms* and

*semantic roles*. GPT4 performs equally or comparably to the best systems in German→English and English→German but falls in the second significance cluster for English→Russian.

**HW-TSC (Chen et al., 2023)** propose a systematic approach to select test sentences with high-level of difficulty from the Wiki Corpus. The strategy considers the difficulty level of a sentence from four dimensions: *word difficulty*, *length difficulty*, *grammar difficulty* and *model learning difficulty*. They open-source two Multifaceted Challenge Sets for Chinese→English and English→Chinese, each of them containing 2,000 sentences. Then, they use these challenge sets to test the shared task systems, presenting results by three automatic metrics.

The resulting system ranks are quite different from the official results. The authors point out that systems that perform well on average test sets may not perform as well on sets with high difficulty. If the ranking difference is caused by domain issues, the top-ranked systems on the official test sets may not be so general. GPT4 is ranked in the first two positions in Chinese→English but its rank in
<table><caption>**Table 8:** Overview of the participating test suites.</caption><thead><tr><th>Test suite</th><th>Directions</th><th>Phenomena</th><th>#Sentences</th><th>Citation</th><th>Link</th></tr></thead><tbody><tr><td>DFKI</td><td>de-en, en-de, en-ru</td><td>110 linguistic phenomena</td><td>11,517</td><td>Manakhimova et al. (2023)</td><td>DFKI-NLP</td></tr><tr><td>HW-TSC</td><td>zh-en, en-zh</td><td>4 difficulty dimensions</td><td>4,000</td><td>Chen et al. (2023)</td><td>HwTsc</td></tr><tr><td>IIIT HYD</td><td>en-de</td><td>5 domains, 5 writing styles</td><td>2,268</td><td>Mukherjee and Shrivastava (2023)</td><td>wmt23</td></tr><tr><td>INES</td><td>en-de</td><td>Inclusive language forms</td><td>162</td><td>Savoldi et al. (2023)</td><td>fbk.eu</td></tr><tr><td>MuST-SHE</td><td>en-de</td><td>Binary gender bias</td><td>200</td><td>Savoldi et al. (2023)</td><td>fbk.eu</td></tr><tr><td>RoCS-MT</td><td>en-de, en-cs, en-uk, en-ru</td><td>Non-standard user-generated content</td><td>1,922</td><td>Bawden and Sagot (2023)</td><td>RoCS-MT</td></tr></tbody></table>

English→Chinese is much lower (ranks 4-9).

## IIIT HYD (Mukherjee and Shrivastava, 2023)

This test suite covers five specific domains (entertainment, environment, health, science, legal) and spans five distinct writing styles (descriptive, judgments, narrative, reporting, technical-writing) for English-German. The authors conduct their analysis through a combination of automated assessments and manual evaluations.

Based on their evaluation, it is evident that both ONLINE-B and ONLINE-Y consistently surpassed other MT systems in performance across a diverse array of writing styles and domains. When focusing on GPT4, whereas it performs comparably to the best systems for most domains and writing styles, it gives considerably worse results when applied to the legal domain, and the writing style of judgments.

### MuST-SHE<sup>WMT23</sup> and INES (Savoldi et al., 2023)
By focusing on the en-de and de-en language pairs, the authors rely on these newly created test suites to investigate systems' ability to translate feminine and masculine gender and produce gender-inclusive translations. Furthermore, they discuss metrics associated with the test suites and validate them by means of human evaluations.

The results indicate that systems achieve reasonable and comparable performance in correctly translating both feminine and masculine gender forms for naturalistic gender phenomena. Instead, the generation of inclusive language forms in translation emerges as a challenging task for all the evaluated MT models, indicating room for future improvements and research on the topic.

Concerning GPT 4, it is noticeable that its overall accuracy is 2% worse than the best MT system, whereas it achieves a relatively low accuracy with regard to the feminine gender, when evaluating whether the first-person singular references to the

speaker are translated according to the speaker's linguistic expression of gender.

### RoCS-MT (Bawden and Sagot, 2023)
The RoCS-MT Challenge Set is designed to test MT systems' robustness to user-generated content (UGC) displaying non-standard characteristics, such as spelling errors, devowelling, acronymisation, etc. It is composed of non-standard English comments from Reddit, manually normalised and professionally translated into four of the WMT 2023 target languages, German, Czech, Ukrainian and Russian, and also French.

Through automatic and manual analysis of system outputs, we find that many of the phenomena remain challenging for most systems, but to varying degrees depending on the phenomenon, the particular instance (notably how frequent the non-standard word is) and the system, especially with respect to the quantity of training data. For example, non-standard instances of words (e.g. through devowelling or through phonetically inspired spelling) are often either omitted in the translation or copied unchanged. When non-standard words are translated, it is often in their standard form, but with some exceptions, for example capitalisation is sometimes preserved. However, there is often inconsistency within a same system's outputs.

GPT4-5shot has a clear lead over all other systems, correctly translating even some of the most challenging examples. It sometimes (although inconsistently) reproduces non-standardness in its outputs, but also does not always remain entirely faithful to the source sentence. However, aside the huge disparity in the amount of training data compared to other systems, notably the constrained ones, the lack of access to its training data is a serious obstacle to any meaningful scientific comparison; we cannot know which phenomena were seen during training and how frequently, and more
crucially, we cannot verify whether RoCS-MT sentences were seen during training.

# 7 Conclusions

The General Machine Translation Task at WMT 2023 covered 14 translation pairs, where the only non-English language pair was Czech→Ukrainian. Source based DA+SQM was the main human golden truth. The evaluation included 72 primary submissions from 17 participants, 6 online systems and 3 additional contrastive systems including GPT4. It was performed by 155 human (semi-)professional annotators, who contributed more than 175,000 judgments altogether. For most language pairs (apart from English→Czech), MT systems produce outputs that cannot be identified as being worse than the manually produced references translations in a statistically significant way, using our current evaluation methods.

It is apparent that this year, the amount of unconstrained submissions are lower thank in past years (27 submissions by 11 participants). Additionally, for some language pairs there are only few submissions by participants, and therefore they are dominated by many online systems, of whom we have no technical descriptions. We are therefore considering ways to encourage participation in the future, whereas redefining the constrained setting may be needed.

It is the first time that Large Language Models (LLMs) are included in the Shared Task as translation systems. Although the technology is very apparent in NLP research, we received only one submission using LLM methods (Lan-BridgeMT), whereas one dominant commercial LLM (GPT4) was included via our own efforts. GPT4 was in the first significance cluster for all systems translating towards English, but fell in the second significance cluster (rank 3-5) for English→Czech, whereas a similar sign was given by one of the test suites for English→Russian (rank 3; Manakhimova et al., 2023). Additionally, test suites providers noted that GPT4 outputs are not always faithful to the source sentence (Bawden and Sagot, 2023) and that they have some issues with speaker gender translation (Savoldi et al., 2023) and specific domains (Mukherjee and Shrivastava, 2023, e.g. legal;). Due to the closed-source nature of commercial tools, it is hard to know the exact reasons for these findings, although they confirm previous observations that GPT models have difficulties with

under-represented languages (Hendy et al., 2023). We believe that a more transparent comparison including open source LLMs should be sought for the future.

# 8 Limitations

We investigated a research question of testing general capabilities of MT systems. However, we have simplified this approach. Firstly, we only used four domains that are not specialized. Secondly, we used only cleaner sentences, avoiding noisy in the source sentences.

Although we accept human judgement as a gold standard, giving us more reliable signal than automatic metrics, we should mention that human annotations are noisy (Wei and Jia, 2021) and their performance is affected by quality of other evaluated systems (Mathur et al., 2020).

Different annotators are using different ranking strategy which may have an effect on the system ranking as we are using raw scores.

# 9 Ethical Consideration

Several of the domains contained texts that included personal data, for example the speech data (See Section 2.4 for more details). Entities were replaced by anonymisation tags (e.g. #NAME#, #EMAIL#) to preserve the anonymity of the users behind the content.

The sentences in Ukrainian datasets were collected with users' opt-in consent, and any personal data related to people other than well-known people was pseudonymized (using random first names and surnames). Sentences where such pseudonymization would not be enough to preserve reasonable anonymity of the users (e.g. describing events uniquely identifying the persons involved) were not included in the test set.

As described in Section 2.2 and in the linguistic brief (Appendix Section B), inappropriate, controversial and/or explicit content was filtered out prior to translation, particularly keeping in mind the translators and not exposing them to such content or obliging them to translate it. A few sentences containing explicit content managed to escape the filter, and we removed these sentences from the test sets without translation.

Human evaluation using Appraise for collecting human judgements was fully anonymous. Automatically generated accounts associated with annotation tasks with single-sign-on URLs were dis-
tributed randomly among pools of annotators and
did not allow for storing personal information. For
language pairs for which we used calibration HITs,
we received lists of tasks completed by an individ-
ual anonymous annotator. Annotators have been
well paid in respect to their countries.

Acknowledgments

This task would not have been possible without the
sponsorship of monolingual data, test sets trans-
lation and evaluation from our partners. Namely
Microsoft, Charles University, Toloka AI, Google,
NTT Resonant, Dubformer, and Centific.

Additionally, we would like to thank Rebecca
Knowles, Sergio Bruccoleri, Mariia Anisimova and
many others who provided help and recommenda-
tions.

Barry Haddow's participation was funded by UK Research and Innovation (UKRI) under the UK government's Horizon Europe funding guarantee [grant number 10039436 – UTTER].

Rachel Bawden's participation was funded by
her chair position in the PRAIRIE institute funded
by the French national agency ANR as part of the
"Investissements d'avenir" programme under the
reference ANR-19-P3IA-0001 and by her Emer-
gence project, DadaNMT, funded by Sorbonne Uni-
versité.

Maja Popović's participation was funded by the ADAPT SFI Centre for Digital Media Technology, funded by Science Foundation Ireland through the SFI Research Centres Programme and co-funded under the European Regional Development Fund (ERDF) through Grant 13/RC/2106.

Martin Popel's participation was funded by
GAČR EXPRO grant LUSyD (GX20-16819X).

Eleftherios Avramidis's participation was funded
by the German Research Foundation (DFG)
through the project TextQ (grant num. MO
1038/31-1, 436813723), and by the German
Federal Ministry of Education and Research
(BMBF) through the project SocialWear (grant
num. 01IW2000).

This work has been using data and tools provided
by the LINDAT/CLARIAH-CZ Research Infras-
tructure (https://lindat.cz), supported by the Min-
istry of Education, Youth and Sports of the Czech
Republic (Project No. LM2023062).

References

Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondřej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. 2021. Findings of the 2021 conference on machine translation (WMT21). In *Proceedings of the Sixth Conference on Machine Translation*, pages 1–88, Online. Association for Computational Linguistics.

Eleftherios Avramidis, Vivien Macketanz, Arle Lommel, and Hans Uszkoreit. 2018. Fine-grained evaluation of quality estimation for machine translation based on a linguistically motivated test suite. In *Proceedings of the AMTA 2018 Workshop on Translation Quality Estimation and Automatic Post-Editing*, pages 243–248, Boston, MA. Association for Machine Translation in the Americas.

Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. 2020. ParaCrawl: Web-scale acquisition of parallel corpora. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 4555–4567. Online. Association for Computational Linguistics.

Loïc Barrault, Magdalena Biesialska, Ondřej Bo-
jar, Marta R. Costa-jussà, Christian Federmann,
Yvette Graham, Roman Grundkiewicz, Barry Had-
dow, Matthias Huck, Eric Joanis, Tom Kocmi,
Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof
Monz, Makoto Morishita, Masaaki Nagata, Toshi-
aki Nakazawa, Santanu Pal, Matt Post, and Marcos
Zampieri. 2020. Findings of the 2020 conference on
machine translation (WMT20). In *Proceedings of
the Fifth Conference on Machine Translation*, pages
1–55, Online. Association for Computational Linguis-
tics.

Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà,
Christian Federmann, Mark Fishel, Yvette Gra-
ham, Barry Haddow, Matthias Huck, Philipp Koehn,
Shervin Malmasi, Christof Monz, Mathias Müller,
Santanu Pal, Matt Post, and Marcos Zampieri. 2019.
Findings of the 2019 conference on machine trans-
lation (WMT19). In *Proceedings of the Fourth Con-
ference on Machine Translation (Volume 2: Shared
Task Papers, Day 1)*, pages 1–61, Florence, Italy. As-
sociation for Computational Linguistics.
Rachel Bawden and Benoît Sagot. 2023. RoCS-MT: Robustness Challenge Set for Machine Translation. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 Workshop on Statistical Machine Translation. In *Proceedings of the Eighth Workshop on Statistical Machine Translation*, pages 1–44, Sofia, Bulgaria. Association for Computational Linguistics.

Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In *Proceedings of the Ninth Workshop on Statistical Machine Translation*, pages 12–58, Baltimore, Maryland, USA. Association for Computational Linguistics.

Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (WMT17). In *Proceedings of the Second Conference on Machine Translation*, pages 169–214, Copenhagen, Denmark. Association for Computational Linguistics.

Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In *Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers*, pages 131–198, Berlin, Germany. Association for Computational Linguistics.

Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 workshop on statistical machine translation. In *Proceedings of the Tenth Workshop on Statistical Machine Translation*, pages 1–46, Lisbon, Portugal. Association for Computational Linguistics.

Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In *Proceedings of the Third Conference on Machine Translation: Shared Task Papers*, pages 272–303,

Belgium, Brussels. Association for Computational Linguistics.

Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (meta-) evaluation of machine translation. In *Proceedings of the Second Workshop on Statistical Machine Translation*, pages 136–158, Prague, Czech Republic. Association for Computational Linguistics.

Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation of machine translation. In *Proceedings of the Third Workshop on Statistical Machine Translation*, pages 70–106, Columbus, Ohio. Association for Computational Linguistics.

Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar Zaidan. 2010. Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In *Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and Metrics-MATR*, pages 17–53, Uppsala, Sweden. Association for Computational Linguistics.

Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 workshop on statistical machine translation. In *Proceedings of the Seventh Workshop on Statistical Machine Translation*, pages 10–51, Montréal, Canada. Association for Computational Linguistics.

Chris Callison-Burch, Philipp Koehn, Christof Monz, and Josh Schroeder. 2009. Findings of the 2009 Workshop on Statistical Machine Translation. In *Proceedings of the Fourth Workshop on Statistical Machine Translation*, pages 1–28, Athens, Greece. Association for Computational Linguistics.

Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar Zaidan. 2011. Findings of the 2011 workshop on statistical machine translation. In *Proceedings of the Sixth Workshop on Statistical Machine Translation*, pages 22–64, Edinburgh, Scotland. Association for Computational Linguistics.

Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. WIT3: Web inventory of transcribed and translated talks. In *Proceedings of the 16th Annual Conference of the European Association for Machine Translation*, pages 261–268, Trento, Italy. European Association for Machine Translation.

Xiaoyu Chen, Daimeng Wei, Zhanglin Wu, Ting Zhu, Hengchao Shang, Zongyao Li, Jiaxin GUO, Ning Xie, Lizhi Lei, Hao Yang, and Yanfei Jiang. 2023. Multifaceted Challenge Set for Evaluating Machine Translation Performance. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 8440–8451. Online. Association for Computational Linguistics.

Jan Christian Blaise Cruz. 2023. Samsung R&D Institute Philippines at WMT 2023. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Hiroyuki Deguchi, Kenji Imamura, Yuto Nishida, Yusuke Sakai, Justin Vasselli, and Taro Watanabe. 2023. NAIST-NICT WMT’23 general mt task submission. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Christian Federmann. 2018. Appraise evaluation framework for machine translation. In *Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations*, pages 86–88, Santa Fe, New Mexico. Association for Computational Linguistics.

Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 878–891, Dublin, Ireland. Association for Computational Linguistics.

Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Graham Neubig, and Andre Martins. 2022. Quality-aware decoding for neural machine translation. In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 1396–1412, Seattle, United States. Association for Computational Linguistics.

Markus Freitag, Isaac Caswell, and Scott Roy. 2019. APE at scale and its implications on MT evaluation biases. In *Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)*, pages 34–44, Florence, Italy. Association for Computational Linguistics.

Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. *Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics*, 9:1460–1474.

Markus Freitag, Nitika Mathur, Chi kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frédéric Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. Results of WMT23 Metrics Shared Task: Metrics might be Guilty but References are not Innocent. In *Proceedings of the Seventh Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Tirthankar Ghosal, Marie Hledíková, Muskaan Singh, Anna Nedoluzhko, and Ondřej Bojar. 2022. The second automatic minuting (AutoMin) challenge: Generating and evaluating minutes from multi-party meetings. In *Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges*, pages 1–11, Waterville, Maine, USA and virtual meeting. Association for Computational Linguistics.

Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. 2012. Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 languages. In *Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)*, pages 759–765, Istanbul, Turkey. European Language Resources Association (ELRA).

Thamme Gowda, Zhao Zhang, Chris Mattmann, and Jonathan May. 2021. Many-to-English machine translation tools, data, and pretrained models. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations*, pages 306–316, Online. Association for Computational Linguistics.

Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In *Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse*, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics.

Yvette Graham, Barry Haddow, and Philipp Koehn. 2020. Statistical power and translationese in machine translation evaluation. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 72–81, Online. Association for Computational Linguistics.

Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are GPT models at machine translation? a comprehensive evaluation. *arXiv preprint arXiv:2302.09210*.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.

Josef Jon, Martin Popel, and Ondřej Bojar. 2023. CUNI-GA submission at WMT23. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. 2020. *The multilingual Amazon reviews corpus*. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 4563–4568, Online. Association for Computational Linguistics.

Ondrej Klejch, Eleftherios Avramidis, Aljoscha Burchardt, and Martin Popel. 2015. MT-ComparEval: Graphical evaluation interface for machine translation development. *Prague Bull. Math. Linguistics*, 104:63–74.

Rebecca Knowles. 2021. On the stability of system rankings at WMT. In *Proceedings of the Sixth Conference on Machine Translation*, pages 464–477, Online. Association for Computational Linguistics.

Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. 2022. Findings of the 2022 conference on machine translation (WMT22). In *Proceedings of the Seventh Conference on Machine Translation (WMT)*, pages 1–45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In *Proceedings of the Sixth Conference on Machine Translation*, pages 478–494, Online. Association for Computational Linguistics.

Tom Kocmi, Martin Popel, and Ondrej Bojar. 2020. Announcing CzEng 2.0 Parallel Corpus with over 2 Gigawords. *CoRR*, abs/2007.03006.

Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In *Proceedings of Machine Translation Summit X: Papers*, pages 79–86, Phuket, Thailand.

Philipp Koehn and Christof Monz, editors. 2006. *Proceedings on the Workshop on Statistical Machine Translation*. Association for Computational Linguistics, New York City.

Keito Kudo, Takumi Ito, Makoto Morishita, and Jun Suzuki. 2023. SKIM at WMT 2023 general translation task. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Ben LI, Yoko Matsuzaki, and Shivam Kalkar. 2023. KYB general machine translation systems for WMT23. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. *Transactions of the Association for Computational Linguistics*, 8:726–742.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach.

Samuel Läubli, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen, and Antonio Toral. 2020. A Set of Recommendations for Assessing Human-Machine Parity in Language Translation. *Journal of Artificial Intelligence Research (JAIR)*, 67.

Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Zettlemoyer Luke. 2022. Mega: Moving average equipped gated attention. *arXiv preprint arXiv:2209.10655*.

Vivien Macketanz, Eleftherios Avramidis, Aljoscha Burchardt, He Wang, Renlong Ai, Shushen Manakhimova, Ursula Strohriegel, Sebastian Möller, and Hans Uszkoreit. 2022. A linguistically motivated test suite to semi-automatically evaluate German-English machine translation output. In *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 936–947, Marseille, France. European Language Resources Association.

Shushen Manakhimova, Eleftherios Avramidis, Vivien Macketanz, Ekaterina Lapshinova-Koltunski, Sergei Bagdasarov, and Sebastian Möller. 2023. Linguistically motivated Evaluation of the 2023 State-of-the-art Machine Translation: Can ChatGPT Outperform NMT? In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 4984–4997, Online. Association for Computational Linguistics.
Luo Min, yixin tan, and Qiulin Chen. 2023. Yishu: Yishu at WMT2023 translation task. In Proceedings of the Eighth Conference on Machine Translation (WMT), Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Alexander Molchanov and Vladislav Kovalenko. 2023. PROMT systems for WMT23 shared general translation task. In Proceedings of the Eighth Conference on Machine Translation (WMT), Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Makoto Morishita, Jun Suzuki, and Masaaki Nagata. 2020. JParaCrawl: A large scale web-based English-Japanese parallel corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3603-3609, Marseille, France. European Language Resources Association.

Ananya Mukherjee and Manish Shrivastava. 2023. IIIT HYD's Submission for WMT23 Test-suite task. In Proceedings of the Eighth Conference on Machine Translation (WMT), Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Mathias Müller, Malihe Alikhani, Eleftherios Avramidis, Richard Bowden, Annelies Braffort, Necati Cihan Camgöz, Sarah Ebling, Cristina España-bonet, Anne Göhring, Roman Grundkiewicz, Mert Inan, Zifan Jiang, Oscar Koller, Amit Moryossef, Annette Rios, Dimitar Shterionov, Sandra Sidler-miserez, Katja Tissi, and Davy Van Landuyt. 2023. Findings of the second WMT shared task on sign language translation (WMT-SLT23). In Proceedings of the Eight Conference on Machine Translation (WMT), Singapore. Association for Computational Linguistics.

Anna Nedoluzhko, Muskaan Singh, Marie Hledíková, Tirthankar Ghosal, and Ondřej Bojar. 2022. ELITR minuting corpus: A novel dataset for automatic minuting from multi-party meetings in English and Czech. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3174-3182, Marseille, France. European Language Resources Association.

Graham Neubig. 2011. The Kyoto free translation task. http://www.phontron.com/kftt.

Mariana Neves, Antonio Jimeno Yepes, Aurélie Névéol, Rachel Bawden, Giorgio Maria Di Nunzio, Roland Roller, Philippe Thomas, Federica Vezzani, Maika Vicente Navarro, Lana Yeganova, Dina Wiemann, and Cristian Grozea. 2023. Findings of the WMT 2023 Biomedical Translation Shared Task: Evaluation of ChatGPT 3.5 as a Comparison System. In Proceedings of the Seventh Conference on Machine Translation (WMT), Singapore, Singapore (Hybrid). Association for Computational Linguistics.

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernán, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume

Wenzek, Al Youngblood, Bapi Akula, Loic Barraault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv preprint arXiv:2207.04672.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Stephan Peitz, Sarthak Garg, Udhay Nallasamy, and Matthias Paulik. 2019. Cross+Self-Attention for Transformer Models. https://github.com/pytorch/fairseq/files/3561282/paper.pdf.

Martin Popel. 2020. CUNI English-Czech and English-Polish systems in WMT20: Robust document-level training. In Proceedings of the Fifth Conference on Machine Translation, pages 269–273, Online. Association for Computational Linguistics.

Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.

Reid Pryzant, Youngjoo Chung, Dan Jurafsky, and Denny Britz. 2018. JESC: Japanese-English subtitle corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).

Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence-
BERT: Sentence embeddings using Siamese BERT-
networks. In Proceedings of the 2019 Conference on
Empirical Methods in Natural Language Processing
and the 9th International Joint Conference on Natu-
ral Language Processing (EMNLP-IJCNLP), pages
3982–3992, Hong Kong, China. Association for Com-
putational Linguistics.

Matīss Rikters. 2018. Impact of Corpora Quality on
Neural Machine Translation. In In Proceedings of
the 8th Conference Human Language Technologies
- The Baltic Perspective (Baltic HLT 2018), Tartu,
Estonia. IOS Press.

Matīss Rikters and Makoto Miwa. 2023. AIST AIRC submissions to the WMT23 shared task. In Proceedings of the Eighth Conference on Machine Translation (WMT), Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Roberts Rozis and Raivis Skadinš. 2017. Tilde MODEL
- multilingual open data for EU languages. In Pro-
ceedings of the 21st Nordic Conference on Computa-
tional Linguistics, pages 263–265, Gothenburg, Swe-
den. Association for Computational Linguistics.

Pavel Rychlý and Yuliia Teslia. 2023. MUNI-NLP submission for Czech-Ukrainian translation task at WMT23. In Proceedings of the Eighth Conference on Machine Translation (WMT), Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Beatrice Savoldi, Marco Gaido, Matteo Negri, and Luisa Bentivogli. 2023. Test Suites Task: Evaluation of Gender Fairness in MT with MuST-SHE and INES. In Proceedings of the Eighth Conference on Machine Translation (WMT), Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2021. Wiki-Matrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1351–1361, Online. Association for Computational Linguistics.

Roman Sudarikov, Martin Popel, Ondřej Bojar, Aljoscha Burchardt, and Ondřej Klejch. 2016. Using MT-ComparEval. In Translation Evaluation: From Fragmented Tools and Data Sets to an Integrated Ecosystem, pages 76–82.

Jörg Tiedemann. 2012. Parallel data, tools and inter-
faces in OPUS. In Proceedings of the Eighth In-
ternational Conference on Language Resources and
Evaluation (LREC’12), pages 2214–2218, Istanbul,
Turkey. European Language Resources Association
(ELRA).

Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the unattainable? reassessing claims

of human parity in neural machine translation. In *Proceedings of the Third Conference on Machine Translation: Research Papers*, pages 113–123, Brussels, Belgium. Association for Computational Linguistics.

Christopher Lemmer Webber, Jessica Tallon, Erin Shep-
herd, Amy Guy, and Evan Prodromou. 2018. Ac-
tivityPub, W3C Recommendation. Technical report,
W3C.

Johnny Wei and Robin Jia. 2021. The statistical advan-
tage of automatic NLG metrics at the system level.
In Proceedings of the 59th Annual Meeting of the
Association for Computational Linguistics and the
11th International Joint Conference on Natural Lan-
guage Processing (Volume 1: Long Papers), pages
6840–6854, Online. Association for Computational
Linguistics.

Johnny Wei, Tom Kocmi, and Christian Federmann. 2022. Searching for a higher power in the human evaluation of MT. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 129–139, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Di Wu and Christof Monz. 2023. Beyond shared vocab-
ulary: Increasing representational word similarities
across languages for multilingual machine translation.
arXiv preprint arXiv:2305.14189.

Di Wu, Shaomu Tan, David Stap, Ali Araabi, and Christof Monz. 2023a. UvA-MT's participation in the WMT 2023 general translation shared task. In Proceedings of the Eighth Conference on Machine Translation (WMT), Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Yangjian Wu and Gang Hu. 2023. Exploring prompt en-
gineering with GPT language models for document-
level machine translation: Insights and findings. In
Proceedings of the Eighth Conference on Machine
Translation (WMT), Singapore, Singapore (Hybrid).
Association for Computational Linguistics.

Zhanglin Wu, Daimeng Wei, Zongyao Li, Zhengzhe YU, Shaojun Li, Xiaoyu Chen, Hengchao Shang, Jiaxin GUO, Yuhao Xie, Lizhi Lei, Hao Yang, and Yanfei Jiang. 2023b. Treating general mt shared task as a multi-domain adaptation problem: Hw-tsc's submission to the WMT23 general mt shared task. In Proceedings of the Eighth Conference on Machine Translation (WMT), Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Yiming Yan, Tao Wang, Chengqi Zhao, Shujian Huang, Jiajun Chen, and Mingxuan Wang. 2023. BLEURT has universal translations: An analysis of automatic metrics by minimum risk training. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5428–5443, Toronto, Canada. Association for Computational Linguistics.
Hui Zeng. 2023. Achieving state-of-the-art multilingual translation model with minimal data and parameters. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Wenbo Zhang. 2023. IOL Research machine translation systems for WMT23 general machine translation shared task. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.

Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. *The United Nations parallel corpus v1.0*. In *Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)*, pages 3530–3534, Portorož, Slovenia. European Language Resources Association (ELRA).

Hao Zong. 2023. GTCOM neural machine translation systems for WMT23. In *Proceedings of the Eighth Conference on Machine Translation (WMT)*, Singapore, Singapore (Hybrid). Association for Computational Linguistics.
A Statistics of training data

This section describes statistics of the training corpora.

<table><caption>**Table 9:** Statistics for parallel training set provided for General/News Translation Task. Suffixes, k, M, and B, are short for thousands, millions, and billions, respectively. Dataset ID is the unique identifier created by MTData, example *mtdata echo <dataset_id>*.</caption><thead><tr><th>Dataset ID</th><th>Segs</th><th colspan="2">Tokens</th><th colspan="2">Chars</th></tr></thead><tbody><tr><td><strong>eng-ces</strong></td><td></td><td><strong>eng</strong></td><td><strong>ces</strong></td><td><strong>eng</strong></td><td><strong>ces</strong></td></tr><tr><td>Facebook-wikimatrix-1-ces-eng</td><td>2.09M</td><td>33.56M</td><td>29.66M</td><td>206.82M</td><td>216.62M</td></tr><tr><td>ParaCrawl-paracrawl-9-eng-ces</td><td>50.63M</td><td>692.12M</td><td>626.34M</td><td>4.33B</td><td>4.68B</td></tr><tr><td>Statmt-commonrawl_wmt13-1-ces-eng</td><td>161.84k</td><td>3.35M</td><td>2.93M</td><td>20.66M</td><td>20.75M</td></tr><tr><td>Statmt-europarl-10-ces-eng</td><td>644.43k</td><td>15.63M</td><td>13.00M</td><td>94.31M</td><td>98.14M</td></tr><tr><td>Statmt-news_commentary-16-ces-eng</td><td>253.27k</td><td>5.46M</td><td>4.96M</td><td>34.58M</td><td>37.97M</td></tr><tr><td>Statmt-wikititles-3-ces-eng</td><td>410.94k</td><td>1.03M</td><td>965.62k</td><td>7.47M</td><td>7.57M</td></tr><tr><td>Tilde-ecb-2017-ces-eng</td><td>3.10k</td><td>52.12k</td><td>45.21k</td><td>327.57k</td><td>339.24k</td></tr><tr><td>Tilde-eesc-2017-ces-eng</td><td>1.33M</td><td>28.78M</td><td>25.63M</td><td>188.53M</td><td>205.14M</td></tr><tr><td>Tilde-ema-2016-ces-eng</td><td>495.23k</td><td>7.64M</td><td>7.28M</td><td>50.31M</td><td>57.01M</td></tr><tr><td>Tilde-rapid-2019-ces-eng</td><td>263.29k</td><td>5.79M</td><td>5.30M</td><td>37.36M</td><td>41.26M</td></tr><tr><td>(Total)</td><td>56.29M</td><td>793.41M</td><td>716.10M</td><td>4.97B</td><td>5.36B</td></tr><tr><td><strong>eng-deu</strong></td><td></td><td><strong>eng</strong></td><td><strong>deu</strong></td><td><strong>eng</strong></td><td><strong>deu</strong></td></tr><tr><td>Facebook-wikimatrix-1-deu-eng</td><td>6.23M</td><td>100.50M</td><td>96.95M</td><td>623.66M</td><td>701.23M</td></tr><tr><td>ParaCrawl-paracrawl-9-eng-deu</td><td>278.31M</td><td>4.27B</td><td>3.99B</td><td>26.37B</td><td>29.46B</td></tr><tr><td>Statmt-commonrawl_wmt13-1-deu-eng</td><td>2.40M</td><td>51.40M</td><td>47.05M</td><td>314.18M</td><td>340.51M</td></tr><tr><td>Statmt-europarl-10-deu-eng</td><td>1.82M</td><td>45.51M</td><td>42.41M</td><td>272.94M</td><td>312.14M</td></tr><tr><td>Statmt-news_commentary-16-deu-eng</td><td>388.48k</td><td>8.55M</td><td>8.77M</td><td>54.40M</td><td>65.94M</td></tr><tr><td>Statmt-wikititles-3-deu-eng</td><td>1.47M</td><td>3.61M</td><td>3.08M</td><td>26.48M</td><td>25.50M</td></tr><tr><td>Tilde-airbaltic-1-deu-eng</td><td>0.84k</td><td>17.60k</td><td>15.08k</td><td>104.34k</td><td>105.52k</td></tr><tr><td>Tilde-czechtourism-1-deu-eng</td><td>6.76k</td><td>128.29k</td><td>114.44k</td><td>769.04k</td><td>829.41k</td></tr><tr><td>Tilde-ecb-2017-deu-eng</td><td>4.15k</td><td>85.52k</td><td>74.81k</td><td>545.51k</td><td>582.63k</td></tr><tr><td>Tilde-eesc-2017-deu-eng</td><td>2.86M</td><td>61.47M</td><td>58.28M</td><td>400.37M</td><td>469.94M</td></tr><tr><td>Tilde-ema-2016-deu-eng</td><td>347.63k</td><td>5.09M</td><td>5.01M</td><td>33.48M</td><td>39.43M</td></tr><tr><td>Tilde-rapid-2016-deu-eng</td><td>1.03M</td><td>20.65M</td><td>19.85M</td><td>134.26M</td><td>158.13M</td></tr><tr><td>Tilde-rapid-2019-deu-eng</td><td>939.81k</td><td>19.90M</td><td>19.30M</td><td>129.03M</td><td>153.08M</td></tr><tr><td>(Total)</td><td>295.81M</td><td>4.59B</td><td>4.29B</td><td>28.36B</td><td>31.73B</td></tr><tr><td><strong>eng-heb</strong></td><td></td><td><strong>eng</strong></td><td><strong>heb</strong></td><td><strong>eng</strong></td><td><strong>heb</strong></td></tr><tr><td>ELRC-wikipedia_health-1-eng-heb</td><td>3.16k</td><td>69.71k</td><td>54.76k</td><td>442.38k</td><td>583.87k</td></tr><tr><td>Facebook-wikimatrix-1-eng-heb</td><td>2.04M</td><td>35.83M</td><td>28.96M</td><td>218.77M</td><td>300.61M</td></tr><tr><td>Neulab-tedtalks_train-1-eng-heb</td><td>211.82k</td><td>4.45M</td><td>3.44M</td><td>22.36M</td><td>29.00M</td></tr><tr><td>OPUS-bible_uedin-v1-eng-heb</td><td>62.20k</td><td>1.55M</td><td>830.23k</td><td>8.16M</td><td>7.46M</td></tr><tr><td>OPUS-ccmatrix-v1-eng-heb</td><td>25.23M</td><td>313.87M</td><td>249.49M</td><td>1.81B</td><td>2.45B</td></tr><tr><td>OPUS-elrc_2922-v1-eng-heb</td><td>3.16k</td><td>69.73k</td><td>54.77k</td><td>442.40k</td><td>583.54k</td></tr><tr><td>OPUS-elrc_3065_wikipedia_health-v1-eng-heb</td><td>3.16k</td><td>69.71k</td><td>54.76k</td><td>442.31k</td><td>583.51k</td></tr><tr><td>OPUS-elrc_wikipedia_health-v1-eng-heb</td><td>3.16k</td><td>69.71k</td><td>54.76k</td><td>442.31k</td><td>583.51k</td></tr><tr><td>OPUS-globalvoices-v2018q4-eng-heb</td><td>1.03k</td><td>20.31k</td><td>15.03k</td><td>122.39k</td><td>158.63k</td></tr><tr><td>OPUS-gnome-v1-eng-heb</td><td>0.15k</td><td>0.42k</td><td>0.40k</td><td>2.89k</td><td>3.96k</td></tr><tr><td>OPUS-kde4-v2-eng-heb</td><td>79.32k</td><td>338.22k</td><td>347.35k</td><td>2.09M</td><td>3.13M</td></tr><tr><td>OPUS-multicaligned-v1-eng-heb</td><td>5.33M</td><td>60.55M</td><td>52.81M</td><td>380.74M</td><td>518.33M</td></tr><tr><td>OPUS-opensubtitles-v2018-eng-heb</td><td>29.89M</td><td>195.98M</td><td>154.25M</td><td>1.03B</td><td>1.40B</td></tr><tr><td>OPUS-php-v1-eng-heb</td><td>27.82k</td><td>83.46k</td><td>93.03k</td><td>498.72k</td><td>789.34k</td></tr><tr><td>OPUS-qed-v2.0a-eng-heb</td><td>464.35k</td><td>6.37M</td><td>4.48M</td><td>34.70M</td><td>42.34M</td></tr><tr><td>OPUS-tatoeba-v20220303-eng-heb</td><td>164.20k</td><td>1.02M</td><td>806.38k</td><td>5.41M</td><td>7.37M</td></tr><tr><td>OPUS-tatoeba-v2-eng-heb</td><td>54.36k</td><td>357.09k</td><td>277.32k</td><td>1.87M</td><td>2.56M</td></tr><tr><td>OPUS-ubuntu-v14.10-eng-heb</td><td>1.44k</td><td>6.13k</td><td>5.78k</td><td>38.78k</td><td>54.69k</td></tr><tr><td>OPUS-wikimedia-v20210402-eng-heb</td><td>226.83k</td><td>8.51M</td><td>7.56M</td><td>57.58M</td><td>78.26M</td></tr><tr><td>OPUS-wikipedia-v1.0-eng-heb</td><td>139.85k</td><td>2.69M</td><td>2.27M</td><td>16.45M</td><td>22.43M</td></tr><tr><td>OPUS-xlent-v1.1-eng-heb</td><td>3.19M</td><td>9.61M</td><td>7.93M</td><td>60.53M</td><td>73.11M</td></tr><tr><td>Statmt-ccaligned-1-eng-heb_IL</td><td>5.33M</td><td>60.55M</td><td>52.81M</td><td>380.76M</td><td>518.34M</td></tr><tr><td>(Total)</td><td>72.46M</td><td>702.05M</td><td>566.59M</td><td>4.04B</td><td>5.45B</td></tr></tbody></table>
<table><caption>**Table 10:** Statistics for parallel training set provided for General/News Translation Task. Suffixes, k, M, and B, are short for thousands, millions, and billions, respectively.</caption><thead><tr><th>Dataset ID</th><th>Segs</th><th>Tokens</th><th>Chars</th><th></th></tr></thead><tbody><tr><td><strong>eng-jpn</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Facebook-wikimatrix-1-eng-jpn</td><td>3.90M</td><td>61.63M</td><td>379.09M</td><td>454.97M</td></tr><tr><td>KECL-paracrawl-3-eng-jpn</td><td>25.74M</td><td>599.02M</td><td>3.69B</td><td>4.58B</td></tr><tr><td>Phontron-kftt_train-1-eng-jpn</td><td>440.29k</td><td>9.74M</td><td>59.91M</td><td>49.08M</td></tr><tr><td>StanfordNLP-jesc_train-1-eng-jpn</td><td>2.80M</td><td>19.34M</td><td>104.00M</td><td>119.62M</td></tr><tr><td>Statmt-news_commentary-16-eng-jpn</td><td>1.84k</td><td>39.50k</td><td>247.70k</td><td>310.56k</td></tr><tr><td>Statmt-ted-wmt20-eng-jpn</td><td>241.74k</td><td>4.03M</td><td>23.02M</td><td>27.32M</td></tr><tr><td>Statmt-wikititles-3-jpn-eng</td><td>757.04k</td><td>1.94M</td><td>13.96M</td><td>18.67M</td></tr><tr><td>(Total)</td><td>33.88M</td><td>695.74M</td><td>4.27B</td><td>5.25B</td></tr><tr><td><strong>eng-rus</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Facebook-wikimatrix-1-eng-rus</td><td>5.20M</td><td>86.79M</td><td>76.48M</td><td>537.73M</td><td>965.44M</td></tr><tr><td>OPUS-unpc-v1.0-eng-rus</td><td>25.17M</td><td>563.82M</td><td>520.71M</td><td>3.70B</td><td>7.31B</td></tr><tr><td>ParaCrawl-paracrawl-1_bonus-eng-rus</td><td>5.38M</td><td>101.31M</td><td>80.41M</td><td>632.54M</td><td>1.06B</td></tr><tr><td>Statmt-backtrans_enru-wmt20-eng-rus</td><td>36.77M</td><td>736.20M</td><td>670.93M</td><td>4.31B</td><td>7.73B</td></tr><tr><td>Statmt-commonrawl_wmt13-1-rus-eng</td><td>878.39k</td><td>18.77M</td><td>17.40M</td><td>116.16M</td><td>214.59M</td></tr><tr><td>Statmt-news_commentary-16-eng-rus</td><td>331.51k</td><td>7.67M</td><td>7.13M</td><td>48.79M</td><td>97.41M</td></tr><tr><td>Statmt-wikititles-3-rus-eng</td><td>1.19M</td><td>3.13M</td><td>2.88M</td><td>22.80M</td><td>39.34M</td></tr><tr><td>Statmt-yandex-wmt22-eng-rus</td><td>1.00M</td><td>21.25M</td><td>18.68M</td><td>130.99M</td><td>250.76M</td></tr><tr><td>Tilde-airbaltic-1-eng-rus</td><td>1.09k</td><td>23.98k</td><td>18.79k</td><td>142.52k</td><td>252.73k</td></tr><tr><td>Tilde-czechtourism-1-eng-rus</td><td>7.33k</td><td>140.09k</td><td>110.10k</td><td>838.09k</td><td>1.50M</td></tr><tr><td>Tilde-worldbank-1-eng-rus</td><td>25.85k</td><td>588.58k</td><td>573.93k</td><td>3.85M</td><td>8.21M</td></tr><tr><td>(Total)</td><td>75.96M</td><td>1.54B</td><td>1.40B</td><td>9.50B</td><td>17.67B</td></tr><tr><td><strong>eng-ukr</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>ELRC-acts_ukrainian-1-eng-ukr</td><td>129.94k</td><td>3.04M</td><td>2.60M</td><td>19.55M</td><td>35.69M</td></tr><tr><td>Facebook-wikimatrix-1-eng-ukr</td><td>2.58M</td><td>41.55M</td><td>35.59M</td><td>257.56M</td><td>447.33M</td></tr><tr><td>ParaCrawl-paracrawl-1_bonus-eng-ukr</td><td>13.35M</td><td>505.83M</td><td>487.47M</td><td>3.28B</td><td>6.04B</td></tr><tr><td>Tilde-worldbank-1-eng-ukr</td><td>1.63k</td><td>36.07k</td><td>34.18k</td><td>237.96k</td><td>477.91k</td></tr><tr><td>(Total)</td><td>16.06M</td><td>550.46M</td><td>525.68M</td><td>3.55B</td><td>6.52B</td></tr><tr><td><strong>eng-zho</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Facebook-wikimatrix-1-eng-zho</td><td>2.60M</td><td>49.87M</td><td>311.07M</td><td>277.84M</td></tr><tr><td>OPUS-unpc-v1.0-eng-zho</td><td>17.45M</td><td>417.25M</td><td>2.75B</td><td>2.14B</td></tr><tr><td>ParaCrawl-paracrawl-1_bonus-eng-zho</td><td>14.17M</td><td>217.60M</td><td>1.34B</td><td>1.18B</td></tr><tr><td>Statmt-backtrans_enzh-wmt20-eng-zho</td><td>19.76M</td><td>364.22M</td><td>2.16B</td><td>1.96B</td></tr><tr><td>Statmt-news_commentary-16-eng-zho</td><td>313.67k</td><td>6.92M</td><td>44.14M</td><td>38.83M</td></tr><tr><td>Statmt-wikititles-3-zho-eng</td><td>921.96k</td><td>2.37M</td><td>17.82M</td><td>16.28M</td></tr><tr><td>(Total)</td><td>55.22M</td><td>1.06B</td><td>6.62B</td><td>5.61B</td></tr><tr><td><strong>ces-ukr</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>ELRC-acts_ukrainian-1-ces-ukr</td><td>130.00k</td><td>2.48M</td><td>2.56M</td><td>19.61M</td><td>35.26M</td></tr><tr><td>Facebook-wikimatrix-1-ces-ukr</td><td>848.96k</td><td>10.43M</td><td>10.07M</td><td>75.97M</td><td>127.31M</td></tr><tr><td>OPUS-bible_uedin-v1-ces-ukr</td><td>7.95k</td><td>140.03k</td><td>132.06k</td><td>904.31k</td><td>1.33M</td></tr><tr><td>OPUS-ccmatrix-v1-ces-ukr</td><td>3.99M</td><td>45.13M</td><td>45.10M</td><td>330.68M</td><td>566.27M</td></tr><tr><td>OPUS-elrc_5179_acts_ukrainian-v1-ces-ukr</td><td>130.00k</td><td>2.48M</td><td>2.56M</td><td>19.61M</td><td>35.26M</td></tr><tr><td>OPUS-elrc_wikipedia_health-v1-ces-ukr</td><td>0.19k</td><td>3.23k</td><td>3.18k</td><td>24.27k</td><td>41.63k</td></tr><tr><td>OPUS-eubookshop-v2-ces-ukr</td><td>1.51k</td><td>23.71k</td><td>19.15k</td><td>187.30k</td><td>275.14k</td></tr><tr><td>OPUS-gnome-v1-ces-ukr</td><td>0.15k</td><td>0.42k</td><td>0.41k</td><td>3.53k</td><td>5.82k</td></tr><tr><td>OPUS-kde4-v2-ces-ukr</td><td>133.67k</td><td>593.82k</td><td>677.35k</td><td>4.45M</td><td>7.97M</td></tr><tr><td>OPUS-multicaligned-v1.1-ces-ukr</td><td>1.61M</td><td>19.75M</td><td>19.77M</td><td>146.44M</td><td>244.36M</td></tr><tr><td>OPUS-multiparacrawl-v9b-ces-ukr</td><td>2.20M</td><td>25.62M</td><td>25.55M</td><td>188.08M</td><td>325.50M</td></tr><tr><td>OPUS-opensubtitles-v2018-ces-ukr</td><td>730.80k</td><td>3.88M</td><td>3.90M</td><td>24.20M</td><td>40.62M</td></tr><tr><td>OPUS-qed-v2.0a-ces-ukr</td><td>161.02k</td><td>2.02M</td><td>2.04M</td><td>13.44M</td><td>22.80M</td></tr><tr><td>OPUS-tatoeba-v20220303-ces-ukr</td><td>2.93k</td><td>10.85k</td><td>11.40k</td><td>68.70k</td><td>118.67k</td></tr><tr><td>OPUS-ted2020-v1-ces-ukr</td><td>114.23k</td><td>1.57M</td><td>1.56M</td><td>10.70M</td><td>17.93M</td></tr><tr><td>OPUS-ubuntu-v14.10-ces-ukr</td><td>0.23k</td><td>1.67k</td><td>1.76k</td><td>13.02k</td><td>20.86k</td></tr><tr><td>OPUS-wikimedia-v20210402-ces-ukr</td><td>1.96k</td><td>39.18k</td><td>34.91k</td><td>285.74k</td><td>414.20k</td></tr><tr><td>OPUS-xlent-v1.1-ces-ukr</td><td>695.41k</td><td>1.78M</td><td>1.58M</td><td>12.92M</td><td>18.30M</td></tr><tr><td>(Total)</td><td>10.76M</td><td>115.95M</td><td>115.57M</td><td>847.58M</td><td>1.44B</td></tr></tbody></table>
B Preprocessing cleanup brief for linguists

Human check briefing

In this task, we wish to check the data to remove all inappropriate content, remove repetitive content, or correct minor problems with the text.

The data is automatically broken down into individual sentences, which may contain wrong sentence splitting that needs to be fixed. Each paragraph is separated by empty lines. Keep the document-separators intact.

We ask you to read each document and either:

* Delete document completely if it contains any of following issues. Be on the save side, rather remove documents where you are uncertain

* Remove documents written in different language (natural code-switching is fine)

* Remove inappropriate content (such as sexually explicit, vulgar, or otherwise inappropriate)

* Remove controversial content (propagandist, controversial political topics, etc.)

* Remove content that is too noisy or doesn't resemble natural text (such as documents badly formatted, hard to understand, containing unusual language, lists of numbers/data, or other structured data generated automatically)

* Keep document while checking

* Fix sentence-breaking, each line must be one sentence (do not reformulate, simply remove or add end of lines on a proper place).

* Remove or move fragments of sentences to previous or following sentence (for example emoticons, one or few words sentences)

* Fix minor issues and keep it (do not spend too much time on fixing it).

* It is fine to keep some errors or problems

* Remove boilerplates (segments that break the document, for example ads, page numbers, signatures, artefacts, ...)

* If a given document has more than around 30 sentences, consider splitting it by adding an empty line on a meaningful place splitting it into paragraphs

This task shouldn't take much longer than reading through documents.
C **Translator Brief for General MT**

Translator Brief

In this project we wish to translate online news articles for use in evaluation of Machine
Translation (MT). The translations produced by you will be compared against the translations
produced by a variety of different MT systems. They will be released to the research
community to provide a benchmark, or "gold-standard" measure for translation quality. The
translation therefore needs to be a high-quality rendering of the source text into the target
language, as if it was news written directly in the target language. However, there are some
constraints imposed by the intended usage:

* All translations should be "from scratch", without post-editing from MT. Using post-editing would bias the evaluation, so we need to avoid it. We can detect post-editing so will reject translations that are post-edited.

* Translation should preserve the sentence boundaries. The source texts are provided with exactly one sentence per line, and the translations should be the same, one sentence per line. Blank lines should be preserved in the translation.

* Translators should avoid inserting parenthetical explanations into the translated text and obviously avoid losing any pieces of information from the source text. We will check a sample of the translations for quality, and we will check the entire set for evidence of post-editing.

* Please do not translate the anonymization tags (e.g. `#NAME#`), but use the same form as in the source text. These tags are used to de-identify names and various other sensitive data. In other words, translation must contain given tag `#NAME#` on a position where it would naturally be placed before anonymization.

* If the original data contain errors, typos, or other problems, do not try to fix them (or introduce them in the translation), instead try to prepare correct translation as if the error wouldn't be in the source.

The source files will be delivered as text files (sometimes known as "notepad" files), with one sentence per line. We need the translations to be returned in the same format. If you prefer to receive the text in a different format, then please let us know as we may be able to accommodate it.
## D Additional statistics of the test sets

Table 11 shows the type-token ratios for the source and target side of each of the test sets, shown for the four main domains. As mentioned previously, texts are tokenised using the language-specific Spacy models (Honnibal and Montani, 2017) where available. For Hebrew, we use the multilingual Spacy model as no language-specific model is available. The type-token ratio is calculated as the number of unique tokens divided by the total number of tokens. The absolute value depends not only on the lexical diversity of the text but also on the morphological complexity of the language in question.

<table><caption>Table 11: Type-token ratio for individual source languages used in the general translation test sets.</caption><thead><tr><th rowspan="2"></th><th colspan="2">manuals</th><th colspan="2">mastodon</th><th colspan="2">news</th><th colspan="2">user_review</th></tr><tr><th>src</th><th>trg</th><th>src</th><th>trg</th><th>src</th><th>trg</th><th>src</th><th>trg</th></tr></thead><tbody><tr><td colspan="9"><strong>From English</strong></td></tr><tr><td>en-cs</td><td>-</td><td>-</td><td>0.30</td><td>0.42</td><td>0.27</td><td>0.39</td><td>0.22</td><td>0.35</td></tr><tr><td>en-de</td><td>-</td><td>-</td><td>0.30</td><td>0.32</td><td>0.27</td><td>0.29</td><td>-</td><td>-</td></tr><tr><td>en-he</td><td>-</td><td>-</td><td>0.30</td><td>0.30</td><td>0.27</td><td>0.29</td><td>0.22</td><td>0.24</td></tr><tr><td>en-ja</td><td>-</td><td>-</td><td>0.30</td><td>0.23</td><td>0.27</td><td>0.19</td><td>0.22</td><td>0.17</td></tr><tr><td>en-ru</td><td>-</td><td>-</td><td>0.30</td><td>0.41</td><td>0.27</td><td>0.38</td><td>0.22</td><td>0.33</td></tr><tr><td>en-uk</td><td>-</td><td>-</td><td>0.30</td><td>0.41</td><td>0.27</td><td>0.38</td><td>0.22</td><td>0.34</td></tr><tr><td>en-zh</td><td>-</td><td>-</td><td>0.30</td><td>0.29</td><td>0.27</td><td>0.26</td><td>0.22</td><td>0.21</td></tr><tr><td colspan="9"><strong>Other language directions</strong></td></tr><tr><td>cs-uk</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.43</td><td>0.41</td><td>-</td><td>-</td></tr><tr><td>de-en</td><td>0.32</td><td>0.23</td><td>0.49</td><td>0.42</td><td>0.34</td><td>0.26</td><td>-</td><td>-</td></tr><tr><td>he-en</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.34</td><td>0.09</td><td>-</td><td>-</td></tr><tr><td>ja-en</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.22</td><td>0.23</td><td>0.22</td><td>0.21</td></tr><tr><td>ru-en</td><td>0.47</td><td>0.28</td><td>-</td><td>-</td><td>0.40</td><td>0.24</td><td>-</td><td>-</td></tr><tr><td>uk-en</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.36</td><td>0.21</td><td>-</td><td>-</td></tr><tr><td>zh-en</td><td>0.25</td><td>0.25</td><td>-</td><td>-</td><td>0.23</td><td>0.19</td><td>0.22</td><td>0.17</td></tr></tbody></table>

## E News Task System Submission Summaries

This section lists all the submissions to the translation task and provides the authors' descriptions of their submission.

### E.1 AIRC (Rikters and Miwa, 2023)

AIRC trained constrained track models for translation between English, German, and Japanese. Before training the final models we first filtered the parallel and monolingual data (Rikters, 2018), then performed iterative back-translation as well as parallel data distillation to be used for non-autoregressive model training. We experimented with training Transformer models, Mega (Ma et al., 2022) models, and custom non-autoregressive sequence-to-sequence models with encoder and decoder weights initialised by multilingual BERT base. Our primary submissions contain translations from ensembles of two Mega model checkpoints and our contrastive submissions are generated by our non-autoregressive models.

### E.2 ANVITA (no associated paper)

ANVITA-ZhJa Machine Translation system for WMT2023 Shared Task:General MT(News). This paper describes ANVITA-ZhJa MT system, architected for submission to WMT 2023 General Machine Translation(News) shared task by the ANVITA team, where the team participated in 4 translation directions: Chinese, Japanese→English and English→Chinese, Japanese. ANVITA-ZhJa MT system comprised of four NMT models.Chinese, Japanese→English and English→Chinese, Japanese multilingual models for primary and Chinese→English and English→Chinese bilingual models for contrastive submissions. Base MT models are built using transformer(base) architecture, trained over the organizer provided parallel corpus and subsequently used deep transformer with added layers and other parameters. We also distilled corpus using heuristics based filtering and used model ensemble for enhanced performance.
### E.3 CUNI-DocTransformer (Popel, 2020)

Exactly the same system as submitted in WMT20, document-level Transformer trained with Block Backtranslation.

### E.4 CUNI-GA (Jon et al., 2023)

Our submission is a result of applying a novel n-best list reranking and modification method on translation candidates produced by two other competing systems, CUNI-Transformer and CUNI-DocTransformer. Our method uses a genetic algorithm and MBR decoding to search for optimal translation under a given metric (in our case, a weighted combination of ChrF, BLEU, COMET22-DA, and COMET22-QE-DA).

### E.5 CUNI-Transformer (Popel, 2020)

The English↔Czech sentence-level models are exactly the same as submitted in WMT20 (Popel, 2020). The Ukrainian↔Czech models are very similar, also trained with Block Backtranslation.

### E.6 GTCOM (Zong, 2023)

GTCOM uses transformer as the basic architecture and leverages multilingual models to improve translation quality. Besides, GTCOM does a lot of data cleaning and data augmentation work.

### E.7 HW-TSC (Wu et al., 2023b)

HW-TSC's submission is a standard Transformer model equipped with our recent technique.

### E.8 IOL-Research (Zhang, 2023)

This paper describes the IOL Research team's submission system for the WMT23 General Machine Translation shared task. We participate in two language translation directions, including English-to-Chinese and Chinese-to-English. Our final primary submissions belong to constrained systems, which means for both translation directions we only use officially provided monolingual and bilingual data to train the translation systems. Our systems are based on Transformer architecture with pre-norm or deep-norm, which has been proven to be helpful for training deeper models. We employ methods such as back-translation, data diversification, domain fine-tuning and model ensemble to build our translation systems. Another important aspect is that we carefully conduct data cleaning and use as much monolingual data as possible for data augmentation.

### E.9 TeamKYB (LI et al., 2023)

We here describe our neural machine translation system for the general machine translation shared task in WMT 2023. Our systems are based on the Transformer with base settings. We trained our model with preprocessed train data. We collect multiple checkpoint from our model and performed inference with several hyperparameter settings. Collected translations were processed via some rule-based corrections. We chose best translation from the results by using N-best ranking method.

### E.10 Lan-BridgeMT (Wu and Hu, 2023)

With the emergence of large-scale models, various industries have undergone significant transformations, particularly in the realm of document-level machine translation. This has introduced a novel research paradigm that we have embraced in our participation in the WMT23 competition. Focusing on advances in models such as chatGPT and GPT4, we have undertaken numerous prompt-based experiments. Our objective is to achieve optimal human evaluation results for document-level machine translation, resulting in our submission of the final outcomes in the general track.

### E.11 MUNI-NLP (Rychlý and Teslia, 2023)

MUNI-NLP system is a standard transformer.
E.12 NAIST-NICT (Deguchi et al., 2023)

In this paper, we describe our NAIST-NICT submission to the WMT'23 English-Japanese general machine translation task. Our system generates diverse translation candidates and reranks them with a two-stage reranking system to find the best translation. We first generate 50 candidates each from 18 different translation methods using a variety of techniques to increase the diversity of the translation candidates. We trained 7 different models per language direction using different combinations of hyperparameters. From these models we used various decoding algorithms, ensembling the models, and using kNN-MT. The 900 translation candidates go through a two-stage reranking system in order to find the most promising candidate. The first step compares the 50 candidates from each translation method using DrNMT and returns the one with the highest score. The final 18 candidates are ranked using COMET-MBR, and the highest scoring is returned as the system output. We found that generating diverse translation candidates improves the translation quality by using the well-designed relanker model.

E.13 PROMT (Molchanov and Kovalenko, 2023)

This paper describes the PROMT submissions for the WMT23 Shared General Translation Task. This year we participated in two directions of the Shared Translation Task: English to Russian and Russian to English. Our models are trained with the MarianNMT toolkit using the transformer-big configuration. We use BPE for text encoding, both models are unconstrained. We achieve competitive results according to automatic metrics in both directions.

E.14 SRPH (Cruz, 2023)

We submit single-model encode-decoder Transformer systems for the constrained English to Hebrew and Hebrew to English translation directions. Our dataset is cleaned and filtered via a combination of heuristic-based, ratio-based, and embedding-based (LaBSE) methods, resulting in a dataset with high alignment. We train models with heavy use of back-translation and decode using Noisy Channel Reranking using a reverse model and a language model trained with contest data.

E.15 SKIM (Kudo et al., 2023)

The SKIM team submission took a standard procedure of building ensemble Transformer models, including base-model training, data augmentation using back-translation of base models, and retraining several final models using back-translated training data. Each final model has its own architecture and configuration, including a 10.5B parameter at most, substituting self and cross sublayers in decoder with cross+self-attention sub-layer (Peitz et al., 2019). We select the best candidate from large candidate pools, namely 70 translations generated from 16 distinct models for each sentence, with an MBR reranking method using COMET and COMET-QE (Fernandes et al., 2022). We also applied data augmentation and selection techniques to training data of the Transformer models.

E.16 UPCite-CLILLF (no associated paper)

In this biomedical shared task, we have created data filters to better "choose" relevant training data for fine-tuning, among provided training data sources. In particular, we have used the textometric analysis tool ITRAMEUR to filter the segments and terms that characterize the test set and then extracted them from training data to fine-tune MBart-50 baseline (decoder_attention ``heads: 16, decoder ``ffn ``dim: 4096, decoder ``layers: 12, encoder ``attention ``heads: 16, encoder ``ffn ``dim: 4096, encoder ``layers: 12, num ``hidden ``layers: 12, max ``length: 200, epoch: 3). In doing so, we hope to meet several objectives: to build feasible fine-tuning strategy to train biomedical in-domain fr<->en models; to specify filtering criteria of in-domain training data and to compare models' predictions, fine-tuning data and test set in order to better understand how neural machine translation systems work. We will also compare the pipeline of the shared task of this year to those of the past 2 years to evaluate the benefits of our training strategies of in-domain machine translation models.
### E.17 UvA-LTL (Wu et al., 2023a)

We present our WMT system, UvA-MT, in the WMT 2023 shared general translation task. This year, we developed a single Multilingual Machine Translation (MMT) system to participate in the two-directional translation track between English and Hebrew. The main architecture is based on the prior work of Beyond Shared Vocabulary (Wu and Monz, 2023). We scaled it up to a transformer-large level (422M parameters). Additionally, we employed back translation to generate synthetic data and labeled them with a new language tag. After convergence, we further fine-tuned the system without using synthetic data. Several domain shift techniques were also introduced, such as the domain-aware language model, to filter monolingual data.

### E.18 YiShu (Min et al., 2023)

Yishu's team participated in WMT23 Machine Translation Competition and adopted the most advanced neural machine translation method. They use Transformer model structure and use large-scale parallel corpus for training. In order to improve the translation quality, the team adopted cutting-edge data preprocessing technology, various attention mechanisms and improved decoding strategies. In addition, they also carried out in-depth parameter adjustment and model optimization. Yishu team incorporated evaluation indicators such as BLEU and TER into the training constraints of the model to achieve better translation performance. They strive for high accuracy and fluency in the competition, and strive to achieve excellent results in the field of translation.

### E.19 LanguageX (Zeng, 2023)

LanguageX's submission is a many-to-many encoder decoder transformer model.
## F Automatic scores

This section contains automatic metric scores. While human judgement is the official ranking of systems and their performance, we share automatic scores to show expected system performance for various testsets.

We use COMET (Rei et al., 2020) as the primary metric and chrF (Popović, 2015) as the secondary metric, following recommendation by (Kocmi et al., 2021). We also present BLEU (Papineni et al., 2002) scores as it is still a widely used metric. The COMET scores are calculated with the default model Unbabel/wmt22-comet-da. The chrF and BLEU scores are calculated using SacreBLEU (Post, 2018). Scores are multiplied by 100. We ranked the systems according to their scores. Unconstrained systems are indicated with a grey background in the tables.

<table><thead><tr><th>System</th><th>COMET</th></tr></thead><tbody><tr><td>CUNI-GA</td><td>90.9</td></tr><tr><td>GPT4-5shot</td><td>90.8</td></tr><tr><td>ONLINE-W</td><td>89.4</td></tr><tr><td>GTCOM_Peter</td><td>88.9</td></tr><tr><td>ONLINE-B</td><td>88.8</td></tr><tr><td>ONLINE-A</td><td>88.2</td></tr><tr><td>CUNI-Transformer</td><td>88.0</td></tr><tr><td>ONLINE-G</td><td>87.7</td></tr><tr><td>MUNI-NLP</td><td>87.0</td></tr><tr><td>ONLINE-Y</td><td>86.5</td></tr><tr><td>NLLB_Greedy</td><td>86.3</td></tr><tr><td>NLLB_MBR_BLEU</td><td>86.3</td></tr><tr><td>Lan-BridgeMT</td><td>86.0</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>chrF</th></tr></thead><tbody><tr><td>GPT4-5shot</td><td>61.0</td></tr><tr><td>CUNI-GA</td><td>57.9</td></tr><tr><td>GTCOM_Peter</td><td>57.6</td></tr><tr><td>CUNI-Transformer</td><td>57.4</td></tr><tr><td>MUNI-NLP</td><td>57.0</td></tr><tr><td>Lan-BridgeMT</td><td>55.7</td></tr><tr><td>ONLINE-W</td><td>55.0</td></tr><tr><td>ONLINE-B</td><td>54.7</td></tr><tr><td>ONLINE-A</td><td>54.4</td></tr><tr><td>ONLINE-G</td><td>53.7</td></tr><tr><td>ONLINE-Y</td><td>53.4</td></tr><tr><td>NLLB_Greedy</td><td>52.5</td></tr><tr><td>NLLB_MBR_BLEU</td><td>52.3</td></tr></tbody></table>

<table><caption>**Table 12:** Scores for the cs→uk translation task: chrF (nrefs:1lcase:mixedleff:yeslnc:6lnw:0lspace:nolversion:2.2.1), BLEU (nrefs:1lcase:mixedleff:noltok:13alsmooth:explversion:2.2.1), COMET (Unbabel/wmt22-comet-da).</caption><thead><tr><th>System</th><th>BLEU</th></tr></thead><tbody><tr><td>GPT4-5shot</td><td>32.8</td></tr><tr><td>CUNI-Transformer</td><td>30.2</td></tr><tr><td>GTCOM_Peter</td><td>29.8</td></tr><tr><td>CUNI-GA</td><td>29.5</td></tr><tr><td>MUNI-NLP</td><td>28.3</td></tr><tr><td>Lan-BridgeMT</td><td>27.5</td></tr><tr><td>ONLINE-W</td><td>26.8</td></tr><tr><td>ONLINE-B</td><td>25.7</td></tr><tr><td>ONLINE-A</td><td>25.4</td></tr><tr><td>NLLB_MBR_BLEU</td><td>25.1</td></tr><tr><td>NLLB_Greedy</td><td>24.9</td></tr><tr><td>ONLINE-G</td><td>24.8</td></tr><tr><td>ONLINE-Y</td><td>24.2</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>COMET</th></tr></thead><tbody><tr><td>ONLINE-W</td><td>91.8</td></tr><tr><td>CUNI-GA</td><td>90.8</td></tr><tr><td>ONLINE-B</td><td>89.9</td></tr><tr><td>GPT4-5shot</td><td>89.4</td></tr><tr><td>ONLINE-A</td><td>88.4</td></tr><tr><td>CUNI-DocTransformer</td><td>88.3</td></tr><tr><td>GTCOM_Peter</td><td>87.7</td></tr><tr><td>ONLINE-M</td><td>87.4</td></tr><tr><td>Lan-BridgeMT</td><td>87.3</td></tr><tr><td>CUNI-Transformer</td><td>87.2</td></tr><tr><td>NLLB_Greedy</td><td>87.1</td></tr><tr><td>ONLINE-Y</td><td>87.0</td></tr><tr><td>NLLB_MBR_BLEU</td><td>86.9</td></tr><tr><td>ONLINE-G</td><td>85.9</td></tr><tr><td>ZengHuiMT</td><td>85.4</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>chrF</th></tr></thead><tbody><tr><td>ONLINE-W</td><td>76.3</td></tr><tr><td>ONLINE-B</td><td>70.4</td></tr><tr><td>ZengHuiMT</td><td>67.5</td></tr><tr><td>ONLINE-A</td><td>66.3</td></tr><tr><td>CUNI-GA</td><td>65.9</td></tr><tr><td>GTCOM_Peter</td><td>65.4</td></tr><tr><td>CUNI-DocTransformer</td><td>65.1</td></tr><tr><td>ONLINE-Y</td><td>64.6</td></tr><tr><td>CUNI-Transformer</td><td>63.9</td></tr><tr><td>Lan-BridgeMT</td><td>63.8</td></tr><tr><td>ONLINE-G</td><td>63.7</td></tr><tr><td>ONLINE-M</td><td>63.2</td></tr><tr><td>GPT4-5shot</td><td>62.3</td></tr><tr><td>NLLB_Greedy</td><td>60.0</td></tr><tr><td>NLLB_MBR_BLEU</td><td>59.1</td></tr></tbody></table>

<table><caption>**Table 13:** Scores for the en→cs translation task: chrF (nrefs:1lcase:mixedleff:yeslnc:6lnw:0lspace:nolversion:2.2.1), BLEU (nrefs:1lcase:mixedleff:noltok:13alsmooth:explversion:2.2.1), COMET (Unbabel/wmt22-comet-da).</caption><thead><tr><th>System</th><th>BLEU</th></tr></thead><tbody><tr><td>ONLINE-W</td><td>59.4</td></tr><tr><td>ONLINE-B</td><td>50.1</td></tr><tr><td>ONLINE-A</td><td>43.4</td></tr><tr><td>CUNI-GA</td><td>43.3</td></tr><tr><td>ZengHuiMT</td><td>43.1</td></tr><tr><td>CUNI-DocTransformer</td><td>42.5</td></tr><tr><td>GTCOM_Peter</td><td>42.3</td></tr><tr><td>CUNI-Transformer</td><td>41.4</td></tr><tr><td>ONLINE-Y</td><td>40.8</td></tr><tr><td>Lan-BridgeMT</td><td>40.7</td></tr><tr><td>ONLINE-G</td><td>39.6</td></tr><tr><td>ONLINE-M</td><td>39.6</td></tr><tr><td>GPT4-5shot</td><td>37.8</td></tr><tr><td>NLLB_Greedy</td><td>35.9</td></tr><tr><td>NLLB_MBR_BLEU</td><td>35.1</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>COMET</th></tr></thead><tbody><tr><td>GPT4-5shot</td><td>86.3</td></tr><tr><td>ONLINE-W</td><td>86.0</td></tr><tr><td>ONLINE-B</td><td>85.6</td></tr><tr><td>ONLINE-A</td><td>85.5</td></tr><tr><td>ONLINE-Y</td><td>84.9</td></tr><tr><td>ONLINE-M</td><td>84.8</td></tr><tr><td>ONLINE-G</td><td>84.6</td></tr><tr><td>GTCOM_Peter</td><td>82.7</td></tr><tr><td>NLLB_MBR_BLEU</td><td>81.4</td></tr><tr><td>ZengHuiMT</td><td>81.1</td></tr><tr><td>Lan-BridgeMT</td><td>80.9</td></tr><tr><td>NLLB_Greedy</td><td>79.9</td></tr><tr><td>AIRC</td><td>78.7</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>chrF</th></tr></thead><tbody><tr><td>ONLINE-W</td><td>72.1</td></tr><tr><td>ONLINE-A</td><td>70.0</td></tr><tr><td>GPT4-5shot</td><td>69.8</td></tr><tr><td>ONLINE-B</td><td>69.1</td></tr><tr><td>ONLINE-G</td><td>69.1</td></tr><tr><td>ONLINE-Y</td><td>68.4</td></tr><tr><td>ZengHuiMT</td><td>67.6</td></tr><tr><td>Lan-BridgeMT</td><td>66.7</td></tr><tr><td>GTCOM_Peter</td><td>66.6</td></tr><tr><td>ONLINE-M</td><td>66.5</td></tr><tr><td>NLLB_MBR_BLEU</td><td>57.6</td></tr><tr><td>NLLB_Greedy</td><td>57.3</td></tr><tr><td>AIRC</td><td>57.2</td></tr></tbody></table>

<table><caption>**Table 14:** Scores for the de→en translation task: chrF (nrefs:1lcase:mixedleff:yeslnc:6lnw:0lspace:nolversion:2.2.1), BLEU (nrefs:1lcase:mixedleff:noltok:13alsmooth:explversion:2.2.1), COMET (Unbabel/wmt22-comet-da).</caption><thead><tr><th>System</th><th>BLEU</th></tr></thead><tbody><tr><td>ONLINE-W</td><td>51.8</td></tr><tr><td>GPT4-5shot</td><td>47.9</td></tr><tr><td>ONLINE-A</td><td>47.9</td></tr><tr><td>ONLINE-B</td><td>46.3</td></tr><tr><td>ONLINE-G</td><td>46.0</td></tr><tr><td>ONLINE-Y</td><td>43.9</td></tr><tr><td>GTCOM_Peter</td><td>42.2</td></tr><tr><td>Lan-BridgeMT</td><td>42.1</td></tr><tr><td>ONLINE-M</td><td>41.3</td></tr><tr><td>ZengHuiMT</td><td>40.8</td></tr><tr><td>NLLB_Greedy</td><td>33.1</td></tr><tr><td>AIRC</td><td>32.4</td></tr><tr><td>NLLB_MBR_BLEU</td><td>32.4</td></tr></tbody></table>
<table><thead><tr><th>System</th><th>COMET</th></tr></thead><tbody><tr><td>ONLINE-W</td><td>85.5</td></tr><tr><td>GPT4-5shot</td><td>85.0</td></tr><tr><td>ONLINE-B</td><td>84.8</td></tr><tr><td>ONLINE-Y</td><td>84.1</td></tr><tr><td>ONLINE-A</td><td>83.7</td></tr><tr><td>ONLINE-G</td><td>82.5</td></tr><tr><td>ONLINE-M</td><td>81.7</td></tr><tr><td>Lan-BridgeMT</td><td>80.4</td></tr><tr><td>ZengHuiMT</td><td>79.4</td></tr><tr><td>NLLB_MBR_BLEU</td><td>78.0</td></tr><tr><td>NLLB_Greedy</td><td>77.9</td></tr><tr><td>AIRC</td><td>72.9</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>chrF</th></tr></thead><tbody><tr><td>ONLINE-W</td><td>71.8</td></tr><tr><td>ONLINE-A</td><td>69.7</td></tr><tr><td>ZengHuiMT</td><td>69.4</td></tr><tr><td>GPT4-5shot</td><td>69.1</td></tr><tr><td>ONLINE-B</td><td>69.1</td></tr><tr><td>ONLINE-Y</td><td>69.1</td></tr><tr><td>ONLINE-G</td><td>69.0</td></tr><tr><td>ONLINE-M</td><td>66.9</td></tr><tr><td>Lan-BridgeMT</td><td>66.1</td></tr><tr><td>NLLB_Greedy</td><td>56.2</td></tr><tr><td>NLLB_MBR_BLEU</td><td>55.4</td></tr><tr><td>AIRC</td><td>52.2</td></tr></tbody></table>

<table><caption>**Table 15:** Scores for the en→de translation task: chrF (nrefs:1lcase:mixedleff:yeslnc:6lnw:0lspace:nolversion:2.2.1), BLEU (nrefs:1lcase:mixedleff:noltok:13alsmooth:explversion:2.2.1), COMET (Unabel/wmt22-comet-da).</caption><thead><tr><th>System</th><th>BLEU</th></tr></thead><tbody><tr><td>ONLINE-W</td><td>47.8</td></tr><tr><td>ONLINE-A</td><td>43.7</td></tr><tr><td>GPT4-5shot</td><td>43.6</td></tr><tr><td>ONLINE-Y</td><td>43.6</td></tr><tr><td>ONLINE-G</td><td>43.2</td></tr><tr><td>ONLINE-B</td><td>42.7</td></tr><tr><td>ONLINE-M</td><td>40.5</td></tr><tr><td>ZengHuiMT</td><td>40.5</td></tr><tr><td>Lan-BridgeMT</td><td>39.4</td></tr><tr><td>NLLB_Greedy</td><td>31.1</td></tr><tr><td>NLLB_MBR_BLEU</td><td>29.6</td></tr><tr><td>AIRC</td><td>26.5</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>COMET</th></tr></thead><tbody><tr><td>ONLINE-B</td><td>89.9</td></tr><tr><td>ONLINE-A</td><td>87.0</td></tr><tr><td>GPT4-5shot</td><td>86.9</td></tr><tr><td>GTCOM_Peter</td><td>86.7</td></tr><tr><td>ONLINE-G</td><td>85.6</td></tr><tr><td>ZengHuiMT</td><td>85.6</td></tr><tr><td>ONLINE-Y</td><td>84.9</td></tr><tr><td>UvA-LTL</td><td>84.7</td></tr><tr><td>NLLB_MBR_BLEU</td><td>82.9</td></tr><tr><td>NLLB_Greedy</td><td>82.8</td></tr><tr><td>Samsung_Research_Philippines</td><td>82.6</td></tr><tr><td>Lan-BridgeMT</td><td>82.4</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>chrF</th></tr></thead><tbody><tr><td>ONLINE-B</td><td>87.5</td></tr><tr><td>ZengHuiMT</td><td>76.3</td></tr><tr><td>GTCOM_Peter</td><td>76.2</td></tr><tr><td>ONLINE-A</td><td>73.3</td></tr><tr><td>GPT4-5shot</td><td>71.4</td></tr><tr><td>UvA-LTL</td><td>70.9</td></tr><tr><td>ONLINE-Y</td><td>70.5</td></tr><tr><td>ONLINE-G</td><td>69.8</td></tr><tr><td>NLLB_Greedy</td><td>64.4</td></tr><tr><td>Lan-BridgeMT</td><td>63.5</td></tr><tr><td>NLLB_MBR_BLEU</td><td>63.0</td></tr><tr><td>Samsung_Research_Philippines</td><td>55.5</td></tr></tbody></table>

<table><caption>**Table 16:** Scores for the he→en (refA) translation task: chrF (nrefs:1lcase:mixedleff:yeslnc:6lnw:0lspace:nolversion:2.2.1), BLEU (nrefs:1lcase:mixedleff:noltok:13alsmooth:explversion:2.2.1), COMET (Unabel/wmt22-comet-da).</caption><thead><tr><th>System</th><th>BLEU</th></tr></thead><tbody><tr><td>ONLINE-B</td><td>76.5</td></tr><tr><td>GTCOM_Peter</td><td>59.2</td></tr><tr><td>ZengHuiMT</td><td>56.6</td></tr><tr><td>ONLINE-A</td><td>53.9</td></tr><tr><td>GPT4-5shot</td><td>51.2</td></tr><tr><td>UvA-LTL</td><td>51.0</td></tr><tr><td>ONLINE-Y</td><td>49.8</td></tr><tr><td>ONLINE-G</td><td>49.3</td></tr><tr><td>NLLB_Greedy</td><td>42.5</td></tr><tr><td>Lan-BridgeMT</td><td>41.4</td></tr><tr><td>NLLB_MBR_BLEU</td><td>40.7</td></tr><tr><td>Samsung_Research_Philippines</td><td>34.0</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>COMET</th></tr></thead><tbody><tr><td>GPT4-5shot</td><td>86.4</td></tr><tr><td>ONLINE-B</td><td>85.6</td></tr><tr><td>ONLINE-A</td><td>85.3</td></tr><tr><td>GTCOM_Peter</td><td>84.5</td></tr><tr><td>ONLINE-G</td><td>84.0</td></tr><tr><td>UvA-LTL</td><td>83.3</td></tr><tr><td>ZengHuiMT</td><td>83.3</td></tr><tr><td>ONLINE-Y</td><td>82.9</td></tr><tr><td>NLLB_MBR_BLEU</td><td>81.8</td></tr><tr><td>NLLB_Greedy</td><td>81.7</td></tr><tr><td>Lan-BridgeMT</td><td>81.3</td></tr><tr><td>Samsung_Research_Philippines</td><td>81.3</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>chrF</th></tr></thead><tbody><tr><td>GPT4-5shot</td><td>69.5</td></tr><tr><td>ONLINE-B</td><td>66.5</td></tr><tr><td>ONLINE-A</td><td>65.6</td></tr><tr><td>GTCOM_Peter</td><td>65.3</td></tr><tr><td>ZengHuiMT</td><td>65.1</td></tr><tr><td>UvA-LTL</td><td>63.3</td></tr><tr><td>ONLINE-G</td><td>62.8</td></tr><tr><td>ONLINE-Y</td><td>62.0</td></tr><tr><td>NLLB_Greedy</td><td>59.6</td></tr><tr><td>Lan-BridgeMT</td><td>59.0</td></tr><tr><td>NLLB_MBR_BLEU</td><td>58.6</td></tr><tr><td>Samsung_Research_Philippines</td><td>51.3</td></tr></tbody></table>

<table><caption>**Table 17:** Scores for the he→en (refB) translation task: chrF (nrefs:1lcase:mixedleff:yeslnc:6lnw:0lspace:nolversion:2.3.1), BLEU (nrefs:1lcase:mixedleff:noltok:13alsmooth:explversion:2.3.1), COMET (Unabel/wmt22-comet-da).</caption><thead><tr><th>System</th><th>BLEU</th></tr></thead><tbody><tr><td>GPT4-5shot</td><td>50.4</td></tr><tr><td>ONLINE-B</td><td>45.0</td></tr><tr><td>GTCOM_Peter</td><td>44.4</td></tr><tr><td>ONLINE-A</td><td>44.4</td></tr><tr><td>UvA-LTL</td><td>41.7</td></tr><tr><td>ZengHuiMT</td><td>41.7</td></tr><tr><td>ONLINE-G</td><td>40.9</td></tr><tr><td>ONLINE-Y</td><td>38.5</td></tr><tr><td>NLLB_Greedy</td><td>37.1</td></tr><tr><td>Lan-BridgeMT</td><td>36.2</td></tr><tr><td>NLLB_MBR_BLEU</td><td>36.2</td></tr><tr><td>Samsung_Research_Philippines</td><td>29.8</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>COMET</th></tr></thead><tbody><tr><td>ONLINE-B</td><td>86.4</td></tr><tr><td>ONLINE-A</td><td>85.7</td></tr><tr><td>GPT4-5shot</td><td>84.9</td></tr><tr><td>GTCOM_Peter</td><td>84.7</td></tr><tr><td>ONLINE-Y</td><td>84.7</td></tr><tr><td>UvA-LTL</td><td>84.2</td></tr><tr><td>Samsung_Research_Philippines</td><td>83.7</td></tr><tr><td>Lan-BridgeMT</td><td>83.0</td></tr><tr><td>NLLB_Greedy</td><td>82.9</td></tr><tr><td>ZengHuiMT</td><td>82.7</td></tr><tr><td>NLLB_MBR_BLEU</td><td>82.5</td></tr><tr><td>ONLINE-G</td><td>82.2</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>chrF</th></tr></thead><tbody><tr><td>ONLINE-B</td><td>66.4</td></tr><tr><td>ZengHuiMT</td><td>62.1</td></tr><tr><td>ONLINE-A</td><td>61.7</td></tr><tr><td>GTCOM_Peter</td><td>61.1</td></tr><tr><td>ONLINE-Y</td><td>60.4</td></tr><tr><td>UvA-LTL</td><td>59.0</td></tr><tr><td>ONLINE-G</td><td>58.1</td></tr><tr><td>Samsung_Research_Philippines</td><td>57.3</td></tr><tr><td>Lan-BridgeMT</td><td>54.9</td></tr><tr><td>NLLB_Greedy</td><td>54.8</td></tr><tr><td>NLLB_MBR_BLEU</td><td>54.3</td></tr><tr><td>GPT4-5shot</td><td>54.0</td></tr></tbody></table>

<table><caption>**Table 18:** Scores for the en→he translation task: chrF (nrefs:1lcase:mixedleff:yeslnc:6lnw:0lspace:nolversion:2.2.1), BLEU (nrefs:1lcase:mixedleff:noltok:13alsmooth:explversion:2.2.1), COMET (Unabel/wmt22-comet-da).</caption><thead><tr><th>System</th><th>BLEU</th></tr></thead><tbody><tr><td>ONLINE-B</td><td>47.8</td></tr><tr><td>ONLINE-A</td><td>38.9</td></tr><tr><td>GTCOM_Peter</td><td>37.2</td></tr><tr><td>ONLINE-Y</td><td>37.2</td></tr><tr><td>ZengHuiMT</td><td>36.5</td></tr><tr><td>UvA-LTL</td><td>35.0</td></tr><tr><td>Samsung_Research_Philippines</td><td>33.3</td></tr><tr><td>ONLINE-G</td><td>33.2</td></tr><tr><td>NLLB_MBR_BLEU</td><td>30.8</td></tr><tr><td>Lan-BridgeMT</td><td>30.5</td></tr><tr><td>NLLB_Greedy</td><td>30.3</td></tr><tr><td>GPT4-5shot</td><td>27.0</td></tr></tbody></table>
<table><thead><tr><th>System</th><th>COMET</th></tr></thead><tbody><tr><td>SKIM</td><td>84.0</td></tr><tr><td>GPT4-5shot</td><td>83.4</td></tr><tr><td>ONLINE-W</td><td>82.3</td></tr><tr><td>NAIST-NICT</td><td>81.9</td></tr><tr><td>ONLINE-Y</td><td>81.6</td></tr><tr><td>ONLINE-B</td><td>81.5</td></tr><tr><td>ONLINE-A</td><td>81.0</td></tr><tr><td>GTCOM_Peter</td><td>80.2</td></tr><tr><td>ANVITA</td><td>79.5</td></tr><tr><td>Lan-BridgeMT</td><td>79.3</td></tr><tr><td>ZengHuiMT</td><td>79.2</td></tr><tr><td>ONLINE-G</td><td>77.8</td></tr><tr><td>ONLINE-M</td><td>77.5</td></tr><tr><td>KYB</td><td>76.6</td></tr><tr><td>NLLB_MBR_BLEU</td><td>75.2</td></tr><tr><td>AIRC</td><td>74.5</td></tr><tr><td>NLLB_Greedy</td><td>74.3</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>chrF</th></tr></thead><tbody><tr><td>ONLINE-W</td><td>51.4</td></tr><tr><td>GPT4-5shot</td><td>51.2</td></tr><tr><td>SKIM</td><td>51.1</td></tr><tr><td>ONLINE-A</td><td>49.6</td></tr><tr><td>NAIST-NICT</td><td>49.5</td></tr><tr><td>ONLINE-Y</td><td>49.5</td></tr><tr><td>ZengHuiMT</td><td>49.5</td></tr><tr><td>ONLINE-B</td><td>49.3</td></tr><tr><td>GTCOM_Peter</td><td>48.7</td></tr><tr><td>Lan-BridgeMT</td><td>47.3</td></tr><tr><td>ANVITA</td><td>46.7</td></tr><tr><td>ONLINE-G</td><td>45.5</td></tr><tr><td>KYB</td><td>43.9</td></tr><tr><td>ONLINE-M</td><td>43.9</td></tr><tr><td>AIRC</td><td>40.5</td></tr><tr><td>NLLB_MBR_BLEU</td><td>39.2</td></tr><tr><td>NLLB_Greedy</td><td>39.0</td></tr></tbody></table>

<table><caption>**Table 19:** Scores for the ja→en translation task: chrF (nrefs:1lcase:mixedleff:yeslnc:6lnw:0lspace:nolversion:2.2.1), BLEU (nrefs:1lcase:mixedleff:noltok:13alsmooth:explversion:2.2.1), COMET (Unabel/wmt22-comet-da).</caption><thead><tr><th>System</th><th>BLEU</th></tr></thead><tbody><tr><td>ONLINE-W</td><td>25.9</td></tr><tr><td>SKIM</td><td>24.8</td></tr><tr><td>GPT4-5shot</td><td>24.1</td></tr><tr><td>ONLINE-B</td><td>23.9</td></tr><tr><td>NAIST-NICT</td><td>23.0</td></tr><tr><td>ONLINE-A</td><td>23.0</td></tr><tr><td>ZengHuiMT</td><td>22.6</td></tr><tr><td>GTCOM_Peter</td><td>22.3</td></tr><tr><td>ONLINE-Y</td><td>22.3</td></tr><tr><td>ANVITA</td><td>20.9</td></tr><tr><td>Lan-BridgeMT</td><td>20.2</td></tr><tr><td>ONLINE-G</td><td>18.3</td></tr><tr><td>KYB</td><td>17.6</td></tr><tr><td>ONLINE-M</td><td>17.2</td></tr><tr><td>AIRC</td><td>14.9</td></tr><tr><td>NLLB_MBR_BLEU</td><td>14.7</td></tr><tr><td>NLLB_Greedy</td><td>14.2</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>COMET</th></tr></thead><tbody><tr><td>ONLINE-B</td><td>88.2</td></tr><tr><td>ONLINE-W</td><td>87.5</td></tr><tr><td>ONLINE-Y</td><td>87.3</td></tr><tr><td>GPT4-5shot</td><td>87.0</td></tr><tr><td>SKIM</td><td>86.6</td></tr><tr><td>NAIST-NICT</td><td>86.2</td></tr><tr><td>ZengHuiMT</td><td>85.3</td></tr><tr><td>ONLINE-A</td><td>85.2</td></tr><tr><td>Lan-BridgeMT</td><td>84.5</td></tr><tr><td>ONLINE-M</td><td>13.9</td></tr><tr><td>ANVITA</td><td>82.7</td></tr><tr><td>KYB</td><td>80.8</td></tr><tr><td>AIRC</td><td>80.7</td></tr><tr><td>ONLINE-G</td><td>80.4</td></tr><tr><td>NLLB_Greedy</td><td>79.3</td></tr><tr><td>NLLB_MBR_BLEU</td><td>77.7</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>chrF</th></tr></thead><tbody><tr><td>ONLINE-B</td><td>35.2</td></tr><tr><td>ONLINE-Y</td><td>34.1</td></tr><tr><td>ONLINE-W</td><td>33.5</td></tr><tr><td>SKIM</td><td>33.5</td></tr><tr><td>ZengHuiMT</td><td>32.9</td></tr><tr><td>NAIST-NICT</td><td>32.0</td></tr><tr><td>ONLINE-A</td><td>31.4</td></tr><tr><td>GPT4-5shot</td><td>31.0</td></tr><tr><td>Lan-BridgeMT</td><td>30.4</td></tr><tr><td>ONLINE-M</td><td>29.6</td></tr><tr><td>ANVITA</td><td>29.3</td></tr><tr><td>KYB</td><td>27.7</td></tr><tr><td>AIRC</td><td>27.6</td></tr><tr><td>ONLINE-G</td><td>27.3</td></tr><tr><td>NLLB_Greedy</td><td>20.9</td></tr><tr><td>NLLB_MBR_BLEU</td><td>18.7</td></tr></tbody></table>

<table><caption>**Table 20:** Scores for the en→ja translation task: chrF (nrefs:1lcase:mixedleff:yeslnc:6lnw:0lspace:nolversion:2.2.1), BLEU (nrefs:1lcase:mixedleff:noltok:ja-mecab-0.996-IPAlsmooth:explversion:2.2.1), COMET (Unabel/wmt22-comet-da).</caption><thead><tr><th>System</th><th>BLEU</th></tr></thead><tbody><tr><td>ONLINE-B</td><td>25.3</td></tr><tr><td>ONLINE-W</td><td>24.5</td></tr><tr><td>ONLINE-Y</td><td>24.5</td></tr><tr><td>SKIM</td><td>24.3</td></tr><tr><td>NAIST-NICT</td><td>22.6</td></tr><tr><td>ZengHuiMT</td><td>22.6</td></tr><tr><td>ONLINE-A</td><td>21.4</td></tr><tr><td>GPT4-5shot</td><td>21.3</td></tr><tr><td>Lan-BridgeMT</td><td>20.5</td></tr><tr><td>ONLINE-M</td><td>19.8</td></tr><tr><td>ANVITA</td><td>19.4</td></tr><tr><td>KYB</td><td>17.8</td></tr><tr><td>AIRC</td><td>17.6</td></tr><tr><td>ONLINE-G</td><td>17.2</td></tr><tr><td>NLLB_Greedy</td><td>11.3</td></tr><tr><td>NLLB_MBR_BLEU</td><td>9.0</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>COMET</th></tr></thead><tbody><tr><td>GPT4-5shot</td><td>83.5</td></tr><tr><td>ONLINE-Y</td><td>82.5</td></tr><tr><td>ONLINE-B</td><td>82.3</td></tr><tr><td>ONLINE-W</td><td>82.2</td></tr><tr><td>ONLINE-G</td><td>82.0</td></tr><tr><td>ONLINE-A</td><td>81.9</td></tr><tr><td>PROMT</td><td>80.9</td></tr><tr><td>ONLINE-M</td><td>80.7</td></tr><tr><td>NLLB_MBR_BLEU</td><td>80.5</td></tr><tr><td>NLLB_Greedy</td><td>80.1</td></tr><tr><td>Lan-BridgeMT</td><td>79.9</td></tr><tr><td>ZengHuiMT</td><td>79.5</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>chrF</th></tr></thead><tbody><tr><td>GPT4-5shot</td><td>60.4</td></tr><tr><td>ONLINE-G</td><td>59.6</td></tr><tr><td>ONLINE-A</td><td>59.4</td></tr><tr><td>ONLINE-B</td><td>59.4</td></tr><tr><td>ZengHuiMT</td><td>58.9</td></tr><tr><td>ONLINE-Y</td><td>58.6</td></tr><tr><td>PROMT</td><td>58.4</td></tr><tr><td>ONLINE-W</td><td>58.3</td></tr><tr><td>Lan-BridgeMT</td><td>57.4</td></tr><tr><td>ONLINE-M</td><td>56.7</td></tr><tr><td>NLLB_MBR_BLEU</td><td>55.8</td></tr><tr><td>NLLB_Greedy</td><td>55.5</td></tr></tbody></table>

<table><caption>**Table 21:** Scores for the ru→en translation task: chrF (nrefs:1lcase:mixedleff:yeslnc:6lnw:0lspace:nolversion:2.2.1), BLEU (nrefs:1lcase:mixedleff:noltok:13alsmooth:explversion:2.2.1), COMET (Unabel/wmt22-comet-da).</caption><thead><tr><th>System</th><th>BLEU</th></tr></thead><tbody><tr><td>ONLINE-B</td><td>34.5</td></tr><tr><td>GPT4-5shot</td><td>34.4</td></tr><tr><td>ONLINE-G</td><td>34.0</td></tr><tr><td>ONLINE-A</td><td>33.8</td></tr><tr><td>ONLINE-Y</td><td>33.2</td></tr><tr><td>ONLINE-W</td><td>33.1</td></tr><tr><td>PROMT</td><td>32.8</td></tr><tr><td>Lan-BridgeMT</td><td>31.8</td></tr><tr><td>ZengHuiMT</td><td>31.3</td></tr><tr><td>NLLB_MBR_BLEU</td><td>31.0</td></tr><tr><td>ONLINE-M</td><td>30.7</td></tr><tr><td>NLLB_Greedy</td><td>30.3</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>COMET</th></tr></thead><tbody><tr><td>ONLINE-G</td><td>86.6</td></tr><tr><td>ONLINE-W</td><td>86.6</td></tr><tr><td>ONLINE-B</td><td>86.2</td></tr><tr><td>GPT4-5shot</td><td>86.1</td></tr><tr><td>ONLINE-Y</td><td>85.5</td></tr><tr><td>ONLINE-A</td><td>85.3</td></tr><tr><td>ONLINE-M</td><td>83.2</td></tr><tr><td>Lan-BridgeMT</td><td>83.1</td></tr><tr><td>NLLB_Greedy</td><td>82.9</td></tr><tr><td>NLLB_MBR_BLEU</td><td>82.7</td></tr><tr><td>PROMT</td><td>82.3</td></tr><tr><td>ZengHuiMT</td><td>81.3</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>chrF</th></tr></thead><tbody><tr><td>ONLINE-B</td><td>61.9</td></tr><tr><td>ONLINE-A</td><td>59.0</td></tr><tr><td>ONLINE-G</td><td>58.9</td></tr><tr><td>ZengHuiMT</td><td>58.8</td></tr><tr><td>ONLINE-W</td><td>56.6</td></tr><tr><td>ONLINE-Y</td><td>56.4</td></tr><tr><td>GPT4-5shot</td><td>56.2</td></tr><tr><td>Lan-BridgeMT</td><td>55.7</td></tr><tr><td>PROMT</td><td>55.4</td></tr><tr><td>ONLINE-M</td><td>55.1</td></tr><tr><td>NLLB_Greedy</td><td>53.3</td></tr><tr><td>NLLB_MBR_BLEU</td><td>53.1</td></tr></tbody></table>

<table><caption>**Table 22:** Scores for the en→ru translation task: chrF (nrefs:1lcase:mixedleff:yeslnc:6lnw:0lspace:nolversion:2.2.1), BLEU (nrefs:1lcase:mixedleff:noltok:13alsmooth:explversion:2.2.1), COMET (Unabel/wmt22-comet-da).</caption><thead><tr><th>System</th><th>BLEU</th></tr></thead><tbody><tr><td>ONLINE-B</td><td>40.4</td></tr><tr><td>ONLINE-A</td><td>34.8</td></tr><tr><td>ONLINE-G</td><td>32.9</td></tr><tr><td>ONLINE-Y</td><td>32.0</td></tr><tr><td>ZengHuiMT</td><td>31.6</td></tr><tr><td>ONLINE-W</td><td>31.4</td></tr><tr><td>ONLINE-M</td><td>30.9</td></tr><tr><td>Lan-BridgeMT</td><td>30.7</td></tr><tr><td>GPT4-5shot</td><td>30.6</td></tr><tr><td>PROMT</td><td>30.5</td></tr><tr><td>NLLB_MBR_BLEU</td><td>28.4</td></tr><tr><td>NLLB_Greedy</td><td>28.2</td></tr></tbody></table>
<table><thead><tr><th>System</th><th>COMET</th></tr></thead><tbody><tr><td>ONLINE-W</td><td>87.5</td></tr><tr><td>GPT4-5shot</td><td>87.1</td></tr><tr><td>ONLINE-B</td><td>86.8</td></tr><tr><td>GTCOM_Peter</td><td>86.3</td></tr><tr><td>ONLINE-A</td><td>86.3</td></tr><tr><td>ONLINE-G</td><td>86.2</td></tr><tr><td>ONLINE-Y</td><td>85.8</td></tr><tr><td>Lan-BridgeMT</td><td>84.8</td></tr><tr><td>ZengHuiMT</td><td>84.4</td></tr><tr><td>NLLB_MBR_BLEU</td><td>84.3</td></tr><tr><td>NLLB_Greedy</td><td>84.2</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>chrF</th></tr></thead><tbody><tr><td>GTCOM_Peter</td><td>69.3</td></tr><tr><td>ONLINE-W</td><td>69.2</td></tr><tr><td>ONLINE-B</td><td>69.0</td></tr><tr><td>ZengHuiMT</td><td>68.5</td></tr><tr><td>ONLINE-A</td><td>68.3</td></tr><tr><td>ONLINE-Y</td><td>68.2</td></tr><tr><td>GPT4-5shot</td><td>68.1</td></tr><tr><td>ONLINE-G</td><td>68.0</td></tr><tr><td>Lan-BridgeMT</td><td>66.2</td></tr><tr><td>NLLB_Greedy</td><td>62.4</td></tr><tr><td>NLLB_MBR_BLEU</td><td>62.4</td></tr></tbody></table>

<table><caption>**Table 23:** Scores for the uk→en translation task: chrF (nrefs: 1lcase:mixedleff:yeslnc:6lnw:0lspace:nolversion:2.2.1), BLEU (nrefs: 1lcase:mixedleff:noltok:13alsmooth:explversion:2.2.1), COMET (Unlabel/wmt22-comet-da).</caption><thead><tr><th>System</th><th>BLEU</th></tr></thead><tbody><tr><td>ONLINE-W</td><td>47.4</td></tr><tr><td>GTCOM_Peter</td><td>46.4</td></tr><tr><td>ONLINE-B</td><td>46.0</td></tr><tr><td>ONLINE-A</td><td>45.9</td></tr><tr><td>ONLINE-Y</td><td>45.7</td></tr><tr><td>ONLINE-G</td><td>44.9</td></tr><tr><td>GPT4-5shot</td><td>43.9</td></tr><tr><td>ZengHuiMT</td><td>43.5</td></tr><tr><td>Lan-BridgeMT</td><td>42.3</td></tr><tr><td>NLLB_MBR_BLEU</td><td>38.1</td></tr><tr><td>NLLB_Greedy</td><td>37.8</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>COMET</th></tr></thead><tbody><tr><td>ONLINE-W</td><td>86.7</td></tr><tr><td>ONLINE-B</td><td>85.6</td></tr><tr><td>GPT4-5shot</td><td>85.3</td></tr><tr><td>ONLINE-G</td><td>85.3</td></tr><tr><td>ONLINE-A</td><td>83.2</td></tr><tr><td>ONLINE-Y</td><td>82.9</td></tr><tr><td>GTCOM_Peter</td><td>82.1</td></tr><tr><td>NLLB_Greedy</td><td>82.1</td></tr><tr><td>NLLB_MBR_BLEU</td><td>81.7</td></tr><tr><td>Lan-BridgeMT</td><td>80.4</td></tr><tr><td>ZengHuiMT</td><td>79.0</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>chrF</th></tr></thead><tbody><tr><td>ONLINE-B</td><td>61.7</td></tr><tr><td>ONLINE-W</td><td>59.2</td></tr><tr><td>ZengHuiMT</td><td>56.4</td></tr><tr><td>ONLINE-G</td><td>56.1</td></tr><tr><td>ONLINE-A</td><td>55.8</td></tr><tr><td>ONLINE-Y</td><td>55.4</td></tr><tr><td>GTCOM_Peter</td><td>54.4</td></tr><tr><td>GPT4-5shot</td><td>53.0</td></tr><tr><td>Lan-BridgeMT</td><td>51.9</td></tr><tr><td>NLLB_Greedy</td><td>50.8</td></tr><tr><td>NLLB_MBR_BLEU</td><td>50.5</td></tr></tbody></table>

<table><caption>**Table 24:** Scores for the en→uk translation task: chrF (nrefs: 1lcase:mixedleff:yeslnc:6lnw:0lspace:nolversion:2.2.1), BLEU (nrefs: 1lcase:mixedleff:noltok:13alsmooth:explversion:2.2.1), COMET (Unlabel/wmt22-comet-da).</caption><thead><tr><th>System</th><th>BLEU</th></tr></thead><tbody><tr><td>ONLINE-B</td><td>39.8</td></tr><tr><td>ONLINE-W</td><td>34.9</td></tr><tr><td>ONLINE-A</td><td>30.3</td></tr><tr><td>ONLINE-Y</td><td>29.5</td></tr><tr><td>ONLINE-G</td><td>28.6</td></tr><tr><td>ZengHuiMT</td><td>27.8</td></tr><tr><td>GTCOM_Peter</td><td>27.5</td></tr><tr><td>GPT4-5shot</td><td>25.2</td></tr><tr><td>NLLB_MBR_BLEU</td><td>24.9</td></tr><tr><td>Lan-BridgeMT</td><td>24.6</td></tr><tr><td>NLLB_Greedy</td><td>24.5</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>COMET</th></tr></thead><tbody><tr><td>HW-TSC</td><td>82.8</td></tr><tr><td>ONLINE-B</td><td>82.7</td></tr><tr><td>Yishu</td><td>82.7</td></tr><tr><td>GPT4-5shot</td><td>81.6</td></tr><tr><td>Lan-BridgeMT</td><td>81.2</td></tr><tr><td>ONLINE-G</td><td>80.9</td></tr><tr><td>ONLINE-Y</td><td>80.6</td></tr><tr><td>ONLINE-A</td><td>80.3</td></tr><tr><td>ZengHuiMT</td><td>79.6</td></tr><tr><td>ONLINE-W</td><td>79.3</td></tr><tr><td>IOL_Research</td><td>79.2</td></tr><tr><td>ONLINE-M</td><td>77.7</td></tr><tr><td>NLLB_MBR_BLEU</td><td>76.8</td></tr><tr><td>ANVITA</td><td>76.6</td></tr><tr><td>NLLB_Greedy</td><td>76.4</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>chrF</th></tr></thead><tbody><tr><td>HW-TSC</td><td>57.5</td></tr><tr><td>ONLINE-B</td><td>57.5</td></tr><tr><td>Yishu</td><td>57.4</td></tr><tr><td>ZengHuiMT</td><td>54.6</td></tr><tr><td>ONLINE-G</td><td>53.9</td></tr><tr><td>ONLINE-A</td><td>53.4</td></tr><tr><td>GPT4-5shot</td><td>53.1</td></tr><tr><td>Lan-BridgeMT</td><td>53.1</td></tr><tr><td>ONLINE-W</td><td>52.5</td></tr><tr><td>IOL_Research</td><td>52.4</td></tr><tr><td>ONLINE-Y</td><td>52.3</td></tr><tr><td>ONLINE-M</td><td>49.7</td></tr><tr><td>ANVITA</td><td>47.1</td></tr><tr><td>NLLB_Greedy</td><td>46.1</td></tr><tr><td>NLLB_MBR_BLEU</td><td>45.8</td></tr></tbody></table>

<table><caption>**Table 25:** Scores for the zh→en translation task: chrF (nrefs: 1lcase:mixedleff:yeslnc:6lnw:0lspace:nolversion:2.2.1), BLEU (nrefs: 1lcase:mixedleff:noltok:13alsmooth:explversion:2.2.1), COMET (Unlabel/wmt22-comet-da).</caption><thead><tr><th>System</th><th>BLEU</th></tr></thead><tbody><tr><td>HW-TSC</td><td>33.6</td></tr><tr><td>ONLINE-B</td><td>33.5</td></tr><tr><td>Yishu</td><td>33.4</td></tr><tr><td>ONLINE-A</td><td>28.3</td></tr><tr><td>Lan-BridgeMT</td><td>27.3</td></tr><tr><td>IOL_Research</td><td>27.2</td></tr><tr><td>ZengHuiMT</td><td>27.0</td></tr><tr><td>GPT4-5shot</td><td>26.8</td></tr><tr><td>ONLINE-G</td><td>26.6</td></tr><tr><td>ONLINE-W</td><td>26.4</td></tr><tr><td>ONLINE-Y</td><td>25.0</td></tr><tr><td>ONLINE-M</td><td>23.5</td></tr><tr><td>ANVITA</td><td>21.8</td></tr><tr><td>NLLB_Greedy</td><td>20.5</td></tr><tr><td>NLLB_MBR_BLEU</td><td>19.8</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>COMET</th></tr></thead><tbody><tr><td>ONLINE-B</td><td>88.1</td></tr><tr><td>Yishu</td><td>88.1</td></tr><tr><td>HW-TSC</td><td>87.3</td></tr><tr><td>GPT4-5shot</td><td>87.1</td></tr><tr><td>ONLINE-W</td><td>86.8</td></tr><tr><td>Lan-BridgeMT</td><td>86.6</td></tr><tr><td>ONLINE-Y</td><td>86.5</td></tr><tr><td>ONLINE-A</td><td>86.2</td></tr><tr><td>IOL_Research</td><td>85.3</td></tr><tr><td>ZengHuiMT</td><td>84.3</td></tr><tr><td>ONLINE-M</td><td>84.2</td></tr><tr><td>ONLINE-G</td><td>83.8</td></tr><tr><td>NLLB_Greedy</td><td>75.7</td></tr><tr><td>ANVITA</td><td>75.6</td></tr><tr><td>NLLB_MBR_BLEU</td><td>71.5</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>chrF</th></tr></thead><tbody><tr><td>HW-TSC</td><td>53.8</td></tr><tr><td>Yishu</td><td>53.0</td></tr><tr><td>ONLINE-B</td><td>52.9</td></tr><tr><td>ONLINE-A</td><td>52.8</td></tr><tr><td>IOL_Research</td><td>51.9</td></tr><tr><td>ONLINE-M</td><td>50.6</td></tr><tr><td>ONLINE-Y</td><td>49.8</td></tr><tr><td>ONLINE-G</td><td>49.4</td></tr><tr><td>ONLINE-W</td><td>47.3</td></tr><tr><td>ZengHuiMT</td><td>47.0</td></tr><tr><td>Lan-BridgeMT</td><td>46.8</td></tr><tr><td>GPT4-5shot</td><td>46.5</td></tr><tr><td>ANVITA</td><td>36.9</td></tr><tr><td>NLLB_Greedy</td><td>26.3</td></tr><tr><td>NLLB_MBR_BLEU</td><td>21.1</td></tr></tbody></table>

<table><caption>**Table 26:** Scores for the en→zh translation task: chrF (nrefs: 1lcase:mixedleff:yeslnc:6lnw:0lspace:nolversion:2.2.1), BLEU (nrefs: 1lcase:mixedleff:noltok:zhlsmooth:explversion:2.2.1), COMET (Unlabel/wmt22-comet-da).</caption><thead><tr><th>System</th><th>BLEU</th></tr></thead><tbody><tr><td>HW-TSC</td><td>58.6</td></tr><tr><td>ONLINE-A</td><td>58.5</td></tr><tr><td>Yishu</td><td>57.6</td></tr><tr><td>ONLINE-B</td><td>57.5</td></tr><tr><td>IOL_Research</td><td>56.9</td></tr><tr><td>ONLINE-M</td><td>54.9</td></tr><tr><td>ONLINE-Y</td><td>54.2</td></tr><tr><td>ONLINE-G</td><td>54.1</td></tr><tr><td>ZengHuiMT</td><td>52.9</td></tr><tr><td>ONLINE-W</td><td>52.1</td></tr><tr><td>Lan-BridgeMT</td><td>50.2</td></tr><tr><td>GPT4-5shot</td><td>49.6</td></tr><tr><td>ANVITA</td><td>38.9</td></tr><tr><td>NLLB_Greedy</td><td>27.4</td></tr><tr><td>NLLB_MBR_BLEU</td><td>19.1</td></tr></tbody></table>
## G Head to head comparisons

Following tables show differences in average human scores for each language pair. The numbers in each of the tables' cells indicate the difference in average human scores for the system in that column and the system in that row.

Because there were so many systems and data conditions the significance of each pairwise comparison needs to be quantified. We applied Wilcoxon rank-sum test to measure the likelihood that such differences could occur simply by chance. In the following tables ⋆ indicates statistical significance at *p* < 0.05, † indicates statistical significance at *p* < 0.01, and ‡ indicates statistical significance at *p* < 0.001, according to Wilcoxon rank-sum test.

Each table contains final rows showing the average score achieved by that system and the rank range according to Wilcoxon rank-sum test (*p* < 0.05). Gray lines separate clusters based on non-overlapping rank ranges.

### Czech→Ukrainian

<table><caption>**Table 27:** Head to head comparison for Czech→Ukrainian systems</caption><thead><tr><th></th><th>ONLINE-B</th><th>GPT4-5shot</th><th>Human-refA</th><th>ONLINE-W</th><th>CUNI-GA</th><th>CUNI-Transformer</th><th>GTCOM_DLUT</th><th>ONLINE-A</th><th>ONLINE-G</th><th>ONLINE-Y</th><th>MUNI-NLP</th><th>Lan-BridgeMT</th><th>NLLB_MBR_BLEU</th><th>NLLB_Greedy</th></tr></thead><tbody><tr><td>ONLINE-B</td><td>—</td><td>0.1</td><td>0.4</td><td>0.9*</td><td>1.3‡</td><td>1.8†</td><td>2.4*</td><td>3.1*</td><td>4.1‡</td><td>5.0‡</td><td>5.0‡</td><td>6.2‡</td><td>6.7‡</td><td>7.0‡</td></tr><tr><td>GPT4-5shot</td><td>-0.1</td><td>—</td><td>0.4</td><td>0.8†</td><td>1.2‡</td><td>1.8‡</td><td>2.3†</td><td>3.1‡</td><td>4.1‡</td><td>4.9‡</td><td>4.9‡</td><td>6.2‡</td><td>6.7‡</td><td>6.9‡</td></tr><tr><td>Human-refA</td><td>-0.4</td><td>-0.4</td><td>—</td><td>0.5‡</td><td>0.9‡</td><td>1.4‡</td><td>1.9‡</td><td>2.7‡</td><td>3.7‡</td><td>4.6‡</td><td>4.6‡</td><td>5.8‡</td><td>6.3‡</td><td>6.6‡</td></tr><tr><td>ONLINE-W</td><td>-0.9</td><td>-0.8</td><td>-0.5</td><td>—</td><td>0.4</td><td>0.9</td><td>1.5</td><td>2.2</td><td>3.2‡</td><td>4.1‡</td><td>4.1‡</td><td>5.3‡</td><td>5.8‡</td><td>6.1‡</td></tr><tr><td>CUNI-GA</td><td>-1.3</td><td>-1.2</td><td>-0.9</td><td>-0.4</td><td>—</td><td>0.6</td><td>1.1</td><td>1.8</td><td>2.9†</td><td>3.7‡</td><td>3.7†</td><td>5.0‡</td><td>5.5‡</td><td>5.7‡</td></tr><tr><td>CUNI-Transformer</td><td>-1.8</td><td>-1.8</td><td>-1.4</td><td>-0.9</td><td>-0.6</td><td>—</td><td>0.5</td><td>1.3</td><td>2.3†</td><td>3.1‡</td><td>3.2‡</td><td>4.4‡</td><td>4.9‡</td><td>5.1‡</td></tr><tr><td>GTCOM_DLUT</td><td>-2.4</td><td>-2.3</td><td>-1.9</td><td>-1.5</td><td>-1.1</td><td>-0.5</td><td>—</td><td>0.8</td><td>1.8‡</td><td>2.6‡</td><td>2.6‡</td><td>3.9‡</td><td>4.4‡</td><td>4.6‡</td></tr><tr><td>ONLINE-A</td><td>-3.1</td><td>-3.1</td><td>-2.7</td><td>-2.2</td><td>-1.8</td><td>-1.3</td><td>-0.8</td><td>—</td><td>1.0‡</td><td>1.9‡</td><td>1.9‡</td><td>3.1‡</td><td>3.6‡</td><td>3.9‡</td></tr><tr><td>ONLINE-G</td><td>-4.1</td><td>-4.1</td><td>-3.7</td><td>-3.2</td><td>-2.9</td><td>-2.3</td><td>-1.8</td><td>-1.0</td><td>—</td><td>0.8</td><td>0.9</td><td>2.1*</td><td>2.6†</td><td>2.8‡</td></tr><tr><td>ONLINE-Y</td><td>-5.0</td><td>-4.9</td><td>-4.6</td><td>-4.1</td><td>-3.7</td><td>-3.1</td><td>-2.6</td><td>-1.9</td><td>-0.8</td><td>—</td><td>0.0</td><td>1.3</td><td>1.8</td><td>2.0†</td></tr><tr><td>MUNI-NLP</td><td>-5.0</td><td>-4.9</td><td>-4.6</td><td>-4.1</td><td>-3.7</td><td>-3.2</td><td>-2.6</td><td>-1.9</td><td>-0.9</td><td>-0.0</td><td>—</td><td>1.2</td><td>1.7</td><td>2.0‡</td></tr><tr><td>Lan-BridgeMT</td><td>-6.2</td><td>-6.2</td><td>-5.8</td><td>-5.3</td><td>-5.0</td><td>-4.4</td><td>-3.9</td><td>-3.1</td><td>-2.1</td><td>-1.3</td><td>-1.2</td><td>—</td><td>0.5</td><td>0.7*</td></tr><tr><td>NLLB_MBR_BLEU</td><td>-6.7</td><td>-6.7</td><td>-6.3</td><td>-5.8</td><td>-5.5</td><td>-4.9</td><td>-4.4</td><td>-3.6</td><td>-2.6</td><td>-1.8</td><td>-1.7</td><td>-0.5</td><td>—</td><td>0.2*</td></tr><tr><td>NLLB_Greedy</td><td>-7.0</td><td>-6.9</td><td>-6.6</td><td>-6.1</td><td>-5.7</td><td>-5.1</td><td>-4.6</td><td>-3.9</td><td>-2.8</td><td>-2.0</td><td>-2.0</td><td>-0.7</td><td>-0.2</td><td>—</td></tr><tr><td>score</td><td>83.7</td><td>83.6</td><td>83.2</td><td>82.8</td><td>82.4</td><td>81.8</td><td>81.3</td><td>80.6</td><td>79.5</td><td>78.7</td><td>78.7</td><td>77.4</td><td>76.9</td><td>76.7</td></tr><tr><td>rank</td><td>1-3</td><td>1-3</td><td>1-3</td><td>4-8</td><td>4-8</td><td>4-8</td><td>4-8</td><td>4-8</td><td>9-11</td><td>9-13</td><td>9-13</td><td>10-13</td><td>10-13</td><td>14</td></tr></tbody></table>
## German→English

<table><caption>Table 28: Head to head comparison for German→English systems</caption><thead><tr><th></th><th>GPT4-5shot</th><th>Human-refA</th><th>ONLINE-A</th><th>ONLINE-B</th><th>ONLINE-W</th><th>ONLINE-Y</th><th>ONLINE-G</th><th>GTCOM_DLUT</th><th>ONLINE-M</th><th>LanguageX</th><th>Lan-BridgeMT</th><th>NLLB_MBR_BLEU</th><th>AIRC</th><th>NLLB_Greedy</th></tr></thead><tbody><tr><td>GPT4-5shot</td><td>—</td><td>0.4</td><td>0.8</td><td>1.2†</td><td>1.5†</td><td>2.3†</td><td>2.6‡</td><td>3.8‡</td><td>5.0‡</td><td>8.5‡</td><td>10.3‡</td><td>10.7‡</td><td>11.5‡</td><td>12.4‡</td></tr><tr><td>Human-refA</td><td>-0.4</td><td>—</td><td>0.4</td><td>0.8*</td><td>1.1*</td><td>1.9†</td><td>2.2‡</td><td>3.4‡</td><td>4.6‡</td><td>8.1‡</td><td>9.9‡</td><td>10.3‡</td><td>11.1‡</td><td>12.0‡</td></tr><tr><td>ONLINE-A</td><td>-0.8</td><td>-0.4</td><td>—</td><td>0.4</td><td>0.7</td><td>1.6*</td><td>1.9†</td><td>3.0‡</td><td>4.2‡</td><td>7.7‡</td><td>9.6‡</td><td>9.9‡</td><td>10.8‡</td><td>11.7‡</td></tr><tr><td>ONLINE-B</td><td>-1.2</td><td>-0.8</td><td>-0.4</td><td>—</td><td>0.3</td><td>1.1</td><td>1.4*</td><td>2.6‡</td><td>3.8‡</td><td>7.3‡</td><td>9.2‡</td><td>9.5‡</td><td>10.3‡</td><td>11.2‡</td></tr><tr><td>ONLINE-W</td><td>-1.5</td><td>-1.1</td><td>-0.7</td><td>-0.3</td><td>—</td><td>0.8</td><td>1.1*</td><td>2.3‡</td><td>3.5‡</td><td>7.0‡</td><td>8.9‡</td><td>9.2‡</td><td>10.0‡</td><td>10.9‡</td></tr><tr><td>ONLINE-Y</td><td>-2.3</td><td>-1.9</td><td>-1.6</td><td>-1.1</td><td>-0.8</td><td>—</td><td>0.3</td><td>1.5‡</td><td>2.7†</td><td>6.2‡</td><td>8.0‡</td><td>8.4‡</td><td>9.2‡</td><td>10.1‡</td></tr><tr><td>ONLINE-G</td><td>-2.6</td><td>-2.2</td><td>-1.9</td><td>-1.4</td><td>-1.1</td><td>-0.3</td><td>—</td><td>1.2‡</td><td>2.4</td><td>5.9‡</td><td>7.7‡</td><td>8.1‡</td><td>8.9‡</td><td>9.8‡</td></tr><tr><td>GTCOM_DLUT</td><td>-3.8</td><td>-3.4</td><td>-3.0</td><td>-2.6</td><td>-2.3</td><td>-1.5</td><td>-1.2</td><td>—</td><td>1.2</td><td>4.7‡</td><td>6.6‡</td><td>6.9‡</td><td>7.8‡</td><td>8.6‡</td></tr><tr><td>ONLINE-M</td><td>-5.0</td><td>-4.6</td><td>-4.2</td><td>-3.8</td><td>-3.5</td><td>-2.7</td><td>-2.4</td><td>-1.2</td><td>—</td><td>3.5‡</td><td>5.3‡</td><td>5.7‡</td><td>6.5‡</td><td>7.4‡</td></tr><tr><td>LanguageX</td><td>-8.5</td><td>-8.1</td><td>-7.7</td><td>-7.3</td><td>-7.0</td><td>-6.2</td><td>-5.9</td><td>-4.7</td><td>-3.5</td><td>—</td><td>1.9</td><td>2.2‡</td><td>3.0‡</td><td>3.9†</td></tr><tr><td>Lan-BridgeMT</td><td>-10.3</td><td>-9.9</td><td>-9.6</td><td>-9.2</td><td>-8.9</td><td>-8.0</td><td>-7.7</td><td>-6.6</td><td>-5.3</td><td>-1.9</td><td>—</td><td>0.3</td><td>1.2*</td><td>2.1</td></tr><tr><td>NLLB_MBR_BLEU</td><td>-10.7</td><td>-10.3</td><td>-9.9</td><td>-9.5</td><td>-9.2</td><td>-8.4</td><td>-8.1</td><td>-6.9</td><td>-5.7</td><td>-2.2</td><td>-0.3</td><td>—</td><td>0.8</td><td>1.7</td></tr><tr><td>AIRC</td><td>-11.5</td><td>-11.1</td><td>-10.8</td><td>-10.3</td><td>-10.0</td><td>-9.2</td><td>-8.9</td><td>-7.8</td><td>-6.5</td><td>-3.0</td><td>-1.2</td><td>-0.8</td><td>—</td><td>0.9</td></tr><tr><td>NLLB_Greedy</td><td>-12.4</td><td>-12.0</td><td>-11.7</td><td>-11.2</td><td>-10.9</td><td>-10.1</td><td>-9.8</td><td>-8.6</td><td>-7.4</td><td>-3.9</td><td>-2.1</td><td>-1.7</td><td>-0.9</td><td>—</td></tr><tr><td>score</td><td>90.3</td><td>89.9</td><td>89.6</td><td>89.1</td><td>88.8</td><td>88.0</td><td>87.7</td><td>86.5</td><td>85.3</td><td>81.8</td><td>80.0</td><td>79.6</td><td>78.8</td><td>77.9</td></tr><tr><td>rank</td><td>1-3</td><td>1-3</td><td>1-5</td><td>3-6</td><td>3-6</td><td>4-7</td><td>6-8</td><td>8-9</td><td>7-9</td><td>10-11</td><td>10-13</td><td>11-14</td><td>12-14</td><td>11-14</td></tr></tbody></table>

## English→Czech

<table><caption>Table 29: Head to head comparison for English→Czech systems</caption><thead><tr><th></th><th>Human-refA</th><th>ONLINE-W</th><th>GPT4-5shot</th><th>CUNI-GA</th><th>ONLINE-A</th><th>CUNI-DocTransformer</th><th>ONLINE-B</th><th>NLLB_MBR_BLEU</th><th>GTCOM_DLUT</th><th>CUNI-Transformer</th><th>NLLB_Greedy</th><th>ONLINE-M</th><th>ONLINE-G</th><th>ONLINE-Y</th><th>Lan-BridgeMT</th><th>LanguageX</th></tr></thead><tbody><tr><td>Human-refA</td><td>—</td><td>1.3*</td><td>3.6‡</td><td>5.0‡</td><td>5.1‡</td><td>6.0‡</td><td>6.6‡</td><td>6.8‡</td><td>7.0‡</td><td>8.0‡</td><td>8.6‡</td><td>9.7‡</td><td>10.2‡</td><td>10.4‡</td><td>10.4‡</td><td>11.3‡</td></tr><tr><td>ONLINE-W</td><td>-1.3</td><td>—</td><td>2.3‡</td><td>3.7‡</td><td>3.8‡</td><td>4.7‡</td><td>5.3‡</td><td>5.5‡</td><td>5.7‡</td><td>6.7‡</td><td>7.3‡</td><td>8.4‡</td><td>8.9‡</td><td>9.1‡</td><td>9.1‡</td><td>10.0‡</td></tr><tr><td>GPT4-5shot</td><td>-3.6</td><td>-2.3</td><td>—</td><td>1.4</td><td>1.5†</td><td>2.5*</td><td>3.0</td><td>3.2‡</td><td>3.4‡</td><td>4.4‡</td><td>5.1‡</td><td>6.1‡</td><td>6.6‡</td><td>6.8‡</td><td>6.8‡</td><td>7.7‡</td></tr><tr><td>CUNI-GA</td><td>-5.0</td><td>-3.7</td><td>-1.4</td><td>—</td><td>0.0‡</td><td>1.0‡</td><td>1.5*</td><td>1.8‡</td><td>2.0‡</td><td>3.0‡</td><td>3.6‡</td><td>4.7‡</td><td>5.1‡</td><td>5.3‡</td><td>5.4‡</td><td>6.3‡</td></tr><tr><td>ONLINE-A</td><td>-5.1</td><td>-3.8</td><td>-1.5</td><td>-0.0</td><td>—</td><td>1.0</td><td>1.5</td><td>1.7‡</td><td>1.9</td><td>2.9*</td><td>3.6‡</td><td>4.7†</td><td>5.1‡</td><td>5.3‡</td><td>5.4*</td><td>6.3‡</td></tr><tr><td>CUNI-DocTransformer</td><td>-6.0</td><td>-4.7</td><td>-2.5</td><td>-1.0</td><td>-1.0</td><td>—</td><td>0.5</td><td>0.7‡</td><td>0.9</td><td>1.9†</td><td>2.6‡</td><td>3.7‡</td><td>4.1‡</td><td>4.3‡</td><td>4.4†</td><td>5.3‡</td></tr><tr><td>ONLINE-B</td><td>-6.6</td><td>-5.3</td><td>-3.0</td><td>-1.5</td><td>-1.5</td><td>-0.5</td><td>—</td><td>0.2‡</td><td>0.4*</td><td>1.4‡</td><td>2.1‡</td><td>3.2‡</td><td>3.6‡</td><td>3.8‡</td><td>3.9‡</td><td>4.8‡</td></tr><tr><td>NLLB_MBR_BLEU</td><td>-6.8</td><td>-5.5</td><td>-3.2</td><td>-1.8</td><td>-1.7</td><td>-0.7</td><td>-0.2</td><td>—</td><td>0.2</td><td>1.2</td><td>1.9</td><td>3.0</td><td>3.4</td><td>3.6*</td><td>3.7</td><td>4.5‡</td></tr><tr><td>GTCOM_DLUT</td><td>-7.0</td><td>-5.7</td><td>-3.4</td><td>-2.0</td><td>-1.9</td><td>-0.9</td><td>-0.4</td><td>-0.2</td><td>—</td><td>1.0</td><td>1.7‡</td><td>2.8†</td><td>3.2‡</td><td>3.4‡</td><td>3.5</td><td>4.3‡</td></tr><tr><td>CUNI-Transformer</td><td>-8.0</td><td>-6.7</td><td>-4.4</td><td>-3.0</td><td>-2.9</td><td>-1.9</td><td>-1.4</td><td>-1.2</td><td>-1.0</td><td>—</td><td>0.7*</td><td>1.7</td><td>2.2†</td><td>2.4‡</td><td>2.4</td><td>3.3‡</td></tr><tr><td>NLLB_Greedy</td><td>-8.6</td><td>-7.3</td><td>-5.1</td><td>-3.6</td><td>-3.6</td><td>-2.6</td><td>-2.1</td><td>-1.9</td><td>-1.7</td><td>-0.7</td><td>—</td><td>1.1</td><td>1.5</td><td>1.7*</td><td>1.8</td><td>2.7‡</td></tr><tr><td>ONLINE-M</td><td>-9.7</td><td>-8.4</td><td>-6.1</td><td>-4.7</td><td>-4.7</td><td>-3.7</td><td>-3.2</td><td>-3.0</td><td>-2.8</td><td>-1.7</td><td>-1.1</td><td>—</td><td>0.4</td><td>0.6†</td><td>0.7</td><td>1.6‡</td></tr><tr><td>ONLINE-G</td><td>-10.2</td><td>-8.9</td><td>-6.6</td><td>-5.1</td><td>-5.1</td><td>-4.1</td><td>-3.6</td><td>-3.4</td><td>-3.2</td><td>-2.2</td><td>-1.5</td><td>-0.4</td><td>—</td><td>0.2</td><td>0.3</td><td>1.1†</td></tr><tr><td>ONLINE-Y</td><td>-10.4</td><td>-9.1</td><td>-6.8</td><td>-5.3</td><td>-5.3</td><td>-4.3</td><td>-3.8</td><td>-3.6</td><td>-3.4</td><td>-2.4</td><td>-1.7</td><td>-0.6</td><td>-0.2</td><td>—</td><td>0.1</td><td>1.0*</td></tr><tr><td>Lan-BridgeMT</td><td>-10.4</td><td>-9.1</td><td>-6.8</td><td>-5.4</td><td>-5.4</td><td>-4.4</td><td>-3.9</td><td>-3.7</td><td>-3.5</td><td>-2.4</td><td>-1.8</td><td>-0.7</td><td>-0.3</td><td>-0.1</td><td>—</td><td>0.9‡</td></tr><tr><td>LanguageX</td><td>-11.3</td><td>-10.0</td><td>-7.7</td><td>-6.3</td><td>-6.3</td><td>-5.3</td><td>-4.8</td><td>-4.5</td><td>-4.3</td><td>-3.3</td><td>-2.7</td><td>-1.6</td><td>-1.1</td><td>-1.0</td><td>-0.9</td><td>—</td></tr><tr><td>score</td><td>85.4</td><td>84.1</td><td>81.8</td><td>80.4</td><td>80.3</td><td>79.4</td><td>78.8</td><td>78.6</td><td>78.4</td><td>77.4</td><td>76.8</td><td>75.7</td><td>75.2</td><td>75.0</td><td>75.0</td><td>74.1</td></tr><tr><td>rank</td><td>1</td><td>2</td><td>3-5</td><td>3-4</td><td>5-8</td><td>5-8</td><td>4-7</td><td>8-14</td><td>6-11</td><td>8-12</td><td>10-14</td><td>9-14</td><td>10-15</td><td>13-15</td><td>8-15</td><td>16</td></tr></tbody></table>
English→German

<table><caption>Table 30: Head to head comparison for English→German systems</caption><thead><tr><th></th><th>GPT4-5shot</th><th>ONLINE-B</th><th>ONLINE-W</th><th>ONLINE-A</th><th>ONLINE-Y</th><th>Human-refA</th><th>ONLINE-M</th><th>ONLINE-G</th><th>Lan-BridgeMT</th><th>LanguageX</th><th>NLLB_MBR_BLEU</th><th>NLLB_Greedy</th><th>AIRC</th></tr></thead><tbody><tr><td>GPT4-5shot</td><td>—</td><td>0.1</td><td>0.7</td><td>0.8</td><td>1.0†</td><td>1.3</td><td>2.3‡</td><td>3.4‡</td><td>5.0‡</td><td>6.3‡</td><td>12.1‡</td><td>13.2‡</td><td>15.4‡</td></tr><tr><td>ONLINE-B</td><td>-0.1</td><td>—</td><td>0.6</td><td>0.7</td><td>0.8*</td><td>1.2</td><td>2.2‡</td><td>3.3‡</td><td>4.8‡</td><td>6.2‡</td><td>12.0‡</td><td>13.1‡</td><td>15.2‡</td></tr><tr><td>ONLINE-W</td><td>-0.7</td><td>-0.6</td><td>—</td><td>0.2*</td><td>0.3‡</td><td>0.6</td><td>1.6‡</td><td>2.7‡</td><td>4.3‡</td><td>5.6‡</td><td>11.5‡</td><td>12.5‡</td><td>14.7‡</td></tr><tr><td>ONLINE-A</td><td>-0.8</td><td>-0.7</td><td>-0.2</td><td>—</td><td>0.1</td><td>0.5</td><td>1.4‡</td><td>2.6‡</td><td>4.1‡</td><td>5.5‡</td><td>11.3‡</td><td>12.4‡</td><td>14.5‡</td></tr><tr><td>ONLINE-Y</td><td>-1.0</td><td>-0.8</td><td>-0.3</td><td>-0.1</td><td>—</td><td>0.3</td><td>1.3†</td><td>2.5*</td><td>4.0‡</td><td>5.3‡</td><td>11.2‡</td><td>12.3‡</td><td>14.4‡</td></tr><tr><td>Human-refA</td><td>-1.3</td><td>-1.2</td><td>-0.6</td><td>-0.5</td><td>-0.3</td><td>—</td><td>1.0‡</td><td>2.1‡</td><td>3.7‡</td><td>5.0‡</td><td>10.8‡</td><td>11.9‡</td><td>14.1‡</td></tr><tr><td>ONLINE-M</td><td>-2.3</td><td>-2.2</td><td>-1.6</td><td>-1.4</td><td>-1.3</td><td>-1.0</td><td>—</td><td>1.1</td><td>2.7†</td><td>4.0‡</td><td>9.9‡</td><td>10.9‡</td><td>13.1‡</td></tr><tr><td>ONLINE-G</td><td>-3.4</td><td>-3.3</td><td>-2.7</td><td>-2.6</td><td>-2.5</td><td>-2.1</td><td>-1.1</td><td>—</td><td>1.5‡</td><td>2.9‡</td><td>8.7‡</td><td>9.8‡</td><td>11.9‡</td></tr><tr><td>Lan-BridgeMT</td><td>-5.0</td><td>-4.8</td><td>-4.3</td><td>-4.1</td><td>-4.0</td><td>-3.7</td><td>-2.7</td><td>-1.5</td><td>—</td><td>1.4*</td><td>7.2‡</td><td>8.3‡</td><td>10.4‡</td></tr><tr><td>LanguageX</td><td>-6.3</td><td>-6.2</td><td>-5.6</td><td>-5.5</td><td>-5.3</td><td>-5.0</td><td>-4.0</td><td>-2.9</td><td>-1.4</td><td>—</td><td>5.8‡</td><td>6.9‡</td><td>9.1‡</td></tr><tr><td>NLLB_MBR_BLEU</td><td>-12.1</td><td>-12.0</td><td>-11.5</td><td>-11.3</td><td>-11.2</td><td>-10.8</td><td>-9.9</td><td>-8.7</td><td>-7.2</td><td>-5.8</td><td>—</td><td>1.1</td><td>3.2‡</td></tr><tr><td>NLLB_Greedy</td><td>-13.2</td><td>-13.1</td><td>-12.5</td><td>-12.4</td><td>-12.3</td><td>-11.9</td><td>-10.9</td><td>-9.8</td><td>-8.3</td><td>-6.9</td><td>-1.1</td><td>—</td><td>2.2‡</td></tr><tr><td>AIRC</td><td>-15.4</td><td>-15.2</td><td>-14.7</td><td>-14.5</td><td>-14.4</td><td>-14.1</td><td>-13.1</td><td>-11.9</td><td>-10.4</td><td>-9.1</td><td>-3.2</td><td>-2.2</td><td>—</td></tr><tr><td>score</td><td>89.0</td><td>88.8</td><td>88.3</td><td>88.1</td><td>88.0</td><td>87.7</td><td>86.7</td><td>85.5</td><td>84.0</td><td>82.7</td><td>76.8</td><td>75.7</td><td>73.6</td></tr><tr><td>rank</td><td>1-5</td><td>1-5</td><td>1-4</td><td>2-6</td><td>4-6</td><td>1-6</td><td>7-8</td><td>7-8</td><td>9</td><td>10</td><td>11-12</td><td>11-12</td><td>13</td></tr></tbody></table>
English→Japanese

<table><caption>Table 31: Head to head comparison for English→Japanese systems</caption><thead><tr><th></th><th>Human-refA</th><th>GPT4-5shot</th><th>ONLINE-B</th><th>ONLINE-Y</th><th>SKIM</th><th>ONLINE-W</th><th>LanguageX</th><th>ONLINE-A</th><th>NAIST-NICT</th><th>Lan-BridgeMT</th><th>ANVITA</th><th>ONLINE-M</th><th>KYB</th><th>AIRC</th><th>ONLINE-G</th><th>NLLB_Greedy</th><th>NLLB_MBR_BLEU</th></tr></thead><tbody><tr><td>Human-refA</td><td>—</td><td>1.2‡</td><td>1.9</td><td>2.1†</td><td>2.2*</td><td>2.3‡</td><td>4.1‡</td><td>4.5‡</td><td>4.6‡</td><td>5.5‡</td><td>7.6‡</td><td>8.1‡</td><td>9.9‡</td><td>11.1‡</td><td>11.1‡</td><td>16.2‡</td><td>19.4‡</td></tr><tr><td>GPT4-5shot</td><td>-1.2</td><td>—</td><td>0.7</td><td>0.9</td><td>1.0</td><td>1.1</td><td>2.9*</td><td>3.3†</td><td>3.4†</td><td>4.3‡</td><td>6.4‡</td><td>6.9‡</td><td>8.8‡</td><td>9.9‡</td><td>10.0‡</td><td>15.0‡</td><td>18.3‡</td></tr><tr><td>ONLINE-B</td><td>-1.9</td><td>-0.7</td><td>—</td><td>0.2</td><td>0.3</td><td>0.4*</td><td>2.3‡</td><td>2.7‡</td><td>2.7‡</td><td>3.6‡</td><td>5.7‡</td><td>6.2‡</td><td>8.1‡</td><td>9.3‡</td><td>9.3‡</td><td>14.3‡</td><td>17.6‡</td></tr><tr><td>ONLINE-Y</td><td>-2.1</td><td>-0.9</td><td>-0.2</td><td>—</td><td>0.1</td><td>0.2</td><td>2.0‡</td><td>2.4‡</td><td>2.5‡</td><td>3.4‡</td><td>5.5‡</td><td>6.0‡</td><td>7.8‡</td><td>9.0‡</td><td>9.0‡</td><td>14.1‡</td><td>17.3‡</td></tr><tr><td>SKIM</td><td>-2.2</td><td>-1.0</td><td>-0.3</td><td>-0.1</td><td>—</td><td>0.1*</td><td>1.9‡</td><td>2.3‡</td><td>2.4‡</td><td>3.3‡</td><td>5.4‡</td><td>5.9‡</td><td>7.7‡</td><td>8.9‡</td><td>8.9‡</td><td>13.9‡</td><td>17.2‡</td></tr><tr><td>ONLINE-W</td><td>-2.3</td><td>-1.1</td><td>-0.4</td><td>-0.2</td><td>-0.1</td><td>—</td><td>1.8†</td><td>2.2†</td><td>2.3‡</td><td>3.2‡</td><td>5.3‡</td><td>5.8‡</td><td>7.6‡</td><td>8.8‡</td><td>8.8‡</td><td>13.8‡</td><td>17.1‡</td></tr><tr><td>LanguageX</td><td>-4.1</td><td>-2.9</td><td>-2.3</td><td>-2.0</td><td>-1.9</td><td>-1.8</td><td>—</td><td>0.4</td><td>0.5</td><td>1.4</td><td>3.5‡</td><td>4.0‡</td><td>5.8‡</td><td>7.0‡</td><td>7.0‡</td><td>12.0‡</td><td>15.3‡</td></tr><tr><td>ONLINE-A</td><td>-4.5</td><td>-3.3</td><td>-2.7</td><td>-2.4</td><td>-2.3</td><td>-2.2</td><td>-0.4</td><td>—</td><td>0.0</td><td>1.0</td><td>3.1‡</td><td>3.5‡</td><td>5.4‡</td><td>6.6‡</td><td>6.6‡</td><td>11.6‡</td><td>14.9‡</td></tr><tr><td>NAIST-NICT</td><td>-4.6</td><td>-3.4</td><td>-2.7</td><td>-2.5</td><td>-2.4</td><td>-2.3</td><td>-0.5</td><td>-0.0</td><td>—</td><td>0.9</td><td>3.0‡</td><td>3.5‡</td><td>5.4‡</td><td>6.5‡</td><td>6.6‡</td><td>11.6‡</td><td>14.9‡</td></tr><tr><td>Lan-BridgeMT</td><td>-5.5</td><td>-4.3</td><td>-3.6</td><td>-3.4</td><td>-3.3</td><td>-3.2</td><td>-1.4</td><td>-1.0</td><td>-0.9</td><td>—</td><td>2.1†</td><td>2.6‡</td><td>4.5‡</td><td>5.6‡</td><td>5.6‡</td><td>10.7‡</td><td>14.0‡</td></tr><tr><td>ANVITA</td><td>-7.6</td><td>-6.4</td><td>-5.7</td><td>-5.5</td><td>-5.4</td><td>-5.3</td><td>-3.5</td><td>-3.1</td><td>-3.0</td><td>-2.1</td><td>—</td><td>0.5</td><td>2.3‡</td><td>3.5‡</td><td>3.5‡</td><td>8.5‡</td><td>11.8‡</td></tr><tr><td>ONLINE-M</td><td>-8.1</td><td>-6.9</td><td>-6.2</td><td>-6.0</td><td>-5.9</td><td>-5.8</td><td>-4.0</td><td>-3.5</td><td>-3.5</td><td>-2.6</td><td>-0.5</td><td>—</td><td>1.9†</td><td>3.0‡</td><td>3.1†</td><td>8.1‡</td><td>11.4‡</td></tr><tr><td>KYB</td><td>-9.9</td><td>-8.8</td><td>-8.1</td><td>-7.8</td><td>-7.7</td><td>-7.6</td><td>-5.8</td><td>-5.4</td><td>-5.4</td><td>-4.5</td><td>-2.3</td><td>-1.9</td><td>—</td><td>1.2</td><td>1.2</td><td>6.2‡</td><td>9.5‡</td></tr><tr><td>AIRC</td><td>-11.1</td><td>-9.9</td><td>-9.3</td><td>-9.0</td><td>-8.9</td><td>-8.8</td><td>-7.0</td><td>-6.6</td><td>-6.5</td><td>-5.6</td><td>-3.5</td><td>-3.0</td><td>-1.2</td><td>—</td><td>0.0</td><td>5.0‡</td><td>8.3‡</td></tr><tr><td>ONLINE-G</td><td>-11.1</td><td>-10.0</td><td>-9.3</td><td>-9.0</td><td>-8.9</td><td>-8.8</td><td>-7.0</td><td>-6.6</td><td>-6.6</td><td>-5.6</td><td>-3.5</td><td>-3.1</td><td>-1.2</td><td>-0.0</td><td>—</td><td>5.0‡</td><td>8.3‡</td></tr><tr><td>NLLB_Greedy</td><td>-16.2</td><td>-15.0</td><td>-14.3</td><td>-14.1</td><td>-13.9</td><td>-13.8</td><td>-12.0</td><td>-11.6</td><td>-11.6</td><td>-10.7</td><td>-8.5</td><td>-8.1</td><td>-6.2</td><td>-5.0</td><td>-5.0</td><td>—</td><td>3.3‡</td></tr><tr><td>NLLB_MBR_BLEU</td><td>-19.4</td><td>-18.3</td><td>-17.6</td><td>-17.3</td><td>-17.2</td><td>-17.1</td><td>-15.3</td><td>-14.9</td><td>-14.9</td><td>-14.0</td><td>-11.8</td><td>-11.4</td><td>-9.5</td><td>-8.3</td><td>-8.3</td><td>-3.3</td><td>—</td></tr><tr><td>score</td><td>80.7</td><td>79.5</td><td>78.8</td><td>78.6</td><td>78.5</td><td>78.4</td><td>76.6</td><td>76.2</td><td>76.1</td><td>75.2</td><td>73.1</td><td>72.6</td><td>70.8</td><td>69.6</td><td>69.6</td><td>64.5</td><td>61.3</td></tr><tr><td>rank</td><td>1-2</td><td>2-6</td><td>1-5</td><td>2-6</td><td>2-5</td><td>4-6</td><td>7-10</td><td>7-10</td><td>7-10</td><td>7-10</td><td>11-12</td><td>11-12</td><td>13-15</td><td>13-15</td><td>13-15</td><td>16</td><td>17</td></tr></tbody></table>
English→Chinese

<table><caption>Table 32: Head to head comparison for English→Chinese systems</caption><thead><tr><th></th><th>Yishu</th><th>Human-refA</th><th>GPT4-5shot</th><th>Lan-BridgeMT</th><th>ONLINE-B</th><th>HW-TSC</th><th>ONLINE-W</th><th>ONLINE-Y</th><th>IOL_Research</th><th>ONLINE-A</th><th>LanguageX</th><th>ONLINE-M</th><th>ONLINE-G</th><th>ANVITA</th><th>NLLB_Greedy</th><th>NLLB_MBR_BLEU</th></tr></thead><tbody><tr><td>Yishu</td><td>—</td><td>0.0</td><td>0.1</td><td>0.2*</td><td>0.3</td><td>0.7</td><td>0.8*</td><td>2.0*</td><td>2.3‡</td><td>2.5‡</td><td>3.6‡</td><td>4.0‡</td><td>5.0‡</td><td>17.7‡</td><td>17.9‡</td><td>25.0‡</td></tr><tr><td>Human-refA</td><td>-0.0</td><td>—</td><td>0.0</td><td>0.1†</td><td>0.3</td><td>0.7</td><td>0.8*</td><td>1.9†</td><td>2.3‡</td><td>2.5‡</td><td>3.6‡</td><td>3.9‡</td><td>5.0‡</td><td>17.7‡</td><td>17.8‡</td><td>25.0‡</td></tr><tr><td>GPT4-5shot</td><td>-0.1</td><td>-0.0</td><td>—</td><td>0.1</td><td>0.3</td><td>0.6</td><td>0.7</td><td>1.9*</td><td>2.3‡</td><td>2.4‡</td><td>3.5‡</td><td>3.9‡</td><td>5.0‡</td><td>17.6‡</td><td>17.8‡</td><td>24.9‡</td></tr><tr><td>Lan-BridgeMT</td><td>-0.2</td><td>-0.1</td><td>-0.1</td><td>—</td><td>0.2</td><td>0.5</td><td>0.6</td><td>1.8</td><td>2.2†</td><td>2.3‡</td><td>3.4‡</td><td>3.8‡</td><td>4.9‡</td><td>17.5‡</td><td>17.7‡</td><td>24.8‡</td></tr><tr><td>ONLINE-B</td><td>-0.3</td><td>-0.3</td><td>-0.3</td><td>-0.2</td><td>—</td><td>0.3</td><td>0.4†</td><td>1.6†</td><td>2.0‡</td><td>2.2‡</td><td>3.2‡</td><td>3.6‡</td><td>4.7‡</td><td>17.3‡</td><td>17.5‡</td><td>24.7‡</td></tr><tr><td>HW-TSC</td><td>-0.7</td><td>-0.7</td><td>-0.6</td><td>-0.5</td><td>-0.3</td><td>—</td><td>0.1</td><td>1.3</td><td>1.7‡</td><td>1.8‡</td><td>2.9‡</td><td>3.3‡</td><td>4.4‡</td><td>17.0‡</td><td>17.2‡</td><td>24.3‡</td></tr><tr><td>ONLINE-W</td><td>-0.8</td><td>-0.8</td><td>-0.7</td><td>-0.6</td><td>-0.4</td><td>-0.1</td><td>—</td><td>1.2</td><td>1.6†</td><td>1.7‡</td><td>2.8‡</td><td>3.2‡</td><td>4.3‡</td><td>16.9‡</td><td>17.1‡</td><td>24.2‡</td></tr><tr><td>ONLINE-Y</td><td>-2.0</td><td>-1.9</td><td>-1.9</td><td>-1.8</td><td>-1.6</td><td>-1.3</td><td>-1.2</td><td>—</td><td>0.4*</td><td>0.5†</td><td>1.6‡</td><td>2.0‡</td><td>3.1‡</td><td>15.7‡</td><td>15.9‡</td><td>23.0‡</td></tr><tr><td>IOL_Research</td><td>-2.3</td><td>-2.3</td><td>-2.3</td><td>-2.2</td><td>-2.0</td><td>-1.7</td><td>-1.6</td><td>-0.4</td><td>—</td><td>0.2</td><td>1.2‡</td><td>1.6‡</td><td>2.7‡</td><td>15.3‡</td><td>15.5‡</td><td>22.7‡</td></tr><tr><td>ONLINE-A</td><td>-2.5</td><td>-2.5</td><td>-2.4</td><td>-2.3</td><td>-2.2</td><td>-1.8</td><td>-1.7</td><td>-0.5</td><td>-0.2</td><td>—</td><td>1.1*</td><td>1.5*</td><td>2.5‡</td><td>15.2‡</td><td>15.4‡</td><td>22.5‡</td></tr><tr><td>LanguageX</td><td>-3.6</td><td>-3.6</td><td>-3.5</td><td>-3.4</td><td>-3.2</td><td>-2.9</td><td>-2.8</td><td>-1.6</td><td>-1.2</td><td>-1.1</td><td>—</td><td>0.4</td><td>1.5</td><td>14.1‡</td><td>14.3‡</td><td>21.4‡</td></tr><tr><td>ONLINE-M</td><td>-4.0</td><td>-3.9</td><td>-3.9</td><td>-3.8</td><td>-3.6</td><td>-3.3</td><td>-3.2</td><td>-2.0</td><td>-1.6</td><td>-1.5</td><td>-0.4</td><td>—</td><td>1.1</td><td>13.7‡</td><td>13.9‡</td><td>21.0‡</td></tr><tr><td>ONLINE-G</td><td>-5.0</td><td>-5.0</td><td>-5.0</td><td>-4.9</td><td>-4.7</td><td>-4.4</td><td>-4.3</td><td>-3.1</td><td>-2.7</td><td>-2.5</td><td>-1.5</td><td>-1.1</td><td>—</td><td>12.6‡</td><td>12.8‡</td><td>20.0‡</td></tr><tr><td>ANVITA</td><td>-17.7</td><td>-17.7</td><td>-17.6</td><td>-17.5</td><td>-17.3</td><td>-17.0</td><td>-16.9</td><td>-15.7</td><td>-15.3</td><td>-15.2</td><td>-14.1</td><td>-13.7</td><td>-12.6</td><td>—</td><td>0.2‡</td><td>7.3‡</td></tr><tr><td>NLLB_Greedy</td><td>-17.9</td><td>-17.8</td><td>-17.8</td><td>-17.7</td><td>-17.5</td><td>-17.2</td><td>-17.1</td><td>-15.9</td><td>-15.5</td><td>-15.4</td><td>-14.3</td><td>-13.9</td><td>-12.8</td><td>-0.2</td><td>—</td><td>7.1‡</td></tr><tr><td>NLLB_MBR_BLEU</td><td>-25.0</td><td>-25.0</td><td>-24.9</td><td>-24.8</td><td>-24.7</td><td>-24.3</td><td>-24.2</td><td>-23.0</td><td>-22.7</td><td>-22.5</td><td>-21.4</td><td>-21.0</td><td>-20.0</td><td>-7.3</td><td>-7.1</td><td>—</td></tr><tr><td>score</td><td>82.2</td><td>82.1</td><td>82.1</td><td>82.0</td><td>81.8</td><td>81.5</td><td>81.4</td><td>80.2</td><td>79.8</td><td>79.7</td><td>78.6</td><td>78.2</td><td>77.1</td><td>64.5</td><td>64.3</td><td>57.2</td></tr><tr><td>rank</td><td>1-5</td><td>1-5</td><td>1-7</td><td>3-8</td><td>1-6</td><td>1-8</td><td>4-8</td><td>5-8</td><td>9-10</td><td>9-10</td><td>11-13</td><td>11-13</td><td>11-13</td><td>14</td><td>15</td><td>16</td></tr></tbody></table>